Pix2seq: A Language Modeling Framework for Object Detection
Authors: Ting Chen
Saurabh Saxena
Lala Li
David J. Fleet
Geoffrey Hinton
Venue: ICLR 2022 Poster (2022)
Keywords: language modeling
object detection

Abstract: We present Pix2Seq, a simple and generic framework for object detection. Unlike existing approaches that explicitly integrate prior knowledge about the task, we cast object detection as a language modeling task conditioned on the observed pixel inputs. Object descriptions (e.g., bounding boxes and class labels) are expressed as sequences of discrete tokens, and we train a neural network to perceive the image and generate the desired sequence. Our approach is based mainly on the intuition that if a neural network knows about where and what the objects are, we just need to teach it how to read them out. Beyond the use of task-specific data augmentations, our approach makes minimal assumptions about the task, yet it achieves competitive results on the challenging COCO dataset, compared to highly specialized and well optimized detection algorithms.

Published as a conference paper at ICLR 2022
PIX2SEQ: A LANGUAGE MODELING FRAMEWORK FOR OBJECT DETECTION
Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, Geoffrey Hinton Google Research, Brain Team
ABSTRACT
We present Pix2Seq, a simple and generic framework for object detection. Unlike existing approaches that explicitly integrate prior knowledge about the task, we cast object detection language modeling task conditioned on the observed pixel inputs. Object descriptions (e.g., bounding boxes and class labels) are expressed sequences of discrete tokens, and train neural network to perceive the image and generate the desired sequence. Our approach is based mainly on the intuition that if a neural network knows about where and what the objects are, we just need to teach it how to read them out. Beyond the use of task-specific data augmentations,
approach makes minimal assumptions about the task, yet it achieves competitive results on the challenging COCO dataset, compared to highly specialized and well optimized detection algorithms
ye., Yeu you 58
Pix25eq
Yru Kewed you Yew 97 motacycle
Cmdd:
K7g Feu 99 72 Person
Figure I: Illustration of Pix2Seq framework for object detection. The neural net perceives an image and generates sequence of tokens that correspond bounding boxes and class labels.
INTRODUCTION
Visual object detection systems aim to recognize and localize all objects of pre-defined categories in an image. The detected objects are typically described by a set of bounding boxes and associated class labels. Given the difficulty of the task, most existing methods, such (Girshick) 2015 Ren cet al. 2015 He et al. 20177 Lin et al.) 2017b Carion et al. 2020), are carefully designed and highly customized, with a significant amount of prior knowledge in the choice of architecture and loss function. For example, many architectures are tailored to the use of bounding boxes (c.g., with region proposals (Girshick 2015 Ren et al.) 2015) and Rol pooling (Girshick et al. 2014 He et al.; (017)). Others are tied to the use of object queries for object binding (Carion et al.] 2020). Loss functions are often similarly tailored the of bounding boxes, such box regression (Szegedy al. 2013
Lin et al. 20177)), set-based matching (Erhan et al. 2014 Carion et al. 2020). or by incorporating
Correspondence to: amt i ngchen@@poogl..ccm
"Code and checkpoints available at http://ggthaccom//ggg--sse//////

---

Published as a conference paper at ICLR 2022
specific performance metrics, like intersection-over-uiiiee 2019). Although existing systems find upplications in myriad domains, from self-driving cars (Sun et al. 2020), medical image analysis (Jaeger al. 2020), agriculture (Sa et al.) 2016), the specialization and complexity make them difficult to integrate into a larger system, or generalize to a much broader array of tasks associated with general intelligence.
bounding boxes (Rezatofighi al.
This paper advocates a new approach, based on the intuition that if a neural net knows about where and what the objects are, we just need to teach it read them out. And by learning to "describe" objects the model can learn to ground the "language" on pixel observations, leading to useful object representations. This is realized with our Pix2Seq framework (see Figure Tp. Given an image. our model produces a sequence of discrete tokens that correspond to object descriptions (c.g., object bounding boxes and class labels), reminiscent an image captioning system (Vinyals et al. 501 Karpathy & Fei-Fei 20155 Xu et al. 2015). In essence, we cast object detection as language modeling task conditioned on pixel inputs, for which the model architecture and loss function are generic and relatively simple, without being engineered specifically for the detection task. As such, one can readily extend the framework to different domains or applications, or incorporate it into perceptual system supporting general intelligence, for which it provides language interface to wide range of vision tasks.
To tackle the detection task with Pix2Seq, first propose a quantization and serialization scheme that converts bounding boxes and class labels into sequences of discrete tokens. We then leverage an encoder-decoder architecture for perceiving pixel inputs and generating the target sequence. The objective function is simply the maximum likelihood of tokens conditioned on pixel inputs and the preceding tokens. While both the architecture and loss function are task-agnostic (without assuming prior knowledge about object detection, e.g., bounding boxes), we can still incorporate task-specific prior knowledge with sequence augmentation technique, proposed below, that alters both input and target sequences during training. Through extensive experimentation, we demonstrate that this simple Pix2Seq framework can achieve competitive results on the COCO dataset compared to highly customized, well established approaches, including Faster R-CNN (Ren et al., (015) and DETR (Carion et al. (2020). By pretraining our model on a larger object detection dataset, its performance can be further improved.
THE PIX2SEQ FRAMEWORK
In the proposed Pix2Seq framework we cast object detection as language modeling task, conditioned on pixel inputs (Figure I. The system consists of four main components (Figure 2):
Image Augmentation: As is common in training computer vision models, we use image augmenta- tions to enrich fixed set of training examples (c.g., with random scaling and crops).
Sequence construction & augmentation: As object annotations for an image are usually represented as set of bounding boxes and class labels, we convert them into sequence of discrete tokens.
Architecture: We use encoder-decoder model, where the encoder perceives pixel inputs, and the decoder generates the target sequence (one token at a time).
Objective/loss function: The model is trained to maximize the log likelihood of tokens conditioned on the image and the preceding tokens (with softmax cross-entropy loss).
Image Augmentation
Image
Ypred
with Causal mask
Encoder
Bbox, class
Sequence
Construction & Augmentation
Yurge:
Objective
(next taken prediction)
Figure 2: Major components of the Pix2Seq learning framework.
2

---

Published as a conference paper at ICLR 2022
2.1 SEQUENCE CONSTRUCTION FROM OBJECT DESCRIPTIONS
In common object detection datasets, such as Pascal VOC (Everingham et al. 2010). COCO (Lin al., 2014), and Openlmages (Kuznetsova et al. 2020), images have variable numbers of objects. represented as sets of bounding boxes and class labels. In Pix2Seq we express them as sequences of discrete tokens.
While class labels are naturally expressed as discrete tokens, bounding boxes are not. A bounding box determined by two of its corner points (i.e., top-left and bottom-right), by its center point plus height and width. We propose to discretize the continuous numbers used to specify the coordinates of corner points (similarly for height and width if the other box format used). Specifically, an object is represented as a sequence of five discrete tokens, i.e. [Jmiin: <mii:: /max <mmxx cl. where each of the continuous corner coordinates is uniformly discretized into an integer between [1. Nains) and is the class index. We use shared vocabulary for all tokens, so the vocabulary size is equal to number of bins + number of classes. This quantization scheme for the bounding boxes allows us to small vocabulary while achieving high precision. For example, a x60 image requires only 600 bins to achieve zero quantization error. This is much smaller than modern language models with vocabulary sizes of 32K higher (Radford et al.) 2018 Devlin et al., (2018). The effect of different levels of quantization on the placement of bounding boxes is illustrated in Figure
With each object description expressed short discrete sequence, we need to serialize multiple object descriptions to form a single sequence for given image. Since order of objects does not matter for the detection task per se, we use a random ordering strategy (randomizing the order objects each time an image is shown). We also explore other deterministic ordering strategies, but we hypothesize that random ordering will work just as well as any deterministic ordering, given capable neural net and autoregressive modeling (where the net can learn to model the distribution of remaining objects conditioned on those observed).
Finally, because different images often have different numbers of objects, the generated sequences will have different lengths. To indicate the end of a sequence, we therefore incorporate an EOS token. The sequence construction process with different ordering strategies is illustrated in Figure
(a) Tibins 10
(b) Thbin 50
(c) Tibins 100
(d) 71bin 500
(e) Original
Figure 3: Applying the proposed discritization of bounding box image of 480 x 640. Only quarter of the image is shown for better clarity. With small number of bins, such as 500 bins (~1 pixel/bin), it achieves high precision even for small objects.
Randem ordering (multiple samples): sur sro 2003
330 9ay 358 1004
508
892
$64 135 987 338 1004
322 570 653 664 5001
508
518
605
892 1004
508 518 905 292 1004
544 155
987
538 1004
537
370
655
444 1001
ordering:
$66 135 987 338 1004 508 518
856 892 1004
327 370
655 444 1001
DisiZari ordering
544 325 1004 $70 655 444 508 892 1004
Figure 4: Examples of sequence construction with Phpins 1000, and 0 is EOS token.
3

---

Published as a conference paper at ICLR 2022
2.2 ARCHITECTURE, OBJECTIVE AND INFERENCE
Treating the sequences that we construct from object descriptions as a "dialect", we turn to generic architectures and objective functions that have been effective in language modeling.
Architecture We use an encoder-decoder architecture. The encoder can be general image encoder that perceives pixels and encodes them into hidden representations, such as a ConvNet (LeCun et 1989, Krizhevsky et al. 2012 He et al., 2016)) Transformer (Vaswani et al. 2017 Dosovitskiy et al. 2020), or their combination (Carion et al. (020). For generation we use a Transformer decoder, widely used in modern language modeling (Radford et al.) 2018 Raffel et al. 2019). It generates one token at time, conditioned on the preceding tokens and the encoded image representation. This removes the complexity and customization in architectures of modern object detectors, e.g., bounding box proposal and regression, since tokens are generated from a single vocabulary with a softmax.
Objective Similar language modeling, Pix2Seq is trained to predict tokens, given an image and preceding tokens, with a maximum likelihood loss, i.e.,
maximize Σ w log P(új x. V1-5-1)
(1)
where æ is a given image, and ÿ are input and target sequences associated with æ, and L is the target sequence length. y and ÿ are identical in the standard language modeling setup, but they can also be different (as in our later augmented sequence construction). Also, wy is pre-assigned weight for j-th token in the sequence. We set w, 1, Vj. however it would be possible to weight tokens by their types (e.g., coordinate vs class tokens), or by the size of the corresponding object.
Inference At inference time, we sample tokens from model likelihood, i.e., P(y, læ, y1g--)) This can be done by either taking the token with the largest likelihood (arg max sampling), or using other stochastic sampling techniques. We find that using nucleus sampling (Holtzman et al. 2019) leads to higher recall than arg max sampling (Appendix C). The sequence ends when the EOS token is generated. Once the sequence is generated, it is straight-forward to extract and de-quantize the object descriptions (i.e., obtaining the predicted bounding boxes and class labels).
2.3 SEQURNCE AUGMENTATION TO INTEGRATE TASK PRIORS
The EOS token allows the model to decide when to terminate generation, but in practice we find that the model tends to finish without predicting all objects. This is likely due to 1) annotation noise (e.g.. where annotators did not identify all the objects), and 2) uncertainty in recognizing or localizing some objects. While this only affects the overall performance by a small percentage (e.g., 1-2% in average precision), it has a larger effect recall. To encourage higher recall rates, one trick is to delay the sampling of the EOS token by artificially decreasing its likelihood. However, this often leads to noisy and duplicated predictions. In part, this difficult trade-off between precision and recall a consequence of our model being task agnostic, unaware of the detection task per se.
To mitigate the problem simply introduce sequence augmentation technique, thereby incorporat- ing prior knowledge about the task. The target sequence ý in conventional autoregressive language modeling (i.e., with sequence augmentation) is the same as the input sequence y. And all tokens in sequence are real (e.g., converted from human annotations). With sequence augmentation, we instead augment input sequences during training include both real and synthetic noise tokens. We also modify target sequences so that the model can learn to identify the noise tokens rather than mimic them. This improves the robustness of the model against noisy and duplicated predictions (particularly when the EOS token is delayed to increase recall). The modifications introduced by sequence augmentation are illustrated in Figure 5 and detailed below.
Altered sequence construction We first create synthetic noise objects to augment input sequences in the following two ways: 1) adding noise to existing ground-trute truth objects (e.g., random scaling or shifting their bounding boxes), and 2) generating completely random boxes (with randomly associated class labels). It is worth noting that some of these noise objects may be identical to, or overlapping with, some of the ground-truth objects, simulating noisy and duplicated predictions, as demonstrated
4

---

Published as a conference paper at ICLR 2022
Target sequence
Yio
end
Input Sequence (reat only)
Ya
3/2
Y/
V6
y7
(a) Conventional autoregressive langage modeling
Target sequence
Yae
n/a
n/a
n/aa
n//a
n/aa
n/a
n/aa
n/a
Input sequence (real + noise)
Ya2
(b) Langage modeling with sequence augmentation (e.g. adding noise tokens)
Figure 5: Illustration of language modeling with without sequence augmentation. With sequence augmentation, input tokens are constructed to include both real objects (blue) and synthetic noise objects (orange). For the noise objects, the model is trained to identify them as the "noise" class, and we set the loss weight of "n/a" tokens (corresponding coordinates of noise objects) zero since we do not want the model to mimic them.
Figure 6: Illustrations of randomly sampled noise objects (in white), vs. ground ground-truth truth objects (in red).
in Figure 6 After noise objects are synthesised and discretized, we then append them in the end of the original input sequence. As for the target sequence, we set the target tokens of noise objects to "noise" class (not belonging any of the ground-trute class labels), and the coordinate tokens of noise objects to "n/a", whose loss weights are set to zero, i.c., setting wj = #"aa"" in Eq [
able substantially delay the EOS token, improving recall without increasing the frequency of noisy and duplicated predictions. Thus, we let the model predict maximum length, yielding fixed-sized list of objects. When we extract the list of bounding boxes and class labels from the generated sequences, we replace the "noise" class label with a real class label that has the highest likelihood among all real class labels. We use the likelihood of the selected class token as (ranking) score for the object.
Altered inference With sequence augmentation,
3 EXPERIMENTS
3.1 EXPERIMENTAL. SETUP
We evaluate the proposed method on the MS-COCO 2017 detection dataset (Lin et al. 2014). containing 118k training images and 5k validation images. To compare with DETR and Faster R-CNN, we report average precision (AP), an integral metric over multiple thresholds, on validation set at the last training epoch. We employ two training strategies: 1) training from scratch on COCO in order to compare fairly with the baselines, and also 2) pretraining wvetraining-finetuniie +finetuning. i.e., pretrain the Pix2Seq model finetune the model on COCO. Since our approach incorporates zero inductive bias prior knowledge of the object detection task, expect the second training strategy to be superior.
larger object detection dataset, namely Objects365 (Shao et al. 2019), and then

---

Published as a conference paper at ICLR 2022
Table I: Comparison of average precision, over multiple thresholds and object sizes, on COCO validation set. Each section compares different methods of the similar ResNet "backbone". Our models achieve competitive results both Faster R-CNN and DETR baselines.
Method
Backbone
#params
AP
APso
AP1s
APs
APM
APi.
Faster R-CNN
R50-FPN
42M
40.2
61.0
43.8
24.2
43.5
52.0
Faster R-CNN+
R50-FPN
42M
42.0
62.1
45.5
26.6
45.4
53.4
DETR
R50
41M
42.0
62.4
44.2
20.5
45.8
Pix2seq (Ours)
R50
37M
43.0
61.0
45.6
25.1
46.9
59.4
Faster R-CNN
RI01-FPN
60M
42.0
62.5
45.9
25.2
45.6
54.6
Faster R-CNN+
R101-FPN
60M
44.0
63.9
47.8
27.2
48.1
56.0
DETR
R101
60M
43.5
63.8
46.4
21.9
48.0
61.8
Pix2seg (Ours)
R101
56M
44.5
62.8
47.5
26.0
48.2
60.3
Faster R-CNN
R50-DC5
166M
39.0
60.5
42.3
21.4
43.5
52.5
Faster R-CNN+
R50-DC5
166M
41.1
61.4
44.3
22.9
45.9
55.0
DETR
R50-DC5
41M
43.3
45.9
22.5
47.3
Pix2seq (Ours)
R50-DC5
38M
43.2
61.0
46.1
26.6
47.0
58.6
DETR
R101-DC5
60M
44,9
64.7
47.7
23.7
49.5
62.3
Pix2seq (Ours)
R101-DCS
57M
45.0
63.2
48.6
28.2
48.9
60.4
For training from scratch, we follow (Carion et al. 2020) using ResNet backbone (He et al. 20166). followed by layers of transformer encoder and layers of (causal) transformer decoder (Vaswani et al. 2017). We resize images (with fixed aspect ratio) the longer side is 1333 pixels. For sequence construction, we use 2000 quantization bins, and we randomize the order objects every time image is shown. We append noise objects real objects such that each image contains 100 objects in total, and hence a sequence length of 500. The model is trained for 300 epochs with batch size of 128.
For pretraining on Objects365 dataset, we use similar settings as above with a few differences. Notably, instead of using the large 1333 1333 image size, we use smaller image size of 640 640, and pretrain the models for 400K steps with batch size of 256. It is worth noting that this pretraining process is even faster than training from scratch due to the use of smaller image size. During the fine- tuning on COCO dataset, only a small number of epochs (e.g., 20 to 60 epochs) are needed to achieve good results. And we could use larger image size during fine-tuning as well. Due to the use of larger pretraining dataset, we also experiment with larger models with Vision Transformers (Dosovitskiy al. 2020).
More details for both training strategies can be found in Appendix B As for ablations, we use ResNet-101 101 backbone with a smaller image size (the longer side 640), and we train the model from scratch for 200 epochs.
3.2 MAIN COMPARISONS
Training from scratch on COCO We mainly compare with two widely recognized baselines: DETR and Faster R-CNN. DETR and our model have comparable architectures, but our Transformer decoder does require learned "object queries" or separated heads for box regression and classifi- cation, since our model generates different types of tokens (e.g., coordinate and class tokens) with single softmax. Faster R-CNN well established method, with optimized architectures such as caature-pyramic networks (FPN) (Lin et al., 2017a). Faster R-CNN is typically trained in fewer epochs than DETR or our model, likely because it explicitly incorporates prior knowledge of the task in the architecture itself. Thus we also include an improved Faster R-CNN baseline, denoted as Faster R-CNN+, from (Carion et al.) (2000), where Faster R-CNN models are trained with the GloU loss (Rezatofighi et al. 2019). train-time random crop augmentations, and the long 9x training schedule.
Results are shown in Table I where each section compares different methods of the same ResNet "backbone". Overall, Pix2Seq achieves competitive results to both baselines. Our model performs comparably to Faster R-CNN small and medium objects, but better larger objects. Compared

---

Published as a conference paper at ICLR 2022
Table 2: Average precision of finetuned Pix2seq models on COCO with different backbone archi- tectures and image sizes. All models are pretrained on Objects365 dataset. As comparison, our best model without pretraining obtains 45.0 AP (in Table I with image size of 1333 1333. The pretraining is with 640 image size while fine-tuning (a few epochs) can use larger image sizes.
Image size during finetuning
Backbone
# params
640x640
1024x 1024
1333x
1333
R50
37M
39.1 1
41.7
42.6
R50-C4
85M
44.7
46.9
47.3
VIT-B
115M
44.2
46.5
47.1
VIT-L
341M
47.6
49.0
50.0
with DETR, model performs comparably slightly worse large and medium objects, but substantially better (4-5 AP) on small objects.
Pretrain on Objects365 and finetune on COCO As shown in Table 2 the performances of Objects365 pretrained Pix2Seq models are strong across various model sizes and image sizes. The best performance (with 1333 image size) is 50 AP which is 5% higher than the best model trained from scratch, and the performance holds up very well even with 640 image size. Notably, with a smaller image size used for pretraining, the pretrain+finetuee process is faster than training from scratch, and also generalizes better. Both factors are crucial for training larger and better models.
3.3 ABLATION SEQUENCE CONSTRUCTION
Figure 7 explores the effect of coordinate quantization on performance. For this ablation we consider images the longest size of which is 640 pixels. The plot indicates that quantization to 500 bins or more is sufficient; with 500 bins there are approximately 1.3 pixels per bin, which does not introduce significant approximation error. Indeed, long as has many bins the number of pixels (along the longest side of the image) there should be no significant error due to quantization of the bounding box coordinates.
We also consider different object ordering strategies in sequence construction during training. These include 1) random, 2) area (i.e., descending object size), 3) dist2ori (i.e., the distance of top-left corner of the bounding box to the origin), 4) class (name). 5) class + area (i.e., the objects are first ordered by their class, and there are multiple objects of the same class, they are ordered by area), and 6) class + dist2ori. Figure [b) shows average precision (AP) and Figure shows average recall (AR) at the top- 100 predictions. Both in terms of precision and recall, the random ordering yields the best performance. We conjecture that with deterministic ordering, it may be difficult for the model to recover from mistakes of missing objects made earlier on, while with random ordering it would still be possible to retrieve them later.
40.0
40
60
37.5
50
35.0
39
32.5
$ 300
03
275
34
25.0
225
200
100
209
509
$609
2006
5000
30
Randon
Ars
Dacoon
Casa
Class + Area
Class Diarzon
30
Random
Area
Diszzn
Class
Clasa Area
Clas Diazoni
Quantizadi bins
(a)
Figure 7: Ablations on sequence construction. (a) Quantization bins vs. performance. (b) and (c) show AP and AR @ 100 for different object ordering strategies.
7

---

Published as a conference paper at ICLR 2022
3.4 ABLATION SEQUENCE AUGMENTATION
Here study the impact of sequence augmentation (i.e., adding the noise objects) for both model training strategies: 1) training from scratch on COCO, and 2) pretraining on Objects365 and finetuning on COCO. Results for training from scratch w/wo sequence augmentation are shown Figure and we find that without sequence augmentation, the AP is marginally worse if one delays the sampling of EOS token during the inference (via likelihood offsetting). but the recall is significantly worse for the optimal AP. Table shows similar results for meetraining-finttniini setting (where we set loss weight 0.1 on ending token instead of tuning their likelihood offset), and we find that AP is not significantly affected while recall is significanty worse without sequence augmentation. It is also worth noting that sequence augmentation is mainly effective during the fine-tuning.
SeqAug in Pretrain
SeqAug in Finetune
AP
@R@ 100
Averago Prectsion
39.2 AP
X
x
43.7
55.4
38
37.6
X
/
44.5
61.6
/
/
44.7
61.7
36
We SegAug SeqAug
Table 3: Impact sequence augmentation when pretraining Objects365 and finetuning COCO. Sequence augmentation has a major im- pact average recall @ 100) but smaller influ- ence on AP. Most improvements can be achieved during fine-tuning.
34
45.0
47.5
50.0
52.5
55.0
57.5
Average
@p100
Figure 8: Impact of sequence augmentation on when training from scratch on COCO.
3.5 VISUALIZATION OF DECODER'S 'S CROSS ATTENTION MAP
When generating new token, the transformer decoder uses self attention over the preceding tokens and cross attention over the encoded visual feature map. Here we visualize the cross attention (averaged over layers and heads) as the model predicts new token. Figure shows cross attention maps as the first few tokens are generated. One can see that the attention is very diverse when predicting the first coordinate token (i.e Ymin), but then quickly concentrates and fixates the object.
(a)
(b)
(c)
Figure 9: Decoder's cross attention to visual feature map when predicting the first objects. (b) we reshape prediction sequence of 25 into a 5x5 grid, so each row represents prediction for tokens [ymin Tmin Fmax maxx c). The attention diverse when selecting the first token of the object, then quickly concentrates the object. (c) Overlay of the cross attention (when predicting the class token) on the original image.
8

---

Published as a conference paper at ICLR 2022
4 RELATED WORK
Object detection. Existing object detection algorithms incorporate explicit prior knowledge about the task in their choice of architecture and loss function. To predict set of bounding boxes, architectures of modern detectors are specifically designed to produce a large set of proposals (Girshick) 2015 Ren
al. 2015 Cai & Vasconcelos, (0188), anchors (Lin et al. 2017b), or window centers (Tian et al. 2019 Zhou et 2019). Non-maximun suppression (Bodla et al. 2017) is often required to prevent duplicate predictions. While DETR (Carion al. 2020) avoids sophisticated bounding box proposals and non-maximum suppression, it still requires set of learned "object queries", specially for object binding. These detectors all require sub-networks (or extra layers) separately for regressing bounding boxes and class labels. Pix2Seq avoids such complexities by having generic image encoder and sequence decoder, with a single softmax for producing coordinate tokens and class labels.
Beyond architectures, the loss functions of existing detectors are also highly tailored for matching bounding boxes. For example, the loss function is often based on bounding box regression (Szegedy et al., 2013: Lin et al. 2017b), intersection over union (Rezatofighi
al., 2019), and set-
matching (Erhan et al., 2014; Liu et al.] 2016 Redmon et al./ [016]] (Stewart et al.] 2016: Carion
al., 2020). Pix2Seq avoids specialized losses, showing that a straightforward maximum likelihood objective with softmax cross entropy can work well.
Our work is also related to recurrent models in object detection (Stewart al. 2016 Park & Berg. 2015; Comera--Parede & Torr 2016; Salvador et al. 2017 Ren & Zemel 2017). in which the system learns to predict one object at time. As above, both architecture and loss functions in these approaches are often tailored to the detection task. Furthermore, these approaches are not based on Transformers, and have not been evaluated against modern baselines on larger datasets.
Language modeling. Our work is inspired by recent success of modern language modeling (Radford) al. 2019; Raffel al., 2019; [Brown et al.) 2020). Although originally intended for natural languages, the underlying methodology has been shown capable of modeling various sequential data, such as machine translation (Sutskever et al. (0144) Bahdanau et al., (014)) image captioning (Vinyals)
et al.. 20 Karpathy & Fei-Fei 2015 Xu et al.. 2015)) and many others (Vinyals et 2015
Huang et al., 2018; Ramesh et al. 2021 Chen et al., 2021). Our work enriches this portfolio and shows that works for even non- senu-eeuuntill data (by turning set of objects into sequence of tokens). We augment both input and target sequences for our model to incorporate task-specific prior knowledge; similar sequence corruption scheme have been used in language models (Devlin) et al. 2018 Clark et al. 2020), and bear some similarity to noise-contrastive learning (Gutmann & Hyvärinen 2010) and the discriminator in GANs (Goodfellow et al. 2014).
CONCLUSION AND FUTURE WORK
This paper introduces Pix2Seq, simple yet generic framework for object detection. By casting object detection as a language modeling task, our approach largely simplifies the detection pipeline, removing most of the specialization in modern detection algorithms. We believe that our framework not only works for object detection, but can also be applied to other vision tasks where the output can be represented by a relatively concise sequence of discrete tokens (e.g., keypoint detection, image captioning, visual question answering). To this end, we hope to extend Pix2Seq as generic and unified interface for solving a large variety of vision tasks.
A major limitation of our approach is that autoregressive modeling is expensive for long sequences (mainly during model inference). Practical measures to mitigate the issue includes: 1) stop inference when the ending token is produced (e.g., in COCO dataset, there are, in average, 7 objects per image, leading to a relatively small number of ~35 tokens), 2) applying it to offline inference, or online scenarios where the objects of interest are relatively sparse (e.g. locate a specific object with language description). However, future work is needed to make it faster for real-time object detection applications. Another limitation is that the current approach for training Pix2Seq is entirely based on human annotation, and by reducing such dependence, it can enable the model to benefit from more unlabeled data.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We specially thank Xiuye Gu for preparing the Objects365 dataset. We thank Mohammad Norouzi, Simon Kornblith, Tsung- Yi Lin, Allan Jabri, and Kevin Swersky for the helpful discussions.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.047. 2014.
Navanceth Bodla, Bharat Singh, Rama Chellappa, and Larry Davis. Soft-mns-improving ohject detection with one line of code. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5561-5569, 2017.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are leameer arXiv preprint arXiv:2005. 14165, 2020.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn:
Delving into high quality object detection. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 6154-6162, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko. End-to-end object detection with transformers. European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael I Laskin, Pieter Ahbeel, Aravind Srinivas, and Igor Mordatch, Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106. 00W:2100..0145.. 2021.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geolfrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in Neural Information Processing Systemy, 33: 22243-22255, 2020b.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020.
Jacob Devlin, Ming-Wir Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04005, 2018.
Alexey Dosovitskiy, Lueas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthiss Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2020.
Dumitru Erhan, Christian Szegedy, Alexander Toshev, and Dragomir Anguelov. Scalable object detection using deep neural networks. In Proceedings of the IEEE Conference Computer Vision and Putiern Recognition, pp. 2147-2154, 2014.
M. Everingham, I.. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2918 2918-2928, 2021.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1440-1444 14481448,410 2015,
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 580-587. 2014.
Ian Goodfellow, Jean Pouget-Ales Rouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair. Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014.
10

---

Published as a conference paper at ICLR 2022
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: estimation principle for unnormal- ized statistical models. In Proceedings of the thirteenth International Conference artificial intelligence and statistics, pp. 297-304, JMLR Workshop and Conference Proceedings, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-can. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2961-2969, 2017.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 8129-8138, 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degencration. arXiv preprint arXiv: 1994.999475// 2019.
Andrew G Howard. Some improvements on deep convolutional neural network based image classification. arXiv preprint arXiv:1312.5402, 2013.
Cheng-Zhi Zhi Anna Huang. Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffmann, Monica Dinculescu, and Douglas Eck. Music transformer. arXiv preprint arXiv: (el::18990044 04281, 2018.
Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European Conference Computer Vision, pp. 646 661. Springer, 2016.
Paul Jaeger, Simon AA Kohl, Sebastian Bickelhaupt, Fabian Isensee, Tristan Anselm Kuder, Heinz-Peter Schlemmer. and Klaus Maier-Hein. Retina u-net: Embarrassingly simple exploitation of segmentation supervision for medical object detection. In Machine Learning for Health Workshop. pp. 171-183, PMLR, 2020.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the HEEE Conference on Computer Vision and Pattern Recognition, pp. 3128-3137, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25:1097-1 1105, 2012.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings. Ivan Krasin, Jordi Pont-Tusee Shahab Kamali, Stefan Popov, Matto Malloci, Alexander Kolesnikov, al. The open images dataset v4. International Journal of Computer Vision, 128(7):19561199) 2020.
Yann LeCun, Berhhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4); 541-551, 1989.
Yi Li, Haozhi Qi. Jifeng Dai. Xiangyang Ji. and Yichen Wei. Fully convolutional instance-aware semantic segmentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 3359-23673 2017.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Tsung-Yl Yi I. Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEFE Conference on Computer Vision and puttern recognition, pp. 2117-2125, 2017a
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2980-2988, 2017b.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In Fumpean Conference Computer Vision, pp. 21-37. Springer, 2016.
11

---

Published as a conference paper at ICLR 2022
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
Funbyung Park and Alexander C Berg. Learning to decompose for object detection and instance segmentation. arXiv preprint L.Xiv:1511.06449,, 2015.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910. 10683, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Hly Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2 /2092, 2021.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2016.
Mengye Ren and Richard Zemel. End-to-end instance segmentation with recurrent attention. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6656-6664, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28:91-99, 2015.
Hamid Rezatofighi. Nathan Tsoi. Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union. June 2019.
Bernardino Romera- Paredes and Philip Hilaire Sean Torr. Recurrent instance segmentation. European Conference on Computer Vision, pp. 312-329. 329. Springer, 2016.
Inkyu Sa, Zongyuan Ge, Feras Dayoub, Ben Upcroft, Tristan Perez, and Chris McCool. Deepfruits: fruit detection system using deep neural networks. sensors, 16(8):1222, 2016.
Amaia Salvador, Miriam Bellver, Victor Campos, Manel Baradad, Ferran Marques, Jordi Torres, and Xavier Giro-i Nieto. Recurrent neural networks for semantic instance segmentation. arXiv preprint 1712.0067 2017.
Shuai Shao. Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu. Xiangyu Zhang, Jing Li. and Jian Sun. Objects365: large-scalc, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1430-8439, 2019.
Russell Stewart, Mykhaylo Andriluka, and Andrew Y Ng. End-to-end people detection in crowded scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2325-2333, 2016.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 2446 2454, 2020.
Ilya Sutskever, Oriol Vinyals, and Quac Le. Sequence sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104-3122, 2014.
Christian Szegedy, Alexander Toshev, and Dumitru Erhan. Deep neural networks for object detection. 2013.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9627-9636, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 2017.
Oriol Vinyals, Fukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. Grammar as foreign language. Advances in Neural Information Processing Systems, 28:2773-2781, 2015a.
12

---

Published as a conference paper at ICLR 2022
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3156- 3164, 2015b.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wen and Ross Girshick. Detectron2. //github. cec..efooocectteeeeerrrreeeeeeee 2019.
at pss
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio, Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pp. 2048-24420 2057. PMLR, 2015.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint 1rXiv:190..07850, 2019.
13

---

Published as a conference paper at ICLR 2022
A QUANTIZATION AND DEQUANTIZATION OF COORDINATES
Algorithm
I and 2
illlustraee the quantization and dequantization process of (normalized) coordinates.
Algorithm 1 Quantization of (normalized) coordi- nates
Algorithm 2 Dequantization of discrete tokens of coordinates
def quant 1 ze (x, bing=1000)
def dequantizs (x, bins=10001 is an intoger betwoon [0, bins-1] returns a real number bet [0, 11
is real rumber betweon [0, returns an Lnteger between [0, bins-1] return int (x (binns - 1)3
return float (x) tbins - 1)
B TRAINING DETAILS
Training from scratch on COCO For baseline architectures, we follow (Carion et al. 2020) using a ResNet backbone (He et al. 2016), followed by layers of transformer encoder and layers of (causal) transformer decoder Vaswani al. 2017). The main dimension of transformer is set to 256 with 8 attention heads, and the dimension of the feed-forward network is set to 1024. We use the stochastic depth (Huang et al. 2016) with rate of 10% to reduce overfitting. Per (Carion et al., 2020), we also experiment with the DC5 variant of ResNet (Li et al.) 2017). which increases the resolution of its output feature map by factor of two [
For image augmentation during training, we perform scale jittering with random crops (Ghiasi et al. 2021 Wu et al. 2019) with strength of [0.1.3] We resize images (with fixed aspect ratio) so the longer side is 1333 pixels. Following (Howard 2013) Chen et al. 2020a b), we also use color distortion with a strength of 0.5, For sequence construction, we use 2000 quantization bins, and we randomize the order of objects every time an image is shown. We append noise objects to real objects such that each image contains 100 objects in total, and hence sequence length of 500.
We train the entire network from scratch for 300 epochs with a batch size of 128. For each image in a mini-batch. we perform two independent augmentations, similar to (Hoffer al.] 2020), re- sulting in 256 effective batch size, which we find helpful to reduce overfitting. We use AdamW optimizer (Kingma& Ba 2014) Loshchilov & Hutter 2018) with a learning rate 0.003 and weight decay of 0.05. We use a learning rate warmup for 10 epochs and then linearly decay the learning rate over the course of training.
Pretraining on Objects365 We explore a wider range of architecture variants including both hybrid ResNet and transformer models (Carion et al.) 2020), as well as pure transformers based on image patches (Dosovitskiy et al. 2020). The details of the architecture can be found in our released code. Since Objects365 dataset is much larger than COCO (1.7M images 118K images), a weaker image augmentation (scale jittering range of [0.3, 2] for ViT backbones, and [0.9, 1.2] for ResNet backbones) without color distortion. For sequence construction, we use 1000 quantization bins. And we still apply sequence augmentation with sampled noise objects added by default.
We use smaller image size of 640x640, and pretrain the models for 400K steps with batch size of 256. We do not perform two augmentations per batch as in training from scratch. And we use smaller learning rate of 0.001 with the same weight decay of 0.05. We use a cosine learning rate decay with a initial warmup of 20K steps.
As for the finetuning on COCO dataset, we use a batch size of 128 for ResNet backbones, and 64 for ViT backbones. Most models are finetuned for 60 epochs with learning rate of 35-5, but even fewer epochs yield similar results. We still use scale jittering with range of [0.3, 2] for image augmentation.
2 "Addinn dilation the last ResNet stage and removing the stride from the first convolution of that stage.
14

---

Published as a conference paper at ICLR 2022
C ABLATION ON INFERENCE (arg max VS NUCLEUS SAMPLING)
Nucleus sampling (Holizman et al. 2019) has been applied to language modeling to reduce duplication and increase diversity in generated samples. Here we study its impact sampling from trained model.
Given the distribution Ply [z. yı.j-i,, apply nucleus sampling, we first define its top-p vocabulary (P) V the smallest set such that
Σ
P(yl y1:j 2
MVV
Let Letp'= =' p x y1.j-1), and we can calcbbiba the conditional likelihood as following for sampling the next token.
P(yllæ, vra-)//))
if yj E V())
(uyla.yy.. 1. y1 (( =
(3)
otherwise.
We vary the inference). When 0, it corresponds to arg max sampling, otherwise samples from a truncated ranked list of tokens that has cumsum larger or equal to In Figure 10] we see that use of nucleus sampling (with p > 0) improves object recall and thus also leads to better average precision. There relatively flat region of AP between 0.2 and 0.5, and we select p to be 0.4 as our default value for other experiments.
of nucleus sampling used in generating the output sequence (during
56
39
Aveerge Recallanon
155
38
54
2
37
Figure 10: Varying parameter p in nucleus sampling during inference results in different AP and AR. With p 0, it is equivalent to argmax sampling. Sampling with p >> 0 is helpful for increasing recall (and precision).
D VISUALIZATION OF SIMILARITY AMONG COORDINATE TOKENS
In our model, bounding box coordinates are not represented as floating points, but encoded as discrete tokens. Here study the similarity among these coordinate tokens via their embeddings. Note that the discrete coordinate tokens and class name tokens are in the same vocabulary and share the same embedding matrix. Specifically, we first slice the learned embedding matrix corresponding to coordinate tokens, and then compute the cosine similarity of embedding vectors for these coordinate tokens.
Figure TT shows cosine similarity among embeddings of coordinate tokens. We can see that nearby coordinates have higher similarities in their token embeddings than far away ones. This emergent property of our model is likely due to the noises / uncertainties in bounding box annotations (i.e. bounding box annotation random sample from a distribution over potential bounding boxes which encodes locality of coordinates).
E THE ABILITY TO DIRECT THE ATTENTION WITH GIVEN COORDINATES
We explore the model's ability to pay attention to a pointed region specified via coordinates. We divide an image evenly into N x N grid of rectangular regions, each specified by sequence of
15

---

Published as a conference paper at ICLR 2022
10
-08
Coordinate tokens
Coordinate tokens
0.5
0.6
0.1
02
Coordinate tokens
Coordinate takens
(a)
(b)
1.0
1.0
1.0
0.8
0.9
0.8
simiarrty
giilerty
0.6
0.6
Cosine Cashae
0.4
0.0
0.4
0.2
0.2
0.0
1500
Coondinate tncens
(C)
(d)
Figure 11: (a) Cosine similarity among embeddings of coordinate tokens. (b) is part of (a) covering only the first 100 tokens. (c), (d) and (e) are the 500-th, 1000-th and 1500-th rows of (a), respectively. Nearby coordinates have higher similarities in their token embeddings.
coordinates bounding box. We then visualize the decoder's cross attention to visual feature map after reading the sequence of coordinates for each region, i.e., [Umin, Tmin; 2maxx Tmmx]. We shuffle the pixels in the image to remove distraction from existing objects, and remove 2% of the top attentions for clarity. Interestingly, shown in Figure 12] it seems the model pay attention the specified region at different scales.
(a) 4x4 grids
(b) 6x6 grids
(c) 8x8 grids
(d) 10x 10 grids
Figure 12: Each grid is a visualization of decoder's s attention after reading a small sequence of coordinates, i.e, (umin Fmins Vmux: ]- Visualization is done for grids of different sizes. The network leams to pay attention to pointed region at different scales.
F MORE VISUALIZATION ON DECODER'S CROSS ATTENTION
In Figure 1 overlay the cross attention (when predicting the class token) the original image for several other images, and it shows that the decoder pays the most attention to the object when predicting the class token.
16

---

Published as a conference paper at ICLR 2022
(A)
(b)
Figure 13: Visualization of Transformer decoder's 's cross attention (when predicting class tokens) conditioned on the given bounding boxes.
G VISUALIZATION OF DETECTION RESULTS
In Figure 14 we visualize detection results one of Pix2seq model (with 46 AP) on a subset of images from COCO validation that contain crowded set of objects.
Aur
Figure 14: Examples of the model's predictions (at the score threshold of 0.5). Original images accessed by clicking the images supported PDF readers.
17


Reviews:
[{"review_id": "lsaPBMcM_fE", "reviewer": "Reviewer_JMKG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**[Strength]**\n1. The paper is generally well written and easy to understand.\n2. The proposed method, Pix2Seq, firstly adopts a language model in object detection.\n3. The proposed method is simple but achieves comparable results with existing methods.\n4. There are various ablation studies for helping understand the embedding of language models on object detection.\n\n\n**[Weakness]**\n1. Inference Time\n\nThe main concern is inference time. Since the model has sequence prediction (“generate one token at a time”), it can take more time compared to existing models. Hence, adding inference time on Table 1 is needed.\n\n2. Training Speed\n\nAnother main concern is training time, 300 epochs. This is one of the major concerns in DETR and may inherit the proposed method. It can be relaxed by using different architecture such as deformable DETR (Zu et al. 2021) or other efficient DETRs.\n\nZhu, Xizhou, et al. \"Deformable detr: Deformable transformers for end-to-end object detection.\" ICLR. 2021.\n\n\n3. Sequence Augmentation\n\n3-1. The authors tackle the exploitation of prior knowledge of object detection in other methods. However, as mentioned in the second paragraph of section 2.3, the proposed method also exploits prior knowledge as a sequence augmentation and it is critical for the performance as shown in Figure 8.\n\n3-2. On the other hand, a curve in Figure 8 is generated by “allowing the model to make more predictions.” However, there is no description of how to do that such as ignoring EOS until K steps. The authors should present or specify the exact performance for a model without Sequence Augmentation and not allow the mode to make more predictions.\n\n4. EOS\n\nSince the model uses a language model, it is possible to emit EOS between some coordinates or class tokens (e.g., y_min, x_min, y_max, EOS). How does the model deal with this? Does the model ignore the bounding box and stop the inference?\n\n\n5. Attention Map\n\nWhat does it mean by columns and rows in Figure 9 (b)? It seems that row means a different set of coordinates and classes (i.e., bbox) and the column means y_min, x_min, y_max, x_max, class. Although it is implicitly described in the context, I would recommend explicitly mentioning it.\n\n6. Similarity among Coordinate Tokens\n\nThe authors only present the correlation matrix and say nearby coordinates have higher similarities in their token embedding. More explanation is needed. Also, the outputs have five (except EOS) different meanings (y_min, x_min, y_max, x_max, class). How is the correlation matrix generated? \n\n7. Change Figure 14\n\n Currently, the authors add hyperlinks in each ‘url’. However, the authors also consider people who read the paper in a hard copy or offline. presenting the original image in the paper or even deleting the captions will be better.\n\n\n\n**[Minor]**\n1. DETR (Carion et al., 2020) does not have a box regression sub-network although it utilizes  GIoU loss. Please change the description in the first sentence of page 9, “These detectors”.\n\n2. Change 43.2, 44.9 in Table 1 to plain text.\n\n\n**[Recommendation]**\n1. Move ablation study for image augmentation to Supplementary material.\n=> Image augmentation is widely used in object detection as the author mentioned and there is no need to incorporate it in the main paper. \n\n2. The author uses only 200 epochs for ablation study while the full model is 300 epochs, which is acceptable but still comparing at the same epochs (300) will be better.\n\n3. In Section 3.3, “class (name)” seems “class + random.” For clarity, adding random or some description will be better.\n\n4. Cross Attention Maps are really good. I suggest present instance or panoptic segmentation performance based on the cross attention map similar to DETR (Carion et al., 2020) paper.\n", "summary_of_the_paper": "\n1. The paper tackles object detection by using the encoder-decoder structure of the language model, Pix2Seq.\n2. The authors argue that the proposed method leverages prior knowledge for object detection less than existing object detection algorithms that exploit box regression, intersection-over-union, and so on.\n3. Pix2Seq is based on maximum likelihood loss and employs sequence augmentation by adding dummy object bounding boxes to delay EOS (End Of Sequence)\nConsequently, The proposed method achieves comparable results with existing methods.\n", "summary_of_the_review": "In general, the proposed method is novel and gets good results. I believe it will get a huge attention and lead to a big change in computer vision community. However, there are a few concerns as mentioned above, especially inference time. I hope the authors resolve my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jG38UnVZbin", "reviewer": "Reviewer_Wvff", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. This paper proposed a novel idea for object detection. Unlike most previous work, the proposed pix2seq model converts the coordinates and labels of objects into a sequence of token and leverage a seq2seq model to complete the prediction. \n\n2. To mitigate the overfitting issue, the authors proposed a few techniques such as training sequence augmentation via noisy annotations. It turns out to be a helpful way to improve object detection performance.\n\n3. The experimental results show that the proposed method achieve comparable performance to two strong baselines, including Faster R-CNN and DETR. Further ablation studies indicate that the sequence augmentation indeed helps and some visualizations align with the intuition behind the model. \n\nCons: \n\n1. The proposed pix2seq is novel in that it uses a sequence generation model to predict object locations and classes, getting rid of the sophisticatedly designed architecture such as Faster R-CNN. However, it still resembles previous works like DETR in that they both exploited an encoder-decoder architecture. DETR decodes the predictions in parallel while the proposed model does it sequentially. Then what are the benefits and unique advantages of modeling it as a sequential decoding problem? I agree that it has the potential to unify different sequential decoding models. However, it is hard to tell in this paper and it is also an open question whether we want to unify all tasks into a sequence generation pipeline because localizing objects seems not to be a sequential task and does not heavily rely on temporal information (Fig. 7(a)(b) shows that random ordering during training obtains the highest mAP.)\n\n2. According to the experimental results, the performance is comparable to Faster R-CNN and DETR while the training/inference may be much more time-consuming. Since the authors did not report the time cost of the proposed method during training and inference. My impression is that such a sequential model may introduce more time cost than parallel ones such as Faster R-CNN or DETR. This leads to the same question asked above. What are the main benefits of modeling object detection as a sequence generation task?\n\n3. The authors claimed that the proposed method is unlike previous highly specified or heavily optimized ones. However, it seems that there is no free lunch. To avoid overfitting, the authors need to use sequence augmentation. To get better decoding results, the model also relies on a better sampling strategy, i.e., nucleus sampling. Even with these two techniques, I do not see a higher ceiling of performance starting from a low floor due to overfitting, which is unlike the one we observed in Vision Transformers for image recognition. I would like to hear more from the authors about what could be potentially applied to further improve the performance.\n\n4. I am curious about whether the proposed method can generalize well across different image sizes during inference. Accordingly, the authors used normalized coordinates. I guess this will not be a big issue. But still, it would be great to see whether the proposed method can perform multi-scale inference giving different image sizes.\n\n5. In the proposed method, the decoder predicts the quantized coordinate tokens given input feature map and preceding tokens. To precisely predict the locations, the model needs to map the heat map into discrete tokens. I am wondering whether the model heavily relies on positional embeddings. If yet, what kind of positional embedding is used for the encoder, and how this will affect the final performance?", "summary_of_the_paper": "In this paper, the authors proposed a novel way of formulating object detection as a seq2seq task. Given an image, the model first uses a visual encoder the obtain a feature map and then sequentially decode the coordinates and label through a decoder. Different from previous conventional object detection pipelines, such as Faster R-CNN, the proposed method does not rely much on the prior knowledge or assumption about the task but lets the model learns by itself from training data. To avoid overfitting, the authors proposed a few techniques, including data augmentation and a more sophisticated decoding strategy. The experimental results show that it can achieve comparable performance with two established baselines, Faster R-CNN and DETR. These results indicate that even with a simple seq2seq pipeline, the model can be still on par with previous strong baseline methods. ", "summary_of_the_review": "I think this paper proposed a novel idea to reformulate object detection to a sequence generation problem. It provides us with a new perspective to think about conventional vision tasks. However, from the paper, I do see several drawbacks of the proposed method. Without seeing strong proofs, it is hard to determine whether the proposed method is a good one for generic object detection. As such, I have a general concern that this paper can bring us a new viewpoint but not new insight to solve object detection problems.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SwMMUxLXVNQ", "reviewer": "Reviewer_bYa9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The submission has merit and a potential to receive considerable attention by the research community due to having an unconventional (but not necessary novel ) view point to the object detection problem with good results. However, I have few major comments on the paper presentation, novelty, limitation, experiments and ablation study listed below:\n\n\n1-\tPresentation: I am not really convinced about the story of the paper and the way it is presented and motivated. As I reflected in the abstract, in my view (though I might misunderstood the technical details), this framework simply formulates the object detection problem as 1) a sequence prediction problem using 2) a transformer-based autoregressive model and 3) the key idea is to ensure each outputted bounding box can be formulated as a set of the token (i.e. by discretising the bounding box states to a set of bins). To this end, the link to a language modelling is weak, strange and arbitrary for the presentation. Considering my viewpoint to the proposed approach (the notes 1-3), the strong argument against limitation of the existing detection techniques and their formulation for the motivation of this work also seem to be invalid as this framework does not really address them. \n2-\tNovelty: Formulating the object detection as the bounding box sequence prediction (note 1) is not very novel (e.g. Stewart et al 2016), and in comparison in this framework, the previous frameworks did not have advantage of using very powerful backbone and decoders as the proposed framework. Bounding box regression is also known to be harder task than classification and to this end, few recent methods have approximated the task by discretising the spaces (e.g. Qiu et al. Offset bin classification network for accurate object detection, CVPR 2020). \n3-\tLimitation: while this strategy may work on 2D object detection framework with axis align bounding boxes, it is not clear how it will perform on (and also computationally scales for) the detection problems such as non-axis aligned object detection (e.g. detection from Satellite image) or 3D object detections (with 6 DoF), where the number of discretising bins (token) can increase exponentially.   \n4-\tExperiments: It is hard to compare which components help more to the Pix2seq framework’s good results from Table 1, e.g. (a) network model: a better decoder (a large transformers encoder and decoder) compared to Faster R-CNN (few MLP layers in the second stage),  (b) Formulation: autoregressive sequence prediction instead of tensor prediction in Faster R-CNN or set prediction in DETR (c.f. Rezatofighi et al. arxiv 2020) or (c) loss variation: avoiding regression loss by discretising the bounding box representation and using softmax loss similar to classification. It is also meaningful to include both inference variations of the framework in Table 1 (argmax sampling & nucleus sampling)\n\nOne  minor comment:\n \nThe Number of parameters in Table 1 should include both backbone and decoder. It would be great if FLOP is also reported \n\nI can understand why random ordering might perform better than a handcrafted (potentially inconsistent) deterministic ordering, but this random strategy also can be sub-optimal. The chain rule decomposition in Eq. 1 can be written in L! ways. While the weight in neural network should learn a joint representation agnostic to these output orders for the same input x, learning this number of combination may not be achievable by a random sampling. In the other sequential techniques, e.g. Vinyals et al, Order Matters, 2015  and Stewart at al. 2016, the best permutation is selected dynamically during training stage by solving an assignment problem between all the predictions and GT before the loss calculation. \n\n", "summary_of_the_paper": "This paper presents a new framework for object detection by casting the problem as an (auto-encoder based) auto-regressive sequence prediction using a CNN based backbone as the encoder to encode visual features and transformer-based encoder & decoder (c.f. section 3.1) as the decoder to predict each bounding box sequentially. All the bounding boxes in an image are generated auto-regressively conditioned on the image features from the backbone and previous predictions (c.f. Eq. 1). The key idea in this paper is to output each axis-aligned bounding box as the set of tokens representing the possible bin locations for its two corners and then to use cross-entropy loss (SoftMax) along with the class token to predict each detection.  The best model is trained by the random ordering of bounding boxes. The approach achieved competitive results on the challenging COCO dataset, compared to Faster R-CNN and DETR.", "summary_of_the_review": "The submission carries an important message for the object detection problem and the framework results are reasonably good, But the paper presentation should be substantially improved to be a reflective of the contribution.  Limitation of this formulation should be clearly elaborated and the ablation study should contain insightful results for the contribution of each proposed module (formulation, architecture or loss). To this end, I rate this submission borderline ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HULd3IVgdZ", "reviewer": "Reviewer_kHNj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Overview\n\n- The idea is novel, the proposed approach is simple and elegant and the paper is well written with most of the technical details with strong experimental results. I really appreciate the authors advocating a totally new approach for object detection based on the intuition that *``if a neural network knows about where and what the objects are, we just need to read them out''*. The proposed model is good proof of that intuition. \n\n- The proposed model and training objective is more general compared to prior models on object detection. Compared to Faster-RCNN, pix2seq gets rids of bounding box proposals, ROI pooling which is highly customized for detection tasks. Compared to the more recent Detr, pix2seq gets rid of the object queries and set-based matching loss. This is a huge improvement towards a more unified model for vision tasks. \n\n- Besides all those pros, there are a few missing details in the paper that needs further clarification (details in the weakness section.\n\n#### Strength\n\n- The idea of *providing a language interface to a wide range of vision tasks* is novel and the proposed model is simple, elegant, and achieves strong performance on object detection benchmark. \n\n- Language modeling with sequence augmentation is novel and useful to encourage higher recall rates. \n\n- Extensive and informative ablation studies on ResNet variant, #bins, different object ordering strategies, image scale augmentation, and sequence augmentation. \n\n#### Weakness\n\n- Position encoding (PE) should be very important for the proposed approach. However, the paper didn't discuss what PE is used in the paper (e.g. absolute, learned, relative) and how to use them (e.g. adding to seq embedding or adding to key, value in attention). The discussion and ablation study on different PEs will be super useful for the reader to replicate the model. \n\n- In altered sequence construction, there are two ways to synthetic the noise sequence. I wonder what is the percentage of different synthetic noise sequences used in the paper? \n\n- In figure 5, the noise token is start with <y_11> but not <end> token. This seems to break the auto-regressive sequence construction. I wonder is there any specific reason to do this? \n\n- What is the inference time of the proposed model. It is known that the auto-regressive model is slow at the decoding stage, but comparing it to the Detr model will be informative for the readers. \n\n- Instance or semantic segmentation (with variable sequence length) seems a natural extension to the proposed model. I wonder is there any comment on this task using the proposed approach? \n", "summary_of_the_paper": "This paper proposed a language modeling framework (pixel2seq) for object detection. The authors cast object detection as language modeling tasks that use a sequence of tokens (x1, y1, x2, y2, c) to describe the bounding box and train an auto-regressive decoder to generate the target sequence. Compared to the existing approach (Faster-RCNN, Detr), the proposed model uses a more general architecture and loss function and achieves state-of-the-art performance on COCO datasets. ", "summary_of_the_review": "The idea is novel, the proposed approach is simple and elegant and the paper is well written with most of the technical details with strong experimental results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "6dqKVt23OE5", "reviewer": "Reviewer_Hjdy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\nSTRENGTHS\n\nS1 The study is novel and scientifically interesting in order to further understand the potential of the language model task beyond textual inputs and outputs.\n\nS2 The loss function is the classic negative log likelihood, so it is not specifically designed for the task, as in other works from the state of the art.\n\nS3 Proposes a approach to encode the continuous coordinates of the bounding boxes to discrete tokens. This simplifies the decoder architecture, compared to DETR.\n\nS4 The work does not require the non-maximum supression post-processing, as in DETR.\n\nS5 The proposed data augmentation approach to avoid the early EOS or repeated detections, observed in previous works, is novel.\n\nS6 It includes ablation studies on the size of the quatization bins and sequence ordering, as well as insightful visualizations of the cross-attention maps.\n\nS7 The manuscript is well written, with multiple figures that facilitate comprehension.\n\n\n\nWEAKNESSES\n\nW1 The obtained metrics in accuracy do not achieve state of the art, but are competitive.\n\nW2 While the work motivates that existing approaches often focus on specific domains (self-driving cars, medical image analysis or agriculture), the results presented focus only on the COCO bechmark. Providing results for some of these specific domains would provide better insights about the potential of Pix2Seq in the referred myriad of domains.\n\n\nW3a The weight w_j for the tokens in Equation 1 is defined but never used or tested. An experimentals analysis of its impact should be included to justify it.\n\nW3b The loss function seems to compare input and target sequences, but actually the order in which the bounding boxes+class labels are generated should not be taken into account. That is, the loss should be invariant to the ordering of the predicted objects. It seems that it will penalize in the proposed set ups, while previous work have used the Hungarian algorithm to match predicted and ground truth detections before computing the loss. It is unclear why Pix2Seq is not adopting this same paradigm and whether it has a negative effect in the results.\n\nW4 An ablation study or disucss about the impact of the sequence augmentation is needed, as this is a mmain contribution of this work.\n\nW5 An analysis or discussion of the computational memory & requirements with respect to Faster R-CNN & DETR is needed to obtain a full picture beyond the accuracy results only.\n\n\nMINOR COMMENTS\n\nC1 In figure 9, what are the columns ? It seems to be the cross-attention maps when predicting the 4 coordinates + class. If so, then the whole output sequence of 25 tokens is the result of reading row by row ? More guidance to the reader may be helpful.\n\nC2 Given the sucess of seq2seq models for image generation (eg, iGPT) when trained with large amounts of data, one woders whether training by just more data results would actually reach state of the art. For example, the OpenImages dataset already provides much more data could may be used to explore the gains in this direction.\n", "summary_of_the_paper": "Pix2Seq provides a simple approach for object detection understood as a sequence generation task, in this case, of the coordinates of the bounding box and the object class. The architecture resembles DETR, but simplifies the decoder thanks to the formulation as a sequence decoding of discretized tokens. The manuscript proposes a scheme to discretize the bounding box coordinates in histogram bins and proposes data augmentation for the sequence, which  address two observed limitations when predictig sequences from images: early EOS and repetition of objects. Results on MS-COCO indicate slightly better results than DETR with the more simple architecture.", "summary_of_the_review": "The proposed ideas are novel and, from my perspective, valuable enough for their publication despite not obtaining new state of the art results for the task. There is already a significant body of works that have addressed the analysis of images as a sequential process in which a predicted token coditions that posterior predictions. Pix2Wav simplifies DETR and improves its performance a bit. However, there are still some doubts listed in the weaknesses that should be clarified before making a final decision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an elegant approach to object detection where an encoder network reads in an image and a decoder network outputs coordinate and category information via a sequence of textual tokens. This method does away with several object detection specific details and tricks such as region proposals and ROI pooling. The paper received positive reviews from all reviewers who agreed that this formulation of object detection was novel and provided a new perspective that may transfer to other computer vision tasks. One common concern amongst reviewers was the slow inference time due to the sequential nature of the decoder -- and this concern was a central point of discussion between the authors and reviewers. My takeaway from this discussion is that this model is certainly slower than traditional computer vision models that can generate boxes in parallel. The slowdown however, is image dependent. Less cluttered environments require shorter output sequences. Moreover, such a model can easily be applied to concept localization, e.g. "Locate the horses", in which cases one can expect fewer objects of the desired category, and hence acceptable inference speeds. Importantly, the contributions of this paper are noteworthy in spite of the proposed architecture having the drawback of being slow. Given this, I recommend accepting this paper for its merits.
Decision: Accept (Poster)