Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a language modeling framework (pixel2seq) for object detection. The authors cast object detection as language modeling tasks that use a sequence of tokens (x1, y1, x2, y2, c) to describe the bounding box and train an auto-regressive decoder to generate the target sequence. Compared to the existing approach (Faster-RCNN, Detr), the proposed model uses a more general architecture and loss function and achieves state-of-the-art performance on COCO datasets. 
main_review: #### Overview

- The idea is novel, the proposed approach is simple and elegant and the paper is well written with most of the technical details with strong experimental results. I really appreciate the authors advocating a totally new approach for object detection based on the intuition that *``if a neural network knows about where and what the objects are, we just need to read them out''*. The proposed model is good proof of that intuition. 

- The proposed model and training objective is more general compared to prior models on object detection. Compared to Faster-RCNN, pix2seq gets rids of bounding box proposals, ROI pooling which is highly customized for detection tasks. Compared to the more recent Detr, pix2seq gets rid of the object queries and set-based matching loss. This is a huge improvement towards a more unified model for vision tasks. 

- Besides all those pros, there are a few missing details in the paper that needs further clarification (details in the weakness section.

#### Strength

- The idea of *providing a language interface to a wide range of vision tasks* is novel and the proposed model is simple, elegant, and achieves strong performance on object detection benchmark. 

- Language modeling with sequence augmentation is novel and useful to encourage higher recall rates. 

- Extensive and informative ablation studies on ResNet variant, #bins, different object ordering strategies, image scale augmentation, and sequence augmentation. 

#### Weakness

- Position encoding (PE) should be very important for the proposed approach. However, the paper didn't discuss what PE is used in the paper (e.g. absolute, learned, relative) and how to use them (e.g. adding to seq embedding or adding to key, value in attention). The discussion and ablation study on different PEs will be super useful for the reader to replicate the model. 

- In altered sequence construction, there are two ways to synthetic the noise sequence. I wonder what is the percentage of different synthetic noise sequences used in the paper? 

- In figure 5, the noise token is start with <y_11> but not <end> token. This seems to break the auto-regressive sequence construction. I wonder is there any specific reason to do this? 

- What is the inference time of the proposed model. It is known that the auto-regressive model is slow at the decoding stage, but comparing it to the Detr model will be informative for the readers. 

- Instance or semantic segmentation (with variable sequence length) seems a natural extension to the proposed model. I wonder is there any comment on this task using the proposed approach? 

summary_of_the_review: The idea is novel, the proposed approach is simple and elegant and the paper is well written with most of the technical details with strong experimental results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a novel way of formulating object detection as a seq2seq task. Given an image, the model first uses a visual encoder the obtain a feature map and then sequentially decode the coordinates and label through a decoder. Different from previous conventional object detection pipelines, such as Faster R-CNN, the proposed method does not rely much on the prior knowledge or assumption about the task but lets the model learns by itself from training data. To avoid overfitting, the authors proposed a few techniques, including data augmentation and a more sophisticated decoding strategy. The experimental results show that it can achieve comparable performance with two established baselines, Faster R-CNN and DETR. These results indicate that even with a simple seq2seq pipeline, the model can be still on par with previous strong baseline methods. 
main_review: Pros:

1. This paper proposed a novel idea for object detection. Unlike most previous work, the proposed pix2seq model converts the coordinates and labels of objects into a sequence of token and leverage a seq2seq model to complete the prediction. 

2. To mitigate the overfitting issue, the authors proposed a few techniques such as training sequence augmentation via noisy annotations. It turns out to be a helpful way to improve object detection performance.

3. The experimental results show that the proposed method achieve comparable performance to two strong baselines, including Faster R-CNN and DETR. Further ablation studies indicate that the sequence augmentation indeed helps and some visualizations align with the intuition behind the model. 

Cons: 

1. The proposed pix2seq is novel in that it uses a sequence generation model to predict object locations and classes, getting rid of the sophisticatedly designed architecture such as Faster R-CNN. However, it still resembles previous works like DETR in that they both exploited an encoder-decoder architecture. DETR decodes the predictions in parallel while the proposed model does it sequentially. Then what are the benefits and unique advantages of modeling it as a sequential decoding problem? I agree that it has the potential to unify different sequential decoding models. However, it is hard to tell in this paper and it is also an open question whether we want to unify all tasks into a sequence generation pipeline because localizing objects seems not to be a sequential task and does not heavily rely on temporal information (Fig. 7(a)(b) shows that random ordering during training obtains the highest mAP.)

2. According to the experimental results, the performance is comparable to Faster R-CNN and DETR while the training/inference may be much more time-consuming. Since the authors did not report the time cost of the proposed method during training and inference. My impression is that such a sequential model may introduce more time cost than parallel ones such as Faster R-CNN or DETR. This leads to the same question asked above. What are the main benefits of modeling object detection as a sequence generation task?

3. The authors claimed that the proposed method is unlike previous highly specified or heavily optimized ones. However, it seems that there is no free lunch. To avoid overfitting, the authors need to use sequence augmentation. To get better decoding results, the model also relies on a better sampling strategy, i.e., nucleus sampling. Even with these two techniques, I do not see a higher ceiling of performance starting from a low floor due to overfitting, which is unlike the one we observed in Vision Transformers for image recognition. I would like to hear more from the authors about what could be potentially applied to further improve the performance.

4. I am curious about whether the proposed method can generalize well across different image sizes during inference. Accordingly, the authors used normalized coordinates. I guess this will not be a big issue. But still, it would be great to see whether the proposed method can perform multi-scale inference giving different image sizes.

5. In the proposed method, the decoder predicts the quantized coordinate tokens given input feature map and preceding tokens. To precisely predict the locations, the model needs to map the heat map into discrete tokens. I am wondering whether the model heavily relies on positional embeddings. If yet, what kind of positional embedding is used for the encoder, and how this will affect the final performance?
summary_of_the_review: I think this paper proposed a novel idea to reformulate object detection to a sequence generation problem. It provides us with a new perspective to think about conventional vision tasks. However, from the paper, I do see several drawbacks of the proposed method. Without seeing strong proofs, it is hard to determine whether the proposed method is a good one for generic object detection. As such, I have a general concern that this paper can bring us a new viewpoint but not new insight to solve object detection problems.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Pix2Seq provides a simple approach for object detection understood as a sequence generation task, in this case, of the coordinates of the bounding box and the object class. The architecture resembles DETR, but simplifies the decoder thanks to the formulation as a sequence decoding of discretized tokens. The manuscript proposes a scheme to discretize the bounding box coordinates in histogram bins and proposes data augmentation for the sequence, which  address two observed limitations when predictig sequences from images: early EOS and repetition of objects. Results on MS-COCO indicate slightly better results than DETR with the more simple architecture.
main_review: 

STRENGTHS

S1 The study is novel and scientifically interesting in order to further understand the potential of the language model task beyond textual inputs and outputs.

S2 The loss function is the classic negative log likelihood, so it is not specifically designed for the task, as in other works from the state of the art.

S3 Proposes a approach to encode the continuous coordinates of the bounding boxes to discrete tokens. This simplifies the decoder architecture, compared to DETR.

S4 The work does not require the non-maximum supression post-processing, as in DETR.

S5 The proposed data augmentation approach to avoid the early EOS or repeated detections, observed in previous works, is novel.

S6 It includes ablation studies on the size of the quatization bins and sequence ordering, as well as insightful visualizations of the cross-attention maps.

S7 The manuscript is well written, with multiple figures that facilitate comprehension.



WEAKNESSES

W1 The obtained metrics in accuracy do not achieve state of the art, but are competitive.

W2 While the work motivates that existing approaches often focus on specific domains (self-driving cars, medical image analysis or agriculture), the results presented focus only on the COCO bechmark. Providing results for some of these specific domains would provide better insights about the potential of Pix2Seq in the referred myriad of domains.


W3a The weight w_j for the tokens in Equation 1 is defined but never used or tested. An experimentals analysis of its impact should be included to justify it.

W3b The loss function seems to compare input and target sequences, but actually the order in which the bounding boxes+class labels are generated should not be taken into account. That is, the loss should be invariant to the ordering of the predicted objects. It seems that it will penalize in the proposed set ups, while previous work have used the Hungarian algorithm to match predicted and ground truth detections before computing the loss. It is unclear why Pix2Seq is not adopting this same paradigm and whether it has a negative effect in the results.

W4 An ablation study or disucss about the impact of the sequence augmentation is needed, as this is a mmain contribution of this work.

W5 An analysis or discussion of the computational memory & requirements with respect to Faster R-CNN & DETR is needed to obtain a full picture beyond the accuracy results only.


MINOR COMMENTS

C1 In figure 9, what are the columns ? It seems to be the cross-attention maps when predicting the 4 coordinates + class. If so, then the whole output sequence of 25 tokens is the result of reading row by row ? More guidance to the reader may be helpful.

C2 Given the sucess of seq2seq models for image generation (eg, iGPT) when trained with large amounts of data, one woders whether training by just more data results would actually reach state of the art. For example, the OpenImages dataset already provides much more data could may be used to explore the gains in this direction.

summary_of_the_review: The proposed ideas are novel and, from my perspective, valuable enough for their publication despite not obtaining new state of the art results for the task. There is already a significant body of works that have addressed the analysis of images as a sequential process in which a predicted token coditions that posterior predictions. Pix2Wav simplifies DETR and improves its performance a bit. However, there are still some doubts listed in the weaknesses that should be clarified before making a final decision.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new framework for object detection by casting the problem as an (auto-encoder based) auto-regressive sequence prediction using a CNN based backbone as the encoder to encode visual features and transformer-based encoder & decoder (c.f. section 3.1) as the decoder to predict each bounding box sequentially. All the bounding boxes in an image are generated auto-regressively conditioned on the image features from the backbone and previous predictions (c.f. Eq. 1). The key idea in this paper is to output each axis-aligned bounding box as the set of tokens representing the possible bin locations for its two corners and then to use cross-entropy loss (SoftMax) along with the class token to predict each detection.  The best model is trained by the random ordering of bounding boxes. The approach achieved competitive results on the challenging COCO dataset, compared to Faster R-CNN and DETR.
main_review: The submission has merit and a potential to receive considerable attention by the research community due to having an unconventional (but not necessary novel ) view point to the object detection problem with good results. However, I have few major comments on the paper presentation, novelty, limitation, experiments and ablation study listed below:


1-	Presentation: I am not really convinced about the story of the paper and the way it is presented and motivated. As I reflected in the abstract, in my view (though I might misunderstood the technical details), this framework simply formulates the object detection problem as 1) a sequence prediction problem using 2) a transformer-based autoregressive model and 3) the key idea is to ensure each outputted bounding box can be formulated as a set of the token (i.e. by discretising the bounding box states to a set of bins). To this end, the link to a language modelling is weak, strange and arbitrary for the presentation. Considering my viewpoint to the proposed approach (the notes 1-3), the strong argument against limitation of the existing detection techniques and their formulation for the motivation of this work also seem to be invalid as this framework does not really address them. 
2-	Novelty: Formulating the object detection as the bounding box sequence prediction (note 1) is not very novel (e.g. Stewart et al 2016), and in comparison in this framework, the previous frameworks did not have advantage of using very powerful backbone and decoders as the proposed framework. Bounding box regression is also known to be harder task than classification and to this end, few recent methods have approximated the task by discretising the spaces (e.g. Qiu et al. Offset bin classification network for accurate object detection, CVPR 2020). 
3-	Limitation: while this strategy may work on 2D object detection framework with axis align bounding boxes, it is not clear how it will perform on (and also computationally scales for) the detection problems such as non-axis aligned object detection (e.g. detection from Satellite image) or 3D object detections (with 6 DoF), where the number of discretising bins (token) can increase exponentially.   
4-	Experiments: It is hard to compare which components help more to the Pix2seq framework’s good results from Table 1, e.g. (a) network model: a better decoder (a large transformers encoder and decoder) compared to Faster R-CNN (few MLP layers in the second stage),  (b) Formulation: autoregressive sequence prediction instead of tensor prediction in Faster R-CNN or set prediction in DETR (c.f. Rezatofighi et al. arxiv 2020) or (c) loss variation: avoiding regression loss by discretising the bounding box representation and using softmax loss similar to classification. It is also meaningful to include both inference variations of the framework in Table 1 (argmax sampling & nucleus sampling)

One  minor comment:
 
The Number of parameters in Table 1 should include both backbone and decoder. It would be great if FLOP is also reported 

I can understand why random ordering might perform better than a handcrafted (potentially inconsistent) deterministic ordering, but this random strategy also can be sub-optimal. The chain rule decomposition in Eq. 1 can be written in L! ways. While the weight in neural network should learn a joint representation agnostic to these output orders for the same input x, learning this number of combination may not be achievable by a random sampling. In the other sequential techniques, e.g. Vinyals et al, Order Matters, 2015  and Stewart at al. 2016, the best permutation is selected dynamically during training stage by solving an assignment problem between all the predictions and GT before the loss calculation. 


summary_of_the_review: The submission carries an important message for the object detection problem and the framework results are reasonably good, But the paper presentation should be substantially improved to be a reflective of the contribution.  Limitation of this formulation should be clearly elaborated and the ablation study should contain insightful results for the contribution of each proposed module (formulation, architecture or loss). To this end, I rate this submission borderline
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 
1. The paper tackles object detection by using the encoder-decoder structure of the language model, Pix2Seq.
2. The authors argue that the proposed method leverages prior knowledge for object detection less than existing object detection algorithms that exploit box regression, intersection-over-union, and so on.
3. Pix2Seq is based on maximum likelihood loss and employs sequence augmentation by adding dummy object bounding boxes to delay EOS (End Of Sequence)
Consequently, The proposed method achieves comparable results with existing methods.

main_review: **[Strength]**
1. The paper is generally well written and easy to understand.
2. The proposed method, Pix2Seq, firstly adopts a language model in object detection.
3. The proposed method is simple but achieves comparable results with existing methods.
4. There are various ablation studies for helping understand the embedding of language models on object detection.


**[Weakness]**
1. Inference Time

The main concern is inference time. Since the model has sequence prediction (“generate one token at a time”), it can take more time compared to existing models. Hence, adding inference time on Table 1 is needed.

2. Training Speed

Another main concern is training time, 300 epochs. This is one of the major concerns in DETR and may inherit the proposed method. It can be relaxed by using different architecture such as deformable DETR (Zu et al. 2021) or other efficient DETRs.

Zhu, Xizhou, et al. "Deformable detr: Deformable transformers for end-to-end object detection." ICLR. 2021.


3. Sequence Augmentation

3-1. The authors tackle the exploitation of prior knowledge of object detection in other methods. However, as mentioned in the second paragraph of section 2.3, the proposed method also exploits prior knowledge as a sequence augmentation and it is critical for the performance as shown in Figure 8.

3-2. On the other hand, a curve in Figure 8 is generated by “allowing the model to make more predictions.” However, there is no description of how to do that such as ignoring EOS until K steps. The authors should present or specify the exact performance for a model without Sequence Augmentation and not allow the mode to make more predictions.

4. EOS

Since the model uses a language model, it is possible to emit EOS between some coordinates or class tokens (e.g., y_min, x_min, y_max, EOS). How does the model deal with this? Does the model ignore the bounding box and stop the inference?


5. Attention Map

What does it mean by columns and rows in Figure 9 (b)? It seems that row means a different set of coordinates and classes (i.e., bbox) and the column means y_min, x_min, y_max, x_max, class. Although it is implicitly described in the context, I would recommend explicitly mentioning it.

6. Similarity among Coordinate Tokens

The authors only present the correlation matrix and say nearby coordinates have higher similarities in their token embedding. More explanation is needed. Also, the outputs have five (except EOS) different meanings (y_min, x_min, y_max, x_max, class). How is the correlation matrix generated? 

7. Change Figure 14

 Currently, the authors add hyperlinks in each ‘url’. However, the authors also consider people who read the paper in a hard copy or offline. presenting the original image in the paper or even deleting the captions will be better.



**[Minor]**
1. DETR (Carion et al., 2020) does not have a box regression sub-network although it utilizes  GIoU loss. Please change the description in the first sentence of page 9, “These detectors”.

2. Change 43.2, 44.9 in Table 1 to plain text.


**[Recommendation]**
1. Move ablation study for image augmentation to Supplementary material.
=> Image augmentation is widely used in object detection as the author mentioned and there is no need to incorporate it in the main paper. 

2. The author uses only 200 epochs for ablation study while the full model is 300 epochs, which is acceptable but still comparing at the same epochs (300) will be better.

3. In Section 3.3, “class (name)” seems “class + random.” For clarity, adding random or some description will be better.

4. Cross Attention Maps are really good. I suggest present instance or panoptic segmentation performance based on the cross attention map similar to DETR (Carion et al., 2020) paper.

summary_of_the_review: In general, the proposed method is novel and gets good results. I believe it will get a huge attention and lead to a big change in computer vision community. However, there are a few concerns as mentioned above, especially inference time. I hope the authors resolve my concerns.
review_title: null

Meta-review: This paper proposes an elegant approach to object detection where an encoder network reads in an image and a decoder network outputs coordinate and category information via a sequence of textual tokens. This method does away with several object detection specific details and tricks such as region proposals and ROI pooling. The paper received positive reviews from all reviewers who agreed that this formulation of object detection was novel and provided a new perspective that may transfer to other computer vision tasks. One common concern amongst reviewers was the slow inference time due to the sequential nature of the decoder -- and this concern was a central point of discussion between the authors and reviewers. My takeaway from this discussion is that this model is certainly slower than traditional computer vision models that can generate boxes in parallel. The slowdown however, is image dependent. Less cluttered environments require shorter output sequences. Moreover, such a model can easily be applied to concept localization, e.g. "Locate the horses", in which cases one can expect fewer objects of the desired category, and hence acceptable inference speeds. Importantly, the contributions of this paper are noteworthy in spite of the proposed architecture having the drawback of being slow. Given this, I recommend accepting this paper for its merits.
Decision: Accept (Poster)