Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a new object-centered framework to unify visual tasks including object detection, key point detection, and multi-label classification. Different from the previous Pix2Seq method, this paper divides the sequence generation problem into an object query generation sub-problem and an attribute sequence generation sub-problem. The method achieves good results on multiple visual recognition tasks on COCO.
strengths_and_weaknesses: Strengths:

- The idea of decoupling the sequence generation process into two steps is new and insightful. 

- The object query generation process allows the use of bipartite matching loss for supervision, which improves the optimization of the original pix2seq and helps the framework achieve competitive results with state-of-the-art methods.

-  The method is evaluated on multiple popular benchmarks and achieves good results. Ablation studies in Section 4.4 make the paper solid.

Weaknesses:

- Since the method uses bipartite matching loss from DETR [4] for supervision and generates many redundant queries (100 queries for each candidate category according to Appendix A) like [4], I think the overall framework can be viewed as an improved DETR with class conditioned query generator, which is close to several previous methods like [28, 43, 26, 49]. The object-centered design also makes the method less general compared to the concurrent work pix2seq v2 [r1] which can be extended to the image captioning task.

-  The two-step design makes the necessity of the sequence generation framework questionable. The sequence generation task in Pix2Seq is design to unify vision and NLP tasks. However, due to the existence of the Prompted Visual Indicator and bipartite matching loss, the proposed method is totally different from sequence-to-sequence autoregression framework in NLP. Therefore, is that possible to further simplify the General Sequence Predictor as a 2-token transformers model where the object query is combined with detection and keypoint task tokens as the inputs and use MLP heads like DETR to obtain the final prediction? 

- The paper emphasizes a unified framework is designed for various visual tasks. However, the experiments didn't show the potential of the proposed framework to perform multiple challenging tasks like [r1]. Is it possible for the proposed framework to perform general object detection and keypoint detection with a single model? Will the model benefit from joint training?

[r1] A Unified Sequence Interface for Vision Tasks, arXiv 2206.07669
questions: Overall, I think this is a good paper with several new and insightful technical contributions and decent results. Although I still have concerns about some designs, I think the proposed object-centered framework might be a new and interesting direction for several visual recognition tasks. Therefore, I lean toward accepting this paper. This paper can be stronger if the questions mentioned in the Weaknesses subsection  can be addressed.    
limitations: The limitations and potential societal impact of the method have been discussed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces a unified architecture similar to M-DETR, but takes object class labels as input, and outputs three components in the same architecture: 
1. the existence of object for multi-label classification 
2. the box coordinates for object detection 
3. keypoint coordinates for human pose estimation. 

The key component of this unified architecture is the sequence decoder, from which the first four outputs are always trained to be box coordinates and the rest outputs are trained to be human pose keypoints. 

strengths_and_weaknesses: The novelty of the paper is limited. The paper claims a "unified framework" (e.g. Figure 1) and "a wide range of vision tasks" (L91) but the experiments only show the tasks of outputting spatial coordinates (box coordinate and keypoint). While some recent work OFA (I’m aware it was not officially published at the submission time) shows a seq2seq framework can be applied to a much wider range of tasks including detection, caption and VQA. Overall I mean more vision tasks are required to support the claim of a “unified framework”. Second, the authors design an architecture that is similar to DETR-family, and the previous work M-DETR uses the architecture for language grounding tasks. This paper takes the class labels as input and the model outputs the bounding box, which feels more like a simplified version of a visual grounding task rather than object detection. Can the authors elaborate more on the difference between the proposed setting and visual grounding tasks?

The overall quality of the paper is good. The experiments are clearly explained and discussed. However, I think the comparison with previous works (Table 1) is unfair since the method uses a retention policy to pre-filter some probable object categories while the normal detection method does not benefit from this. Can the authors discuss this comparison more?

In general, the paper is very clearly written and well organized. The figures are clear and easy to understand. 

The experiments in object detection show the proposed method outperforms previous methods but due to the retention policy (mentioned above), I doubt the fairness of this comparison and more discussion is required. The experiments in human pose estimation show the proposed method is effective when compared with previous works. 

questions: 1. Are the Figure (3) a & b actually the same figure? I.e. the x-axes have one-to-one mapping and can be plotted in the same figure. 
2. What if there are multiple objects of the same category in the image? From Figure 2 it looks like there are 3 object queries l for each class embedding output. Does it work if the image has more than 3 people? Could the authors add more details about how to handle these multi-instance cases? 

limitations: The paper discusses three limitations: (1) not exploring the design of the feature extractor, (2) not exploring other tasks, (3) only taking object class as input and not exploring more detailed indicators. I agree with these limitations as potential future works, but I think point (2) has a higher priority given the paper claims a "unified framework". 

Additionally, I think one limitation of the model is that the object categories are limited by the input categories, and the model cannot discover new categories.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper is a nice try towards unified interface for various object related computer vision tasks. It use image and class as prompt, achieving SOTA detection and pose estimation performance in the same interface. The architecture designs of the Prompted Visual Indicator and Sequence Predictor are sound. The major debates lie in the usage of class as prompt and the object-level query. These two might be beneficial to get SOTA object-level tasks, but might also be the blocker for more general vision task interface.
strengths_and_weaknesses: Strengths:
1. This effort addresses an ambitious goal of getting a unified interface for computer vision tasks. Obj2seq unifies object detection, pose estimation and image classification.
2. The designed architecture achieves SOTA performance on object detection and pose estimation.

Weaknesses:
1. Class as prompt blocks its way to generalize to open-vocabulary applications. The goal of the unified interface is to utilize the scaling law of large scale model to utilize large dataset and address real-world open-vocabulary challenges.
Debatable:
1. The architecture keeps the object query, while pix2seq has already gotten rid of that. This is debatable. I don't have an obvious answer whether that part is necessary to achieve SOTA performance on several object-level tasks considering object is the unit that interaction or reasoning works on.
questions: The limitation of Pix2seq mentioned in this paper that "Pix2Seq is not aware of the desired targets before inference, and the output sequence might become extremely long under sophisticated scenes." This is not convincing enough. We could argue this actually shows the flexibility of Pix2seq. May I bother the author to elaborate more on this? What are the specific scenarios Pix2Seq may meet trouble and Obj2Seq is better.
limitations: Yes, the author mentioned the limitation of this work.
Class as prompt is not only a limitation to satisfy finer requirements, but also cause trouble on open-vocabulary applications.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new framework for vision tasks, called Obj2Seq. It takes prompts of classes as input then filters the non-exist categories and keeps the most confident K' categories for each image by the Prompted Visual Indicator module. After that, they will be passed to the Object Transformer Decoder module together with the encoded image features to get the object queries. The final General Sequence Predictor takes these queries as input to generate a sequence of desired results, e.g., the location (x, y), size (w, h), or even the other for fine-grained information like keypoints (x_nose, y_nose, x_eye1, y_eye1, .....). The experiments on MS COCO demonstrate that the proposed Obj2Seq can achieve comparable results with other task-specific methods.

strengths_and_weaknesses: [+] This paper proposes a new framework for object-level visual prediction tasks, e.g., object detection, multi-label classification or human pose estimation.
[+] According to the experimental results, the proposed Obj2Seq can achieve comparable performances on each task with their task-specific methods.

[-] However, I don't think Obj2Seq is a unified vision framework for most vision tasks, which is mainly designed to solve the object-level prediction. In my opinion, the capacity of the proposed Obj2Seq is equal to the conventional Mask R-CNN framework, because Mask R-CNN is also able to equip new prediction heads based on the downstream tasks. Therefore, I prefer to consider Obj2Seq as another version of "Mask R-CNN" rather than a unified framework for vision tasks. To better understand what a unified vision framework is supposed to look like, please refer to the [1, 2], which are capable of both localization tasks and understanding tasks. Note that I'm NOT suggesting the authors to compare the proposed Obj2Seq with these methods, as [2] and this paper are contemporary works. What I want to say is that the proposed Obj2Seq is far from a unified vision framework. 


[1] Lu, Jiasen, et al. "12-in-1: Multi-task vision and language representation learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[2] Zhang, Haotian, et al. "GLIPv2: Unifying Localization and Vision-Language Understanding." arXiv preprint arXiv:2206.05836 (2022).
questions: According to the authors, the main contribution of this paper is to provide a unified visual framework, but I only perceive it as an object-level prediction framework, which is equivalent to the conventional Mask R-CNN framework in terms of potential capacity. Therefore, I want to know some specific examples of tasks that can only be formulated by Obj2Seq but not Mask R-CNN during rebuttal.
limitations: The authors have already addressed the limitations.
review_title: null

Meta-review: metareview: The paper proposes an approach for formulating a few visual tasks as sequence prediction with class prompt. Reviewers are overall positive about the paper, especially the direction towards a unified vision model where the paper is exploring. However, it is also pointed out the paper should be more explicit about how the sequence is modeled with object queries and bipartite graph matching loss, which are significant differences from standard sequence modeling, as presented in language models, or Pix2Seq v1/v2. The authors should consider point out these differences in the abstract, Figure 1/2, to avoid misleading readers into thinking this is just like language modeling with autoregressive loss. Overall I’d recommend accepting the paper given it is a good attempt towards a unified vision model, but also encourage the authors to further improve the writing and clarify the sequence modeling part as mentioned above.

confidence: Certain
Decision: Accept