Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed a unified sequence interface (Pix2seq++) for computer vision tasks. The authors extend Pix2seq which can do only object detection to more vision tasks such as segmentation, keypoints, and image captioning and train a single model for 4 different tasks. Pix2seq++ unified the decoder outputs by treating these tasks as sequence prediction tasks. The authors use batch mixing instead of data mixing which implied data augmentation for different tasks. The proposed method achieves state-of-the-art performance on 4 benchmarks. 
strengths_and_weaknesses: [Strengths]
- The proposed method is the first one that unified "core" computer vision approaches into a sequence and shows it can achieve state-of-the-art performance on these tasks. This has a huge impact on the computer vision community, as the diverse nature of the computer vision tasks leads to diverse architecture designs, and the proposed model unifies/simplifies the design of the decoder. I believe this has a broad impact on vision and unified-model communities. 
- The proposed method achieves good performance on 4 of the benchmark compared to existing classical methods. 
- The authors explore different sampling strategies and mixing ratios for the tasks in the supplementary.

[Weaknesses]
- The keypoint tokenization is not clear to me. From L69, the paper mentions the key points are represented as a sequence of quantized image coordinates. However, from Figure 1, it seems the output is <label, ymin, xmin>, I wonder which one is the correct format for key points. 
- For instance segmentation task, how to make the polygon as a sequence -- is there any order to pick the start point, or is there any constraint if the sequence is too long? 
- In section 2.3, the authors mention different image augmentation for different tasks. Could the authors describe the difference between different image augmentation methods for different tasks? 
- In table 1, there is no citation for the Transformer-based caption method, and it would be better to report image captioning metrics such as cider score for the tasks. 
- For instance segmentation, I have a question and want to hear the author's insight. Some concurrent work uses VQ-VAE to serialize the image output such as (UViM and Unified-IO), however, the proposed pix2seq++ directly uses points. I wonder what is the authors' insights for different methods? It will be great to discuss this somewhere in the paper. 
questions: - What is the correct format for key points?
- For instance segmentation task, how to make the polygon as a sequence -- is there any order to pick the start point, or is there any constraint if the sequence is too long? 
- Could the authors describe the difference between different image augmentation methods for different tasks? 
- I wonder what is the authors' insights for different methods on instance segmentation (vq-vae vs. seq)? It will be great to discuss this somewhere in the paper. 
limitations: the authors adequately addressed the limitations and potential negative societal impact
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The work proposes an unified framework/interface for multiple vision and vision+language tasks, and achieves comparable performance on multiple tasks compared to other strong baselines. The work also shows benefits of multitask learning on some of the tasks. There are some concerns about novelty and whether the work is just incremental compared to previous work of pix2seq[6]. In addition training(pretraining) setup is slightly confusing and performance on few tasks are not so encouraging when compared to other state-of-the-art works. 
strengths_and_weaknesses: ### Strengths

- The proposed unified interface is simple and beneficial for a multitask learning. 
- Unifying the interface for these tasks enables same input and output for all tasks which makes it easily scalable to multiple tasks.
- Multitask learning shows improvements on some of the tasks.

### Weaknesses

- Authors say the models are initialized from pretrained checkpoint trained on the object detection task with the Objects365 dataset. Are the other baselines that are compared in Table 1 also initialized from pretrained object detection checkpoints on Objects365? If not then this is not a fair comparison. 
- The work seems to be incremental and just adding more tasks on top of previous work of pix2seq[6]. If the main message of this work is multitask learning, then it needs to be highlighted properly and the benefits of doing multitask with a unified framework needs to be analyzed in detail. What is the advantage of this setup compared to other frameworks that have task specific input and output parameters/heads and a common  shared base for all tasks? More specifically the work should ablate why a unified interface important compared to architectures having task specific input/output heads.
- Multitask learning for Instance Segmentation and Keypoint detection are not helping compared to specialized single task models. The work lacks details and analyses for the potential causes.
- Authors mention that one benefit of this work is that the loss function can be shared. Why is that? The work misses ablation study on the benefit of a common loss function.
questions: - How are the batches for different tasks ordered? Does that have any effect on the performance? Does curriculum learning help here?  If we order the batches for different tasks in a particular sequence, does their learning behavior and performance change? How do the training/validation plots look like? Does any task overfit when other tasks are still improving?
- How does multitask performance vary if we pretrain the model on other tasks than object detection? Is there a relation between high performance on obj detection in the unified setup to the fact that the model was initialized from pretrained weight for that task; however the other vision only tasks do not show good performance. 
- How does the current setup compare to other works in terms of training/inference efficiency? 
- What if the multitask trained model is again finetuned on individual tasks? Does multitask training as a pretraining step help?

limitations: Yes the authors have touched upon the limitations of this work. However I would encourage the authors to add more details about any potential negative societal impact this type of work can have.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a unified sequence-to-sequence generation interface for 4 core vision tasks: object detection, instance segmentation, keypoint detection and image captioning. The model is designed with a single encoder-decoder architecture, with no task-specific heads. To differentiate among different tasks, descriptive task prompts are added as the starting token to generate the output sequence. Experimental results demonstrate a proof-of-concept that such a unified interface can achieve competitive performance on 4 tasks evaluated, compared to some well-established task-specific models.
strengths_and_weaknesses: - Strength

In general, the paper is clearly written, easy to follow. Experimental results indeed show a proof-of-concept that this is possible to tackle vision tasks with a unified interface. 

- Weakness

One concern I have is that the only technical contribution of this paper seem to be extending pixel2seq to the 3 other vision tasks. 

- Minor comments

The important details on training and inference may be better to put into the main paper.
questions: - Results on training from imagenet-pretrained weights? This can be important to understand the current advantage over the compared baselines. The baselines seem to be all trained from imagenet-pretrained weights?
- More details on the segmentation inference strategy. How the multiple samples are generated during inference? What kind of sampling strategy are used? How was the threshold decided? 
- Any insights on the choices of  loss weights, as greedy search may not be feasible when scaling up the number of tasks.
limitations: Yes. The authors have discussed the limitations and societal impacts in Section 5.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper addresses the unification of 4 specific vision and vision-language tasks (namely object detection, instance segmentation, human pose estimation, and image captioning) with a transformer framework and a sequence modeling objective. The experiments show that their method achieves competitive results compared to well-established task-specific models.
strengths_and_weaknesses: Strengths:

1. This paper follows the promising trend of task-unification under a transformer framework with sequence modeling, and the authors extends Pix2Seq model to learn 4 specific tasks in COCO datasets. 
2. The proposed method simplifies the training objective and model architecture design.
3. With the nature of transformer architecture and seq. modeling, it allows text-prompting to activate desired functionality (although the text prompt is pre-defined). This helps create a friendly user interface to access the model.
4. Although the proposed model does not establish new state-of-the-art performance across the 4 specific tasks, the results are still very competitive. In my opinion, the results are promising as it could initiate a new research line unifying different computer vision tasks. The proposed model shows great potential for the ambitious goal of the all-in-one design; however, I also have several concerns about this paper, and they are listed below.

Weaknesses:

In NLP and vision-language fields, a single sequence modeling objective shows great advance as most of their tasks can be easily formulated as a sequence prediction task. However, for computer vision tasks, it remains unclear whether most of the vision problems can be solved with such a sequence modeling. While the authors demonstrated promising results on 3 selected vision tasks and 1 vision-language task, many other vision problems developed in the community are not clearly addressed. While it is really an open research question, the title of this paper "A Unified Sequence Interface for Vision Tasks" is indeed overclaimed. I would suggest rewrite as "A Unified Sequence Interface for COCO Dataset" to better reflect what you have proposed and what you have done in the paper. 

To be more specific, the authors claimed that they consider "core vision tasks", but task definition seems bias as it is unclear what should be core, and which should not. In my view, there are only 3 pure vision tasks (detection, segmentation, human pose estimation), and additionally the authors consider a vision-and-language (V+L) task (i.e, image captioning). This introduces many concerns. It is unclear why the authors didn't include other popular V+L tasks such as image-text retrieval and visual question answering as they are commonly evaluated in the foundation model literature. Similarly, there are many popular pure vision tasks ignored in this paper, such as image/video recognition, NeRF, etc. Overall, this gives the impression that the task unification is achieved just because of a careful selection of downstream tasks in the experiments. 

The authors compare their method with MaskRCNN, and show that their proposed method can additionally work for image captioning. But no comparison with V+L foundation models is provided. Since image captioning has been well-studied in the unification literature in V+L field, the main new thing here is probably about unifying 3 selected pure vision tasks. However, pioneer works (such as MaskRCNN, HRNet, and others) already unify the three tasks very well. It is unclear what is the advantage of the proposed method (i.e., using the sequence modeling for the three tasks). 

From modeling perspective, it remains challenging for transformers to learn fine-grained spatial relationship, and such spatial info is critical to the selected 3 tasks. The motivation of using sequence modeling for the three tasks is somewhat weak.

From the perspective of efficiency, the proposed unification removes the need of task-specific modules design for COCO dataset. However, due to the nature of autoregressive prediction, the proposed model will be slow. For example, the bottom-up approaches in human pose estimation literature can perform both multi-person pose estimation and instance segmentation at the same time with just a single forward-pass, but in this paper, the inference cost could grow linearly to the number of people. The inference interface is kind of restricted: (1) only a single task is supported at a time; (2) it predicts token-by-token slowly. 

From the perspective of training, the proposed method doesn't show great benefit. In NLP and V+L research, people use a single loss function, so that it is easier and scalable when performing large-scale pre-training. However, in this paper, the tasks considered require fine-grained annotations, and thus they do not have large-scale pre-training data (except for image captioning). For the considered downstream dataset, it is unclear if single-objective is more effective than multi-objective training.

Overall, the model design is neat and elegant. But it is not well-grounded to the considered downstream tasks. This paper is more like a system paper integrating a few known components together to achieve an expected "okay" performance. Considering the technical novelty, the contribution of this paper is relatively weak. 

The proposed method may bring significant advantage with large-scale pre-training. But given the current paper content, it is difficult to reach a conclusion at this point.
questions: see weakness
limitations: N/A
review_title: null

Meta-review: metareview: Four reviewers provided reviews for this submission. Several reviewers felt that the idea to unify core vision tasks into a sequential output format is interesting and an entirely new approach and can have a large impact on how we train vision models in the future. There were a few concerns discussed between reviewers and authors. One concern was the comparison to past works that pre-trained on ImageNet vs the proposed model that was pre-trained on Objects365. The second concern was differentiating this work with Pix2Seq. In my opinion, the authors were able to answer both questions well. Overall, given the positive reviews, novelty of the work, potential to cause a significant shift in the approach of future modeling and discussion, I recommend acceptance. 

confidence: Certain
Decision: Accept