A Unified Sequence Interface for Vision Tasks
Authors: Ting Chen
Saurabh Saxena
Lala Li
Tsung-Yi Lin
David J. Fleet
Geoffrey Hinton
Venue: NeurIPS 2022 Accept (2022)
Keywords: generalist model
computer vision multi-task
sequence modeling

Abstract: While language tasks are naturally expressed in a single, unified, modeling framework, i.e., generating sequences of tokens, this has not been the case in computer vision. As a result, there is a proliferation of distinct architectures and loss functions for different vision tasks. In this work we show that a diverse set of "core" computer vision tasks can also be unified if formulated in terms of a shared pixel-to-sequence interface. We focus on four tasks, namely, object detection, instance segmentation, keypoint detection, and image captioning, all with diverse types of outputs, e.g., bounding boxes or dense masks. Despite that, by formulating the output of each task as a sequence of discrete tokens with a unified interface, we show that one can train a neural network with a single model architecture and loss function on all these tasks, with no task-specific customization. To solve a specific task, we use a short prompt as task description, and the sequence output adapts to the prompt so it can produce task-specific output. We show that such a model can achieve competitive performance compared to well-established task-specific models.

A Unified Sequence Interface for Vision Tasks
Ting Chen
Saurabh Saxenal Lala Lif
Tsung- Yi Lin"
David J. Fleet
Geoffrey Hinton
Google Research, Brain Team
(iamtingchen, srbs, Ggoolleegogge..oom
Abstract
While language tasks are naturally expressed in single, unified, modeling frame- work, i.e., generating sequences tokens, this has not been the case computer vision. As a result, there is proliferation of distinct architectures and loss func- tions for different vision tasks. In this work we show that a diverse set of "core" computer vision tasks can also be unified if formulated in terms of shared pixel- to-sequence interface. We focus on four tasks, namely, object detection, instance segmentation, keypoint detection, and image captioning, all with diverse types of outputs, e.g., bounding boxes or dense masks. Despite that, by formulating the output of each task sequence of discrete tokens with unified interface, show that one can train a neural network with a single model architecture and loss function on all these tasks, with task-specific customization. To solve specific task, we use a short prompt as task description, and the sequence output adapts the prompt so it can produce task-specific output. We show that such model can achieve competitive performance compared to well-established task-specific models.
1 Introduction
Training a single neural network model capable of performing myriad tasks is major step towards artificial general intelligence. In recent years, with the rise of big language models [34] 35 Transformers [4]] many different language and related tasks are unified under single modeling framework, where language model is trained to predict the solution (in text tokens) given prompt of a task description (also in text tokens). This is only possible because these tasks (both task description and solution) can be expressed in the same, rich language interface.
using
This can be naturally extended to some vision tasks such as image captioning or visual question answering where the solution is given in natural language, but the majority of "core" computer vision tasks have diverse outputs that are not readily expressed in terms of natural language. The object detection task produces a set of bounding boxes and their corresponding class labels, often associated with scores for ranking. The output for instance segmentation is a set of segmentation masks corresponding to image regions. The output of keypoint detection
of keypoints in an 15 have developed specialized architectures
image. As such, existing methods [33 37 15 28 and sophisticated loss functions for each of these complex tasks.
An ambitious goal, in the pursuit of artificial general intelligence, is a simple interface that allows one express seemingly disparate vision tasks in unified framework. This would simplify the design of architectures and loss functions for new tasks. It would enable greater degrees of fea- ture/representation sharing across many different tasks, thereby avoiding the need for sophisticated
Equual contributions. Work done at Google.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Yrur Xya 653 653 Tea
[dotect]
We8 syed y x xyx57
[Segmend You H0
Pix2Seq (V2)
X5. 18u
Xea 57
[Keypoin]] 334
Kay E9,4 #5.8
person working shop mopeds
Deeccribe)
Figure 1:
An illustration of the proposed framework. An image and sequence task prompt is given, the model produce sequence of discrete tokens corresponding the desired output.
output head for each task. would also facilitate adapting of existing models to new tasks, and potentially unlock new capabilities with zero few demonstrations.
To this end, we propose an approach to unify four seemingly different vision tasks in single pixel-to-sequene interface. In effect, this is an extension of Pix2Seq (7] for object detection to broader set of tasks. As proof of concept, we focus on four core vision tasks, namely, object detection, instance segmentation, human keypoint detection, and image captioning. We first show how to unify these tasks into a single shared interface, and then train neural network with shared architecture and objective function. To solve a specific task, instead of using specific head for that task, we use a prompt to specify the task, and the sequence output adapts to the prompt so can produce task-specific output given the task description. This makes multi-task learning more efficient and scalable. We conduct experiments on challenging COCO dataset, and show that simultaneously solve all four tasks well, without specialized architectures loss functions.
2 Approach
In our approach, we cast computer vision tasks as one of translating pixel inputs (along with some descriptions of the task) into sequences of discrete tokens (see Figure D As proof of concept, focus on four core vision tasks: object detection, instance segmentation, keypoint detection, and image captioning; but we believe it is relatively straightforward to include many more tasks.
2.1 A unified interface with tokenization
The vision tasks we consider are diverse, and traditionally have been formulated quite differently. Object detection requires the model to produce bounding boxes for all objects without duplication. Instance segmentation requires the model produce a dense pixel-wise mask for each identified object instance. Human keypoint detection requires the model to generate points corresponding to specific positions of landmarks body parts for person instances (e.g., head, eyes). Image captioning requires the model to produce sequence of words corresponding to a natural language description of the image. Given the significant differences in the form of the outputs, customized models with specialized architectures and loss functions are designed for each task.
To solve these tasks using single model, we advocate the transfornation/ookeniiz and outputs into a unified interface. In this work, we propose sequence interface for the purpose, where both task descriptions and outputs expressed as sequences of discrete tokens:
of task inputs

---

Task prompt
Task output
Output visualization
Yni Xmi
[Detect]
Y65x 1653 Xnax7444
person
Input image
[Segment] Ymin#505 503
ye= Xp=599
Xain 518 YBax=805
Yj=788 y1=664
Ymax 892 Motocycle
[Keypoin]] Ymin
Nose Yeu=1 Xmn left eye
Xmin=37 =633 Xmax 444 person ......
person working mechanical shop with two mopeds outside.
[Describe]
Figure 2:
An illustration of sequence interface for four tasks. The model takes input image, task prompt tokens and produce task output tokens, which be decoded/ddtokeenizee into required task output for visualization.
For object detection, we follow 7 and convert bounding boxes and object descriptions into a sequence of discrete tokens by quantizing the continuous image coordinates. Specifically, object is represented as a sequence of five discrete tokens, i.e. [ymin Tmin ymax Tmax c), and multiple objects randomly ordered each time training image is sampled and serialized into single sequence.
For instance segmentation, instead of per-pixel masks, we predict the polygon corresponding to the instance masks as a sequence of image coordinates conditioned on given object instance. Again, we quantize the coordinates into discrete tokens. And to turn polygon into a sequence, we randomly select starting point for the start token each time training image is sampled. If there are multiple polygons for the same instance, we concatenate sequences of individual polygons with separator token in between, so that every instance has single corresponding sequence.
For keypoint prediction, we predict a set of keypoints as sequence of quantized image coordinates conditioned on a given person instance. Specifically, the sequence of keypoints can be encoded as
[/keypoint| I, -Kkeypoit 1, 1/keypoitt 2; 4keyyoint 2, ..... One may also use keypoint label (e.g., nose, let eye, right eye) before each (y, x)) -ro-cdnnnssssss so their ordering does not need be fixed but we opt for simplicity given that there are only small fixed set of 14 person keypoints in the COCO dataset we consider. When certain keypoints are occluded, their coordinate tokens are replaced with special occlusion token.
For captioning, we directly predict text tokens given caption is sequence of discrete tokens.
It is worth noting that all four tasks share single vocabulary. The specific prompts and output sequences are illustrated in Figure 2
2.2 Unified architecture and objective function
We need a flexible and expressive architecture that can deal with image input and sequence output with complex semanties. Thus we follow 77] and use an encoder-decoder architecture, with an image encoder and sequence decoder. The image encoder perceives pixels and maps them into
3

---

Y2
Decader
Xinag
Y pred
Encoder
with causal mask
Y constructed seq
Objective (weighted next token prediction)
Figure 3: An illustration of our architecture and training objective. Note that Y consiructed sey encapsulates both task prompt tokens and task output tokens. Token weights are set to zero the target token within the prompt so the model is only trained to predict desired output tokens.
hidden representations, which can be instantiated as a ConvNet (23 22 [4], Transformer 44 or their combination
4 The transformers-basee sequence decoder, widely used in modern language 35 generates one token time, conditioned on the preceding tokens and the encoded image representation. This removes the complexity and customization in architectures of
modeling A1
modern neural networks for these vision tasks (such as per-task specific heads or necks [15] 30)).
Unlike 7 where the decoder produces the output tokens directly for the single object detection task, here also conditions on a task prompt so that the model can produce outputs adapted to the task of interest. During training, we concatenate both prompt and desired output into a single sequence, but leverage a token weighting scheme to ensure that the decoder only trained to predict the desired output but not the prompt tokens. During inference, the prompt is given and fixed, so the decoder only needs to produce the rest the sequence. Similar to 77, the training objective is to maximize the likelihood of tokens conditioned on an image and preceding tokens, i.e..
maximize
wy log P(y; F. y1-j-1)
where æ is the input image, is a length-L sequence associated with æ. As mentioned, the initial part of the sequence y is a prompt, for which we set the weight wy to zero so it is not included the loss.
2.3 Training
Each task has its own paired image-sequence training data. There are two ways one can combine tasks and perform the joint training.
Data mixing. We can create dataset with mixed image-sequence pairs drawn from different tasks, balanced account for different dataset sizes and task difficulties. This construction extremely simple conceptually. but image augmentations can be difficult to incorporate as they may also require change to their associated sequences non-trivia ways.
Batch mixing. For each batch, we can sample images with annotations for single task, perform image augmentations appropriate for this task, and convert the augmented data into image-sequence pairs. 'he model computes the loss and gradient for each task separately, and we can combine gradients from task-specifie batches with appropriate weighting.
Algorithm Training based on data mixing
A Agorithm 2 Training hased on batch mixing
1: Tokenize annotation into sequences of tokens, Mixing images sequences from tasks,
1: Sample batches of data from all tasks,
2:
Tokenize annotation into sequences of tokens,
3: Sample a batch, compute the loss, and update the model.
3:
Compute the loss for each task, aggregate their gradicnts, and update the model.
Algorithm D and give summaries of both training strategies. We use batch mixing strategy in this work as simplifies handling of image augmentations, but are hopeful that data mixing

---

can be employed in the future to further simplify the pipeline and allow more tasks to be added straightforwardly.
Both data mixing and batch mixing require that we specify the portion or weighting for each task. a greedy strategy by adding one task at time. Every time
This is empirical matter, and when we add a task, we adjust the weighting of the new task while keeping the relative weighting among the existing task fixed. We fix the sum of the weights across all tasks be one.
2.4 Inference and de-tokenization
At inference time, we sample tokens from the model likelihood, given a prompt at the start of the sequence, i.e., P(yj x. 31-j-1). We currently use nucleus sampling [16] but other techniques such as beam search could also be used. Once the tokens are generated, they can be decoded for each task. In the same way that different tasks require specific tokenization schemes generate token sequences, the decoding (de-tokenization) process is also specific to each task. A more detailed description of inference decoding for each task is given below.
For bounding boxes, following 77. we split predicted sequences into tuples of 5 tokens to get coordinate tokens and class token, and dequantize coordinate tokens get the bounding boxes. For instance segmentation, we dequantize the coordinate tokens corresponding each polygon, and then convert them into dense masks. The model not trained with any geometry-speciffe regularizers per se, and as such the output polygonal masks can be somewhat noisy. To reduce the noise we find it helpful to sample multiple sequences and then average the masks, followed by a simple threshold to obtain single binary mask.
For keypoint detection, we directly dequantize the image coordinate tokens of the keypoints.
For captioning, we directly map the predicted discrete tokens into text.
3 Experiments
3.1 Experimental settings and implementation details
We evaluate the proposed method on the widely used MS-COCO 2017 dataset 26). containing 118k training images and 5k validation images, spanning the four tasks we consider. An image in the dataset typically has annotations for object bounding boxes, segmentation masks for object instances, keypoint for person instances, and a few captions. Following 77 we use a Vision Transformer (ViT-B) encoder [1 41, and a Transformer autoregressive decoder [41] This model has a total of 132M parameters. To initialize the model, we use pretrained checkpoint from 77] trained on the object detection task with the Objects365 dataset [39]; this is useful as COCO relatively small and our model has less task-specific prior. For training a learning rate of Le -4, and we train the model for 100 epochs. We use a single vocabulary of 35K, with 32K text tokens, IK coordinate quantization bins, and few other class labels. We use maximum sequence length of 512. Our backbone model is pretrained with 640x640 image size, and is fine-tuned in 640x640 or x024x 1024 resolutions.
COCO, we use batch size of 128 images,
Object detection. We follow and use sequence augmentation during training, and use class token probability at inference time for scoring. We also use scale jittering as in 7 (scaling images randomly without changing aspect ratio, crop a fixed size region randomly, and then pad to the maximum size).
Instance segmentation. We set the maximum points of polygons to 128. We find that asking the model generate multiple samples during the inference time and average the generated masks be beneficial. More specifically, when multiple samples are independently drawn, we convert each of them into semantic mask for the prompted object. We then average the masks by setting a (50%) threshold, and pixels with more than 50% times of being on will be selected for that instance. We find that 8 samples are sufficient to provide good performance (~6 AP better than using a single sample), and beyond 12 samples we do not see performance boost. Additionally, during inference, we also evaluate on the cropped regions of the image containing the prompted object instance, by replacing the original input image with new image only containing the cropped region. With smaller image size of 640 x600, this yields 1.3 AP improvement, but with larger image size of x024x 1024, this does not seem to help much.
5

---

Keypoint detection. We train and evaluate on cropped regions of the image containing person instances (following the common practice in the community). During training, these regions are provided by ground-truth annotations, and during inference these regions are provided by the object detection model. We choose this region to be twice the size of the provided bounding box. We find that this works better than training with larger crop size or cropping to the exact bounding box. Using our optimal crop we get ~9 AP improvement over using an extremely large crop (~~00 times the box size which can be considered a close approximation to using the entire image). We also use special token to represent invisible token coordinates in the quantized sequence. At training time we small loss weight of 0.1 for these tokens. While using larger weight doesn't 't affect AP much (lower by at weight 1.0) using a weight of 0.0 does much worse (12 AP lower). At inference time invisible tokens are replaced with the model's best guess of the keypoints' coordinates.
Four-tasks joint training. We use mixed weighting of 0.1782, 0.7128, 0.099, 0.01 for object detection, instance segmentation, image captioning, and keypoint detection respectively. This set of weight is searched greedily by adding one task at a time (while keeping the weighting ratio of existing tasks unchanged), Ablations task weighting are shown in the quantitative results below.
Baselines. We compare with a few well-known task-specific baselines. For object detection we compare with a strong 2-stage detector, Faster R-CNN (37]. and a more recent Transformer-ban detector, DETR 4. Both Faster R- R-CNN and DETR use task-specific priors in their design, such as non-maximum suppression in Faster R-CNN and bipartite graph matching with generalized intersection- cinrrcction-ove--uiinn in DETR. Due to their customized architectures and loss functions, extending them to wider spectrum of tasks is non-trivial and may require a new model design. Mask R- CNN 15 advocates a design to extend Faster R-CNN to incorporate segmentation masks and keypoints. While Mask R-CNN is able to perform three out of our four tasks, it still requires the same set of task-based customizations as in Faster R-CNN. We also consider an improved version of Mask R-CNN with non-local architectures 43 which incorporates an attention mechanism, similar to Transformers. The above methods cannot do image captioning, so we train a Transformer- based caption model [40 32) which is specialized for the task. This model is similar to the proposed method trained for caption single task but is using superviid pretrained visual encoder [6] with a high dropout rate.
3.2 Quantitative results
Table 1: COCO results for object detection, instance segmentation and keypoint detection are expressed in terms of AP. For Image Captioning we report BLEU score. Single task results for instance segmentation and keypoint detection are based on detected bounding boxes from single task detection model.- indicates the model is not able to solve the task without modifications.
Object det. Instance seg.
Keypoint det. Captioning
Faster R-CNN 37
42.0
Faster R- CNN++
67
44.0
DETR
4
44.9
Mask R-CNN
15
39.8
37.1
63.1
Mask R-CNN (non-local)
45.0
40.3
66.5
Transformer-based captioner
32
34.3
Pix2Seq v2 single task (640x640)
43.8
37.3
68.0
33,9
Pix2Seq v2 single task (1024x 1024)
45.6
38.7
67.4
34.0
Pix2Seq v2 multi-tasks (640x640)
44.2
36.9
65.0
34.3
Pix2Seq v2 multi-tasks (1024x 1024)
46.5
38.2
64.8
34.9
Our main results are summarized in Table where we report baselines and two variants of our model: (1) single task models where the model is trained on single task (still with the same architecture and objective function), so each task has its own network weights; and (2) a multi-task model, where a single set of network weights is used for all four tasks. We can see that despite without task-specific priors architecture and loss function, our model can still achieve competitive results for each individual task compared to strong specialized baselines (even with a smaller image size). When we train a single model on all tasks, our model is able to address these tasks relatively well, despite the model size being kept the same. We also observe that, with larger image sizes, the performances

---

generally improved. One exception is keypoint detection, which already uses cropped region of interest for detecting key points, thus scaling up the image size not necessarily helpful and can lead to overfitting in case of limited labeled data.
40
40
50
30
30
40
20
20
20-
20
Object dite on (4P)
Object detect (4P1
Instance segimercation (6P)
Object detect on (AP)
Instance sepmeraation (AP)
10
Image capiion ng (BLEUM)
Instance segmentation (AP)
Imagl capiioo ny (BLEUM
Keypo detection (AP)
00
62
0.8
0.01
0.05
02
0.4
0.8
3.01
0.02
0.05
0.3
0.5
Task weight (objact detection)
weight (mmgg captioning)
weight (Kaypoint ditection)
(a) Object detection instance (b) Detection and segmentation image captioning.
(c) Detection, segmentation and captioning vs. keypoint detection.
segmentation.
Figure 4: Performance with different task weighting when new task is added into an existing task mixes.
Figure 4 shows how select appropriate loss weighting for each task using a greedy strategy. More specifically, we first search the weight ratio between object detection and instance segmentation and results are shown in Figure 4a| We observe that for relatively wide range of weighting ratios, the performance of both tasks are near their peak, so we simply choose the 2:8 weighting ratio for these two tasks. After that, we add image captioning task, and the performances under different weighting of the captioning task can be found in Figure (4b] where we find that the 9:1 weighting ratio for existing tasks and image captioning tasks to be appropriate. Finally, adding the keypoint detection task, in Figure 4c we find its weight can be set relatively small and we choose to use 0.01.
3.3 Qualitative results
To demonstrate the capability and performance of our model in more visual and intuitive way, we show the outputs from our multi-task model on selected images from the COCO validation set, for each of the four tasks, i.c., object detection, instance segmentation, keypoint detection, and image captioning. Figure 5 shows results for the object detection task. The model successfully detects objects of different sizes in cluttered scenes with significant occlusion. Empirical results on instance segmentation and keypoint detection are shown in Figures 6 and 7 For both tasks, the multi-task model produces well localized and accurate predictions. We also demonstrate some captions generated by the model in Table 2 With these results, we note that our model has not been pre-trained using large-scale image-text datasets, which expected to significantly improve the captioning performance of the model.
4 Related work
Decoding visual concepts: Image understanding involves extracting visual concepts from images. The formats of these concepts vary according to the given task. Image captioning uses a sequence of words to describe an image (9. Object detection, on the other hand, represents objects with labels and bounding boxes. Depending on the granularity of localization, visual concepts can be expressed as boxes, pixel segmentation, or keypoints (26). Decoding localized visual concepts often requires tailored methods. For example, image segmentation uses per-pixel classification. Object detection uses sliding window with non-maximum suppression to detect boxes. Person keypoint detection uses part models to assemble detected parts into whole body B.
Recently, DETR 4 is proposed as an end-to-end object detection approach based on a Transformer decoding scheme (removing complexity on bounding box proposal and non-maximum suppression). MaskFormer [10] further shows that object detection and segmentation can share the same decoding scheme. Pix2seq [7] demonstrates that boxes and labels can be treated as a sequence of discrete
tokens, thereby sharing the same training and decoding interface language models [33 35). In our
7

---

Figure 5: Visualization the object detection results, with predicted bounding boxes on the input images.
work, we push the envelope further in the unification of language and different visual localization tasks to share the same interface, architecture and training objective.
Generalist vision models: Learning a generalist model capable of performing multiple tasks is widely assumed be a path toward general intelligence. visual recognition, multi-task learning has shown great success by sharing backbone model, followed by multiple independent heads [I5]
19 30).
Models with a shared backbone can learn general features which are transferable across tasks when scaling up with training tasks, model capacity and data (20) 24 [2] Nevertheless, often the task specific backbone models are designed carefully, particularly for tasks that require accurate localization
(38 27
With the invention of Transformers 41 recently being adopted for image classification [T]. the research community has seized on the opportunity to unify the backbone design for vision tasks 2g 25 Perceivers [8 [7 and OFA (42 demonstrate an architecture for multi-task and multi- modal across vision and language. Notably, OFA designs a unified sequence-to- se-quueece decoding architecture for both language and object detection tasks. Flamingo [I] and related methods also focus on universal API that produces a natural language output for a variety of tasks given image input. This line of work shares a common motivation to our work in this paper, however they focus on higher level tasks for which natural language inherently the desired output. In this paper

---

WINNESS DRAUGHT
Figure 6: Visualization of the instance segmentation results, with predicted semantic masks overlaid on the input images.
Figure 7: Visualization of the Human keypoint detection results, with predicted stick figures on the input images.

---

Table Image captioning results.
A group of teddy bears sitting next to each other.
Three teddy bears sitting on blanket.
A group of teddy bears sitting on a blanket with bowls of food.
A herd of elephants standing inside of fenced in area.
A group of elephants standing in a fenced area.
A herd of elephants standing behind a fence.
A man riding skateboard over block of cement.
man doing trick on skateboard in the street.
A man flying through the air while riding a skateboard.
A row of motorcycles parked on grass covered field.
A motorcycle with a helmet on the side of it.
A motorcycle parked in a grassy area with other motorcycles.
demonstrate that one can express a variety of "core" computer vision tasks in universal interface, and the learned model exhibits strong grounding capability of the tokens they produce actual visual concepts. Concurrently to our work, Gato [36] unifies series of vision and control tasks into single sequential prediction problem, and UViM [1 and Unified-IO [1 propose using learned discrete codes for unifying set of vision tasks.
5 Conclusion
In this work, we explore a unified sequence interface for tackling a diverse set of "core" vision tasks, where both the task description (prompt) and task output expressed discrete sequences of tokens. This a significant departure from conventional norms of multi-task vision models that both architecture and loss functions shared among the tasks. We show that such model can achieve competitive performance compared to well-established task-specific models.
Our work is not without limitations. Due to the significant departure from conventional approaches, we believe both architectures and other training techniques can further improved challenge the state-of sar---ffthheeee the-art of specialized systems. We also believe model significantly benefit from scaling up, both in pretraining on larger datasets (e.g.. image-text pairs) and/or using larger model sizes. Another limitation is the inference speed can be potentially slower (for longer sequences particularly) compared to the specialized systems as our approach is based on autoregressive modeling. There few ways improve the efficiency, including using non-autoregressive sequence modeling (which we leave as future work). In this work, we exploit parallel querying for speeding up our model inference. For example, predicting multi-person poses be done independently by prompting the model with independent bounding boxes (detected by the model itself or pre-given), so the only sequential prediction is limited to a single person with few keypoints. The same strategy can be applied to instance segmentation as well.
While the optimal implementation of unified interface still requires more research and the sequence interface explored this work is only one potential implementation, we believe the interface of how different tasks are formulated would play increasing important role in general-purpose intelligent systems going forward.
10

---

Acknowledgements
We specially thank Wei Li for their helpful feedback on the initial draft. We also thank Xiaohua Zhai, Alexander Kolesnikov, Lucas Beyer, Neil Houlsby, Simon Kornblith and Mohammad Norouzi for some early discussions.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han. Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. arXiv preprint or Xiv:2204. 14198, 2022.
[2] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv: 2005. 1...0055..1.... 14165, 2020.
[3] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi pelltipperoonnae 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291-7299, 2017.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, pages 213-229. Springer, 2020.
[5] Lluis Castrejon, Kaustav Kundu, Raquel Urtasun, and Sanja Fidler. Annotating object instances with polygon-mnn Proceedings the IEEE conference on computer vision and pattern recognition, pages 5230-5238, 2017.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International Conference on Machine Learning, pages 15597-6077. PMIR, 2020.
[7] Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton. Pix2seq: language modeling framework for object detection. arXiv preprint 2tti::20009.088822 2021.
[8] Wuyang Chen, Xianzhi Du, Fan Yang, Lucas Beyer, Xiaohua Zhai, Tsung- Yi Lin, Huizhong Chen, Jing Li, Xiaodan Song. Zhangyang Wang, et al. A simple single-scale vision transformer for object localization and instance segmentation. arXiv preprint urXiv:2112.097477 2021.
[9] Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. 2015.
[10] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34, 2021.
[11] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[12] Golnaz Ghiasi, Barret Zoph, Ekin Cubuk. Quoc Le, and Tsung- Yi Lin. Multi task self self-training for learning general representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[13] Ross Girshick. Fast r-enn. In Proceedings of the IEEE International Conference on Computer Vision, pages 1440 1448, 2015.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[15] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference Computer Vision, pages 2961-2969, 2017.

---

[16] Ari Holtzman. Jan Buys. Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text 2019.
degeneration. arXiv preprint
[17] Andrew Jaegle, Sebastian Borgeaud, Jean- Can-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding. Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2 14795, 2021.
[18] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference on Machine Learning, pages 4651 16644, PMLR, 2021.
[19] Tasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid- and high- level vision using diverse datasets and limited memory. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 6129-6138, 2017.
[20] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. European conference on computer vision, pages 491-507. Springer, 2020.
[21] Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: A unified modeling approach for vision with learned guiding codes. arXiv preprint arXiv:2205. 10337, 2022.
[22] Alex Krizhevsky, Tlya Sutskever, and Geoffrey E Hinton, Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 1097-1111 2012.
[23] Yann LeCun, Bernhard Boser, John S Deaker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, I (4):541-55) 5511, 1989.
[24] Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint
2021.
[25] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. arXiv preprint arXiv:2203. 1652,7 2022.
[26] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740-755. Springer, 2014.
[27] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference Computer Vision and pattern recognition, pages 21 17-2125, 2017.
[28] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, pages 2980-2988. 2017.
[29] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[30] Jiasen Lu, Vedanuj Goswami. Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2020.
[31] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206. 089106 2022.
[32] Yingwei Pan, Ting Yao. Yehao Li, and Tao Mei. X-linear attention networks for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10971- 10980, 2020.
[33] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[34] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
12

---

[35] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint 19x.::::100000000
2019.
[36] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg. Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. arXiv 2rXi::2005.06175 2022.
[37]
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28:91-99, 2015.
[38] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assister intervention, pages 234-241. Springer, 2015.
[39] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: large-scale, high-quality dataset for object detection. Proceedings of the IEEE/CVF international conference on computer vision, pages 4430-8439, 2019.
[40] Piyush Sharma, Nan Ding, Sehastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2556-2565, 2018.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008, 2017.
[42] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through simple sequence-to- sequence learning framework. arXiv preprint arXiv:2202.03005, 2022.
[43] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794-7803, 2018. 8.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Conclusion section
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work at its current form does not increase the risk of negative social impacts of those existing specialized systems.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] We will opensource our code at
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] Some the experiments are expensive to run multiple times, and the standard emrrr are usually pretty small.
13

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] It's trained on 32-129 128 Cloud TPUs. Depending on architectures, and tasks, generally takes 4-12 hours.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] It is pretty obvious from the dataset website, and it's well known dataset.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you' 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "mAgy3x1hyQJ", "reviewer": "Reviewer_vjx3", "summary": "This paper addresses the unification of 4 specific vision and vision-language tasks (namely object detection, instance segmentation, human pose estimation, and image captioning) with a transformer framework and a sequence modeling objective. The experiments show that their method achieves competitive results compared to well-established task-specific models.", "questions": "see weakness", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper follows the promising trend of task-unification under a transformer framework with sequence modeling, and the authors extends Pix2Seq model to learn 4 specific tasks in COCO datasets. \n2. The proposed method simplifies the training objective and model architecture design.\n3. With the nature of transformer architecture and seq. modeling, it allows text-prompting to activate desired functionality (although the text prompt is pre-defined). This helps create a friendly user interface to access the model.\n4. Although the proposed model does not establish new state-of-the-art performance across the 4 specific tasks, the results are still very competitive. In my opinion, the results are promising as it could initiate a new research line unifying different computer vision tasks. The proposed model shows great potential for the ambitious goal of the all-in-one design; however, I also have several concerns about this paper, and they are listed below.\n\nWeaknesses:\n\nIn NLP and vision-language fields, a single sequence modeling objective shows great advance as most of their tasks can be easily formulated as a sequence prediction task. However, for computer vision tasks, it remains unclear whether most of the vision problems can be solved with such a sequence modeling. While the authors demonstrated promising results on 3 selected vision tasks and 1 vision-language task, many other vision problems developed in the community are not clearly addressed. While it is really an open research question, the title of this paper \"A Unified Sequence Interface for Vision Tasks\" is indeed overclaimed. I would suggest rewrite as \"A Unified Sequence Interface for COCO Dataset\" to better reflect what you have proposed and what you have done in the paper. \n\nTo be more specific, the authors claimed that they consider \"core vision tasks\", but task definition seems bias as it is unclear what should be core, and which should not. In my view, there are only 3 pure vision tasks (detection, segmentation, human pose estimation), and additionally the authors consider a vision-and-language (V+L) task (i.e, image captioning). This introduces many concerns. It is unclear why the authors didn't include other popular V+L tasks such as image-text retrieval and visual question answering as they are commonly evaluated in the foundation model literature. Similarly, there are many popular pure vision tasks ignored in this paper, such as image/video recognition, NeRF, etc. Overall, this gives the impression that the task unification is achieved just because of a careful selection of downstream tasks in the experiments. \n\nThe authors compare their method with MaskRCNN, and show that their proposed method can additionally work for image captioning. But no comparison with V+L foundation models is provided. Since image captioning has been well-studied in the unification literature in V+L field, the main new thing here is probably about unifying 3 selected pure vision tasks. However, pioneer works (such as MaskRCNN, HRNet, and others) already unify the three tasks very well. It is unclear what is the advantage of the proposed method (i.e., using the sequence modeling for the three tasks). \n\nFrom modeling perspective, it remains challenging for transformers to learn fine-grained spatial relationship, and such spatial info is critical to the selected 3 tasks. The motivation of using sequence modeling for the three tasks is somewhat weak.\n\nFrom the perspective of efficiency, the proposed unification removes the need of task-specific modules design for COCO dataset. However, due to the nature of autoregressive prediction, the proposed model will be slow. For example, the bottom-up approaches in human pose estimation literature can perform both multi-person pose estimation and instance segmentation at the same time with just a single forward-pass, but in this paper, the inference cost could grow linearly to the number of people. The inference interface is kind of restricted: (1) only a single task is supported at a time; (2) it predicts token-by-token slowly. \n\nFrom the perspective of training, the proposed method doesn't show great benefit. In NLP and V+L research, people use a single loss function, so that it is easier and scalable when performing large-scale pre-training. However, in this paper, the tasks considered require fine-grained annotations, and thus they do not have large-scale pre-training data (except for image captioning). For the considered downstream dataset, it is unclear if single-objective is more effective than multi-objective training.\n\nOverall, the model design is neat and elegant. But it is not well-grounded to the considered downstream tasks. This paper is more like a system paper integrating a few known components together to achieve an expected \"okay\" performance. Considering the technical novelty, the contribution of this paper is relatively weak. \n\nThe proposed method may bring significant advantage with large-scale pre-training. But given the current paper content, it is difficult to reach a conclusion at this point.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "l4itDbVQh9", "reviewer": "Reviewer_iNjL", "summary": "This paper presents a unified sequence-to-sequence generation interface for 4 core vision tasks: object detection, instance segmentation, keypoint detection and image captioning. The model is designed with a single encoder-decoder architecture, with no task-specific heads. To differentiate among different tasks, descriptive task prompts are added as the starting token to generate the output sequence. Experimental results demonstrate a proof-of-concept that such a unified interface can achieve competitive performance on 4 tasks evaluated, compared to some well-established task-specific models.", "questions": "- Results on training from imagenet-pretrained weights? This can be important to understand the current advantage over the compared baselines. The baselines seem to be all trained from imagenet-pretrained weights?\n- More details on the segmentation inference strategy. How the multiple samples are generated during inference? What kind of sampling strategy are used? How was the threshold decided? \n- Any insights on the choices of  loss weights, as greedy search may not be feasible when scaling up the number of tasks.", "limitations": "Yes. The authors have discussed the limitations and societal impacts in Section 5.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strength\n\nIn general, the paper is clearly written, easy to follow. Experimental results indeed show a proof-of-concept that this is possible to tackle vision tasks with a unified interface. \n\n- Weakness\n\nOne concern I have is that the only technical contribution of this paper seem to be extending pixel2seq to the 3 other vision tasks. \n\n- Minor comments\n\nThe important details on training and inference may be better to put into the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fXXaFc5VU5S", "reviewer": "Reviewer_KcLQ", "summary": "The work proposes an unified framework/interface for multiple vision and vision+language tasks, and achieves comparable performance on multiple tasks compared to other strong baselines. The work also shows benefits of multitask learning on some of the tasks. There are some concerns about novelty and whether the work is just incremental compared to previous work of pix2seq[6]. In addition training(pretraining) setup is slightly confusing and performance on few tasks are not so encouraging when compared to other state-of-the-art works. ", "questions": "- How are the batches for different tasks ordered? Does that have any effect on the performance? Does curriculum learning help here?  If we order the batches for different tasks in a particular sequence, does their learning behavior and performance change? How do the training/validation plots look like? Does any task overfit when other tasks are still improving?\n- How does multitask performance vary if we pretrain the model on other tasks than object detection? Is there a relation between high performance on obj detection in the unified setup to the fact that the model was initialized from pretrained weight for that task; however the other vision only tasks do not show good performance. \n- How does the current setup compare to other works in terms of training/inference efficiency? \n- What if the multitask trained model is again finetuned on individual tasks? Does multitask training as a pretraining step help?\n", "limitations": "Yes the authors have touched upon the limitations of this work. However I would encourage the authors to add more details about any potential negative societal impact this type of work can have. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The proposed unified interface is simple and beneficial for a multitask learning. \n- Unifying the interface for these tasks enables same input and output for all tasks which makes it easily scalable to multiple tasks.\n- Multitask learning shows improvements on some of the tasks.\n\n### Weaknesses\n\n- Authors say the models are initialized from pretrained checkpoint trained on the object detection task with the Objects365 dataset. Are the other baselines that are compared in Table 1 also initialized from pretrained object detection checkpoints on Objects365? If not then this is not a fair comparison. \n- The work seems to be incremental and just adding more tasks on top of previous work of pix2seq[6]. If the main message of this work is multitask learning, then it needs to be highlighted properly and the benefits of doing multitask with a unified framework needs to be analyzed in detail. What is the advantage of this setup compared to other frameworks that have task specific input and output parameters/heads and a common  shared base for all tasks? More specifically the work should ablate why a unified interface important compared to architectures having task specific input/output heads.\n- Multitask learning for Instance Segmentation and Keypoint detection are not helping compared to specialized single task models. The work lacks details and analyses for the potential causes.\n- Authors mention that one benefit of this work is that the loss function can be shared. Why is that? The work misses ablation study on the benefit of a common loss function.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bKHtJAvaZxd", "reviewer": "Reviewer_X67c", "summary": "This paper proposed a unified sequence interface (Pix2seq++) for computer vision tasks. The authors extend Pix2seq which can do only object detection to more vision tasks such as segmentation, keypoints, and image captioning and train a single model for 4 different tasks. Pix2seq++ unified the decoder outputs by treating these tasks as sequence prediction tasks. The authors use batch mixing instead of data mixing which implied data augmentation for different tasks. The proposed method achieves state-of-the-art performance on 4 benchmarks. ", "questions": "- What is the correct format for key points?\n- For instance segmentation task, how to make the polygon as a sequence -- is there any order to pick the start point, or is there any constraint if the sequence is too long? \n- Could the authors describe the difference between different image augmentation methods for different tasks? \n- I wonder what is the authors' insights for different methods on instance segmentation (vq-vae vs. seq)? It will be great to discuss this somewhere in the paper. ", "limitations": "the authors adequately addressed the limitations and potential negative societal impact", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- The proposed method is the first one that unified \"core\" computer vision approaches into a sequence and shows it can achieve state-of-the-art performance on these tasks. This has a huge impact on the computer vision community, as the diverse nature of the computer vision tasks leads to diverse architecture designs, and the proposed model unifies/simplifies the design of the decoder. I believe this has a broad impact on vision and unified-model communities. \n- The proposed method achieves good performance on 4 of the benchmark compared to existing classical methods. \n- The authors explore different sampling strategies and mixing ratios for the tasks in the supplementary.\n\n[Weaknesses]\n- The keypoint tokenization is not clear to me. From L69, the paper mentions the key points are represented as a sequence of quantized image coordinates. However, from Figure 1, it seems the output is <label, ymin, xmin>, I wonder which one is the correct format for key points. \n- For instance segmentation task, how to make the polygon as a sequence -- is there any order to pick the start point, or is there any constraint if the sequence is too long? \n- In section 2.3, the authors mention different image augmentation for different tasks. Could the authors describe the difference between different image augmentation methods for different tasks? \n- In table 1, there is no citation for the Transformer-based caption method, and it would be better to report image captioning metrics such as cider score for the tasks. \n- For instance segmentation, I have a question and want to hear the author's insight. Some concurrent work uses VQ-VAE to serialize the image output such as (UViM and Unified-IO), however, the proposed pix2seq++ directly uses points. I wonder what is the authors' insights for different methods? It will be great to discuss this somewhere in the paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept