Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a unified framework for vision, language and multi-modal tasks. Instead of directly predicting labels with different formats, a unified task representation is designed to enable joint training of multiple tasks. The method exhibits state-of-the-art performance on the GRIT benchmarks and comparable results with specialized state-of-the-art methods.
strength_and_weaknesses: Strengths:

- The paper presents one of the early attempts to develop one single unified model for major vision tasks including classification, dense predictions, locations, and vision-language tasks and some NLP tasks. The method also achieves top performance on GRIT benchmark and comparable results with specialized state-of-the-art methods.

- The paper provides a practical solution to combine multiple datasets and supervisory signals with discrete encoding and decoding.  

- The method shows good scaling ability for very large models.

Weaknesses:

- The technical novelty of the proposed method is somehow limited although the whole framework is new and the results are significant.  The idea of encoding labels to discrete tokens is similar to earlier work UViM. [r1] also uses the VQ-GAN encoder to convert dense labels to discrete tokens. 

[r1] Visual Prompting via Image Inpainting, NeurIPS 2022.

- Some implementation details are missing. For example, the training details of the VQ model are not mentioned in both the main paper and the supplementary materials. 

Minor issues:

- In Sec. 1 and Fig. 1, it is mentioned a "VQ-VAE" model is used for image serialization. But in Sec. 3, only "VQ-GAN" is mentioned. I understand it seems the VQ-GAN architecture is used but the adversarial loss is not added during encoding images. But different terms in different sections may make the readers hard to follow. 
clarity,_quality,_novelty_and_reproducibility: - **Clarity**: The paper is well-organized and overall easy to follow.

- **Quality**: Thorough experiments are conducted to validate the proposed framework. The presentation is clear. Most related methods have been discussed and compared.

 - **Novelty**: The technical novelty of the proposed method is relatively low. The whole framework is new.

- **Reproducibility**: It would be difficult to fully reproduce the results considering the missing details and requirements of enormous computational resources. It would be appreciated if the authors could make the code/pre-trained models public which might be very useful for future research and applications. 
summary_of_the_review: The paper presents one of the early attempts to develop one single unified model for almost all major vision tasks and some NLP tasks. The method clearly outperforms previous frameworks on the GRIT benchmark. Although there is still a notable performance margin between unified models and specialized models, this paper presents a solid attempt toward unified visual perception. Therefore, I recommend accepting this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a unified learner for various vision and language tasks that involve dense and sparse prediction. To be able to represent languages and various vision tasks in a homogeneous way, the authors employed the tokenized representation where the discrete codebooks for words and visual patches are learned by the VQ-GAN framework and used in the sequence-to-sequence model. When learned with a gigantic dataset of visual and linguistic tasks, the proposed model demonstrated outstanding performance on various downstream tasks without task-specific retraining.  
strength_and_weaknesses: Strength:
The idea of building a unified model for vision and language based on a powerful sequence-to-sequence model has been around recently, but the paper greatly scaled it up to a large number of tasks and datasets, nicely demonstrating that it can achieve decent performance on general vision tasks without task-specific inductive biases. The implication of this result is huge in my opinion.

Weakness:
I do not have a major concern in general since the experiments and results are strong and convincing. One weakness observed from Table 4 is that the multi-task pre-training often hinders the performance of individual downstream tasks, which implies that improving the performance of the model is not simply about scaling up the pre-training data but requires more careful considerations in choosing the pre-training tasks and data. However, I believe that this is not the main scope of this work hence do not have a major concern. Other than that, I would like to suggest authors include some discussions and results that might help readers to understand the limitation/capability of this work better, which are summarized below.

1. Impact of multi-task learning

Other than efficiency, one of the benefits of having a single parameterization of multiple tasks is that it might learn to understand tasks across domains. For instance, some tasks used in the pre-training have strong biases in their domains (e.g., pose is mostly labeled for humans and the surface normal is labeled mostly for rigid indoor objects). Yet, an ideal multi-task learner might be able to disentangle the task and domain, and perhaps generalize the task across domains in other tasks. Many works on the vision-language model often demonstrate this out-of-distribution generalization in a limited context (e.g., in the context of open-vocabulary learning where the pre-trained model is applied to downstream tasks of unseen object categories), but it would be interesting to see how much a massive multi-task learner such as the proposed model can extend it to more difficult cases (e.g., estimating a pose of animal or surface normal of human).  


2. Performance gap in zero-shot generalization

The authors demonstrated that the proposed method exhibits compelling performance to the SOTA models specialized in individual tasks. Although it is quite encouraging, it does not directly show the upper-bound performance of the proposed model on individual tasks (i.e., how powerful the architecture and pre-training strategy for each task and how much performance we need to compromise by eliminating the test-time finetuning). Ablation studies with and without fine-tuning in downstream tasks might be helpful in understanding this gap.

clarity,_quality,_novelty_and_reproducibility: The overall clarity, quality, and novelty of this work are great in general.
summary_of_the_review: In general, I am positive about this paper due to the strong and convincing evaluation results. Although the technical contribution of this work is not ground-breaking, the authors executed the idea of building a unified model for vision and language based on a powerful sequence-to-sequence model well and achieved encouraging results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes UNIFIED-IO, a unified model for a large variety of vision, language and V+L tasks. By formulating all inputs as sequences of embeddings, and all outputs as sequences of discrete tokens, UNIFIED-IO is able to use a simple Seq2Seq model to handle most V&L tasks such as image synthesis, depth estimation, object detection, segmentation, VQA, and more. It uses the T5 architecture as the Seq2Seq model, the SentencePiece tokenizer for encoding/decoding text tokens, a pretrained VQ-GAN for decoding image tokens into images. The model is trained with two stages: firstly pretrained unsupervised with masked token prediction objective on image, text and image-text pair data, then trained with supervision on a large number of tasks and datasets. The model is then evaluated on the recently proposed GRIT benchmark and 16 additional tasks to show competitive results on most of these tasks without finetuning for any specific task.

**Major contributions of this paper:**
- UNIFIED-IO is the first model to combine a large number of V&L tasks.
- It shows the capability of a transformer Seq2Seq model to learn and perform a large variety of V&L tasks.

strength_and_weaknesses: **Strengths:**
1. The method unifies input and output of V&L tasks to sequences in a clean and relatively simple way.
2. The main architecture that follows T5 is also conceptually simple and clean, consisting of a transformer encoder and decoder without much tweaking/modification.
3. No task-specific augmentations are used.
4. It is non-trivial in terms of engineering to combine so many datasets and tasks together, as well as evaluation on a large variety of tasks.
5. This work is a demonstration of clever combination of existing techniques into a generic framework. The framework only consists of a few well-established components (T5, Seq2Seq, VQ-VAE as used in UViM, masked token prediction, coordinates tokenization as proposed by Pix2Seq) while obtaining decent performance.

**Weaknesses:**
1. The need for VQ-VAE may limit the quality of image synthesis (the model is only trained on image synthesis but not evaluated on such tasks).
2. The model is only evaluated on two dense output tasks: depth estimation and segmentation (in GRIT). 
  - For depth estimation, it is compared with UViM but there is no comparison of model size.
  - For segmentation, there is only a mask R-CNN baseline for GRIT (without model size comparison), but there are better-established benchmarks (such as COCO, Cityscape) where more methods have reported on.
  - Results on more dense output tasks such as panoptic segmentation would be interesting.
3. For tasks that require coordinates generation, the model seems to do better in short-length or fixed-length outputs (e.g. keypoint), but not in free-length outputs such as object detection. There is also no evaluation on certain core vision tasks such as object detection, even if the model is trained on it.
4. Overall, the benchmarks could provide more comparison to be more convincing. Right now the results shows that UNIFIED-IO can do a large variety of tasks, but only some of them competitively.
  - GRIT is relatively new, so it would be nice to evaluate on salient benchmarks on some of the tasks without much comparison in GRIT (e.g. segmentation).
  - Additional tasks are mostly with text outputs or image classification, except one task for depth estimation. More tasks with coordinates generation and dense output would be nice.
  - No model comparison in Table 3 and Table 5, so it is hard to separate gains from model scaling and methodology. 
5. Known concern for speed of autoregressive generation.
6. Strictly speaking, all components and techniques have been proposed in previous or concurrent work, so it is a bit weak on novelty.

**Actionable items:**
- More baseline on tasks with coordinates output (object detection) or dense output (segmentation) on popular benchmarks such as COCO. I’d suggest that have object detection, semantic and/or panoptic segmentation, keypoint detection on COCO validation/test set for easier comparison with existing methods and concurrent work.
- Add model size comparison in Table 3 and 5 to help explain how much gain comes from model scaling vs methodology. The XL version with 2.9B parameters is likely much larger than most of the methods in comparison.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, and is the first implementation of a simple framework that can handle a large variety of vision and language tasks.
summary_of_the_review: This paper proposes a simple and generic framework that can perform on a large variety of vision, language and V+L tasks. Even though it is not topping in technical novelty, it is a good proof of concept, and the amount of tasks and datasets combined is impressive. In addition it is not trivial to combine so many datasets and tasks altogether in terms of engineering. I would recommend it to be accepted, with some reservation, mainly in terms of evaluation of the model. I am happy to change my score if the evaluation of the model is more rounded (on a more balanced set of tasks, on salient benchmarks, for easier comparison with existing methods).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes Unified-IO, a single Seq2Seq transformer model that can perform a wide range of tasks in vision, NLP, and vision-language domains. The unification of tasks is achieved by converting the input&output of each task into a sequence of discrete tokens. The model is trained in two stages: pre-training with masked language/image denoising and multi-task learning. The resulting model achieves competitive performance on downstream tasks without per-task finetuning.
strength_and_weaknesses: Strength:
- The number of tasks supported by Unified-IO is impressive. It would require an enormous amount of engineering effort to unify the data format for tasks of diverse nature, not to mention setting up the multi-task training. The community would benefit a lot if the authors could open-source their code for data preprocessing and multi-task training.
- It is interesting to see that a unified model can achieve competitive performance on multiple tasks without per-task finetuning. In particular, Table 4 sheds some light on how various tasks may benefit or conflict each other. 
- The paper is very well-written with a clear structure and flow.


Weaknesses:
- In NLP, task unification enables **zero-shot** generalization to novel tasks unseen during training. Papers such as Flamingo have also shown that models trained on image captioning can solve zero-shot VQA. Unified-IO shows the capability to generalize to novel concepts, but concept-generalization is much easier than task-generalization. Have the authors observed any capability of Unified-IO to generalize across tasks? This might be the most important advantage that is expected from task unification.
- Besides reducing multiple models to a single one, what other advantage does multi-task learning bring? Table 4 shows that different tasks may benefit or harm each other. It would be good to see a more principled analysis of task relationships. If one is interested in a particular task, what is the optimal multi-task selection strategy?
- How important is the pre-training stage? It would be good to see some ablation experiments to study the effect of pre-training and analyze each proposed pre-training objective.
- The paper claims the pre-training stage to be ``webly supervised''. However, Imagenet21k has human-annotated class labels.
clarity,_quality,_novelty_and_reproducibility: The work is clearly presented. The concept is not novel, but the execution of the concept is better than existing methods. The implementation details are provided, but the code and model do not seem to be open-sourced.
summary_of_the_review: Unified-IO demonstrates the possibility of unifying a wide range of different tasks into a single model with competitive performance. This contribution alone can justify acceptance in my opinion. The paper is a bit lacking in terms of more in-depth anaylsis of pre-training and multi-task learning. The work will also be much more valuable if the code and model can be open-sourced.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers liked the paper. Scores are good! Accept! Congrats!! 
justification_for_why_not_higher_score: Seems interesting but not really ground breaking. But reviewers liked the paper. 
justification_for_why_not_lower_score: Seems interesting and pretty good but maybe it will get more interest than a regular poster i guess.
Decision: Accept: notable-top-25%