Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed perceiver IO, which is a general architecture for structured input & output. Perceiver IO is based on the perceiver architecture which scales linearly with the size of inputs and outputs and augments with a flexible querying mechanism similar to NeRF. Different from prior work, perceiver IO directly operates in the raw input space -- UTF-8 bytes for language, xy coordinates in optical flow, raw audio, etc. The output can be arbitrary in size and different structures, and with non-autoregressive decoding, the model can handle large input and output sizes. The proposed model is tested on variety of tasks, including language modeling, optical flow, video audio class autoencoding, image classification, and starcraft II, and achieves superior performance. 
main_review: #### Strength
- The idea of perceiver IO is novel and solid -- a general architecture capable of handling general-purpose inputs and outputs across different tasks and modalities. This is very promising to simplify the construction of highly tuned task-specific neural pipelines and improve the multimodal and multi-task problems. 

- Each component in perceiver IO is necessary and well defined for the proposed tasks. 2D byte array input enables the task agnostic input for different modalities. Perceiver network that scales linearly with the size of input and output, and non-auto regressive decoding make the decoding to raw output possible. 

- The proposed architecture is tested on massive experiments including language understanding tasks, optical flow, video audio class autoencoding, image classification, and starcraft II and achieves superior performance. Each task is supported with a detailed ablation study to shed light on future research.

#### Weakness
- In Table 1, the WNLI task is excluded from the GLUE benchmark, I wonder what is the reason this task is removed? 

- The non-autoregressive decoding enables fast decoding for large outputs. However, for tasks that require more information on prior decoding context (machine translation or image generation tasks), does the proposed model can still perform well on those tasks? 

- Although perceiver IO removes the task-specific pre-processing (tokenization, patch embedding, etc.), the model still requires huge engineering efforts to adapt the model for different tasks. For example, the model hyper-parameters are quite different for different tasks. 

- One huge benefit of perceiver IO is to train different tasks together and explore the transfer between different tasks/modalities, which is not explored in this paper.  
summary_of_the_review: The paper is well written, the idea is solid and novel, supported with massive and strong experimental results. Overall, the is a strong submission and I would recommend accepting the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors proposed a new general architecture called Perceiver IO for various tasks with different types of inputs and outputs. The Perceiver IO employes a read-process-write architecture, in which input arrays are first projected to the latent space through cross-attention and then the outputs are generated by querying the latent space through some output query arrays. Such a generic pipeline can be applied to various tasks spanning from single modality language tasks, multi-modal tasks, dense prediction tasks, and even symbolic predictions. The main benefit of the proposed Perceiver IO is that it decouples the inputs and outputs using a latent processing module so that it can cope with the arbitrary length of input arrays while outputting an arbitrary number of predictions. Extensive epxeriments are conducted on language understanding and masked language learning, optical flow estimation, image classification, multi-modal autoencoding, etc, and showed that the proposed Perceiver IO can achieve comparable performance to strong baselines in different domains.
main_review: Pros:

1. This paper introduced a generic architecture for coping with different tasks with various input and output lengths. It is different from Transformer architecture and Perceiver architecture in that it uses a latent process to compress the larger number of input tokens into a hidden space, and then decode the output from this hidden space. This way it can decouple the inputs and outputs which has the potential to significantly reduce the cost brought by a large number of input and output tokens. 

2. The authors presented a thorough study of Perceiver IO across different application scenarios, such as language, vision, multi-modality. Extensive experiments demonstrate that the proposed method can achieve comparable or even better performance than the baselines. To facilitate the adaptation, the authors suggested a number of good ways to convert the inputs/outputs into certain formats. 

Cons:

1. The authors should list the sizes of inputs, latent, and output arrays for each of the downstream tasks to give the audience a better sense of the complexity. For some of the tasks, such as ImageNet classification, the authors should also report the FLOPs in comparisons with prior arts, e.g., ViTs. 

2. The main merit of Perceiver IO compared with Perceiver is that it further introduces a cross-attention-based output decoder on top of the latent arrays. Though Perceiver was originally proposed for classification tasks, it can be also used to cope with structured outputs. For some of the tasks except for ImageNet classification listed in this paper, I am curious whether Perceiver can be applied and how it performs compared with Perceiver IO.

3. Perceiver IO is proposed as a generic architecture for various tasks by modeling them as a read-process-write process. One question is that whether we should reformulate all these tasks as the way shown in this paper. This paper does show some encouraging results on various tasks compared with the baselines. However, they are only compared with baseline methods and still underperforms established methods in specific domains. As we know, different tasks can still benefit a lot from the specific domain knowledge (e.g., 2D spatial for image recognition). Actually, we can also think of the Transformer encoder as a generic architecture in numerous domains. Then, do we need to build up a higher-level generic architecture upon the Transformer encoder? 

4. A good part of unifying architectures for various tasks is that we can train the same set of parameters using different tasks. In this paper, the authors demonstrated the effectiveness of Perceiver IO across different settings, I am wondering whether such a generic architecture can be used for multi-task training so that it can leverage the training data from different tasks. 
summary_of_the_review: Overall, I like the idea of Perceiver IO considering it is a good way of modeling arbitrary numbers of input and output tokens. Based on the experimental results on various downstream tasks, it indeed shows some superiorities over the strong baselines. The experiments are solid enough to justify the main claim. However, as discussed above, my main question is that whether we want to develop a unified read-process-write architecture for all these tasks. What are the main merits of this act? I will hold my score to the borderline and wait for more feedback from the authors.

[Post-rebuttal comments]

After reading the authors' feedbacks to all reviewers including myself, I think they had addressed most of the concerns and questions. All reviewers think this work is novel and appreciate its solid contributions. As such, I raised my rating to "accept", and look forward to the furture work of applying it to learn from multiple tasks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a general-purpose neural network architecture named Perceiver IO. With only small modifications on the query side, Perceiver IO can handle various inputs, such as languages, images, videos, point clouds, optical flows, or game agents. Perceiver IO is hugely dependent on the previous work, Perceiver (Jaegle et al.). The main difference between Perceiver and Perceiver IO is the final decoding module. Instead of using a classification head, Perceiver IO uses an additional cross-attention module with an output array. For example, if a user wants to make Perceiver IO be a classification model, the output array becomes 1 X # classes. Perceiver IO shows impressive results in the experiments compared to domain-specific designed architectures in various domains (language, optical flow, audio-video autoencoding, image classification).

Perceiver IO shows its flexibility on various input domains and domain knowledge. For example, Perceiver IO shows even comparable performances to the BERT baseline on the GLUE benchmark without the conventional tokenization but with only UTF-8 bytes (Perceiver IO with tokenization shows 81.1 and Perceiver IO with UTF-8 bytes shows 81.0 while BERT baseline with comparable FLOPs shows 81.1). On the other hand, in the vision domain, by using 2D convolution and max-pooling preprocessing, Perceiver IO shows better top-1 accuracy compared to 2D Fourier features (82.1 for conv preprocessing and 79.0 for 2D Fourier features).

main_review: The main strength of Perceiver IO is its high flexibility to various input domains. While keeping the main architecture, we can focus on engineering the input query preprocessing with domain-specific knowledge. Perceiver IO not only shows flexibility but also shows high performance. The results are comparable to or sometimes outperform previous domain-specific methods. For example

- Table 1 shows that Perceiver IO works well even without tokenization, while BERT + UTF-8 bytes show a significant performance drop (81.0 -> 71.5). It may show that Perceiver IO can perform well even without minimal domain knowledge (tokenization).
- Table 2 shows that the input query engineering can improve the target task (e.g., single-task query to 8 multi-task queries).
- Table 5 shows that the small input preprocessing engineering can improve the performance a lot. For example, Perceiver IO shows 82.1 top-1 accuracies for conv preprocessing while Perceiver IO with Fourier features shows 79.0.

I have minor concerns about ImageNet experiments.

- Although this paper is not aiming to state-of-the-art ImageNet performances, I think Perceiver IO is not yet comparable to state-of-the-art models on ImageNet image classification (as described in the last paragraph of Section 1). Perceiver IO still needs tremendous FLOPs compared to vision-specific models (Table 17). For example, ResNet shows 78.6 top-1 accuracy with 4.1 GFLOPs, while Perceiver IO Fourier shows 79.0 top-1 accuracy with 407 GFLOPs (99 times larger FLOPs). Stronger Perceiver IO (conv) shows 82.1 top-1 accuracy with 369 GFLOPs, where CaiT-M48 448 shows 86.5 top-1 accuracy and 329.6 GFLOPs.
- "Perceiver IO (pretrained)" and "Perceiver IO" in 2D Fourier features are different models. For example, "Perceiver IO (pretrained)" does not use weight sharing, and uses larger channel sizes. As a result, Perceiver IO (pretrained) has 4.38 times larger parameters compared to Perceiver IO (but half GFLOPs). The current text can make a reader mislead that "Perceiver IO (pretrained)" is the same model as "Perceiver IO", but the only difference is the JFT pretraining, which is not true.

> Unlike in the other ImageNet experiments, we do not share weights in the latent self-attention process modules, but use a 16-layer latent network with no weight sharing in depth. Unlike the other ImageNet experiments, the process-module MLPs use a hidden layer with 4× the number of channels (rather than 1× as on other ImageNet experiments)

To avoid confusion, I suggest replacing Table 5 with Table 17. Also, I would like to suggest to clarify that "Perceiver IO (pretrained)" and "Perceiver IO" are not the same model.

## Questions

- Perceiver IO shows better performances consistently compared to Perceiver IO baseline on GLUE (Table 13), Audio Set (Table 7), and ImageNet (Table 17). To my knowledge, there is no specific reason to the proposed output array cross-attention works better than the conventional classification head. I wonder what is the opinion by the authors what is the source of the improvements.
- In Table 5, I found that using 2D + maxpool preprocessing harms Perceiver performances (78.6 -> 77.4), but it helps Perceiver IO a lot (79.0 -> 82.1). As far as the reviewer understood, the main difference between Perceiver and Perceiver IO is the last decoding layer, not the input processing. I wonder what is the opinion of the authors why Perceiver and Perceiver IO show contradictory results on ImageNet Fourier features and conv preprocessing.
- I wonder why there is no Perceiver IO (conv) + pretraining result. Do the authors have any plan to add Perceiver IO (conv + pretrained)? (this is not a mandatory experiment for the rebuttal)

## Minor comments

- Many details are missing in the main text. To understand the method, the readers should read through the heavy appendix and the previous Perceiver paper. I fully understand that this is because of the page limitation, but this paper is somewhat hard to understand at first glance without any prior knowledge.
- In my opinion, it would be very helpful to readers if Table 8 and 9 are in the main text.
- pg 3. Typo -- unnecessary space *( "inducing points")*
- pg 29. Missing hyperlink (Tab. ??)
- In my opinion, it will be very interesting if there is one unified Perceiver model pretrained on various domain data. For example, one can train a Perceiver model trained by language and vision modalities, where each modality has different input and output arrays.
summary_of_the_review: This paper provides very a strong empirical contribution to a unified architecture for various input domains. Not only showing flexibility, but Perceiver IO also shows powerful performances in various domains. Although I have a small concerns that ImageNet results (a reader can mislead the results) I think this paper is a good paper and indeed recommend acceptance.

--------------------
Post-rebuttal comments.

The authors addressed my concerns very well in the rebuttal. I will keep my decision as "8: accept, good paper".
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a general purpose architecture(Perceiver IO) which can take any arbitrary type of inputs and produce arbitrary outputs. The architecture enables this while scaling linearly with input embedding size and output size. Perceiver IO performs comparably with SOTA results on a variety of tasks from language, vision and multimodal domains, which speaks about the generalizability of this architecture. 
main_review: #### Strengths

- Major contribution is ability to scale the output to arbitrary length using an output query array. This enables supporting different types of output and scales well to different tasks. 
- Byte level performance of Perceiver IO is impressive compared to BERT baseline. Byte level embeddings can be fed to Perceiver IO without any tokenization and this setting gets comparable performance with BERT sentence piece tokenization. 
- Shows good results with multitask learning, using both single and task specific tokens.
- Authors do a thorough evaluation of this architecture on a wide variety of tasks, and achieves comparable results on multiple tasks in language, vision, multimodal, RL domains. 

#### Weaknesses

- Novelty over Perceiver : Although I find this work quite impressive as it scales linearly with output embedding sizes as well as arbitrary types of outputs, the overall architecture seems incremental compared to Perceiver. One important change is there is no cross attention in the intermediate layers. I would love to see more detailed analysis why that is removed in this architecture and empirical evidence to support this change. 

- How does choosing values for N,D affect the performance on various tasks. If we decouple input representation, output representation and base architecture, how does model performance scale with depth, N, D values.  Ablations on these hyper-parameters are required for more clarity. 

- Given Perceiver can handle multiple modalities, I would be interested to see how it performs on multimodal tasks like VQA, Image/Text Retrieval, Hateful Memes etc and comparison to multimodal models like CLIP[1], ALIGN[2], SimVL[3].

  - [1] Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry et al. "Learning transferable visual models from natural language supervision." arXiv preprint arXiv:2103.00020 (2021).
  - [2] Jia, Chao, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. "Scaling up visual and vision-language representation learning with noisy text supervision." arXiv preprint arXiv:2102.05918 (2021).
  - [3] Wang, Zirui, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. "SimVLM: Simple visual language model pretraining with weak supervision." arXiv preprint arXiv:2108.10904 (2021).
 
summary_of_the_review: Perceiver IO extends the Perceiver architecture to scale to arbitrary outputs and output lengths. The proposed method is evaluated on a wide variety of tasks and achieves good results comparable to SOTA in most of them. I recommend the current score, with minor reservations over added novelty over Perceiver and also I would encourage the authors to include more analysis and comparison with other SOTA multimodal models.
review_title: null

Meta-review: This paper proposes Perceiver IO, a general neural architecture that handles general purpose inputs and outputs. It operates directly in the raw input domains, and thus does away with modality specific architecture components. The paper contains extensive experiments showing the capabilities of this architecture in different domains. The paper received very positive reviews from all reviewers. Some concerns included a need for additional details such as a missing task from GLUE, FLOPs comparisons to past works, nomenclature for the versions of Perceiver IO, etc. These concerns were well addressed by the authors. Others concerns by reviewers were the lack of experiments in a multi task setting. However, it was acknowledged by the authors and reviewers that this is an open area of research and is a good fit for future work. Given this high quality submission, strong reviews and a very positive discussion amongst authors and reviewers, I recommend accepting this paper.
Decision: Accept (Spotlight)