Perceiver IO: A General Architecture for Structured Inputs & Outputs
Authors: Andrew Jaegle
Sebastian Borgeaud
Jean-Baptiste Alayrac
Carl Doersch
Catalin Ionescu
David Ding
Skanda Koppula
Daniel Zoran
Andrew Brock
Evan Shelhamer
Olivier J Henaff
Matthew Botvinick
Andrew Zisserman
Oriol Vinyals
Joao Carreira
Venue: ICLR 2022 Spotlight (2022)
Keywords: Perceiver
BERT
natural language processing
optical flow
computer vision
multimodal
GLUE
ImageNet
StarCraft

Abstract: A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible. Current architectures, however, cannot be applied beyond a small set of stereotyped settings, as they bake in domain & task assumptions or scale poorly to large inputs or outputs. In this work, we propose Perceiver IO, a general-purpose architecture that handles data from arbitrary settings while scaling linearly with the size of inputs and outputs. Our model augments the Perceiver with a flexible querying mechanism that enables outputs of various sizes and semantics, doing away with the need for task-specific architecture engineering. The same architecture achieves strong results on tasks spanning natural language and visual understanding, multi-task and multi-modal reasoning, and StarCraft II. As highlights, Perceiver IO outperforms a Transformer-based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence.

Published as a conference paper at ICLR 2022
PERCEIVER I0: A GENERAL ARCHITECTURE FOR STRUCTURED INPUTS & OUTPUTS
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin lonescu,
David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff,
Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, João Carreira
DeepMind
ABSTRACT
A central goal of machine learning the development systems that can solve many problems in many data domains as possible. Current architectures, how- ever, cannot be applied beyond small set stereotyped settings, as they bake domain & task assumptions scale poorly to large inputs or outputs. In this work, we propose Perceiver I0, a trary settings while scaling linearly with the size of inputs and outputs. Our model augments the Perceiver with a flexible querying mechanism that enables outputs of various sizes and semantics, doing away with the need for task-specific architecture engineering. The same architecture achieves strong results on tasks spanning natu- ral language and visual understanding, multi-task and multi-modal reasoning, and StarCraft II. As highlights, Perceiver IO outperforms a Transformer based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-offHe of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence.
architecture that handles data from arbi-
INTRODUCTION
Humans have a remarkable ability to take in data from many sources, integrate it seamlessly, and deploy it in the service of range of goals. Most machine learning research focuses on building bespoke systems to handle the stereotyped inputs and outputs associated with single task. This is true even for models that handle multiple modalities. A typical approach independently processes each input with modality specific architecture (for example using a 2D ResNet (He et al. 2016) for vision and a Transformer (Vaswani et al. (017) for language), integrates them afterwards using a third fusion network, and reads out the result in task-specific manner. The complexity of systems like this can grow dramatically as the inputs or outputs grow more diverse (c.g. Abramson et al. 2020 Vinyals et 2019 Ramesh et al. 2021). and the structure of task's 's inputs and outputs may place strong constraints on how data is processed, making adaptation to new settings difficult.
Is the development of problem-specifi models for each new set of inputs and outputs unavoidable? Life would be drastically simpler if a single neural network architecture could handle a wide variety of both input modalities and output tasks. In this work, we propose such an architecture, with the ultimate goal of building a network that can easily integrate and transform arbitrary information for arbitrary tasks. Our starting point is the Perceiver (Jaegle et al. 202T), an architecture which has demonstrated remarkable ability to handle data from many modalities with changes to the network architecture. The Perceiver uses attention to map inputs of a wide range of modalities to a fixed-size latent space that is further processed by deep, fully attentional network. This process decouples the bulk of the network's 's processing from the size and modality-speciin details of the input, allowing it scale to large and multimodal data.
But the Perceiver can only handle simple output spaces like classification. Much of the complexity of real-world tasks comes from the variety, size, and structure of their outputs, and in this regard

---

Published as a conference paper at ICLR 2022
Eoral:
redizied endlety
tene
Sentamend
Crammatioal
Eraalnent
Lubel Drunnring
Figure 1: The Perceiver IO architecture can be used domains with wide variety of input and output spaces, including multi-task language understanding, dense visual tasks like optical flow, hybrid dense/sparse multimodal tasks such vidco+audio+class autoencoding, and tasks with symbolic outputs like StarCraft II. See Tables 3 and for details of all domains considered here.
the original Perceiver can't 't be considered general purpose. this work, we develop a mechanism for decoding structured outputs language, optical flow fields, audiovisual sequences, symbolic unordered sets, etc. directly from the Perceiver latent space, which allows the model to handle a host of new domains without sacrificing the benefits of deep, Nomain-agnosse this, we produce each output by attending to the latent array using an output query that specifies the semantics of that particular output. For example if we wanted the model to predict optical flow one particular pixel we could compose a query from the pixel's xy coordinates plus an optical flow task embedding: the model would then attend using the query and produce single flow vector. As result, our architecture can produce many outputs, each with arbitrary shape and structure, and yet the latent features in our architecture remain agnostic to the shape and structure of the outputs.
processing. To do
Perceiver IO does this using fully attentional read-process-wiie write architecture: inputs are encoded (read) to a latent space, the latent representation refined (process) via many layers of processing. and the latent space is decoded (write) to produce outputs. This approach inherits the best features of both Transformers - which leverage domain agnostic primitives for nonlocal processing inputs and the encoder-decoder architectures (e.g. [Ronneberger et al. 20155 Newell et al. (016) that are in widespread use in high-bandwidth domains such as computer Vision or multimodal processing. This approach allows to decouple the size of elements used for the bulk of the computation (the latent) from the size of the input and output spaces, while making minimal assumptions about the spatial or locality structure of the input and output.
Perceiver IO's decoding procedure uses an attention mechanism to map from latents to arbitrarily sized and structured outputs using a querying system that can flexibly specify the semanties needed for outputs on a wide range of domains, including dense and multitask settings. This decoder allows Perceiver 10 to use on set of challenging domains, while improving performance on tasks like classification that could be handled by the Perceiver.
as drop-in replacement for wide range of specialist networks currently in
The proposed architecture can be applied with unprecedented levels of generality. Perceiver IO can replace the Transformers used in BERT (Devlin al. 2019) and AlphaStar (Vinyals al. 2019). At the same time, Perceiver IO produces state-of art results on the Sintel optical flow benchmark (Butler et al., (012) and good results on ImageNet image classification (Deng al. 2009). Perceiver 10 produces compelling results even when handling highly diverse multimodal data, such as on joint (video, audio, label) autoencoding in Kinetics (Smaira et al. 2020) and joint audio-video classification AudioSet (Gemmeke et al.) 2017). Perceiver 10 allows to simplify pipelines and remove domain-specific assumptions: we process language without tokenizers without a performance speed hit, fine-tune on multiple classification tasks simultaneously and without the need for [CLS] tokens (Sec. [..). estimate optical flow without relying on explicit architectural features for multiscale correspondence (Sec. (.2). learn joint representations of video, audio, and labels without separate network trunks (Sec. 4.3 and perform image classification with no information about the 2D structure of images (Sec.
2 RELATED WORK
Neural network research has long sought architectures that can handle large, arbitrarily structured inputs and outputs. Autoencoding (Hinton & Zemel 1994) was among the first attempts build
ii

---

Published as a conference paper at ICLR 2022
representation which could encode and reproduce gigh-dimensional inputs like images. As hardware grew more powerful, neural nets led to breakthroughs in image understanding (Krizhevsky et al. 2012
Zeiler & Fergus 2014; Szegedy et al., 2015) and interest intensified: autoregressive models that could
process and complete samples of handwriting were developed (Graves) 2013), and new convolutional
network designs led to good results in structured output spaces like semantic segmentation (Farabet
et al., 2012 Long et al. 2015 Ronneberger et al. (2015), pose estimation (Toshey & Szegedy (2014). detection (Sermanet et al., 2014), captioning (You et al, 2016), and optical flow (Fischer al. 2015). At the same time, natural language applications research has made extensive progressive in capturing the structured nature of language, typically via autoregressive models (Collobert al..
2011 [Sutskever et al.) 2014 Vaswani et al., 2017 Radford et al.) 2019; Brown et al. (020) or context
prediction (Mikolov et al.
2013 Pennington et al. 2014 Devlin et al. 2019).
Similar to our work, several groups have proposed to solve tasks in multiple domains (e.g. Kaiser jt al. 2017; Alayrac al. 2200 Akhari et al. 202T), but typically across a fixed and predefined set of modalities by means of domain-specific networks. Although single-task specialist networks remain dominant in vision, multi-task learning has become popular (Misra et al. 2016 Doersch & Zisserman
2017 Kokkinos 2017; Zamiir et al.] 2018) and individual models achieve generality in restricted domain: e.g. Mask-RCNN (He et al. 2017) handles object detection, segmentation, and pose estimation. In language, training evaluation on multiple tasks has also become common (Collobert) & Weston| 2008 Luong et al. 2016 Devlin et al. [2019] Liu et al., [2019] [Raffel et al., (020). Several groups have demonstrated that Transformers (originally designed for language) can be used or adapted to non-language tasks (e.g. Chen et al. 2020 Lu et al. 2021), but the limited scalability of Transformers limits their usefulness as general general-purpose purpose architectures.
Several groups have proposed to use attention to manipulate the size of arrays or to introduce bottlenecks in processing. Set Transformers and related work (Lee et al. 2019 Goyal et al., [022) use a learned query ("inducing points") to induce local bottlenecks by mapping set back and forth from a set with fewer elements and learned decoder queries ("seed vectors") to map to outputs ("pooling by multiheaded attention").. Each layer of these networks has complexity linear in the input size, while Perceivers use a deep latent network with complexity independent of the input and output. Our work uses attention over inputs and outputs of different sizes in part to produce an efficient attention architecture, and several other efficient attention architectures have been proposed, largely for language small-scale problems (e.g. Xiong al. 2021; Wang et al. 2020 Tay al. 2021a Beltagy et al. 2020 and see Tay et al. 2021b). The focus of our work is developing an architecture that efficient and also performs well in many settings with a wide range of inputs and outputs. Several works use attention to process latent spaces that interface with input/output data using task- or domain-specific architectures (Carion al. 2020 Locatello et al., 2020; Wang et al. 202I). and tross-attention itself is widely used to produce outputs in of different size or structure from inputs (Dai et af., 2019 Desai & Johnson 2021 Miech et al. [202 Vaswani et al., 2017 Raffel et al.,
2020; (Santoro et al.) 2018) Hudson& & Zitnick 2021 Ma et al. (022). Perceiver 10 builds on this body of work to produce a general purpose architecture that can be easily and widely applied.
THE PERCEIVER IO ARCHITECTURE
The Perceiver IO architecture builds on the Perceiver (Jaegle et al. 2021 which achieved its cross- domain generality by assuming that its input simple 2D byte array: set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise). each described by a feature vector. The model then encodes information about the input array using a smaller number of latent feature vectors, using Transformer-styy attention, followed by iterative processing and final aggregation down to category label.
Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its outputs as the Perceiver has with respect to its inputs: that is, it should produce arbitrary output arrays. We can predict each element of the output array using another attention module by querying the latent array using a query feature vector unique to the desired output element. In other words, define query array with the same number of elements the desired output. The queries may be hand -esigned,, learned embeddings, or a simple function of the input. They attend to the latents to yield output array of the desired shape.

---

Published as a conference paper at ICLR 2022
Input array
Process
K
K
V
Attention
scores
Latent array
Q
Attention
Attention
K V
Output
array E
Attention
Output
Encode
query array
Decode
Figure 2: The Perceiver IO architecture. Perceiver IO maps arbitrary input arrays to arbitrary output arrays in domain agnostic process. The bulk of the computation happens in latent space whose size typically smaller than the inputs and outputs, which makes the process computationally tractable even for very large inputs outputs. See Fig. 5 for more detailed look at encode, process, and decode attention.
3.1 ENCODING, PROCESSING, AND DECODING
Fig. 2 Qillustrates the Perceiver I0. We first encode by applying an attention module that maps input arrays RMxC to arrays in latent space R We next process the latents by applying series of modules that take in and return arrays in this latent space. Finally, we decode by applying an attention module that maps latent arrays output arrays y ROXE M. C. O, and E are properties of the task data and can be very large (Tab. 5). while N and D are hyperparameters and can be chosen make model computation tractable. Following the design of the Perceiver, we implement each of the architecture's components using Transformer-style attention modules.
Each of these modules applies global query-key-value (QKV) attention operation followed by multi-layer perceptron (MLP). As usual in Transformer-style architectures, apply the MLP independently to each element of the index dimension. Both encoder and decoder take in two input arrays, the first used as input to the module's key and value networks, and the second used input to the module's query network. The module's 's output has the same index dimension (the same number of elements) the query input.
The Perceiver architecture builds on primitives similar to those in Transformers. Why aren't Transformers all you need? Transformers scale very poorly in both compute and memory (Tay et al., 2020). Because Transformers deploy attention modules homogeneously throughout its architecture, using its full input generate queries and keys every layer. This means each layer scales quadratically in compute and memory, which makes it impossible to apply Transformers on high- dimensional data like images without some form of preprocessing. Even on domains like language where Transformers shine, preprocessing (e.g. tokenization) is often needed to scale beyond short input sequences. Perceiver IO uses attention non-homogencousll by mapping inputs space, processing in that latent space, and decoding to an output space. Perceiver 10 has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. F.2) Because of the corresponding reduction compute and memory requirements, Perceiver 10 scales to much larger inputs and outputs. While Transformers typically used in settings with data preprocessed to contain at most a few thousand dimensions (Brown et al. 20000] Raffel al. 2020), show good results domains with hundreds of thousands of dimensions.
latent
This architecture can be applied to inputs of any shape or spatial layout including inputs or outputs with different spatial structure (e.g. sound and video). In contrast to latent spaces typically used in vision (e.g. |Ronneberger et 2013) the latent does not explicitly share the structure (spatial or otherwise) of the inputs. To decode this information, we query for it using attentnn.
iv

---

Published as a conference paper at ICLR 2022
Masked language modeling
Multimodaa
antoencoding
Classification task_id
Video quories
position
Sa video
Optical flow
61602,866 postions
222048 positions
Audio queries Is ooudoo
Multi-task classification
@014406 poshiors
embedding
task__dd
@1900 posibion:
Label query
label
antiice
tanks
Figure We construct queries with output-specific features to produce outputs with different semantics. For settings where each output point differs only in its position, like language, a position embedding can be used. Input features for the target output can also be used to query, either alone (as for StarCraft II) or alongside position features (as for flow). For multi-/task, modal) settings we use one embedding for each [task, modality) } instead of each position. A single learned embedding suffices for simple classification tasks, like ImageNet. For tasks with heterogeneous outputs like multimodal autoencoding, features that are specific to some queries (like xy position) can be combined with modality embeddings, which also pad embeddings to fixed length.
3.2 DECODING THE LATENT REPRESENTATION WITH A QUERY ARRAY
Our goal is to produce a final output array of size O E, given a latent representation of size N x D. We produce an output of this size by querying the decoder with an array of index dimension O. To capture the structure of the output space, we use queries containing the appropriate information for each output point, e.g. spatial position or its modality.
We construct queries by combining (concatenating containing all of the information relevant for one of the O desired outputs. This process is analogous to the way that positional information is used to query implicit functions like NeRF (Mildenhall et al. 2020). We illustrate the query structure for the tasks we consider here in Fig. B For tasks with simple outputs, such classification, these queries can be reused for every example and can be learned from scratch. For outputs with a spatial or sequence structure, we include a position encoding (e.g. a learned positional encoding Fourier feature) representing the position be decoded in the output. For outputs with a multi-task or multimodal structure, we learn a single query for each task or for each modality: this information allows the network to distinguish one task or modality query from the others, much as positional encodings allow attention to distnguish one position from another. For other tasks, the output should reflect the content of the input the query location: for instance, for flow we find it helpful to include the input feature at the point being queried, and for StarCraft II we use the unit information to associate the model's 's output with the corresponding unit. We find that even very simple query features can produce good results, suggesting that the latent attention process able to learn to organize the relevant information a way that's easy to query.
adding) set of vectors into query vector
Each output point depends only its query and the latent array, allowing to decode outputs in parallel. This property allows us to amortize model training on datasets of very large output size. For example, Kinetics consists of labels, video voxels, and audio samples which together come to over 800,000 points (Tab. 5). which prohibitively expensive to decode at once, even with linear scaling. affordable subset
Instead, we subsample the output array at training time and compute the loss of points. At test time, we generate outputs in batches to produce the full output array.
EXPERIMENTS
To probe the generality of Perceiver IO, we evaluate on several domains including language under- standing (Wikipedia+C4 masked language modeling), visual understanding (Sintel/KITTI optical flow and ImageNet classification), multi-modal (Kinetics autoencoding and AudioSet classifica- tion) & multi-task settings (multi-task GLUE), and symbolic representations for games (StarCraft II). All experiments were conducted using JAX (Bradbury et al.] 2018) and the DeepMind JAX ecosystem (Babuschkin et al. 2020).

---

Published as a conference paper at ICLR 2022
Model
Tokenization
M
N
Depth
Params
FLOPs
SPS
Avg.
BERT Base (test)
SentencePiece
512
512
12
110M
109B
81.0
BERT Base (ours)
SentencePiece
512
512
12
110M
109B
81.1
Perceiver 10 Base
SentencePicce
512
256
26
223M
119B
7.4
81.2
BERT (matching FLOPs)
UTF-8 bytes
2048
2048
6
20M
130B
2.9
71.5
Perceiver IO
UTF-8 bytes
2048
256
26
201M
113B
7.6
81.0
Perceiver 10++
UTF-8 bytes
2048
256
40
425M
241B
4.2
81.8
Table 1: Perceiver 10 on language: results on the GLUE benchmark (Avg. = average performance, higher better). Following Devlin al. (2019) we exclude the WNLI task. We use Pearson correlation STS-B, Matthews correlation on CoLa and accuracy on remaining tasks. BERT Base (test) performance is reported Devlin et al. (2019). SPS train-time steps per second. M # inputs and N latents.
4.1 LANGUAGE
We first compare Perceiver 10 to standard Transformers for language. Although Transformers were originally developed for language, their quadratic complexity makes them difficult to use on language inputs without tokenization, which typically shortens the length of input sequences by factor of -4. But unlike Transformer-bes
models such as BERT (Devlin et al., 2019) or XLNet (Yang et al. 2019), Perceiver IO scales linearly with input length. Our experiments focus showing that Perceiver IO performs as well as or better than Transformers for masked language modeling (MLM) while removing tokenization (which is hard maintain, introduces engineering overhead, and adds needless complexity to language models (Bostrom & Durrell 2020 Clark et al. 2022)).
We compare results for given FLOPSs budget rather than given parameter budget as the former grows quadratically with sequence length but the latter is independent (except for positional encodings). From a practioner's 's perspective, FLOPs matter more than parameters since FLOPs directly relate to training time. We evaluate the quality of the learned representation on the GLUE benchmark (Wang) al. 2019) and report our results Tab. [ We find that at a given FLOPs budget, Perceiver IÖ trained without tokenization matches the performance of a strong Transformer-basee model trained with SentencePiece tokenization (Sennrich et al., 2016 Kudo Richardson 2018).
Pretraining. We pretrain on the Masked Language Modeling (MLM) task proposed in Devlin et al. (2019) using a large text corpus obtained by combining English Wikipedia and C4 (Raffel al. 2020). For both the SentencePiece and the byte-level models, mask 15% of the words, where a word is defined as a space-delimited scquence of characters. As a token contains many bytes on input
average, need increase the sequence length to input similar amount of text: sequence lengths of 512 SentencePiece tokens or 2048 UTF-8 bytes. For the SentencePiece models we use a vocabulary size of 32, 000 following Devlin al. (019)). For the byte-level models, the vocabulary size is much smaller: 256 bytes and 4 special tokens ((PAD), [MASK). [CLS), [SEP) 1). Perceiver IO produces one output vector per masked input by using learnable position-dependent vectors to query the output of the final latent processing layer. We then apply position-wise linear layer on top of these output vectors and train the model using a softmax cross-entropy loss predict the original mon-masked input as target. The full details of the architecture are given in Sec. F22 See Appendix Fig. 7 for analysis and visualization of the learnt features.
Finetuning, We finetune Perceiver 10 on the GLUE Benchmark Wang et al. (2019), reporting the best performance on the dev set for fixed size sweep of finetuning hyperparameters. Individual task results and hyperparameters are given in Sec. F.
Perceiver 10 on SentencePiece tokens. We first observe that Perceiver 10 applied on SentencePiece tokenized input sequences slightly outperforms strong BERT baseline applied on the same inputs (81.2 vs 81. As a result the reduced latent size of 256 we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining similar FLOPs budget.
Perceiver 10 on UTF-8 bytes. Next, we show that we can leverage Perceiver IO to run on much longer sequences than regular Transformer. Rather than using fixed, handcrafted vocabulary, model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string. Perceiver 10 significantly outperforms byte-level BERT baseline at the same FLOPSs
vi

---

Published as a conference paper at ICLR 2022
budget, demonstrating the real advantage of Perceiver 10 architecture for language Remarkably, the bytes Perceiver I0 is on par with BERT running on SentencePiece tokens, showing that Perceiver IO is also competitive against strong baselines relying on handcrafted tokenizers. The performance of Perceiver I0 on bytes scales well with more FL.OPs where we obtain 81.8 on the GLUE benchmark.
The byte-level Perceiver IO shares some similarities with the concurrent CANINE work (Clark) et al.] 2022). Chilk/Clark et al. (2022) rely on a relatively sophisticated pipeline that maps Unicode codepoints hash embeddings (Svenstrup al. (017), we embed raw UTF-8 bytes directly. Clark] et al. (2022) also uses bottleneck architecture to scale to longer text inputs, but their upsampling strategy differs from ours: they concatenate raw inputs with their aligned downsampled latent representation, apply a 1D convolution and then run a shallow transformer stack on the resulting upsampled sequence. Their approach scales quadratically with respect to the original input length while Perceiver IO's decoder scales linearly with respect to the target output size. Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.
Multitask
Perceiver
10.
We
use multitask queries as described in Sec. 3.2
to finetune on all 8 GLUE tasks simultaneously using the UTF- 8 byte model (results in Tab. [ We compare to results from the single task regime where the model is trained independently on each task. We also compare to an approach analogous to BERT's [CLS] token that prepends special token to the input and uses the position corresponding to this token to query the task logits. We do this either by sharing a single token among tasks (Shared inpul token) or using task-specific tokens (Task-speciftt input token). In both cases, we use a 2-layer task-specifie MLP head to generate output logits for each task. We observe that our multitask approach outperforms single-task approaches and matches the approach that uses 8 task-specific input tokens. Our approach is more generic as it decouples the output array from the input array by not relying on [ [CLS] tokens. This is especially appealing when the tasks are many or inhomogeneous, as we show in Sec. 4.3
Method
Avg.
Singlc-task query
81.0
Multitask
Shared input token
81.5
Task-specific input tokens
81.8
Multitask query
81.8
Table 2: Multitask Perceiver IO. Re-
sults use the same metric as Tab. T (higher better).
4.2 OPTICAL FLOW
Optical flow is a decades-old open problem in computer vision (Lucas & Kanade 1981 Horn & Schunck 1981). Given two images of the same scene (e.g. two consecutive frames of a video), the task is to estimate the 2D displacement for each pixel in the first image. This has many broader applications, such as navigation and visual odometry in robots (Campbell et al., 2004), estimation of 3D geometry (Ranfil et al. 2020), and even to aid transfer of more complex. learned inference such as 3D human pose estimation from synthetic to real images (Doersch & Zisserman 2019). Optical flow is challenging for neural networks for two reasons. First, optical flow relies on finding correspondence: a single frame provides no information about flow, and images with extremely different appearance can produce the same flow. Second, flow is extremely difficult to annotate, and the few datasets with realistic images and high-quality ground truth are small and biased. While is straightforward to generate large synthetic datasets as training data, e.g. AutoFlow (Sun et al. 2021) there is still a large domain gap.
Algorithms for optical flow thus must learn to accomplish several steps in a way that transfers from synthetic to real data. First, the algorithm must find correspondence between points. Then it must compute their relative offsets. Finally it must propagate flow across large regions of space, including to parts of the image which have no texture for correspondence. To generalize to real data, the learned procedure needs work for objects and textures that weren't seen in the training data.
These difficulties have led flow researchers to develop some of the most involved architectures in the computer vision literature. State of the art algorithms, such as PWCNet (Sun et al., 2018), RAFT (Teed & Deng, 2020) or GMA (Jiang et al., (022)), use explicit machinery to ensure each of these steps is performed correctly even on out-of-domain data. Expensive global correlation
Despite its greater depth, Perceiver 10 is also faster than the Transformer-hase BERT baselines in real wall-clock terms by over factor of 2 for the byte-based models as shown in Tab.
vii

---

Published as a conference paper at ICLR 2022
volumes explicitly compare features within spatiotemporal neighborhood across images to find correspondences. Flows are computed iteratively and hierarchically in 2D space using explicit lookup operators to verify correctness, leading to slow performance on TPUs (Jouppi et al. 2017
Perceiver 10 on Flow In contrast, we apply Perceiver 10 to flow in a straightforward manner. We concatenate the frames along the channel dimension and extract a x 3 patch around each pixel (leading to 3 3 x x 2 = 54 values for each pixel). We concatenate fixed position encoding to these features and then apply Perceiver IO. To decode, we query the latent representation using the input encoding. See Sec. for training details and results with various forms of pre- and postpproesssing, which typically perform similarly. We also test a version with convolutional downsampling and RAFT-style upsampling, which performs only slightly worse while improving computation time.
It may seem counter-intuitive to append the images along the channel dimension, as large motions might result in pixels on entirely different objects being concatenated. However, this kind of operation isn't unpreceednte::: one of the earliest optical flow algorithms, Lucas-Kanade (Lucas & Kanade 198T,, makes explicit use of the temporal image gradient, which is approximated by the difference in intensities at a given pixel across two frames. The algorithm uses the fact that the temporal gradient of the image approximates the spatial gradient times the spatial velocity, if lighting effects are ignored. The approximation is even better for image regions with very little texture. Such regions are challenging for algorithms that attempt to find explicit correspondence in feature space, especially if feature encoding involves any normalization operations, which may destroy intensity information.
Results Tab. 3 shows our results, follow- ing the standard protocol for training on AutoFlow (Sun et al. 2021). We compare
Network
Sintel.clean
Sintel.finll
KITTI
PWCNet (Sun 2018 RAFT (Teed & Deag Z020
2.17
2.91
5.76
to PWCNet and RAFT baselines trained by
1.95
2.57
4.23
Perceiver 10
1.81
2.42
4.98
the AutoFlow authors. On Sintel (Butler
et allaal221 (012), our results are slightly better than RAFT on Sintel and outperform PWC- Net on KITTI (Menze & Geiger) 2015). As far
Table 3: Optical Flow evaluated on Sintel (Butler et al.
2012) and KITTI with average end-point error (EPE) (lower better). Baselines are reported from Sun 2202
aware, this result is state of the art on Sintel fintel.final (GMA Jiang et al. (202l) produces slightly better numbers the somewhat easier Sintel.clean evaluation set using different training data). This is surprising considering how different our architecture is from PWCNet and RAFT and how little tuning for flow Perceiver IO required. We use cost volumes explicit warping, our model is not explicitly hierarchical, and the latent representation doesn' 't even maintain the 2D layout of the inputs. Also note that we reuse RAFT's AutoFlow augmentation parameters, which were tuned specifically for RAFT using population- based training (Sun et al. 2021). As shown in Appendix Fig. []] qualitatively Perceiver IO is good at following object boundaries, and can easily propagate motion across image regions with little texture.
4.3 MULTIMODAL AUTOENCODING
We explore using Perceiver 10 for audio-video-label multimodal autoencoding on the Kinetics- 700-2020 dataset (Smaira et al., 2020). The goal of multimodal autoencoding is to learn model that can accurately reconstruct multimodal inputs in the the presence of bottleneck induced by an architecture. This problem has been previously studied using techniques such as Restricted Boltzmann Machines (Ngiam et al., 2011), but on much more stereotyped and smaller scale data.
K00-22c--700-2020 has video, audio, and class labels. We wish train model reconstruct all modalities simultaneously. With traditional autoencoding models like convolutional encoder-decoders, is not obvious how to combine these modalities, because each uses data of different dimensions 3D (video), ID (raw audio), and OD (class labels) and with wildly different numbers of elements. With Perceiver IO, we pad the inputs with modality-speciif embeddings, serialize them into a single 2D input array and query outputs using queries containing position encodings (for video and audio) and modality embeddings.
We train 16 frames 224 x 224 resolution, preprocessed into 50k 4x4 patches well as 30k raw audio samples, producing a total of 1920 16-d vectors and one 700-d one-hot class label. We decode directly into pixels, raw audio, and the one-hot label without any post- post-processing. To prevent the
viii

---

Published as a conference paper at ICLR 2022
Figure 4: Multimodal audio-video-label autoencoding with 88x compression. Side-by-side: inputs on left, reconstructions right. See the supplemental material for example output video and audio.
model from encoding the label directly into one of the latent variables, we mask the class label 50% of the time in training. Due to the scale of imputs and outputs in this task we subsample decoding in training, while fully decoding in testing: we sampled 512 audio samples and 512 pixels and the class label for every training example. This allows us to directly decode to a video-sized array, which would otherwise be infeasible given memory constraints. We used a latent array with 512 channels and 784, 392, and 196 latents, resulting in compression ratios of 88x, 176x, and 352x respectively.
We show results in Tab. 4 and reconstructions in Fig. 4 By masking the classification label during evaluation, our autoencoding model becomes Kinet- ics 700 classifier. Latent variables are shared across modalities, the quality of reconstructions for each modality is sensitive to the weight of its loss term and other training hyperparameters. Tab. 4 shows one tradeoff, where we emphasized video and au- dio PSNR the expense of classification accuracy. By putting stronger weight on the class loss, we can reach 45% top-1 accuracy while maintaining 20.7 PSNR for video (Sec. []. This strongly suggests that Perceiver IO can jointly represent modalities with very different properties.
Compression Ratio
Audio PSNR
Video PSNR
Top-1
Accuracy
88x
26.97
24.37
10.2%
176x
25.33
24.27
8.6%
352x
14.15
23.21
11.5%
Table 4: Multimodal autoencoding results. Higher is better for accuracy and PSNR.
4.4 IMAGENET, STARCRAFT II, AND AUDIOSET
Please read the Appendix for results on ImageNet (Sec.(A.. StarCraft II (Sec. B), and AudioSet (. We have omitted these results from the main paper to make the exposition as clear as possible within pages (the ICLR camera ready page limit). As highlights of these experiments: (1) on ImageNet, Perceiver IO surpasses 80% top-1 accuracy (84.5% top-1) without using 2D convolutions after pretraining on JFT. (2) When used to replace AlphaStar's 'ss entity Transformer, Perceiver IO obtains 3.5x reduction in FLOPs while preserving StarCraft II 87 % win rate and parameter count, after only experimental runs. (3) On AudioSet, Perceiver IO consistently outperforms the original Perceiver when using the same training protocol multimodal video + audio classification. The Appendix includes additional details of the experimental domains included in the main paper.
5 CONCLUSION
In this work we introduce Perceiver 10, an architecture capable of handling general purpose inputs and outputs while scaling linearly in both input and output sizes. As we show, this architecture achieves good results in wide variety of settings, making it a promising candidate for a general purpose neural network architecture, Perceiver 10 leverages the expressive power of latent attention and uses learned queries to expose simple and unified interface that can handle multimodal and multitask settings. Overall, Perceiver 10 offers a promising way to simplify the construction of sophisticated neural pipelines and facilitate progress on multimodal and multiask problems.
ACKNOWLEDGMENTS
We are grateful to Ankush Gupta and Adrià Recasens Continente for reviewing drafts of this paper and to Deqing Sun for sharing code and helpful advice the optical flow experiments.
ix

---

Published as a conference paper at ICLR 2022
REFERENCES
Josh Abramson, Arun Ahuja, Arthur Brussce, Federico Carnevale, Mary Cassin, Stephen Clark, Andrew Dudzik, Petko Georgiev, Aurelia Guy, Tim Harley, Felix Hill, Alden Hung, Zachary Kenton, Jessica Landon, Timothy Lillicrap, Kory Mathewson, Alistair Muldal, Adam Santoro, Nikolay Savinov, Vikrant Varma, Greg Wayne, Nathaniel Wong, Chen Yan, and Rui Zhu. Imitating interactive intelligence. arXiv preprint arXiv:2012.05672, 2020.
Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. VATT: Transformers for multimodal self-supervised learning from video, audio and text. In Proceedings of Neural Information Processing Systems (NeurIPS), 2021.
Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelovic, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multi- modal versatile networks. In Proceedings of Neural Information Processing Systems (NeurIPS), 2020.
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Claudio Fantacci, Jonathan Godwin, Chris Jones, Tom Hennigan, Matteo Hessel, Steven Kapturowski, Thomas Keck, lurii Kemaev, Michael King, Lena Martens, Vladimir Mikulik, Tamara Norman, John Quan, George Papamakarios, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojciech Stokowiec, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http:///ithui: com wtt.:ccmm/eeeeee deepmind
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document Transformer. arXiv preprint urXiv:2004. 0ttt200040.55500 0500 2020.
Kaj Bostrom and Greg Durrett. Byte pair encoding suboptimal for language model pretraining. In Proceedings of the Annual Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
James Bradbury. Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http: /ttp://githuse ggt.bbb..b....gggggg com/ com//oooge//aa....o
Andrew Brock, Soham De, Samuel L Smith, and Karen Simonyan. high-performance large-scale image recognition without normalization. In Proceedings of International Conference on Machine Learning (ICML), 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter. Christopher Hesse, Mark Chen. Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of Neural Information Processing Systems (NeurIPS), 2020.
Daniel J. Butler, Jonas Wulff, Garrett B. Stanley, and Michael J. Black. naturalistic open source movie for optical flow evaluation. In Proceedings of European Conference on Computer Vision (ECCV), 2012.
Campbell, R. Sukthankar, and Nourbakhsh. Techniques for evaluating optical flow for visual odometry in extreme terrain. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2004.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with Transformers. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proceedings of International Conference on Machine Learning (ICMI.,, 2020. 2020.

---

Published as a conference paper at ICLR 2022
Jonathan H. Clark, Dan Garrette, Iulia Ture, and John Wieting. CANINE: pre-training effi- cient tokenization-fret free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73-91, 2022.
Ronan Collobert and Jason Weston, A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of International Conference on Machine Learning (ICML), 2008 2008.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu. and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 2011.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. RandAugment: Practical automated data augmentation with a reduced search space. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V. Le, and Ruslan Salakhutdinov. Transformer-Xe XL: Attentive language models beyond a fixed-length context. In Proceedings the Annual Meetings of the Association for Computational Linguisties (ACL), 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Karan Desai and Justin Johnson. VirTex: Learning Visual Representations from Textual Annotations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional Transformers for language understanding. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), 2019.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of IEEE International Conference Computer Vision (ICCV), 2017.
Carl Doersch and Andrew Zisserman. Sim2real transfer learning for 3D human pose estimation: motion to the rescue. Proceedings of Neural Information Processing Systems (NeurIPS), 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In Proceedings of International Conference Learning Representations (ICLR), 2021.
Clement Farabet, Camille Couprie, Laurent Najman, and Yann LeCun. Learning hierarchical features for scene labeling. IEEE Transactions Pattern Analysis and Machine Intelligence, 2012.
Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazırbaş, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. FlowNet: Learning optical flow with convolutional networks. In Proceedings of IEEE International Conference on Computer Vision (ICCV), 2015.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio Set: An ontology and human- human-labeled dataset for audio events. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, and Yoshua Bengio. Coordination among neural modules through a shared global workspace. In Proceedings of International Conference on Learning Representations (ICLR), 2022.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint 2013.
xi

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of IEEE International Conference on Computer Vision (ICCV), 2017.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint arXiv: (N0v:/666.0844 2016.
Geoffrey E Hinton and Richard S Zemel. Autoencoders, minimum description length, and Helmholtz free energy. In Proceedings of Neural Information Processing Systems (NeurlPS), 1994.
Berthold KP Horn and Brian G Schunck. Determining optical flow, Artificial Intelligence, 1981.
Drew A. Hudson and C. Lawrence Zitnick. Generative adversarial Transformers. In Proceedings of International Conference on Machine Learning (ICML., 2021.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver: General perception with iterative attention. In Proceedings of International Conference on Machine Learning (ICML), 2021.
Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley. Learning to estimate hidden motions with global motion aggregation. In Proceedings of IEEE International Conference on Computer Vision (ICCV), 2021.
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of Tensor Processing Unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, 2017
Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. arXiv preprint arXiv:1706.05137, 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001. 000361. 2020.
lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid- and high-level vision using diverse datasets and limited memory. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep con- volutional neural networks. Proceedings of Neural Information Processing Systems (NeurIPS), 2012.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, In Proceedings of the Annual Meetings of the Association for Computational Linguistics (ACL), 2018
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set Trans- former: A framework for attention-based permutation-innaarian neural networks. In Proceedings of International Conference on Machine Learning (ICML), 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: 190.: 2019..
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Proceedings of Neural Information Processing Systems (NeurlPS), 2020.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
xii

---

Published as a conference paper at ICLR 2022
Iyy Loshchilov and Frank Hutter. SGDR:
Stochastic gradient descent with warm restarts. In Proceedings of International Conference on Learning Representations (ICLR), 2017.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained Transformers as universal computation engines. arXiv preprini arXiv:2103.05244 2021.
Bruce D Lucas and Takeo Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), 1981.
Minh-Thane Thang Luong, Quoc V Le, llya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence to sequence learning. In Proceedings of International Conference Learning Represen- tations (ICLR), 2016.
Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke 7ettle- moyer. LUNA: Linear unified nested attention. In Proceedings of Neural Information Processing Systems (NeurIPS), 2021.
Moritz Menze and Andreas Geiger. Object scene flow for autonomous vehicles. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2015.
Antoine Miech, Jean- Mann-Bapistte Alayrae, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Thinking fast and slow: Efficient text-to-visual retrieval with Transformers. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed represen- tations of words and phrases and their compositionality. In Proceedings of Neural Information Processing Systems (NeurIPS), 2013.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorth, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition 2016.
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In Proceedings of European Conference on Computer Vision (ECCV), 2016.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. In Proceedings of International Conference on Machine Learning (ICML), 2011.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. GloVe: Global Vectors for word representation. In Proceedings of the Annual Conference on Empirical Methods Natural Language Processing (EMNLP)), 2014.
Hieu Pham, Zihang Dai, Qizhe Xie, Minh-Thame Thang Luong, and Quoe Le. Meta pseudo labels. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Dean A. Pomerleau. ALVINN: An autonomous land vehicle in a neural network. In Proceedings of Neural Information Processing Systems (NeurIPS), 1989.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019.
Colin Raffel Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text Transformer. Journal of Machine Learning Research, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to- text-to-image generation. In Proceedings of International Conference Machine Learning (ICMI.), 2021.
Xiii

---

Published as a conference paper at ICLR 2022
Rene Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions Pattern Analysis and Machine Intelligence, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAT), 2015.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. In Proceedings of Neural Information Processing Systems (NeurIPS), 2018.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the Annual Meetings of the Association for Computational Linguistics (ACL), 2016.
Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. In Proceedings of International Conference on Learning Representations (ICLR), 2014.
Lucas Smaira, João Carreira, Eric Noland, Ellen Clancy, Amy Wu, and Andrew Zisserman. A short note the Kinetics- 700-2000 human action dataset. arXiv preprint arXiv:2010.10864 2020.
Kenneth O. Stanley. Compositional pattern producing networks: A novel abstraction of development.
Genetic programming and evolvable machines, 8(20):131 162, 2007.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data deep learning era. In Proceedings of IEEE International Conference on Computer Vision (ICCV), 2017.
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
Deqing Sun, Charles Herrmann, Varun Jampani, Michael Krainin, Forrester Cole, Austin Stone, Rico Jonschkowski, Ramin Zabih, William T Freeman, and Ce Liu. TF-RAFT: tensorflow implementation of RAFT. In ECCV Robust Vision Challenge Workshop, 2020.
Deqing Sun, Daniel Vlasic, Charles Herrmano, Varun Jampani, Michael Krainin, Huiwen Chang, Ramin Zabih, William Freeman, and Ce Liu. AutoFlow: Learning a better training set for optical flow. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In Proceedings of Neural Information Processing Systems (NeurfPS), 2014.
Dan Svenstrup, Jonas Meinertz Hansen, and Ole Winther. Hash embeddings for efficient word representations. In Proceedings of Neural Information Processing Systems (NeurIPS), 2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2015.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Kridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Proceedings of Neural Information Processing Systems (NeurIPS), 2020.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient Transformers: A survey, arXiv preprint ar XwX.:i00090075550 2020.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, and Che Zheng Zhe Zhao. Synthesizer: Rethinking self-attention in Transformer models. Proceedings of International Conference on Machine Learning (ICML), 2021a.
xiv

---

Published as a conference paper at ICLR 2022
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sebastian Ruder, and Donald Metzler. Long range arena: benchmark for efficient Transformers. In Proceedings of International Conference on Learning Representations (ICLR), 2021b.
Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character Transformers via gradient-based subword tokenization. In Proceedings of International Conference on Learning Reprexentations (ICLR), 2022.
Zachary Teed and Jia Deng. RAFT: Recurrent All-pairs Field Transforms for optical flow. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
Alexander Toshev and Christian Szegedy. DeepPose: Human pose estimation via deep neural networks. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2014.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image Transformers & distillation through attention. In Proceedings of International Conference on Machine Learning (ICML), 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image Transformers. In Proceedings of IEEE International Conference on Computer Vision (ICCV), 2021b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of Neural Information Processing Systems (NeurIPS), 2017.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly, Pointer networks. In Proceedings of Neural Information Processing Systems (NeurIPS), 2015.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dario Yogatama, Dani Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Apps Chris, and David Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7882): 350-354, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In Proceedings of International Conference on Learning Representations (ICLR), 2019,
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang- Ching Chen. Max-deeplab End- to-end panoptic segmentation with mask Transformers. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04770 047688 2020.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung. Yin Li, and Vikas Singh. Nyströmformer: A Nyström-based algorithm for approximating self-attention. In Proceedings of AAAI Conference on Artificial Intelligence, 2021.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 2022.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. In Proceedings of Neural Information Processing Systems (NeurIPS), 2019.
XV

---

Published as a conference paper at ICLR 2022
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang. and Jiebo Luo. Image captioning with mantic attention. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In Proceedings of International Conference on Learning Representations (ICLR), 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Junsuk Choe Sanghyuk Chun, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceedings of IEEE International Conference Computer Vision (ICCV), 2019.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of IEEE Conference on Computer Vision and Putlern Recognition (CVPR), 2018.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. Proceed- ings of European Conference on Computer Vision (ECCV), 2014.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Proceedings of International Conference on Learning Representations (ICLR), 2018
xvi

---

Published as a conference paper at ICLR 2022
APPENDIX
Modalities
Tasks
Preprocessing
Postprocssing
Inputs
Outputs
Text
Token-level pred.
Tokenization: Emreed.
Lincar projection
768
768
Text
Ryze-level prod.
Emhhad.
Nore
2,048 778
2, 2,048 x
Text
Muhti-tak tasles]
Embed.
Noce
2, 2,048 x 708
8.X 768
Video
Flow prediction
None
Norr
365, 056 61
182, 528
Video
Flow prediction
Concat
None
184,628 64
182, 028 64
Video
Flow prediction prediction
Con-+maxpol
RAFT upsamplinng
Com--maxpolseoocc
RAFT upsamplling
11,408
VideonnAudioLLaeel
Autoencoding
Patch: lxcxc Vid, Aud
None
50, 63.
803, x 512
Inage
Clavsification
None
None
60, 176
1 1.1.1.000 1,000
Image
Clacsification
Lincar projection Con--maapool
None
50, 176 x 256
x 1,000
Image
Classilicasio
Nore
3, 136 64
1 1,000
StarCraft Unic Set
Enondiing and Classification
Tokenization
Pointer nerwork
512 206
502 128
Video+Audio
Clastifiction
Patch; 2 x x Vid, 128 Aud
Novr
13, 024 487
1 527
Vudeo+Audio
Classilicaión
mol- spectroam
Noms
17,341
Table 5: Details of each of the tasks use evaluate Perceiver IO here. The positional and task embeddings appended inputs for each case are listed Tab.
In the following sections, we describe experiments on three additional domains (ImageNet, StarCraft II, and AudioSet) and provide additional details for the methods and experiments described in the paper. For ease of reference and comparison across domains, we describe the input and output size and processing used in all experiments in Tab. 5 and provide details of input key/value, position encoding, and output queries used in all experiments in Tab. 6
On all domains but StarCraft II, we include experiments with several input configurations, ranging from domain adaptation (e.g. tokenizer-free language, flow from raw pixels, ImageNet with no convolutional or patch-based preprocessing and fully learned position encodings) to moderate domain adaptation (e.g. SentencePiece language understanding, flow from conv-maapool-ppoceess. images and with RAFT upsampling, ImageNet with features). These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling range of tasks, and its flexibility to work as part of domain- adapted system.
and 2D Fourier
Inpur
lapel
quers Inpec
quers
Langrage (30 (8)
Tias
hysthsker mocding laared por
765
leamed pos
1260
(Paeciver 10++ MLM
Toot
Inychakke crondeg kusned
768
Icaned
L536
Laagange
byycl crasddng Justel pro
Claw qaery (ro
Eaggage
(Peroline five
bypetoker mooding laarned pos
7W0
Class spery (pe))
1534
Optical How
(omsat tomee
Jona Lineay jonno HCM,) 20 101
302
[Lasenn 22 [VN)
Opsical Hew
Viceo
(ere or LiscatkGBB, Pia
431
[cen Stataro, 3D FFX)
Kirtkss
Vides,
Ipedted KGB, 3D FFs, icaanol usdality feel
704
30 FPu, lcarad cxadaity feal.
1026
Aadi
[pacted exrst prscass ID IV, loome modaary
701
[IB RII leareed redillry foa
Imarche: (ID Rin
Image
[RGB,, 23 P3g]
261
Clanogaeyy lxagle
D034
[lase::RdB)
512
Toogaher (eoes)
Trage
[Cees sostsss 3D FF))
525
Clasoqury (single)
1024
Sortist It
SC2 entitec
178
Aucishiet
Video,
taacted MGltt, 17% teamed medaltt teasee)
Clas quary (omgle)
AucieSet
Viles
lsustid RGB, 30 Ehs, Icasad modaltz (enuar)
467
Clangyay (xaglee)
1024
Malopesttignaa
Incloppterrpaa (casao. Fb, loanet modility fcature)
Table 6: Table best viewed on screen. The structure and size of the positional and task embeddings used to construct Perceiver IO's encoder key-value inputs and decoder query inputs, for each domain described in the main text. "[x, yl" indicates that x's and y's features are concatenated, while "x +y" indicates that x's and y's features are added to produce the full featurization. "FF" = Fouricr features as in Jaogle et al. (2021)
A IMAGE CLASSIFICATION
Perceiver did well on ImageNet Deng et al. 2009) classification without using 2D structure in the design of the architecture, but generated class scores using a simple average project decoder (see [ec. and Fig. for a diagram illustrating the difference between the two forms of decoder). We now evaluate the effect of this more general decoder. See Sec. for similar validation AudioSet.
C
xvii

---

Published as a conference paper at ICLR 2022
Model
Pretrained?
Accuracy
FLOPss
Params
ConvNet baselines
ResNer-50 (He et al. 2016
N
78.6
4.1B
26M
NFNet-F6+SAM (Brock
2021
86.5
377.3B
438.4M
Meta Pseudo Labels (Pham al. 2021
Y
90.2
480M
VIT baselines
ViT-B/16 (Dosovitskiy et al.
2021
77.9
55.4B
86M
ViT-H/14 (Dosovitskiy al.
2021
Y
88.6
632M
DeiT 1000 epochs (Touvron et al.
2021a)
85.2
87M
CaiT-M48 448 (Touvron al. 202
86.5
329.6B
356M
w/ 2D Fourier features Perceiver
78.6
404B
42. IM
Pereeiver IO, config
N
79.0
407B
48.4M
Perceiver IO, config B (pretrained)
Y
84.5
213B
212M
w/ learned position features Pereeiver (learned pos)
N
67.6
404B
55.9M
Perceiver I0, config A (Icarned pos)
N
72.7
407B
62.3M
w/ 2D conv + maxpool preprocessing Perceiver (conv)
N
77.4
367B
42.1M
Perceiver 1O, config A (conv)
N
82.1
369B
48.60
Perceiver IO, config (conv) (pretrained)
86.4
176B
212M
Table 7: Results ImageNet image classification (top-1 accuracy, higher is hetter). "L" indicates value we could not find reported classification the primary focus of this work is generality, rather than speed on images Perceiver IO uses comparable FLOPs to attention basedd image classification models, especially for the more compact configuration pretrained on JFT. The positional encoding does not significantly change model FLOPS.
literature. We did not extensively tune our models efficiency on image
Results Tab. 7 shows our results alongside representative numbers from the literature. Perceiver and Perceiver IO differ in their decoder, and neither model uses convolutional preprocessing by default. Perceiver 10 consistently outperforms the original architecture. After pretraining on JFT (Sun et al.) 2017). Perceiver IO performs in the ballpark of models designed primarily for image classification. Perceiver 1O is competitive with members of the Vision Transformer (ViT) (Dosovitskiy et al. 2021) family even without relying on 2D convolutions, Perceiver IO also compatible with convolutional preprocessing: adding a 2D conv+maxpool preprocessing stage leads to a moderate increase in efficiency and bump performance.
While neither the Perceiver and Perceiver 10 incorporate any 2D spatial structure architecturally. they use positional features that inject 2D spatial information (Sec. 3.2 and Appendix sec. of Jaegle et al. 2021). By replacing these 2D position features with fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images. This positional encoding is an array of shape 50,176 x 256, which randomly initialized using a truncated Gaussian distribution with scale 0.02. ImageNet networks that use this positional encoding are given no information about 2D image structure. For these experiments, we additionally use 1D convolutional network to project the RGB at each point to 256 before concatenating with the learned positional encoding. The results of this experiment shown in Tab. 7 (w/ learned position features). To our knowledge, this is the best result by any model on ImageNet without 2D architectural feature information.
A.1 DETAILS OF IMAGENET TRAINING
2019) and MixUp (Zhang al. 2018 For ImageNet experiments, we CutMix (Yun et al.
regularization, in addition to RandAugment (Cubuk et al. 2020 as used in Jaegle et al. (2021). We
observed only marginal improvements in performance from this change, but brings the augmentation
strategy more in line with the strategy used elsewhere in the literature (Brock al. 202 Touvron et al. 2021a In all experiments,
RandAugment with 4 layers at magnitude (as Jaegle
Xviii

---

Published as a conference paper at ICLR 2022
Model
Train steps/sec
Perceiver (2D FF)
4.73
Perceiver IO (2D FF)
4.85
Perceiver (learned pos)
4.16
Perceiver IO (learned pos)
4.14
Perceiver (conv)
4.73
Perceiver 10 (conv)
5.58
Perceiver IO (pretrained)
6.41
Table 8: ImageNet model training speed. The model used for pretraining faster because it uses only 16 process modules. We did not reimplement basclines, so we report only the training speed of Perceiver and Perceiver 10 models.
et al. 2021) and CutMix with a ratio of 0.2. In early experiments, we found that higher weight decay and moderate gradient clipping contributed better generalization: we use a weight decay 0.1 and clip to maximum glohal gradient norm of 10. We use no dropout. We use architecture with weight sharing in depth: the latent (processing) component of the architecture includes blocks of attention modules each, and weights are shared between the corresponding modules in each block. We omit the repeated encoder cross-attends used in Jaegle et al. (2021) as we found these to lead to relatively small performance improvements but to significantly slow down training: using 8 encoder cross-attention instead adds an additional 303 billion FLOPs. The FLOPs for all ImageNet models presented here given in Tab. 7 and the model training step time on 64 TPUv3 given in Tab.
For all ImageNet experiments, we train for 110 epochs, using batch size of 1024 and 64 TPUs. We use LAMB with a simple learning rate schedule consisting of a flat learning rate x 10-3 for 55 epochs, after which the learning rate is decayed to 0 over the final 55 epochs following a cosine decay schedule (Loshchilov & Hutter 2017). We found cosine learning rate decay schedule simpler to tune than the step decay schedule used in Jaegle et al. (202I) and that beginning the decay process halfway through training generally led to good performance without introducing instability. We found important omit an initial learning rate warm-up period, as this often prevented models from training when using LAMB.
A.2 LARGE-SCALE PRETRAINING
As reported in Jaegle et al. 2021 Perceiver models are able to easily overfit ImageNet-scale datasets without regularization. For this reason, we explored pretraining a model on JFT, a large-scale, multi-labeled internal dataset with 300 million images spanning approximately 18,000 classes (Sun) ct al. 2017). We pretrain on this dataset at the same resolution used on ImageNet (224 x 224) using base learning rate of 3 10-4 and cosine decay schedule, decaying 0 over 14 epochs. We omit all augmentation except basic cropping, resizing, and left-right flipping. We use a weight decay of 0.1. We use larger batch size of 8192 and train on 256 TPUs. Images in this dataset come with variable number of labels, so we use a cross-entropy loss with a multi-one-hot representation of the targets. Unlike in the other ImageNet experiments, we do share weights in the latent self-attention process modules, but use 16-layer latent network with no weight sharing in depth. Unlike the other ImageNet experiments, the process-modull MLPs use a hidden layer with 4x the number of channels (rather than x as other ImageNet experiments). When pretraining the 2D FF model, we use ID convolutional network to project input RGB at each point 256 before concatenating with the positional encoding (a 2D Fourier frequency positional encoding). When pretraining the manv-ma model, we instead use the initial convolutional preprocessing described in Sec. A.3 below.
To evaluate transfer, we fine-tune our pre- mre-trained model on ImageNet. We replace only the final linear layer of the decoder to produce the required 18,000 classes. For 2D FF fine-tuning, optimizer and augmentation settings as with our from-scratch ImageNet training: 1024 batch size on 64 TPUs, 131K 11K steps with LAMB using flat base LR of 0.002 for the first 70K steps and cosine
used similar
xix

---

Published as a conference paper at ICLR 2022
Entity encoder
Win rate
Params (M)
FLOPs
Train steps/sec
Transformer (Vinyals et al., 2019) Perceiver 10
0.87
144
3.3B
2.9
0.87
140
0.93B
2.9
Table 9: We evaluate Perceiver IO StarCraft 11 by using it to replace the well-tuned Transformer entity encoder. Perceiver 10 matches the performance of the original Transformer despite using fewer FLOPs and parameters and requiring essentially no tuning. Note that the training steps/sec the overall system does not change because the entity encoder is not the speed bottleneck.
learning rate decay for the last 61K steps. We use identical settings for monx-ol fine-tuning with the exception of the base learning rate, which we set to 0.0002, training with the higher 0.002 rate was unstable.
A.3 2D CONVOLUTIONAL PREPROCESSING ON IMAGENET
In other image settings discussed here, we optionally use simple pre- and post-processing steps to reduce the size of very large inputs and outputs. Because ImageNet data points are relatively small (Tab. 5). we are able to process full images without convolutional pre- and post-processing. Consequently, we can use this dataset to probe the sensitivity of the model to convolutional pre- processing. Incorporating a single convolution + max pooling leads to a moderate improvement in the performance of the architecture: this is perhaps unsurprising, as convolutional pre-processing injects information about the 2D structure of images into the architecture. By comparison ViT first processes images by applying a 2D convolution with matched kernel and stride to downsample its inputs (referred to as a "linear projection of flattened patches" in that work and throughout the ViT literature). As in other experiments, we find that incorporating an attention-based decoder (Perceiver I0) leads to better results than averaging and pooling the output (Perceiver). Using convolutional preprocessing leads training speed in some configurations (Tab. (). The input to the network after preprocessing is 56 x 56 instead of 224 x 224 as in the experiments directly on pixels.
moderate reduction the number of FL.PSs used by the model (Tab. 7 and
B STARCRAFT II
To further demonstrate Perceiver IO's capabilities on discrete modalities and as drop-in replacement for Transformers, we plug in Perceiver 10 in place of AlphaStar's Transformer. AlphaStar (Vinyals) et al. 2019) is the state-of-the-at system for the challenging real-time strategy game of StarCraft IT.
At its core, AlphaStar represents the units in the game discrete, unordered set of symbols (the "units"). These units are represented by vector of properties including unit type, position, and health. At each timestep. the architecture encodes units with entity encoder, which in the original model was parameterized using a vanilla Transformer.
The entity encoder takes as input of 512 entities (referred as grrbedded_entitee in Vinyals et al. (219)) and produces as output an embedding for each entity (entity (entity_embeddiinn emw.dddinessss and ID embedding reduced over entities ceabbedded_entttty
These 512 entities repre- sent the units and other entities that are present in the game: unused entity slots are masked. entityyyst entttt_eeeeeddeee www.eeeeediin igg is produced by passing the outputs of the entity encoder through a ReLU and a ID convolution with 256 channels. embedded_ent leteiedd_etiiie ity is produced by averaging the (unmasked) entity encoder outputs and passing through a linear layer with 256 units and a ReLU,
In the original AlphaStar system, the entity encoder consisted of Transformer with 3 attention layers, each of which used heads and a feature dimension of 128. The output of each attention layer is projected to 256 and followed by an 2-layer MLP with hidden size 1024 and output size 256. This architecture was arrived by an extensive tuning process as reported in Vinyals et al. (2019).
The representation produced by the entity encoder is used both as summary of the state (after pooling) and as a rich representation the units. This representation is used by a pointer network (Vinyals) et al. 2015)) to assign probability to each possible unit selection, in the process parameterizing the agent's 's unit selection policy. For this reason, we view AlphaStar as an important test case for Perceiver IO's ability to function as general-purpont tool for processing symbolic set-valued
XX

---

Published as a conference paper at ICLR 2022
Model
Input
mAP
Latent channels (D)
Params (M)
FLOPs
Train steps/sec
Perceiver
Raw audio video
42.4
21.0
52.3B
3.8
Perceiver IO
Raw audio video
43.3
512
25.0
52.9B
3.8
Perceiver
mel-spectrogram video
43.6
512
21.0
60.7B
3.8
Perceiver IO
mel-spectrogram video
44.9
1024
88.2
129.5B
3.8
Table 10: Perceiver IO multimodal (audio video) AudioSet classification (mAP average precision, higher hetter). All models have similar runtimes despite FL.OPs differences because the bottleneck data loading and preprocessing rather than model forward/backward passes.
data. If the question "can Perceiver IO serve as a replacement for a well-tuned Transformer as a symbolic processing engine?" then the answer is yes:
We obtained StarCraft results by using Perceiver IO instead of a Transformer for the AlphaStar entity encoder. We replaced the Transformer with Perceiver 10 with latent of index dimension 32, keeping the input and output size of 512 units. We performed no tuning beyond sweeping the size of the latent index dimension (we tried values of 32 and 64): Perceiver IO works out of the box. We observed that the resulting agent reached the same level of performance as the original AlphaStar agent, reaching an 87% win-rate versus the Elite bot after behavioral cloning (Pomerleau 1989) on human data, while also leading to 3x decrease in FLOPs (Tab. 9
We replaced this Transformer with a 3-layer Perceiver IO with latent index dimension 32. We tuned only the size of the index dimension (sweeping values of 32 and 64), but otherwise used the same hyperparameters ImageNet.
C AUDIOSET
We seek to confirm that the the attention- decoder helps even on classification, where the original Perceiver's decoder could be used. We show that the trend identified on ImageNet holds more generally, by revisiting
multimodal AudioSet classification domain. AudioSet is large-scale event classification dataset containing 1.7 million training examples, each consisting of 10s long video and audio. Each example is labeled with several labels drawn from 527 classes.
We perform experiments using the protocol described in Jaegle et al. 2021), training models for 100 epochs using 32-frame clips at train time and 16 overlapping 32-freme clips at test time. As in the ImageNet experiments, We compare the performance of Perceiver and Perceiver IO using models that are matched except for the decoder (we use an average + project decoder for Perceiver and query-based attention decoder for Perceiver IO, see Sec. E.3 and Fig. 6 All models an architecture with 12 processor modules and a latent index dimension N of 512 (we omit the repeated cross-attends used in Jacgle et al. (202T)). We compare models taking video and either raw audio or mel-spectrogram (pre-processed audio) as input. For all four model settings, we swept the number of latent channels (using (512, 1024)) and report the best value for each setting. We performed no additional tuning.
Results of this experiment are shown in Tab. 10 using the attention-based decoder leads to small but consistent improvements over the less generally applicable average + project decoder. Because Perceiver IO introduces no domain assumptions not present in the original Perceiver, this is evidence that Perceiver I0 is strictly more general model.
We find that as in the ImageNet experiments,
D FLOPS CALCULATION
In all cases, we report theoretical FLOPs with multiplies and accumulates counted as separate operations. This the strategy used in Kaplan et al. (2020) and elsewhere the literature. We use this strategy consistently here to allow comparisons between the models propose and develop (including our BERT reimplementation) Note that some papers in the literature report FLOPs using fused multiply-accumulle using this strategy will cut the figures reported here in half.
xxi

---

Published as a conference paper at ICLR 2022
Encode
Input array
K
Latent
Latent array
array
Attention scores
Process
V
Latent array
K
Latent array
Attention scores
Decode
V
Latent
array
K
Output
Output
query
Q
array
array
AEtention
scores
Figure 5: Schematic depiction of encode, process, and decode attention. Each attention module uses the same operations, differs in which inputs are used to generate key/values queries and in the output shape. Encode attention can be viewed as mapping an input latent space, typically with a smaller index dimension (fewer elements). Decode attention can be viewed as mapping latent to output space, often with larger index dimension (more elements). Both of these forms of cross-attention Process attention (self-attention) preserves the input index dimension (same elements). Red and blue dashed lines used highlight the matrix multiplications used in QKV attention, as described in the text.
E ARCHITECTURAL DETAILS
Perceiver 1O constructed from GPT-2-style (Radford et al.) 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs and outputs from the QKV attention and MLP take desired sizes. Using the array sizes of the encoder attention, the QKV attention takes in two two-dimensional arrays, key-value input array XxV IR MC and query input array XQ R N and maps them to an array X XQKV R Nx sharing the shape of the query input (after projection). XQкV is used as input to an MLP, which is applied independently to each element of the index dimension (i.e. convolving the MLP with input along the first dimension), producing final array X MLLP ND
While we describe attention as taking two inputs, in standard Transformers it is typically described mapping one input to output of the same size. This is because all modules of standard Transformer use self -eff-attention, where the same input is used for both key-value inputs and query inputs. The view of attention that describe encompasses both cross- cross-attention and self-attention, both of which are specific ways of using OKV-attention. Perceiver I0 uses cross-attention for encoder and decoder attention modules and uses self-attention for the latent processing modules. These modules differ primarily in terms of what shape data they ingest and produce (Fig.) 5)
We now describe the structure of QKV attention and the ML. in detail.
xxii

---

Published as a conference paper at ICLR 2022
E ATTENTION MODULE INTERNALS
QKV attention takes in two two-dimensional arrays, a query input Xę E R and a key-value input Xxv E RMxx The output of QKV attention is an array with the same index (first) dimension as the query input and channel (second) dimension determined by output projection:
Q = fo(x))): == V
(1)
XQK softmax(QK*/ /VF Aum(XQ, (annXX,XKV) = XQKV fo(XoKV),
(2)
(3)
where XΩк is an array of attention maps E RNXM, and XΩKV is an anay E RNXD The functions f(Q.K.V) are linear layers mapping each input to shared feature dimension F and fo linear layer projecting the output to a target channel dimension, which is often the same size as Xq s. All linear layers are applied convolutionally over the index dimension (the first dimension of their inputs). We have omitted batch and head dimensions (in the case of multi-headed attention) for readability. QKV attention is followed by a two-layer MLP with a GELU (Hendrycks & Gimpel 2016) nonlinearity following the first layer. The full module has the following structure:
XQRV=
Atm Atm((ayerNorm(Xx)) MayeeooooX layerNorm(Xxv))
(4)
XQxV = XΦxv + XQ
(5)
XQRV= = XΦRv + MLPP(layeNNom((qKv))
(6)
slightly abusing notation for simplicity and to emphasize the residual structure. "Attn" refers QKV as described above.
In the context of decoder attention, sometimes find helpful omit the second step (XQRV XΦxv + XQ), as it involves adding the model output with a query. Queries sometimes include features inherited from the input space (Tab. (6). and this residual connection may make learning unnecessarily difficult. For example, for optical flow. including this residual connection forces the network to produce optical flow output by adding RGB and Fourier features to the model's output.
E.2 COMPUTATIONAL COMPLEXITY
The computational complexity of each attention module is dominated by the two matrix multiplica- tions in QKV attention. Still using the shapes of the encoder attention, these two matrix multiplies involve matrices of shape M x F' and complexity of O(MN F). Let M, N, and O be the index dimensions for the input, latent, and output arrays, and to simplify the analysis let be the feature size for all layers. The KV and Q sizes for the encoder, latent transformer, and decoder will then be M x F and N x F (for the encoder), N x F and x F (for the latent transformer), and N x and O F (for the decoder). model with latent attention blocks has complexity O[M + O LN) In other words, Perceiver 10 has complexity lincar in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadrate scaling with input/output size, Perceiver 10 is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution). For further discussion of these points, see Sec. 2 and Sec. A of Jaegle al. (202T)
and M N and N F. giving overall time and memory
E.3 USING THE DECODER FOR CLASSIFICATION / REGRESSION
As we show in ImageNet and AudioSet experiments, the attentional decoder used here can be used in settings where standard average + project decoders are applicable. We find that the attentional decoder typically produces somewhat better results than the standard decoder. This likely occurs because attentional decoding is more expressive than average + project decoding. To make this clear, illustrate the two pooling schemes in Fig. Both decoders be viewed as first averaging the
xxiii

---

Published as a conference paper at ICLR 2022
Latents
Attention Decoder
Latents
Avg+proj Decoder
V
K
Output
Output
MLP
proj
Query
Pooling weights
Pooling weights
(attention scores)
(1/N)
Figure 6: Single-query attention decoder (left), as used Perceiver IO for classification tasks and standard average project decoder (right), as used in Jaegle et al. (202T). Both modules can be seen as first aggregating latents by weighted averaging (learned, data-dependent weighting for the attention decoder, uniform weights for the average project decoder) and then projecting to an output channel dimension (linear value projection + MLP for the attention decoder; simple linear projection by the average + project decoder). Attentional decoding more expressive than average project decoding and follows the same architectural template as encoder and processor modules.
latents and then projecting them target shape, but decoder attention uses more expressive modules for each of these operations. Instead of uniformly weighting each input in the averaging operation, decoder attention uses the attention scores as data-dependent weights for each input point. Instead of projecting the raw averaged input to target dimensionality, decoder attention first projects inputs via a value layer and then processes them with an MLP. In addition to its greater expressivity, decoder attention has the advantage of being easily generalizable dense outputs (by increasing the number of queries) and of reusing the same architectural pattern used for the encoder and processor modules.
LANGUAGE: ADDITIONAL DETAILS
OTHER TOKENIZER- MODELS
One application of Perceiver I0 is byte-level language processing, which has concurrently been addressed by several other groups Clark et al. (2022) trains models on Unicode code points and shows results competitive with subword-based models on multilingual question answering dataset. Tay al. (2022) trains on UTF-8 bytes directly by introducing hand-desigma module that is trained end-to-end to perform subword tokenization and produces results on-par with and sometimes better than subword- based models. Xue al. (2022) trains encoder-decoder models on UTF-8 bytes directly and shows that making the encoder 3x deeper than the decoder leads to comparable performance with subword baselines.
F.2 ARCHITECTURE DETAILS
The architecture hyperparameters and the training speed for the Perceiver IO used in the language experiments are given in Tab.
Model
BERT Base
BERT matching FLOPS
Perceiver 1O Base
Perceiver 10
Perceiver 10++
Tokenizer
bytes
SentenedPicx
bytes
bytes
Number of inputs (M)
512
2048
512
2048
2048
Input embedding size (C)
768
768
768
768
768
Number Process layers
26
26
latents (N)
256
Lateni size (D)
1280
1280
FTW hioden dimension for latents
1280
1280
1536
Number of outpot queries during pretraining
512
2048
2048
queries
FFW hickon dimension for outputs
768
768
Training steps/second
7.3
2.9
7.4
7.6
4.2
Table 11: Perceiver IO architecture details for language experiments.
xxiv

---

Published as a conference paper at ICLR 2022
F.3 MLM PRETRAINING
We pretrain all models on a mixture of the C4 dataset (Raffel et al. 2020) and English Wikipedia, where 70% of the training tokens are sampled from the C4 dataset and the remaining 30% from Wikipedia. We concatenate 10 documents before splitting into crops to reduce wasteful computation on padding tokens. We use the same masking strategy for SentencePiece and byte-level experiments: each word is masked independently with probability 15% where word boundaries are defined using white-space boundaries.
The pretraining hyperparameters are given in Tab. 12 For the BERT (matching FLOPs) model trained on bytes, we reduce the model width from 768 to 512, the feed-forward hidden size from 3072 to 2048, the number of layers from 12 to 6 and the number of attention heads from 12 to 8. Given the longer sequence length of 2048 bytes, this model has about the same number of inference FLOPs as BERT Base model on sequence length of 512 tokens.
In order to decode, we use learned queries of the same dimension of the input array (Tab. I1.. We have many output queries inputs to be able to predict the masked token at all positions in the sentence (M=0).
To get insight into the learnt queries visualize the attention weights in the first cross attention layer on a small paragraph (Fig. 77. We discover that the model has learnt both position and content based look-ups. The caattion-based look-ups can be either very sparse and precise more distributed and periodic. This second mode appears somewhat less often and is more efficient because more data is being attended to the averaged: this acts as a learned pooling. The content based retrievals focus mostly on syntactic elements like capital letters and punctuation (colon, exclamation marks, quotation marks, etc). This probably because these are good word delimiters and can help the model reduce prediction uncertainty.
time, but also more distributed, since the values subsequently
Training steps
500,000
Batch size
2048
Masking strategy
Words
Optimizer
LAMB
You et al. 2021
Learning rate
0.00125
Linear warmup steps
1,000
Cosine cycle decay
500,000
Weight decay
0.01
Table 12: Hyporparameters for masked language modelling (MLM) pre-training experiments
F.4 GLUE FINETUNING
Following Devlin et al. 2019), we specify a fixed-size byperparameter grid and select the best dev performance across that grid for each task independently (Tab. 12. The full GLUE results are shown in Tab. 14 Following Devlin et al. (019) we exclude the WNLI task. We use accuracy for all tasks expect STS-B and CoLA where we use Pearson correlation and Matthews correlation respectively. The average is computed by first averaging the results of MNLI-matched and MNLI-mismatehed, which is then counted as single task in the overall average.
For single-task experiments, we do not require [CLS] token as we use single decoding query vector. both single-task and multi-task experiments an extra 2-layer MLP with a hidden size of E and a tanh activation is used to map the the Perceiver 10 outputs to the class logits (or regression target for STS-B).
F.5 ABLATION ON THE NUMBER OF LATENTS
For given FLOPs budget, there is a trade off between the number of latents N and the width D of the latents. We ablate this in Tab. [5] by varying the number of latents between 128, 256 (best), and 512. We adapt the latent dimension accordingly to match the FLOPS budget.
XXV

---

Published as a conference paper at ICLR 2022
A bear walks into a restaurant. He tells his waiter, "I want a grilled...heese!. The waiter says, "What's with the pause?" "Whaddya mean?" the bear replies. "I'm a bear!" A bear walks into restaurant. He tells his waiter, "I want gcheeed The waiter says, "What's with the pause?" "Whaddya mean?" the bear replies. "I'm a bear!"
(a) Very sharp location based attention.
A bear walks into restaurant. He tells his waiter, "I want grilled...cheesse " The waiter says, "What's with the pause?" "Whaddya mean?" the bear replies. "I'm bear!" A bear walks into restaurant. He tells his waiter, "I want gceelld...cheese! The waiter says, "What's with the pause?" WWhddyy mean?" the bear replies. "I'm bear!"
(b) more efficient more distributed "periodic" location based attention.
A bear walks into restaurant. He tells his waiter, "I want grilled..cceesee"" The waiter says, "What's with the pause?" "Whaddya mean?" the bear replies. "I'm bear!"
A bear walks into restaurant. He tells his waiter, "I want grilled.. says, "What's with the pause?" "Whaddya mean?" the bear replies. "I'm bear!" A bear walks into restaurant, He tells his waiter, "I want grilled...chess./ The waiter says, "What's 's with the pause?" "Whaddya mean?" the bear replies. "I'm bear!"
The waiter
(c) Content based attention for syntactic elements like punctuation and capital letters.
Figure 7: Visualization of attention weights for few queries in the initial cross- rross-attention layer. We use the color to convey the weight of the attention and normalize by the maximum weight to make them easier to visualize. Best viewed in color.
Training epochs
10
Batch size
[16, 32, 64)
Optimizer Learning rate
LAMB
(1x10-4,5x 10-5 5, 2 2x110-, 1x10-5
Linear warmup steps Weight decay
200
0.01
Table 13: Hyperparameters for GLUE finetuning experiments. We sweep over the values in brackets.
G POSITIONAL ENCODINGS FOR IMAGE AND AUDIO EXPERIMENTS
For all image experiments (with the exception of the ImageNet experiment that uses learned positions, Sec.AA.1. we use 2D Fourier feature positional encoding (Vaswani et al. 2017 Stanley 2007] Mildenhall et al. 2020; Tancik al. 2020) using sine and cosine bands with frequencies spaced linearly from a minimum frequency to maximum frequency. We use 64 sine/cosine bands per dimension in all settings. The minimum frequency is always
the minimum frequency of the input signal, corresponding to a single full oscillation over the input dimension. The maximum frequency typically set the input's Nyquist frequency (e.g. 112 cycles for an image with 224 pixels per dimension). The input position used to construct the Fourier frequencies is scaled to [-1, 1] for each input dimension. For example, the upper left corner of an image position [-1,-1] while the bottom right corner is at position [1. 1]. We follow the same strategy using 1D and 3D Fourier feature positional encoding for audio's time and video's spatiotemporal inputs, respectively.
H OPTICAL FLOW: ADDITIONAL DETAILS AND RESULTS
Pre- and post-processing can provide non-trivial inductive biases when processing image data and also change computation time. In this section, ablate these choices. The network in the main paper concatenates the two frames frames before extracting 3D patches around each pixel, each of size 3x3x2. Tab. [6 shows few alternative designs for patch extraction. 1x1 1 means that only a single pixel (or pair of pixels) is used for each input element. 'Separate frames' means that the frames are not concatenated, but rather, input array elements extracted independently from the
Xxvi

---

Published as a conference paper at ICLR 2022
Medel
Tokemizer
Molti-task
Cal.A
RTF
STS-B
Averagge
(eet)) (Devlin 2019
SentencePPece
64.0083.00
66.40
80.95
(ours)
SemsencePacce
61.72
58,04
Perosiver Base
SentenccPPece
84.53785.08
57:25
92.12
65.25
94.38
58.15
81.16
(matching FLOPS)
76.11/75:55
77.00
53.91
$2.84
71.45
Penosiver
UTF-- Bytes
83 23.22/83.89
91.71
98.17
56.81
80L93
Perceiver 10++
UTT-8 Dytes
No
52.56
84.13/84.91
96.03
92.06
90.96
66.54
90.98
87.93
81.76
Percetver 10 (Shredd input token)
UTT-8 Dytes
47.43
82 82.00/82.60 65
89. 39.58
90.18
89.20
$2.03
93.17
97.95
81.49
Percewver 10 Tlsk specine ingut token)
UTE-8 Bytes
Yes
49.06
N2. 1214482244
89.38
90..5
89.40
79.69
93.17
80.02
81.76
Percewerr 10 (Mutarask quesy)
UTF-8 Bytes
Yes
47.88
N2.05/82.77
90.36
90..37
89.49
90.018
93.75
79.95
81.79
Table 14: Full GLUE results (higher is better). The first 3 models use SentencePicce tokens, the latter 3 usc UTF-8 bytes directly.
Number of latents (N)
Latent width (D)
FLOPs
Average GLUE score
128
1920
120B
75.84
256
1280
113B
80.95
512
896
125B
80.92
Table 15: Ablation on the UTF-8 Bytes Perceiver 10 latent width versus depth.
two frames (thereby doubling the number of input elements). the case of separate frames, 1x1 means essentially no preprocessing: each pixel becomes its own element with spatio-temporal context whatsoever.
We also performed experiments with less expensive input model which uses 7x7 convolution to 64 channels, followed by a max pool, similar to the one used in our ImageNet experiments. After feeding this through the Perceiver 10 architecture (including querying with the same convolutional features used as input). we have an output feature grid with stride 4 and 64 channels, on top of which we apply a RAFT upsampling layer. This involves a linear projection from 64 dimensions to 2, which is the parse-resolution optical flow estimate. We then upsample this flow for given pixel in the high-resolution flow map by applying attention over a neighboring 3x3 block of the low-resolution flow map, following the uppsampling approach in RAFT (Teed & Deng] 2020).
We found that concatenating frames led to a non-trivial performance improvement across the more difficult Sintel.final and KITTI Flow 2015 (Menze & Geiger, 2013) datasets. Spatial context helps, and the impact of frame concatenation is larger when more context is available, suggesting that the algorithm is comparing spatial and temporal gradients. Convolutional downsampling and RAFT upsampling provide even more spatial context for both the input features and the queries, but this doesn't make up for the loss of resolution and overall performs slightly worse than using the full resolution.
Perceiver I0 is somewhat slower on traditional GPUs than our baseline RAFT model, but find that the trend reverses on TPUs, which is the target architecture for our work. For ease of comparison, report inference speed on 1088 436 images, using tiled inference setup. Our most expensive model achieves approximately 0.8 frames/sec on 2017 TITAN Xp, and our lightweight model (with conv downsampling and RAFT-style upsampling) achieves 3.3 frames/sec, which is far from the 10 frames per second reported for RAFT (Teed & Deng, 2020). On the v3, however, our most expensive model achieves 4.4 frames/sec on single TPU core, and 17.8 frames/sec for the lightweight model. An efficient Tensorflow implementation of RAFT (Sun et al.) 2020) (received courtesy the authors) achieves only 1.6 frames/sec on the same hardware. We suspect that the difference is due to the gather operations required for RAFT but not for Perceivers, which are slow on TPU due to their poor memory locality properties.
TPU
Fig. 8 shows some results example image pairs from the Sintel.final dataset. We that the algorithm is capable of dealing with heavy occlusion, and can propagate optical flow across large regions with very little texture. The network also deal with very large motions and very small objects.
Finally, to verify that Perceiver 10 performs well real-world data despite being trained only on synthetic imagery, we applied it to a small number (roughly 10) real videos taken from Getty images (www.gettt)) wages..aa......m com). Perceiver IO typically performs very well out-of-domain,
xxvii

---

Published as a conference paper at ICLR 2022
Method
Patch size
Concat. Trames
Downsample
Depth
Latents
SinteLelean
Sintel.final
KITTI
PWCNet (un
( et
2018
2.17
2.91
5.76
RAFT Deng 2020
1.95
4.23
Pereciver 10
3x3
Yes
No
24
2048
181
2.42
4.98
Pereeiver
No
No
1.78
6.19
Perceiver 10
1x1
Yes
No
24
2048
1.91
2.56
5.39
Perceiver 10
NIA
Yes
Yes
24
2048
1.84
2.52
4.83
Pereoiver 10
No
Yes
2048
1,90
2.53
6.66
Perceiver 10
NIA
Yos
Yes
16
1024
2,06
2.67
6.12
Table 16: Ablated Optical Flow results (end-point error. lower is better). The top Perceiver I0 results show configuration from the main paper. We ablate 1) patch size for the context surrounding each pixel, 2) whether the two frames are concatenated downsampled by factor of 4 using convolution, and then subsequently upsampled with RAFT, and finally the number of self aelf-attention modules (depth) and number elements the latent array, resulting in bottom-row network which substantially less expensive than the original model.
input separately to the Perceiver, 3) whether the inputs and queries are
Figure 8: Qualitative examples of optical flow. For each image pair, show the frames (top), and then the estimated flow (bottom left) and the ground-truth flow (bottom right), In the left example, we see one person under heavy occlusion where the correct flow is propagated into a region with few details. Another person in the foreground has clothes with little texture and substantial blur, and yet the algorithm can propagate the flow across the entire region. In the center example, we see very large motions from both the dragon and the person, yet many fine structures preserved like the pole. On the right, we see forest scene with a few extremely small objects with very subtle motions (circled) which our algorithm is able to detect and segment correctly.
although some failure cases remain: for instance, shadows tend to be interpreted as objects (Autoflow contains no shadows), and large regions with compression artifacts but no other texture may result in hallucinated flow (Autoflow contains no video compression artifacts). We include three challenging examples in the supplementary zip file, each of which depict complex motion and small objects. Perceiver IO can pick up remarkably small objects such as the water droplets thrown by the girl's shoe pigeon. mp or the confetti thai thai_dance mp4.
Implementation details: Our experiments with pixels and patches use sine and cosine position encoding with 64 bands for both features concatenated to the pixel patch values. For experiments without concatenated frames, have an additional time dimension which must be encoded with positional encoding, and for this also use 64 sine and cosine bands (which are highly redundant, there's only two frames). For this version, only the elements associated with the first frame are included as queries for the decoder. For both input and query, we project these concatenated features to 64 dimensions before inputting them into the transformer. We use latent array with 2048 elements and 512 channels and 24 self- attention modules, each with 16 self-attention heads, unless otherwise noted. Our experiments with convolutional downsampling and RAFT upsampling use settings that are mostly similar, although we use no additional projection as the output of the convolutional network is already 64 channels. For these experiments, the output of the perceiver decoder's cross attend is 64 channels, which is fed into a RAFT-style upsampling operation. For the pixel- and patch-base models, total computational complexity for forward pass 368 x 496 image is roughly 987 billion FLOPs, and there roughly 27.9 million parameters.
Y, plus the raw and Y values resulting 258 extra
In all cases, train the AutoFlow dataset (Sun et al. 2021 which consists of 400, 000 image pairs, for 480 epochs using a cosine learning rate schedule which starts at learning rate of 4e-4. We batch size of 512. We the LAMB (You et al. (202I) optimizer. We also the default curriculum for AutoFlow, which gradually increases the severity of the augmentations over time. We find that naïve training Floo does train, use additional phase in this curriculum,
xxviii

---

Published as a conference paper at ICLR 2022
Params
FLOPs (train)
FLOPs (eval)
Train steps/sec
20.0M
310B
6.85T
4.4
Table 17: Additional details of the model used for Multimodal autoencoding.
where we completely disable all augmentations. Furthermore, for this phase, we feed every image pair twice in a batch: once forward, and once reversed. As the inverse flow is not currently available for AutoFlow, this inverse flow was computed via an approximation which averages all the flows terminating at a given pixel.
The evaluation datasets have different resolution, so we evaluated in a tiled manner, using six evenly-spaced tiles. For pixels that are covered by multiple tiles, we average the predictions, weighted proportional the distance to the nearest edge of the respective tile (as we expect predictions nearer to the tile edges to be less accurate). We leave the possibility of making Perceiver I0 invariant to input shape to future work.
I MULTIMODAL AUTOENCODING: ADDITIONAL DETAILS
For the multimodal autoencoding experiments, we patch preprocessing for both images and audio, and we embed the labels as one-hot labels. The patch size is 1 x 4 4 for video and 16 for audio. The audio is sampled at 48kHz, or 1920 samples per frame. The decoder outputs 16 x 224 x 224 + 16 1920/16 - vectors with 512 channels, that is, one element for each pixel in the video, one element for each audio patch, and one element for the classification label. These are then linearly projected to the appropriate channel size for each modality: 3 for videos, 16 for audio and 700 for classification (the logits for each of the 700 classes in Kinetics700). Finally, we un-patch the audio to arrive the output audio. We note that we read and generate the audio waveform directly in the time domain; we do not transform first to spectrogram.
We use 387 dimensional 3D Fourier position embedding for each input video patch and 385 dimensional ID Fourier position embedding for each audio patch (385 to ensure the input dimensions Perceiver IO match for all elements). addition, we pad all imput elements with learned vector representing the modality; inputs from the same modality share the same token. In particular, we add a 317 dimensional modality embedding to video elements, a 319 dimensional modality embedding to audio elements, and 4 dimensional modality embedding to the label, so that all elements have 704 features.
The decoder queries are also constructed from Fourier position embeddings for video and audio and a learned positional embedding for label: 387 features for video, 385 features for audio, and 1024 learned features for the label. We pad the queries for each modality with different learned vector for each modality, so that the final feature size for the queries 1026.
We train Kinetics 700 (Smaira et al. 2020). We use batch size of 1024, and learning rate of le-3. The training loss is a weighted sum of the L1 loss for video, the L1 loss for audio, and the cross entropy loss for the label. The weightings are 0.03 for video, for audio, and 0.0001 for the label; the loss weights are imbalanced in favor of audio because it is more difficult to obtain audio of high perceptual quality by directly outputting the waveform. We also tried different weighting (0.03 for video, 1 for audio. and 1 for the label) to obtain higher classification accuracy. Additional model details are given in Tab. 17
To help verify the quality of Perceiver IO's outputs on real-world data, we applied it small number of real videos (~10) with audio taken from Getty Images. Perceiver I0 is able to capture the structure of both video and audio inputs, despite encoding both jointly with single network. The model introduces blurriness to both video and audio: this may be partially attributable the preprocessing which included coarse patching (Tab. (5) for both modalities due to the very high computational cost of processing raw video and audio inputs (which amount to over 2 million raw points). Although decoding can be done in parallel, allowing us to decode very large output arrays in sequential batches, Perceiver I0 requires all points are encoded simultancously. Addressing this limitation and scaling to even larger inputs is important direction for future work.
xxix


Reviews:
[{"review_id": "WGNpfTgIIY4", "reviewer": "Reviewer_qg6s", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. This paper introduced a generic architecture for coping with different tasks with various input and output lengths. It is different from Transformer architecture and Perceiver architecture in that it uses a latent process to compress the larger number of input tokens into a hidden space, and then decode the output from this hidden space. This way it can decouple the inputs and outputs which has the potential to significantly reduce the cost brought by a large number of input and output tokens. \n\n2. The authors presented a thorough study of Perceiver IO across different application scenarios, such as language, vision, multi-modality. Extensive experiments demonstrate that the proposed method can achieve comparable or even better performance than the baselines. To facilitate the adaptation, the authors suggested a number of good ways to convert the inputs/outputs into certain formats. \n\nCons:\n\n1. The authors should list the sizes of inputs, latent, and output arrays for each of the downstream tasks to give the audience a better sense of the complexity. For some of the tasks, such as ImageNet classification, the authors should also report the FLOPs in comparisons with prior arts, e.g., ViTs. \n\n2. The main merit of Perceiver IO compared with Perceiver is that it further introduces a cross-attention-based output decoder on top of the latent arrays. Though Perceiver was originally proposed for classification tasks, it can be also used to cope with structured outputs. For some of the tasks except for ImageNet classification listed in this paper, I am curious whether Perceiver can be applied and how it performs compared with Perceiver IO.\n\n3. Perceiver IO is proposed as a generic architecture for various tasks by modeling them as a read-process-write process. One question is that whether we should reformulate all these tasks as the way shown in this paper. This paper does show some encouraging results on various tasks compared with the baselines. However, they are only compared with baseline methods and still underperforms established methods in specific domains. As we know, different tasks can still benefit a lot from the specific domain knowledge (e.g., 2D spatial for image recognition). Actually, we can also think of the Transformer encoder as a generic architecture in numerous domains. Then, do we need to build up a higher-level generic architecture upon the Transformer encoder? \n\n4. A good part of unifying architectures for various tasks is that we can train the same set of parameters using different tasks. In this paper, the authors demonstrated the effectiveness of Perceiver IO across different settings, I am wondering whether such a generic architecture can be used for multi-task training so that it can leverage the training data from different tasks. ", "summary_of_the_paper": "In this paper, the authors proposed a new general architecture called Perceiver IO for various tasks with different types of inputs and outputs. The Perceiver IO employes a read-process-write architecture, in which input arrays are first projected to the latent space through cross-attention and then the outputs are generated by querying the latent space through some output query arrays. Such a generic pipeline can be applied to various tasks spanning from single modality language tasks, multi-modal tasks, dense prediction tasks, and even symbolic predictions. The main benefit of the proposed Perceiver IO is that it decouples the inputs and outputs using a latent processing module so that it can cope with the arbitrary length of input arrays while outputting an arbitrary number of predictions. Extensive epxeriments are conducted on language understanding and masked language learning, optical flow estimation, image classification, multi-modal autoencoding, etc, and showed that the proposed Perceiver IO can achieve comparable performance to strong baselines in different domains.", "summary_of_the_review": "Overall, I like the idea of Perceiver IO considering it is a good way of modeling arbitrary numbers of input and output tokens. Based on the experimental results on various downstream tasks, it indeed shows some superiorities over the strong baselines. The experiments are solid enough to justify the main claim. However, as discussed above, my main question is that whether we want to develop a unified read-process-write architecture for all these tasks. What are the main merits of this act? I will hold my score to the borderline and wait for more feedback from the authors.\n\n[Post-rebuttal comments]\n\nAfter reading the authors' feedbacks to all reviewers including myself, I think they had addressed most of the concerns and questions. All reviewers think this work is novel and appreciate its solid contributions. As such, I raised my rating to \"accept\", and look forward to the furture work of applying it to learn from multiple tasks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RAq_R3k8iF", "reviewer": "Reviewer_MYnu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strength of Perceiver IO is its high flexibility to various input domains. While keeping the main architecture, we can focus on engineering the input query preprocessing with domain-specific knowledge. Perceiver IO not only shows flexibility but also shows high performance. The results are comparable to or sometimes outperform previous domain-specific methods. For example\n\n- Table 1 shows that Perceiver IO works well even without tokenization, while BERT + UTF-8 bytes show a significant performance drop (81.0 -> 71.5). It may show that Perceiver IO can perform well even without minimal domain knowledge (tokenization).\n- Table 2 shows that the input query engineering can improve the target task (e.g., single-task query to 8 multi-task queries).\n- Table 5 shows that the small input preprocessing engineering can improve the performance a lot. For example, Perceiver IO shows 82.1 top-1 accuracies for conv preprocessing while Perceiver IO with Fourier features shows 79.0.\n\nI have minor concerns about ImageNet experiments.\n\n- Although this paper is not aiming to state-of-the-art ImageNet performances, I think Perceiver IO is not yet comparable to state-of-the-art models on ImageNet image classification (as described in the last paragraph of Section 1). Perceiver IO still needs tremendous FLOPs compared to vision-specific models (Table 17). For example, ResNet shows 78.6 top-1 accuracy with 4.1 GFLOPs, while Perceiver IO Fourier shows 79.0 top-1 accuracy with 407 GFLOPs (99 times larger FLOPs). Stronger Perceiver IO (conv) shows 82.1 top-1 accuracy with 369 GFLOPs, where CaiT-M48 448 shows 86.5 top-1 accuracy and 329.6 GFLOPs.\n- \"Perceiver IO (pretrained)\" and \"Perceiver IO\" in 2D Fourier features are different models. For example, \"Perceiver IO (pretrained)\" does not use weight sharing, and uses larger channel sizes. As a result, Perceiver IO (pretrained) has 4.38 times larger parameters compared to Perceiver IO (but half GFLOPs). The current text can make a reader mislead that \"Perceiver IO (pretrained)\" is the same model as \"Perceiver IO\", but the only difference is the JFT pretraining, which is not true.\n\n> Unlike in the other ImageNet experiments, we do not share weights in the latent self-attention process modules, but use a 16-layer latent network with no weight sharing in depth. Unlike the other ImageNet experiments, the process-module MLPs use a hidden layer with 4× the number of channels (rather than 1× as on other ImageNet experiments)\n\nTo avoid confusion, I suggest replacing Table 5 with Table 17. Also, I would like to suggest to clarify that \"Perceiver IO (pretrained)\" and \"Perceiver IO\" are not the same model.\n\n## Questions\n\n- Perceiver IO shows better performances consistently compared to Perceiver IO baseline on GLUE (Table 13), Audio Set (Table 7), and ImageNet (Table 17). To my knowledge, there is no specific reason to the proposed output array cross-attention works better than the conventional classification head. I wonder what is the opinion by the authors what is the source of the improvements.\n- In Table 5, I found that using 2D + maxpool preprocessing harms Perceiver performances (78.6 -> 77.4), but it helps Perceiver IO a lot (79.0 -> 82.1). As far as the reviewer understood, the main difference between Perceiver and Perceiver IO is the last decoding layer, not the input processing. I wonder what is the opinion of the authors why Perceiver and Perceiver IO show contradictory results on ImageNet Fourier features and conv preprocessing.\n- I wonder why there is no Perceiver IO (conv) + pretraining result. Do the authors have any plan to add Perceiver IO (conv + pretrained)? (this is not a mandatory experiment for the rebuttal)\n\n## Minor comments\n\n- Many details are missing in the main text. To understand the method, the readers should read through the heavy appendix and the previous Perceiver paper. I fully understand that this is because of the page limitation, but this paper is somewhat hard to understand at first glance without any prior knowledge.\n- In my opinion, it would be very helpful to readers if Table 8 and 9 are in the main text.\n- pg 3. Typo -- unnecessary space *( \"inducing points\")*\n- pg 29. Missing hyperlink (Tab. ??)\n- In my opinion, it will be very interesting if there is one unified Perceiver model pretrained on various domain data. For example, one can train a Perceiver model trained by language and vision modalities, where each modality has different input and output arrays.", "summary_of_the_paper": "This paper proposes a general-purpose neural network architecture named Perceiver IO. With only small modifications on the query side, Perceiver IO can handle various inputs, such as languages, images, videos, point clouds, optical flows, or game agents. Perceiver IO is hugely dependent on the previous work, Perceiver (Jaegle et al.). The main difference between Perceiver and Perceiver IO is the final decoding module. Instead of using a classification head, Perceiver IO uses an additional cross-attention module with an output array. For example, if a user wants to make Perceiver IO be a classification model, the output array becomes 1 X # classes. Perceiver IO shows impressive results in the experiments compared to domain-specific designed architectures in various domains (language, optical flow, audio-video autoencoding, image classification).\n\nPerceiver IO shows its flexibility on various input domains and domain knowledge. For example, Perceiver IO shows even comparable performances to the BERT baseline on the GLUE benchmark without the conventional tokenization but with only UTF-8 bytes (Perceiver IO with tokenization shows 81.1 and Perceiver IO with UTF-8 bytes shows 81.0 while BERT baseline with comparable FLOPs shows 81.1). On the other hand, in the vision domain, by using 2D convolution and max-pooling preprocessing, Perceiver IO shows better top-1 accuracy compared to 2D Fourier features (82.1 for conv preprocessing and 79.0 for 2D Fourier features).\n", "summary_of_the_review": "This paper provides very a strong empirical contribution to a unified architecture for various input domains. Not only showing flexibility, but Perceiver IO also shows powerful performances in various domains. Although I have a small concerns that ImageNet results (a reader can mislead the results) I think this paper is a good paper and indeed recommend acceptance.\n\n--------------------\nPost-rebuttal comments.\n\nThe authors addressed my concerns very well in the rebuttal. I will keep my decision as \"8: accept, good paper\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "HkGOlWnfwux", "reviewer": "Reviewer_2doh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strength\n- The idea of perceiver IO is novel and solid -- a general architecture capable of handling general-purpose inputs and outputs across different tasks and modalities. This is very promising to simplify the construction of highly tuned task-specific neural pipelines and improve the multimodal and multi-task problems. \n\n- Each component in perceiver IO is necessary and well defined for the proposed tasks. 2D byte array input enables the task agnostic input for different modalities. Perceiver network that scales linearly with the size of input and output, and non-auto regressive decoding make the decoding to raw output possible. \n\n- The proposed architecture is tested on massive experiments including language understanding tasks, optical flow, video audio class autoencoding, image classification, and starcraft II and achieves superior performance. Each task is supported with a detailed ablation study to shed light on future research.\n\n#### Weakness\n- In Table 1, the WNLI task is excluded from the GLUE benchmark, I wonder what is the reason this task is removed? \n\n- The non-autoregressive decoding enables fast decoding for large outputs. However, for tasks that require more information on prior decoding context (machine translation or image generation tasks), does the proposed model can still perform well on those tasks? \n\n- Although perceiver IO removes the task-specific pre-processing (tokenization, patch embedding, etc.), the model still requires huge engineering efforts to adapt the model for different tasks. For example, the model hyper-parameters are quite different for different tasks. \n\n- One huge benefit of perceiver IO is to train different tasks together and explore the transfer between different tasks/modalities, which is not explored in this paper.  ", "summary_of_the_paper": "This paper proposed perceiver IO, which is a general architecture for structured input & output. Perceiver IO is based on the perceiver architecture which scales linearly with the size of inputs and outputs and augments with a flexible querying mechanism similar to NeRF. Different from prior work, perceiver IO directly operates in the raw input space -- UTF-8 bytes for language, xy coordinates in optical flow, raw audio, etc. The output can be arbitrary in size and different structures, and with non-autoregressive decoding, the model can handle large input and output sizes. The proposed model is tested on variety of tasks, including language modeling, optical flow, video audio class autoencoding, image classification, and starcraft II, and achieves superior performance. ", "summary_of_the_review": "The paper is well written, the idea is solid and novel, supported with massive and strong experimental results. Overall, the is a strong submission and I would recommend accepting the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "BNEcks2-bA", "reviewer": "Reviewer_h233", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths\n\n- Major contribution is ability to scale the output to arbitrary length using an output query array. This enables supporting different types of output and scales well to different tasks. \n- Byte level performance of Perceiver IO is impressive compared to BERT baseline. Byte level embeddings can be fed to Perceiver IO without any tokenization and this setting gets comparable performance with BERT sentence piece tokenization. \n- Shows good results with multitask learning, using both single and task specific tokens.\n- Authors do a thorough evaluation of this architecture on a wide variety of tasks, and achieves comparable results on multiple tasks in language, vision, multimodal, RL domains. \n\n#### Weaknesses\n\n- Novelty over Perceiver : Although I find this work quite impressive as it scales linearly with output embedding sizes as well as arbitrary types of outputs, the overall architecture seems incremental compared to Perceiver. One important change is there is no cross attention in the intermediate layers. I would love to see more detailed analysis why that is removed in this architecture and empirical evidence to support this change. \n\n- How does choosing values for N,D affect the performance on various tasks. If we decouple input representation, output representation and base architecture, how does model performance scale with depth, N, D values.  Ablations on these hyper-parameters are required for more clarity. \n\n- Given Perceiver can handle multiple modalities, I would be interested to see how it performs on multimodal tasks like VQA, Image/Text Retrieval, Hateful Memes etc and comparison to multimodal models like CLIP[1], ALIGN[2], SimVL[3].\n\n  - [1] Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry et al. \"Learning transferable visual models from natural language supervision.\" arXiv preprint arXiv:2103.00020 (2021).\n  - [2] Jia, Chao, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. \"Scaling up visual and vision-language representation learning with noisy text supervision.\" arXiv preprint arXiv:2102.05918 (2021).\n  - [3] Wang, Zirui, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. \"SimVLM: Simple visual language model pretraining with weak supervision.\" arXiv preprint arXiv:2108.10904 (2021).\n ", "summary_of_the_paper": "This paper proposes a general purpose architecture(Perceiver IO) which can take any arbitrary type of inputs and produce arbitrary outputs. The architecture enables this while scaling linearly with input embedding size and output size. Perceiver IO performs comparably with SOTA results on a variety of tasks from language, vision and multimodal domains, which speaks about the generalizability of this architecture. ", "summary_of_the_review": "Perceiver IO extends the Perceiver architecture to scale to arbitrary outputs and output lengths. The proposed method is evaluated on a wide variety of tasks and achieves good results comparable to SOTA in most of them. I recommend the current score, with minor reservations over added novelty over Perceiver and also I would encourage the authors to include more analysis and comparison with other SOTA multimodal models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Perceiver IO, a general neural architecture that handles general purpose inputs and outputs. It operates directly in the raw input domains, and thus does away with modality specific architecture components. The paper contains extensive experiments showing the capabilities of this architecture in different domains. The paper received very positive reviews from all reviewers. Some concerns included a need for additional details such as a missing task from GLUE, FLOPs comparisons to past works, nomenclature for the versions of Perceiver IO, etc. These concerns were well addressed by the authors. Others concerns by reviewers were the lack of experiments in a multi task setting. However, it was acknowledged by the authors and reviewers that this is an open area of research and is a good fit for future work. Given this high quality submission, strong reviews and a very positive discussion amongst authors and reviewers, I recommend accepting this paper.
Decision: Accept (Spotlight)