Transframer: Arbitrary Frame Prediction with Generative Models
Authors: Charlie Nash
Joao Carreira
Jacob C Walker
Iain Barr
Andrew Jaegle
Mateusz Malinowski
Peter Battaglia
Venue: Accepted by TMLR ()

Abstract: We present a general-purpose framework for image modelling and vision tasks based on probabilistic frame prediction. Our approach unifies a broad range of tasks, from image segmentation, to novel view synthesis and video interpolation. We pair this framework with an architecture we term \modelname, which uses U-Net and Transformer components to condition on annotated context frames, and outputs sequences of sparse, compressed image features. Transframer is the state-of-the-art on a variety of video generation benchmarks, is competitive with the strongest models on few-shot view synthesis, and can generate coherent 30 second videos from a single image without any explicit geometric information. 

A single generalist Transframer simultaneously produces promising results on 8 tasks, including semantic segmentation, image classification and optical flow prediction with no task-specific architectural components, demonstrating that multi-task computer vision can be tackled using probabilistic image models.

Our approach can in principle be applied to a wide range of applications that require learning the conditional structure of annotated image-formatted data.

Under review as submission to TMLR
Transframer: Arbitrary Models
Frame Prediction with Generative
Anonymous authors Paper under double-blind review
Abstract
We present general-purpose framework for image modelling and vision tasks based probabilistic frame prediction. Our approach unifies broad range of tasks, from image segmentation, to novel view synthesis and video interpolation. We pair this framework with an architecture we term Transframer, which uses U-Net and Transformer components to condition on annotated context frames, and outputs sequences of sparse, compressed image features. Transframer is the state-of-the-art on a variety video generation benchmarks, is competitive with the strongest models on few-shot view synthesis, and can generate coherent 30 second videos from single image without any explicit geometric information. A single generalist Transframer simultancously producos promising results mantic segmentation, image classification and optical flow prediction with no task-specific architectural components, demonstrating that multi-task computer vision can be tackled using probabilistic image models. Our approach can in principle be applied to wide range of applications that require learning the conditional structure of annotated image-formatted data.
tasks, including
1 Introduction
We introduce framework that unifies wide range of tasks under the umbrella of conditional frame predic- tion, and demonstrate its efficacy on an extensive range of challenging image modelling and computer vision lasks.
Our proposed model, which we call Transfrumer, trained to predict the probability arbitrary frames conditioned on one or more annotated context frames. Context frames could be previous video frames, along with time annotations, likelihood-based autoregressive method analogous to WaveNet (Oord et al. 2016 and large-scale language
views of scene with associated camera annotations. Transframer is
models Brown et al., 2020 implemented with Transformers (Vaswani et al. 2017 and U-Nets (Ronneberger et al. 2015), whose underlying representation of the data uses the sparse discrete cosine transform (DCT) image representation introduced by Nash et al. (Nash et al. 2021).
Unlike deterministic methods for frame-to-framee modelling EEienn Pergus 2015 Kokkinos 2017 Trans-
framer maps context frames to probability distribution, which allows uncertainty to be quantified and diverse outputs to be obtained through sampling. While autoregressive models typically expensive prohibitive evaluate and sample from, Transframer's sparse representations allow more efficient computa- tion than raw pixels, which we demonstrate by sampling coherent 30 second-long videos.
We applied Transframer to a wide range of tasks, ranging from video modelling, novel view synthesis, semantic segmentation, object recognition, depth estimation, optical flow prediction, etc., and found promising results in all cases.
Our work makes the following key contributions:
Transframer's quantitative performance is stato-of-tho-art on video modelling and competitive on novel view synthesis.

---

Under review as submission to TMLR
time=1 1
view=v
Context
time=2 2
view=Va V2
task depth
task segment
task=detes
time=3
view=V3
dept-
task segment
task=detect
Target
Video
View Synthesis
Multi-task Vision
Prediction
Figure framework for general visual prediction. Given a collection context images with associ- ated annotations (time-stamps, camera viewpoints, etc. and a query annotation, the task is to predict probability distribution over the target image. This framework supports range of visual prediction tasks, including video modelling, novel view synthesis, and multi-lask vision.
We introduce residual sparse DCT representations basis for generative video modelling. Transframer is general-purpose and can be trained on wider variety of modelling tasks than are Iraditionally lackled by more domain-specific methods.
We show how to generate long video sequences (30 seconds) using our probabilistic framework.
2
Background
To contextualize this work, we first provide a description of the prior work build on, focusing on generative models of compressed image representations, and the DCTransformer generative image model (Nahh et al, 2021 An additional discussion of related work generative video modelling, view synthesis, and alternative frameworks for image-to-image translation can be found in the Appendix A.
Generative modelling with compressed image representations To deal with the high dimensionality of image, audio and video data, number of generative models operate on compressed data representa- tions fvan don Oord et al.] likelihood- generalive models, which can otherwise waste their capacily on perceptually unimportant
2017 Essor et al. 2021 Nash et al. 2021). This is particularly important for
details (van den Oord et al. 2017 For images I the general approach is as follows:
Convert images compressed codes using an encoder code Encoder(I),
Train a generative model p(Iodde) on the image codes, 3. Decode samples Ioode (Icoee) from the model back to images i, Decoder
2

---

Under review as submission to TMLR
VQ-VAE (van den Oord et al. 2017 the most well-known of these approaches, uses vector-quantized autoencoder network to spatially downsample and compress image data, reducing the size of the image rep- resentation while maintaining the most important perceptual details. The resulting image codes are modelled post-hoc using a PixelCNN-style autoregressive model, resulting in samples that are more globally coherent than samples from models trained on raw pixel data. VQ-GAN (Esser et al. 2021 uses an adversarially trained decoder to achieve higher compression rates and image reconstructions with improved perceptual quality. Transformer-basee based autoregressive architectures are an effective alternatives to PixelCNN, forming
the basis of prior generative models in mumber of works (Esser et al. 2021 Wu et al. 2021b
Sparse DCT image representations In this work we use the DCT-based image representation proposed by Nash et al. (Nash et al.) 2021], rather than raw pixels. The DCT-based image representation is heavily inspired by the JPEG codec (Wallace 1991;; and it encodes data in several stages:
1. The DCT is applied to non-overlapping image patches for each channel in the YUV color space. The UV colour components are optionally 2x downsampled before applying the block-DCTT
2. The resulting DCT data are lossily quantized using a quality-paramettrized quantization matrix. Lower quality settings result in stronger quantization, and a greater loss of information.
The quantized DCT data are converted a sparse representation: a list consisting of tuples d (c,p,v) v) of DCT channels, positions, and values for the non-zero elements.
4. The list is sorted for the purposes of generative modelling. As in Nash et al. we order from low-to- high frequency DCT channels, interleaving luminance and color channels. Within DCT channels, list elements are sorted by spatial position in raster-scan order.
The resulting image representation has number of key properties. It consists of variable-length sequences [dil'n where the length L depends on the content of the input image. Images with more higher-frequency content result in longer sequences. This contrasts with the compressed representations of VQ-VAE and VQ- GAN, which are fixed size regardless of image content. In addition, the size of the representation and quality of image reconstructions controllable via the quantization quality setting. We can choose low-quality settings for more efficient training and inference, or high-quality settings for increased image quality.
DCTransformer To model the above data representation, Nash et al. Nash ct al. 2021 proposed the DCTransformer architecture. DCTransformer is an autoregressive model that predicts elements of sparse DCT sequences conditioned all prior elements:
p((di)) = II p(d dzi)
(1)
p(d dz)) p(1ll Pl, dei)p(pile d<))p((/de)
(2)
The architecture has an encoder-decoder structure, where the encoder processes arbitrarily long sequences of DCT data, and the decoder autoregressively predicts fixed-size chunks of DCT data, conditioned on the encoder's outputs.
During training a larget chunk [d of length C randomly selected from the DCT sequence, and all prior elements in the sequence [db] are treated as inputs. The sparse inputs are converted to a dense DCT image. DCT images are 3D arrays, where each spatial location corresponds to a pixel block of size BxB B, and channels correspond to different DCT components. For images of size H HxW x3 and DCT block size the corresponding DCT images will therefore be of size H/BxW/BBa Bx332 Input DCT images are partially observed as they consist only of data from the partial DCT inpul sequence [dr] rather than the full data sequence. The use of DCT images enables the encoder to process arbitrarily long input sequences, as regardless of the length they are converted to fixed size DCT image arrays. This contrasts with typical sequence-modelling approaches, where memory and computation scales with sequence length.
encoder processes partial DCT images, and a sequence of output embeddings is passed to cansally-masked Transformer docoder. The Transformer decoder embeds and processes target sequences 1CC and uses channel, position and value heads to output predictions for the corresponding list elements. The list elements are treated discrete values, and predicted using a softmax distribution, with an autore-
A
3

---

Under review as submission to TMLR
Table
Target, context, and annotation specification for different task types.
Task type
larget
Description
Query an image desired time, ltrgett- The C contains the sequence of fuarget -1 previous images and associated timestamps,
Video
harget
Maaget
modelling
Query an image at desired time, Larget- The C contains a sequence of immediately preceding frames starting at time, to < Ieagget: future frame at time, T > harget and associated timestamps, t.
Video
((T, harget U ((Ir,T))
ttarett
interpolation
Query an image from a desired viewpoint, Vtarget The C contains N other images and associated viewpoints, Vn-
View
synthesis
Vtarget
{IIn: vn))
Multi-format
Query desired image format, ftarget e.g., RGB, depth channel, semantic segmentation, etc. The C contains an RGB image, annotated fRGB.
image
Starget
{(TRCDD SRGB))
translation
gressive cross-entropy training objective. Writing 0c(-), 0p(-) and (.(.) for the channel, position and value softmax parameters respectively the objective for target chunk dil is:
t+C Σ log (q.P.. vi lo de) Pr + log lo2e)(de)) c
C=
where Vi, Pt and C₁ are one-hot vectors associated with indices vt, and respectively. In this work we use the same decoder and training objective, but augment the encoder to additionally process annotated context images.
3 Methods
Here, we present two main components of our work. (1) A framework for probabilistic vision that generalizes video prediction, few-shot view synthesis, and other computer vision tasks, (2) An architecture and modelling paradigm that implements that framework called Transframer, which builds on the DCTransformer (Nash
et al. 2021 generative image model and data representation.
3.1 Generalized conditional image modelling
Our probabilistic formulation is as follows: given an annotation a target and context information C, the task is predict distribution over target image target
(I larget
a larget C)
(4)
where C- (L,, n is a set of context images In and associated annotations a An annotation is metadata about the image such video timestamps, camera coordinates, image channels, etc. This framework phrase them in terms of a target and C. Some of the tasks we consider, such as novel view synthesis and video generation have previously been expressed in this manner (Eslami el al. 2018 Kumar el al. 2018). Other lasks, including semantic seg- mentation, depth estimation, or optical flow estimation are frequently modeled within the image prediction framework but typically using task-specific losses. Finally, tasks such as object detection and image classifi- cation are generally approached using specialized representations and losses, without any image generation components. Our framework enables to perform all these tasks using a single unified training objective.
be applied to wide range of frame- prade-preiitionn tasks, as long as
4

---

Under review as submission to TMLR
In some cases, we want a predictive distribution over multiple target images
target
conditioned on queries
For example, to predict multiple consecutive frames of video, or to synthesize views at multiple query viewpoints. The simplest approach is to treat the target images as conditionally independent:
{ [ak J
sarget] 1
p
( target
a
target
k C)
= III k
target
a
target C):
(5)
This enables parallel generation of the target images, but does not capture any additional dependencies between the larget images. In cases where the largel images are constrained by the the context, such as view synthesis with a large collection of input views, this not a significant problem. However, whenever there exist strong dependencies between target images, as in video modelling for example, sampled output images are likely to be inconsistent with one another. To address this issue, we can instead predict target images sequentially, updating the context set with newly generated images at each step. This corresponds to the following autoregressive model:
p
[ target
| f
a
target
C)
III target
a Cx)
target
(6)
Cs ([I,, an)
[[Iparget a
barget)) pcck" )
(7)
Predictions for each larget image will take into account all previously generated images, and can therefore capture the required dependencies.
A unified task formulation We apply framework to four different general task specifications, which are distinguished by their target annotations a target and context set C, as described in Table |
Training objective A range of modelling paradigms be used to estimate the predictive distributions in Equation 4 with DCTransformer with sparse DCT representations being a particular example. Other examples include conditional VAEs such as GQN (Eslami et al. 2018). conditional GANs as in pix2pix [Isola
et al. 2017), or conditional diffusion-based models (Ho el al. 2020). The kind of training objective depends on the model class used. In this work, we build on the DCTransformer, and therefore use the corresponding autoregressive training objective described in Section
3.2 Transframer architecture
To estimate predictive distributions over target images, we require an expressive generative model that can produce diverse, high-quality outputs. DCTransformer (Soction 2 Nash al. 2021 produces compelling results single image domains, but is not designed to condition
the multi-image context sets ((I,, a,)),
that we require. As such, extend DCTranfformer (Section 2 Nash et al. 2021 to enable image and
annotation-condiiioona predictions. In particular, we replace DCTransformer's 's V Tision-Transformeesssyyy en- coder, which operates on single (partially observed) DCT images, with multi-frame U-Net architecture, that processes a set of annotated context: frames along with the partially observed target frame. As in DC'Trans- former, we pass output embeddings from the encoder to Transformer- based decoder, and autoregressively predict sparse DCT tokens (Figure
Multi-frame U-Net The input to the U-Net is a sequence consisting of N context DCT frames (See Section 2 and a partially observed target DCT frame. Annotation information provided the form of vectors associated with each input frame. Initially, the DCT image channels are linearly projected to an embedding dimensionality E using 1 x 1 convolutions. We add horizontal and vertical positional embeddings to the embedded DCT images, and incorporate annotation information a by adding linearly projected annotation vectors.
The core component of our U-Net is a computational block that first applies a shared NF- NF-ResNet convo- lutional block (Brock et al. 2021) to each input frame, and then applics a Transformer-style self-attention block to aggregate information across frames (Figure 2)) Ne-ResNet blocks consist of grouped convolu- tions followed by squeeze-and-exxite layer (Hu et al. 2018 and designed for efficient performance
5

---

Under review as submission to TMLR
d
2
d3
d4
dp
Transformer Decoder
2
(4)
2/2
(4)
Z3
(4)
d1
d3
d4
MLP
MLP
MLP
Multi-Frame U-Net
Multi-frame self-attention
a1
82
a.T
NF-Block
NF-Block
NF-Block
(1-1)
(1-1)
(l-1)
Z
Z
Z
(a) Transframer architecture.
(b) Multi-frame U-Net block.
Figure 2: (a) Transframer takes input context DCT-images (left and middle), well as partially obscrved DCT-image of the target (right) and additional annotations a. The inputs are processed by a multi-frame U-Net encoder, which operates at number of spatial resolutions. U-Net outputs are passed to a DCTransformer decoder via cross-attention, which autoregressively generates a sequence of DCT tokens corresponding to the unseen portion of the target image (shown in green). (b) Multi-frame U-Net blocks consist of NF-Net convolutional blocks, multi-frame self-attention blocks which exchange information across input frames and Transformer-styl residual MLP. 2! is a representation from the th block and i-th frame.
on TPUs (pu). The self-attention block is based on standard Transformer block (Vaswani et al. 2017 with a self-attention layer applied before an MLP. We use one of two self-attention types: the first case, self-attention is applied to corresponding spatial locations across the frame feature maps. In the latter case, self-attention is applied both spatially and across frames. In some cases, we omit self-attention entirely to reduce memory consumption. We use multi-query attention (Shazeer 2019h, rather than the standard multi-head attention in order to improve sampling speed.
As in standard U-Net, our multi-frame variant has a downsampling stream, where feature maps are spatially downsampled for more efficient processing, and an upsampling stream where feature maps are spatially upsampled back to the input resolution. Skip-connections are used to propagate information from the downsampling stream to the upsampling stream directly, bypassing low-resolution information bottlenecks. In both streams, multiple convolutional blocks are applied a given resolution before an upsampling downsampling block changes the spatial resolution. We use symmetrical structure with the same number of block at each resolution in both streams.
3.3 Residual DCT representations
Consecutive video frames often exhibit substantial temporal redundancy. As an extreme case, in videos with static cameras and backgrounds, pixel data may be almost entirely unchanged across adjacent frames. In the DCTransformer data format (Section 2 a sparse image representation is obtained by storing the positions and values of non-zero DCT coefficients. When working with video data, we can potentially increase the degree of sparsity by storing DCT data only where it changes relative to the previous frame.
Figure 3 visualizes the sparsity of absolute and residual DCT representations for two datasets. The sparsity gains of the residual representation are significant the static-camera RoboNet dataset (Dasari al. 2019h. However, for the KITTI dataset. (Geiger et. al. 2012 which features continuously moving camera,

---

Under review as submission to TMLR
1.00
1.0
Absolute
1.75
Dits Dim.
1.05
0.7
Residual DCT
1.55
0.5
2.50
global step 1e4
global step 1e4
global steple4
(b) BAIR (61x61)
DCT mode
Bits Dim.
Absolute DCT
Absolute
PSNP
1.5
Residual
0.0
3.4
Residual DCT
1.16
0.4
3.3
global step le4
global step le4
global steple4
(a) Sparsity heatmaps
(c) KITTI (64x64)
Figure 3: (a) Comparing the sparsity of absolute and residual DCT representations for RoboNet (128x128; top) and KITTI videos (bottom). RoboNet has static videos with only a few moving elements, so residual frame representations yield a substantial increase in sparsity. KITTI videos typically feature a moving camera, resulting in differences almost everywhere consecutive frames. this case the sparsity benefits of residuals is minor. (b) Evaluation metries for absolute and residual representations BAIR and KITTI.
the sparsity gains are diminished. In our experiments, we use residual DCTs for video prediction, and absolute DCTs otherwise.
4 Experiments
To illustrate the flexibility and universality of our framework, and to test the performance of Transframer, range of datasets and tasks, covering predictive video modelling, few-shot novel view synthesis, and number of classic computer vision tasks. Our focus is on generality, and the case of expressing given task in the presented framework. Please see the appendix for model hyperparameters, training details, and additional ablation studies. selection of example output videos can be seen at htttps///iite: google. com/viex/transfrrnce
4.1 Video modelling
We train Transframer predict the next frame given series of input video frames. We supply one-hot time-steps as annotations described in Table [ (row 1), where time is specified relative to the target frame. For each video modelling task, we specify minimum and maximum number of context frames. During training, for each element of batch, we uniformly at random choose the number of context frames in the specified range, and pad the sequence to the maximum number of frames. In general, we choose the minimum number of frames to be equal to the number of context frames provided at test time.
Metrics For video generation we use the following metrics: Structural Similarity Index Measure (SSIM, (Wang et al. 2004)), Peak Signal-to-noise Ratio (PSNR, (Huynh-Thu & Ghanbari, 2008 Learned Perceptual Image Patch Similarity (LPIPS, (Zhang et al. 2018) and Fréchet Video Distance (FVD, (Un- terthiner al. 2019)) ). FVD measures distributional similarity between sets of ground truth and generated videos, and is sensitive to image quality and temporal coherence. For RoboNet. and KITTI we follow the

---

Under review as submission to TMLR
Table 2: Fréchet video distance for generated videos on BAIR and Kinetics600 datasets. For BAIR, we use single context frame and generate 15 frames. For Kinetics600, we use context frames and generate frames.
BAIR (64x64)
FVD4
DVD-GAN-FP Clark et al.
2019
109.8
VideoGPT Yan et al.
2021
103.3
THVD-GAN-FP Luc et al. Transframer (Ours)
2020
103.3
100.0
Video Transformer (Weissenborn et al.
2020
94.0
FitVid Babacizadeh et al. 2021
93.6
Kinctics600 (64x64)
FVD4
Video Transformer (Weissenborn el al.
2020
170.0
DVD-GAN-FP (Clark et al.] 2019)
69.1
Video VQ-VAE Walker et al. 2021
64.3
TrIVD-GANFPP
Luc et al. 2020
27.7
Transframer (Ours)
25.4
Table 3: Sample metrics for generated vidcos on KITTI and action-conditional RoboNet datasets. For KITTI, we use 5 context frames and generale 25 frames. For RoboNel, we use context frames and generate 10 frames.
KITTI (64x64)
RoboNet (64x64)
FVDJ
PSNR1
SSIMf
LPIPS}
FVD.
PSNR
SSIM
LPIPS
SVG (Villegas et al. 2019
1217.3
15.0
41.9
0.327
123.2
23.9
87.8
0.060
GHVAE Wu et al.)
2021a
552.9
15.8
51.2
0.286
95.2
24.7
89.1
0.036
FitVid BB Babacizadeh et al.
2021
884.5
17.1
49.1
0.217
62.5
28.2
89.3
0.024
Transframer (Ours)
260.4
17.9
54.0
0.112
53.0
31.3
94.1
0.013
literalure and report the best SSIM, PSNR, and LPIPS scores over 100 trials for each video. For consistency with previous work we only report test-set FVD for Kinetics600 and BAIR.
Datascts We first evaluate our model on BAIR Ebert et al. 2017 which one of the most well studied video modelling datasets, consisting of short video clips of a single robot arm interactions the unconditional setting. Table 2 shows the FVD score obtained by our model. While our model improves on some strong baselines such THVDDGAN, is out-performed by FitVid and Video Transformer. However, BAIR's small test set consisting of only 256 samples makes precise comparisons challenging, discussed by Lue al. [Luc al. 2020). Moreover, both methods under-perform Transframer in all other benchmarks, with larger test sets.
Kinetics600 (Carreira et al. 2018a) is an action recognition dataset, consisting of video clips of dynamic human actions across 600 activities, such as sailing, chopping, and dancing. Tt is a challenging dataset for video modelling, as the scenes fcature complex motion, as well as sudden scene transitions. We use the same experimental set-up DVD-GAN CClak al.) 2019), operating at 64x64 resolution, and generate 11 frames given 5 context frames. We report test-set FVD, for 50k frames, sampling the context frames uniformly random from the test-sct clips. Our model obtains an FVD of 25.4, improving over the previous state-of- the-art model THVD-GAN-FP Luc et al.
2020
The KITTI dataset (Geiger et al.)
2012) contains long video clips of roads and surrounding areas taken from the viewpoint of car driver. is less challenging than Kinetics600 due smoother camera motion, and a more constrained visual setting (roads, cars, etc.). However it is a relatively small dataset, which makes generalization challenging. As such, employ the augmentation methods of Babaeizadeh et al.
8

---

Under review as submission to TMLR
Context (t=0)
Context (t=1)
Prediction (t=6)
Actual (t=6)
Errors (t=6)
Prediction (t=11)
Target (t=11)
Errors (t=11)
video generation on the RoboNet test set 128x128 resolution. The action soquence for the output frames is provided to the model, and the frames are autoregressively generated, with context length of up to 4.
Figure 4:
[Babaeizadeh et al. 2021 to reduce overfitting. We follow the evaluation regime of Villegas et al. (Villegas et al. 2017), operating at 64x64 resolution, and generate 25 frames given 5 context frames, with test clips laken from 3 longer test clips al 5-frame strides. Table 3 shows that our model's performance improves on comparable methods on all metrics, with the improvement in LPIPS and FVD being the most substantial.
To evaluate our model in the cotion-conditional sctting, we use RoboNet (Dasari et al. 2019). which consists of short video clips of robot arms interacting with objects, and provides robot action annotations 5-dimensional vectors. The actions relate to the robot end-effector and gripper joint. We train at 64x64 and 128x128 resolutions, and evaluate using 2 context frames and 10 sampled frames on the test set specified by Vid (Babaeizadeh et al.] 2021 To process the input actions, we linearly project the 5-dimensional action-vectors and add them to input DCT image embeddings in the encoder. Table 3 shows the model's performance in comparison to existing methods, and Figure ! shows an example of our model's outputs at 128x128 resolution. At 64x64 resolution, our model improves on alternatives every metric by a large margin. At 128x128, could not find comparable previous work, we report our results in Appendix E for future comparisons.
Long range video generation As additional proof-of-concept, we push the temporal limits of our model and generate 30 second videos at 25 frames per second (fps) conditioned on single input image. At this frame rate, videos consist of 750 frames total. Generating these frames sequentially is challenging for a number of reasons. First, our architecture would need to process a large number of frames to model long- range dependencies; scaling such a model would be computationally expensive. Second, simply unrolling a conventional model with only a few frames of context often leads to globally incoherent generations or collapse of the output as gencration errors compound over time-stops.
We address these issues with two-stage procedure, where first generate low fps videos using one model, and then interpolate to obtain the desired frame rate with another. We train the low fps model fps videos with up to 15 context frames. At evaluation time we condition on single context image, and sequentially predict 29 frames to yield a 1 fps video that is 30 seconds long in real time. We train a separate model using the same architecture to interpolate between consecutive fps frames at 25 fps. During training, the interpolation model predicts target frame conditioned on the penultimate frame as well as a single future frame. At test time the model takes consocutive 1fps frames as context, then generates the 25 fps frames successively. The first two rows in Table I describe both stages, and Figure 5 shows some example outputs.
9

---

Under review as submission to TMLR
Context
1s
2s
48
8s
16s
30s
Given a single context frame we generate 30 seconds of video 25fps. Full videos (including interpolated frames) and attributions for the context images available at https://sites google com/ viex/transfrrmer
Figure
Context
SRN
PNeRF
Ours
Actual
Context
Ours
uctual
(a) ShapeNet
(b) Objectron
Figure 6: View synthesis on ShapeNet (a) and Objectron (b). For ShapeNet we also show the corresponding outputs from SRN (Vincent. Sitzmann, 2019) and PixelNeRF YYu al. 2021
Impact of residual DCT representations As described in Section for video datasets where consec- utive frames have high level of temporal redundancy, images be more compactly represented using the difference between DCT values in adjacent frames. We compare the data representations in Figure B with training curves for residual and absolute DCT sequences for BAIR and KITTI. Our experiments on both datasets suggest that residual representations provide an advanlage, which greater on BAIR, likely due lo the static backgrounds and associated sparsity.
4.2
Novel view synthesis
We train Transframer for novel view synthesis by conditioning on the target camera position, as well as collection of annotated input views. We supply camera views as context and target annotations as described in Table I (row 3), and sample uniformly a number of context views, up to specified maximum.
We follow Yu et al. YYu et al. 2021 in using PSNR and SSIM to evaluate our model's few-shot view synthesis performance. These metrics are informative, but problematic in the case where the target view is not well determined by the input views. these cases, the single prediction that optimizes PSNR will be blurred average possible target views. This penalizes models such ours that make probabilistic
Metrics
10

---

Under review as submission to TMLR
Table 4: View synthesis results on ShapeNet.
Chair (1 view)
Chair (2 view)
FID4
PSNRT
SSIM
LPIPS/
FID4
PSNRT
SSIM1
LPIPS/
PixeINeRF
132.86
23.72
0.91
0.13
101.22
26.20
0.94
0.08
Transframer (Ours)
51.09
22.85
0.91
0.07
46.17
23.77
0.92
0.06
Car (1 view)
Car (2 view)
FID
PSNR1
SSIM
LPIPS|/
FID4
PSNR
SSIM
LPIPS|
SRN
142.85
22.25
0.89
0.13
133.42
24.84
0.92
0.11
PixeINeRF
161.51
23.17
0.90
0.15
129.65
25.66
0.94
0.10
Transframer (Ours)
74.94
21.94
0.89
0.10
62.70
23.34
0.91
0.08
predictions, rather than a single point estimate. Hence, we introduce LPIPS and Fréchet Inception Distance (FID, (Heusel ct al. (017)) as additional metrics, both of which correlate with human percoptual preferences to a greater degree than the alternatives. We calculate per-scene FID between two sets: The ground truth set of the combined input and target views, and the sampled set consisting of all generated views combined with the input views.
Datasets We evaluate our model on ShapeNet benchmarks, in particular the chair and car subsets used by Yu et al. (Yu et al., 2021). The dataset consists of renders of 3D objects from the ShapeNet database. We train single model across both classes, and don't provide class labels input. As in Yu et al. (Yu et all 2021 we evaluate using either or 2 context views, and predict the remaining views in 251-frame test-set. To process the input views we flatten and linearly project the 9-dimensional camera matrix and add them to input DCT image embeddings in the encoder. Table ! compares our model to alternative approaches, and Figure B shows example results. Our model achieves worse PSNR and SSIM scores than baseline methods, but performs better on LPIPS and FID. As mentioned earlier, our model outputs samples, rather than a point estimate. As such it is unsurprising that it is outperformed on PSNR and SSTM by methods that output point estimates. Qualitatively, observe that our model produces sharper outputs that capture the input views' textures, compared to somewhat blurry outputs for unseen views for PixeINeRF and SRN.
As a more realistic view-synthesis task, we train and evaluate on the Objectron dataset. Objectron consists of short, object-centered clips, with full object and camera pose annotations. The dataset features diverse array everyday objects, such as bottles, chairs and bikes, recorded in a variely of settings. To the best of our knowledge, no prior works have trained view-synthesis models in the few-shot setting on this dataset. We compare the performance of our model for varying numbers of context views, and describe experimental setup in detail in Appendices F and G for future reference. Figure B shows example oulputs for two scenes. When given a single input view, the model produces coherent outputs, but misses some features such as the crossed chair legs. Given two views, these ambiguities are resolved to a large extent.
4.3 Multi-task Vision
Different computer vision tasks are commonly handled using specialized (often intricately so) architectures and loss functions. In this section, we report experiment where jointly trained single Transframer model using the same loss function 8 different tasks and datasets: optical flow prediction from single image, object classification, detection and segmentation, semantic segmentation (on two datascts), future frame prediction and depth estimation. We reduce all tasks to the same next-frame prediction task and do not adapt losses or architectural components. For the classification task, we use the ImageNet dataset (Deng et al. 2009) and consider prodicting one-hot images, where cach class is encoded as 8x8 white patch on black background. As image annotations we use one-hot vector that determines the desired type of outputs, e.g., depth or optical flow. In all cases used 256x256 resolution frames.
11

---

Under review as submission to TMLR
Context
Predictions
Actual
Context
Predictions
Actual
Chussifi oet lion (ImogeNec)
(Citscape))
Somant
Segns
Segi. (PASCAL)
Semartic
Flow predio tion (K600)
Detection (COCO)
ciling (YTSM)
Video mode
Doyth rstimation (NYU:Depth)
Scate parsing (COCD)
Figure 7: Example samples the validation set of our multi-task setup, obtained using single model. The model has no task specific paramcters; in all cases it makes predictions conditioned on context frame and a task embedding veclor.
We trained model using batches composed of randomly sampled elements from each dataset, rebalancing them slightly to accommodate their varied sizes. Several qualitative samples from the validation set are shown in Figure i It shows that the model learns to generate diverse samples across completely different tasks. On some tasks, such as Cityscapes, the model produces qualitatively good outputs, but model outputs on lasks like future frame prediction and bounding box detection are of variable quality, suggesting these are more challenging to model in this setting. Full experimental details can be found in Appendix H, along with quantitative performance metrics for the ImageNet classification and CityScapes semantic segmentation.
5 Conclusion
We introduced a gencral-purpose framework for conditional image prediction which unifics a range imago modelling and vision tasks. Our architecture, Transframer, models the conditional probability of image frames given target annotations and one or more context. frames. Our results on video modelling and novel view synthesis confirm the quantitative strength of our approach, while our experiments applying the model to 8 standard vision tasks, including depth estimation, object detection, and semantic segmentation, demonstrate its broad applicability. These results show that general-purpose probabilistic image models can support challenging image and video modelling tasks, as well as multi-task computer vision problems traditionally addressed with task-specific methods.
References
Google cloud tpus. https://cloud..ooggll http://oggoogooo/op/////////uuua
Sami Abu-FalHaija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijaynnaraiimhan. Youtube-Sm: A large-scale video classification benchmark. ar preprint arXiv:1699. 08675, 2016.
Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. Objectron: A large scale dataset of object-centric videos in the wild with pose annotations. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2021.
12

---

Under review as submission to TMLR
Mohammad Babacizadch, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. In TCLR, 2018.
Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. FitVid: Overfitting pixcl-level video prediction. CoRR, abs/21006.1311 2021.
Andy Brock, Soham De, Samuel L. Smith, and Karen Simonyan. High-performance large-scale image recog- nition without normalization. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 1059-1071. PMLR, 2021.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NevirIPS, 2020.
João Carreira, Eric Noland, Andras
Chloc Hillier, and Andrew Zisserman. A short note about Sinetics-600. CoRR, abs/1808.01340,, 2018a.
Joao Carreira, Eric Noland, Andras Banki-Horvathy Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint orXiv:1808.011 013400 2018b.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint (cXi::190700577 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understand- ing. In Proceedings of the IEFE conference on computer vision and pattern recognilion, pp. 3213 3-233, 2016.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. RoboNet: Large-scale multi-robot learning. CoRL, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hicrarchical image database. In CVPR, 2009.
Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind. Unconstrained scene generation with locally conditioned radiance fields. ICCV, 2021.
Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh M. Susskind, and Qi Shan. Equivariant neural rendering. In ICML, 2020.
Frederik Ebert, Chelsca Finn, Alex X. Leo, and Sergey Levine. Self-supervised visual planning with temporal skip connections. CoRL, 2017.
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi- scale convolutional architecture. In ICCV, 2015.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 160(6334):1204412222
2018.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pp. 12873 12883. Computer Vision Foundation IEEE, 2021.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303 338, 2010.
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vegan and time-sensitive transformer. orXiv preprint NrXiv:2204.0366 2022.
13

---

Under review as submission to TMLR
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the KITTI vision benchmark suite. In CVPR, 2012.
Pengsheng Guo, Miguel Angel Bautista, Alex Colburn, Liang Yang, Daniel Ulbricht, Joshua M. Susskind, and Qi Shan. Fast and explicit neural view synthesis. arXiv preprint orXiv:2107.05775, 2021.
Martin Housel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NearIPS, 2020.
Jie Hu, Li Shen, and Gang Sun. Squceza-and-exciration networks. In CVPR, pp. 7132-7141, Computer Vision Foundation / IEEE Computer Society, 2018.
Quan Huynh-Thu and Mohammed Ghanbari. Scope of validity of PSNR in image/viden quality assessment. Electronies letters, 44(13):800-804 2008.
Eddy 1lg, Özgün Çiçek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, and Thomas Brox. Uncertainty estimates for optical flow with multi-hypotheses networks. CoRR, abs/ abs/1802.07095, 2018.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. CVPR, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR (Poster), 2015.
Simon Kohl, Bernardino Romera-Paredes, Clemens Meyer, Jeffrey De Fanw, Joseph R. Ledsam, Klaus H. Maier-Hein, S. M. Ali Eslami, Danilo Jimenez Rezende, and Olaf Ronneberger. A probabilistic U-Net for segmentation of ambiguous images. In NeurIPS, pp. 6963-6975, 2018.
Simon A. A. Kohl, Bernardino Romera-Pareder Klaus H. Maier-Hein, Danilo Jimenez Rezende, S. M. Ali Eslami, Pushmeet Kohli, Andrew Zisserman, and Olaf Ronneberger. A hicrarchical probabilistic U-Net for modeling multi-scale ambiguities. ar preprint arXi::1955.1000 13077, 2019.
lasonas Kokkinos. UberNet: Training universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In CVPR, 2017.
Adam R. Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Jimencz Rezende. NcRF-VAE: A geometry aware 3d scene generative model. In ICML, 2021.
Ananya Kumar, S. M. Ali Eslami, Danilo J. Rezende, Marta Garnelo, Fabio Viola, Edward Lockhart, and
Murray Shanahan. Consistent generative query networks. arXie preprint wxXi:1807.02005 2018.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. In NIPS, pp. 6402 6413, 2017.
Alex X. Lee, Richard Zhang, Frederik Ebert, Picter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXte:1804.01522 2018.
Stefan Tee, Senthil Purushwalkam, Michael Cogswell, David J. Crandall, and Dhruv Batra. Why M heads are better than one: Training diverse ensemble of deep networks. CoRR, Abs//511.0033 2015.
Tsung-Y: Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on compuler vision, pp.
Springer, 2014.
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation-Dasee based adversarial video prediction on large-scale data. Xixiv preprint arXiv:2003.04085, 2020.
14

---

Under review as submission to TMLR
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV (1), volume 12346 of Lecture Notes in Computer Science, pp. 405 421. Springer, 2020.
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. ICML, 2021.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint 0rXic:1609.034463 2016.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. In VISIGRAPP (5: VISAPP), pp. 101-112. SCITEPRESS, 2021.
Konstantinos Rematas, Ricardo Brualla and Vittorio Ferrari. Sharf: Shape-conditioned radiance fields from single view. In JCML, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
Christian Rupprecht, Iro Laina, Robert S. DiPietro, and Maximilian Baust. Learning in an uncertain world: Representing ambiguity through multiple hypotheses. In ICCV, pp. 3611 3620. JEEE Computer Society, 2017.
Noam Shazeer. Fast transformer decoding: One wead is all you need. CoRR, abs/1911.021 2019.
Joshua Tobin, Wojeiech Zaremba, and Pieter Abbeel. Geometry-aware neural rendering. In NewrIPS, 2019.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In CVPR, pp. 1526 1535. Computer Vision Foundation / IEEE Computer Sociely, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. ICLR Workshops, 2019.
Aäron den Oord, Oriol Vinyals, and Koray Kavukeuoglu. Neural discrete representation learning. In NewrlPS, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In ICML, 2017.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In NewrIPS, pp. 81 91, 2019.
Gordon Wetzstein Vincent Sitzmann, Michael Zollhöfer. Scene representation networks: Continuous 3D- structure-aware neural scene representations. In NeurlPS, 2019.
Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with VQVAE. ar preprint arXiv:2103.0195: 2021.
Gregory K. Wallace. The JPEG still picture compression standard. Commun. ACM, 34(4):30 44, 1991.
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEFE Trans. Image Process., 13(4):600-61) 662, 2004.
15

---

Under review as submission to TMLR
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. In ICLR, 2020.
Bohan Wu, Suraj Nair, Roberto Martin-Martinn Fei-Fei, and Chelsea Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. In CVPR, 2021a.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. CoRR, abs/2111.12417, 2021b.
Wilson Yan, Yunzhi Zhang, Picter Abbeel, and Aravind Srinivas. VideoGPT: Video generation using VQ- VAE and transformers. arXiv preprint Xr 10157, 2021.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelNeRF: Neural radiance fields from one few images. In CVPR, 2021.
Christopher Zach, Thomas Pock, and Horst Bischof. A duality based approach for realtime tv-l optical flow. In Joint pallern recognition symposium, pp. 214-223. Springer, 2007.
Richard Zhang, Phillip Isola, Alexci A. Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effective- ness of deep features as a perceptual metric. In CVPR, 2018.
Jun- Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli Shecht- man. Toward multimodal image-to-image translation. In NIPS, pp. 476, 2017.
16


Reviews:
[{"review_id": "4aptednyrO", "reviewer": "Reviewer_Kiok", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pros:**\n* The experiments are exhaustive and support the claims very well. The results are very competitive across various tasks and extremely impressive, especially in video modeling and novel view synthesis.\n* The paper is very well structured; makes it easy to follow\n* The core idea is simple yet effective\n* The authors have provided training details that should help with reproducibility \n* Long-range video generation using two-step generation+interpolation is smart\n\n**Cons:**\n* Most of the components used by the method exist in the literature which reduces the novelty. (This is not a criterion that would affect my decision, I am merely stating it as an observation)\n* Section 3.2 can do with improved details. The authors just textually describe the model which makes it really hard to pass without multiple iterations, improved figures that show more details of the shared U-Net (NF block), and how the representations are attentions are calculated would be really helpful.\n* Although the paper is well structured, equations (5, 6, 7) could do with improved symbols. it is quite difficult to understand what $\\{I_k^{target} \\}_k$ means without multiple iterations.\n* Section 3.1 mentions framework allows for object detection and recognition but the authors have not provided any performance metrics (just visualizations in the appendix)\n* Is there a specific reason why the sparse DCT list needs to be sorted? The operations look equivariant/invariant. \n* The results in the appendix showing the results with respect to the DCT quality are nice. I am curious as to what happens when the DCT quality is different between training and testing. Does the model still generalize well?\n* Some results on training/inference wall-clock time would be nice\n* I could not find any code/ checkpoints for reproducibility", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a framework to perform various computer vision tasks under the umbrella of conditional frame prediction using a generative model that operates on compressed data representations (DCT). The authors combine this with a new architecture named the Transframer to obtain competitive performance on multiple computer vision tasks including coherent video generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think most of the points listed in the cons above can be addressed to secure my recommendation", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hIfw3epm7n", "reviewer": "Reviewer_L2ne", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Providing a unified framework for different problems is an important question and will be of interest to the community.\n\n- Combining a transformer with a U-Net to learn the conditional probabilities is a reasonable idea.\n\n- Extensive empirical results are provided to demonstrate the proposed methods on a variety of computer vision tasks.\n\nWeaknesses\n- Can the authors be more specific about the difference between the proposed framework and the existing frameworks? As far as I understand, using a model to learn p(target| context) seems a standard way to make predictions on many computer vision tasks. The authors mentioned that the same framework has been used in novel view synthesis and video generation but not semantic segmentation, depth estimation, or optical flow estimation. It will be better to give more explanation on the differences. Since even task-specific loss functions are used, they might be equivalent to some conditional probability and thus still fit into the probabilistic framework. I think the authors may also have to use different loss functions for different tasks (though all of them can be interpreted as conditional probabilities).   \n\n- How does the proposed framework estimate uncertainty? The authors claimed that the proposed probabilistic framework can provide a distribution over predictions and thus offer uncertainty estimation. It is unclear what the outputs are from the proposed model and how it outputs a distribution instead of a point estimation. In all experiments, the authors do not show the uncertainty estimation results.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Transframer, a unified probabilistic framework for conditional image prediction and generation. Specifically, Transframer utilizes a transformer and a U-Net to learn the conditional probability of the target given known context and target annotations. The proposed framework has been tested on many computer vision tasks including video modeling, novel view synthesis, semantic segmentation, image classification, and optical flow prediction.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Provide a clear explanation of the differences between the proposed method and existing methods, especially the loss functions (critical).\n\n- Provide an explanation of how the proposed method provides uncertainty estimation and provide empirical demonstrations (critical).\n", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eOJeJWncHE", "reviewer": "Reviewer_1DLG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- Interesting idea of modeling video generation with a sparse image representation DCT and the autoregressive probabilistic model.\n- Interesting extension of DCTrasnformer to video generation tasks.\n- Outstanding performance compared to previous baselines on the video modeling and novel view synthesis tasks.\n- Potential to be extended to solve various vision tasks with a unified framework and loss function.\n\n### Weaknesses\n\nPlease consider that I may miss some critical flaws in the experimental setups since I am not an expert on video modeling and novel view synthesis tasks.\n\n- Some of the writings are not clear. See the requested changes part for details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes the general framework Transframer for various vision tasks based on the DCTransformer.\n\nBesides the concept of sparse DCT image representations, the suggested idea in this paper is simple: the authors try to model a variety of vision works into conditional image modeling tasks.\n\nSpecifically, Transframer is trained to predict the distribution of the target image, given the annotations of the target image and the context information (set of images and corresponding annotations).\n\nIn experiments, Transformer performs well on the video modeling and novel view synthesis task. Moreover, it shows its capability to handle various vision tasks through multi-task learning without any need for task-specific loss or architecture.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Figure 2: What is z? Are they representations of each element from the previous block?\n- Figure 3: In (a) Sparsity heatmaps, it is hard to know which image indicates videos from RoboNet or KITTI. In caption, (b) → (b, c).", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has been reviewed by four reviewers. Three recommended "leaning accept" and one recommended "accept". 

The general consensus is that the experiments in the paper are thorough and the performances of the proposed method are impressive. The reviewers also find the proposed transframer architecture interesting. 

The reviewers asked the authors to clarify the novelty compared to existing work, and suggested improvements on presentation. The rebuttal addresses these points. 

Overall this is an interesting and useful paper. I recommend acceptance. 
Decision: Accept (poster)