Does Visual Pretraining Help End-to-End Reasoning?
Authors: Chen Sun
Calvin Luo
Xingyi Zhou
Anurag Arnab
Cordelia Schmid
Venue: NeurIPS 2023 poster (2023)
Keywords: visual reasoning
self-supervised learning
TL;DR: Self-supervised visual pretraining helps end-to-end visual reasoning

Abstract: We aim to investigate whether end-to-end learning of visual reasoning can be achieved with general-purpose neural networks, with the help of visual pretraining. A positive result would refute the common belief that explicit visual abstraction (e.g. object detection) is essential for compositional generalization on visual reasoning, and confirm the feasibility of a neural network ''generalist'' to solve visual recognition and reasoning tasks. We propose a simple and general self-supervised framework which ''compresses'' each video frame into a small set of tokens with a transformer network, and reconstructs the remaining frames based on the compressed temporal context. To minimize the reconstruction loss, the network must learn a compact representation for each image, as well as capture temporal dynamics and object permanence from temporal context. We perform evaluation on two visual reasoning benchmarks, CATER and ACRE. We observe that pretraining is essential to achieve compositional generalization for end-to-end visual reasoning. Our proposed framework outperforms traditional supervised pretraining, including image classification and explicit object detection, by large margins.

Does Visual Pretraining Help End-to-End Reasoning?
Chen Sun Brown University, Google
Calvin Luo Brown University
Xingyi Zhou Google Research
Anurag Arnab Google Research
Cordelia Schmid
Google Research
Abstract
We aim to investigate whether end-to-end learning of visual reasoning can be achieved with general-purpose neural networks, with the help of visual pretraining. A positive result would refute the common belief that explicit visual abstraction (e.g. object detection) is essential for compositional generalization soning, and confirm the feasibility of a neural network "generalist" to solve visual recognition and reasoning tasks. We propose a simple and general self-supervised framework which "compresses" each video frame into small set of tokens with transformer network, and reconstructs the remaining frames based on the com- pressed temporal context. To minimize the reconstruction loss, the network must learn a compact representation for each image, well as capture temporal dynam- ics and object permanence from temporal context. We perform evaluation on two visual reasoning benchmarks, CATER and ACRE. We observe that pretraining is essential to achieve compositional generalization for end-to-end visual reasoning. Our proposed framework outperforms traditional supervised pretraining. including image classification and explicit object detection, by large margins.
visual rea-
Introduction
This paper investigates an end-to-end trained neural network is able to solve challenging visual reasoning tasks (23 65 66] that involve inferring causal relationships, discovering object relations, and capturing temporal dynamics. A prominent approach [49 for visual reasoning is to construct a structured, interpretable representation from the visual inputs, and then apply symbolie programs 442 or neural networks 177 to solve the reasoning task. Despite their appealing properties, such as being interpretable and enabling the injection of expert knowledge, is practically challenging to determine what kinds of symbolic representations to use and how to detect them reliably from visual data. In fact, the most suitable symbolic representation may differ significantly across different tasks: the representation for modeling single human's 's kinematics (e.g. with body parts and joints) is unlikely be the same that for modeling group social behaviors (e.g. each pedestrian can be viewed as its own complete, independent entity). With the success of unified neural frameworks for multi-task learning 77. it is desirable utilize a unified input interface (e.g. raw pixels) and to have the neural network learn to dynamically extract suitable representations for different tasks. However, how to learn distributed representation with deep neural network that generalizes similarly learning methods based on symbolic representation [66] for visual reasoning remains an open problem.
The key hypothesis we make in this paper is that general-purpose neural network, such as Transformer [55], can be turned into an implicit visual concept learner with self-supervised pre- training. An implicit visual concept refers network, which can be "finetuned" directly on the downstream tasks. Some of the learned implicit representations may be discretized into human- inttrpr-attreee symbols for the purposes of human understanding of and feedback to the model. Others may correspond to part of, or combination of
vector-based representation in an end-to-end neural
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Siage 2
Stage 2
Single stage
Answer
Answen (4.9)
Pnswer (4.y)
Symbolic Programs
Neural Networks
Neural Networks
argeet
Oiject
rxprssmtaiion
Token
represerator
Neural Networks
Neural Networks
Neural Networks
Stage
Stage
(a) Neuro- Neuro-Symbolic Concept Learning
(b) Attention over Object Embeddings
(e) Implist Vieual Concept Learinng (ours]
Figure I: Comparison between a neuro-symbolic approach, hybrid approach with learned object embeddings [17, and our proposed approach for visual reasoning. The illustration of each model family flows upwards, where visual inputs are encoded by neural networks (stage 1), and then processed by symbolic programs or another neural network to generate reasoning predictions (stage 2). Compared to (a) and (b), our approach does not require a separate "preprocessing" stage to extract the symbolic representation from visual inputs, and the self-supervised pretrained neural network can be end-to-end "finetuned" to the downstream visual reasoning tasks.
human-interpretable symbols. As opposed to explicit symbolic representation (e.g. object detection), implicit visual concepts do not require pre-defining a concept vocabulary or constructing concept classifiers, and also do not suffer from the carly commitment loss of information issues which may happen when visual inputs are converted into explicit symbols or frozen descriptors (e.g. via object detection and classification). A comparison between approach and those that utilize explicit symbols under a pipeline-stye styyled framework is visualized in Figure]
Our proposed representation learning framework, implicit visual concept learner (IV-CL) consists of two main components: first, a single image is compressed into a small set tokens with neural network. This is achieved by vision transformer (ViT) network [19 with multiple "slot" tokens (e.g. the [CLS] token in ViT) that attend to the image inputs. Second, the slot tokens are provided as context information via temporal transformer network for other images in the same video, where the goal is to perform video reconstruction via the masked autoencoding [27] objective with the temporal context. Despite its simplicity, the reconstruction objective motivates the emergence of two desired properties in the pretrained network: first, to provide context useful for video reconstruction, the image encoder must learn compact representation of the scene with its slot tokens. Second, to utilize the context cues, the temporal transformer must learn to associate objects and their implicit representation across time, and also capture the notion of object permanence the existence of an object even when is occluded from the visual observations.
We conduct extensive ablation experiments on the Compositional Actions and TEmporal Reasoning (CATER) 23 benchmark and the Abstract Causal REasoning (ACRE) [66] benchmark. To better understand if and how end-to-end pretraining helps visual reasoning, also consider the supervised pretraining paradigm, where the slot tokens in the Transformer network are pretrained to "decode" image-level labels object locations and categories. Specifically, we adopt the Pix2Seq objective [13] which formulates object detection as an autoregressive "language" modeling task.
Our experimental results reveal the following observations: first, IV-CL learns powerful implicit representations that achieve competitive performance on CATER and ACRE, confirming that visual pretraining does help end-to-end reasoning. Second, the pretraining objective matters: networks pretrained on large-scale image classification benchmarks [L5 52 transfer poorly to the visual reasoning benchmarks, while object detection learns better representation for reasoning. However,

---

both are outperformed by IV-CL by large margins. Finally, we observe that the network inductive biases, such as the number of slot tokens per image, play an important role: on both datasets, we observe that learning small number of slot tokens per image (1 for CATER and 4 for ACRE) lead to the best visual reasoning performance. To the best of our knowledge, our proposed framework the first to achieve competitive performance on CATER and ACRE without the need to construct explicit symbolic representation from visual inputs.
In summary, our paper makes the following two main contributions: First, unlike common assump- tions made by neuro- syuro-yymoolic approaches, we demonstrate that compositional generalization for visual reasoning can be achieved with end-to-end neural networks and self-supervised visual pretrain- ing. Second, we propose IV-CL. a supersueeee representation learning framework, and validate its effectiveness on the challenging CATER and ACRE visual reasoning benchmarks against supervised visual pretraining counterparts. Code and checkpoints will be released.
2 Related Work
Neural Network Pretraining. Huge progress has been made towards building unified learning
frameworks for a wide range of tasks, including natural language understanding 16
48 g 40).
visual recognition [36 35 63 22 and multimodal perception [33
Unfortunately, most of the "foundation models" 7 for visual data focus on perception tasks, such as object classification, detection, or image captioning. Despite improved empirical performance on the visual question answering task [32 64), visual reasoning remains challenging when measured on more controlled benchmarks that require compositional generalization and causal learning 66 23] 14. is commonly believed that symbolic or neurosymbolic methods 42 62, 37 4, as opposed to general-
50
38
24
3
purpose neural networks, are required to achieve generalizable visual reasoning [6] 66 To our knowledge, our proposed framework is the first to demonstrate the effectiveness of general-purpont end-to-end neural network on these visual reasoning benchmarks.
Self-supervieed Learning from Images and Videos. Self-supervised learning methods aim to learn strong visual representations from unlabelled datasets using pre-text tasks. Pre-text tasks were initially hand-designed to incorporate visual priors [18 69 [0]. Subsequent works used contrastive formulations which encourage different augmented views of the same input to map to the same feature representation, whilst preventing the model from collapsing to trivial solutions AS 12
28 26 2).
One challenge of the contrastive formulation the construction of positive and negative views, which has been shown to critically impact the learned representation [12 59 51). Whereas contrastively learned representations may not easily transfer across domains [6) our pretraining successfully generalizes to visually different datasets, such as from ACRE to RAVEN.
Our work is most related masked self-supervised approaches. Early works in this area used stacked autoencoders (56) or inpainting tasks (7) with convolutional networks. These approaches have seen a resurgence recently, inspired by BERT [6] and vision transformers
[19]. BEiT ( encodes masked patches with discrete variational autoencoders and predicts these tokens. Masked Autoencoders (MAE) (27). on the other hand, simply regress the pixel values of these tokens. MAE has been extended to regress features (57] and to learn video representations [53] 20). Our training objective is different, as is predictive coding based on compressed video observations. We confirm empirically that the proposed method outperforms MAE and its video extension by large margins.
Object-centric Representation for Reasoning. Most of the existing neuro-symbolic 42 62 and neural network [17 based visual reasoning frameworks require a "preprocessing" stage of symbolic representation construction, which often involves detecting and classifying objects and their attributes from image or video inputs. Our proposed framework aims to investigate the effectiveness of single-stage, end-to-end neural networks for visual reasoning, which is often more desirable than the two-stage frameworks for scenarios that require transfer learning or multi-task learning. In order to obtain the object-centric, or symbolic representation in the preprocessing stage, one can rely on a supervised object detector [42] 54], such as Mask R-CNN [2]. An alternative approach is to employ self supervuper objectives and learn low-level features that are correlated with objects, such as textures [21] 30 44. or objects themselves
9 II. In practice, supervised or self self-supervised approaches for object detection and object-centric representation learning may suffer from the lack of supervised annotations, or from noisy object detection results. For example, it was previously observed that object-centric representations are beneficial for transfer learning to temporal event classification only when ground truth object detections are used for training and evaluation [68].
3

---

Siot tokens
Un-masked pixel tokens
Masked pixel tokkens
Image Decoder
Image Decoder
Image Decoder
Temporal Transformer
ViT-B Encoder
VIT-B Encoder
VIT-B Encoder
ViT-B Encoder
ViT-B Encoder
context frames
spatially masked frames
Figure 2: IV-CL self-supervised pretraining. We consider the video reconstruction objective via masked autoencoding: A ViT-B image encoder is tasked (I) extract visual representations (orange) for the unmasked patches per image and (2) compress image into a small set of slot tokens (blue). temporal transformer then propagates the information from the slot representations and patch-leve representations from neighboring frames, which are essential for successful reconstruction.
3 Method
now introduce the proposed implicit visual concept learning (IV-CL) framework. follow the pretraining and transfer learning paradigm: during pretraining (Figure 2 we task shared image encoder to output patch-level visual embeddings along with a small set of slot tokens that compress the image's 's information. The pretraining objective is inspired by masked autoencoding (MAE) for unlabeled video frames, where the aim is to reconstruct a subset of "masked" image patches given the "unmasked"" image patches context. Compared to the standard MAE for images 27 the image decoder has access to two additional types of context information: (1) The encoded patch embedding from the unmasked image patches of the neighboring frames; (2) The encoded slot tokens from a subset of context frames. The context information encoded and propagated by a temporal transformer network. To successfully reconstruct masked frame, the image encoder must learn compact representation of the full image via the slot tokens, and the temporal transformer has to learn capture object permanence and temporal dynamics.
After pretraining, the image decoder is discarded, and only the image encoder and temporal trans- former are kept for downstream visual reasoning tasks. The inputs to the temporal transformer the slot tokens encoded from individual, unmasked video frames. We use the full finetuning strategy where the weights of both the newly added task decoder (e.g. a linear classifier), and the pretrained image and temporal transformers are updated during transfer learning.
We adopt the Vision Transformer (ViT) backbone to encode each image inde- pendently: an input image broken into non-overlapping patches of 16x pixels, which are then linearly projected into patch embeddings as inputs to the transformer encoder. Spatial information is preserved by sinusoidal positional encodings. We use the standard ViT-Base configuration which has 12 Transformer encoder layers. Each layer has hidden size of 768. MLP projection size of 3072, and 12 attention heads. During pretraining, a subset video frames are spatially masked randomly given masking ratio, only the unmasked image patches are fed into the VIT-B encoder. For context frames and during transfer learning, all image patches are provided as inputs to the image encoder.
Image Encoder:
In the seminal work by Locatello et al. 41, slot tokens are defined soft cluster centroids that group image pixels, where the goal is unsupervised object detection. Each slot token repeatedly attends to the raw image inputs and is iteratively refined with GRU network. We borrow their terminology, and use slots to denote the representational bottleneck in which we hope to encode implicit visual concepts, such object-centric information. We generalize their slot update rules
Slot Tokens:

---

by: (1) iteratively updating the visual representation with layers of the Transformer encoder (ViT); (2) replacing cross-attention with multi-headed self-attention; (3) using MLP layers with untied weights to update the intermediate slot representation as opposed to a shared GRU network. These two modifications allow us implement "slot attention" directly with Transformer encoder, simply by prepending slot tokens as additional inputs to the encoder (similar to [CLS] tokens). The initial slot embeddings at the input of the visual encoder are implemented as a learnable embedding lookup table. To compare the effectiveness of different methods to aggregate "slot" information, we also explore single-headed soft attention and Gumbel-max attention as used by 66].
Temporal Transformer:
To propagate temporal information across frames, we use another trans- former encoder (with fewer layers than the ViT-B image encoder) which takes the tokens encoded by the image encoder as its inputs. During pretraining, the slot tokens from context frames, along with the unmasked patch tokens from the query frames are concatenated together and fed into the temporal transformer. For each query image, the temporal transformer outputs its corresponding unmasked patch tokens contextualized from both the unmasked patches from neighboring query frames and the slot tokens from context frames. The contextualized patches are then fed into the image decoder to compute the reconstruction loss. To preserve temporal position information, we use learned positional embeddings (implemented with an embedding lookup table). When finetuned on reasoning task, the temporal transformer takes the slot tokens encoded by the image encoder as its inputs. Putting the image encoder and the temporal transformer together. the overall video encoder used for finetuning can be viewed as a factorized space-time encoder proposed by [5]. It is more parameter-effiee effficiiee than the vanilla video transformer used by [53].
We use the same image decoder as in [27]. The query images are decoded independently given the contextualized unmasked patch tokens. The image decoder is implemented with another transformer, where masked patch tokens are appended to the contextualized unmasked patch tokens as inputs to the image decoder. Sinusoidal positional encodings are used to indicate the spatial locations of individual patch tokens. We use the same number of layers, hidden size, and other hyperparameters as recommended by [27]. During pre-training, we use mean squared error to measure the distance between the original query image patches and the reconstructed patches.
Image Decoder for Pre-training:
Transfer Learning: As the goal of pre- veetrraining is to learn the slot tokens which we hope to compress an input image into a compact set of implicit visual concept tokens, we only ask the image encoder to generate the slot tokens during finetuning, which are fed to the temporal transformer as its inputs. We then average pool the output tokens of the temporal transformer and add a task specispei decoder to make predictions. Both benchmarks used in our experiments can be formulated as multi-class classification: for CATER, the goal is to predict the final location of the golden snitch, where the location is quantized into one of the 6x6 positions; in ACRE, the goal is to predict whether the platform is activated, unactivated, or undetermined given a query scenario. We use linear classifiers as the spsc-sp decoders with standard softmax cross-entropy for transfer learning.
Supervised Pretraining Baselines: To better understand if visual pretraining helps end-to-end reasoning, we consider two types of supervised pretraining baselines. The first is the "classical" image classification pretraining which often exhibits scaling laws (52) when transferred to other visual recognition benchmarks. The second is the object detection task, which intuitively may also encourage the emergence of object-centric representations (per task requirement) inside the neural network. Both pretraining objectives can be directly applied on the same Transformer architecture as utilized in IV-CL, with different designs on the task specific decoders (which are discarded for visual reasoning finetuning). For image classification, we directly treat the slot token as a [CLS] token and add a linear classifier on top of it. For object detection, to make minimal modification to our framework, we follow the design proposed by Pix2Seq (13), which parameterizes the bounding box annotations as discrete tokens, and formulates the training objective as an autoregressive sequence completion task. The inputs to the autoregressive decoder are the encoded slot tokens. We adopt the same sequence construction and augmentation strategies as in Pix2Seq.
Experiments
4.1 Experimental Setup
Benchmarks: In the classic "shell game", a ball is placed under cup and shuffled with other empty cups on a flat surface; then, the objective is to determine which cup contains the ball. Inspired by
5

---

+ objects. A this, CATER is a dataset composed of videos of moving and interacting CLEVR
special golden ball, called the "snitch", present in each video, and the associated reasoning task is to determine the snitch's 8 position at the final frame. Solving this task is complicated by the fact that larger objects can visually occlude smaller ones, and certain objects can be picked up and placed down to explicitly cover other objects; when an object is covered, it changes position in consistence with the larger object that covers it. In order to solve the task, a model must learn to reason not only about objects and movement, but also about object permanence, long-term occlusions, and recursive covering relationships. Each video has 300 frames, and we use the static camera split for evaluation.
The ACRE dataset tests a model's ability to understand and discover causal relationships. The construction of the dataset is motivated by the Blicket experiment in developmental psychology [25). where there is a platform as well as many distinct objects, some of which contain the "Blicketness" property. When at least one object with the "Blicketness" property is placed on the platform, music will be played; otherwise, the platform will remain silent. In ACRE, the platform is represented by a large pink block that either glows or remains dim depending on the combination of CLEVR objects placed on it. Given six evidence frames of objects placed on the platform, the objective of the reasoning task is to determine the effect a query frame, containing a potentially novel object combination, would have on the platform. Possible answers including activating it, keeping in inactive, or indeterminable.
We use the unlabeled videos from the training and validation splits of the CATER dataset for pretraining. Both the static and moving camera splits are used, which contains 9,304 videos in total. In our experiments, we observe that ACRE requires higher resolution inputs during pretraining and finetuning. Our default preprocessing setup is to randomly sample 32 frames of size 64x 64 for pretraining the checkpoints that are transferred to CATER, and 16 frames of size x24x224 for pretraining the checkpoints that are transferred to ACRE. The randomly sampled frames are sorted to preserve the arrow of time information. No additional data augmentations are performed.
Pretraining data:
Transfer learning: For CATER, we evaluate on the static split which has 3,065 training, 768 valida- tion, and 1645 test examples. We select the hyperparameters based on the validation performance, then use both training and validation data to train the model to be evaluated on the test split. By default, we use 100 randomly sampled frames of size 64x 64 during training, and 100 uniformly sampled frames of stride during evaluation. For ACRE, we explore all three splits, all of which contain 24,000 training, 8,000 validation, and 8.000 test examples. We use the validation set to select hyperparameters and use both training and validation to obtain the models evaluated on the test split.
Default hyperparameters: We use the Adam optimizer for pretraining with learning rate of 10-3, and the AdamW optimizer for transfer learning with a learning rate of x 10-5, The pretraining checkpoints are trained from scratch for 1,000 epochs using a batch size of 256. For transfer learning, we finetune the pretrained checkpoints for 500 epochs using batch size of 512. All experiments are performed on TPU with 32 cores. Below we study the impact of several key model hyperparameters.
4.2 IV-CL vs. Supervised Pretraining
We first compare our proposed IV-CL to traditional supervised pretraining on both detection and classification tasks. For classification, we consider the same ViT-B visual encoder trained on ImageNet-21 K [15] and JFT [52]. For object detection, we consider an in-domain object detection benchmark dataset called LA-CATER 49]. LA-CATER matches the visual appearance of CATER; it was created to study the benefit of modeling object permanence and provides frame-level bounding box annotations for all visible and occluded objects. We validated the correctness of our object detector on the COCO benchmark, which achieves comparable performance to the original Pix2Seq implementation. On the LA-CATER validation set, we observe 82.4% average precision (AP) at an IOU threshold of 50%. Whereas one might expect almost perfect performance on such a synthetic environment, this can be explained by the inherent properties of the dataset; frame-level object detection on LA-CATER also evaluates the detection of occluded and invisible objects, which is indeterminable when given only single, static images as inputs. We also consider classification pretraining baseline to count the number of unique objects in LA-CATER frames.
We note three remarkable trends when inspecting the results in Table | of the models pretrained with supervision outperforms their self-supervised counterpart. Instead, their performance both CATER and ACRE fall behind IV-CL by large margins. Second, when
First, we observe that none

---

Table Self-supervised visual pretraining vs. supervised pretraining. We compare our proposed pretraining with traditional supervised classification or detection pretraining.
Objective
Pretrain data
CATER
ACRE (comp)
Random Init. Detection Classification
3.34%
38.78%
LA- LA-CATER
56.64%
67.27% %
LA-CATER
41.48%
64.78%
Classification
ImageNet-21k
55.58%
60.73%
Classification
JFT
54.07%
48.32%
IV-CL
CATER 70.14 (+0059)%
93.27 (=0022))%
comparing the detection and classification objectives, we observe that the detection pretraining outperforms classification pretraining significantly. This can be potentially explained by the domain gap between natural image datasets and CLEVR-style datasets, encourages the learning of object-centric representations in the slot tokens. To better understand this, perform addition ablations by replacing the object detection dataset with COCO 39], which is natural image dataset. We observe similar transfer learning performance as LA-CATER pretraining. Additionally, perform probing experiment where ask the object detection decoder make predictions with a single randomly sampled slot token. We empirically observe that each token appears focus on one or small subsets of the objects in the scene, and different tokens complementary to each other. Both observations indicate that the stronger performance of object detection pretraining is likely due to the "object-centric" objective itself. Finally, observe counterexample of the "scaling law": larger scale classification pretraining (JFT) leads to significantly worse performance than smaller scale pretraining (ImageNet-21k).
that object detection objective
4.3
Visualizations of the Learned Slots
To help understand what visual concepts are implicitly captured by IV-CL, we visualize the attention heatmaps from each learned slot token back the image pixels. This is implemented with the attention rollout technique M Figure shows examples of the attention heatmaps after (a) self-supervised pretraining on CATER, and after (b) finetuning for visual reasoning on ACRE.
(a) Pretrained CATER
(b) Finetuned ACRE
Figure Visualization slots of an IV-L model after pretraining on CATER (left) and finetuning on ACRE (right). Each heatmap is generated by attention rollout []] to the input pixels. brighter color indicates higher attention weight.
We observe two general patterns by inspecting the pretrained slot heatmaps: first, a subset of the heatmaps exhibit object-centric behavior, as in Figure 3Ja). Each slot tends to focus on an individual object, and most of the objects in an image are covered by combining the heatmaps from all four slots. However, we also observe that sometimes the slots are not completely disentangled with respect to individual objects, which indicates that the implicit representations obtained after IV-CL pretraining do not learn perfectly disentangled visual concepts, and further finetuning is necessary to achieve compositional generalization on visual reasoning. We then inspect the heatmaps after finetuning for visual reasoning ACRE in Figure b). We observe some slots model relationships among objects

---

Table 2: CATER Prefraining with different mask ratios, context sizes, and frame lengths.
(a) Mask ratio
(b) Context size
(c) Frame length
Ratio
Acc.
Size
Acc.
Length
Acc.
37.5%
70.14%
8
70.14%
32
70.14% %
12.5%
66.35%
65.35%
8
62.28%
50%
66.57%
4
67.4 67.47%
16
66.63%
87.5%
61.94%
16
64.34%
64
68.25%
Table 3: CATER Pretraining with different number of slots, and pooling strategies
(a) Number of slots
(b) Where to pool
(c) How pool
# slots
Acc.
Layer
Acc.
Method
Acc.
70.14%
11
70.14%
Slice 70.14% %
2
66.52%
55.80 33.80%
Soft 64.23%
8
64.45%
9
68. 86%
Hard
65.90%
and the platform, and some focus on individual objects. Intuitively, both types of information are needed to solve the ACRE benchmark. Finally, we visualized the attention heatmap of ImageNet- 2 21 pretrained model after finetuning on ACRE. We observe that the heatmaps often "collapse" on a small subset of the same objects, which coincides with its lower reasoning performance.
4.4 Ablation Study
Next, we ablate key design choices. We present our ablation study on CATER in Table
Masking ratio: Contrary to the large masking ratio (75%) employed in vanilla MAE (7], we found that the optimal masking ratio was 37.5% on downstream CATER accuracy. This is perhaps due to the fact that CATER is designed to test "compositional generalization", and so the spatial context provides less information than in natural images and video.
We also study the impact of the number of frames IV-CL is pretrained on, and find the best performance on 32 frames. Fixing the total number of pretraining frames, we then ablate over the number of context frames, which are the frames from which slot representations are generated. When no context frame is used, we essentially utilize only pateh-level representations to perform reconstruction with the temporal transformer (simulating per- frame MAE followed by temporal transformer). We find that the best performance is achieved with context frames, which balances the number of slot representations with patch-level representations.
Number of Total Frames and Context Frames:
Number of Slot Tokens: Another useful ablation is on the impact of the number of slots used for CATER and ACRE. CATER, we find that only 1 slot token per frame is enough to solve the reasoning task. We believe that this may be due to how the reasoning objective of CATER is designed: to successfully perform snitch localization, the model need only maintain an accurate prediction of where the snitch actually or potentially is, and can ignore more detailed representation of other objects the scene. Under the hypothesis that the slot tokens represent symbols, perhaps the singular slot token is enough to contain the snitch location. On the other hand, when ablating over the number of tokens for the ACRE task (see Appendix), we find that a higher number of tokens is beneficial for reasoning performance. This can potentially be explained by the need to model multiple objects across evidence frames order to solve the final query; under our belief that slot tokens are encoding symbols, multiple may be needed in order to achieve the best final performance.
We ablate over which layer to pool over to generate the slot tokens. The patch tokens are discarded after the pooling layer, and only the slot tokens are further processed by the additional Transformer encoder layers. As expected, layers to process both slot and patch tokens. Additionally, we also study the impact of slot pooling method, and observe that adding additional headed soft attention and Gumbel-ma attention outperformed by simply using the slot tokens directly.
Slot Pooling Layer and Method:
desirable to use all image encoder

---

Table 4: Results CATER (static). IV-CL performs the best among non-object-centric methods, and performs competitively with methods with object-supervision.
Method
Object-centric
Object superv.
Top-1 Acc. (%)
Top-5 Acc. (%)
OPNet
74.8
Hopper 70
73.2
93.8
ALOE 17
/
70.6
93.0
Random Init.
X
X
3.3
18.0
MAE (Image) 27
X
27.1
47.8
MAE (Video)
X
63.7
82.8
IV-CL (ours)
X
70.1 0.6
88.3 0.2
Table Results on ACRE compositionality, systematicity, and LLD.. splits. IV-CL performs the best among all methods on the compositionality split, and performs competitively on other splits.
Method
Object-centric
Object superv.
comp (%)
sys (%)
iid (%)
NS-OPT 66
69.04
67.44
66.29
AL.OF 17
X
91.76
93.90
Random Init.
X
38.78
38.57
38.67
MAE (Image) 27
X
80.27
76.32
80.81
MAE (Video)
X
78.85
71.69
77.14
IV-CL (ours)
X
93.27
0.22
92.64
0.30
92.98
0.80
4.5 Comparison with State-of-the-Art
We compare our IV-CL framework with previously published results. As most of the prior work require explicit object detection and are not end-to-end trained, we reimplement an image-based based MAE 27 and a video-based MAE [53] baseline and analyze the impact of inductive biases (using slot tokens not) as well as pretraining objectives (predictive coding given compressed context, or autoencoding the original inputs) on the reasoning performance. Our reimplementation of image and video MAEs achieve very similar performances on their original benchmarks. However, for video-based MAE, we observe that the "un-ffctorized" backbone leads to training collapse on CATER. We hence adjust the backbone to be "factorized" as illustrated in Figure 2 pretraining and hyperparameter selection procedures as for IV-CL.
We follow the same
Table compares the result of IV-CL against other satte-of-the-ate models on CATER. We also compare IV-CL on ACRE against other existing models in Table We cite the comparable results reported by the original authors when available. IV-CL achieves the best performance among the approaches that do not depend on explicit object-centric representation, and overall state-of-the-an performance on ACRE.
4.6 Transfer Learning RAVEN
We explore generalization to visually different reasoning benchmark, RAVEN (65). Inspired by Raven's 's Progressive Matrices (RPM). its goal is to evaluate machine learning model's 's structural, relational, and analogical reasoning capabilities. The reasoning task determine which of eight candidate geometrical figures naturally follow the patterned sequence of eight context figures. We explore all seven reasoning scenarios and perform finetuning on all training and validation examples (56,000 examples). The pretraining and finetuning hyperparameters exactly match those for ACRE, the model now takes in images as input (8 for context, for answers). report generalization performance on RAVEN in Table 6 We observe that the pretrained representation is generalizable, as IV-CL achieves competitive performance on RAVEN (55) with the same pretrained model and finetuning hyperparameters as ACRE, despite the different visual appearances across the datasets.

---

Table 6: Transfer learning results RAVEN. We follow the same pretrained representation and finetuning hyperparameters as for ACRE.
Method
Average
Center
2x2 Grid
3x3 Grid
L-R
U-D
O-IC
O-IG
LSTM
13.1
13.2
14.1
13.7
12.8
12.4
12.2
13.0
ResNet DRT 665
59.6
58.1
46.5
50.4
65.8
67.1
69.1
60.1
CoPINet 67
91.4
95.1
77.5
78.9
99.1
99.7
98.5
91.4
SCL. 58
91.6
98.1
91.0
82.5
96.8
96.5
96.0
80.1
IV-CL (ours)
92.5
98.4
82.6
78.4
96.6
97.2
99.0
95.4
4.7 Generalization to Real Videos
Finally, we attempt to answer the question: Would our proposed IV-CL framework work on real videos? We consider the Something- Ellse benchmark [43], which consists of short videos capturing the interactions between human hands and different objects. This benchmark focuses relational reasoning, especially on compositional generalization across different object categories. We consider the base split and the "compositional" split. The base split contains 112,397 training videos and 12,467 validation videos, across 88 categories. The compositional split contains 54,919 training fine-
videos and 57,876 validation videos, across 174 categories. Each category corresponds grained activity that requires spatiotemporal relation reasoning. The compositional split is designed include disjoint object types for each category between the training set and the validation set.
Table 7: Performance Evaluation on Something-Elss We consider the base and compositional
splits. *: Uses groundtruth box annotations during evaluation.
Model
Split
Object Supervision
Top-1 Acc. (%)
Top-5 Acc. (%)
STRG, STIN+OIE+NL 4
Base
78.1
94.5
ORVIT 331
Base
/
87.1
97.6
IV-CL (Ours)
Base
X
79.1
95.7
STRG, STIN+OIE+NL 43
Comp
V
56.2
81.3
ORVIT [3T
Comp
V
69.7
90.1
IV-CI (Ours)
Comp
X
59.6
85.6
Due to the large domain gap between CATER and Something-Eles videos, we choose to perform pretraining directly on the corresponding training splits of the Comething-Elss benchmark. We use the same pretraining and finetuning hyper parameters sampled at stride size of during finetuning. During both pretraining and finetuning, we apply the standard video data augmentation techniques used by prior work (e.g. (5). In Table we observe that our method generalizes well to real videos, and it achieves competitive performance compared to methods that use annotated boxes during training (STRG, STIN+OIE+NL) and evaluation (ORViT).
in ACRE, except that we use 16 frames
5 Conclusion and Future Work
In this work we demonstrate that competitive visual reasoning can be achieved in general-purpose end-to-end neural network, with the help self-supervised visual pretraining. Our proposed implicit visual concept learner (IV-CL) framework leverages a Transformer encoder to "compress" visual in- puts into slot tokens, and trained with a self-supervised video reconstruction objective. Quantitative and qualitative evaluations confirm the effectiveness of IV-CL CATER and ACRE visual reasoning benchmarks, when compared to supervised visual pretraining and neuro neuro-symbolic approaches. A limitation of our work is that evaluations are performed purely on synthetic reasoning tasks. We believe extending evaluation to large-scale natural video reasoning benchmarks, building joint model for visual recognition and reasoning, and exploring how to incorporate explicit object-centric knowledge when such knowledge is available are interesting future directions to pursue.
Acknowledgements: Part of the research was conducted while C.I.. worked student researcher at Google. C.S. and C.I., are in part supported by research grants from Honda Research Institute, Meta AI, Samsung Advanced Institute of Technology, and Richard B. Salomon Faculty Research Award.
10

---

References
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. arXiv preprint S.ii:2000500922.. 2020.
[2] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervvsed learning from raw video, audio and text. In NeurlPS, 2021.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
[4] Jacob Andreas. Measuring compositionality in representation learning. arXiv preprint wwXiv:/902.0718/ 2019.
[5] Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: video vision transformer. In ICCV, 2021.
[6] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. ICLR, 2022.
[7] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2 07258, 2021.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
[9] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint arXiv: 11390, 2019.
[10] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
[11] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[12] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[13] Ting Chen, Saurabh Saxena, Lala Li, David J Flect, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. In ICLR, 2022.
[14] Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B Tenenbaum, and Chuang Gan. Comphy: Compositional physical reasoning of objects and events from videos. In ICLR, 2022.
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidircctional transformers for language understanding. In NAACL, 2019.
[17] David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt Botvinick. Attention over learned object embeddings enables complex visual reasoning. In NeurIPS, 2021.
[18] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
[19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010.11929, 2020.
[20] Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners. arXiv preprint arXiv:2205.091 13, 2022.
[21] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.1222 2018.
11

---

[22] Golnaz Ghiasi, Barret Zoph, Ekin D Cubuk, Quoc V Le, and Tsung-Y. Yi Lin. Multi-task self- training for learning general representations. In ICCV, 2021.
[23]
Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and temporal reasoning. arXiv preprint arXiv:1910.047/440 2019.
[24] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In CVPR, 2022.
[25] Alison Gopnik and David M Sobel. Detecting blickets: How young children use information about novel causal powers in categorization and induction. Child development, 7165):1205-1222.
2000.
[26] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self supervisedd learning. In NeurlPS, 2020.
[27] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[28] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[29] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[30] Katherine Hermann, Ting Chen, and Simon Korbblith. The origins and prevalence of texture bias in convolutional neural networks. In NeurIPS, 2020.
[31] Roei Herzig, Elad Ben-Avrabam, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Object-region video transformers. CVPR, 2022.
[32] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
[33] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In ICML, 2021.
[34] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
[35] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In CVPR, 2018.
[36] lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In CVPR, 2017.
[37] Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequeee recurrent networks. In ICML, 2018.
[38] Valerii Likhosherstov, Anurag Arnab, Krzysztof Choromanski. Mario Lucic, Yi Tay, Adrian Weller, and Mostafa Dehghani. Polyvit: Co-training vision transformers on images, videos and audio. arXiv preprint (rXiv:2111.12991 2021.
[39] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[40] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[41] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurlPS, 2020.
[42] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv: 190..189444,2884, 2019.
12

---

[43] Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something-elsse Compositional action recognition with spatial temporal interaction networks. In CVPR, 2020.
[44] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. http::/ddistll.pub///0 7/ffettt htt/ffaaturr--isuaalinnt
[45] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint wwXiv:1807.03744 2018.
[46] Namuk Park. Wonjae Kim, Byeongho Hco, Tackyung Kim, and Sangdoo Yun. What do self-supervised vision transformers learn? arXiv preprint arXiv:2305.0000 0072500079, 2023.
[47] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
[48] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[49] Aviv Shamsian, Ofri Kleinfeld, Amir Globerson, and Gal Chechik. Learning object permanence from video. In ECCV, 2020.
[50] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, 2019.
[51] Chen Sun, Arsha Nagrani, Yonglong Tian, and Cordelia Schmid. Composable augmentation encoding for video representation learning. In ICCV, 2021.
[52] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017.
[53] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders data-efficient learners for self-super video pre-training. arXiv preprint 2202 2022.
[54] Manuel Traub, Sebastian Otte, Tobias Menge, Matthias Karlbauer, Jannik Thuemmel, and Martin V Butz. Learning what and where: Disentangling location and identity tracking without supervision. In ICLR, 2023.
[55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz. Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[56] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre- Antorn-Attinee Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 2010.
[57] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuaa Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022.
[58] Yuhuai Wu, Honghua Dong, Roger Grosse, and Jimmy Ba. The scattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning. arXiv preprint arXiv:2007.04212. 2020.
[59] Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. arXiv preprint arXiv:2008. 20X0::0008.0565, 2020.
[60] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 2022.
[61] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint
1rX.v:191000144.. 2019.
[62] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. In NeurIPS, 2018.
[63] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018.
[64] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In CVPR, 2019.
13

---

[65] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song- Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In CVPR, 2019.
[66] Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu, Acre: Abstract causal reasoning beyond covariation. In CVPR, 2021.
[67] Chi Zhang, Baoxiong Jia, Feng Gao. Yixin Zhu, Hongjing Lu, and Song-Chun Chun Zhu. Learning perceptual inference by contrasting, In NeurIPS, 2019.
[68] Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Is an object-centric video representation beneficial for transfer? In ACCV, 2022.
[69] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
[70] Honglu Zhou, Asim Kadav, Farley Lai, Alexandru Niculescu-Mizil, Martin Renqiang Min, Mubbasir Kapadia, and Hans Peter Graf. Hopper: Multi-hop transformer for spatiotemporal reasoning. In ICLR, 2021.
14

---

Additional Experimental Details
Transfer Learning Framework. In Figure we visualized our proposed self-supervised pretraining framework. Once the representation network has been pretrained, we discard the image decoder and only use the ViT-B image encoder, along with the pretrained temporal transformer, illustration of the transfer learning process is shown in Figure A1
Question Decoder
Snitch: (x. y)
Temporal Transformer
ViT-B Encoder
VIT-B Encoder
ViT-B Encoder
VIT-B Encoder
Figure A1: An illustration of the transfer learning process. Both the ViT-B image encoder and the temporal transformer are transferred to downstream visual reasoning tasks to encode video inputs. Unlike pretraining, only the slot tokens are provided as inputs to the temporal transformer.
t=0
t=5
t=10
t=300
CATER: Predict the location of the (possibly occluded) golden enitch in finel frame temporel context.
Evidence
Evidence
Evidence
Query
ACRE: Predict whether query object(s) activate platform undetermined based evidence
Figure A2: Illustration of the CATER (top) and ACRE (bottom) benchmarks.
Illustration of the Benchmarks. In Figure A2 we provide the illustrations of the CATER benchmark and the ACRE benchmark, respectively. The CATER benchmark features special golden ball, called the "snitch", and the associated reasoning task is to determine the snitch's 's position at the final frame despite occlusions. Object locations in the CATER dataset are denoted by positions on an invisible 6-by-6 grid; therefore, in essence, the CATER task boils down to a 36- way classification problem. The CATER dataset features a split where the camera is statically fixed to a particular angle and position throughout the videos, well moving camera split where the viewing angle is able to change over time. We use the static split for evaluation. Each video has 300 frames.
The ACRE benchmark is inspired by the research on developmental psychology: Given few context demonstrations of different object combinations, as well as the resulting effect, young children have been shown to successfully infer which objects contain the "Blicketness" property, and which combinations would cause the platform to play music. ACRE explicitly evaluates four kinds of reasoning capabilities: direct, indirect, screened-off, and backward-blockke frame be a combination that was explicitly provided the context frames tests a model's 's direct reasoning ability. Indirect reasoning be tested by novel query combination, the effect of which
Having the query
15

---

requires understanding multiple context frames to deduce. In screen-off off questions, the model must understand that as long as a singular Blicket object is placed on the platform, the entire combination would cause it to light up. In backward-blocs effect of query combination cannot be determined from the provided context frames. Furthermore, ACRE features three different dataset splits to test model generalization: Independent and Identically Distributed (I..D.),, compositionality (comp). systematicity (sys). the compositionality split, the shape-material-cole combinations of the CLEVR objects in the test set are not seen before in the train split; therefore, the model must learn to generalize across object attributes. In the systematicity split, the evidence frames of the train split contain three lit up examples, whereas the evidence frames of the test split contain four.
questions, the model must recognize when the
Number of the Slot Tokens. In Table A1 we provide ablation experiment on the impact of the number of slot tokens for the reasoning performance on all splits. Unlike CATER, whose goal is to infer the position of a single object, the "snitch", the ACRE benchmark requires reasoning over combinations of objects, and their relationship with the platform. that more slot tokens are needed to achieve optimal performance. We observe that the performance starts to saturate given four or eight slots.
result, we generally observe
Table A1: ACRE # tokens. We show results compositionality (comp), systematicity (sys), and L.D.. (iid) splits.
# slots
comp
sys
iid
1
91.75%
90.34%
90.96%
2
90.82%
88.21 %
88.73%
4
93.27%
92.64%
92.9
8
95.54%
86.18%
88.97%
64
90.45%
80.07%
90.82%
(a) Pretrained CATER
(b) Finetuned ACRE
Figure A3: Visualizations of the Slot Tokens. The row corresponds to the attention heatmaps from the slot tokens after pretraining on CATER, and the bottom row corresponds the heatmaps after finetuning on ACRE.
16

---

Visualizations of Slot Tokens. Figure A3 A3 additional visualizations of the slot token attention heatmaps after pretraining on CATER, and finetuning on ACRE, respectively. We follow the same attention rollout technique in Figure 3. For ACRE, we show the example when the platform is visible (context information) on bottom left, and when the platform is invisible (question) on the bottom right. We observe consistent trend that a subset of the heatmaps exhibit object-centric behavior, especially before finetuning on ACRE. After finetuning, we observe that some slots remain focusing on individual objects, while the others attempt model the relationships among different objects and the platform.
Our MAE baselines are pretrained with the same hyper parameters (e.g. optimization and mask ratio) IV-CL, which we have observed be optimal based on the validation set performance. The image encoders for all methods are based on ViT-B, hence the total model sizes are comparable.
17


Reviews:
[{"review_id": "1hOKoLGoUe", "reviewer": "Reviewer_zViR", "summary": "This paper investigates if an end-to-end trained neural network is able to solve challenging visual reasoning tasks. A new representation learning framework named implicit visual concept learner (IV-CL) is proposed. It consists of two main components. First, a single image is compressed into a small set of tokens with a neural network. Second, the slot tokens are provided as context information for other images in the same video, where the goal is to perform video reconstruction via the masked autoencoding objective with the temporal context. The method outperforms traditional supervised pretraining works, and several observations are summarized based on experiment results.", "questions": "Discuss the question in the “Weakness” section: Why IV-CL learns \"implicit\" \"concept\"? Or why does the pre-training on current datasets make sense?", "limitations": "The paper has discussed realistic limitations. However, an important limitation should have been considered (please refer to “Weakness” section).\n\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is with good motivation and the analysis of current problems of visual reasoning is reasonable and insightful. Different from explicit representations, implicit representations may correspond to part of, or a combination of human-interpretable symbols, or may be discretized and adjusted based on human understanding or model feedback. It has several advantages: free from a pre-defined concept vocabulary or concept classifiers, and avoids early loss of information.\n\nThus, the paper proposes a new framework named implicit visual concept learner (IV-CL) and summarizes several observations based on experiment results.\n", "weaknesses": "The proposed implicit visual concept learner (IV-CL) is not a sound solution to the motivation of “implicit representation” of the “explicit” reasoning process, because the model based on self-supervised pretraining is totally an implicit one. \n\nThe image is compressed into a small set of tokens, the slot tokens are provided as context information via a temporal transformer network for other images in the same video, and the learning process is supervised by masked autoencoding and video reconstruction. \nThe explicit information adopted in previous works (e.g., [1,2]) is abandoned. The information includes localized objects, object attributes (e.g., color, shape), and spatial relationships. Reasonable implicit reasoning should imply the information. For example, the visual patches attention for each slot is in correspondence with one piece of information. The visualization in Fig.3 shows some interpretable results obtained from the self-supervised training. However, without any supervision, it is not controllable and stable, and difficult to generalize to more complex visual reasoning applications.\n\nTherefore, method design should be reflected and refined to match the goal proposed in the introduction (L36-39): “Some of the learned implicit representations may be discretized into human-interpretable symbols for the purposes of human understanding of and feedback to the model. Others may correspond to part of or a combination of human-interpretable symbols.”\n\n\nAlternatively, it is also reasonable to study the effect of pretraining on visual reasoning. However:\n\n1.\tThe introduction starts with the problem of existing explicit reasoning (L33-44). If the core contribution is a new pre-training paradigm, the demonstration should be adjusted.\n\n2.\tMore importantly, the paper is built upon small, simplified synthetic data, which have a huge gap with practical visual reasoning applications, e.g., VQA in real scenes with diverse visual concepts. Thus, the result and performance of pre-training on these benchmarks are not that convincing and lack generalization. The limitation is discussed in the paper, but I regard it as a crucial problem that should be considered in this paper.\n\n\n[1] Mao, J., Gan, C., Kohli, P., Tenenbaum, J. B., & Wu, J. (2019). The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904.1258\n\n[2] Yi, K., Wu, J., Gan, C., Torralba, A., Kohli, P., & Tenenbaum, J. (2018). Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 31.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer zViR for the in-depth, constructive feedback.  Below we seek to address the reviewer’s concerns:\n\n> **On explicit information from supervision**\n\nThe reviewer suggests that “without any supervision, [the learned representation] is not controllable and stable, and difficult to generalize to more complex visual reasoning applications”.  In our work, we demonstrate the opposite findings, by comparing our proposed IV-CL framework against a supervised learning paradigm, where slot tokens in the Transformer network are pretrained with supervised visual recognition tasks (e.g. decoding image-level labels or performing object detection). In alignment with the reviewers suggestion, we do present a study on the effect of visual pretraining on visual reasoning in Table 1, which demonstrates the effectiveness of IV-CL over its supervised pretraining counterparts.\n\n> **On the “core contribution”**\n\nThe core contribution is not limited to a pretraining paradigm, but also a singular unified framework for learning to reason end-to-end without explicit supervision.  This framework enables the learning of powerful implicit representations, which emerge during visual pretraining and are maintained and refined for reasoning.  The benefit of self-supervised pretraining and finetuning is its generality; compared to explicit representation learning utilized for existing reasoning approaches, our approach requires no explicit human-level supervision.  Therefore, the representations are not fixed and frozen as in prior work, but can flexibly adapt to extract the information useful for the reasoning task at hand during fine-tuning.\n\n> **On synthetic datasets**\n\nAlthough the datasets are synthetic in nature, the tasks they evaluate require higher-level reasoning capabilities to solve.  The difficulty in these tasks is less about understanding the visual details of the elements in the scene but more about how they relate to and interact with each other.  Therefore, prior works on spatiotemporal reasoning (e.g. OPNet[48], Hopper[69], and ALOE [17]) also perform evaluations on such synthetic benchmarks, which we demonstrate new SOTA for.  However, we do agree that spatiotemporal reasoning on natural datasets would be an interesting and worthwhile endeavor in the future, both in terms of constructing appropriate datasets and exploring our approach on them. In Table A2 of the supplementary, we also report IV-CL’s performance on the Something-Else (Materzynska et al., CVPR 2020) benchmark. Something-Else is a real-world video benchmark focusing on spatiotemporal relationship reasoning, and compositional generalization. IV-CL outperforms the competitive method (STIN+OIE+NL) without using object bounding boxes as its supervision. We note that the current state-of-the-art, ORViT (Herzig et al., CVPR 2022), uses ground-truth object bounding boxes during both training and evaluation, and thus is not directly comparable to IV-CL.\n\n> **On why IV-CL learns “implicit” concepts** \n\nAs for what IV-CL learns to encode in the slot tokens, we believe they are implicit concepts.  We believe they are implicit, because no explicit supervision is utilized to learn the concepts.  We believe it is learning concepts through both performant downstream reasoning performance (Tables 4 and 5) and through probing, which verifies object-centric encoding (Figure 3). We will clarify further in the final version.\n\n> **On why pre-training on current datasets make sense**\n\nWe believe that the data IV-CL is pretrained on helps it to demonstrate higher-level reasoning capabilities.  Explicitly pretraining on an environment that has CLEVR objects interacting with each other promotes the learning of object-centric representations in a self-supervised way, which also helps it to unlock reasoning capabilities in reasoning tasks that also utilize CLEVR objects.  During finetuning, such implicitly learned object-centric representations are flexibly adapted to the task under our framework; as we demonstrate in Figure 3 (b), after finetuning on ACRE, the slots evolve to capture object relations beyond individual objects, which help it solve the reasoning task at hand.\n"}, {"review_id": "KlnQ7MaydQ", "reviewer": "Reviewer_t1ps", "summary": "This paper explores with the need of explicit visual abstraction for visual reasoning tasks.Instead, this paper explores the use of implicit representations learned through a slot attention module.The proposed approach uses slot attention in a frame prediction frame-work that aims to compress visual input information to set of representative tokens/slots. The method is evaluated on CATER, ACRE and RAVEN tasks.", "questions": "* An clear explanation of the advantages/limitations with respect to  \"Learning What and Where: Disentangling Location and Identity Tracking Without Supervision, ICLR 2023\", would be greatly appreciated.\n\n* Details of the evaluation protocol on CATER should be discussed in more detail.\n\n* Details of the applicability of the proposed method to the moving camera split of CATER should be discussed in more detail.\n", "limitations": "Currently, the limitations are not discussed is detail. While evaluation on synthetic data is definitely a limitation, an analysis of failure cases would be helpful.Furthermore, broader societal impacts are also not discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper presents an interesting method that seems to generalize across a diverse set of datasets.\n* The results on RAVEN are very promising. \n* The paper is well written.\n", "weaknesses": "* The proposed method has lot of similarities to prior work: \"Learning What and Where: Disentangling Location and Identity Tracking Without Supervision, ICLR 2023\", which also uses slot attention to learn implicit representations.\n\n* The above mentioned work achieves 90.7% Top-1 accuracy on CATER, significantly better than the 70.14% reported in this work.\n\n* Therefore, a detained discussion about the similarity and differences to prior work is necessary.\n\n* The proposed method seems to be designed for image sequences. Could the proposed implicit representation be extended to visual reasoning problems which take a single image as input e.g. CLEVR?\n\n* Important details about the evaluation protocol is unclear:: for evaluation on CATER in L242 it is stated that \"100 uniformly\nsampled frames of stride 3 during evaluation\". Why is the full sequence not utilized during evaluation?\n\n* The paper does not report results on the moving camera split of CATER. Why is this the case? Is the method limited to only static camera sequences?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer t1ps for supplying useful feedback in response to our work, as well as relevant prior work, which we address:\n\n> **CATER moving camera performance**\n\nWe conduct the moving camera experiments by adopting the same pretrained checkpoint as for the static camera experiments (which was pretrained on videos with static and moving cameras), and using the same hyperparameters for finetuning. On the moving CATER test split, IV-CL achieves 59.1% top-1 accuracy and 89.5% top-5 accuracy. Please also find the results in Table R1 of the rebuttal PDF. The performance is on par with the best ALOE method with auxiliary losses, and better than other baseline methods.  We will update the manuscript with the results of this experiment for the final camera-ready.\n\n> **Comparison with Loci** \n\nThe “Learning What and Where: Disentangling Location and Identity Tracking Without Supervision, ICLR 2023\" (Loci) paper is indeed relevant to our approach, and demonstrates strong tracking results on CATER. We will discuss Loci as a related work in our final version. We share similar high-level motivations, but highlight the important distinctions: (1) Our work is more general; this is not only reflected in our evaluation, which includes multiple visual reasoning benchmarks beyond CATER, but is also apparent in the design. We do not train IV-CL using specialized task-specific losses, whereas Loci uses three losses specific to tracking and object permanence. IV-CL is general in that it is pretrained using a visual reconstruction task with minimal assumptions, and is finetuned generally on reasoning tasks directly without injecting information about how the task “should be” solved via auxiliary objectives.  (2) Loci assumes static backgrounds and cameras, whereas our approach works with moving cameras and backgrounds. We consider two benchmarks with moving cameras and backgrounds: (1) CATER moving camera split (see our response above), and (2) Something-Else (Materzynska et al., CVPR 2020), a real-world video benchmark focusing on spatiotemporal relationship reasoning and compositional generalization. Table A2 of supplementary shows that IV-CL outperforms the competitive baseline (STIN+OIE+NL) without using object bounding boxes as its supervision. We note that the current state-of-the-art, ORViT (Herzig et al., CVPR 2022), uses ground-truth object bounding boxes during both training and evaluation, and thus is not directly comparable to IV-CL.  \n\n> **Extension to single image inputs**\n\nThe proposed implicit representation could be extended to visual reasoning problems which take a single image as input, such as CLEVR, by treating the image as a video with a single frame.\n\n> **Frame sampling strategy**\n \nWe choose uniformly sampled frames during evaluation to balance memory usage with meaningful frames. Each CATER video has 300 frames. By sampling every 3 frames, the effective video still displays the same actions, but the amount of memory utilized is substantially smaller. Frame sampling during evaluation is a common practice, for example, ALOE sample 80 frames for evaluation.\n"}, {"review_id": "ra1KBVxUw6", "reviewer": "Reviewer_e8rc", "summary": "The authors investigate whether unsupervised pre-training yields representations that are suitable for solving reasoning tasks. To that end, they propose a new transformer-based model inspired by masked autoencoders that is pre-trained on CATER. The authors find that the proposed model outperforms supervised pre-training counterparts and it is competitive with respect to object-centric approaches. Interestingly, the representations learned on CATER transfer well to RAVEN, achieving state-of-the-art results. In addition to the main experiments, the authors include multiple ablations and visualizations of the learned slots. Additional experimental details, ablations on the number of slot tokens, visualizations, and pre-training data are provided in the Appendix.", "questions": "* Could you clarify the conceptual difference between unsupervised slot attention methods and the proposed method (see clarity above)?\n* Is [A] relevant to your work?\n\n[A] Faulkner, Ryan, and Daniel Zoran. \"Solving Reasoning Tasks with a Slot Transformer.\"", "limitations": "The authors include discussion on the limitations in the last section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\n========\n* The proposed pre-training scheme is novel to the best of my knowledge\n* The finding that representations learned on CATER transfer to RAVEN is novel and interesting\n\nQuality\n======\n* The proposed method is technically sound\n* The model design choices are properly justified with a good number of ablations (6 tables in the main text and one in the appendix).\n* Besides comparing with state of the art methods, the authors provide additional baselines such as supervised pretraining, leading to the finding that pre-training on JFT is not significantly better than pre-training on ImageNet.\n* The slot visualizations provide additional insights on the inner mechanisms of the learned model.\n\nClarity\n=====\n* The text is well-written and easy to read.\n\nSignificance\n==========\n* The fact that unsupervised pre-training on CATER results in representations that compete with dedicated slot-attention methods and transfer to RAVEN matrices is an important finding that will have an impact in the research community", "weaknesses": "Originality\n========\n* Unsupervised pre-training for solving reasoning tasks has already been explored in the past (slot attention) and the proposed method is close to existing methods like MAE. However, I think the originality of this work lies on the empirical evidence obtained from training the proposed method in different ways applied to reasoning tasks.\n\nQuality\n======\n* The authors make a clear distinction between object-centric two-stage methods using e.g. slot attention, and their proposed approach. However, isn’t the proposed approach also slot-based and two-stage (pre-training + fine-tuning)?\n\nClarity\n=====\n* Figure 1 shows a difference between existing slot attention methods (b) and the proposed method (c) in the fact that the proposed method is single-stage. However, does not the method have a pre-training and fine-tuning stage?\n* In Figure 2, the slot tokens suddenly appear at the output of the encoders, should not they be in the input too (like the [CLS] tokens)?\n\nMinor\n=====\nLine 173: reasonign", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer e8rc for their review, and seek to address their listed questions and feedback:\n\n> **On relevant work**\n\nWe do agree that the “[A] Solving Reasoning Tasks with a Slot Transformer” is relevant and related to our approach.  We would like to highlight that our IV-CL approach is more general, and utilizes a simpler architecture.  The Slot Transformer work [A] utilizes a slot attention encoder, a temporal transformer, as well as a variational autoencoder in terms of architecture, whereas IV-CL simply utilizes a temporal transformer and an encoder with slot tokens.  Furthermore, the Slot Transformer utilizes two auxiliary losses, whereas IV-CL is a singular reconstruction loss with minimal assumptions during pretraining.  Lastly, the performance that Slot Transformer achieves on CATER (the only overlapping reasoning task) is 62.9% (static camera) / 35.8% (moving camera), whereas IV-CL substantially outperforms it with 70.1% (static) / 59.1% (moving) performance. Please find the results in Table R1 of the rebuttal PDF. We will add the discussion and comparison in the final version.\n\n> **Conceptual difference between Figure 1b (ALOE) and Figure 1c (IV-CL)**\n\nOur definition of “stages” is based on how the input information is processed during inference.  In prior approaches, such as neuro-symbolic (Figure 1a), attention-over-objects (Figure 1b), the input information must be first explicitly organized into objects and their corresponding descriptors (perception stage), before the reasoning stage can be performed. They are hence “two-staged”; the perception model is trained first and frozen, and the reasoning model is trained separately.  The perception model is therefore brittle and inflexible for reasoning, as it is trained without understanding any information about what representations are useful for the downstream reasoning tasks.  In contrast, IV-CL uses end-to-end neural networks without explicitly enforcing any interpretable intermediate representations.  IV-CL can be end-to-end trained to jointly learn the perception model and reasoning model together.\n\n> **On Figure 2**\n\nWe agree that the slot tokens should be input as well, but did not visualize them as part of the input for clarity. We have included the revision in the Figure R1 of the rebuttal PDF, will amend this for the final version.\n"}, {"review_id": "FnYs7Q6NBr", "reviewer": "Reviewer_PrZv", "summary": "- The authors explore whether self-supervised visual pretraining can allow end-to-end methods to be more general purpose.\n- The paper proposes a self-supervised representation learning framework for this task, based on slot tokens, and evaluates performance on multiple benchmarks. \n", "questions": "See above ^", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written, studies an interesting problem, and proposes a satisfying solution.\n- The idea of an implicit visual concept learner with self-supervised pre-training, that retains the same benefits as fully symbolic methods, is compelling and will be of interest to the neuro-symbolic & end-to-end learning communities. \n", "weaknesses": "- W1. A key weakness of the paper is that the evaluation datasets are based on CLEVR objects in *synthetic* images. I would like to see experiments on real-world images; I’m not yet convinced that this self-supervised objective with slot tokens would work in natural images that may contain clutter, etc. I am happy to raise my score if authors add evaluation on a real-world dataset.\n\n- W2. I am interested in the performance of using slot-based inputs, instead of explicitly object-centric representations, for neuro-symbolic methods. The proposed methods leverages these softer slot tokens + end-to-end methods; would the change from object-centric input to slot tokens decrease performance of neuro-symbolic methods significantly? Comparison to the NS-CL [1], a neuro-symbolic approach, with slot tokens may be interesting.\n\n[1] The Neuro-Symbolic Concept Learner, ICLR 2019\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer PrZv for their thorough review.  We seek to address their listed considerations below:\n\n> **On synthetic evaluation**\n\nWe primarily evaluate the reasoning capabilities for IV-CL over established benchmarks for spatiotemporal reasoning, which are largely synthetic due to the difficulty of generating natural-looking video reasoning datasets.  Although these benchmarks are synthetic, they fundamentally succeed at testing reasoning capabilities (such as causal inference, object permanence, etc.) beyond simple recognition.  Furthermore, we outperform other approaches from prior work, which also solely evaluate on such synthetic benchmarks.  We take our reported results as strong preliminary evidence that such models can learn to reason over space and time, and as an encouraging signal that such fundamental reasoning capabilities can transfer over into the future for a natural visual domain. In Table A2 of the supplementary, we also report IV-CL’s performance on the Something-Else benchmark (Materzynska et al., CVPR 2020). Something-Else is a real-world video benchmark focusing on spatiotemporal relationship reasoning, and compositional generalization. IV-CL outperforms the competitive baseline (STIN+OIE+NL) without using object bounding boxes as its supervision. We note that the current state-of-the-art, ORViT (Herzig et al., CVPR 2022), uses ground-truth object bounding boxes during both training and evaluation, and thus is not directly comparable to IV-CL.\n\n> **Using implicit concepts in a neuro-symbolic framework**\n\nWe agree that exploring whether softer slot tokens would be beneficial for neuro-symbolic approaches would indeed be interesting. However, this would require translating softer slot tokens into “explicit” concepts as required by the symbolic program, as illustrated in Figure 1(a). A naive quantization approach is likely to cause loss of information, and we thus believe that combining soft concepts with symbolic programs is an interesting but challenging future work.\n"}]

Author rebuttal:
We thank the reviewers for their thorough, thoughtful, and constructive comments.  We are happy to hear that the reviewers consistently agree that the proposed work is interesting, well-presented, and presents novel insights.  In particular, the reviewers highlighted that the proposed method is technically sound, that the results on transferring CATER pretraining to RAVEN are both intriguing and novel, and that learning implicit visual concepts is compelling and of interest to the community.

A common comment from the reviewers was the use of synthetic data in our evaluation:

- First, we would like to highlight that it is a common practice (OPNet[48], Hopper[69], ALOE[17]) to evaluate visual reasoning ideas on synthetic datasets. This is due to the difficulty of generating natural-looking video reasoning datasets with sufficiently large scale and accurate annotations. Although these established benchmarks (such as CATER and ACRE) are synthetic, they fundamentally succeed at isolating out and testing reasoning capabilities (such as causal inference, object permanence, etc.) beyond simple visual recognition.

- Second, we have in addition evaluated our method on a real-world dataset Something-Else, in the supplement Table A2. Something-Else is a real-world video benchmark focusing on spatiotemporal relationship reasoning, and compositional generalization. IV-CL outperforms the competitive baseline (STIN+OIE+NL) without using object bounding boxes as its supervision, especially on the "Comp" split which requires compositional generalization.

We would also like to reiterate the core contributions of our work:

- We propose a unified framework for learning to reason end-to-end without explicit supervision.  This framework enables the learning of powerful implicit representations, which emerge during visual pretraining and are maintained and refined for reasoning.  

- We demonstrate that a visual pretraining task of masked reconstruction, which makes minimal assumptions about downstream reasoning tasks, is useful in learning powerful representations that capture implicit visual concepts, such as objects and object relations.  

- We show via extensive ablation studies and evaluations that our framework can be fine-tuned directly on reasoning tasks without the need for additional task-specific auxiliary losses or inductive biases, and achieves competitive performance or establishes new state-of-the-art on four visual reasoning benchmarks.

We thank the reviewers for their time and careful consideration,

The Authors


Meta-review: The paper investigates the role of visual pretraining in end-to-end learning for visual reasoning tasks. It introduces a self-supervised framework that compresses video frames into a set of tokens using a transformer network and evaluates its performance across various benchmarks, notably outperforming traditional methods.

Overall, the reviewers commend the paper for its novel pre-training scheme, promising results on diverse datasets, and technical soundness. The idea of an implicit visual concept learner is particularly well-received for its potential impact on both the neuro-symbolic and end-to-end learning communities. However, there are areas for improvement: the method's originality compared to prior work needs clarification, and the evaluation could be strengthened by including real-world datasets. Additionally, the paper could benefit from revising its alignment with its stated goals and making the implicit representations more interpretable. Some of these weaknesses have been partially addressed in the rebuttal. Given the paper's strengths and its potential for improvement through revisions, the committee leans towards an accept decision.
Decision: Accept (poster)