Towards Learning Implicit Symbolic Representation for Visual Reasoning
Authors: Chen Sun
Calvin Luo
Xingyi Zhou
Anurag Arnab
Cordelia Schmid
Venue: Submitted to ICLR 2023 (2023)
Keywords: visual reasoning
self-supervised learning
implicit symbolic representation

Abstract: Visual reasoning tasks are designed to test a learning algorithm's capability to infer causal relationships, discover object interactions, and understand temporal dynamics, all from visual cues. It is commonly believed that to achieve compositional generalization on visual reasoning, an explicit abstraction of the visual scene must be constructed; for example, object detection can be applied to the visual input to produce representations that are then processed by a neural network or a neuro-symbolic framework. We demonstrate that a simple and general self-supervised approach is able to learn implicit symbolic representations with general-purpose neural networks, enabling the end-to-end learning of visual reasoning directly from raw visual inputs. Our proposed approach ``compresses'' each frame of a video into a small set of tokens with a transformer network. The self-supervised learning objective is to reconstruct each image based on the compressed temporal context. To minimize the reconstruction loss, the network must learn a compact representation for each image, as well as capture temporal dynamics and object permanence from temporal context. We evaluate the proposed approach on two visual reasoning benchmarks, CATER and ACRE. We observe that self-supervised pretraining is essential to achieve compositional generalization for our end-to-end trained neural network, and our proposed method achieves on par or better performance compared to recent neuro-symbolic approaches that often require additional object-level supervision.

Under review as a conference paper at ICLR 2023
TOWARDS LEARNING IMPLICIT SYMBOLIC REPRESENTATION FOR VISUAL REASONING
Anonymous authors
Paper under double-blind review
ABSTRACT
Visual reasoning tasks are designed to test a learning algorithm's capability infer causal relationships, discover object interactions, and understand temporal dynamics, all from visual cues. commonly believed that to achieve compo- sitional generalization on visual reasoning, explicit abstraction of the visual scene must be constructed; for example, object detection can be applied the visual input to produce representations that are then processed by neural net- work or a neuro-symbolic framework. We demonstrate that a simple and general self- solf-supervised approach is able to learn implicit symbolic representations with general-purpose neural networks, enabling the end-to-end learning of visual rea- soning directly from raw visual inputs. Our proposed approach "compresses" each frame of a video into a small set of tokens with a transformer network. The self-supervised learning objective is reconstruct each image based on the com- pressed temporal context. To minimize the reconstruction loss, the network must learn compact representation for each image, well as capture temporal dy- namics and object permanence from temporal context. We evaluate the proposed approach on two visual reasoning benchmarks, CATER and ACRE. We observe that self-superviser pretraining is essential to achieve compositional generaliza- tion for our end-to-end trained neural network, and our proposed method achieves par better performance compared to recent neuro-symbolic approaches that often require additional object-level supervision.
INTRODUCTION
This paper investigates an end-to-end trained neural network is able to solve challenging visual reasoning tasks (Zhang et al. 2021 Girdhar & Ramanan 2019; al. (2019) that involve inferring causal relationships, discovering object relations, and capturing temporal dynamics. A prominent approach (Shamsian et al. 2020) for visual reasoning is to construct a structured and interpretable representation from the visual inputs, and then apply symbolic programs (Mao et al.) 2019) or neural networks (Dig al. T to solve the reasoning task. Despite their appealing properties, such as being interpretable and easier to inject expert knowledge into the learning framework. it is practi- cally challenging to determine what types of symbols use and how to detect them reliably from visual data. In fact, the suitable symbolic representation for a single scene may differ significantly across different tasks: the representation for modeling a single human's kinematics (e.g. with body parts and joints) is unlikely to be the same as that for modeling group social behaviors (e.g. each pedestrian can be viewed
whole entity). With the success of unified neural frameworks for multi-task learning (Bommasani et al., 2021). it is desirable to have a unified input interface (e.g. raw pixels) and have the neural network learn dynamically extract suitable representations for different tasks. However, how to learn distributed representation with deep neural network that be- haves and generalizes similarly to learning methods based on symbolic representation (Zhang al.) 2021 for visual reasoning remains open problem.
The key hypothesis we make in this paper is that a general- purpose neural network, such as Trans- formers Vaaswani et al. 2017). can be turned into implicit symbolic concept learner with self- supervised pre-training. For reasoning with image and video cues, the concepts are often organized object-centric, objects usually serve the basic units in visual reasoning tasks. Our proposed approach is inspired by the success of self-supervised learning of object detectors with neural net- works (Burgess al. 2019 Locatello et al. 2020 Niemeyer & Geiger 2021) and the emergence of

---

Under review as a conference paper at ICLR 2023
Stage
Stage 2
Single stage
Answer: (x.
Answer: (x, y)
Answer: (x.
Symbolic Programs
Neural Networks
Neural Networks
Object
representation
Distributed representation
Tokan
Distribuned representation
Neural Networks
Neural Networks
Neural Networks
Stage
Stage
End-to-end trained
(a) Neuro-Symbolic Concept Learning
(b) Attention over Object Embeddings
(c) Implicit Symbolic Concept Learning (ours)
Figure Comparison between neuro-symbolic approach (c.g. Mao al. (019 ) a hybrid ap- proach with learned object embeddings (e.g. Ding et al. (202T)) and our proposed approach for visual reasoning. The illustration of each model family flows upwards, where visual inputs are en- coded by neural networks (stage 1), and then processed by symbolic programs or another neural network to generate reasoning predictions (stage 2). Compared to (a) and (b), our approach does not require separate "preprocessing" stage to extract the symbolic representation from visual inputs, and the self-supervised pretrained neural network can be end-to-end "finetuned" to the downstream visual reasoning tasks.
object masks in self-supervisee classification networks (Caron et al. 2021 It is also motivated by concept binding neuroscience (Treisman 1996 Roskies] [999] Feldman 2013) and in machine learning (Greff et al.) 2020), where concept binding for raw visual inputs refers to the process of segregating and representing visual scenes into a collection (distributed) concept representation, which can be composed and utilized solve downstream recognition and reasoning tasks. The concepts are bound in an object-centric fashion, where attributes (c.g. colors, shapes, sizes) of the same objects associated via dynamic information routing. Different from explicit symbolic rep- resentation, implicit symbolic representation via dynamic information binding in a neural network does not require predefining the concept vocabulary or the construction of concept classifiers. The implicit representation can also be "finetuned" directly on the target tasks, it does not suffer from the early commitment or loss of information issues which may happen when visual inputs are converted into symbols and frozen descriptors (e.g. via object detection and classification).
Our proposed representation learning framework, implicit symbolic concept learner (IS-CL) con- sists of two main components: first, single image is compressed into small set of tokens with neural network. This is achieved by a vision transformer (ViT) network (Dosovitskiy et al., 2020) with multiple "slot" tokens (e.g. the [CLS] token in ViT) that attend to the image inputs. Sec- ond, the slot tokens are provided as context information via a temporal transformer network for other images in the same video, where the goal is to perform video reconstruction via the masked autoencoding (e et al. 2022) objective with the temporal context. Despite its simplicity, the re- construction objective motivates the emergence of two desired properties in the pretrained network: first. to provide context useful for video reconstruction, the image encoder must learn a compact representation of the scene with its slot tokens. Second, utilize the context cues, the temporal
2

---

Under review as a conference paper at ICLR 2023
transformer must learn to associate objects and their implicit representation across time ("implicit tracking"). and also capture the notion of object permanence - the existence of an object even when it is occluded from the visual observations. One intuitive way view our proposed IS-CL frame- work is from the perspective of Slot Attention model by Locatello et al. (2020) Instead of using a shared slot attention module to iteratively refine the encoded tokens, image encoder is imple- mented as a stack of Transformer encoder layers with dedicated "slot" tokens. This generalization enables us directly transfer the pretrained implicit symbolic representation encoded by expressive VIT backbones directly to downstream reasoning tasks.
To validate our proposed framework, we conduct extensive ablation experiments on the Compo- sitional Actions and TEmporal Reasoning (CATER) (Girdhar & Ramanan 2019) benchmark and the Abstract Causal REasoning (ACRE) /Zhag et al., 2021) benchmark. We observe that the self-supervised representation learned by IS-CI. indeed behave likes the symbolic representation, in the sense that when finetuned on CATER and ACRE, our learned representation achieves com- petitive or better generalization performance when compared with the frameworks that use explicit object-centric representation. Intriguingly, we observe that the network inductive biases, such as the number of slot tokens per image, play important role on transfer learning performance: On both datasets, we observe that a small number of slot tokens per image (1 for CATER and 4 for ACRE) lead the best transfer learning performance knowledge, our proposed framework is the first to achieve competitive performance on CATER and ACRE without the need to construct explicit symbolic representation from visual inputs.
visual reasoning tasks. To the best of our
In summary, our paper makes the following two main contributions: tions made by neuro-symbolic approaches, we demonstrate that compositional generalization for visual reasoning can be achieved with end-to-end neural networks and implicit symbolic represen- tations. Second, we propose a eelf-supervise representation learning framework IS-CL, learn implicit symbolic representation with uct, we show that the learned representation achieves competitive performance on the challenging CATER and ACRE visual reasoning benchmarks. The code and pretrained checkpoints will be released upon paper acceptance.
First, unlike common assump-
Transformer neural networks. As byprod-
2 RELATED WORK
Neural Network Pretraining. We have collectively made huge progress towards building unified
learning frameworks for a wide range of tasks, including natural language understanding (Devlin
al.
2018 Radford et al.
2019 Brown et al. 2020, [ Liu et al. 2019). visual recognition (Kokkinos)
2017
Kendall et al.] 2018 Zamir at 2018 Ghiasi et al. 2021) l). and multimodal perception (Jaegle
ct al.
2021: Sun et al. 2019) Likhosherstov et al. 2021 Girdhar et al. 2022, Alayrac et al. 2022)
As this
addptation learning paradigm gains momentum, researchers at Stanford (Bom-
masani et al. 2021) have even coined the term "foundation models" to refer to these pretrained neural
networks. Unfortunately, most of the "foundation models" for visual data focus perception tasks, such as object classification, detection, or image captioning.
Despite improved empirical performance on the visual question answering task (Hudson & Man- ning. 2019) Antol et al. 2015) Zellers et al. 2019),, visual reasoning remains challenging when mea- sured on more controlled benchmarks that require compositional generalization and causal learn- ing Zhang et al.) 2021 Girdhar & Ramanan 2019; Chen et al.] 2022). It is commonly believed that symbolic or neurosymbolic methods (Mao et al.) 2019 Yi et al. 2018 Lake & Baroni 2018 Andreas 2019), as opposed to the general purpose neural networks, are required to achieve gen- general-purpose
eralizable visual reasoning Yi et al. (2019); Zhang al. (202T). To our knowledge, our proposed framework is the first to demonstrate the effectiveness of implicit symbolic representation on these visual reasoning benchmarks.
Self-supervised Learning from Images and Videos. Self-supervised learning methods aim to learn strong visual representations from unlabelled datasets using pre-text tasks. Pre-text tasks were initially hand-designed to incorporate visual priors (Doersch et al. 2013 Zhang al. 2016 Caron et al. 2018). Subsequent works used contrastive formulations which encourage different augmented views of the same input to map to the same feature representation, whilst preventing the model from
collapsing to trivial solutions (Oord et al. 2018 Chen et al. 2020 He et al. 20200 Grill et al. 2020
Akbari et al. 2021
3

---

Under review as a conference paper at ICLR 2023
Our work is most related to masked self-supervised approaches. Early works in this area used stacked autoencoders (Vincent et al. 2010) or inpainting tasks (Pathak et al., 2016) with convo- lutional networks. These approaches have seen resurgence recently, inspired by BERT (Devlin
et al. 2018) and vision transformers (Dosovitskiy et al., 2020). BEiT (Bao et al.) 2022) encodes
masked patches with discrete variational autoencoders and predicts these tokens. Masked Autoen- coders (MAE) (He et all, 2022). on the other hand, simply regress to the pixel values of these tokens.
Masked Feature Prediction (Wei al., 2022) (MFP) also regresses to pixelwise targets, but feature
transformations of them as opposed to the direct RGB values as MAE. MAE and MFP have also
both been extended to video too (Tong et al. 2022, Feichtenhoter et al. 2022). and are shown to
be effective in object detection et al. (2022). The video reconstruction objective is also based on masked autoencoding, however, the goal is to learn a compact "implicit symbolic" representation for reasoning opposed to generic visual descriptors for recognition tasks. We confirm empirically
that the proposed method outperforms MAE and VidcoMAE pretraining methods by large margins on the CATER and ACRE benchmarks.
Object-centric Representation for Reasoning. Most of the existing neuro-symbolic (Mao et al., 2019 Yi et al. 2018) and neural network (Ding et al. 2021] based visual reasoning frameworks quire a "preprocessing" stage of symbolic representation construction, which often involves detect- ing and classifying objects and their attributes from image video inputs. Our proposed framework aims to investigate the effectiveness of single-stage, end-to-end neural networks for visual reason- ing, which is often more desirable than the two-stage frameworks for scenarios that require transfer learning or multi-task learning. In order to obtain the object-centric, or symbolic representation in the preprocessing stage, one can rely on supervised object detector (Mao et al. 2019) such as Mask R-CNN (He et al. 2017). An alternative approach is to employ self-supervised objectives and learn levelevel features that are correlated with objects, such as textures (Geirhos et al. 2018 |Her-
mann et al. 2020 Olah et al. 2017). or objects themselves (Burgess et al. 2019] Locatello et al.)
2020 Caron et al. 2021). ). In practice, supervised or self-supervised approaches for object detection and object-centric representation learning may suffer from the lack of supervised annotations, the noisy object detection results. For example, Zhang et al.] (2022) observed that object-centric rep- resentation is beneficial for transfer learning to temporal event classification only when the ground truth object detections are used.
METHOD
We now introduce the proposed implicit symbolic concept learning (IS-CI.) framework. We follow the pretraining and transfer learning paradigm: During pretraining (Figure we task a shared image encoder to output patch-level visual embeddings along with a small set of slot tokens that compress the image's 's information. The pretraining objective is masked autoencoding (MAE) for unlabeled video frames, namely reconstructing the pixel values for a subset of "masked" image patches, given the "unmasked" image patches as context. Compared to the standard MAE for images (He et al. 2022), the image decoder has access to two additional types of context information: (1) The encoded patch embedding from the unmasked image patches of the neighboring frames; (2) The encoded slot tokens from a subset of context frames. The context information is encoded and propagated by temporal transformer network. To successfully reconstruct a masked frame, the image encoder must learn a compact representation of the full image via the slot tokens, and the temporal transformer has to learn to capture object permenance and temporal dynamics.
During transfer learning (Figure D., the image decoder can be discarded, and only the image encoder and temporal transformer need to be transferred. The inputs to the temporal transformer are the slot tokens encoded from individual, unmasked video frames. We consider the full finetuning strategy where the weights of both the newly added task decoder (e.g. a linear classifier), and the pretrained image and temporal transformers are updated during transfer learning.
We adopt the Vision Transformer (ViT) backbone to encode each image indepen- dently: An input image is broken into non-overlapping patches of 16x 16 pixels, which are then linearly projected into patch embeddings as inputs to the transformer encoder. Spatial information is preserved by sinusoidal positional encodings. We use the standard ViT-Base configuration which has 12 Transformer encoder layers. Each layer has hidden size of 768, MLP projection size of 3072, and 12 attention heads. During pretraining, subset of video frames are spatially masked randomly
Image Encoder:
4

---

Under review as a conference paper at ICLR 2023
Un-masked pixel tokens
Masked pixel tokens
Image Decoder
Image Decoder
Image Decoder
Temporal Transformer
VIT-B Encoder
ViT-B Encoder
ViT-B Encoder
ViT-B Encoder
ViT-B Encoder
context frames
spatially masked frames
Figure 2: An overview of the proposed implicit symbolic concept learner (IS-CL) framework for self-supervised pretraining. We consider the video reconstruction objective masked autoencod- ing: A ViT-B image encoder is tasked to (1) extract visual representation (orange) for the unmasked patches per image and (2) compress an image into small set of slot tokens (blue). A temporal trans- former then propagates the information from slot representation and patch-level representation from neighboring frames, which are essential for successful reconstruction. We hypothesize that implicit symbolic representation emerges automatically in the slot tokens by self-supervised pre-training
given a masking ratio. As illustrated in Figure i ViT-B encoder. For context frames and during transfer learning, all image patches are provided as inputs the image encoder.
only the unmasked image patches are fed into the
In the seminal work by Locatello et al. (2020), slot tokens are defined as the represen- tational bottleneck in an image autoencoder, where the slot representations are iteratively updated with GRU after the slots attend to the visual inputs in each iteration. We borrow their terminology, and also use slots denote the representational bottleneck which we hope to encode symbolic, or object-centric information. We generalize their slot update rules by: (1) iteratively updating the input representation from raw pixels to visual representation encoded by the Transformer encoder (ViT); (2) replacing cross-attention with multi-headed self-attention; (3) using MLP layers with untied weights to update the intermediate slot representation as opposed to a shared GRU network. These two modifications allow to implement "slot attention" directly with Transformer encoder, simply by prepending slot tokens as additional inputs to the encoder (similar to [CLS] tokens). The learnable embedding
Slot Tokens:
initial slot embeddings the input of the visual encoder implemented lookup table. To compare the effectiveness of different methods to aggregate "slot" information, we also explore single-headed soft attention and Gumbel-max attention as used by Xu al. (2022).
Temporal Transformer: To propagate temporal information across frames, we use another trans- former encoder (with fewer layers than the ViT-B image encoder) which takes the tokens encoded by the image encoder as its inputs. During pretraining, the slot tokens from context frames, along with the unmasked patch tokens from the query frames are concatenated together and fed into the temporal transformer. For each query image, the temporal transformer outputs its corresponding unmasked patch tokens contextualized from both the unmasked patches from neighboring query frames and the slot tokens from context frames. The contextualized patches are then fed into the image decoder to compute the reconstruction loss. To preserve temporal position information, we learned positional embeddings (implemented with embedding lookup table). During transfer learning, the temporal transformer takes the slot tokens encoded by the image encoder as its inputs.
Putting the image encoder and the temporal transformer together, the overall video encoder used for transfer learning can be viewed as an factorized space-time encoder proposed by Armab et al. (202T) It is parameter--fffcient than the vanilla video vision transformer used by Tong et al. (002

---

Under review as a conference paper at ICLR 2023
Question Decoder
Snitch: (x, y)
Temporal Transformer
ViT-B Encoder
ViT-B Encoder
ViT-B Encoder
VIT-B Encoder
Figure 3: An illustration of the transfer learning process: Both the ViT-B image encoder and the temporal transformer are transferred to downstream visual reasoning tasks to encode video inputs. Unlike pretraining, only the slot tokens are provided as inputs to the temporal transformer.
Image Decoder for Pre-training: We use the same image decoder as in (He al. 2022) As illus- trated in Figure 2 the query images are decoded independently given the contextualized unmasked patch tokens. The image decoder is implemented with another transformer, where masked patch tokens are appended to the contextualized unmasked patch tokens as inputs to the image decoder. Sinusoidal positional encodings are used to indicate the spatial locations of individual patch tokens. We use the same number of layers, hidden size, and other hyperparameters as recommended by He measure the distance between the original query image patches and the reconstructed patches.
al. (2022). For pre-training purpose, we use
squared
Transfer Learning: As the goal of pre- pre-training is learn the slot tokens which we hope compress an input image into several implicitly symbolic tokens, we only ask the image encoder to generate the slot tokens during finetuning (Figure B). which are fed the temporal transformer as its inputs. We then average pool the output tokens of the temporal transformer and add a task-specifie decoder make predictions. Both benchmarks used in our experiments can be formulated as multi-class classification: For CATER, the goal is to predict the final location of the golden snitch (Figure + top), where the location is quantized into one of the 6x6 positions; For ACRE, the goal is predict whether the platform will activate, not activate, undetermined given query scenario (Figure bottom). We hence use linear classifiers as the task-specific decoders and the standard softmax cross-entropy for transfer learning.
4 EXPERIMENTS
We present results on CATER Girdhar & Ramanan 2019 and ACRE (Zhang et al. 2021
4.1 EXPERIMENTAL SETUP
Benchmarks: In the classic "shell game", ball is placed under cup and shuffled with other empty cups on flat surface; then, the objective is to determine which cup in the final shuffled configuration contains the ball. Inspired by this, CATER et al., 2017) objects as they move around the scene. A special golden ball, called the "smitch", present in each video, and the associated reasoning task is to determine the snitch's position at the final frame. Object locations in the CATER dataset are denoted by positions an invisible 6-by-6 grid; therefore, essence, the CATER task boils down to a 36-way classification problem. Solving this task is complicated by the fact that larger objects can visually occlude smaller ones, and certain objects can picked up and placed down to explicitly cover other objects; when an object is covered, it changes position in consistence with the larger object that covers it. Therefore, in order to solve the task successfully, a model must learn to reason not only about objects and movement, but also about object permanence, long-term occlusions, and recursive covering relationships.
dataset composed of videos of CLEVR (Johnson]

---

Under review as a conference paper at ICLR 2023
t=0
1=5
[=10
[=300
CATER: Predict (poseiblyy occluded) golden final given
Evidence
Evidence
Evidence
Query
ACRE: Predict whether query object(s) activate platform or undetermined on evidence Images.
Figure 4: Illustration the CATER and ACRE benchmarks.
The CATER dataset features split where the camera is statically fixed to particular angle and position throughout the videos, as well as moving camera split where the viewing angle is able to change over time. We the static split for evaluation. Each video has 300 frames. A visualization of CATER video and the associated snitch localization task is shown in Figure (top).
The ACRE dataset tests model's ability understand and discover causal relationships. The con- struction of the dataset is motivated by the Blicket experiment in developmental psychology (Gopnik Sobel 2000), where there a platform as well as many distinct objects, some of which contain the "Blicketness" property. When at least one object with the "Blicketness" property is placed on platform, music will be played; otherwise, the platform will maintain silence. Given few con- text demonstrations of different object combinations, as well as the resulting effect, young children have been shown to successfully infer which objects contain the "Blicketness" property, and which combinations would cause the platform to play music. In ACRE, the platform is represented by large pink block that either glows or remains dim depending on the combination of CLEVR ob- jects placed on it. Given six evidence frames of objects placed the platform, the objective of the reasoning task is to determine the effect a query frame, containing a potentially novel object com- bination, would have the platform. Possible answers include lighting up the platform, keeping the platform dim, or unable to be determined with the given evidence frames. A visualization of an example ACRE sample is shown in the bottom row Figure (bottom).
Pretraining data: We use the unlabeled videos from the training and validation splits of the CATER dataset for pretraining. Both the static and moving camera splits are used, which contains 9,304 videos in total. In our experiments, we observe that ACRE requires higher resolution inputs during pretraining and finetuning. Our default preprocessing setup to randomly sample 32 frames of 64x64 for pretraining checkpoints to be transferred to CATER, and 16 frames of 224 x24x224 for pretraining checkpoints to transferred to ACRE. The randomly sampled frames are sorted to preserve the arrow of time information. No additional data augmentations are performed.
Table 1: Pretraining ablation experiments on CATER
(a) Impact of the mask ratio
(b) Impact of pretrain context
(c) Impact of pretrain frames
Mask ratio
Accuracy
context
Accuracy
frames
Accuracy
37.5%
69.48%
69.48%
32
69.48 %
12.5%
66.35%
65.35%
8
62.28%
50%
66.57%
67.69%
16
66.63%
75%
64.12%
67. 67.47%
64
68.25%
87.5%
61.94%
16
64.34%
Transfer learning: For CATER, evaluate the static split which has 3,065 training, 768 vali- dation, and 1645 test examples. We select the hyperparameters based on the validation performance, then use both training and validation data train the model be evaluated on the test split. By de-

---

Under review as a conference paper at ICLR 2023
Table 2: Ablation experiments on CATER tokens
(a) Impact of the slot token
(b) Impact of slot pool layer
number
(c) Impact of slot pool method
Pool layer
Accuracy
# slots
Accuracy
Pool method
Accuracy
11
69.48%
69.48%
Slice
69.48% %
2
66.52%
5
55.80%
Soft 64.23%
62.67%
4
64.90%
Hard 65.90%
68.86%
8
64.45%
fault, we use 100 randomly sampled frames of 64 during training, and 100 uniformly sampled frames of stride 3 during evaluation. For ACRE, we explore all three splits, all of which contain 24,000 training, 8,000 validation, and 8,000 test examples. We again use the validation set to select hyperparameters and use both training and validation to obtain the models evaluated on the test split. We use all seven frames of 224 x 224 during training and evaluation.
Default hyperparameters: We Adam optimizer for pretraining at learning rate of 10-3, and Adam W optimizer for transfer learning at learning rate of 5 x 10- The pretraining checkpoints trained from scratch for 1,000 epochs at batch size of 256. For transfer learning, we finetune the pretrained checkpoints for 500 epochs at batch size of 512. All experiments are performed on TPU with 32 cores. Below we study the impact of several key model hyperparameters.
4.2 ABLATION STUDY
We use CATER for ablation study in Table D and reuse the optimal hyperparameters in ACRE experiments. The impact of the number of slot tokens for ACRE is studied separately in Table Q
Masking ratio: Contrary to the large masking ratio employed in vanilla MAE, we found that the optimal masking ratio was 37.5% on downstream CATER accuracy. This is perhaps due to the fact that CATER is designed to test "compositional generalization", and so the spatial context provides less information than in natural images and video.
We also study the impact of the number of frames the implicit symbolic concept learner is pretrained on, and find the best performance on 32 frames. Fixing the total number of pretraining frames, we then ablate over the number of context frames, which are the frames from which slot representations are generated. When O context frames are used, we essentially utilize only patch-level representations to perform reconstruction with the temporal transformer (simulating per-frame MAE followed by temporal transformer). We find that the best performance is achieved with context frames, which balances the number of slot representations with patch-level representations.
Number of Total Frames and Context Frames:
Number of Slot Tokens: Another useful ablation on the impact of the number of slots used for CATER and ACRE. In CATER, we find that only 1 slot token per frame is enough to solve the reasoning task. We believe that this may be due to how the reasoning objective of CATER is designed: to successfully perform snitch localization, the model need only maintain an accurate prediction of where the snitch actually or potentially is, and can ignore more detailed representation of other objects in the scene. Under the hypothesis that the slot tokens represent symbols, perhaps the singular slot token is enough to contain the snitch location. On the other hand, when ablating over the number of tokens for the ACRE task (Table 3. we find that a higher number of tokens is beneficial for reasoning performance. This can potentially be explained by the need to model multiple objects across evidence frames in order to solve the final query; under our belief that slot tokens encoding symbols, multiple may be needed in order to achieve the best final performance.
We ablate over which layer to pool over to generate the slot tokens. The patch tokens are discarded after the pooling layer, and only the slot tokens are further processed by the additional Transformer encoder layers. As expected, it is desirable to use all image encoder layers process both slot and patch tokens. Additionally, we also study the impact of slot pooling method, and observe that adding additional single-heade headed soft attention and Gumbel-max attention outperformed by simply using the slot tokens directly.
Slot Pooling Layer and Method:
8

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation on ACRE compositionality (comp), systematicity (sys), and L.I.D (iid) splits.
slots
comp
sys
iid
1
91.75%
90.34%
90.96%
2
90.82%
88.21%
88.73%
4
93.03%
92.36% %
92.13%
8
95.54%
86.18%
88.97%
64
90.45%
80.07%
90.82%
Table 4: Benchmark results on CATER (static).
Method
Object-centric
Object superv.
Top-1 Acc.
Top-5 Acc.
OPNet (Shamsian et al. 2020
74.8% %
Hopper Zhou et al. 2021)
73.2%
93.8%
ALOE base Ding ct al.
202
X
60.5%
84.5%
+LOE++ Ding et al.
2021
X
74.0%
90.4%
Random Init. R3D LSTM
X
3.3%
18.0%
X
60.2%
81.8%
R3D + NL LSTM
X
x
46.2%
66.9%
MAE (He et 2022
X
27.1%
47.8%
VideoMAE (Tong et al. 2022
X
X
63.7%
82.8%
IS-CL (ours)
X
69.5%
88.3%
4.3 COMPARISON TO THE STATE-OF-THE-AA
Table compares the result of IS-CL against other state-of-the-ane art models on CATER snitch local- ization. We also compare IS-CL on ACRE against other existing models in Table 5 We pretrain MAE and VideoMAE ourselves on the same pretraining dataset and searched for their correspond- ing optimal hyperparameters. We observe that the spacetime ViVIT used by VideoMAE leads to collapsed training, and modified it to use factorized encoder. Other results are cited from the pub- lished results. IS-CL achieves the best performance among the approaches that do not dependent on explicit object-cen abject-centric representation, it also achieves overall state-of-the performance on the comp and iid splits of ACRE.
Table 5: Results on ACRE compositionality (comp), systematicity (sys), and L.I.D. (iid) splits.
Method
Object-centric
Object superv.
comp
sys
iid
NS-OPT Zhang et al.
2021
69.04%
67.44%
66.29%
ALOE
Ding et al
2021
X
91.76 %
93.90%
Random Init.
X
X
38.78% %
38.57%
38.67%
CNN-BERT (Zang al.
2021
X
X
43.79%
39.93%
43.56%
MAE He al. 2022
80.27%
76.32%
80.81%
VideoMAE long et al. 2022
X
78.85%
71.69%
77.14%
JS-CL
X
93.03%
92.36%
92.13%
CONCLUSION AND FUTURE WORK
In this work we propose the implicit symbolic concept learner (IS-CL) framework, which trains neural network end-to-end to solve complex visual reasoning tasks, without explicitly constructing an object-centric representation. IS-CL learns such implicit symbolic representations as slot embed- dings in pretraining step through self-supervised video reconstruction objective via masking. We observe the exciting results that the learned representation behave like their symbolic counterparts, when measured compositional generalization performance CATER and ACRE benchmarks. Future work includes probing experiments to understand the information encoded by the slot tokens, and applying IS-CL to large-scale natural image and video datasets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34, 2021.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204. 141992 2022.
Jacob Andreas. Measuring compositionality in representation learning.
arXiv preprint
arXiv:1902.07181, 2019.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. ICLR, 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint axXiv2108.0025. 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, volume 33, pp. 1877 1901, 2020.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint 1901.11900.113330 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for un- supervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 149, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B Tenenbaum, and Chuang Gan. Comphy: Compositional physical reasoning of objects and events from videos. arXiv preprint urXiv:2205.0106 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18Xx:/811000480, 2018.
David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt Botvinick. Attention over learned object embeddings enables complex visual reasoning. Advances in Neural Information Processing Systems, 34, 2021.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp- 1422-1430, 2015.
10

---

Under review as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders spa- tiotemporal learners. arXiv preprint arXiv:2205.091 2022.
Jerome Feldman. The neural binding problem (s). Cognitive neurodynamics, 7(1):1-11, 2013.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cans are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv: 1t:::/881.122111 2018.
Golnaz Ghiasi, Barret Zoph, Ekin D Cubuk, Quoc V Le, and Tsung- Yi Lin. Multi-task self-training for learning general representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8856 8865, 2021.
Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and tem- poral reasoning. arXiv preprint arXiv: arXiv:1910.04744 2019.
Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Mísra. Omnivore: A single model for many visual modalities. I Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16102-16112, 2022.
Alison Gopnik and David M Sobel. Detecting blickets: How young children use information about novel causal powers in categorization and induction. Child development, 71 1055:1205-1 2000.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem artificial neural networks. arXiv preprint arXiv:2012.052008 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9729 9738-. 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked toencoders are scalable vision learners. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Katherine Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. Advances in Neural Information Processing Systems, 33:19000- 19015, 2020.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference on Machine Learning. pp. 4651-4664 PMLR. 2021.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
11

---

Under review as a conference paper at ICLR 2023
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482-7491, 2018.
lasonas Kokkinos. Ubernet: Training universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition, pp. 6129-6138, 2017.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sen to-sequeneee recurrent networks. In International conference on machine learning, pp. 2873-2882. PMLR, 2018.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer back- bones for object detection. ECCV, 2022.
Valerii Likhosherstov, Anurag Amah, Krzysztof Choromanski, Mario Lucie, Yi Tay, Adrian Weller, and Mostafa Dehghani. Polyvit: Co-training vision transformers on images, videos and audio. arXiv preprint arXiv:2111.12911 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in Neural Information Processing Systems, 33:1 1525-1188188, 2020.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv: 12584, 2019.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11453-11464 2021.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.2391 3/distill.0000..
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 107 2018
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue. Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Adina L Roskies. The binding problem. Neuron, 24(1):7-9, 1999.
Aviv Shamsian, Ofri Kleinfeld, Amir Globerson, and Gal Chechik. Learning object permanence from video. In European Conference on Computer Vision, pp. 35-50. Springer, 2020.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: joint model for video and language representation learning. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 7464-7473, 2019.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203. /2602, 2022.
12

---

Under review as a conference paper at ICLR 2023
Anne Treisman. The binding problem. Current opinion in newrobiology, 6(2):171-178, 1996.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
Chen Wei, Haoqi Fan, Saining Xic, Chao- Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14668-14678, 2022.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong
Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18134-18144 2022.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv: 19X0001910:01422 2019.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE con- ference on computer vision and puttern recognition, pp. 3712-3722, 2018
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6720-6731. 6731, 2019.
Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu. Acre: Abstract causal reasoning beyond covariation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10643-10653, 2021.
Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Is an object-centric video representation beneficial for transfer? arXiv preprint arXiv:2207.10075. 2022.
Richard Zhang, Phillip Isola, and Alexci A Efros. Colorful image colorization. In European confer- on computer vision, pp. 649-666. Springer, 2016.
Honglu Zhou, Asim Kadav, Farley Lai, Alexandru Niculescu-Mizil. Martin Renqiang Min, Mub- basir Kapadia, and Hans Peter Graf. Hopper: Multi-hop transformer for spatiotemporal reasoning. arXiv preprint arXiv:2103.10: 10574, 2021.
13


Reviews:
[{"review_id": "xmT9T3EIROd", "reviewer": "Reviewer_YXhg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an end-to-end implicit symbolic representation learning framework for visual reasoning tasks. It wisely adopts slot tokens for its bottleneck information properties, masked autoencoding objective, and transformers to learn implicit representations in a self-supervised way. The learned implicit representation can later be applied to solve specific visual reasoning tasks with proper head fine-tuning with target data. Such a learning framework has a very advantage: it does not require specific object abstraction (e.g., detection mask) and thus serves more general purposes. Results on two common benchmarks (CATER and ACRE) show consistent improvements achieved by the proposed framework. ", "summary_of_the_review": "In general, the reviewer believes the proposed approach takes a step towards visual reasoning with deep learning, while the analysis is intuitive and probably lacks some empirical analysis for strong support.\n\n--------------\nAfter discussing with AC and all other reviewers, we generally believe the current manuscript lacks some thorough analysis of the proposed framework, especially for 1 slot token works pretty well in many scenarios. I encourage the author to take all the points raised in the review in their revision and also keep improving the performance in various benchmarks. Reviewer ejDU lists the key points we discussed in our meeting, and hope them can help contribute to a better revision. \n\nI appreciate the efforts made by the authors. ", "strength_and_weaknesses": "The studied direction is important, and the proposed method took a further step toward a general solution to visual reasoning. Figure 1 clearly demonstrates the difference and advantages between the proposed method and the previous approaches: the proposed framework can perform visual reasoning with neural networks and non-predefined implicit tokens. All the used components: slot token, MAE objective, and transformers (encoder, temporal) are reasonable. Overall the paper is easy to follow.\n\nExtensive experiments are conducted in CATER and ACRE datasets and show the proposed framework consistently outperforms the previous approaches. Ablation studies are conducted thoroughly around masking ratio, total frame numbers, context frame numbers, slot token numbers, slot pool layer, and slot pool methods.\n\nMany analyses and explanations in the current manuscript are intuitive and without supporting empirical evidence. For example, in \"Number of Slot Tokens\" section, experiments found 1 works the best in the CATER benchmark, and the author explained why: \"the model need only maintain an accurate prediction of where the snitch actually or potentially is\". If we add golden snitches up to 2/3/4 (if possible, or in other sims), will the best performance achieve by 2/3/4 implicit slot accordingly? or the best performance is still achieved by 1 slot.\n\nWhat will the performance change if we set the slot token number to 100? will the proposed framework lose the \"representational bottleneck\" properties and lead to drastic performance drops? \n\nIn the transfer learning setting, the multi-class classification formulation for the goal of both CATER and ACRE tasks encodes strong human priors. The tested visual reasoning tasks are actually solved with specific designs for specific tasks in the end, while the studied implicit representations are uniform, and thus, the contribution did not weak a lot.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kb3sZmNTpJ", "reviewer": "Reviewer_ejDU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the framework \"Implicit Symbolic Concept Learner\" (IS-CL), a transformer-based architecture that is capable of reasoning about videos. The framework follows a pretraining-transfer-learning pipeline. That is, the model is first pretrained on a collection of unlabelled videos (by the MAE objective), and then finetuned on the target task (e.g., predicting the location of a certain object).", "summary_of_the_review": "Overall I think this paper is a good contribution to the community. However, there are still a few issues that can be addressed by the authors before publication.", "strength_and_weaknesses": "The paper presentation is clear and well-connected with existing works such as NS-CL and ALOE (Figure 1). There are a few places where the authors connect their work with Slot Attention, which has greatly helped me to understand the connections and differences between the proposed framework and others. The proposed framework is also relatively straightforward, with clear motivation. Experimental results have shown the success of the proposed framework. Furthermore, the ablation studies seem very adequate.\n\nTalking about weaknesses, I will start with a few conceptual ones.\n\nFirst, the idea of \"concept learning,\" following the notations from Mao et al, primarily refers to the correspondence between linguistic units (words, phrases) and visual representations (e.g., red, cubic, left-of, etc.). This seems to be different from the definition of \"concepts\" or \"implicit symbolic concept\" in the paper, which, in my understanding, refers to \"implicitly defined objects.\"\n\nSecond, when we talk about \"visual concepts,\", they are usually more \"abstract\" than \"pixel reconstruction.\" However, since the overall training paradigm (pretraining time) is to reconstruct the pixels, I don't think there is evidence that the model is capable of discovering \"symbolic concepts,\" e.g., colors, shapes, etc., from the pertaining. With all that, I am wondering if the title/model should be better phrased as \"implicit object learning\" or similar phrases?\n\nHere're a few technical questions.\n\nFirst, I am wondering why the authors have not tried visual question-answering benchmarks such as CLEVRER, especially given that based on the ablation study (Table 2a), CATER does not require an \"implicit symbolic\" representation (slot =1 then basically it's just a per-image representation). It seems that the framework can be easily extended to that, as in ALOE. And I think extending the framework to that can significantly improve the completeness of the paper.\n\nSecond, in Table 4, it seems that the model still underperforms several baselines. Although authors may argue that those better-performing algorithms are \"object-centric\" I think it is still completely reasonable to compare IS-CL with ALOE, because ALOE also does not use any object-detection labels during training. Again, I think adding a new benchmark that showcases the advantage of the proposed method will be ideal.\n\nThird, more of a clarification question, regarding the comparison between VideoMAE and IS-CL. Is the only difference that you are using a \"single-token\" embedding for each frame whereas VideoMAE uses all visual tokens? Because these two models have very similar training objectives and similar architectures.\n\nFourth, is there any way that we can visualize the learned \"implicit slot tokens?\" For example, can you visualize the attention maps?\n\nFinally, you mentioned that you have changed the slot-attention-style encoding with a customized transformer-style encoding. Do you have any ablation studies on that?\n\nClairification questions:\n\n1. Page 5:  allowing the slots to attend not only to raw visual inputs, but also to the encoded patch-level representation. Can you be more specific about this?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "W8C6eaOsDD", "reviewer": "Reviewer_Cwxs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A self-supervised training over video is proposed for learning compositional scene representations. The paper explores the approach on the CATER and ACRE datasets (video versions of CLEVR), and show its benefits in the domain of visual reasoning and question answering over these videos.  ", "summary_of_the_review": "The paper shows a new approach in a growing relatively under-explored domain. Multiple experiments are presented on two datasets, the paper is easy to follow and has good visualizations, but the main issues are 1) the bit limited novelty of the idea in light of the ALOE paper that explored self-supervised transformer based approaches for both the dataset explored here and an additional one (see details above), and 2) The fact that the paper doesn’t explore other datasets beyond the CLEVR family. I therefore overall recommend weak rejection at this point and recommend exploring how to extend the approach for additional datasets, which may lead also to new modeling ideas that could strengthen the technical contribution of the paper.", "strength_and_weaknesses": "**Strengths**: \n* **Self-supervised**: The approach is self-supervised and so doesn’t require annotations like alternative approaches for building compositional scene representations such as object detectors etc.\n* **dynamics for object-centered representations**: The approach encourages temporal dynamics to be captured in the representations, by using a self-supervised approach over videos. This is an important and still underexplored signal in the domain of self-supervised compositional learning that could greatly help discover objects. \n* **Compositional Generalization**: The approach explores and show good results on compositional and systematic generalization, which gives good indication that the representations learned indeed disentangle the dimensions needed for the downstream reasoning tasks.\n* **Extensive set of experiments on CLEVR-based data**: see quality point in the question below.\n\n**Weaknesses**:\n* **Experiments on synthetic datasets only**: The model is explored on CLEVR-based datasets only. While CLEVR-based data is definitely a great and suitable for exploring the presented approach, I would strongly encourage the authors to explore the approach either on new synthetic multi-object datasets with greater realism and diversity.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "129CAPg3MAw", "reviewer": "Reviewer_LKo8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for compositional visual reasoning in videos, based on recent advances in self-supervised pretraining. The method first trains a spacial-temporal transformer that reconstructs the video frames under the masked autoencoder paradigm, then performs reasoning via transfer learning. The authors claim and demonstrate via extensive experimental study on the CATER and ACRE datasets, that a generalizable compact representation is learnt, superior to other approaches based on object-centric representation. Ablation studies are conducted.", "summary_of_the_review": "This paper is an interesting paper that applies state-of-the-art self-supervised pretraining to compositional visual reasoning. Despite the reasonable results achieved on two challenging datasets, it is unfortunate that the paper does not provide a deeper look into the learned representation in order to back the claimed novelty. There is definitely a lot of room for improvement here. My score is borderline reject.\n", "strength_and_weaknesses": "Strength:\n\n1. This paper proposes a powerful end-to-end network for compositional visual reasoning, and achieved nice results on two datasets that were regarded as (and designed to be) challenging for end-to-end systems. \n\n2. The paper also demonstrates self-supervised pre-training can lead to useful representations for visual reasoning, which sheds light to further research.\n\n3. The paper is well written and backed by rich experimental results.\n\nWeakness\n\n1. The paper claims that the model learns \"implicit symbolic representation\" without a clear definition and elaboration. What is the difference between \"symbolic representation\" vs. a regular latent representation learned by other self-supervised model? Moreover, there isn't sufficient study on the representation itself other than transfer learning results to back these claims. How does the same architecture perform without the representation?\n\n2. The experimental results have not established, decisive evidence that end-to-end method is superior to object-centric representations. Specifically, under same supervision, the model does not significantly outperform ALOE and ALOE++. The paper also does not provide evaluation on the CLEVRER dataset, a video reasoning benchmark where object ALOE performs nicely.\n\n3. Minor: The data flow in figures goes upwards instead of downwards, which might cause unnecessary confusions to readers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject