SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We first compare our proposed IV-CL to traditional supervised pretraining on both detection and classification tasks. For classification, we consider the same ViT-B visual encoder trained on ImageNet-21K [15] and JFT [52]. For object detection, we consider an in-domain object detection benchmark dataset called LA-CATER [49]. LA-CATER matches the visual appearance of CATER; it was created to study the benefit of modeling object permanence and provides frame-level bounding box annotations for all visible and occluded objects. We validated the correctness of our object detector on the COCO benchmark, which achieves comparable performance to the original Pix2Seq implementation. On the LA-CATER validation set, we observe  $82.4\%$  average precision (AP) at an IOU threshold of  $50\%$ . Whereas one might expect almost perfect performance on such a synthetic environment, this can be explained by the inherent properties of the dataset; frame-level object detection on LA-CATER also evaluates the detection of occluded and invisible objects, which is indeterminable when given only single, static images as inputs. We also consider a classification pretraining baseline to count the number of unique objects in LA-CATER frames.
We note three remarkable trends when inspecting the results in Table 1. First, we observe that none of the models pretrained with supervision outperforms their self-supervised counterpart. Instead, their performance on both CATER and ACRE fall behind IV-CL by large margins. Second, when
Table 1: Self-supervised visual pretraining vs. supervised pretraining. We compare our proposed pretraining with traditional supervised classification or detection pretraining.
Objective	Pretrain data	CATER	ACRE (comp)
Random Init.	-	3.34%	38.78%
Detection	LA-CATER	56.64%	67.27%
Classification	LA-CATER	41.48%	64.78%
Classification	ImageNet-21k	55.58%	60.73%
Classification	JFT	54.07%	48.32%
IV-CL	CATER	70.14 (±0.59)%	93.27 (±0.22)%
comparing the detection and classification objectives, we observe that the detection pretraining outperforms classification pretraining significantly. This can be potentially explained by the domain gap between natural image datasets and CLEVR-style datasets, or that object detection objective encourages the learning of object-centric representations in the slot tokens. To better understand this, we perform addition ablations by replacing the object detection dataset with COCO [39], which is a natural image dataset. We observe similar transfer learning performance as LA-CATER pretraining. Additionally, we perform a probing experiment where we ask the object detection decoder to make predictions with a single randomly sampled slot token. We empirically observe that each token appears to focus on one or a small subsets of the objects in the scene, and different tokens are complementary to each other. Both observations indicate that the stronger performance of object detection pretraining is likely due to the "object-centric" objective itself. Finally, we observe a counterexample of the "scaling law": larger scale classification pretraining (JFT) leads to significantly worse performance than smaller scale pretraining (ImageNet-21k).
4.3 Visualizations of the Learned Slots
To help understand what visual concepts are implicitly captured by IV-CL, we visualize the attention heatmaps from each learned slot token back to the image pixels. This is implemented with the attention rollout technique [1]. Figure 3 shows examples of the attention heatmaps after (a) self-supervised pretraining on CATER, and after (b) finetuning for visual reasoning on ACRE.
Figure 3: Visualization of 4 slots of an IV-CL model after pretraining on CATER (left) and finetuning on ACRE (right). Each heatmap is generated by attention rollout [1] to the input pixels. A brighter color indicates higher attention weight.
We observe two general patterns by inspecting the pretrained slot heatmaps: first, a subset of the heatmaps exhibit object-centric behavior, as in Figure 3(a). Each slot tends to focus on an individual object, and most of the objects in an image are covered by combining the heatmaps from all four slots. However, we also observe that sometimes the slots are not completely disentangled with respect to individual objects, which indicates that the implicit representations obtained after IV-CL pretraining do not learn perfectly disentangled visual concepts, and further finetuning is necessary to achieve compositional generalization on visual reasoning. We then inspect the heatmaps after finetuning for visual reasoning on ACRE in Figure 3(b). We observe some slots model relationships among objects
Table 2: CATER Pretraining with different mask ratios, context sizes, and frame lengths.
(a) Mask ratio
Ratio	Acc.
37.5%	70.14%
12.5%	66.35%
50%	66.57%
87.5%	61.94%
(b) Context size
Size	Acc.
8	70.14%
0	65.35%
4	67.47%
16	64.34%
(c) Frame length
Length	Acc.
32	70.14%
8	62.28%
16	66.63%
64	68.25%
Table 3: CATER Pretraining with different number of slots, and pooling strategies
(a) Number of slots
# slots	Acc.
1	70.14%
2	66.52%
8	64.45%
(b) Where to pool
Layer	Acc.
11	70.14%
5	55.80%
9	68.86%
(c) How to pool
Method	Acc.
Slice	70.14%
Soft	64.23%
Hard	65.90%
and the platform, and some focus on individual objects. Intuitively, both types of information are needed to solve the ACRE benchmark. Finally, we visualized the attention heatmap of a ImageNet-21k pretrained model after finetuning on ACRE. We observe that the heatmaps often "collapse" on a small subset of the same objects, which coincides with its lower reasoning performance.
Next, we ablate our key design choices. We present our ablation study on CATER in Table 2.
Masking ratio: Contrary to the large masking ratio (75%) employed in vanilla MAE [27], we found that the optimal masking ratio was 37.5% on downstream CATER accuracy. This is perhaps due to the fact that CATER is designed to test "compositional generalization", and so the spatial context provides less information than in natural images and video.
Number of Total Frames and Context Frames: We also study the impact of the number of frames IV-CL is pretrained on, and find the best performance on 32 frames. Fixing the total number of pretraining frames, we then ablate over the number of context frames, which are the frames from which slot representations are generated. When no context frame is used, we essentially utilize only patch-level representations to perform reconstruction with the temporal transformer (simulating a per-frame MAE followed by a temporal transformer). We find that the best performance is achieved with 8 context frames, which balances the number of slot representations with patch-level representations.
Number of Slot Tokens: Another useful ablation is on the impact of the number of slots used for CATER and ACRE. In CATER, we find that only 1 slot token per frame is enough to solve the reasoning task. We believe that this may be due to how the reasoning objective of CATER is designed: to successfully perform snitch localization, the model need only maintain an accurate prediction of where the snitch actually or potentially is, and can ignore more detailed representation of other objects in the scene. Under the hypothesis that the slot tokens represent symbols, perhaps the singular slot token is enough to contain the snitch location. On the other hand, when ablating over the number of tokens for the ACRE task (see Appendix), we find that a higher number of tokens is beneficial for reasoning performance. This can potentially be explained by the need to model multiple objects across evidence frames in order to solve the final query; under our belief that slot tokens are encoding symbols, multiple may be needed in order to achieve the best final performance.
Slot Pooling Layer and Method: We ablate over which layer to pool over to generate the slot tokens. The patch tokens are discarded after the pooling layer, and only the slot tokens are further processed by the additional Transformer encoder layers. As expected, it is desirable to use all image encoder layers to process both slot and patch tokens. Additionally, we also study the impact of slot pooling method, and observe that adding additional single-headed soft attention and Gumbel-max attention are outperformed by simply using the slot tokens directly.
Table 4: Results on CATER (static). IV-CL performs the best among non-object-centric methods, and performs competitively with methods with object-supervision.
Method	Object-centric	Object superv.	Top-1 Acc. (%)	Top-5 Acc. (%)
OPNet [49]	✓	✓	74.8	-
Hopper [70]	✓	✓	73.2	93.8
ALOE [17]	✓	✗	70.6	93.0
Random Init.	✗	✗	3.3	18.0
MAE (Image) [27]	✗	✗	27.1	47.8
MAE (Video)	✗	✗	63.7	82.8
IV-CL (ours)	✗	✗	70.1 ± 0.6	88.3 ± 0.2
Table 5: Results on ACRE compositionality, systematicity, and I.I.D. splits. IV-CL performs the best among all methods on the compositionality split, and performs competitively on other splits.
Method	Object-centric	Object superv.	comp (%)	sys (%)	iid (%)
NS-OPT [66]	✓	✓	69.04	67.44	66.29
ALOE [17]	✓	X	91.76	93.90	-
Random Init.	X	X	38.78	38.57	38.67
MAE (Image) [27]	X	X	80.27	76.32	80.81
MAE (Video)	X	X	78.85	71.69	77.14
IV-CL (ours)	X	X	93.27 ± 0.22	92.64 ± 0.30	92.98 ± 0.80
4.5 Comparison with State-of-the-Art
We compare our IV-CL framework with previously published results. As most of the prior work require explicit object detection and are not end-to-end trained, we reimplement an image-based MAE [27] and a video-based MAE [53] baseline and analyze the impact of inductive biases (using slot tokens or not) as well as pretraining objectives (predictive coding given compressed context, or autoencoding the original inputs) on the reasoning performance. Our reimplementation of image and video MAEs achieve very similar performances on their original benchmarks. However, for video-based MAE, we observe that the "un-factorized" backbone leads to training collapse on CATER. We hence adjust the backbone to be "factorized" as illustrated in Figure 2. We follow the same pretraining and hyperparameter selection procedures as for IV-CL.
Table 4 compares the result of IV-CL against other state-of-the-art models on CATER. We also compare IV-CL on ACRE against other existing models in Table 5. We cite the comparable results reported by the original authors when available. IV-CL achieves the best performance among the approaches that do not depend on explicit object-centric representation, and overall state-of-the-art performance on ACRE.
4.6 Transfer Learning to RAVEN
We explore generalization to a visually different reasoning benchmark, RAVEN [65]. Inspired by Raven's Progressive Matrices (RPM), its goal is to evaluate a machine learning model's structural, relational, and analogical reasoning capabilities. The reasoning task is to determine which of eight candidate geometrical figures naturally follow the patterned sequence of eight context figures. We explore all seven reasoning scenarios and perform finetuning on all training and validation examples (56,000 examples). The pretraining and finetuning hyperparameters exactly match those for ACRE, but the model now takes in 16 images as input (8 for context, 8 for answers). We report generalization performance on RAVEN in Table 6. We observe that the pretrained representation is generalizable, as IV-CL achieves competitive performance on RAVEN [65] with the same pretrained model and finetuning hyperparameters as ACRE, despite the different visual appearances across the datasets.
Table 6: Transfer learning results on RAVEN. We follow the same pretrained representation and finetuning hyperparameters as for ACRE.
Method	Average	Center	2×2 Grid	3×3 Grid	L-R	U-D	O-IC	O-IG
LSTM	13.1	13.2	14.1	13.7	12.8	12.4	12.2	13.0
ResNet + DRT [65]	59.6	58.1	46.5	50.4	65.8	67.1	69.1	60.1
CoPINet [67]	91.4	95.1	77.5	78.9	99.1	99.7	98.5	91.4
SCL [58]	91.6	98.1	91.0	82.5	96.8	96.5	96.0	80.1
IV-CL (ours)	92.5	98.4	82.6	78.4	96.6	97.2	99.0	95.4
4.7 Generalization to Real Videos
Finally, we attempt to answer the question: Would our proposed IV-CL framework work on real videos? We consider the Something-Else benchmark [43], which consists of short videos capturing the interactions between human hands and different objects. This benchmark focuses on relational reasoning, especially on compositional generalization across different object categories. We consider the base split and the "compositional" split. The base split contains 112,397 training videos and 12,467 validation videos, across 88 categories. The compositional split contains 54,919 training videos and 57,876 validation videos, across 174 categories. Each category corresponds to a fine-grained activity that requires spatiotemporal relation reasoning. The compositional split is designed to include disjoint object types for each category between the training set and the validation set.
Table 7: Performance Evaluation on Something-Else. We consider the base and compositional splits. *: Uses groundtruth box annotations during evaluation.
Model	Split	Object Supervision	Top-1 Acc. (%)	Top-5 Acc. (%)
STRG, STIN+OIE+NL [43]	Base	✓	78.1	94.5
ORViT [31]*	Base	✓	87.1	97.6
IV-CL (Ours)	Base	X	79.1	95.7
STRG, STIN+OIE+NL [43]	Comp	✓	56.2	81.3
ORViT [31]*	Comp	✓	69.7	90.1
IV-CL (Ours)	Comp	X	59.6	85.6
Due to the large domain gap between CATER and Something-Else videos, we choose to perform pretraining directly on the corresponding training splits of the Something-Else benchmark. We use the same pretraining and finetuning hyper parameters as in ACRE, except that we use 16 frames sampled at stride size of 2 during finetuning. During both pretraining and finetuning, we apply the standard video data augmentation techniques as used by prior work (e.g. [5]). In Table 7, we observe that our method generalizes well to real videos, and it achieves competitive performance compared to methods that use annotated boxes during training (STRG, STIN+OIE+NL) and evaluation (ORViT).
In this work we demonstrate that competitive visual reasoning can be achieved in a general-purpose end-to-end neural network, with the help of self-supervised visual pretraining. Our proposed implicit visual concept learner (IV-CL) framework leverages a Transformer encoder to "compress" visual inputs into slot tokens, and is trained with a self-supervised video reconstruction objective. Quantitative and qualitative evaluations confirm the effectiveness of IV-CL on CATER and ACRE visual reasoning benchmarks, when compared to supervised visual pretraining and neuro-symbolic approaches. A limitation of our work is that evaluations are performed purely on synthetic reasoning tasks. We believe extending evaluation to large-scale natural video reasoning benchmarks, building a joint model for visual recognition and reasoning, and exploring how to incorporate explicit object-centric knowledge when such knowledge is available are interesting future directions to pursue.
Acknowledgements: Part of the research was conducted while C.L. worked as a student researcher at Google. C.S. and C.L. are in part supported by research grants from Honda Research Institute, Meta AI, Samsung Advanced Institute of Technology, and a Richard B. Salomon Faculty Research Award.
[65] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In CVPR, 2019.
[66] Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu. Acre: Abstract causal reasoning beyond covariation. In CVPR, 2021.
[67] Chi Zhang, Baoxiong Jia, Feng Gao, Yixin Zhu, Hongjing Lu, and Song-Chun Zhu. Learning perceptual inference by contrasting. In NeurIPS, 2019.
[68] Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Is an object-centric video representation beneficial for transfer? In ACCV, 2022.
[69] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
[70] Honglu Zhou, Asim Kadav, Farley Lai, Alexandru Niculescu-Mizil, Martin Renqiang Min, Mubbasir Kapadia, and Hans Peter Graf. Hopper: Multi-hop transformer for spatiotemporal reasoning. In ICLR, 2021.
6 Additional Experimental Details
Transfer Learning Framework. In Figure 2, we visualized our proposed self-supervised pretraining framework. Once the representation network has been pretrained, we discard the image decoder and only use the ViT-B image encoder, along with the pretrained temporal transformer. An illustration of the transfer learning process is shown in Figure A1.
Figure A1: An illustration of the transfer learning process. Both the ViT-B image encoder and the temporal transformer are transferred to downstream visual reasoning tasks to encode video inputs. Unlike pretraining, only the slot tokens are provided as inputs to the temporal transformer.
Figure A2: Illustration of the CATER (top) and ACRE (bottom) benchmarks.
Illustration of the Benchmarks. In Figure A2, we provide the illustrations of the CATER benchmark and the ACRE benchmark, respectively. The CATER benchmark features a special golden ball, called the "snitch", and the associated reasoning task is to determine the snitch's position at the final frame despite occlusions. Object locations in the CATER dataset are denoted by positions on an invisible 6-by-6 grid; therefore, in essence, the CATER task boils down to a 36-way classification problem. The CATER dataset features a split where the camera is statically fixed to a particular angle and position throughout the videos, as well as a moving camera split where the viewing angle is able to change over time. We use the static split for evaluation. Each video has 300 frames.
The ACRE benchmark is inspired by the research on developmental psychology: Given a few context demonstrations of different object combinations, as well as the resulting effect, young children have been shown to successfully infer which objects contain the "Blicketness" property, and which combinations would cause the platform to play music. ACRE explicitly evaluates four kinds of reasoning capabilities: direct, indirect, screened-off, and backward-blocking. Having the query frame be a combination that was explicitly provided in the context frames tests a model's direct reasoning ability. Indirect reasoning can be tested by a novel query combination, the effect of which
requires understanding multiple context frames to deduce. In screen-off questions, the model must understand that as long as a singular Bicket object is placed on the platform, the entire combination would cause it to light up. In backward-blocking questions, the model must recognize when the effect of a query combination cannot be determined from the provided context frames. Furthermore, ACRE features three different dataset splits to test model generalization: Independent and Identically Distributed (I.I.D.), compositionality (comp), systematicity (sys). In the compositionality split, the shape-material-color combinations of the CLEVR objects in the test set are not seen before in the train split; therefore, the model must learn to generalize across object attributes. In the systematicity split, the evidence frames of the train split contain three lit up examples, whereas the evidence frames of the test split contain four.
Number of the Slot Tokens. In Table A1, we provide ablation experiment on the impact of the number of slot tokens for the reasoning performance on all splits. Unlike CATER, whose goal is to infer the position of a single object, the "snitch", the ACRE benchmark requires reasoning over combinations of objects, and their relationship with the platform. As a result, we generally observe that more slot tokens are needed to achieve optimal performance. We observe that the performance starts to saturate given four or eight slots.
Table A1: ACRE # tokens. We show results on compositionality (comp), systematicity (sys), and I.I.D. (iid) splits.
# slots	comp	sys	iid
1	91.75%	90.34%	90.96%
2	90.82%	88.21%	88.73%
4	93.27%	92.64%	92.98%
8	95.54%	86.18%	88.97%
64	90.45%	80.07%	90.82%
(a) Pretrained on CATER
Figure A3: Visualizations of the Slot Tokens. The top row corresponds to the attention heatmaps from the slot tokens after pretraining on CATER, and the bottom row corresponds to the heatmaps after finetuning on ACRE.
(b) Finetuned on ACRE
Visualizations of Slot Tokens. Figure A3 provides additional visualizations of the slot token attention heatmaps after pretraining on CATER, and finetuning on ACRE, respectively. We follow the same attention rollout technique as in Figure 3. For ACRE, we show the example when the platform is visible (context information) on bottom left, and when the platform is invisible (question) on the bottom right. We observe a consistent trend that a subset of the heatmaps exhibit object-centric behavior, especially before finetuning on ACRE. After finetuning, we observe that some slots remain focusing on individual objects, while the others attempt to model the relationships among different objects and the platform.
Our MAE baselines are pretrained with the same hyper parameters (e.g. optimization and mask ratio) as IV-CL, which we have observed to be optimal based on the validation set performance. The image encoders for all methods are based on ViT-B, hence the total model sizes are comparable.