SlowGuess/NeurIPS_2023_OCR huggingface dataset:
not quantify (e.g., via  $\alpha$ -rating) the degree to which OOD inputs systematically differ from the training [98, 109] or tackle only visually simplistic and single-object scenes [98, 109, 97, 74, 73], unlike ours. Furthermore, [45], relying on RL, lacks the simplicity of our one-step prediction framework. Higgins et al. [46] test the importance of disentanglement in a symbol-to-image task, however, the learning is partly supervised by symbol labels. Zhao et al. [116] study generative models (e.g., VAE) and whether the learned distributions capture OOD regions of the observation space, however, they lack a focus on perception unlike ours. Another line of representation methods seeks object-centric representations [35, 27, 66, 34, 26, 68, 89, 86]. Although these are considered promising to support OOD generalization [36, 90], very few studies investigate this potential. Such studies tend to lack well-defined and systematic factor recombination for OOD testing [23, 24, 111, 70, 102] or rely on RL [24, 111, 70, 102], lacking the simplicity of one-step prediction.
Visual Reasoning. In the visual domain, several studies test reasoning abilities by inferring the underlying rule from a support set and then finding images from a candidate set that conform to that rule [10, 113, 48, 14, 47, 96, 49]. Alternatively, there are odd-one-out tasks that find the rule-violating image from an input image set without requiring a separate support set [112, 71] and tasks that categorize input images into several groups [29, 39, 99, 75, 52]. However, their learning setup is discriminative and not generative, unlike ours. Some reasoning tasks require learning to generate [19, 103, 80, 114], however, these tasks fall outside the scope of visual systematic generalization. Assouel et al. [4] improve upon the ARC benchmark [19] by properly defining elementary primitives and procedurally generating the dataset to make it suitable to study systematic generalization. However, the primitives and their combinations are concerned with scene dynamics and not the visual observation itself, unlike ours. Furthermore, both benchmarks [19, 4] use coarse and simplistic observations, whereas ours provide visually complex high-resolution images.
We test the five baselines as outlined in Section 4: Image-to-Image CNN, Image-to-Image ViT, SSM-VAE, SSM-Slot, and Oracle on the 12 benchmark tasks outlined in Section 2.3. In Figures 4, 10, and 11, we report the LPIPS, the MSE, and the systematic generalization gaps, respectively. Since LPIPS better reflects the human judgment of qualitative performance, we base our conclusions in this section on LPIPS. In our analysis, we consider Oracle's performance as a yardstick for task success. In Fig. 5 and 14, we show samples of the model predictions.
6.1 Effect of  $\alpha$  on Systematic Generalization
Performance at  $\alpha = 0.0$ . When  $\alpha = 0.0$ , the training split only includes the core combinations. In all tasks, we note that all baselines fail completely for  $\alpha = 0.0$ . This is supported by the qualitative results in Fig. 5, where most predicted images are garbled blobs. This can be expected since at  $\alpha = 0.0$ , the factors are highly correlated in the training split, making it difficult for the learner to infer the true causal parent in the rule. For instance, in the Shape-Swap task in SVIB-dSprites: the learner sees only the core combinations: {green-circle-tiny, blue-square-small, magenta-square-medium, orange-star-large} where every shape value is always shown with a specific color value and a specific size value, making it impossible (even for the Oracle) to determine whether the true causal parent is color, shape, or size.
Performance with Increasing  $\alpha$ . As  $\alpha$  is increased from 0.0, the difficulty of systematic generalization becomes less severe and we see a consistent downward slope in the error plots across baselines and tasks. This is also supported by the qualitative results in Fig. 5, where, with increasing  $\alpha$ , a greater number of baselines are able to improve their predictions. Since training splits with larger  $\alpha$  expose more combinations, there is more opportunity for the baselines to overcome learning spurious correlations and discover the compositional primitives and the causal rule to apply.
Performance at  $\alpha = 0.6$ . Our easiest training split corresponds to  $\alpha = 0.6$  which exposes the largest fraction of combinations during training. Yet, on most tasks, most baselines are not able to reach the performance of Oracle—although it does happen in some select cases. From this, we can say that our benchmark is largely unsolved with significant room for progress.
Performance of Oracle. We note that Oracle is the best performer as it is the only baseline that can solve the tasks as early as at  $\alpha = 0.2$ . In comparison to Oracle, other baselines show a much flatter
Figure 4: Systematic Generalization in Visual Imagination. We plot LPIPS with respect to  $\alpha$  for all benchmark tasks and baselines on the systematically out-of-distribution (OOD) test set. Lower is better. For SVIB-dSprites, SVIB-CLEVR, and SVIB-CLEVRTex, we evaluate the tasks: single atomic (S-A), single non-atomic (S-NA), multiple atomic (M-A), and multiple non-atomic (M-NA).
Figure 5: Qualitative Performance of the baselines on SVIB-dSprites and SVIB-CLEVR. We illustrate the predictions made by various baselines on the Shape-Swap task, where the task is to swap the shapes of the two objects in the input image. We show predictions on a test episode for various baselines for varying  $\alpha$  of the training split.
downward slope. This can be explained by considering that SVIB requires systematic perception i.e., learning to perceive the input in terms of useful tokens—explicitly or implicitly—while also generalizing systematically to OOD inputs. Oracle is designed with access to the ground-truth tokens directly and it does not need to learn systematic perception via training. On the contrary, other baselines carry the additional burden of learning systematic perception along with rule learning—contributing to their worse performance.
Performance on SVIB-CLEVRTex. The performance of baselines on SVIB-CLEVRTex requires a special mention as we see that the curves with respect to  $\alpha$  are almost flat and the slope is not
Table 1: LPIPS Performance on SVIB. This table summarizes the LPIPS performance results of various models evaluated across all 12 tasks of SVIB. The values are computed by averaging the results from four tasks for each difficulty level.
Models	SVIB-dSprites	SVIB-CLEVR	SVIB-CLEVRTex
Easy	Medium	Hard	Easy	Medium	Hard	Easy	Medium	Hard
I2I-CNN	0.0643	0.1251	0.2678	0.0593	0.1265	0.2181	0.2938	0.3014	0.3210
I2I-ViT	0.0084	0.0359	0.2326	0.0583	0.1263	0.2153	0.2529	0.2614	0.2950
SSM-VAE	0.1284	0.1515	0.2410	0.0763	0.0961	0.1884	0.3819	0.3789	0.3745
SSM-Slot	0.0534	0.0817	0.1834	0.0596	0.0533	0.1139	0.3429	0.3357	0.3590
Oracle	0.0006	0.0006	0.0064	0.0107	0.0141	0.0122	0.1414	0.1444	0.1404
as large as that in the other simpler environments i.e., SVIB-dSprites and SVIB-CLEVR. That is, the baselines struggle to systematically generalize in SVIB-CLEVRTex. A notable exception is Oracle which can solve the tasks starting at  $\alpha = 0.2$ . What distinguishes Oracle from other baselines is its systematic perception ability. As such, we can attribute the degraded performance of other baselines in SVIB-CLEVRTex to their poor systematic perception. This shows that higher visual complexity can make systematic perception more difficult in comparison to visually simple scenes of SVIB-dSprites and SVIB-CLEVR. This observation is also supported by the predictions visualized in Fig. 14. It is noteworthy that our proposed benchmark provides a unique opportunity to study systematic imagination in visually complex scenes as the existing image-to-image benchmarks provide visually toy-like scenes [19, 4].
6.2 Comparison of Baselines
We now compare the baselines in more detail. We draw our conclusions based on tasks with ratings of  $\alpha = 0.2$  and 0.4 since the tasks are too difficult for  $\alpha = 0.0$  and, in select cases, too easy for  $\alpha = 0.6$ .
Image-to-Image Models. In Image-to-Image models, we compare two encoders: CNN and ViT. In SVIB-dSprites, we note that ViT is generally superior to CNN. This is also evidenced in the qualitative results of SVIB-dSprites in Fig. 5 where ViT can correctly predict for  $\alpha \geq 0.4$  while CNN continues to predict the wrong object shapes on all  $\alpha$  values. It is somewhat surprising that ViT, which lacks any explicit design to encourage systematic generalization, can generalize at all. However, in SVIB-CLEVR and SVIB-CLEVRTex, the performances of CNN and ViT are comparable. This suggests that although ViT is more capable than CNN on SVIB-dSprites, the greater visual complexity of SVIB-CLEVR and SVIB-CLEVRTex can make ViT struggle similarly to CNN.
State-Space Models. Comparing SSM-VAE and SSM-Slot, we find that the SSM-Slot generally outperforms the SSM-VAE. This is further evidenced by the qualitative results of SVIB-dSprites in Fig. 5 where SSM-VAE predicts an incorrect shape even at  $\alpha = 0.6$  while SSM-Slot is able to predict correctly for  $\alpha \geq 0.4$ . However, both SSMs still remain significantly worse than the Oracle.
State-Space Modeling versus Image-to-Image Modeling. We compare our best-performing image-to-image model with our best-performing state-space model i.e., I2I-ViT versus SSM-Slot. In SVIB-dSprites, both models perform comparably. We believe this is due to the visual simplicity of SVIB-dSprites which allows ViT to learn systematic perception implicitly without any explicit inductive biases such as those imposed on the SSMs. In SVIB-CLEVR, SSM-Slot seems to generalize significantly better than I2I-ViT. This is also evidenced in the qualitative results in Fig. 5, where SSM-Slot can correctly generate the red monkey for  $\alpha \geq 0.4$  while I2I-ViT fails for all  $\alpha$  values. We hypothesize that since SSMs enforce an inductive bias on the latent, this can eventually help in learning systematic perception. In SVIB-CLEVRTex, both baselines fail completely. In fact, SSM-Slot fails worse than I2I-ViT. We think this is due to a familiar problem of the conventional Slot Attention-based models that they cannot extract objects properly in visually complex scenes [55, 91].
6.3 Summary of Experiment Results
We summarize the main experimental findings and pointers for making progress on our benchmark:
1. All baselines fail completely at  $\alpha = 0.0$ . Although most baselines demonstrate improved performance with increasing  $\alpha$ , only a select few can eventually solve some of the tasks and reach the performance of the Oracle. This highlights the challenging nature of our tasks and indicates a substantial opportunity for improvement.
2. Systematic perception is central to solving our tasks. However, visual complexity can make perception and, as a result, task-solving much more difficult. Our benchmark provides a unique opportunity to study systematic generalization along the axis of visual complexity.
3. Excluding the Oracle, in SVIB-dSprites and SVIB-CLEVR, ViT outperforms all other baselines on the Easy and Medium training splits but loses to the object-centric baseline SSM-Slot on the Hard training split. In SVIB-CLEVRTex, ViT outperforms all other baselines but significantly falls short of reaching the successful performance of the Oracle.
4. Models based on single-vector representations (e.g., I2I-CNN and SSM-VAE) tend to perform worse than their multi-vector counterparts (e.g., I2I-ViT and SSM-Slot).
We introduced the Systematic Visual Imagination Benchmark (SVIB)—a novel image-to-image generation benchmark that focuses on visual compositionality and systematic generalization. SVIB is procedurally generated and contains multi-object scenes, with each object compositionally constructed from well-defined visual primitives. The underlying image-to-image mapping rule is designed to require extracting the underlying visual factors. Our experiments showed that our benchmark is yet to be solved and highlights an important limitation of the current models. We also showed that systematic perception and visual complexity are important aspects of this problem. We hope that our benchmark will enable the development of more capable world modeling and perception approaches.
Limitations and Future Extensions. Our benchmark makes several simplifying choices to support faster model development and ease of analysis. However, this leaves several limitations and avenues for extending the benchmark. First, from a world modeling perspective, a richer benchmark may be constructed by introducing action-conditioning, stochasticity, longer episodes, occlusions, 3D viewpoints, etc. Second, our dynamics are fixed within a task. Our benchmark can be extended to have distinct dynamics in each episode constructed compositionally using dynamical primitives. Third, our benchmark can be extended by introducing greater realism e.g., greater visual complexity, more objects, more primitives, etc. Fourth, our dynamics involve symmetric rules and a future extension may introduce non-symmetric rules. Another avenue is to increase emphasis on relational abstractions. Lastly, future works may also consider investigating whether large pre-trained models (e.g., [43]) can provide necessary priors for systematic generalization.
This work is supported by Brain Pool Plus Program (No. 2021H1D3A2A03103645) through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT.
[1] Kelsey R Allen, Anton Bakhtin, Kevin Smith, Joshua B Tenenbaum, and Laurens van der Maaten. Ogre: An object-based generalization for reasoning environment. In NeurIPS Workshop on Object Representations for Learning and Reasoning, 2020.
[2] Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019.
[105] Zhengxuan Wu, Elisa Kreiss, Desmond C Ong, and Christopher Potts. Reascan: Compositional reasoning in language grounding. arXiv preprint arXiv:2109.08994, 2021.
[106] Zhengxuan Wu, Christopher D Manning, and Christopher Potts. Recogs: How incidental details of a logical form overshadow an evaluation of semantic interpretation. arXiv preprint arXiv:2303.13716, 2023.
[107] Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.05861, 2022.
[108] Sirui Xie, Ari S. Morcos, Song-Chun Zhu, and Ramakrishna Vedantam. Coat: Measuring object compositionality in emergent representations. In International Conference on Machine Learning, 2022.
[109] Zhenlin Xu, Marc Niethamme, and Colin Raffel. Compositional generalization in unsupervised compositional representation learning: A study on disentanglement and emergent language. arXiv preprint arXiv:2210.00482, 2022.
[110] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019.
[111] Jaesik Yoon, Yi-Fu Wu, Heechul Bae, and Sungjin Ahn. An investigation into pre-training object-centric representations for reinforcement learning. ArXiv, abs/2302.04419, 2023.
[112] Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre. A benchmark for compositional visual reasoning. arXiv preprint arXiv:2206.05379, 2022.
[113] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5317-5327, 2019.
[114] Chi Zhang, Baoxiong Jia, Song-Chun Zhu, and Yixin Zhu. Abstract spatial-temporal reasoning via probabilistic abduction and execution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9736-9746, 2021.
[115] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[116] Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon. Bias and generalization in deep generative models: An empirical study. Advances in Neural Information Processing Systems, 31, 2018.