SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yeongbin Kim* KAIST
Gautam Singh* Rutgers University
Junyeong Park KAIST
Caglar Gulcehre† EPFL & Google DeepMind
Sungjin Ahn\* KAIST
Systematic compositionality, or the ability to adapt to novel situations by creating a mental model of the world using reusable pieces of knowledge, remains a significant challenge in machine learning. While there has been considerable progress in the language domain, efforts towards systematic visual imagination, or envisioning the dynamical implications of a visual observation, are in their infancy. We introduce the Systematic Visual Imagination Benchmark (SVIB), the first benchmark designed to address this problem head-on. SVIB offers a novel framework for a minimal world modeling problem, where models are evaluated based on their ability to generate one-step image-to-image transformations under a latent world dynamics. The framework provides benefits such as the possibility to jointly optimize for systematic perception and imagination, a range of difficulty levels, and the ability to control the fraction of possible factor combinations used during training. We provide a comprehensive evaluation of various baseline models on SVIB, offering insight into the current state-of-the-art in systematic visual imagination. We hope that this benchmark will help advance visual systematic compositionality. https://systematic-visual-imagination.github.io/
Constructing a mental model of the world, known as a world model, in a composable way is a crucial aspect of human intelligence [40, 12, 72]. This ability enables humans to adapt to novel situations and problems in a zero-shot manner by envisioning the possible future [79, 7]. Studies in neuroscience and cognitive science suggest that the key to this ability is the process of acquiring abstract, conceptual, and reusable pieces of knowledge from past experiences and applying them in new configurations to comprehend a novel situation [30, 15, 76, 7]. For instance, a person who understands the implications of a scene involving a "big dog" and a "small cat", e.g., how they may interact and what the subsequent scene may look like, can also reasonably understand the implications of an unfamiliar scenario involving a "big cat" and a "small dog". While this capability, termed systematic compositionality, is fundamental to human intelligence, how neural networks can acquire such an ability remains one of the grand challenges in machine learning [63, 6, 36].
The notion of systematic compositionality originates from the fields of philosophy of language and linguistics [95, 30]. Language inherently provides a compositional representation using token
This work was partly done while C.G. was at Google DeepMind. C.G. is currently affiliated with EPFL.
$\ddagger$  Correspondence to sungjin.ahn@kaist.ac.kr.
This is the official project page. It provides links to the benchmark and the code.
structures, such as words or characters, which simplifies addressing compositional systematicity. As a result, the AI community has also made efforts to address this problem predominantly in the language domain [22, 63, 59, 88, 67, 2, 57, 28, 87, 106]. One notable milestone driving progress in this field recently is the development of the SCAN benchmark [63] posing a sequence-to-sequence translation task. The authors have demonstrated that RNNs fail catastrophically at test time when presented with a compositionally novel input, i.e., an unknown composition of known concepts.
However, the problem is even more elusive when it comes to imagining the dynamical implications of a visual observation, a problem referred to in this work as the compositional or systematic visual imagination. One reason for this is that, unlike language, images do not naturally provide such a token-based compositional representation, making the problem significantly more challenging. To tackle this problem, one must not only learn how to utilize compositional representations for systematic composition but also obtain such representations from unstructured, complex, high-dimensional pixels. Another reason is the lack of an appropriate benchmark to directly address it. Although several prior works have explored related problems, none have tackled it head-on.
Specifically, the prior research related to systematic visual imagination can be grouped into three categories. The first group [83, 105] tackles this issue by providing language alongside images, e.g., Visual Question Answering (VQA) [54, 6, 16, 5, 38], thus pursuing an indirect task that bypasses the challenge of learning compositional representation by leveraging the inherent token-based compositional nature of language. The second group [85, 13, 46, 116, 74, 109] also focuses on an indirect task, i.e., obtaining disentangled representations. The underlying hypothesis is that a disentangled representation such as those from variational autoencoders [60, 44] or object-centric representations [68, 17, 66, 53, 107, 89, 91, 90, 86] will naturally lead to a solution to the systematic generalization problem as well. However, it restricts exploring potential solutions not relying on disentanglement [108], and recent studies suggest that disentanglement may not necessarily lead to systematic generalization [74, 109]. The third group focuses on visual reasoning [19, 103, 80, 114, 4] while ignoring the problem of visual perceptual systematic generalization, essential for zero-shot problem-solving. For instance, the two closest visual reasoning benchmarks to ours, ARC [19] and Sort-of-ARC [4], exhibit these limitations. ARC lacks suitability for evaluating systematic generalization due to its non-procedural generation and lack of access to underlying component factors and rules. Sort-of-ARC, on the other hand, emphasizes inferring the underlying rule composition from a few-shot support set while ignoring the zero-shot systematic visual imagination ability—a cornerstone of our proposed benchmark.
In this paper, we introduce a new benchmark called the Systematic Visual Imagination Benchmark (SVIB), the first benchmark to rigorously address the compositional visual imagination problem. The SVIB poses the task as a minimal world modeling problem: one-step image-to-image generation. The objective is to generate the subsequent scene image from the current one. The underlying world dynamics operate as a relational function of object-centric factors. The benchmark presents a subset of possible combinations of the visual primitives during training, and during testing, exclusively presents compositionally novel input images to assess a model's systematic imagination ability.
SVIB offers numerous benefits for studying the compositional visual imagination problem. Our proposed task framework provides a means to jointly optimize for systematic perception and imagination, instead of optimizing an indirect objective such as disentanglement or object segmentation quality. Unlike many works in disentanglement literature focusing on a single-object scene [46, 74, 109, 98], SVIB incorporates multi-object scenes where each object is constructed via composition rules in terms of well-defined factors like color, shape, and size. SVIB also provides various difficulty levels, accommodating visual complexity from simplistic 2D images to realistically textured 3D scene images, and varying complexity of the world dynamics. Additionally, SVIB provides control over the fraction of possible factor combinations used in training, thus serving as a systematic generalization difficulty meter. The simplicity of one-step generation allows us to concentrate on the systematic imagination ability, sidestepping the computational demands of long-range generation abilities that video generation tasks often entail.
Our paper makes two main contributions: First, we propose the first benchmark to foster research on the systematic or compositional visual imagination problem, providing several unique benefits as outlined above. Second, we conduct comprehensive empirical evaluations of various baseline models on SVIB, shedding light on the current state-of-the-art capabilities in this field.
Figure 1: Compositional Visual World in SVIB: Our benchmark provides 2-frame videos of an underlying compositional visual world. The scenes contain two objects, each composed using intraobject factors such as color, shape, size, material, etc. In this figure, we show episodes following the Shape-Swap rule i.e., the input state is transformed to the target state by swapping the shapes of the two objects. We illustrate episodes from three tasks having different visual complexity levels.
2 SVIB: Systematic Visual Imagination Benchmark
In this section, we describe the Systematic Visual Imagination Benchmark or SVIB that we propose. The benchmark is designed to train models to perform visual imagination and measure their systematic generalization ability. Our benchmark provides a catalog of 12 visual imagination tasks<sup>5</sup>. Each task contains 64K episodes for training a model and 8K systematically out-of-distribution episodes for testing the model. Each episode is a two-frame video as illustrated in Fig. 1. We denote the two frames as the input image x and the target image y. Using the training episodes, a model must learn to map the input image x to the target image y, with a focus on accurately predicting the target images for the systematically out-of-distribution input images in the testing episodes.
2.1 Compositional Visual World
For each task, the episodes are generated from an underlying compositional visual world defined by 1) a visual vocabulary—a library of visual primitives that act as building blocks for constructing the scenes, and 2) a rule that determines the mapping from the input scene to the target scene.
Scene Compositionality and Visual Vocabulary. In SVIB, each scene is composed of two objects. The objects are composed of intra-object factors such as color, shape, size, etc. These factors take their values from a collection of visual primitives called a visual vocabulary. For example, if an object can be described by its color, shape, and size then we can define a visual vocabulary containing colors {green, blue, magenta, orange}, shapes {circle, triangle, square, star}, and sizes {tiny, small, medium, large}. Under a given vocabulary, we refer to any composition of factors that completely describes an object's appearance as a combination. Given our example vocabulary, two possible combinations are: blue-circle-tiny and orange-triangle-large. By placing the objects defined by these combinations inside a scene, we can construct an example scene containing a blue-circle-tiny at position (0.1, 0.4) and a orange-triangle-large at position (0.7, 0.1).
Rule. A rule is a function that transforms the input scene state to the target scene state. In SVIB, the rule is fixed for all episodes within a task. The rule modifies specific factor values of both objects in the scene. The rule is symmetric, i.e., the same rule is executed on both objects. An example of a rule in SVIB is the Shape-Swap rule that swaps the shapes of the two objects to transform the input scene state to the target scene state. In SVIB, we identify two axes of rule complexity and our tasks span these two complexity axes. The first axis is the number of modified factors per object. We call a rule that modifies only a single factor per object as a single rule. If a rule modifies multiple factors per object, we call it a multiple rule. The second axis is the number of parents i.e., the number of factors that determine the modified factor value. If only one parent is involved per factor, we call the rule an
See our project page at https://systematic-visual- imagination.github.io.
Rule Category	Rule Description	SVIB-dSprites	SVIB-CLEVR	SVIB-CLEVRTex
input	target	input	target	input	target
Single Atomic	other[shape]→self[shape]
Single Non-Atomic	other[shape]→self[shape]
Multiple Atomic	self[shape]→self=color]
other=color]→self[size]
Multiple Non-Atomic	self[shape]→self=color]
other=color]→self.size]
Figure 2: SVIB Tasks and the Transformation Rules. In this illustration, we show sample episodes of all 12 tasks in our benchmark. For each instance, we show the input and target images. We describe the underlying rule that governs the transformation of the input image to the target image. In the rule description diagrams, each factor value is considered to be an integer index in its corresponding vocabulary. A direct arrow indicates a simple assignment operation and a  $\oplus$  symbol indicates a summation operation. All assignments are performed modulo the vocabulary size of the target factor. As the rules are applied symmetrically to both objects, we describe the rules with respect to the sel.f object while other denotes the other object.
atomic rule else we call it a non-atomic rule. With these two axes, we define 4 rule categories with increasing complexity: single atomic, multiple atomic, single non-atomic, and multiple non-atomic. For example, the aforementioned Shape-Swap rule can be categorized as single atomic because only one factor (i.e., shape) is modified per object whose new value depends on only one parent, i.e., the shape of the other object.
2.2 Systematic Training and Testing Splits
In this section, we describe how we construct the training and the testing episodes of a task. As a benchmark for studying systematic generalization, our task episodes satisfy the following three conditions: 1) each primitive in the visual vocabulary is shown individually in the training episodes, 2) a subset of combinations is reserved for testing and 3) the fraction of combinations exposed during training can be controlled, offering a control knob to adjust the difficulty of generalization. To satisfy these, we proceed as follows.
Core Combinations and Testing Combinations. We first construct a set of core combinations—the smallest set that contains each primitive in the visual vocabulary at least once. For instance, for our previous example of a visual vocabulary, we can define the set of core combinations as follows: {green-circle-tiny, blue-square-small, magenta-square-medium, orange-star-large}. Next, from all the remaining combinations of primitives, we reserve  $20\%$  of the combinations as testing combinations. For more details, see Appendix D.2.
The  $\alpha$ -Rating and Training Combinations. Each training split in our benchmark is associated with an  $\alpha$ -rating. To create a training split having a specific  $\alpha$ -rating, we randomly select an  $\alpha$  fraction of the combinations that remain after setting aside the core and the testing combinations. These selected combinations are then added to the core combinations to obtain the set of training combinations. The  $\alpha$ -rating acts as a control knob over how difficult it is to generalize for a model trained on this split. A higher  $\alpha$ -rating corresponds to exposing more combinations in the training split, thereby providing an easier generalization setting. Similarly, a low  $\alpha$ -rating corresponds to a more difficult generalization setting. In Figure 3, we provide an illustration.
Generating the Episodes. To sample an episode given the training combinations, we randomly sample two combinations and position the objects randomly to create an input scene. Next, the task
Figure 3: The  $\alpha$ -Rating of Training Splits in SVIB. Left: In this illustration, we showcase a library of 4 shape primitives and 4 color primitives that result in a total of 16 possible combinations or objects. Right: In training episodes, we present each shape and color primitive individually, however, we do not show all possible combinations. To control the ease of solving the generalization task, we define an  $\alpha$ -rating of the training split, the fraction of the combinations presented during training. A higher  $\alpha$ -rating corresponds to more combinations shown in the training split. In testing episodes, we present only the held-out combinations.
rule is applied to the input scene to obtain the target scene. The input and the target scenes are rendered to generate the two frames of an episode. By sampling episodes repeatedly, we obtain the complete training split. The testing combinations are used to generate the testing split analogously.
2.3 Contents of the Benchmark
Tasks. In SVIB, we provide 12 tasks organized into 3 subsets based on the visual complexity of their underlying visual world: 1) SVIB-dSprites is based on visually simple 2D scenes; 2) SVIB-CLEVR is based on visually simple 3D scenes; and 3) SVIB-CLEVRTex is based on 3D scenes with complex textures. This gradation in visual complexity is illustrated in Figure 1. Within each of these three subsets, we provide 4 tasks corresponding to 4 rules with increasing rule complexity: Single Atomic (S-A), Single Non-Atomic (S-NA), Multiple Atomic (M-A), and Multiple Non-Atomic (M-NA). These 4 rules are illustrated in Figure 2. For their formal definitions, see Appendix D.1.1. For each rule, we provide 3 training splits with distinct  $\alpha$ -ratings denoted as Easy split ( $\alpha = 0.6$ ), Medium split ( $\alpha = 0.4$ ) and Hard split ( $\alpha = 0.2$ ) and a common testing split. Each training split contains 64K episodes and each testing split contains 8K episodes. For each episode, we provide the input image, the target image, the ground-truth scene descriptions, and the ground-truth object masks. In Figure 6, we provide the complete directory structure of the benchmark.
Omni-Composition Datasets. In SVIB, we provide an omni-composition dataset for each of the 3 visual worlds i.e., SVIB-dSprites, SVIB-CLEVR, and SVIB-CLEVRTex. An omni-composition dataset is a dataset containing unpaired images that capture all possible combinations of primitives under the visual vocabulary of its visual world. The use of omni-composition datasets for solving the benchmark tasks is optional.
3 Metrics
MSE. A natural metric for measuring the systematic generalization performance of a model is the mean squared error (MSE) between predicted and target images on the testing set episodes (denoted as  $\mathrm{MSE}^{\mathrm{OOD}}$ ). We can also compute how much worse the OOD MSE is relative to the in-distribution MSE via the systematic generalization gap:  $\log \mathrm{MSE}^{\mathrm{OOD}} - \log \mathrm{MSE}^{\mathrm{ID}}$ , where  $\mathrm{MSE}^{\mathrm{ID}}$  denotes the in-distribution MSE.
LPIPS. A key limitation of MSE is that it can overly focus on low-level errors, e.g., small deviations that do not significantly impact human perception. Therefore we recommend reporting LPIPS [115], an error metric shown to better correspond to human judgments. LPIPS works by taking two images as input, extracting their features via a pre-trained image classification model, and computing a weighted distance between the extracted features. In our experiments, we use the official implementation<sup>6</sup>.
<sup>6</sup>https://github.com/richzhang/PerceptualSimilarity
In this section, we describe the baselines that we evaluate in our experiments. These were selected considering two main implications of our benchmark: evaluating systematic generalization in vision models and world models. We evaluate two categories of baselines: 1) Image-to-Image Models, and 2) State-Space Models. Additionally, we evaluate an Oracle model to obtain the best-case performance and to set a milestone of success on the benchmark tasks.
4.1 Image-to-Image Models
In this category, we consider end-to-end neural network models that take the input image  $\mathbf{x}$  and try to predict the target image  $\mathbf{y}$ . Within the model, the input image is mapped to an intermediate hidden representation  $\mathbf{e}$  which is used to generate the prediction  $\hat{\mathbf{y}}$ . This can be described as follows:
$$
\mathbf {e} = \operatorname {E n c o d e r} _ {\theta} (\mathbf {x}) \quad \Longrightarrow \quad \hat {\mathbf {y}} = \operatorname {D e c o d e r} _ {\gamma} (\mathbf {e})
$$
We test two variants that capture the two extremes of how expressive the intermediate representation is. The first variant employs a CNN as the encoder to map the input image to single-vector representation  $\mathbf{e}$ , thus creating a bottleneck of low expressiveness. We denote this baseline as Image-to-Image CNN or I2I-CNN. The second variant employs a Vision Transformer or ViT as the encoder and produces a representation that is a collection of multiple vectors—as many as the patches in the image [25]. We denote this baseline as Image-to-Image ViT or I2I-ViT.
Oracle. To set a milestone of success on our benchmark, we construct a baseline where instead of employing an image encoder to obtain the representation  $\mathbf{e}$ , we directly stack the vector representations of the ground truth scene factors and provide it to the decoder. For more details, see Appendix E.3.
4.2 State-Space Models
State-Space Models (or SSMs) are a category of models that first encode the input image  $\mathbf{x}$  to a latent representation  $\mathbf{z}_{\mathbf{x}}$  and then apply a latent-level dynamics model to predict the target scene latent.
$$
\mathbf {z} _ {\mathbf {x}} = \operatorname {E n c o d e r} _ {\phi} (\mathbf {x}) \quad \Longrightarrow \quad \hat {\mathbf {z}} _ {\mathbf {y}} = \operatorname {D y n a m i c s} _ {\theta} (\mathbf {z} _ {\mathbf {x}}) \quad \Longrightarrow \quad \hat {\mathbf {y}} = \operatorname {P r o b e} _ {\gamma} (\hat {\mathbf {z}} _ {\mathbf {y}}),
$$
where  $\mathbf{z}_{\mathbf{x}}$ ,  $\hat{\mathbf{z}}_{\mathbf{y}} \in \mathcal{Z}$ . We test two variants that employ two different kinds of latent structures. The first variant adopts Variational Auto-Encoding or VAE for obtaining latent scene representations—a popular representation method for world modeling [60, 42, 41]. The encoded latent is a single vector where factors of scene variation are expected to be disentangled and captured by individual dimensions of the vector. We denote this variant as SSM-VAE. The second variant adopts Slot Attention Video or SAVi—a popular object-centric representation method [68, 61, 107]. SAVi represents each frame by a set of  $N$  object vectors, also known as slots. Furthermore, the slot order is preserved across frames, facilitating the training of a slot-level dynamics model. We denote this variant as SSM-Slot. Finally, since it is not possible to compute the MSE or LPIPS performance from the predicted latent, we train a probe  $\mathrm{Probe}_{\gamma}(\cdot)$  that takes the predicted latent  $\hat{\mathbf{z}}_{\mathbf{y}}$  and decodes it to generate the target image  $\hat{\mathbf{y}}$ . To ensure a fair performance comparison, we keep the probe's architecture to be the same as that of the decoder used in the Image-to-Image baselines and the Oracle. For more details, see Appendix E.4.
Systematic Generalization in Language. In the language domain, a substantial number of studies, including SCAN [63], have examined systematic generalization through sequence-to-sequence tasks that translate different sequence forms [22, 59, 88, 67, 2, 57, 28, 87, 106]. Some other benchmarks like gSCAN [83] and ReaSCAN [105] utilize additional inputs such as images and videos alongside language [81, 92, 69]. In Visual Question Answering or VQA, studies test systematicity by providing images with unseen object combinations [54, 6] or presenting questions in unseen formats [16, 5, 38]. However, all of these either test systematic generalization in language or utilize language inputs when testing visual compositionality.
Representation Learning. Another group of works questions specific representation learning methodologies and whether they support systematic generalization. One line of representation methods seeks disentangled single-vector representations [44, 58, 62]. However, studies that investigate it either do