Compositional-ARC: Assessing Systematic Generalization in Abstract Spatial Reasoning
Authors: Philipp Mondorf
Shijia Zhou
Monica Riedler
Barbara Plank
Venue: ICLR 2026 Poster (2026)
Keywords: Systematic Generalization
Abstract Spatial Reasoning
ARC
Meta-Learning for Compositionality
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: In this study, we extend meta-learning for compositionality to the domain of abstract spatial reasoning, demonstrating that neural networks can systematically generalize from known geometric transformations to novel combinations.

Abstract: Systematic generalization refers to the capacity to understand and generate novel combinations from known components. Despite recent progress by large language models (LLMs) across various domains, these models often fail to extend their knowledge to novel compositional scenarios, revealing notable limitations in systematic generalization. There has been an ongoing debate about whether neural networks possess the capacity for systematic generalization, with recent studies suggesting that meta-learning approaches designed for compositionality can significantly enhance this ability. However, these insights have largely been confined to linguistic problems, leaving their applicability to other tasks an open question. In this study, we extend meta-learning for compositionality to the domain of abstract spatial reasoning. To this end, we introduce $\textit{Compositional-ARC}\textemdash{}$a dataset designed to evaluate the capacity of models to systematically generalize from known geometric transformations (e.g., translation, rotation) of abstract two-dimensional objects to novel combinations of these transformations (e.g., translation+rotation). Our results show that a small transformer-based encoder-decoder model, trained via meta-learning for compositionality, can systematically generalize to previously unseen transformation compositions. Notably, despite having only 5.7M parameters, this model significantly outperforms state-of-the-art LLMs$\textemdash{}$including o3-mini, GPT-4o, and Gemini 2.0 Flash, which fail to exhibit similar systematic behavior$\textemdash{}$and performs on par with the winning model of the ARC prize 2024, an 8B-parameter LLM trained via test-time training. Our findings highlight the effectiveness of meta-learning in promoting systematicity beyond linguistic tasks, suggesting a promising direction toward more robust and generalizable models.

Published as a conference paper at ICLR 2026
COMPOSITIONAL-ARG, ASSESSING SYSTEMATIC GENERALIZATION IN ABSTRACT SPATIAL REASONING
Philipp Mondorf
Shijia Zhou Monica Riedler
Barbara Plank 1.2
MaiNLL, Center for Information and Language Processing, LMU Munich, Germany Munich Center for Machine Learning (MCML), Munich, Germany
{p. m.nndderf, zhou. shi shijia, menica... conica.riedler, riedler, b.plank)@lmu.c de
ABSTRACT
Systematic generalization refers to the capacity understand and generate novel combinations from known components. Despite recent progress by large language models (LLMs) across various domains, these models often fail to extend their knowledge to novel compositional scenarios, revealing notable limitations in sys- tematic generalization. There has been an ongoing debate about whether neural networks possess the capacity for systematic generalization, with recent studies suggesting that meta-learning approaches designed for compositionality can signif- icantly enhance this ability. However, these insights have largely been confined to linguistic problems, leaving their applicability to other tasks an open question. this study, extend meta-learning for compositionality to the domain of abstract spatial reasoning. To this end, we introduce Compositional-ARe signed to evaluate the capacity of models to systematically generalize from known geometric transformations (e.g., translation, rotation) of abstract two-dimensional objects to novel combinations of these transformations (e.g.. translation--rotaiion)) Our results show that small transformer-based encoder-decoder model, trained via meta- meta-learning for compositionality, can systematically generalize to previ- ously unseen transformation compositions. Notably, despite having only 5.7M parameters, this model significantly outperforms state-of-the-an L.MMs-incudding o3-mini, GPT-4o, and Gemini 2.0 Flash, which fail to exhibit similar systematic behavior- and performs on par with the winning model of the ARC prize 2024, 1B-paramettr LLM trained via test-time training. Our findings highlight the effectiveness of meta-learning in promoting systematicity beyond linguistic tasks, suggesting promising direction toward more robust and generalizable models.
dataset de-
INTRODUCTION
fundamental aspect of human cognition is the ability to systematically generalize from known components novel combinations (Marcus 2003 Lake et al. 2017). This capacity is particularly evident in language, where an infinite number of new sentences can be constructed and interpreted by extracting meaning from previously acquired expressions and rules (Chomsky 2002 Szabó 2012).
Similarly, our spatial perception relies on systematic generalization, enabling individuals to compose learned spatial principles into novel configurations (Zhou et al.) 2024; Dautriche & Chemla 2025).
For instance, once person understands how to translate and rotate an object, they can apply these transformations in combination irtnaattonttaansaattin have never encountered such a composed transformation before (Fife et al. 2019).
translating and rotating the object simultancously- even if they
Despite its central role in human cognition, systematic generalization remains a significant challenge in artificial intelligence (Lake & Baroni 2018 Loula et al. 2018 Hupkes et al. 2020). While large
language models have demonstrated notable progress across various domains (OpenAl 2024 Guo
al. 2025). they often fail to combine acquired knowledge in novel scenarios, demonstrating notable
difficulties with systematic generalization (Dziri et al. 2023 Ismayilzada et al., 2025 Gendron et al.
2024). The question of whether neural networks
achieve systematicity has been the subject of extensive debate (Fodor & Pylyshyn 1988 Brakel & Frank 2009) Calvo & Symons] 2014 inter
alia). Recent research by Lake & Baroni (2023) demonstrates that a raasformer-basee encoder-

---

Published as conference paper at ICLR 2026
Primilive Transforuations
Transformation Composition (level=1)
Transformation Composition (level=2)
Irx Uni
Coal
UMiza One
(a) Shaps based ITanstation dowa)
(d) supercokk ThunsarionRelllciii
Iaza Oldd
Ouces Drad
Sest Dod
Outza
lapa Crid
Depea Crd
(1) culor-hoccal (Rellection boriscontll
(c) disppeaccghhbrr (Tramelaiin
(g) stapeechh--nniille...
(Trnnsition Roffection-Estteeeeo
Ioqu Card
neighboe-hooo (Fistonsion
(A) (nonnnnnnigh (Resfaction Externiio)
Figure 1: A conceptual overview of the data in Compositionul-ARC. Primitive transformations refer basic geometric transformations (e.g., translation, reflection, extension) based on an object's (a) shape, (b) color, (c) proximity to neighboring object. Pairs of these indicators, such (d) shape+color (e) nhiph-nr or (f) color can be combined to form level-1 transfor- mation compositions. Finally, all three indicators can be combined to form level-2 transformation compositions, based on the object's (g)
decoder model, trained via meta- for compositionality (MLC), can achieve human-like systematic generalization in processing instructions expressed in a pseudolanguage. By training the model to combine basic units of pseudolanguage into novel sequences over stream of dynamically changing grammars, [ake Baroni (2023) show that this model can generalize previously unseen compositions of language (see Section 2b. While this approach presents promising direction for systematicity in neural networks, its applicability beyond linguistic contexts remains an open question.
In this study, we extend the MLC framework proposed by Lake & Baroni (2033) to the domain of abstract spatial reasoning. Inspired by the Abstraction and Reasoning Corpus (ARC) (Chollet) 2019), introduce Compositional-ARC
dataset for assessing systematic generalization in abstract spatial reasoning. Compositional-ARC presents examples of basic geometric transformations (e.g., translation, rotation) applied abstract two-dimensional objects and tests generalization to previously unseen compositions (e.g.. translation+rotttion;, see Figure T]. Using MLC, we train small encoder-decoder model on samples from Compositional-ARC and demonstrate that it can systematically generalize to unseen transformation compositions. To the best of our knowledge, this the first application MLC abstract spatial reasoning. In summary, our contributions are:
1. We introduce Compositional-ARCC novel dataset, inspired by ARC (Chollet 2019) that evaluates systematic generalization abstract spatial reasoning. The dataset includes examples of basic geometric transformations applied to abstract two-dimensional objects and tests generalization to unseen transformation compositions (see Figure !
2. We demonstrate that MLC enables transformer-based models to generalize unseen positions of geometric transformations, demonstrating its potential beyond linguistic tasks.
3. We show that a 77raamameterr mecoder-decoder model trained via MLC significantly outperforms saae--of-the-art general-purpose LLMMs such as 03-mini (OpenA] (025). GPT- 4o (Achiam al. 2023), and Gemini 2.0 Flash (DeepMind) 2024), which fail to exhibit comparable systematic behavior on Compositional-ARC.
4. We find that the same MLC model performs on par with the winning model the ARC Prize 2024, an 3B-parameter LLM trained via test-time training (Franzen et al.] 2024).
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND: META-LEARNING FOR COMPOSITIONALITY
When learning a new language, humans rely on their ability to recombine known words and expres- sions to interpret novel sentences (Chomsky et al. 1976; De Beule & Bergen 2006). For instance, someone who understands the meanings of "cats drink water" and "dogs like to play" will typically also understand the meanings of "dogs drink water" and "cats like play" (Hinzen et al. 2012).
Whether language models possess a comparable degree of systematicity remains an open question, as current models, including LLMs, still struggle with tests of systematic generalization (Ismay- ilzada et al. 2025 Dziri et al. 2023) To address these limitations, Lake & Baroni (0233) propose meta-learning for compositionality (MLC), a framework designed to model human- like systematic generalization in learning pseudolanguage instructions. Through a series of experiments, the authors
show that models trained via MILC can achieve levels of systematicity comparable to those of humans when interpreting previously unseen pseudolanguage inputs.
Task setup. their study, Lake & Baroni (2023) examine few-shot compositional tasks in which instructions, represented as sequences of pseudowords (e.g., "dax," "lug." "fep"), must be mapped to corresponding sequences of abstract symbols (see Figure 2 for an example). To understand the meaning of such instructions, interpretation grammar needs to be deduced from limited number of study examples. This grammar maps pseudowords to their symbolic representation through a set of compositional rewrite rules. For instance, if "dax" corresponds to a green circle, "dax fep" to three green circles, and "zup" to a red circle, then "zup fep" would denote three red circles. Importantly, the examples are designed be highly systematic, progressing from primitive mappings more complex compositions. The core challenge lies in the ability to generalize systematically, i.e., to reuse and combine components from the study examples (left side of Figure to generate correct outputs for novel query instructions (right side of Figure
Algorithmic approach. To achieve systematic generalization in the instruction-leane task, Lake & Baroni (2023) train transformer- based encoder-decoder model through meta-learning for compo- sitionality. The key idea is train the model on dataset of dynamically changing interpretation grammars, where the mappings from input sequences to output symbols differ across training sam- ples. This forces the model to rely on the information conveyed in the study examples to infer the appropriate grammar of given sample, rather than memorizing static input-output mappings across the dataset. This flexibility enables the model to adjust to novel scenarios governed by new sets of examples and rules. Moreover, the compositional structure of both study examples and queries encourages the model to internalize mechanisms for composing elements presented in the examples. After training the model over a set of 100,000 distinct interpretation grammars, it demonstrates the capacity to generalize to previously unseen instructions and grammars. For specific details regarding training procedures, we refer Appendix C.3 and the original paper (Lake & Baroni
2023
While Lake & Baroni (2023) also evaluate MLC on COGS (Kim & Linzen 2020) and SCAN LLkke & Baroni 2018), which test systematic lexical generalization novel word combinations, their
'For extended literature review on systematic generalization in LLMs, please refer Appendix A
Study instructions
Query Instructions
Primitives
Function 3
Target Responses
dax wif
lug kiki wif
zup fep
zup lug
dax kiki lug
zup kiki dax
Function
Function compositions
dax blicket zup
wif fep
lug fep kiki wif
zup fep kiki lug
dax lep
lug kiki wif fep
wif kiki dax blicket lug wif blicket dax kiki lug
Function
lug blicket wif
wif blicket dax
Figure 2: An example of the few-shot instruction learning task adapted from Lake & Baroni 2003)) Study instructions illustrate the mapping of pseudolanguage expressions abstract symbols.
3

---

Published as a conference paper at ICLR 2026
experiments are confined to the linguistic domain. In the following section, we propose Compositional- ARC to show how MLC can be extended to support systematic generalization in abstract spatial reasoning, demonstrating its potential beyond linguistic tasks.
3 METHOD
3.1
COMPOSITIONAL-ARC
To test systematicity in abstract spatial reasoning, we leverage the closure property of combined geometric transformations, where the composition of two valid transformations- -ucch as transla- tion, rotation, and siaection-yclloe another valid geometric transformation (Brannan et al. [001]) Drawing inspiration from the Abstraction and Reasoning Corpus (ARC) (Chollet (019)), we design task in which abstract objects, defined in a two-dimensional grid environment, are subjected to basic geometric transformations and their compositions (see Figure 1 for examples). We use fixed-size 10x x 10 grids, each of which can be represented as a two-dimensional array of integers, where different values correspond to distinct colors. We use integers from 0 to 9, with O denoting black background and the remaining integers mapping to unique colors (see Appendix B. for more details). Objects are defined based on color connectivity; that is, each object comprises a group of connected cells sharing the same color. Connectivity is determined by the Moore neighborhood (Bays) 2010), meaning that cells are considered connected if they are directly or diagonally adjacent. Each grid contains either one or two objects. A transformation is represented as pair of grids, with the input grid displaying the objects before, and the output grid showing them after the geometrie transformation. Each transformation affects only one of the objects in the grid. For example, in Figure Ia a single L-shaped yellow object is translated one step downward. In Figure Ic a square blue object in the bottom-right expands toward the neighboring top row. Objects never occlude another nor extend beyond the boundaries of the 10 x 10 grids.
We limit our dataset to five basic geometric transformations and their compositions: i) translations, ii) rotations, iii) reflections, iv) extensions, and v) color changes. For our experiments, we constrain the configurations of these transformations to establish controlled setup. Translations are limited to movements of one cell to the right or one cell downward. Rotations are restricted to 90 degrees clockwise counterclockwic around the top-left corner of the object. We consider horizontal and vertical reflections across the object's 's central axis. Extensions mean that the object grows in certain direction, and are limited to neighboring cells either leftward or upward. Color changes are restricted to changing the object's 's color to either red or orange. Experiments under more relaxed conditions are presented in Section 5.3 Detailed definitions of each transformation can be found in Appendix [B.2]
To signal which objects undergo which transformations, we consider three types of indicators: i) shape- based transformations, which affect objects of a particular shape; ii) color-based transformations, which affect all objects of specific color; and iii) neighbor-based transformations, where objects are transformed when a second, indicator object is present. For instance, in Figure all L-shaped objects (similar to the object in Figure Tab undergo one-step downward translation. All green objects undergo horizontal reflection, and any object sharing a grid with the gray diagonal object (c.g., as seen in Figure Tch expands into the neighboring top row. This indicator-based approach enables the definition of transformation compositions. For example, objects that are both L-shaped and green undergo one-step downward translation together with a horizontal reflection (see Figure Id for an example). We also define different levels of composition: level I combines two indicators (c.g., when an object matches the indicated shape and color, but lacks the proximity to a neighboring object, as illustrated in Figure Id) while level 2 combines all three indicators, specifying the object's 's shape, color, and proximity to an indicator object (see Figure
To test systematicity, we present few-shot examples of primitive transformations and their level- / compositions, and evaluate models on previously unseen level-2 compositions. For instance, in Figure 3] models are asked to infer the correct transformation for a previously unseen level-2 composition of indicators, given a set of 12 study examples illustrating primitive transformations and their level-1 compositions. Conceptually, our setup is similar to the few-shot compositional task introduced by Lake & Baroni (2023) (see Section 2). but it replaces the lexical interpretation grammar with visual interpretation grammar. Specifically. models need to infer which indicator maps to which transformation, and how to compose them to deduce the correct final transformation. For a detailed description of how we algorithmically generate dataset samples, please refer to Appendix B.3
4

---

Published as a conference paper at ICLR 2026
Study Examples
Query
Predictions
Primitive Transformations
Osd
Composition
(level-2)
Laana
Orpa
ed-minl
Liama (TTT)
Transformation Compositions (eeve=1)
Query Target
GPT-4o
Mistral
CLEW
Gemini
Mistral (TTT)
Figure 3: An episode from Compositional-ARC. Given set of study examples with primitive transformations and level-1 transformation compositions, models must predict the output grid for an unseen level-2 transformation composition. Visual grammar: shape - clockwise rotation, color translation to right, neighbor - leftward extension. Model predictions are presented to the right.
3.2 META-LEARNING FOR COMPOSITIONALITY ABSTRACT SPATIAL REASONING
To systematically generalize from known geometric transformations to previously unseen trans- formation compositions, we extend the meta-learning for compositionality (Lake & Baroni 2023) framework described in Section 2 As in the original MLC approach, we train a transformer based encoder-decodee model on dataset of dynamically changing interpretation grammars. However, instead of mapping pseudolinguistic instructions to sequences of abstract symbols, we consider a visual interpretation grammar that associates visual indicators (object shape, color, proximity indicator object) with specific geometric transformations, as described in Section B.I An episode in Compositionl--RC is defined along with query inputs for which the correct outputs must be inferred. For instance, the episode in Figure contains study examples: primitive transformations (two per indicator type) and six level-1 compositions (two per composition type). Given the study examples, the model is asked to predict output grids for previously unseen level-2 compositions. By training over series of episodes with changing visual interpretation grammars, the model needs to abstract and recombine information from examples in order to predict the correct query transformation composition, as cannot rely on fixed mappings from indicators to transformations.
set of study examples that illustrate the underlying grammar,
Encoding and positional embedding. Each episode is presented to the model as a sequence of onputt grid pairs (study examples), followed by a query input grid, for which the model must generate the corresponding output grid (see Figure 3 To encode the two-dimensional grids, we divide each 10 grid (Dosovitskiy et al. 202T). Each patch is mapped to unique embedding vector. Since each grid cell can take integer values from 0 to 9, 2 x patch can yield up to 10,000 distinct configurations, resulting in 10,000 possible embedding vectors. Two special tokens, and -, are introduced to mark the boundaries between study examples and the input-output grids, respectively. The decoder vocabulary comprises two additional tokens for the start and end of sequence (SOS and EOS). To encode positional information, we use standard learnable ID positional embeddings that capture the order of grid pairs, well second set of learnable 2D positional embeddings applied to grid
grid into 2 patches (left to right, top to bottom), yielding 25 patches per

---

Published as a conference paper at ICLR 2026
patches. These 2D embeddings are decomposed into separate row and column components, which are added to each patch embedding to capture two-dimensional spatial information.
Training procedure. The model is trained on large set of episodes, each defined by unique visual interpretation grammar. In each episode, the model is provided with sequence of study examples and tasked with predicting the output grid for a given input query (see Figure 3 Following Lake & Baroni (023), we include auxiliary copy task during training. in which the model must also reproduce the output grids of each study example. We employ a model with three layers each in the encoder and decoder, eight attention heads per layer, input and hidden embeddings of size 128, feedforward hidden size of 768, and GELU (Hendrycks & Gimpel 2016) activations. In total, the model has 5.7 million trainable parameters. To promote robustness in the decoder, we introduce minor perturbations by randomly altering the color of individual cells in the target output query with small probability (0.001). Unlike Lake & Baroni 2023), we do not incorporate systematic noise to model inductive biases observed in human learning. Further implementation details regarding the training procedure and hyperparameters can be found in Appendix C
EXPERIMENTAL SETUP
4.1 TASK SETUP
We consider two task setups in this work. The first, denoted "3-Shot," is standard few-shot learning task where models must generate an output grid for a query input that performs a level-2 transformation composition. This prediction is based on three examples illustrating the same level-2 transformation. A visual representation of this setup is provided in Figure i the Appendix. This task evaluates the model's ability to infer geometric transformations from limited set of examples.
The second setup, denoted as "Systematicity." focuses on compositional generalization and differs from the first in the type of few-shot examples presented. As mentioned in Section B.1 the idea is to test whether models can infer novel compositions from known geometric transformations. To this end, we replace the level-2 few-shot examples with set of primitive transformations plus level- / transformation compositions, and query the model to predict the previously unseen level-2 trans- formation composition, as illustrated in Figure 3 Specifically, we present six primitive transforma- tions-two examples for each indicator (shape-based, color-based, neighbor-based) and six level-1 transformation compositions, two examples for each level-1 indicator composition (shape+color shape+neighbor, color+
We generate 100,000 episodes, each comprising three few-shot examples for the "3-Shot" task, 12 systematic study examples for the "Systematicity" setup, and ten query input-output grid pairs demonstrating the final level-2 transformation composition. Each episode is characterized by a unique visual interpretation grammar. For instance, in one episode, yellow objects are translated downward by a single cell, while in another, yellow objects are reflected horizontally. To train our encoder-decoder model via MLC, we split the data into 82,908 training, 8,546 validation, and 8,546 test episodes. Importantly, the data splits are constructed such that the geometric transformations involved in the final query level-2 compositions differ between the training and evaluation sets. For instance, while the model is trained on basic transformations and a series of transformation compositions (e.g., +roplaationotttttonnnn reffaattiontrfectiionn reflection), it tested out-of-distribution on compositions not seen during training (e.g., translation votaaltiioooootooon dataset splits, please refer to Appendix B.4 especially Table
For comprehensive statistics on the
4.2 LARGE LANGUAGE MODELS
General-purpose LLMss. In addition to the model trained via MICC, we evaluate three state-of-the art pprposs L.MSs on the test set of our proposed dataset: 03-mini (low) (OpenAT) (025). GPT-4o (Achiam et al. 2023), and Gemini 2.0 Flash (DeepMind] (024).. To textually prompt the models for given episode, we represent grids two-dimensional arrays, consistent with prior work (Moskvichev et al. 2023). We also test a multimodal setup in which both an image of the study examples and the input query are provided alongside the text prompt. Due to financial constraints, each model is evaluated on a single test query for each of the 8.546 episodes in the test set. All textual and visual prompts, specific model versions, and decoding parameters are detailed in Appendix D.2
6

---

Published as a conference paper at ICLR 2026
Table 1:
Comparison of model performance across the different task setups. We report exact
match accuracy, color accuracy, and shape accuracy as described in Section 4.3
Model
Match Accuracy
Accuracy 1%
Shape Accuracy
GPT-4o
22.28
99.67
57,02
19.42
99.75
54.56
Gemini 2.0 Flash
30.08
99.92
52.34
3-Shot
Image
35.86
03-mini (low)
85.85
98.57
86.05
Mistral-NeMO-MMuiinn Saanitro--B-Full
95.71
99.85
96.78
MLC (ours)
99.92
100.00
99.92
GPT-4o
image
0.86
97.94
7.50
Gemini 2.0 Flash
2.66
99.68
12.81
Systematiicity
image
2.05
99.28
9.60
03 mini (low)
0.53
99.10
5.65
Llama--3.2-38 3.2-BBR-RB
2.54
test-fime training
73.70
100.00
86.88
Mistral-NeMO Minitrom Melrral-eMO-Miiitmm Siittrom------ 8B-Full
0.70
99.99
9.75
Test-fime traüning
78.20
100.00
88.26
MLC (ours)
Domain-specine LLMs. We further consider two LLMs specifically tailored ARC-style data: fine-tuned on the re-ARC dataset (Hodel (202) an extension of 1,000 additional generated examples per sample in ARC- and (ii) Mistral- NeMO-Miiittron-SB- -ull trained on a broad range of ARC-style data, including re-ARC, Concept-ARC, (Moskvichey et al. 2023) and
(i) 1aama-3.2-3B-Raaa
ARC-Heavy (Li al. 2025). These models were proposed by Franzen al. (2024) and placed 1st in the ARC prize 2024 Note that in addition to fine-tuning, these models use an ARCCcustomized tokenizer, extensive data augmentation during training and inference, generation procedure that leverages depth-first search to produce multiple solution candidates, and a refined candidate-selectinn step. The authors also employ test-time training (TTT), which further fine-tunes models on the few- grid pairs from the final test set. We use both models with their default parameters.
shot
For additional details, please refer to original paper (Franzen et al. 2024 or Appendix D.2
4.3 EVALUATION METRICS
To evaluate the quality of the generated output grids, three different metrics: i) exact match accuracy, ii) color accuracy, and iii) shape accuracy. Exact match accuracy requires that a prediction accurate only if every cell matches the target grid. Color accuracy checks whether predicted objects match target colors, ignoring shape and location. Shape accuracy checks whether predicted objects match target shapes, ignoring color and location. Formal definitions are provided in Appendix D.I
RESULTS
In T report the performance of the model trained via MICC, alongside the LLMS evaluate on the two task setups, as described in Section
Standard few-shot learning task. We begin by examining model performance on the "3-Shor" task, where models are given three input-output examples illustrating the final transformation composition (see Figure 5 in the Appendix). Despite this guidance and the relatively simple transformations involved, peneroe-purpose LLMs such as GPT-4o and Gemini 2.0 Flash struggle with the task: GPT-4o reaches an accuracy only 22.28%, while Gemini 2.0 Flash performs slightly better at 30.08%. The tong-chhii-of--houuh model o3-mini achieves modest accuracy of 64.04%. In

---

Published as a conference paper at ICLR 2026
contrast, domain domain-specific models such as Llama-3.2-33 -e,RRR and Mistral-NeMM--Mniiii
Full perform significantly better. While Llama-3.2-3B 3.2-3B-ReARC achieves an accuracy of 85.85%. reaches up 95.71%. Note that we do not employ test-time training in this setup, as it would contradict the out-of-distribution test setup described in Section 4.1 Notably, the 5.7M-parameter encoder-decoder model trained via MLC outperforms both general-purpar domain-specifte LLMs, with an accuracy of 99.92%, despite having only a fraction of the parameters. We further find that all models predict object color nearly perfectly, For GPT-4o and Gemini 2.0 Flash, we observe that shape accuracy is significantly higher than exact match accuracy. This discrepancy suggests that while these models are often able to predict the correct shape and color of an object, they frequently fail to accurately predict its final position. Interestingly, both models show lower accuracy when visual input is added to the textual prompt, likely due to modality alignment challenges (Masry et al. 2025 or limitations in leveraging the visual content for reasoning.
and
Systematicity task. In the "Systematicity" task, models are asked to infer the correct final transfor- mation composition from a set of study examples that represent more basic, decomposed transforma- tions (see Figure for an example). As shown in Table all general- purpose LLMs perform poorly on this task. For instance, GPT-4o achieves an accuracy of 0.99%, while Gemini 2.0 Flash reaches 2.66%. Interestingly, 03-mini, the best-performing forms worst in this setting, with an accuracy of only 0.53%. For the domain-specitf LLMs, we find that test-time training (TTT)-wher models are additionally fine-tuned on the study examples' input- output grid pairs of the test set si--ssnniiicaatlly improves performance. While Llama-3.2 2--3--ReAR--- achieves only 0.70% accuracy without TTT, performance increases 73.70% with TTT. Similarly, Saitraa-eMO-MMnnitee Fu-Fll''s accuracy increases from 0.70% to 78.20% with TTT. We hypoth- esize that training on the systematic study examples of the test. data (demonstrating primitive and level-/ transformations) teaches the models how to abstract and compose transformations for the final input query. We further find that the much smaller 5.7M-parameter MLC model performs on par with the domain-specific LLMs trained via TTT, slightly outperforming with an accuracy of 78.26%. Importantly, as described in Section 4.1 the MLC model has never seen the specific level-2 compositions of the test data during training. but instead optimized distinct set of transformation compositions (see data split for seed 1860; Table 7 in the Appendix). Consistent with our findings from the 3-shot learning task, models generally succeed in predicting the correct object colors. However, shape accuracy declines markedly. A qualitative example of the models' predictions is shown in Figure 3 with additional examples in Figures & 9 in the Appendix. The strong performance of the small MLC model highlights the effectiveness of this training strategy in promoting systematic generalization to novel transformation compositions. The model not only learns to infer visual interpretation grammar from a limited number of study examples but also generalizes to novel transformation compositions that it has never encountered during training.
parproe model on the "3-Shot" task, per-
5.1 CONSISTENCY ACROSS DATA SPLITS
To ensure that the strong performance of MLC, as reported in Table is not the result of a favorable data split, train and evaluate the model on three additional, independently generated data splits for each task configuration-rrsulle these data splits are provided in Table 7 in the Appendix. Table summarizes the average accuracy and corresponding standard deviation across all four splits. For the standard three-shot learning task,
in four distinct models per task setup. Detailed descriptions of
Table 2: Average accuracy and standard deviation across the four different data splits. For the systematicity task, we ablate different components of the training procedure assess their individual contributions and overall impact.
Model
Match Accuracy [S]]
Color Accuracy [%]]
Shape Accuracy [%
MLC (3-Shol)
98.78 :: 1.99
100,00 :: 0.00
98.79 1: 1.98
MLC (Systematicity)
86.73 + 6.03
99.36 : 0,70
87.55 + 5.45
copy task
69.05 1 9.23
99.43 1: 0.38
70.60 1 9.23
primitives
75.27 = 12.95
99,56 0.50
76.92 + 11.23
level-1 compositions
21.01 # 19.07
94.72 + 7.41
23.03 + 19.08
8

---

Published as a conference paper at ICLR 2026
MLC consistently achieves high aceuracy, with mean of 98.78% and a standard deviation of 1.99%. Similarly. for the systematicity task, the model demonstrates robust generalization, achieving an even higher average accuracy than the initial data split, with mean of 86.73%.
Ablation studies. To gain deeper insights into the factors influencing model performance, we conduct a series of ablation studies. First, we evaluate the impact of removing the auxiliary copy task from the training objective- a setup in which the model is trained not only to predict the output grid for a given imput query but also to reproduce the output grid of each study example (refer to Section (.2). Removing this auxiliary task results in notable decrease in accuracy from 86. to 69.05%. This decline underscores the importance the copy task in promoting systematic generalization, aligning with the findings of Lake & Baroni (2023). Subsequently, we assess the role of study examples in model performance. Removing primitive transformations from the study examples (see Figure 3 results in moderate reduction in performance, with average accuracy of 75.27%. This suggests that examples involving only level-/ transformation compositions are, to some extent, sufficient for allowing the model to generalize more complex level-2 compositions. However, removing level-/ transformation compositions leads to severe performance degradation, reducing accuracy to 21.01 %. We hypothesize that this is due to the increased complexity of composing three primitive operations directly into level-2 transformation, as opposed to building on intermediate level-1 I compositions.
5.2 ERROR ANALYSIS
To characterize model behavior on the systematicity task, we analyze the models' prediction errors. Figure 4 shows the relative frequency of common error types across models. We consider the following error categories: (i) Format errors (output not a valid 10 x 10 grid with cell values in 0,...,91 9); (ii) No Transformation (output identical input); (iii) Primitive (a primitive transformation is applied instead of the target level-2 composition); (iv) Level-/ (a level-1 composition is applied instead of the level-2 composition); (v) Invalid Position (correct color and shape, wrong position); (vi) Invalid Shape (correct color, incorrect shape); and (vii) Other (e.g., wrong number of objects, or objects with both incorrect shape and color). Models show distinet error profiles. General-purpose LLMs (GPT-4o, Gemini 2.0 Flash, 03-mini) most often predict incorrect shapes that do not match any primitive or level-1 outcome; for 03-mini, over 30% of errors involve applying a primitive instead of level-2 composition, and with image input more than 20% of GPT--4''s errors are format violations. I.amm- 3.2ma-3.2-3B---- 2-3B-ReARC mainly copies the input (no transformation), whereas Mistral- NeMO-Minitro- -BB--ullFll most often applies primitive instead of the target level-2 composition. After test-time training on the study examples (Section 4.2), errors of both domain-specific LLMs most often involve level-1 predictions. The MLC model rarely produces primitive or level-1 outputs; instead, it fails mainly by predicting an incorrect shape. Exact percentages by model and error type, and breakdown of primitive and level-1 errors, reported in Tables and 6 in the Appendix.
Gemini 2.0 Flash
Liama 3.2.3333..33...... 2--3B-RnARC
03 03-mini (low)
KZ
Image
GPT-4a
Mistrai-NeMo Www-eeeMinnttr-BB-....
MLC (ours)
test-time training
100
60
20
Format
No Tansformation
Primitive
Level-1
Invalld Posttion
Invalid Shape
Other
Figure 4: Error distribution by error category across models. Bars show the fraction of prediction errors assigned to each error category.

---

Published as a conference paper at ICLR 2026
5.3 INCREASING DATASET COMPLEXITY
So far, Compositional-ARG has been restricted to translations of one cell to the right downward; ii 90-degree clockwise or counterclockwise rotations; iii) horizontal and vertical reflections; iv) extensions neighboring cells leftward upward; and v) color changes to red or orange. We analyze whether the MLC model still systematically generalizes when we increase the variety of transformations, and therefore the dataset complexity. To this end, we introduce a new dataset that additionally allows translations of one or two cells in any direction (left, right, upward, downward), extensions to neighboring cells in any direction, and color changes red, orange. yellow, and green. We generate 100,000 episodes and split the data as described in Section 4.1 exact dataset statistics are given in Table in the Appendix. We then train an MLC model following the procedure in Section 3.2 Even on this more diverse dataset, the MLC model systematically generalizes to unseen transformation compositions, achieving an exact match accuracy of 88. 10% on the test set, a color accuracy of 99.83%, and a shape accuracy of 88.25%.
6 RELATED WORK
Meta-learning. Meta-learning aims to improve model's ability to adapt to novel tasks by leverag-
ing experience over multiple training episodes (Thrun & Prati 1998 Hospedales al. 2022). It has
been successfully applied to various tasks, such as few-shot learning (Mishra et al. 2018). continual learning (Javed & White, 2019; Lee et al. 2023 Irie et al. 2025). and reinforcement learning (Duan) et al. 2016 Wang et al.
et al. 2018). Related to our work, meta-learning has been used to improve systematic generalization Lake & Baroni 2018) showed that traditional sequence-to- sequence models struggle with compositional skills, but incorporating meta-learning can significantly improve performance (Lake) 2019; Conklin et al. 2021). Recent work argues that giving models the opportunity to practice skills via meta- meta-lcarning is crucial for addressing challenges such as systematic generalization, among others (Irie et al. 2025). Our method builds on meta-learning strategies inspired by Lake & Baroni (023), extending them to the domain of abstract spatial reasoning.
ARC-like puzzles. The abstraction and reasoning corpus (ARC) (Chollet 2019) is benchmark designed to evaluate a model's capacity to generalize to novel scenarios with limited to no prior knowledge, Based on a set of few-shot examples, models are required to infer transformations of abstract objects patterns within two-dimensional grids. Unlike ARC, which encompasses broad range of complex transformations, our work deliberately narrows the scope to the five fundamental geometric transformations described in Section 3.1 focusing instead on the aspect of systematicity. Several ARC variants have been proposed, including 1D-ARC (Xu et al.) 2024). Mini-ARC (Kim) eet al. 2022), ConceptARC (Moskvichev al. 2023) and MC-LARC (Shin al. 2024). However, to the best of our knowledge, Compositional-ARC is the first to focus on compositional generalization.
7 CONCLUSION
In this work, extend the meta-learning for compositionality framework proposed by Lake & Baroni (2023) to the domain of abstract spatial reasoning. To this end, we introduce Compositional- ARC- novel dataset designed to evaluate systematicity in this field. Our experiments demonstrate that models trained via MLC can systematically generalize to novel compositions of geometric transformations. Moreover, a small MLC model outperforms state-of-the-un general-purpose LLMs on Compositional-ARe and performs on par with domain-specing LLMs trained via test-time training. Our findings suggest that MILC presents a promising direction for enabling systematic generalization in language models across diverse domains.
Limitations & Future directions. A notable limitation of the current version of Compositional- ARC is its restriction to fixed-size grids and limited number of transformations. While it is possible to extend the dataset to more diverse grid setups, it is currently unclear how MLC would perform on more complex transformations. A promising direction for future work is to train additional model that learns how to decompose complex ARC-like problems into primitive transformations, and then train MI. these primitives generalize to unseen, more complex transformation compositions.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we make all code publicly available at: http:://ithubb hutps://gtuub..om//aaiiip com/mainlp/CCAACC This enables users to reproduce the data described in Section 3.] and train models via MLC for the task, as outlined in Section 3.2 Details about the training proce- dures and hyperparameters are provided in Section 3.2 and Appendix Specifics on prompts, model versions, and decoding parameters are given in Appendix D.2 Our datasets are publicly available at: https//huggingfaee http:///wggnnnecc..oo//aattt///s hatpaaasewemmnnlooooooooooonaaae
with further details presented in Sec-
tion 3.1] Section 4.1 and Appendix B Finally, Appendix C.2 outlines the software and computational
resources used for model training.
ACKNOWLEDGMENTS
We express our gratitude to the members of the MaiNLP lab for their invaluable feedback. Fur- thermore, we thank the anonymous reviewers for their insightful comments and suggestions. We gratefully acknowledge that experiments involving API calls to GPT-44 and 03-mini were supported by compute grant from OpenAI. The authors also acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR @ FAU) of the Friedrich-Alest Awwexneer--niierreeeee
Nürnberg (FAU) under the NHR project b217dd. NHR funding is provided by federal and Bavarian state authorities. NHR @ FAU hardware is partially funded by the German Research Foundation (DFG) 440719683, Finally, we acknowledge the support for BP through the ERC Consolidator Grant DIALECT 101043235.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv: 2023.
Anthropic. Introducing the next generation of claude.
claude- 3- 33-famill ly March 2024. Accessed: 2025-11-21.
Carter Bays. Introduction to Cellular Automata and Conway's Game of Life, pp. 1-7. Springer London, London, 2010. ISBN 978 1-84996-217-99 doi: 11007/978-1-8499-227---77 URL
Philémon Brakel and Stefan Frank. Strong systematicity in sentence processing by simple recurrent networks. Proceedings of the Annual Meeting of the Cognitive Science Society, volume 31, 2009.
David Brannan, Matthew F Esplen, and Jeremy Gray. Geometry. Cambridge University Press, 2011.
Paco Calvo and John Symons. The architecture of cognition: Rethinking Fodor and Pylyshyn's systematicity challenge. MIT Press, 2014.
Hoyeon Chang, Jinho Park, Hanseul Cho, Sohee Yang. Miyoung Ko, Hyeonbin Hwang, Seungpil Won, Dohaeng Lee, Youbin Ahn, and Minjoon Seo. The coverage principle: A framework for understanding compositional generalization. arXiv preprint arX1v:2505.22227, 2025.
François Chollet. On the measure of intelligence, 2019. URL https://arxiv..o abs/
1911. 01547
Noam Chomsky, Syntactic structures. Mouton de Gruyter, 2002.
Noam Chomsky et al. Reflections on language. Temple Smith London, 1976.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2/ 10. 14168, 2021.

---

Published as a conference paper at ICLR 2026
Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. Meta-learning to compositionally generalize. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 3322-3335, Online, August 2021. Association for Computational Linguistics. doi: 00.18653/v1/2022 ..cl-long. 258. URL //Rlittss///ac.aatoolooy/ otw.:gg..rg////o acc::11.ccl--m -ong. .588
Isabelle Dautriche and Emmanuel Chemla. Evidence for compositional abilities in one-year- old infants. Communications Psychology, 3(1):37, 2025. ISSN 2731-9121. doi: 10. s44271-005--0222-9, URL
000---255-00000 00222-9
Joachim De Beule and Benjamin K Bergen. On the emergence of compositionality. The Evolution of Language, pp. 35-42. World Scientific, 2006.
Google DeepMind. Gemini 2.0 flash, 2024. URL https:://eet nd goog le/ Accessed: 2025-03-19.
Sechnolooiies/geee
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL http::t:/pppteeeeetee
net/ / fute/forum? wdiiiicbydVVTTT
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song. Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language models. In The Eleventh International Conference on Learning Representations, 2023. URL https 7 //operrvvew. net forum?id d=gg2mmd-gg@
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Picter Abbeel, RI:: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.02779, 2016.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang (Lorraine) Li, Liwei Jiang. Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 70293-70332. Curran Asso- URL ciates, Inc., / igs neur ps /ppprrrr..pppp..pppppppe for___illee files/ppppsssee 2023/ Fi ftt.f/eeeeeebeec 22222
James H. Fife, Kofi James, and Malcolm Bauer. A learning progression for geometric transformations. ETS Research Report Series, 2019(1):1-16,, 2019. doi: https://bo.oog///0000///222222 URL
wi http:///ww..aaa.......oooo///aaa
Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A crit-
ical analysis. Cognition, 28(1):3-71, 1988. ISSN 0010-0277. doi: https://doi.or//10
1016/0010--2277088900033
URL httpes//ww..ceeeecddeeer... wttp:///eeeeeee...ccccceeeeeeeee ceteet.comscccineec
art articie/p11/000002777855
Daniel Franzen, Jan Disselhoff and David Hartmann. The Ilm architect: Solving the are challenge is a
matter of perspective. https: t/ggttttuuut/u com http://aafr//rr--reeee2222////
the www.aacchhcttt... pdf 2024. Accessed: 2025-09-23.
Gaäl Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. Large language models are not strong abstract reasoners. In Proceedings of the Thirr International Joint Conference Artificial Intelligence, IJCA '24, 2024. ISBN 978-1-956792-04-11 doi: (0.24963/jcai.22244693 URL httphttss://oii.orpttt wwt.:ooooorggg// 200..244433/11
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The la herd of models. arXiv preprint vrXXiv2407.21788 2024.
12

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseck-rl incentivizes reasoning in Ilms through rein- forcement learning. Nature, 645(8081):633 638, Sep 2025. ISSN 1476-4687. doi: 10.1038/ 1588-0025-09422-2 htt.htttss://dd. wrgg 1038/s4111 58 025- -2222--992222
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus).
arXiv preprint
arXiv:1606.0888 08415, 2016.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track net
(Round 2), 2021. URL WtLpttts:///peett
httpl//wwweeeetttueeeeeeeeeeeeee forum? id= Bywt 2mQsCe
Wolfram Hinzen, Edouard Machery, and Markus Werning. The Oxford Handbook of Com- positionaliity Oxford University Press, 02 2012. ISBN 9780199541072. doi: 10.1093/
oxforuhbb977801195411772..0 10001. 9780199541072.001.00
URL
Michael Hodel. Addressing the abstraction and reasoning corpus via procedural example generation. arXiv preprint arXiv:2404.07353, 2024.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-Learning in Neural Networks: A Survey IEEE Transactions on Pattern Analysis & Machine Intelligence. 44(09):
5149-5169, September 2022. ISSN 1939-3539. doi: 10.1 10.1109/TPMMI.20 2021 .3079209. URL.
https: www.eeeeeeeeeeeeeeee.....ee wtt.eeerrrreeet......o....
wtw.:iitttt.................. wrt. wttp////..1////PAA//... 1109/TPAM9MTIIT 2021 3079209
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795,
Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. Metalearning continual learning algo- rithms. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https
//openreview.net net
Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Duygu Ataman, and Lonneke Van Der Plas. Evaluating morphological composi- tional generalization in large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1270-1305, Albuquerque, New Mexico, April 2025. Association for Computa- tional Linguisties. ISBN 770-88887776--88988
doi: 20118653/v122222 haa0055.anaaaooonn5555 URL
kttps://aclantholooo m-c.::/ww......2mmmmmmaa-------- www.naaalnnnnn... 597
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey,, al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Khurram Javed and Martha White. Meta-learning representations for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL /tttps:/ http:///nnn..euuupsecc//pa/eeee co/papr files:/pape:/2019/ Ff
.pdf
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Mare van Zee, and Olivier Bousquet. Measuring compositional generalization: A comprehensive method on realistic data. In International Conference on Learning Representations, 2020. URL net/ htt:://uummmddyycccneeee
https: /tttss:/cpenrevee/
Najoung Kim and Tal Linzen. COGS: A compositional generalization challenge based semantic interpretation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference
Empirical Methods in Natural Language Processing (EMNLP), pp. 9087- 9105, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/200 emnlp-main.77 7311. URL WtR.ttttt... ttatta://aci htt.://ppe//aallaatooooooo.ooooo
anthelogy. orgp///oooyro//22000/// emn 1p n. 7317
13

---

Published as a conference paper at ICLR 2026
Subin Kim, Prin Phunyaphibarn, Donghyun Ahn, and Sundong Kim. Playgrounds for abstraction and reasoning. In NeurlPS 2022 Workshop on Neuro Causal and Symbolic Al (nCSI), 2022.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large language models are zero-shot reasoners. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.). Advances in Neural In- formation Processing Systems, volume 35, pp. 22199-22213. Curran Associates, Inc.. 2022. URL httpp:://proooodingg..... neur cc/ wwtpppppppeeeeeeeeeeeeeeeeeeeeee http://ppeee...eeee///r//////22/ file/ http://cccccccccccoeeeeeeeeeeee.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequeee recurrent networks. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2873-28838 PMLR, 10-15 Jul 2018. URL https: http:::/pprcceeddngg.
mlr. www.s/ees/.... v80 Lakel8a html
Brenden M Lake. Compositional generalization through meta sequence-to to-ice-n--e learn- ing. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019, htl.atppe///pooeeeinnggerreooioo 2019 o013//ilee4002221117733777333333
Brenden M. Lake and Marco Baroni. Human-like systematic generalization through meta- learning neural network. Nature, 623(79855::15--12,, 2023. ISSN 1476-4687. doi: 10.1038/
$41586-023-06668-3 /tLpttss: / /ttp:/a.......1...01111111111111
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017. doi: 10.1017/801 401055011105111100011 60018377.
Soochan Lee, Jachyeon Son, and Gunhee Kim. Recasting continual learning as sequence mod- eling. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 70433-70452 Curran Asso- ciates, Inc 2023. /Rtp///psse/ppreeeeccssseeeeeee htt3://eeee222ccccccaccfcccaffa3
files/paper)
Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M. Dunn, Hao Tang, Wei-Long Zheng, Yewen Pu, and Kevin Ellis. Combining induction and transduction for abstract reasoning. In The Thirteenth International Conference on Learning Representations,
2025. URL htt.ttttt///peerev////
n Foruum??---mmmooAAAVV
João Loula, Marco Baroni, and Brenden Lake. Rearranging the familiar: Testing compositional generalization in recurrent networks. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 108 -14,, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18--1 5413. URL https:://ccaatthtt
brr//18-- 54333/
Amogh Mannekote. Towards compositionally generalizable semantic parsing in large language models: A survey, arXiv preprint arXiv:2404.1307. 2024.
Gary F Mareus. The algebraic mind: Integrating connectionism and cognitive science. MIT press, 2003.
Ahmed Masry, Juan A. Rodriguez, Tianyu Zhang, Suyuchen Wang, Chao Wang, Aarash Feizi, Akshay Kalkunte Suresh, Abhay Puri, Xiangru Jian, Pierre-André Noël, Sathwik Tejaswi Mad- husudhan, Marco Pedersoli, Bang Liu, Nicolas Chapados, Yoshua Bengio, Enamul Hoque, Christo- pher Pal, Issam H. Laradji, David Vazquez, Perouz Taslakian, Spandana Gella, and Sai Rajeswar. Alignvlm: Bridging vision and language latent spaces for multimodal understanding, 2025. URL https://arxx: www.:oorggssooooo 2534 01341
14

---

Published as a conference paper at ICLR 2026
Pablo Mendes, Max Jakob, and Christian Bizer. DBpedia: A muitilingual cross-domain knowledge base. In Nicoletta Calzolari, Khalid Choukri. Thierry Declerck, Mehmet Uğur Dogan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC" 12), pp. 1813 1817, Istanbul, Turkey, May 2012. European Language Resources Association (ELRA). URL httpps://acaaatholooggooo org. 1323/
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In International Conference on Learning Representations, 2018.
Arsenii Kirillovich Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The conceptARC benchmark: Evaluating understanding and generalization in the ARC domain. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https::/
Eorum??dd-BykyCttte.
OpenAl. Openai ol system card, 2024.
OpenAL. Openai 03-mini system card, January 2025. URL https::/cdn.opentt com/ 03 mini system-------- card-aebaldd feb10.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Flavio Petruzzellis, Alberto Testolin, and Alessandro Sperduti. Benchmarking GPT-4 algorithmic problems: A systematic evaluation of prompting strategies. In Nicoletta Calzolari, Min- Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp- 2161-2177, Torino, Italia, May 2024. ELRA and ICCL. org 2024 lrec-main. man.cmmain........
URL
Yusuke Sakai, Hidetaka Kamigaito, and Taro Watanabe. Revisiting compositional generalization capability of large language models considering instruction following ability. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11121933123,, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0, doi: 18.186533v 1/t/////2222522 http:///11222...21oo...... 1508. URL http:: crq/2025..cl. acnn 1508/]
David Maria Schmidt, Raoul Schubert, and Philipp Cimiano. Compost: A benchmark for analyzing the ability of llms to compositionally interpret questions in a qald setting. In Daniel Garijo, Sabrina Kirrane, Angelo Salatino, Cogan Shimizu, Maribel Acosta, Andrea Giovanni Nuzzolese, Sebastián Ferrada, Thibaut Soulard, Kouji Kozaki, Hideaki Takeda, and Anna Lisa Gentile (eds.), The Semantic Web ISWC 2025, pp. 3-22, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-032-09527-5.
Donghyeon Shin, Seungpil Lee, Klea Lena Kovacec, and Sundong Kim. From generation to selection: Findings of converting analogical problem-solving into multiple-choie questions. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 6696-6700 6708, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 118855311/2224 htw.:///ww//222..nnnnnnn-n-oo findings-emnlp.922 URL hittps://aclantien waaphhoooyyy.oo .rg. 22eeg/2224.40 ftddi.......i......ee emn 19.3
Zoltán Gendler Szabó. The case for compositionality. In Markus Werning, Wolfram Hinzen, and Edouard Machery (eds.), The Oxford Handbook of Compositionalityy Oxford University Press, 2012.
Gemini Team, Rohan Anil. Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katic Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. /800, 2023.
15

---

Published as a conference paper at ICLR 2026
Jonathan Thomm, Giacomo Camposampiero, Aleksandar Terzic, Michael Hersche, Bern- hard Schölkopf, and Abbas Rahimi. Limits of transformer language models on learn- ing to compose algorithms. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Pa- quet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp- 7631-7674. Curran Associates, Inc., 2024. doi: 10.52202/ 079017-0245. URL
neurips wwtpaper files/paper Coreeeeeee pdf
2024/Eile/0e (t5.://.2.6111711111992222222222 Pwwpll9t22-aaaeeere Conferenceeeeeeeeeeeeeeeee
Sebastian Thrun and Lorien Pratt. Learning to Learn: Introduction and Overview, pp. 3-17, Springer doi: 10. 00110077978811445555555-----
US. Boston, MA, 1998. ISBN 978-1-4615-55229
URL
https://doi. org 1007 978 1- 4615 5529-2222222 2
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Jane Wang, Zeb Kurth-Nelson, Hubert Soyer, Joel Leibo, Dhruva Tirumala, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 39, 2017.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In The Eleventh International Conference on Leurning Representations, 2023a. URL https: / wtt.e/eww.....ooooeodd--1-1111
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 13484- 13508. Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10.18653/ v1/2023.aclllong.. long.
URL htt.ttt:: / ac www.ennntolggy..... org. 2020g/22222222 acl- 754/
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language mod- cls. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 24824424837. Curran Asso- ciates, Inc., 2022. URL UtL.htppss//prooeeddssseeeerrree 2122://e222255133334444111111022
http://pppr__eeesseeeeeepeee files/paper)
Sondre Wold, Lucas Georges Gabriel Charpentier, and Étienne Simon. Systematic generalization in language models scales with information entropy. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 1807-1819, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979- 999999966-655 doi: 10.18653/v 10t0l/.1/22...ff http:2/...niins-aae acl.90.. URL https
//aclantho ogy. org/ 2025. findings-aan acl. 907
Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, and Elias Boutros Khalil. LLMs and the abstraction and reasoning corpus: Successes, failures, and the importance object- based representations. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URL
net otg.ppreeeeeeeeneeeeoooooooooooo wow.:roiid-emoooeeeeoo
Haoran Yang, Hongyuan Lu, Wai Lam, and Deng Cai. Exploring compositional generalization
of large language models. In Yang (Trista) Cao, Isabel Papadimitriou, Anaelia Ovalle, Marcos Zampieri. Francis Ferraro, and Swabha Swayamdipta (eds.). Proceedings of the 2024 Conference of
the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 4: Student Research Workshop). pp. 16-24, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.118653/vll 1/202//1222222mm// naa.-...aal--ss.33.. URI
https: http:///aaaaathhhhcc wrg. 2024. www.naaalaa..... www. 37
Jun Zhao, Jingqi Tong, Yurong Mou, Ming Zhang, Qi Zhang, and Xuanjing Huang. Exploring the compositional deficiency of large language models in mathematical reasoning through trap problems. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024
16

---

Published as a conference paper at ICLR 2026
Conference on Empirical Methods in Natural Language Processing, pp. 16361-16376. Miami, Florida, USA, November 2024, Association for Computational Linguistics. doi: 10.18653/v1/ 2024.emmlpy 4024.emll-mmann9115 m15 DtL.ttoess//ccaattoooo.y...
Yanli Zhou, Reuben Feinman, and Brenden M. Lake. Compositional diversity in visual concept learning. Cogmition, 244:105711 2024, ISSN 0010-0277, doi: http:///bi.rrg11.0000.....g..ioo
2023. 105711. URL https:
SDC10027723003451
A SYSTEMATIC GENERALIZATION IN LLMS
The question of whether neural networks, and more recently large language models, have the capacity to generalize systematically from known components to novel combinations has been, and continues to be, the subject of extensive debate (Fodor& & Pylyshyn, 1988 Brakel & Frank 2009) Lake & Baroni 2023 Mannekote| 2024, inter alia). This section offers an extended literature review on systematic generalization LLMs, presenting overview of recent studies that assess systematicity in current language models.
Following Hupkes et al. (2020). different aspects of compositionality need be distinguished. Sys- tematicity refers to the capacity to recombine known parts and rules into novel combinations (Szabó) 2012 Hupkes et al. 2020
Lake Baroni 2023). More formally, this capacity can be defined as:
Definition (Systematic generalization). The capacity to recombine previously observed or learned parts and rules, i.c., primitives e1, .2,...,, C,, to generalize to novel, previously unseen compositions of them (e.g., x 62).
According to Hupkes et al. 22020), systematicity is different from other aspects of compositionality, such productivity: the capacity to predict expressions beyond the length of those already encoun- tered, or substitutivity: the ability to handle synonym substitutions. For more details on the different aspects of compositionality, we refer to the original study by HHukess et al. (2020).
Systematicity in current LLMs. A growing body of research evaluates whether large language models satisfy the criteria of systematicity. i.c., whether they can generalize systematically from known or previously seen components to novel combinations. Thomm et al. (2024) study whether LLMs such as LLaMA (Touvron al., 2023), GPT-4 (Achiam et al.) 2003), ), and Gemini-Pro (Team) et al. 2023) can solve compositional algorithmic tasks by reusing previously encountered primitives. Training& a small LLaMA- style model from scratch on four compositional algorithmic tasks shows that, while the model is able to learn all sub-tasks or primitives reliably, it fails to properly compose them. Instead, the model exhibits extreme sample inefficiency: is able to solve the compositional task only when the amount of training data is increased by almost one order of magnitude. The authors further present a complexity-theoreiic theoretic argument that gradient-descent training of fixed-depth feedforward models is asymptotically data-inefficient evaluations of GPT-4 and Gemini-Pa further show that these models struggle with the tasks, even when strong hints are provided or techniques such as 2022) are used. Dziri et al. (2023) also highlight compositional limits of LLMs on multiplication, logic-grid puzzles, and mnamic-programminn task: performance is near-perfect in-distribution but collapses as computation graphs deepen or branch beyond training complexity, [Petruzellis llis et al. (2024) complement these findings with systematic study of LLMs' performance on algorithmic tasks where both the number of operands per operation and their nesting depth can be controlled. While the authors show that I.LMs such as GPT-4 fail on highly nested, multi-operand formulas, they find that advanced prompting strategies such as zero-shot CoT (Kojima et al. 2022) with self-consistency (Wang et al., 2023a) can improve performance on less complex compositions.
combinatorial problems. Prompt-based
(CoT) prompting (Wei et al.)
Zhho et al. (2024) introduce MATHTRAP, where standard problems from GSM8K (Cobbe et al. 2021) and MATH (Hendrycks et al. 2021) are modified with logical "traps" (e.g., undefined con- competence with the capacity to identify such inconsistencies. The authors show that LLMs such as Llama-3 (70B) (Grattafiori al. 2024), Claude3-Opus (Anthropic) 2024), and GPT-4 score well on the original
cepts, missing conditions, contradictions) that require combining ordinary
17

---

Published as a conference paper at ICLR 2026
problems and on standalone questions about the logical inconsistencies presented, yet their accuracy drops dramatically on such trap problems. Prompting that warns about traps, few-shot demonstrations, fine-tuming, and OpenAI's ol "slow thinking" (Jacch et al. 2024) improve performance but still leave a substantial gap to human performance on the task.
In semantic parsing, where natural language must be translated into structured (often symbolic) form, systematic generalization plays an important role when novel compositional queries that have not been seen during training are introduced (Mannekote 2024). SShhmidt al. (2025) propose CompoST, a controlled semantic-parsing question answering over linked data from DBpedia (Mendes al. 2012). Models need to map natural- Natural-language questions to SPARQL queries where all atomic graph-pattern constituents have been presented, while novel combinations appear at test time. Across zero-shot, few-shot, and fine- tuned settings on three difficulty splits, performance drops sharply as structural deviation increases. The authors conclude that current L.MMs struggle to systematically recombine known SPARQL constituents into correct queries, indicating weak systematic generalization in this domain. In contrast, Drozdov et al. (2023) show that models can achieve high systematic generalization on semantic parsing datasets such as CFQ (Keysers et al., 2020) and COGS (Kim & Linzen 2020) when explicitly prompted to decompose problems. The authors introduce dynamic least-to- most prompting, where models first decompose the input and then solve subproblems sequentially. Compared to standard few-shot prompting, least-to-most prompting achieves near-SOTA OOD performance on CFQ and COGS, suggesting that systematicity is not reliably expressed by default but can be elicited. Yang et al. (2024) study "order-n" compositional instructions derived via self-instruct (Wang) al. 2023b). Training on higher-order compositions improves performance on lower-order ones, but training on simpler orders does not transfer to longer compositions, revealing an asymmetry typical of non-systematic learners. /Sakai all (2025) introduce Ordered CommonGen, where four known concepts must be embedded in a sentence in specified order across permutations. While unordered concept coverage is high, ordered coverage remains substantially lower even for LLMs, indicating difficulty in faithfully recombining familiar concepts under novel structural constraints, [Imayilzadd ict al. (2025) extend systematicity tests to morphology in agglutinative languages (Turkish, Finnish): LLMs struggle to generate validate novel morpheme compositions, particularly for nonce roots and longer affix chains, and performance degrades with compositional length.
benchmark for evaluating systematic generalization in
Two data-centric accounts help explain when LLMs succeed fail. Wold et al. (2025) argue that systematic generalization scales with the information entropy of the training distribution over primi- tives; in modified SCAN, higher-entropy coverage of verbs and contexts yields smooth improvements in systematic generalization. Chang et al.] (2025) formalize the coverage principle, showing that systematicity in transformer-basse models largely reduces substituting functionally equivalent fragments observed in shared contexts. They show that data requirements for multi-hop systematicity
grow at least quadratically in component set size and are largely insensitive to parameter scaling.
Summary. Overall, the surveyed studies suggest that current LLMs do not not reliably exhibit human- like systematic generalization under standard training and evaluation: performance often correlates with training data coverage and degrades novel compositions. However, prompting techniques for explicit decomposition and compositional training curricula might be able to elicit systematicity, consistent with view that compositional abilities are partly latent but not automatically deployed.
B DATASET
In this work, we present CompositionllAARC dataset designed to study systematicity in abstract spatial reasoning. As outlined in Section 3.1 Compositional-ARC evaluates a model's capacity to systematically generalize learned geometric transformations (e.g., translation, rotation) of two- dimensional objects to novel compositions of these transformations (c.g.. translation-rottaionn) The subsequent sections offer detailed description of the dataset, including formal definitions of the grid-based environment and the set of transformations it includes.
B.1
GRID SETUP
We define the structure of the 10 x 10 grid environment and the notion of objects within it. Each grid is represented as matrix X N10x10 where each element corresponds to cell with discrete
18

---

Published as a conference paper at ICLR 2026
color value. Objects defined based color connectivity using the Moore neighborhood (Bays 2010).
Definition 2 (Grid & Object). Let X N10x10 be matrix with rows i and columns referred to as a grid, where each element X; E fo....,. .9. The value X,j = represents a background cell, and values X; E [l...., represent object colors.
An object is a set of coordinates
① S {0..... .........) 9)2
such that each (i,j) O satisfies X;j c, and the elements in O form single connected component.
Two elements Xij and Xat considered connected if:
max(li kl, l ()) 2
We define the following color mapping: 0 + black, 1 + red, 2 + orange, 3 + yellow, 4 - green, blue, 6 purple, pink, - cyan, and 9 gray.
B.2 GEOMETRIC TRANSFORMATIONS
We formally define the five basic geometric transformations used in dataset: translation, rotation, reflection, extension, and color change. Each transformation operates on objects within the grid environment defined in Appendix [.I A transformation is considered valid if all transformed coordinates lie within the grid bounds and do not overlap with existing objects in the original grid.
Translation. Moves object by cell along specified direction (downward rightward). A formal definition is given in the text box below.
Definition (Translation). Let O S (O...., 9]2 be an object in grid X N10x10 and let (V1, v2) {(1, (0, 1)) be the translation direction (downward rightward).
The translated object is:
Trans, (0) {ii v1.j v2) (i.j) 0)
The translation is valid if:
v(i'j') T Tuans, (0), 0 < i'jj' < 10, X,y 0
19

---

Published as a conference paper at ICLR 2026
Rotation. Rotates object 90º clockwise counterelockwise around the top-left of its bounding box. A more formal definition is given in the text box below.
Definition 4 (Rotation). Let S (o,..., 9)2 be a set of grid cells with row-column coordinates (i, Let ig (o and jo (ij)e j. We set the pivot P (io, jo) as the top-left of the bounding box.
For each (i,j) O, we specify the offset from the pivot as:
(Ai, Aj) = (i Aminn j - jmin).
We define a rotation by +190 as:
Ro (Ai, Aj) (Áj, -ii), R- (Ai, (-Aj, Ai),
where +900 is clockwise and -90 is counterclockwise under the row-down convention.
Given a 90 rotation, either clockwise or counterclockwise, the rotated object is:
+90 (0) (émin
-i, Jmin Aj) (i.j) E0).
The rotation is valid if: v(i', To To), 0 0 V' < 10, Tzy =
Reflection. Reflects an object across its vertical or horizontal axis, reversing the relative positions of its coordinates while preserving overall structure.
Definition (Reflection). Let O .o...., 9] be object in a grid X 100 and let E (horizontal, vertical) indicate the axis of reflection.
Let:
Amnn = minfi (1,J) E 0},
imax max(i (1,j) E 0}
Jmin min (1,j) 0), Jmax max (i.j) E 0)
Then the reflected object is:
TO)) = {(imax (i - imin); j) (i,j) 60) if horizontal
(((i, Jmax - (j - jmin)) (i,j) 6
if d = vertical
The reflection is valid if:
"(i'j') E (O), 0si,j' < 10,
X =0
20

---

Published as a conference paper at ICLR 2026
Extension. Adds new cell in the upward or leftward direction for each coordinate in the object.
Definition 6 (Extension). Let O S (o...., 9}2 be an object in a grid X E N10x10, with color e <>> 0. Let d E {up, left) indicate the extension direction.
Let the set of new cells adjacent to the object in direction d be:
{ [ -1--1,,) $ 0 (i.j) 0, i> 0, T1-1. = 0} if d up Na(O)
( ((i.j- & 0 (i.j) E O j > 0, Fis-1 = 0} if d left
Then the extended object is:
Tex. (O) - -
The extension is valid if:
V(V.)) E No(O), 0 osú,j' < 10,, X =0
All new cells (f',)) E Na(O) assigned the color of the original object:
X/3
Color change. Alters the color of an object to either red orange, without changing its structure position.
Definition (Color Change). Let S (0,...,99) 9]2 be an object in a grid X E N10x10, with color >>> O. Let d' E (1,2) be the new color (representing red or orange).
The resulting grid X' given by:
d'
if (1.)) E otherwise
X
Xi
B.3 DATASET GENERATION
To generate episodes that comprise primitive transformations, level-1 transformation compositions, and level-2 transformation compositions, we developed script that systematically generates the corresponding input-output grid pairs for each transformation. The complete code repository for data generation is publicly available at: http://ggthub..commmmmpp/////
the following, we provide a brief overview of the procedure used generate input-output grid pairs for each sample within an episode. As detailed in Section 3.1 and Appendix B.2] we consider five basic geometric transformations, along with three types of transformation indicators: shape-based, color-based, and neighbor-based. These allow us to define a total of ten distinct transformation triplets, each mapping the indicators corresponding transformations (e.g., shape-based, translation. color-based: reflection, neighbor- extension). For each episode, a transformation triplet is uniformly sampled from this set to define the visual interpretation grammar of the episode. Once the transformations determined, we randomly assign specific shape for the shape-based transformation, a specific color for the color-based transformation, and an indicator object for the neighbor- based transformation. Importantly, the indicator object is constrained to neither share the shape associated with the shape- based transformation nor the color linked to the color-based transformation.
Using these specifications, we generate input-output grid pairs representing primitive, level-1, and level-2 transformations. For each transformation mapping, we randomly place an object on a 10 x 10 grid, ensuring it possesses the designated shape, color, and/or proximity to the indicator object as required. The specified transformation then applied to this object. If the resulting transformed object remains within the grid bounds and does not overlap with any other object, the corresponding input-output grid pair is accepted as a valid sample for the episode. Otherwise, a new object location sampled and the process is repeated until valid pair is obtained. Finally, make that each
21

---

Published as a conference paper at ICLR 2026
episode follows a unique grammar, ie., that no two combinations of shape, color, and indicator objects correspond to the same set of transformations within the dataset.
Once the dataset is generated, we apply a systematicity- data split into training, validation, and test sets. As mentioned before, the five basic geometric transformations, along with three types of transformation indicators, allow
define total of ten distinct transformation triplets (e.g.. shape-based translation, color- based: reflection, neighbor- based: extension). We split the data as follows: we randomly designate 20% of all triplets as test-only and 80% as train-only (see Table (7). This means that the geometric transformations involved in the final query level-2 compositions differ between the training and evaluation sets. For instance, for seed 1860, all episodes whose triplet falls in the train set yield 82,908 training episodes, and evaluation-one onlly triplets form a 17.092-episode pool, which we split evenly into 8,546 validation and 8,546 test episodes. All data is publicly available at: htps://huggingface httpss//wggiingffaeeooddasstttii
B.4 DATASET STATISTIC'
Table 7 presents detailed statistics for the datasets used in this study. As outlined in Section 5.1 we train and evaluate models via MLC across four distinct dataset splits to mitigate the influence of randomness in the data split process. The table includes the number of training, validation, and test samples for each split. Additionally, it provides information on the query transformation compositions present in the training and test sets, along with the frequency of each basic geometric transformation within the train dataset.
In a similar vein, Table 3 shows the statistics for the dataset version that includes more diverse transformations, as described in Section 5.3 The table provides information on the query transforma- tion compositions present in the training and test sets, along with the frequency of each geometric transformation within the training dataset.
C TRAINING DETAILS
As outlined in Section 3.2 we transformer- bassed encoder-decoder model trained using MLC to predict the correct output grid for a given input query, given a set of study examples. Specifically, generate dataset of 100,000 episodes and split it into train, validation and test sets (for more information see Section 4.1 and Table 7). The model is optimized using cross-entropy loss, averaged over the predicted patch embeddings, non-background regions, patches corresponding exclusively to black 2 x 2 cells are down-weighted by a factor of 0.2 during loss computation.
described in Section 3.2 To place greater emphasis on
Each episode includes collection of study examples and queries. In the standard few-shot learning task (Section 4.1,, the model receives three input-output grid pairs, along with the input query. For the systematicity task, 12 systematic study examples are provided. In both tasks, the model is required to predict the correct output grid for ten distinct input queries.
Training is conducted over 200 epochs with batch size of 200 for the standard few-shot learning task (i.e., 200 10 = 2000 queries per batch), and over 300 epochs with the same batch size for the systematicity task. A learning rate of 0.01 is used in both cases. Following the approach of Lake & Baroni 2023), we apply a warm-up phase during the first episode, beginning with a learning rate of 1 x 10= followed by linear decay to 5 1 over the course of training. Additional hyperparameter settings are provided in Section C. and summarized in Table 3
C.1 HYPERPARAMETERA
To identify suitable hyperparameters for model training, we conduct Bayesian search over a predefined
range of values: learning rate E [1 x 10 2 10 3, 10-4], final learning rate after linear decay E [1 x 10 x 10_4], dropout rate E [0.0, 0. 0.2], gradient accumulation over k E [1, 2] batches, cell color perturbation probability Pnoise E [0.0, 0.01, 0.001], feedforward hidden dimension E [512, 768], loss weighting for background (all-black) patches E [0.2, 0.4, 1.0), number of encoder layers E [2, 3, 4), and number of decoder layers E 2, 3,3,4. 4.
22

---

Published as a conference paper at ICLR 2026
Table Hyperparameter configuration for models trained via MLC.
Parameter
Value
Parameter
Value
number layers in decoder
learning rate after training
x 10
number layers in decoder
dropout
number attention heads
weight decay
0.01
hidden dimension feedforward hidden size
128
noise probability
0.001
768
gradient accumulation over & batches
learning rate
0.01
background patch loss weight
0.2
For the hyperparamter search, the model trained for 40 epochs on the systematicity task and evaluated on its corresponding validation set. Across 25 independent runs, select the configuration that achieves the highest validation accuracy. The final hyperparameter settings, presented Table employed consistently across both task setups.
C.2 IMPLEMENTATION DETAILS
All experiments were conducted using PyTorch (Paszke al. 2019 as the primary development framework. Comprehensive details regarding supporting software and versioning are available in our code repository. Experiments were executed on NVIDIA A100 and H200 GPUs. Training models with MLC on the standard three-shot learning task over 200 epochs required approximately 40 GPU hours on single A 100 GPU. For the systematicity experiments with 12 study examples, training over 300 epochs the designated dataset consumed roughly 100 GPU hours on a single H200 GPU.
C.3 ORIGINAL MLC TRAINING
In the original MLC setup, Lake & Baroni (023) train standard seq2seq transformer (3-layer encoder/decoder, 8 attention heads, hidden dimension 128) with Adam on 100,000 dynamically generated episodes of pseudo-language instructions. Each episode is defined by latent compositional grammar and contains multiple study examples and queries concatenated into single input sequence, described in Section - Training minimizes token-level cross-entropy for 50 epochs using a batch size of 25 episodes, a learning rate of 10 with one-epoch warm-up followed by linear decay, and dropout of 0.1. Compared our approach, their model targets linguistic sequences rather than visual grids and does not use patch-wise losses or background reweighting. Additionally, we train longer with larger batches and loss designed emphasize con-background spatial structure.
D EXPERIMENT DETAILS
This section provides further details regarding our experimental setup. Specifically, Section D. presents formal definitions of the evaluation metrics, while Section Scction[DDDoottiie outlines additional information on how interact with API-based LLMs.
D. EVALUATION METRICS
As described in Section 4.3 we use three different evaluation metrics assess model performance: exact match accuracy, ii) color accuracy, and iii) shape accuracy. These metrics are formally defined based on the grid-based environment X and the concept of object O, as specified in Definition 2
Let X Larget
pred E N10x10 denote the target and predicted grids, respectively. Each cell X target
(or X prred) contains an integer in 0, .,..., 9, where represents the background and values from I to correspond to cells occupied by colored objects. The set of objects defined as maximal connected cells of consistent color under the Moore neighborhood (see Section 3.1 extracted from X target and X pred are denoted P tarter and P P(XPrrdd) respectively. For each object in grid E P(X),
assign color label c00) E 1 E1...,, and define its normalized shape follows:
23

---

Published as a conference paper at ICLR 2026
S(D) = {(i - imin j - jmin) (i,j) E 0)
(1)
where
min
minfi: (i.j) E 0) and Juin - min (i,)) E O}.
This transformation "anchors" the object to the top-left corner by translating it to coordinate system with its minimum row and column indices zero.
Accuracy. The exact match accuracy evaluates whether the predicted grid X ared is identical the target grid X target on a cell-by-cell basis:
1, if X i bargget (i, (.....,.)'. ,933,
Accuracy (X
X xtarget)
0, otherwise.
In other words, this metric yields a value of 1 if and only if the entire predicted grid matches the target grid exactly, i.e., X Xlarget Xrred The mean accuracy over the dataset 'D is then defined as:
Accuracy
Σ
Accuracy X pred
X targett
(4)
D
(Xpred XeD
Color accuracy. Color accuracy assesses whether the predicted grid contains the same number of objects of each color as the target grid, irrespective of their locations or shapes. For a given color E ,,...,9. 9, let
m(c, x)= = |10 P(X) c(0) = c]/.
(5)
denote the number objects of color in grid X. Then, color accuracy is defined as:
Color Accuracy(X**) pred
targett) = 1 Vc (1....,9)} m(c, X.red) = m((c X target }.
(6)
where 1. is the indicator function, returning if the condition is satisfied for all colors and otherwise. The mean color accuracy over the dataset D is given by:
Color Accuracy = [D] pred X JDD
Color Accuracy
(erracy(X*)) X pred X target
Shape accuracy. Shape accuracy measures the agreement in object shapes between the predicted and target grids, independent of color and position. For each object in grid O P(X), we consider its normalized shape S(O) as defined in Equation The count of objects with a specific normalized shape in grid X is given by: n(8, X) 110 P(X): :(0) = s}|-
(8)
Accordingly, shape accuracy is defined as:
Shape Accuracy( pred X Xtarget) = =[ n(s, X = n(s,, X xargeet
(9)
That is, the predicted grid pred has perfect shape accuracy if the number of objects corresponding to each normalized shape identical to that in the target grid X target Finally, the mean shape accuracy the dataset D is given by:
Shape Accuracy
= [D]
1
Σ
Shape Accuracy( Cecuracy(XPPa) pred X Xlarget)
(10)
(X Xtarsett
24

---

Published as a conference paper at ICLR 2026
D.2 MODEL INFORMATION
General-purpose LLMs. As described in Section 4.2 we evaluate three different general-
purpose LLMs Compositional-ACC Specifically, we assess the performance of o3-
mini (OpenAl 2025) (version 0
GPT-4o (Achiam et al. 2023
(version 20- -08-2002-08---- 08-06- and Gemini 2.0 Flash (DeepMind) 2024) (version gemini-2. 0-
All models are accessed via their respective batch APIs, enabling us process multiple samples per request. Unless otherwise specified, we employ the default API settings. For GPT-4o and 03-mini, this corresponds to a temperature and top-p value of 1.0 Due to
financial constraints, the 03-mini model is configured with a "low" reasoning effort. For Gemini 2.0 Flash, the provider does not disclose default parameter settings.
The complete set of prompts used in our evaluations is presented Figures 12 h 15 To ensure consistency and facilitate meaningful comparisons, we apply the same prompts across all models. The standard few-shot learning prompt appears in Figure 12] while the prompt used for the systematicity task is shown in Figure 14 For Gemini 2.0 Flash, we add the instruction: "Do not generate any code to solve the task" to the output requirements, as the model otherwise does not adhere to the required output format. As outlined in Section 4.2 we additionally evaluate GPT-4o and Gemini 2.0 Flash in multimodal configuration, in which both an image of the study examples and the input query are provided alongside the text prompt (text+imagcc). The multimodal prompt for the few-shot learning task is shown in Figure 13 with the accompanying image illustrated in Figure TO The corresponding multimodal prompt for the systematicity task is depicted Figure [5] with the associated image presented in Figure 11] For the textual prompts, represent grids as two-dimensional arrays, consistent with prior work (Moskvichev et al. (023)). For instance, the final query input grid in Figure 5 would be represented as:
Prompts.
[[O, 0, 0,
O
0,
0, 0]
[0, 0, 0
0
0
0,
0, 01,
[0, 0, 0
0
0,
0,
0 0, C].
[0, 0, 0,
0,
O
O,
0, 0, 0O,
[0, 5, 0, 0,
0, 01,
[0, 5,
0,
0, 01,
[5, 5, 0, 0, 0, 0, 0, 0, C],
[0, 5, 0, 0, 0, 0, 0, 0, 0, 00,
[0, 0, 0, 0,
1, 0, 0 0, 01,
[0, 0, 0, 1, 1, 0, 0, 0]]
Model responses are parsed using regular expressions identify the expression "output:". followed by a two-dimensional array of the form "[[-. as specified in the input prompt. If a response does not contain this pattern, it is excluded from further analysis and omitted from accuracy computations. Table Jummarisemaa the proportion of valid responses for each model.
As mentioned in Section 4.2 we also evaluate two LLMs proposed by al. (2024) that are specifically tailored A ARC-style data: La (ver- 4bit] and (ii) Mistral-NeMO-MMiinne 8B-Full (version Mi Mistral -Neoo--MMMM--o -MMMoMMMiiiroo----- -B-Chii-e--ACCheeet--eeeeee www.Aetteettruulllaaaa
Domain-specific LLMs.
sion Llama 3 2 3B-a Serchhitetts-Rerecce
4 We with
use the original code provided by the authors to run their models on Compositional-AP default parameters. This means that the models perform augmented inference on the test set with rotations and transpositions over all symmetries, in addition to color permutations and example
https:/ppatfoomm..o www.:mmwwoppii.iimmmmmmm http:///wwwmmmm.omommmoioooc/ee/ www.///sss//meeeellloiiiiii
25

---

Published as a conference paper at ICLR 2026
shuffling. Candidate pruning is further applied with minimum probability of 0.1. For models evaluated with test-time training, we follow the authors' one-epoch LoRA adaptation on the study examples of the test data repeated 48 times with the same augmentations described before. LoRA targets the attention and MIP modules, as well as the emheddings, with r r= 64, a = 16, and dropout set O. The models are trained with batch size of 16, gradient accumulation set to cosine learning rate of x 10-4 (with x 10 for embeddings), and a warmup ratio of 0.25. The resulting weights are then used for inference with the same default settings described earlier.
E ADDITIONAL RESULTS
In this section, present additional results for the experiments conducted in this study. First, present additional qualitative results related to the model predictions on the standard few-shot learning and the systematicity task. Figures through iillustrrate representative episodes from the standard few-shot learning task. Model predictions are shown adjacent to each query, with results for GPT-4o and Gemini 2.0 Flash corresponding text-only prompts. Across all three episodes, the model trained using MLC consistently predicts the correct output grid. In contrast, GPT-4o and Gemini 2.0 Flash frequently fail identify the correct transformation-eiitee of the transformed object or incorrectly predicting its final position. Notably, o3-mini successfully and 7 but fails on the example in Figure Figures and BaaddDhighee highliggh episodes from the systematicity task. As shown, all general -prppos LLMs fail to produce accurate transformations, often misplacing the transformed object within the grid. In contrast, the model trained via MLC consistently predicts the correct transformation.
misrepresenting the shape
predicts the correct output for the episodes in Figures and
Response rates, As outlined in Section the general-purpose LLMs we evaluate are instructed to present their final output grid predictions using the keyword "output:", followed by two-dimensional array of size 10 x 10 the format "I[...II". Responses that do not conform to this expected pattern excluded from subsequent analyses and are not included in accuracy calculations. Table provides an overview of the proportion of valid responses for each model. In the standard few-shot learning setting, all models demonstrate very high valid response rates, exceeding 99%. However, in the systematicity task, a slight decrease in valid responses is observed for Gemini 2.0 Flash when additional visual input (text+image) introduced, with the rate falling 94.09%. More significantly. GPT-40 exhibits notable drop in valid response rate to 77.24% under multimodal conditions. We hypothesize that this decline may be attributed the increased context length resulting from the additional image input.
Error Analysis. As described Section 5.2 we analyze the models' predictions and compare them with common failure modes. Table S shows the percentage of each eπror type described in Section 5.2 across models. For errors related to the models predicting a primitive or level-1 transformation instead of the desired level-2 transformation composition, further illustrate which specifie primitive or level-1 transformation was applied in Table Specifically, this table shows whether the primitive
Table 4: The proportion of valid responses generated by the different models reported for the standard three-shot learning task and the systematicity task. For general-purpose purpose LLMs, valid responses must contain the string "output:", followed by two-dimensional 10 x 10 array of the form "[[...]"
Valid Respomes (3-Shot)
Responses (Systematicily)
GPT-4o
99.95%
99.40%
image
99.80%
77.24%
Gemini 2.0 Flash
99.92%
99.74%
image
99.51%
94.09%
03-mini (low)
100%
100%
Lluna: 1lamm-3.223BB ReARC
3.2-3BBRARCCC
100%
100%
training
100%
Misttal-ccO--MMntton-------
100%
100%
test test-time training
MLC (ours)
100%
100%
26

---

Published as a conference paper at ICLR 2026
Table 5: Error distribution by error category across models. Values denote the percentage (%) of prediction errors assigned to each error category.
Model
Format
No Transform
Primitive
Level-1
Invalid Position
Invalid Shape
Other
GPT-4o
7.71
image
22.91
4,09
4.26
59.84
Gemini 2.0 Flash
0.26
1.56
11.41
22.41
5.73
58.32
0.30
image
7.60
0.72
9.05
15.52
4.60
61.83
03-mini (low)
0.00
5.06
30.86
13.08
0.79
49.31
0.91
L.amm--3.2-3B.. 3.2.BBBReARC
55.28
test-fime training
0.44
67.13
15.52
Mistral Mimi--on SB-Full
0.00
0.54
53.41
28.89
0.12
17.03
0.01
test time training
0.00
2,47
5.05
71.55
13.31
7.62
0.00
MLC (ours)
0.05
3.34
9.11
3.02
70.57
transformation applied was based on the object's shape, color, or neighboring object. Similarly, the table illustrates which specific level-1 transformation composition was applied.
Training on static data. In addition to the model trained via MLC on stream of dynamically
changing visual interpretation grammars, as described in Section 3.2 we adopt the approach of Lake (2019) and train transformer-Dos baasod encoder-decoder on a dataset governed by fixed visual grammar (referred to basic seq2seq). This means that the anlicator-transfrmmaae whole dataset. For instance, yellow objects translate one step downward, then this applies to all data samples across the dataset. Instead of episodes with few-shot examples, this dataset comprises individual input-output grid pairs, where the objective to predict the output grid corresponding to a closely resembles standard training approach.
mappings static across
given input grid. This
We construct a dataset of 1,300 grid pairs, partitioned into 1 1,260 training samples, 20 validation samples, and 20 test samples. Samples represent primitive transformations, well as level-1 and level-2 transformation compositions. As with our other experiments, the test set includes level-2 transformation compositions that were not observed during training-only their constituent components and level-1 compositions were seen during training. For instance, the test set might include transformations composed of shape-based downward translation, color-based horizontal reflection, and neighbor-based upward extension. However, only their decomposed elements have been shown during training.
The model is trained for 200 epochs on the dataset using the parameters specified in Section While it successfully fits the training data (with an accuracy over 99%), it fails generalize to the -utoo--distributinn test set, achieving a test accuracy of 0.0%. This demonstrates that traditional model training, sample by sample, does not encourage systematic generalization to unseen composi-
Table 6: Percentages of errors falling into each primitive and level-1 transformation error category.
Primitive Transformations
Level-1 Transformations
Model
Shape
Color
Neighbor
Shapo+Color
Shigh-Neighbor
Color-Neighbor
GPT-4o
2.09
1.56
0.93
4.68
2.10
0.92
image
1.97
1.57
0.66
2.65
1.01
0.42
Gemini 2.0 Flash
4.22
4.70
16.17
4.05
2.19
image
2.96
3.66
03-mini (low)
16.67
13.07
1.12
11.16
0.99
0.93
Lluma 3,2-3B 22-3B-BB REARC
14.22
6.48
4.43
2.80
1.74
0.24
training
0.27
5.16
0.76
35.68
22.64
Minitron
7.46
best toiiing
MLC (ours)
0.16
1.08
2.32
5.98
2.16
0.97
27

---

Published as a conference paper at ICLR 2026
tions. Instead, systematicity requires training procedure with examples over dynamically varying interpretation grammars, as described in Section 3.2
F USE OF AI ASSISTANTS
We used GitHub Copilot for parts of the project's 'ss code, and ChatGPT for minor language revisions.
28

---

Published as a conference paper at ICLR 2026
Table Summary of dataset statistics across different dataset splits, each determined by distinct random seed. Listed are the number of episodes in the training, validation, and test sets. Additionally, the final query transformation compositions (level 2) are reported for both the training and evaluation datasets. The rightmost column details the frequency of each basic geometric transformation present the training dataset.
Data Split
No. Episodes
Query Transformations
Basic Transformations
Set
No.
Type
Composition
Transformation
Freq.
Train
82908
faunhation-ccleciioo-oooadi
red coloring
35828
Val
8546
relistion--oattooo-cctttnooi
orange coloring
35819
Test
8546
down translation
23398
cnnelaaion-nnaation--cloiii
right translation
27021
Train
redlction-colorinn-ctttiioo
leftward extension
22140
seed 1860
refection--oiaiionncoloinnnn
upward extension
21806
connlation-coloring-eeeee
ew. rotation
19551
ccw. rotation
19394
translation cranlatioo-roiaaioo-ccttttsosooo
horizontal reflection
21967
Test
vertical reflection
21800
Train
83481
translation www.arionnnaattiineeeeeeeeee
red coloring
27603
Val
8259
translation hat.laaooo-rccccccccoooottttiooo
orange coloring
27525
Test
8260
down translation
31385
reflectin=-clorrngge coloringsexteesiion
right translation
36126
Train
translation--eelll www.reeleeeeoneeeeeeeiiiie
leftward extension
26501
seed 1870
translation -rooaationncoooorinng
upward extension
25913
+edlection+coolriing
rotation
15421
translation tramslaiom-eooloring--- ecoloring+eetteeeeeee
cow. rotation
15283
rolation motation+coloria www.ooooooooooeteenaate
horizontal reflection
22366
Test
reflection eootaioon-colorinng
vertical reflcction
22320
Train
80035
translation rranshtionncoolrinngeee colerennn
red coloring
25850
Val
9982
translation-roiian cot.aoooooiiiiitotttsiiooooo
orange coloring
25832
Test
9983
translation aetaaaaiooortatiioonoooorngg
down translation
31385
inflctionrroatiion-cttessss
right translation
36126
Train
-reflecciionnncooorinn
leftward extension
24821
seed 1880
connlation-rcflectio---c-o
upward extension
24147
tramslation iaailatoo--rdleeccioooooaaooo
cw. rotation
19734
rotation-colorinneectteeooo
cew. rotation
19594
horizontal reflection
16331
Test
reflection cerertiooncoooinnggetttniooooo
vertical reflection
16285
Train
80557
translation coolorng iextensiniitttssiionn
red coloring
30227
Val
9721
orange coloring
30255
Test
9722
rotation coloring
down translation
23279
right translation
24789
Train
reflection teeerneii
leftward extension
26483
seed 1890
upward extension
26277
cw. rotation
13949
refection--oiation-ccolinnng
cew. rotation
13831
horizontal reflection
26329
Test
tramslation twaaainneo
vertical reflection
26252
29

---

Published as a conference paper at ICLR 2026
Table 8: Statistics of the dataset version including more diverse transformations. Listed are the number of episodes in the training, validation, and test sets. Additionally, the final query transformation compositions (level 2) are reported for both the training and evaluation datasets. The rightmost column details the frequency of each basic geometric transformation present in the training dataset.
Data Split
No. Episodes
Query Transformations
Basic Transformations
Set
No.
Type
Composition
Transformation
Freq.
Train
85528
red coloring
18376
Val
5472
orange coloring
18627
Test
5473
yellow coloring
18961
trunsation-rroaaiioo--olliit
green coloring
18491
Train
aclletiioo-ollecnn--eccioooooon
1-step left translation
6471
rellectioo-rtaaaiioosccoornnnn
2-step left translation
3671
1-step right translation
7942
2-step right translation
5438
1-step up translation
6780
Test
seed 1860
2-step up translation
4051
1-step down translation
6686
2-step down translation
4022
leftward extension
11742
rightward extension
12071
upward extension
11801
downward extension
12909
rotation
20797
ccw. rotation
20799
horizontal reflection
23536
vertical reflection
23413
30

---

Published as a conference paper at ICLR 2026
Study Examples
Query
Predictions
Few-Sbot Examples
63-mini
GPT-4o
Gemini 2.0
Orypa Gild
Target
Llama
Mistral
Figure 5: An example of the few-shot learning task. Models are provided with three study examples that demonstrate the transformation that needs to inferred for the final input grid. Model predictions displayed the right.
Study Examples
Query
Predictions
Examples
Esd
Cape
Cald
Coll
Gold
Dupe:CCo
Ohipa God
Oepa G11
Duper Cas
Orraa Ci1
Query Targer
MLC (ours)
Lianaa
Mistral
Figure 6: A second example of the few-shot learning task. Models are provided with three study examples that demonstrate the transformation that needs be inferred for the final input grid. Model predictions are displayed to the right.
Study Examples
Query
Predictions
Few:Shot Examples
Ouraa
Ouiput
clst
63-mini
GPT-4o
Geminl 2.0
(ours)
1.lama
Mistral
Figure 7: third example the few-shot learning task. Models are provided with three study examples that demonstrate the transformation that needs to be inferred for the final input grid. Model predictions are displayed the right.
31

---

Published as a conference paper at ICLR 2026
Study Examples
Query
Predictions
Primitive Transformations
Orgga CSL
Capa Cast
Composition
(level-2)
MLC (ours)
Llama
+ Cut CN
(TTT)
Transformation Compositions (level=
Ospa
Csst
Query Target
GPT-4o
Mistral
Figure 8: An episode from the systematicity task. Given set of study examples comprising primitive transformations and level-1 transformation compositions, models are asked predict the output grid for previously unseen level-2 transformation composition. Predictions of different models are presented to the right.
Study Examples
Query
Predictions
Primitive Transformations
Oppa Det
Composition
(level=2)
ML. (omrs)
Lluma
Transformation Compositions
Depedaal
Query Tarpet
GPT-4o
Mistral
Mistral
Figure 9: Another episodes from the systematicity task. Given a set of study examples comprising primitive transformations and level-1 transformation compositions, models asked predict the output grid for previously unseen level-2 transformation composition. Predictions of different models are presented the right.
32

---

Published as a conference paper at ICLR 2026
Study Examples
Figure 10: An exemplary visual input used in the multimodal prompt for the 3-shot learning task.
Study Examples
Courade
Ponohwrss.c
Paaral
Figure 11: An exemplary visual input used in the multimodal prompt for the systematicity task.
33

---

Published as a conference paper at ICLR 2026
Text-Only 3-Shot Prompt
### Task Description:
You must solve an abstract visual reasoning task by identifying geometric transformations (c.g., rotation, translation, color changes, etc.) applied to objects within a 10x 10x10 grid.
To infer the correct geometric transformation, are given series of 3 pairs of input-output examples. Each example pair consists of:
An input grid: 10x 10 list of lists (2d array), where each element integer (0-9).
A corresponding output grid: a 10x10 list of lists (2d array) that has undergone transformation based on a specific geometric rule.
For the prediction you need to understand the transformations displayed in the provided examples and apply them to the final input grid.
### Your Task:
Analyze the example pairs infer the transformation rules applied each input grid.
2. Identify how these transformations are applicd to generate the output grids.
Apply the deduced transformations to final input grid.
4. Output the correctly transformed 10x 10 grid.
##!! Output Requirements:
Return only the final output grid.
Do not include extra text, explanations, comments.
The output must be formatted exactly as: 'output: [[...]]]
The output grid must 10x of lists containing only integers between 0 and (inclusive).
. Do not include unnecessary line breaks or additional text beyond the specified format.
### Input Format:
You will receive the following data:
Study examples: list 3 few-shot example pairs, formatted
"example input [I...II, example output I: II...II,, ... example input II...II, example output II...II'
2. Final input: A single 10x10 list of lists on which must upply the inferred transformation(s).
Your goal is to determine the correct transformation and return the final output grid.
### Input:
Study examples:
example input <2-222iimeeen dimenssiil array representing the input grid of example 1> example output 1: <2 <2-dimensional array representing the output grid of example 1>
example input <2-dimensional urray representing the input grid of exumple 3> example output 3: <2-dimensional array representing the output grid of example 3>
Final input: <2-dimensional array representing the final query input grid
The prompt used for the few-shot experiment when instructing LLMs in (text-only) mode. Text enclosed in sharp brackets < <.. is replaced by the actual examples.
Figure 12:
34

---

The prompt used for the few-shot experiment when instructing LLMSs in (text+image) replaced by the actual examples. Additionally, the Published as a conference paper at ICLR 2026
Text+Image 3-Shot Prompt
### Task Description:
You must solve an abstract visual reasoning task by identifying geometric transformations (c.g.. rotation, translation, color changes, etc.) applied to objects within a 10x10 grid.
To infer the correct geometric transformation, you are given series of pairs of input-output examples. Each example pair consists of:
An input grid: 10x10 list of Lists (2d array). where each element is integer (0-9).
A corresponding output grid: a 10x10 list of lists (2d array) that undergone transformation based a specific geometric rule.
For the prediction you need to understand the transformations displayed in the provided examples and apply them to the final input grid.
### Your Task:
Analyze the example pairs infer the transformation rules applied each input grid.
2. Identify how these transformations are applied to generate the output grids.
3. Apply the deduced transformations the final input grid.
4. Output the correctly transformed 10x 10 grid.
#M! Output Requirements:
Return only the final output grid.
Do not include any extra text, explanations, comments.
The output must formatted exactly as: "ourput: I[...III
The output grid must be 10x10 list of lists containing only integers between 0 and (inclusive).
Do not include unnecessary line breaks or additional text beyond the specified format.
### Input Format:
You will receive the following data:
Study examples: A list of 3 few-shot example pairs, formatted
"example input [[-..], example output 1: [[--.]], ... example input 3: [[-..], example output 3: [[...]]'
2. Final input: single 10x 10 list of lists on which you must apply the inferred transformationss.. Image input: Additionally, you receive an image that visualizes the few-shot example pairs and imput query.
Your goal is to determine the correct transformation and return the final output grid.
### Input:
Study examples:
example input I: <2-dimensionnl array representing the input grid of example >>>
example output 1: <2-dimensional array representing the output grid of example
example input 3: < 2-dimensional urray representing the input grid of example 3>
example output 3: 2-dimennsional array representing the output grid of example 3>
Final input: <2-dimensional array representing the final query input grid
Figure 13:
mode. Text enclosed in sharp brackets < .... > model is provided with the image in Figure 10
35

---

Published as a conference paper at ICLR 2026
Text-Only Systematicity Prompt
### Task Description:
You must solve an ahstract visual reasoning task by identifying geometric transformations (c.g., rotation, translation, color changes, etc.) applied to objects within a 10x grid.
To infer the correct geometric transformation, you are given a series of 12 pairs of input-output ex- amples. Each example pair consists of:
An input grid: 10x10 list of lists (2d array). where each element is integer (0-9).
A corresponding output grid: a 10x10 list of lists (2d array) that has undergone transformation based on a specific geometric rule.
The first 6 example pairs demonstrate primitive transformations based on the object's color, shape, the presence of an additional object. For instance, objects of certain color within the 10x input grid might undergo translation, while objects of a certain shape (distinct numerical pattem) are being rotated.
The latter example pairs involve composite transformations. meaning multiple transformations are applied simultaneously. For instance. for objects that have the appropriate color and shape, both a translation and rotation upplied simultaneously
For the final prediction need to understand and further combine the transformations displayed in the provided examples and apply them to the final input grid.
#н Your Task:
Analyze the example pairs infer the transformation rules applied each input grid.
2. Identify how these transformations might combine to generate the output grids.
Apply the deduced transforaations to final input grid.
4. Output the correctly transformed 10x 10 grid.
## Output Requirements:
Return only final output grid.
Do not include any extra text, explanations, comments.
The output must be formatted exactly as: 'output: [[...]]]
The output grid must 10x10 list of lists containing only integers between 0 and (inclusive).
* Do not include unnecessary line breaks or additional text beyond the specified format.
### Input Format:
You will receive the following data:
Study examples: list 12 study example pairs, formatted
"example input [I-..I,, example output 1: II...II, ... example input 12: [[...II, example output 12: II...II'
2. Final input: single 10 list of lists on which must apply the inferred transformation(s).
Your goal is to determine the correct transformation and return the final output grid.
##H Input:
Study examples:
example input <2-dimension urray representing the input grid of example >> example output I: <2-dimensional array representing the output grid of example
example input 12: <2-dimensional array representing the input grid example
example output 12: <2ddimensia urray representing the output grid of example 12>
Final input: <2-dimensional array representing the final query input grid
Figure 14: The prompt used for the systematicity experiment when instructing LLMs in (text-only) mode. Text enclosed in sharp brackets < ... is replaced by the actual examples.
36

---

Published as a conference paper at ICLR 2026
Text+Inage Systematicity Prompt
### Task Description:
You must solve an abstract visual reasoning task by identifying geometric transformations (c.g. rotation, translation, color changes, etc.) applicd to objects within a 10x grid.
infer the correct geometric transformation. you are given series of pairs of input-output ex- amples. Each example pair consists
An input grid: a 10x10 of lists (2d array), where each element an integer (0-9).
corresponding output grid: 10x10 list of lists (2d arry) that has undergone transformation based specific geometric rule.
The first 6 example pairs demonstrate primitive transformations based object's color, shape, the presence additional object. For instance, objects of certain color within the 10x10 input grid might undergo translation, while objects of certain shape (distinct numerical patterm) being rotated.
The latter 6 example pairs involve composite transformations, meaning multiple transformations applied simultaneously, For instance, for objects that have the appropriate color and shape, both translation and rotation applied simultaneously.
For the final prediction you need to understand and further combine the transformations displayed in the provided examples and apply them the final input grid.
### Your Task:
1. Analyze the example pairs infer the transformation rules applied each input grid.
2. Identify how these transformations might combine to generate the output grids.
3. Apply deduced transformations final input grid.
4. Output the correctly transformed 10x10 grid.
#tHi Output Requirements:
Return only the final output grid.
Do not include any extra text, explanations, or comments.
output must be formatted exactly as: 'output: [[...]]'
The output grid must be 10x 10 list of lists containing only integers between 0 and 9 (inclusive).
Do not include unnecessary line hreaks additional text beyond the specified format.
### Input Format:
You will receive the following data:
1. Study examples: A list of 12 study example pairs, formatted as:
'example input I: [I...I], example output I: [[...], ... example input 12: [I...I), example output 12: I[...]I"
2. Final input: A single 10x10 of lists on which you must upply inferred transformationss)
3. Image input: Additionally, you receive image that visualizes the 12 study example pairs and the final input query.
Your goal determine the correct transformation and return the final output grid.
##H Input:
Study examples:
example input I: <2-dimensional arry representing the input grid of example 1>
example output I: d-dimennso array representing the output grid of example 1>
example input 12: <2ddinensional array representing the input grid of example 12> example output 12: <2-dimensional array representing the output grid of example 12
Final input: <2-dimensional array representing the final query input grid
Figure 15:
The prompt used for the systematicity experiment when instructing LLMs in (text+image)
mode. Text enclosed in sharp brackets < replaced by the actual examples. Additionally, the model is provided with the image Figure
37


Reviews:
[{"review_id": "VJ7O2Ra2jx", "reviewer": "Reviewer_14KV", "summary": "The paper proposes Compositional-ARC: a dataset for evaluating LLMs capacity for systematic generalization & compositionality with a methodology similar to that in Lake & Baroni (2023), but in a visual domain. The authors provide a number of empirical results to support their MLC-based approach.", "questions": "What is your explanation for imperfect color accuracy in systematicity evaluation part of the proposed model? While 97+ is quite high, compared to other models, this component seems to be lacking.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is focused on an important and relevant task\nThe paper is clearly written and is a pleasure to read.\nThe results are promising.", "weaknesses": "The main weaknesses, in my view, are:\n\n- Slightly misleading performance comparisons. The authors compare a model specifically trained for this ARC subtype with more generalist models, or with models designed to perform well on ARC (but not on the new Compositional ARC). Still, Compositional ARC is a reasonable subset of all possible \"ARC-lke\" problems, so this is not a disqualifying consideration.\n\n- Low practical applicability. While the question of compositionality is fundamental, the authors substantially narrowed the scope of an already artificial dataset (ARC). This limits the potential impact of the paper and introduces the risk of creating models that are increasingly detached from real-world applications. Still, such research has lots of value and so again, this is not a disqualifying weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wkZbvFOrvW", "reviewer": "Reviewer_minw", "summary": "The study proposed a dataset to study the compositionality of transformers. In some way, the proposed Compositional-ARC dataset is similar to the ARC dataset, which contains pixel objects on a grid. The task is to infer the transformation given an input grid. The ground truth transformation depends on the “cues” in the input, such as the shape or color of the object. The authors study whether transformers can learn to compose transformations indicated by the composed cues, and with several extra few-shot composition examples. The authors train a transformer that can achieve this task almost perfectly. However, given the simplicity of the dataset and task, it is hard to infer the actual compositionality ability of transformer models.", "questions": "For my main concerns, see \"weakness\".\n\nWhy is the task considered meta-learning? Is it fair to say this task is supervised learning with online adaptation? The task may be considered as learning a \"meta-skill\", which is not the same as the common definition of \"meta-learning\".", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The composition ability is one of the most fascinating properties of LLMs. The authors study this property in a controlled toy setting. A small transformer model, after training, can solve this toy task perfectly.", "weaknesses": "I have major concerns that the setup and analysis are too simple, such that the conclusions and implications are limited. \n\n**Simplicity**\n\nUnlike the original ARC task, the number of possible rules underlying the transformations is small. All the primitive transformations are about shift, rotation, and extension. Given 3-shot examples in some of the testing cases further simplifies the task. It is expected that a small transformer can learn this task almost perfectly. It is hard to say how much of such “systematic generalizability” is in real-world data and models. Though the authors do not provide any theoretical analysis, it may not be difficult to formulate this task such that all the transformation rules are orthogonal in some linear space, then the compositionally is simply a consequence of retrieving the cues and adding them together. The failure of this task in the frontier model could be due to the number of examples provided in the context. Since frontier models are not trained on the task directly, it may not be a fair comparison between them and MLC (author’s).\n\n**overly claimed LLM failures**\n\nThe authors overly emphasize the failure case of compositionality and generalizability in LLMs (e.g. Line 115). The cited studies, Ismayilzada et al., 2025; Dziri et al., 2023, use more or less simple synthetic settings. On the large scale, frontier LLMs can consistently show surprising compositionality and generalizability. For example, a user can ask an LLM to answer a question in a specific tone or with other constraints. In a way, it is because of the compositionality and generalizability of LLM that alignment, such as RLHF, can be feasible and effective. Authors should introduce and discuss what is known about LLM’s compositionality and generalizability, rather than introducing only the pessimistic side under simplified unrealistic settings.\n\n**Clearity**\n\nIn the introduction, the authors motivate the study with “systematic generalization” in human and AI. Without a definition or more context, it is hard to know what “systematicity” refers to. How is “systematic generalization” different from naive “generalization”?\n\nSimilarly, meta-learning for compositionality (MLC) is not defined and formalized. Providing general setups of MLC would be helpful.\n\n**Structure**\n\nThe subtitle “Task setup” in the section Background is confusing. Overall, the background section is cumbersome, with too many details on a small number of studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7q9IhT03AB", "reviewer": "Reviewer_6Vzw", "summary": "Introduces Compositional-ARC, a new dataset and benchmark to study systematic generalization in abstract spatial reasoning. Focuses on geometric transformations - translation, rotation, reflection, extension and color - and they are applied on 2D grid objects\nUses meta-learning for compositionality (MLC) framework from Lake & Baroni (2023) paper to extend the approach from linguistic domain to the spatial reasoning domain. The evaluations includes both general-purpose and ARC-specific LLMs.", "questions": "- What is the training procedure (meta-learning part). It will be good to have some details in this paper as well, instead of having to refer to the other paper\n- Line 266 - can you explain the two special tokens and how they mark the boundaries?\n- Auxiliary task - What is intuition behind this? In ablation this is adding more value, which is quite strange\n- Some figures are overloaded (Fig. 3) to understand all the actions happening easily", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Systematic generalization is important challenge and the motivation is well explained\n- The dataset is carefully formulated and explained in a good detailed manner\n- Good empirical analysis and evaluations and ablations are reported as well", "weaknesses": "- Core algorithm (MLC) is adopted unchanged; and is applied to a new domain. It is more of a new dataset/benchmark track? Need some clarification on the position of the study\n- The Scope is quite narrow and restrictive - 10x10 grids, rotation angles, number of objects, occlusion, boundaries\nIt is unclear how insights generalize to richer or naturalistic visual domains (e.g., CLEVR-style reasoning\n- Evaluation fairness? - Many comparisons with LLMs use text-formatted grids as prompts. These models are not optimized for symbolic grid reasoning, so the Claim that small MLC model can outperform SOTA LLMs is needs more validation\n- Analysis of failure modes? Where it fails and why\n\n\n[1] CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d6oBmnhluf", "reviewer": "Reviewer_USJo", "summary": "This paper is a straightforward application of a meta-learning method --- developed by Lake and Baroni --- to the task of compositional transformations in 2D grid patterns.  The authors created a dataset of such transformations that involves training and testing \"episodes\" using different transformation grammars, and Lake and Baroni's meta-learning approach to train a small transformer network to perform such compositional transformations.  This method produces a model (MLC) that outperforms much larger state-of-the-art LLMs (ones that are not specifically trained on this dataset / task).", "questions": "In Figure 1, for level 2, the ordering of primitive transformations matter, correct?  What is this ordering? \n\nThe task addressed in this paper has several constraints: 10x10 grids, 10 colors, only one or two objects, a small repertoire of transformations, and restrictions on those transformations (e.g., rotations are only 90-degrees).   To what extent does MLC rely on (overfit to) these constraints, and to what extent could it generalize beyond them?  If MLC fails on larger grids or on larger numbers of objects, does that mean that it hasn't actually learned any general compositional abilities?  If these restrictions were relaxed, how would that affect the training / success of the network?  A discussion of all this, and discussion of possible future research directions, would be helpful.  \n\nFor Figure 3, it would be useful to specify what the primitive transformations are that are illustrated.  This would make the set-up clearer to readers. \n\nThe authors state: \"we consider a visual interpretation grammar that associates visual indicators (object  shape, color, or proximity to an indicator object) with specific geometric transformations\".  -- Why refer to these patterns as \"visual\", when the model is given the grids in text format?  Why not just say these are 2D patterns?  It's not clear that the MLC model \"perceives\" objects, shapes, etc., or even that it needs to in order to perform the task correctly.  In discussing ARC, Chollet is adamant in stating that ARC tasks are not *visual* tasks, but rather 2D pattern-recognition tasks.  It would be useful to comment on this in the paper. \n\nWhy does the model need the \"auxiliary copy task\" to perform well?  \n\nThe authors state, \"\"we split the data into 82,908 training, 8,546 validation and 8,546\ntest episodes.\"   -- Why use these values?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written, understandable, and offers a solid contribution to the literature on meta-learning and systematicity / compositionality  in AI models.", "weaknesses": "While the results are impressive, it's not clear from the paper how such results could be incorporated in models that use this ability for compositional reasoning in less narrow tasks.  For example, how would the authors use these results in service of, say, solving ARC tasks?  How could the MLC model be used as a component of such a system, or as a way to help train such a system?   It would be useful if the authors discussed this point -- where to go from here.   The authors state \"\"Our findings suggest that MLC presents a promising direction for enabling systematic generalization in language models across diverse domains.\"   Please say more about what these directions are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)