Im-Promptu: In-Context Composition from Image Prompts
Authors: Bhishma Dedhia
Michael Chang
Jake Snell
Thomas L. Griffiths
Niraj Jha
Venue: NeurIPS 2023 poster (2023)
Keywords: in-context learning
compositionality
generative models
TL;DR: We show an isomorphism between in-context learning over natural language prompts and implicit understanding of composition rules over visual entities

Abstract: Large language models are few-shot learners that can solve diverse tasks from a handful of demonstrations. This implicit understanding of tasks suggests that the attention mechanisms over word tokens may play a role in analogical reasoning. In this work, we investigate whether analogical reasoning can enable in-context composition over composable elements of visual stimuli. First, we introduce a suite of three benchmarks to test the generalization properties of a visual in-context learner. We formalize the notion of an analogy-based in-context learner and use it to design a meta-learning framework called Im-Promptu. Whereas the requisite token granularity for language is well established, the appropriate compositional granularity for enabling in-context generalization in visual stimuli is usually unspecified. To this end, we use Im-Promptu to train multiple agents with different levels of compositionality, including vector representations, patch representations, and object slots. Our experiments reveal tradeoffs between extrapolation abilities and the degree of compositionality, with non-compositional representations extending learned composition rules to unseen domains but performing poorly on combinatorial tasks. Patch-based representations require patches to contain entire objects for robust extrapolation. At the same time, object-centric tokenizers coupled with a cross-attention module generate consistent and high-fidelity solutions, with these inductive biases being particularly crucial for compositional generalization. Lastly, we demonstrate a use case of Im-Promptu as an intuitive programming interface for image generation.

Im-Promptu: In-Context Composition from Image Prompts
Bhishma Dedhia', Michael Chang", Jake C. Snell", Thomas L. Griffiths Niraj K. Jha'
'Department of Electrical and Computer Engineering, Princeton University
Department of Computer Science, University California Berkeley
Department of Computer Science, Princeton University
'Department of Psychology, Princeton University (bdedhia, js2523, tomg, jhal @princetinnnnn edu, mbchang@berkeley...o edu
Abstract
Large language models are few-shot learners that can solve diverse tasks from a handful of demonstrations. This implicit understanding of tasks suggests that the attention mechanisms over word tokens may play role in analogical reasoning. In this work, we investigate whether analogical reasoning can enable in-context composition over composable elements of visual stimuli. First, we introduce a suite of three benchmarks test the generalization properties of visual in in-context learner. We formalize the notion of an maalogy-based in in-context learner and use it design mneaallernnng framework called Im-Promptu. Whereas the requisite token granularity for language is well established, the appropriate compositional granularity for enabling in-context generalization visual stimuli is usually unspec- ified. To this end, we use Im- Im-Prompfu to train multiple agents with different levels of compositionality, including vector representations, patch representations, and object slots. Our experiments reveal tradeoff's between extrapolation abilities and the degree compositionality, with con-compositional representations extending learned composition rules to unseen domains but performing poorly combinato- rial tasks. basch representations require patches to contain entire objects for robust extrapolation. At the same time, object-centric tokenizers coupled with cross- tross-attention module generate consistent and high-fidelity solutions, with these inductive biases being particularly erucial for compositional generalization. Lastly, we demonstrate a use case of Im-Promptu as an intuitive programming interface for image generation.
Introduction
No thought can formed that isn't informed by the past: or. more precisely, we think only thanks to analogies that link our present to our past.
D. Hofstadter and E. Sander, Surfaces and Essences [1]
Humans represent complex concepts by combining and organizing simpler concepts [2, 3]. This endows us with an uncanny skill for constructing an unlimited number of concepts by composing relatively small set of previously learned basic building blocks, an ability more formally called compositional generalization. For example, we can generate sentences by combining dictionary of words in different ways and solve wide range of mathematical problems using small set of basic arithmetic operations and variables. In the case the visual world, objects form naturally
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

composable entities and object-centriy primitives can be flexibly combined using abstract syntactic rules to yield novel and counterfactual scenes.
The growing body of work in object-centric learning methods [4, 5, 6] enable the extraction of latent object representations from perceptual scenes. However, slot composition methods have been limited to ad-hoc composition rules based on hard-coding slots into concept libraries [7, 8J. A central challenge in learning generative model that explicitly models the compositional grammar of the visual world is that the seemingly unlimited number of composition rules that undergird real-life visual entities prevents the explicit specification of each rule. For example, a warm spring day consists of blooming flowers and freshly verdant expanses, but come autumn, the same scene stipulates the presence of golden meadows carpeted with fallen leaves. State-of-the-ae
rext-to-image models [9, 10. 11, 12] circumvent this bottleneck by grounding images in natural language. This aligns the representations of entities in an image to the compositional structure of language. While the visual quality of these models is stunning, they often require engineering complex language prompts to elicit the desired output. They also suffer from limited understanding of object relations [13]. Moreover, studies [1 [14] have shown that infants as young as six months old can extract compositional structure from sequences of visual events, even before they acquire language, Such compositional abstraction independent of language faculty exhibited by humans remains elusive for contemporary models.
Large language models (LLMs) [15, 16, 17, 18, 19, 20). on the other hand. demonstrate implicit composition skills. Pre-trained on large text cor- pora, these models have impressive emergent few-shot learning capabilities. They can solve novel tasks on the fly without gradient updates by following examples shown in the instruction prompts, a process known in-context learn- ing. Under the hood, in-context learning uses the inherently compositional nature of the input text to implicitly infer the task structure and how to apply the task structure to query [21]. In the most simple form, the prompt-basc few-shot be formalized as analogy solving of the type A : B :: C D, where A represents an example task, B represents the ex- ample solution, and C denotes the query. The generated solution is the analogy completion D. Analogies over compositional elements, there- fore, provide a direct isomorphism between in- context learning for language and learning implicit composition rules for visual objects, as depicted in Fig. 1. While LLMs use pre-specified tokenizers [22, 23, 24], the compositional granularity for implicit visual understanding is unspecified. On the other hand, slot methods induce object-level representations but fail to learn composition rules. In this work, we then ask:
Translate Englich to Feanch
Nodiity blue abject
abject:
aut ottec Lostze nac
pluch glraae to gizaae palocha
pepperaint es zanthe poivrtes
Support Prompta
Lange Langsage
Viaual In-Gontext
Model
Seaznes
cheepe 1le Econsgu)
Quary
learning paradigm
Figure I: LLMs perform in-context understanding of a task from a few examples in an analogical manner (left). We use analogy-makinn itly understand the composition rules over visual stimuli made of like entities (right).
to implic-
Can analogy-solving enable in-context generalization over object-centric compositional elements of visual entities?
While many linguistic corpora have been extracted from the Internet for text-based tasks, to the best of our knowledge, no datasets exist for measuring in-context composition capability from visual prompts. To this end, we introduce a suite of three diverse benchmarks in Section 3 built on top of rich combinatorial spaces systematically test image-basee in-context learning abilities: (a) 3D Shapes [25], (b) BitMoji Faces, and (c) CLEVr Objects [26]. If the attention over word tokens enables analogical understanding in LLMs,
transfer this mechanism We formulate a unifying framework in Section 4.1 to formalize this notion. We present a visual analogy- based meta-learming algorithm called Im-Promptu in Section 4.2 and use it to train agents with a cross-attention module. How does one tokenize visual inputs? To answer this, we model several generative agents in Section 5, encompassing a continuum ranging from strong compositional learners using object-centric abstractions to non-compositionall monolithic representation learners. Extensive experiments presented in Section 6 show the dependence of robust in-context learning for visual stimuli on object-centric slots as visual tokenizers and cross- cross-attention. We demonstrate the use
visual analogies?

---

of our visual analogy framework in task extrapolation (Section 6.2), combinatorial generalization (Section 6.3). and generating counterfactual images (Section 6.5).
2 Related Work
In this section, we discuss prior works and draw connections to relevant research areas.
2.1 Object-Centric Learning: This growing body of work targets unsupervised decomposition of scenes into set of object-centric vectors from raw perception data. A common approach involves modeling an autoencoder consisting of a latent factorization of object slots that are independently decoded to reconstruct the scene [4, 5, 6, 27, 28, 29]. These methods effectively segment scenes into representations of underlying objects but do not learn the flexible combination of object represen- tations. On the other hand, the authors in [7] replace the independent slot decoding bias with an Image-GPT [30] and demonstrate novel slot composition. While this work largely answers how to compose a given set of visual entities, their slot selection is performed manually through organization of slots into clustered libraries. In our work, we use a meta-learning approach to help the learner implicitly understand the composition task. Recently, the authors of [31] recast the iterative slot refinement problem as finding of fixed points through implicit differentiation to improve the stability and computational cost of the slot-attention (SA) technique. The authors of [32] use specific inductive biases to disentangle object contents from spatial information that enables relational reasoning. Our work produces a more generalized composition beyond spatial relations.
2.2 Analogical Reasoning:
The ability to make analogies has been shown to have important implica- tions in various domains, such as problem-solving, creativity, learning, and counterfactual thinking [1, 33, 34, 35, 36). Various models have been proposed to explain the underlying mechanisms of analogy-making
Popular symbolic approaches include ARGUS [37]. Transfer Frames [38], and Structural Mapping Engine [39], connectionist models include LISA [40] and Star-1 [41]. and models that occupy a middle ground like Copycat [42]. Contemporary deep-learning models [43, 44, 45] attempt solve Raven's Progressive Matrices and Bongard Problems, types of visual analogies present in IQ tests. The authors of [46] propose a generative deep visual analogy-making model based on parallelogram regularizer the latent space to disentangle variation factors and traverse transformation manifolds. Most similar to us, a recent work [47] reformulates the analogy-making task as Inpainting in an image grid and proposes an auto-encoder pre-training task to that end.
2.3 Compositional Generative Models: Prior works [48, 49] use energy energy-based models to represent labelss composable factors of variation of scene. However, these methods use concept labels or are grounded in text and are limited to few concept composition operations. A follow-up work [50] describes unsupervised energy-basse based concepts; however, the re-composition performed via manually-picked concepts. Recent works in text-to-image models use language grounding to induce compositionality. DALL-E [9] uses language modeling over the joint space of the text and image tokens to facilitate image generation. The latest text-to-image models [10. 11, 12] use diffusion modeling to guide the image generated from the text. A growing body of work [51, 52, 53] explores text-grounded controllable image editing via prompt engineering, where most parts of the image are preserved, but specific objects are re-composed.
2.4 Modular Representations and Attention: A swath of recent works proposes neural methods that combine modular primitives with recurrent mechanisms and cross-attention. The authors of [54] propose the Recurrent Independent Mechanisms (RIMs) to learn weakly coupled dynamical systems with independent compositional neural modules that communicate via a sparse attention bottleneck. Another work [55] uses object-centric abstractions as 'files' to store factorized declarative knowledge and the dynamics of the environment. A follow-up work [56] introduces neural abstractions of the classical Production Systems framework to learn rule-based templates, thereby facilitating compositional dynamics of the environment that can quickly adapt to novel stimuli. Most recently. the authors in [57] propose an iitention-based architecture to learn in-context causal relationships between entities.
3 Benchmarks
Humans can contextualize past visual stimuli in the face of everyday experiences and even use them for imagining counterfactuals. Having seen an oak table, we can, with little difficulty, look at a metal armchair and understand the concept of an oak armehair. How do we then systematically test such
3

---

Primitive Tasks
Composite Tasks
proupt
Support Proupe
Query
solution
Support Prospe
Query
Solution
Task: Rodify blue object green object
3D Shapes
Task:
floor blue fleor
<and> hair style
BitMoj1 Faces
Tank: [Add red rubber cylindar cands Delate metal cube cands
Task: [Add gray antal sphara to the acens]
GLEVY Objects
Delete blue rubber cube]
We use combinatorial image spaces to set up visual in-context learning tasks. Each task requires the generation solution from query and supporting examples. The latent description of the task is descrihed within the square parentheses. Left: Primitive tasks modify composable element in isolation. Right: Composite tasks combine primitives to test the combinatorial generalization skills of the learner.
Figure 2:
abilities of visual in-context learner? In this work, posit that an answer to this question lies in combinatorial visual spaces with controllable factors of variation, which provide a semantically rich, simple, scalable, and composable avenue for setting up such tasks. We create suite of three benchmarks (Fig. 2) from compositional image creators that include (a) 3D Shapes [25], (b) BitMoji Faces, and (c) CLEVr Objects [26]. Each benchmark has split primitive training tasks and out-of- of-diitribuutiin test tasks. The primitive tasks (Fig. 2, left) constitute an analogy where a single composable element of the visual stimulus is modified in isolation. The description of each benchmark with its primitive set of tasks is given next:
3.1 3D Shapes Image Prompts: This dataset consists of static scenes of various objects lying on colored floor in front of colored wall viewed from different orientations. To up primitive tasks, we consider the four properties orientation), where each property take multiple values within domain. For each such task, the latent relation r(.) for the analogy prompt takes a property E and modifies it from a source domain value largel value. This benchmark comprises 80000 such tasks with maximum of four supporting image pairs for each instance and with a roughly equal split across properties.
(object color, wall color, floor color, scene
3.2 BitMoji Faces Image Prompts: BitMoji is an avatar creator service for social media users that allows them to create intricate cartoon faces. We queried the BitMoji API [58] to collect faces with four underlying dynamic elements P (skin tone, facial hair style, hair style, eyewear), }. Much like the previous case, primitive tasks are generated by intervening a source value of property and modifying demonstrations available per task.
target value. We populate training set with 80000 tasks, with four
3.3 CLEVr Objects Image Prompts: CLEVr is a popular visual question-answwe the visual component consisting of multiple objects lying in scene. We use the CLEVr rendering engine to set up primitive tasks that include adding and deleting the same object across various scenes. In this fashion, generate 60000 tasks with three examples per task.
dataset with
composable image space allows the concurrent modification of individual visual elements. One can construct composite tasks (Fig. 2, right) out of a combination of primitives shown in the training set. A k-composite task denoted as R² (.) 71 o 72 o ..... o Tx where rı, ........ Th E R. We provide details of the k-compositeness test of each of our agents in Section 6.3. Finer details about each

---

benchmark, along with visual examples from every task, are detailed in Appendix A. We will release the complete dataset of primitive and composite tasks upon the publication of this article.
4 Methods
To learn compasitional structure over visual stimuli, we begin by formulating a unified mechanistic interpretation of an in-context compositional learner using insights from design elements of LLMs.
4.1 In-Context Learning as Analogy Completion: A general compositional in-context learner M..a.o()) can be formalized via the following three key components:
An encoder E.).) that maps the input space X' to the compositional space - Es(.)
X' c
An executor ........) that maps compositional entities from example tasks, example solutions, and query task to the query solution _ (...,.)) : C x C x C - C
A decoder D.(.) that maps the compositional space back to the input space - Da(.):
C++ X'
Coupling the above functions together yields the learner:
M Da (To (Eg()), Eg(()) X X' X
(1)
The inclusion of (.(. and (.(.) is crucial for compositional abstraction since the imput space may not be a priori composable in the case of images. Beyond this, we do not place any strong parametric constraints on the model. Let R denote a task set. For any task r E R. define the task analogy ( over
input pairs 31, 22 as I1 : r((x)) :: z2 r(x2) & A: : C: D. An effective in-context learner over R then satisfies the following property [59]:
V R, xç-p(r.x)) [C (D; o (A, (.BBB)) C)) Se
(2)
Here, P denotes the distribution of the analogies resulting from the input space X' and task r(.), Z() denotes loss metric, and é is an error bound. The above property simply states that a good in-context learner solves analogies defined over a task set within some finite error bound. In Appendix B, we describe LLMS as an instantiation of this framework.
4.2 Im-P Im-Promptu Learning: Pre-training LLMs on large-scale text data leads to the emergence of highly transferable internal representations that allow them to adjust to novel tasks by simple priming few demonstrations. However, it is still unclear whether the underlying structure of the training data, architecture, or inductive biases instilled by choice of pre-training learning algorithms cause these intriguing properties emerge. How can we encourage the emergence of such a general-purpose representation for visual data? In this work, we take an explicit approach based on the formalism presented in the previous section. Having established the desideratum (Bq. 2), we set up learning procedure called Im-Promptu that trains the model to generate analogy completions from image prompts. To this end, given a task set R, in each minibatch, we sample an analogy A A: B :: C:: D that follows a latent primitive r(.) E R. Given a loss criterion, L, we make the following stochastic gradient updates of the model parameters:
fatt1 (+1 0++1 = (0' a
V oxt go (D, at, (A, B,
(3)
where the step size is a hyperparameter. The full Im- Promptu algorithm is laid out in Algorithm 1.
5 Learning Agents
In this section, we explore various modeling choices for visual in-context learner ranging from simple pixel-space rules to object-centric learners (OCLs).
5.1 Pixel Baseline: This non-parametric baseline manipulates the pixel space using simple addition rule. We denote the support prompt as A B, the query as C, and the solution as D. Then the Pixel baseline predicts ò as the outcome of a linear transformation, ò = C (B - A).
5.2 Monolithic Learner: Our first learning agent, inspired by [46], is entirely non-compositional and uses latent monolithic vectors to execute the image prompts as follows:
V = fa (Aea(A), eg (B))), D da (hg (MV (C')))
(4)
5

---

Algorithm Im-Promptu Learning Algorithm
Task set R, Step Size a Initialize parameters 0, at Φ of model M
Require:
while not done do
Sample primitive task r(.) Sample N input image pairs X X2 Pirain
for in 1... N do
Sample analogy Ç* 21 r(æ)) m² r(x)) - A:B:C:D end for Update (0.a,0) a,, xb N Vo... EM()()) end while
The encoder (-)-) and decoder d.(-) are convolutional and deconvolutional networks, respectively. The inference network (-) and executor network h-(-) modeled multi-layered perceptrons. Here, [] denotes the concatenation operator. This architecture is detailed in Appendix D.
Inspired by recent work on inpainting for In-Context learning [47], this agent (Inpaint., visualized in Appendix D.2) is modeled via an autoencoder architecture that fills in missing patches of an image. The architecture is composed of a discrete variational autoencoder (dVAE) that discretizes the input patches and a Vision Transformer [60] auto-encoder to impute missing values. While the monolithic agent separately encodes the components of the visual analogy, this model jointly represents them as 2 x 2 image grid. The model is pre-trained using the masked autoencoder reconstruction task [61]. Subsequently, at inference time, the analogy solution represented as missing image in the grid. Details have been given in Appendix D.2.
5.3 Inpainting Model:
5.4 Object-Centric Learner (OCL): This agent, as depicted in Fig. 3, is a strongly compositional learner that uses object-centric inductive biases. The encoder and decoder of the framework in Section 4.1 are parametrized using Slot Attention Transformer (SLATE) [7] (see Appendix C.2) as follows:
(-) _ S (J2 (.)), Do(.)
A 98 AF GPT (.)
(5)
Context Encoder Transformer (CET) & .......) (.) with interleaved layers of: (a) cross-attention on the context slots and (b) self-attention, induces sparse modifications on the query slots to complete the analogy.
N = EA)) SP.N (B), Si. = EC)
SP.N = CET (query =S.. keys, values =S SP.N
(7)
The latent sequence 2n predicted by Image-GPT mapped by the dVVAE back the pixel space D. 2D = 99 GPT (PPPPP D 9ō AVAE (2p)
(8)
The CET forward pass is shown in Appendix D.3.
5.5 Sequential Prompter: This (Seq., visualized in Appendix D.4) is an ablation over the OCL that replaces CET with an LLM--like sequence prompt obtained from simple concatenation of slots of images A, B, and C, injected with position embeddings Pi.
(.,..., .) - [ SP,N, SC,] +
(9)
While the OCL explicitly encodes the entire context into a fixed-length slot sequence via the CET, the ablated agent models a longer concatenated sequence.
5.6 Patch Learner: This agent straddles the compositionality extremes of Monolithic Learners and OCLs, and uses image patch abstractions. The SA module from OCL is ablated to get discrete sequence of 4 x 4 patch latents as the degree of compositional granularity Ex(.) A S (.VAE(.). A larger image implies longer sequences for this modeling choice.
6 Experiments
In this section, we set up experiments to answer (1) whether analogical reasoning enables in- context generalization, (2) does such generalization require key mechanisms from LLMs, namely compositionality and attention, (3) if compositionality granularity for visual tokens, and (4) what are the limits to the generalization.
key ingredient, what the correct.

---

Solutiion
8
Ed(.)
Ta(:1.3..
(a(.)
Context Encader
cross Attonttoo
Transforner
SLATE encoder
SLATE decoder
C
Trage-GPT
prompt
Impar proupt
Figure 3: The OCL uses compositional object vectors to answer image prompts. (1) The encoder (.). is pre-trained SLATE [7] encoder that uses dVAE to obtain a latent discrete sequence for the input image and runs SA [6] over the sequence to extract object-centric slots. (2) The CET modifies query slots SINV via cross-attention over the support prompt slots 1: S (3) The modified slots
are used to prompt an Image-GIP [30] that outputs the latent discrete sequence of the task solution D that is decoded by the dVAE decoder.
We trained each agent the primitive of tasks across the three benchmarks using Im-Promptu (Section 4.2). The SA-based object-centric encoders used in the OCL and Sequential Prompter
6.1 Training Setup:
pre-trained as SLATE autoeneoder, as described by the authors in [7]. On the other hand, the Patch Learner was trained completely from scratch. The loss metric L used to train the agents cross-entropy (CE) loss between the true latent sequence Zn and the predicted solution ZD obtained from Image-GPT, i... Compromptu CE(ZD, ZD).
In addition to the above loss, the dVAE was trained using the mean- seaan- error (MSE) loss over the raw pixel space to yield the full loss function C Limpromptx + M MSE(D, D). For inference, answers of the transformer-baset agents were sampled from the Image-GPT decoder using top-k nucleus sampling [62]. Hyperparameters for training and inference have been laid out in Appendix E.
In the first distt---istiibutiinn paradigm, we tested the ability of agents to apply learned rules to different domain pairs. In order to do this, we held out 20% of source-target pairs (see Section 3 for the definition of source and target) from the training primitives and tested the ability of agents to generalize from learned rules to unseen source-targe pairs. For example, the object color property the 3D Shapes benchmark can take unique values and 10 9 90 source-target combinations. Thus, agent was trained only 90 0.8 72 pairs for object-hue primitives. We made sure that each value in domain set was shown at least once as either the target or the source.
6.2 Primitive Task Extrapolation:
Fig. 4(a) plots scores of different agents across benchmarks against two key metries: (1) MSE (lower is better) that quantitatively compares the construction against the ground truth and (2) Fréchet inception distance (FID, lower better) score measure the perceptual quality of the composition. We make several interesting observations:
(R1.1) Simple pixel manipulation produces implausible outputs. The pixel baseline has poor MSE scores (~~ 138 for 3D Shapes, ~ 895 for BitMoji) with clearly apparent artifacts in the output. While linear pixel transformations is unable to model complex dependencies (see Fig. ((b).dd) under 'Pixel').
sufficient when object is independent of the remaining entities,
(R1.2) Inpainting model struggles to generalize beyond i..d. inputs. We observed that the Inpainting model is able to cogently complete analogies on the i..d. validation set (see Appendix F.I for outputs). However, when tested extrapolated primitives, it only fills in the high-level structure (see Fig. 4 (b), (c)) and, in the case of more complex inputs, produces entirely incoherent outputs (Fig. 4 (d)). We posit that top-down method like inpainting pre-ordains larger datasets for improved generalization and, as such, suffers from sample inefficiency.
(R1.3) Monolithic Learners are reasonably effective at generalizing from learned rules on structured stimuli. These agents have the lowest MSE scores on Shapes and BitMoji Faces, both spatially consistent datasets. However, the output deteriorated over CLEVr Objects where the
7

---

MEndiNG:
1452
7.
-uppport Prompt-
Quary
FO::
(a)
Ieaent
MMR 14277.
Inseler.
MSE: 240 104.0
treart IIE 200 106
(b)
Ground Truth
Pixel
Inpainting
1.0
M5E
Seg. Pronpter
3D Shapes
BitMoji Faces
CLEVr Objects
Ground Truth
Pixol
Inpainting
Monolithie
Patch
OCL
(c)
(d)
Mizsing
objeete
Poned
Figure 4: Primitive task extrapolation: (a) Plot showing scores where the left y-axis and the bars represent the MSE scores, while the right y-axis and the dotted line denote the FID scores. (b)-(d) Comparison of agent solutions with the dotted red circle representing anomalies with respect to the ground truth.
latent scene configuration is diverse. The edges of the generated shapes and lighting details were significantly blurred, and occluded objects were often fused together (FID 92, worse than Pixel baseline, Fig. 4(d) under "Monolithic')..
(R1.4) Patch Learner and Sequential Prompter benefit from additional examples. For these agents, the addition of context examples was markedly apparent from the significant MSE drops for shot (see Fig. 4(a)). We posit that since these models lear from longer contexts, additional examples are particularly effective in regularizing the output sequence manifold of Image-GPT.
6.3 Composite Task Extrapolation: In this generalization paradigm, we probed the compositional generalization capabilities of the agents. A k-composite task (k: 2 2, see Section 3) is an unordered composite function of k primitives. The value of ls varied across each benchmark, where higher values of & yield visually complex contexts. To solve such task, the agent must identify the underlying primitive relations and combinatorially modify the query.
In Fig. 5(a), we note (R1.1) and (R1.3) from the previous setup in this extrapolation paradigm as well, but we make additional key observations next:
(R2.1) The effect of object-centric biases is strong. This is indicated by the widening MSE gap between the OCL and monolithic agent for increasing values of & across all three benchmarks (see Fig. (()) Moreover, the OCL and the Sequential Prompter generate outputs with consistent perceptual quality (lower FID scores).
(R2.2) Monolithic and Patch Learners apply shorteuts. The monolithic agent tends to produce Frankenstein-liee image, as if superposition of context images (see Fig. 5(d)-(f) under 'Monolithic"), In the case of the CLEVr Objects dataset, both the Monolithie and Patch Learners often simply filled in 'blobs' agnostic to the color and shape of objects (Fig. 5(f), Appendix F4), with missing shadows leading to lower MSE scores but poor visual quality.
6.4 Analysis: Next, discuss key elements required for in-context composition.
(A1) In-context generalization critically hinges on compositional granularity. As evident from (R1.2), monolithic representations suffice for reliably learning primitive composition rules over simple contexts. However, they suffer from fidelity issues when the visual space becomes more complex. Beyond primitive generalization, patch abstractions effective when the underlying space

---

(a)
(d)
Qway
W... PILL 200. 10.89
WONEA MIL 213 FD 3L33
175
325
FID
3D
Fatch
Fec. Proeppac
20
10
(b)
(e)
Peot SIIKAN (N60 nas 200-34
FINT LLD (RER 2111E FD.
Fegginr Priegg
peacy
Bi BitMojs Faces
70
252
10
350
Seg. Tammplas
(c)
k
(E)
PHLT TMER ENN IN,
HAMIN 19364
FRRE REORLYD FO.
CLEVE Objects
100
MSS
50
k
Figure 5: Composite task extrapolation: (a)-(c) Plots showing scores across benchmarks where the left y-axis and the bars represent the MSE scores, while the right y-axis and the dotted line denote the FID scores. & denotes the level of compositeness. (d)-(f) Comparison of agent solutions with the dotted red circle representing anomalies with respect the ground truth.
has object-patch equivalence, as evident from the CLEVr Objects dataset (Fig. 5(c)). However, even with object-patch equivalence, the Patch Learner omits the inter-object dependencies (c.g., object occlusions and shadows). It further fails to learn complex and diverse objects spanning multiple patches reliably (does poorly on the BitMoji Faces dataset with diverse facial components). Slot- based compositionality with the CET not only enables the generalization of the learned primitives but also enables combinatorial generalization to composite tasks that require abstracting out the composition rule over each entity in isolation as well as composing the resultant interplay between them to generate globally consistent output.
(A2) Encoding the context via cross-attention leads to sample efficiency, Longer slot sequences in the Sequential Prompter require multi-shot contexts across all benchmarks. The sample inefficiency was particularly observed on the CLEVr Objects dataset that uses six slots and, hence, much longer sequences. Cross-attention via CET is essential for implicit inference, enabling the learner encode specific objects from the context.
'Counterfactual Prompt Engineering' with in-Promptu: LLMs have spurred interest in several priming techniques [63, 64] that can 'engineer' the model into following user instructions. While this natural-language-baate based programming language provides an intuitive interface for tasks that have fundamental degree of language abstraction (writing code, summarization, symbolic

---

Pronpe: (add netal blue sylinder).
Pronge: (add rubber purple gube)
Pronpt: (add rubber red ErelAndee)
Proepti (add rubber silver cube).
(a)
OCL
OCC
OCT
OCL
Troept: releee
steoppt fadd: rabber aed cylinded ielighsly sptexee
(b)
mosppt
ophinent
DALL-B
DALL-B
Figure 6: Scene creation from object components: (a) OCL trained via Im-Promptu is used to generate scene of objects. Image-Prompis engineered using existing assets reliably generate the scene object by object. (b) The same scene is generated via language grounding and in-painting features of DALL-E [9]. Object properties are distorted and specifying the location of objects tedious via language.
manipulation, etc.), it is not a priori obvious that textual descriptions should also scaffold image generators. Coming up with natural-language prompts for complex visual entities is often tedious and unintuitive. Instead, explaining over them't To this end, Im- Im-Promplu provides an avenue for generating counterfactuals via image prompts. In Fig. 6(a), we demonstrate the creation of scene from its object components the OCL. As point of comparison, in Fig. 6(b), using image-inpainting and natural language prompts with DALL-E [9] yields an unreliable output with distorted objects. More practically, the OCL reduces human effort significantly in rendering images with desired properties by using existing visual assets. The procedure to generate scenes from OCL can be found in Appendix G.
image is naturally easier from its entities and composition
7
Conclusion
This work investigated whether amalogy-solving over visual entities. We first designed a new test suite. We probed if attention and compositionality. key ingredients in LLMss, also play crucial role in the visual domain. We transported these ingredients from language to model several agents. Our experiments showed that non-compositional agents do not generalize well beyond primitive task extrapolation. Even with compositionality baked in, the performance of patch-based learners depends on idiosyncrasies found that object-centric biases consistently facilitate an implicit understanding of composition rules and generate outputs with global semantic consistency. Our ablation of the CET suggested that cross-altention plays crucial role, just like language models, which posits the importance of analogy and attention in understanding the underpinnings of in-context learning. Future research challenges include collecting labels, improving object-centri, inductive biases for real-life entities, and designing better prompting techniques. We hope that our work will spur further research on visual in-context learning.
enable in-context compositional generalization
the visual structure. However,
primitives from photo-realistic graphic engines or human
Acknowledgments and Disclosure of Funding
We would like to thank Sreejan Kumar for helpful discussions and insights throughout the course of this work. The experiments reported in this article were performed on the computational resources managed and supported by Princeton Research Computing at Princeton University. This project was funded by NSF Grant No. CCF-2203399 and ONR Grant No. N00014-18-1-221 acknowledges financial support from the Schmidt DataX Fund Princeton University made possible through a major gift from the Schmidt Futures Foundation.
JS gratefully
"No one is an artist unless he carries his picture in his head before painting it, and is sure of his method and composition" Claude Monet
10

---

References
[1] Douglas R. Hofstadter and Emmanuel Sander. Surfaces and Essences: Analogy as the Fuel and Fire of Thinking. Basic Books, 2013.
[2] Brenden M. Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One Shot Learning of Simple Visual Concepts. Cognitive Science, 33, 2011.
[3] Brenden M. Lake, Tal Linzen, and Marco Baroni. Human Few-Shot Learning of Compositional Instructions. CoRR, abs//abls.a./ 1901.0401.4487 2019.
[4] Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised Scene Decomposition and Repre- sentation. CoRR, 00522333.00877, 2019.
[5] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-Object Representation Learning with Iterative Variational Inference. CoRR, abs/1903.004500 2019.
[6] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object- Centric Learning with Slot Attention. CoRR, abs/2006.15055 2020.
[7] Gautam Singh, Fei Deng. and Sungjin Ahn. Illiterate DALL-E Learns to Compose. CoRR, abs/2110.1 1405, 2021.
[8] Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-Centric Slot Diffusion. CoRR, abs/2303. 108444 2023.
[9] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and lla Sutskever. Zero-Shot Text-to-Image Generation. CoRR, abs/2102. 12092, 2021.
[10] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents. CoRR, abs/2204.061255 2022.
[11] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. CoRR, abs/2205/1 1487, 2022.
[12] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. CoRR, abs/2112.10752. 2021.
[13] Gary Marcus, Ernest Davis, and Scott Aaronson. A Very Preliminary Analysis of DALL-E2 2. CoRR. abs/2202.13807, 2022.
[14] Fei Xu, Elizabeth S. Spelke, and Sydney Goddard. Number Sense in Human Infants. Develop- mental Science, 8(1):88-100) 2005.
[15] Tom B. Brown et al. Language Models are Few-Shot Learners. CoRR, abs/2005. 2020.
[16] Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https::/gitee
com com//xinggfllll//eett... www.ssoorrtrrrrrorrrreeee. jjxx May 2021.
[17] Aakanksha Chowdhery et al. PaLM: Scaling Language Modeling with Pathways. CoRR, abs/2204.02311, 2022.
[18] Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-1; Technical Details and Evaluation. /sttps:/uppoads--- webflow. com/ /000/44440038844457880000000 611 1111892422a09111000offff_eeeeee2 pape
paper pdf, 2022.
[19] Teven Le Scao et BLOOM: Paramear Open-Access Multilingual Language Model. CoRR, 0bs/2211.051000 2022.
[20] OpenAl GPT-4 Technical Report. CoRR, abs/2303.08774,

---

[21] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger Language Models do In-context Learning Differently. CoRR, abs/2303.03846, 2023.
[22] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural Machine Translation of Rare Words with Subword Units. CoRR, abs/1508.07991 000800 2016.
[23] Mike Schuster and Kaisuke Nakajima. Japanese and Korean Voice Search. In IEEE International Conference Acoustics, Speech and Signal Processing, pages 5 5149-5152, 2012.
[24] Taku Kudo and John Richardson. SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. CoRR, abs/1808.06226, 2018
[25] Chris Burgess and Hyunjik Kim. 3D Shapes Dataset. https//gghhhbbbcommddeppmniillls dataset/, 2018.
[26] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. CoRR, abs/1612.06890, 2016.
[27] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the Binding Problem in Artificial Neural Networks. CoRR, abs/2012.05208, 2020.
[28] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations. CoRR, abs/1907.13055 2019.
[29] Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. GENESIS- V2: Inferring Unordered Object Representations without Iterative Refinement. CoRR, abs//90 2021.
[30] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative Pretraining From Pixels. In Hal Daumé III and Aarti Singh, editors, Pro- ceedings of the 37th International Conference on Machine Leurning, volume I 119 of Proceedings of Machine Learning Research, pages 1691-1703. PMLR, 13-18 July 2020.
[31] Michael Chang, Thomas L. Griffiths, and Sergey Levine. Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation. CoRR, 0bs//220.000877 2022.
[32] James C. R. Whittington, Rishabh Kabra, Loic Matthey, Christopher P. Burgess, and Alexander Lerchner. Constellation: Learning Relational Abstractions over Objects for Compositional Imagination. CoRR, abs/2107.11153, 2021.
[33] J. P. Guilford. Creativity. American Psychologist, 5(9):444, 1950.
[34] Dedre Gentner. Structure-Mapping: A Theoretical Framework for Analogy. Cognitive Science, 152):155- 1983.
[35] Arthur B. Markman and Dedre Gentner. Structural Alignment in Analogy and Similarity. American Psychologist, 48(1):40, 1993.
[36] Keith J. Holyoak and Paul Thagard. Mental Leaps: Analogy in Creative Thought. MIT Press, 1995.
[37] W. R. Reitman. Cognition and Thought: An Information Processing Approach. John Wiley and Sons, 1965.
[38] Patrick H. Winston. Learning by Creatifying Transfer Frames. Artificial Intelligence, 10(2):147 172, 1978.
[39] Brian Falkenhainer, Kenneth D. Forbus, and Dedre Gentner. The Structure-Mapping Engine: Algorithm and Examples. Artificial Intelligence, 41(1):1-63 63, 1989.
[40] Jeffrey F. Hummel and Keith J. Holyoak. Distributed Representations of Structure: A Theory of Analogical Access and Mapping. Psychological Review, 104(3):4774666, 1997.
12

---

[41] Graeme S. Halford, William H. Wilson, Jian Guo, Ross W. Gayler, Janet Wiles, and J. E. M. Stewart. Connectionist Implications for Processing Capacity Limitations in Analogies, pages Ablex Publishing, 1994.
[42] Douglas R. Hofstadter. Fluid Concepts and Creative Analogies: Computer Models of the Fundamental Mechanisms of Thought. Basic Books, 1995.
[43] Stefan Depeweg, Constantin A. Rothkopf, and Frank Jäkel. Solving Bongard Problems with a Visual Language and Pragmatic Reasoning. CoRR, abs/1804.04453 2018.
[44] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu, RAVEN: A Dataset for Relational and Analogical Visual rEasoNing. CoRR, abs/1903.02744 2019.
[45] Felix Hill, Adam Santoro, David G. T. Barrett, Ari S. Morcos, and Timothy Lillicrap. Learning to Make Analogies by Contrasting Abstract Relational Structure. CoRR, abs/1992.001200, 2019.
[46] Scott E. Reed, Yi Zhang, Yuting Zhang. and Honglak Lee. Deep Visual Shalogy-Makking In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[47] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei A. Efros. Visual Prompting via Image Inpainting. CoRR, 0bss/20998.00447, 2022.
[48] Yilun Du, Shuang Li, and Igor Mordatch. Compositional Visual Generation with Energy Based Models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6637-6647. Curran Associates, Inc., 2020.
[49] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional Visual Generation with Composable Diffusion Models. CoRR, abs/2206.01714, 2022.
[50] Yilun Du, Shuang Li, Yash Sharma, Joshua B. Tenenbaum, and Igor Mordatch. Unsupervised Learning of Compositional Energy Concepts. CoRR, abs/21 11 1.03042, 2021.
[51] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Crmmpt-to-Prompee Image Editing with Cross Attention Control. CoRR, http/2208.062252 2022.
[52] Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan. UniTune: Text-Driven Image Editing by Fine Tuning an Image Generation Model Single Image. CoRR, abs/2210.09477, 2022.
[53] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-Based Real Image Editing with Diffusion Models. CoRR. abs/2210.09276 2022.
[54] Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent Independent Mechanisms. CoRR, abs/1909. 10893, 2020.
[55] Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blundell, Yoshua Bengio, and Michael Mozer. Object Files and Schemata: Factorizing Declara- tive and Procedural Knowledge in Dynamical Systems. CoRR, abs/2006. 1100066.6222 16225, 2020.
[56] Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael Mozer, and Yoshua Bengio. Neural Production Systems: Learning Rule-Governed Visual Dynamics. CoRR, abs/2103.01931 2022.
[57] Nan Rosemary Ke, Silvia Chiappa, Jane Wang, Anirudh Goyal, Jorg Bornschenn Melanie Rey, Theophane Weber, Matthew Botvinic, Michael Mozer, and Danilo Jimenez Rezende. Learning to Induce Causal Structure. CoRR, abs/2204.04875, 2022.
[58] Snap Inc Bitmoji. https://wwwt http://www..tttmoe bitmoji.i comp///.ooomm..mooommmmmm///.. accessed 2023,
[59] Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What Can Transformers Learn In- In-Context? A Case Study of Simple Function Classes. CoRR, abs/2208 000666 2022.
13

---

[60] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR, abs/2010.11929, 2021.
[61] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. CoRR, abs/21 11.0677,, 2021.
[62] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The Curious Case of Neural Text Degeneration. CoRR, hes/1904.09755 2020.
[63] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. CoRR, abs/2107. 2021.
[64] Simon Willison. In Defense of Prompt Engineering. http:://simonvillsson net//023/
hebb/t/innddeeesee www.oopp---o-oooooooooooooooooet grrmmtt-engineeringg/ February 2023.
[65] Jason Tyler Rolfe. Discrete Variational Autoencoders. CoRR, htss/1609.02200, 2016
[66] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learning. CoRR, abs/1711 2017.
14


Reviews:
[{"review_id": "wjhslBG1Vm", "reviewer": "Reviewer_4dpY", "summary": "The paper investigates whether analogical reasoning can enable in-context compositional generalization over visual entities. The authors construct three benchmarks to test the compositional generalization on visual analogy-making, including 3D shapes, BitMoji Faces, and CLEVR. The authors also present a visual analogy-making algorithm called Im-Promptu. The authors test the proposed method on the constructed benchmarks with various visual representations, including vector representations, patch representations, and object slots. The experiments demonstrate the tradeoffs between extrapolation abilities and the compositionality degree of the visual representations.", "questions": "1. What is the fundamental difference between the proposed Im-Promptu learning in Section 4 and visual analogy-making?\n\n2. What is the main technical contribution of this paper?\n\n3. The method of this paper works better on spatially consistent datasets, so how to reflect the statement \"produces a more generalized composition beyond spatial relations\" in the paper?\n\n4. Why are there some results with lower MSE and higher FID?", "limitations": "The authors do not discuss the limitations of the proposed method.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The paper constructs three benchmarks to test the compositional generalization on visual analogy-making, including 3D shapes, BitMoji Faces, and CLEVR.\n\n2. The paper conducts experiments using various visual representations, including vector representations, patch representations, and object slots, and provides insights into the impact of the compositionality degree of the visual representations on compositional generalization. The results demonstrate the effectiveness of the object-centric representation for the compositional visual analogy-making.", "weaknesses": "1. The technical novelty of the proposed framework (Im-Promptu) is limited. The formulation in Section 4 is similar to visual analogy-making [1,2]. It is inappropriate to rename it as in-context learning or Im-Promptu learning, if there is no fundamental difference. Section 5 lists several model variants for visual analogy-making by composing existing modules. It is unclear what is the paper's contribution to the methodology.\n\n2. The constructed benchmarks are all synthetic domains with little variance. This is a weakness considering that previous works [2,3] have already studied similar tasks using natural images.\n\n3. The experimental results are inadequate. The authors only evaluated the proposed framework on the self-constructed benchmarks, while previous works [1,2,3] have introduced several benchmarks for visual analogy-making. It is unclear whether the proposed framework generalizes to other realistic benchmarks.\n\n4. The paper lacks reference to important related works [2,3].\n\n[1] Reed, Scott E., et al. \"Deep visual analogy-making.\" Advances in neural information processing systems 28 (2015).\n\n[2] Sadeghi, Fereshteh, C. Lawrence Zitnick, and Ali Farhadi. \"Visalogy: Answering visual analogy questions.\" Advances in Neural Information Processing Systems 28 (2015).\n\n[3] Bar, Amir, et al. \"Visual prompting via image inpainting.\" Advances in Neural Information Processing Systems 35 (2022): 25005-25017.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and constructive feedback. We respond to the questions and concerns below.\n\n> What is the fundamental difference between the proposed Im-Promptu learning in Section 4 and visual analogy-making?\n\nWe agree that our work bears similarity to visual analogy-making in the sense of being able to complete an analogy, having observed a transformation over context examples. However, our work is interested in asking the broader question of what enables the in-context compositional generation over composable attributes in the visual domain. Such a compositional learning ability has been well investigated for the language modality. This question can then be disentangled into the interplay of three sub-questions - (1) What composable abstractions can help learn implicit composition? (2) Which objective optimizes learning such rules? (3) How to model the composition from entities? In this context, visual analogy as an auto-encoding reconstruction only answers the second question by formulating a reliable optimization objective via analogy solving for learning transformations. Im-Promptu, on the other hand, supports broader compositional learning by unifying these components under a common mechanistic framework (See Equations 1 and 3).\n\n>The technical novelty of the proposed framework (Im-Promptu) is limited. The formulation in Section 4 is similar to visual analogy-making [1,2,3]. It is inappropriate to rename it as in-context learning or Im-Promptu learning if there is no fundamental difference.\n\nFirst, Ref. [2] focuses on solving intelligence test types of analogies where the solver has to classify the correct answer from a bunch of different options, and as such, doesn't support generation. Ref. [1] uses visual analogy as a proxy to learn input-invariant transformation vectors that can traverse continuous manifolds (e.g., geometric transformations, animation frames) via simple rules. The authors demonstrate limited generalization to new inputs but with latent transformations within the training distribution. In fact, our monolithic agent is directly instantiated from their MLP formulation (See Equation 5.3 in Ref. [1]). Our experiments reinforce the findings of [1] and demonstrate the ability of such representations to generalize to single-order transformations (Sec. 6.2) at test time. However, these don't extend to a higher-order compositional generation where multiple composite attributes are modified and therefore don’t support higher-order visual understanding (Sec. 6.3).\n\nWe would like to thank the reviewers for pointing us toward the in-painting technique used in [3]. We have now implemented the technique with the slight modification of using an MAE -VQVAE ensemble (instead of MAE-VQGAN). The pretraining involves masked auto-encoder (MAE) reconstruction on 2x2 analogy grids described in [3]. See the global response section for exact quantitative and qualitative results. Even though past works have found in-painting as a simple and effective self-supervision task for learning general-purpose representations, we find that it cannot generate compositions beyond the training distribution (see attached pdf for generated outputs).\n\n> Section 5 lists several model variants for visual analogy-making by composing existing modules. It is unclear what is the paper's contribution to the methodology.\n\nWe believe that our work takes a crucial step towards solving visual analogies that can understand and compose higher-order relations than the ones they were trained on, a notion largely unexplored in visual analogy models. To this end, we make several key technical contributions:\n\n1. Benchmarks for the systematic understanding of in-context learning in the visual domain. This is important to understand the underpinnings of in-context learning, the emergence of which is less than completely understood in the language domain.\n2. Extensive exploration of model abstractions across the compositionality continuum. While we use existing modules to learn compositional abstractions, it is not a priori straightforward that strong compositional abstractions should also support in-context compositional generation. Ours is the first work that utilizes these modules with a range of context aggregation techniques to understand the implications of various design choices on generalization.\n3. From a more practical standpoint, we provide empirical training recipes for OCL that could be utilized to scale it while doing in-context learning over real data.\n\n> The constructed benchmarks are all synthetic domains with little variance. This is a weakness considering that previous works [2,3] have already studied similar tasks using natural images. It is unclear whether the proposed framework generalizes to other realistic benchmarks\n\nPlease see the global response section under the heading “Concern surrounding the absence of natural images”. While other works [3] that use realistic benchmarks show decent performance across task-specific metrics, the degree of generalization that can be obtained via in-painting is, as such, unmeasurable. In fact, our experiments show that the techniques of both Refs. [1] and [3] are not amenable to compositional generalization.\n\n> The method of this paper works better on spatially consistent datasets, so how to reflect the statement \"produces a more generalized composition beyond spatial relations\" in the paper?\n\nHere we refer to the aforementioned papers' limited ability to generalize to spatial “positional” relations by separating the position of an object from its content. Our method, in contrast, can be used to simultaneously modify the geometry of the scene with other compositional attributes. We apologize for the confusion.\n\n>  Why are there some results with lower MSE and higher FID?\n\nThis is possible if the model generation gets the larger structure of the output right but poorly captures finer aspects like shadows and edge transitions.\n\n\n\n\n\n\n\n\n\n"}, {"review_id": "fLRJS6Liib", "reviewer": "Reviewer_Lzbc", "summary": "This work investigates whether analogical reasoning can enable in-context composition over composable elements of visual stimuli. First, they introduce a suite of three benchmarks to test the generalization properties of a visual in-context learner. They formalize the notion of an analogy-based in-context learner and use it to design a meta-learning framework called Im-Promptu. To this end, they use Im-Promptu to train multiple agents with different levels of compositionality, including vector representations, patch representations, and object slots. Lastly, they demonstrate a use case of Im-Promptu as an intuitive programming interface for image generation.", "questions": "I have the following questions:\n1. Since the number of in-context demonstrations is one important parameter in in-context learning fields, is there any ablation study for the number of demonstrations? \n2. The input and output of these three tasks are all images. One question is can the text be the output of the task (e.g., image classification)?\n3. In the proposed method, there are some components involved. What's the difference if we use models from scratch instead of pre-trained?\n", "limitations": "Compared with in-context learning in language models, this paper only considers three proposed computer vision tasks. Thus the setting of in-context learning may be limited.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In-context learning is a novel learning paradigm recently. While in-context learning is popular with large language models (LLM), it has not been discovered in computer vision fields. This paper provides three benchmarks for in-context learning for computer vision and introduces some algorithms for proposed benchmarks. Since in-context learning is open for computer vision, the attempt is valuable and makes further research possible. ", "weaknesses": "In-context learning is a general learning paradigm for diverse tasks. In the computer vision field, there are a lot of tasks (e.g., classification, object detection, segmentation). However, this paper only considers three tasks: 3D Shapes Image Prompts, BitMoji Faces Image Prompts, CLEVr Objects Image Prompts. Thus, this benchmark may be not sufficiently comprehensive from this perspective. On the other hand, the performance versus the number of in-context examples is one important aspect of in-context learning. This paper doesn't provide any insight from this view.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and positive feedback. We address the various concerns below.\n\n> Since the number of in-context demonstrations is one important parameter in in-context learning fields, is there any ablation study for the number of demonstrations?\n\nWe agree with the reviewer that the number of context examples is a critical factor to in-context learning ability. We have, in fact, run ablations on the number of examples by exploring k-shot variants of the models. The plots in Figure 4 and Figure 5  show the model performance against the number of examples under “k-shot” labels. In addition, observations on the effect of context size have been described in R1.3.  We find the performance of the sequential prompter and patch-based agent to significantly improve with additional examples.\n\n> The input and output of these three tasks are all images. One question is can the text be the output of the task (e.g., image classification)?\n\nWe think this is possible by jointly modeling slots and language tokens and doing the context aggregation over the joint space.\n\n> In the proposed method, there are some components involved. What's the difference if we use models from scratch instead of pre-trained?\n\nAs such, we found that symmetry breaking among slots converged extremely slowly when trained from scratch and was sensitive to hyperparameters. We found it helpful to pre-train the encoder and bias the slots to capture object abstractions before learning to compose.\n\n"}, {"review_id": "dFniih5DGS", "reviewer": "Reviewer_1AXW", "summary": "This work proposes a framework for in-context image generation/composition from image prompts. The in-context learner resembles analogy completion and is simply optimized with a reconstruction objective. Several variants in terms of visual representations (pixels/latent vectors), the compositionality of the vectors (monolithic/object-centric), and the scheme of integrating prompts (cross-attention/simple concatenation) are explored to indicate the key ingredients of a successful visual in-context learning model. In addition, a benchmark consisting of three object/attribute-combinatorial datasets is created to evaluate visual in-context learning tasks.", "questions": "In terms of the experiments, I have two questions:\n(a) How to implement the multi-shot in-context generation?\n(b) In Figure 6, the objects in the generated image seem to be placed in the same position as they are in the image prompt. How is this position-preserving functionality realized by OCL? I would like to see some insights into this from the authors.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The main idea of introducing in-context inference ability (which is text-free) into the object-centric learning framework is interesting.\n\n+ The proposed Object-Centric Learner (OCL) is technically sound and the experimental results demonstrate its validity for in-context composition from image prompts.\n\n+ Extensive evaluations regarding different levels of compositionality and different means of context aggregation are investigated.\n", "weaknesses": "- The main concern comes from the confusing contribution clarification of this paper. I do not see which specific/major contributions the authors want to emphasize. Text-free in-context generation compared to text-to-image generation models such as DALL-E? Endowing SLATE with the in-context composition capability by inserting a context encoder transformer?\n(a) If the answer is the first one, I would say it is over-claimed. Although Figure 6 to some extent showcases that DALL-E requires tedious prompt engineering and some external techniques like inpainting to achieve scene composition, the proposed text-free in-context generation method is not more appealing since 1) it can only be applied in the toy scenario (while DALL-E supports open-vocabulary generations with high fidelity), and 2) one needs to have each component in advance (e.g., an image of a rubber purple cube in the desired position), and this is not feasible in practical scenarios.\n(b) If the answer is the latter, I would say the claimed contribution is not well supported. The proposed OCL demonstrates its ability of in-context composition from image prompts, but SLATE also supports compositional generation using the concept library built from the training datasets. I think some comparisons and discussions with SLATE would help contextualize this paper.\n\n- Overall, the paper is not easy to follow because: 1) there are many long sentences with less specific descriptions, and 2) many technical details of the in-context learner are missing and Section 4 is redundant (e.g., Equations 2 and 3 could be moved to the appendix). In addition, the authors should consider adding some preliminaries of slot attention and SLATE, making the paper more self-contained.\n\n- The proposed method is only evaluated on toy synthetic datasets. Although previous related works (e.g., SLATE) generally struggle to generalize to real-world data with diverse scenes, I would like to see some experiments on natural images.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and constructive feedback. We respond to the questions and concerns below.\n\n> The main concern comes from the confusing contribution clarification of this paper. I do not see which specific/major contributions the authors want to emphasize. Text-free in-context generation compared to text-to-image generation models such as DALL-E? Endowing SLATE with the in-context composition capability by inserting a context encoder transformer?\n\nThe goal of the work is the latter, i.e., enabling in-context composition for visual agents, with the former being a demonstration of how compositional visual understanding could be useful for downstream tasks. \n\n>  If the answer is the latter, I would say the claimed contribution is not well supported. The proposed OCL demonstrates its ability of in-context composition from image prompts, but SLATE also supports compositional generation using the concept library built from the training datasets. I think some comparisons and discussions with SLATE would help contextualize this paper.\n\nLLMs have the uncanny ability to implicitly compose outputs using human-designed abstractions (words, BPE, sentences). On the other hand, methods like SLATE learn their own abstractions; however, composition follows via explicitly constructed concepts by the human user.  Such rules can be tedious to specify as they scale exponentially with the number of stored concepts. Our work then combines the best of both approaches by simultaneously learning the compositional abstractions and the composition process. Moreover, the implication of this goes beyond simply inserting a context encoder over SLATE. While SLATE is a state-of-the-art slot learning method, it does not a priori follow that slot abstractions can enable state-of-the-art in-context compositional generation. Our work rigorously compares slots against the other possible composition choices to test this hypothesis. \n\n> If the answer is the first one, I would say it is over-claimed. Although Figure 6 to some extent showcases that DALL-E requires tedious prompt engineering and some external techniques like inpainting to achieve scene composition, the proposed text-free in-context generation method is not more appealing since 1) it can only be applied in the toy scenario (while DALL-E supports open-vocabulary generations with high fidelity)\n\nWe agree with the reviewer and do not claim to supersede DALL-E at generating open-vocabulary generations. The example shows how in-context visual understanding using the model's own concepts could facilitate relational understanding of the scene in comparison to grounding concepts in text. Future image-generation systems could then use text for high-fidelity generations of individual concepts in conjunction with architectures like OCL to facilitate the understanding of the interplay of these concepts. \n\n> The proposed method is only evaluated on toy synthetic datasets. Although previous related works (e.g., SLATE) generally struggle to generalize to real-world data with diverse scenes, I would like to see some experiments on natural images.\n\nPlease refer to the section “Concern surrounding the absence of natural images” in the global response.\n\n> How to implement the multi-shot in-context generation?\n\nThe context encoder collectively attends to the set of all context examples in the multi-shot case. Simple example encodings (implemented as positional embeddings) help the encoder differentiate between examples.  We will add this point to the text in addition to releasing our code, where the implementation will be made much more clear.\n\n> In Figure 6, the objects in the generated image seem to be placed in the same position as they are in the image prompt. How is this position-preserving functionality realized by OCL? I would like to see some insights into this from the authors.\n\nNote that the OCL is trained on object addition and deletion analogies (see Figure 9 in the supplementary). The image prompt then corresponds to an analogy of simply adding the object at a particular position in the scene. We will add a section in the appendix specifying the algorithm used to generate the prompts and the corresponding compositions. \n\n> Many technical details of the in-context learner are missing and Section 4 is redundant (e.g., Equations 2 and 3 could be moved to the appendix). In addition, the authors should consider adding some preliminaries of slot attention and SLATE, making the paper more self-contained.\n\nWe have added detailed background for slot attention and SLATE in the Appendix (see Section D in Appendix). All technical details have been specified in Table 6 of the Appendix. If the reviewer could be more specific about the details they are looking for, we would be happy to provide the necessary information.\n\n\n"}, {"review_id": "RgQHrIyHsf", "reviewer": "Reviewer_FiDL", "summary": "This paper focuses on the problem of using composable elements of visual modality to perform in-context referring for analogical reasoning like LLMs do. Firstly, the authors constructed three benchmarks to evaluate the generalization capacities of visual in-context learner. Then, the authors unified the formula of in-context analogy learning, and designed a framework called Im-Promptu. Different levels of agents, ranging from simple pixel-space rules to object-centric learners are trained for exploring the granularity of composition in visual modality. Lastly, the paper pointed out the characteristics of different degrees of compositionality via experiments.", "questions": "1.\tJust one question: The proposed three benchmarks are very different from actual scenes, are there any plans to introduce more real-scene images to the benchmark?", "limitations": "The authors discussed limitations and there is no potential negative societal impact of their work from my perspective.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.\tThe paper proposes an interesting topic of considering the in-context composition learning from visual prompts. Similar to the composition skills of LLMs, visual generative models’ compositionality is largely under explored. The motivation of this paper is meaningful and insightful.\n2.\tThis paper is of great novelty. First, it formulates the notion of analogy-based in-context learner and proposed a novel training framework called Im-Promptu based on that. What’s more, the paper provides a detailed and insightful analysis of different granularities’ influence on the compositionality and whether analogical reasoning enables in-context generalization. \n3.\tThe paper contributes three benchmarks for evaluating the generalization properties of visual in-context learners.\n", "weaknesses": "1.\tThere is no obvious weakness I have found in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and positive feedback. We address the various concerns below.\n\n> Just one question: The proposed three benchmarks are very different from actual scenes, are there any plans to introduce more real-scene images to the benchmark?\n\nWe are experimenting with photorealistic graphics engines that can render textures, near-natural objects, and finer lighting to close the sim-to-real gap. "}, {"review_id": "ljwBIAlAmo", "reviewer": "Reviewer_JrRD", "summary": "This paper presents a comprehensive study of in-context learning of compositionality in images. It introduces a simple benchmark of compositional language-driven visual transformations and explores under what conditions a model can perform in-context analogies from image pairs.", "questions": "None.", "limitations": "Limitations of the work are adequately spelled out.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Very interesting study, with relevant conclusions that can be easily turned into hypotheses for examining how one can achieve in-context analogical reasoning in a more real-world, less synthetic context. The scope of the study is well delineated, and the hypotheses that are tested are very clear. This is a very important topic beyond the narrow setting of 'can we do visual analogies', because of the widespread believe that analogy making is a great proxy for broader understanding of the visual world.\nThis study further reinforces recent observations from a number of researchers that object or attribute-centric representations are very effective at enabling higher level visual reasoning and deserve more attention (pun intended), and confirms that cross-attention is a strong contender for enabling compositional learning beyond just a single modality.\n", "weaknesses": "One could argue that results obtained on synthetic data always come with the question of whether the findings will transfer to more real-world settings. In this case, the diversity of synthetic data sources partially addresses the concern.\n\nNits:\n- Fig 1: girafe has two 'f' in English.\n- Suggest shortening refs [17] and [19] if permitted byt the NeurIPS style guide. A whole printed page for a single reference is a bit over the top.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and positive feedback."}]

Author rebuttal:
We would like to thank all reviewers for raising a number of great questions, adding detailed comments, and drawing our attention to related works. We will incorporate the feedback in the next iteration of the manuscript. We address some common concerns below and introduce additional results.

### Concern surrounding the absence of natural images

Formulating a method that can solve in-context downstream computer vision tasks is a worthwhile goal, but the scope of this work lies in understanding the computational underpinnings of compositional generalization. Uncovering the process by which humans form and modify flexible representations from few-shot examples is key to abstract reasoning and visual understanding [1][2]. While training large models on natural images is an appealing avenue for studying the emergence of large-scale in-context learning, this by itself tells us very little about the acquisition and the generalization limits of composition skills. Therefore, in this work, we construct synthetic benchmarks with composable attributes to systematically find effective inductive biases for compositional generalization.  We believe that the key to scaling up our models lies in designing natural image benchmarks where compositional attributes of the visual stimuli are modified. Generating compositional stimuli for real images is a significant engineering effort, and we leave this to future work.

### Additional Inpainting Results

We thank Reviewer 4dpy for pointing us to recent work [3] on visual prompting through in-painting. The authors represent an analogy via an image grid and use inpainting to fill in the missing solution. We implemented this methodology to investigate its ability to acquire composition skills. The results are as follows: 

**Primitive Task Extrapolation**:

|           | OCL-1 shot MSE | In-Painting MSE | OCL-1 shot FID | In-Painting FID |
| --------- | ----------------- | --------------- | ----------------- | --------------- |
| Shapes 3D | 4.72 ∓ 0.46       | 142.77 ∓ 1.31   | 34.77 ∓ 0.06      | 31.20 ∓ 1.86    |
| BitMoji   | 6.91 ∓ 0.54       | 260.94 ∓ 8.27   | 8.96 ∓ 0.05       | 104.84 ∓ 12.23  |
| CLEVr     | 37.99 ∓ 1.91      | 200.64 ∓ 3.75   | 38.99 ∓ 0.57      | 186.34 ∓ 5.33   |

**Composite Task Extrapolation**:

|                             | OCL-1 shot MSE   | In-Painting MSE | OCL-1 shot FID   | In-Painting FID |
| --------------------------- | ------------ | ------------------ | ------------ | ------------------ |
| Shapes 3D - Two Composite   | 6.22 ∓ 2.76  | 194.6 ∓ 1.56       | 34.18 ∓ 0.02 | 31.82 ∓ 2.45       |
| Shapes 3D - Three Composite | 7.96 ∓ 2.93  | 268.6 ∓ 0.92       | 35.12 ∓ 0.05 | 41.84 ∓ 1.51       |
| Shapes 3D - Four Composite  | 12.62 ∓ 4.41 | 292.84 ∓ 1.12      | 36.82 ∓ 0.06 | 51.34 ∓ 1.97       |
| BitMoji - Two Composite     | 5.72 ∓ 0.18  | 284.9 ∓ 7.35       | 8.78         | 103.74 ∓ 10.54     |
| BitMoji - Three Composite   | 5.96 ∓ 0.23  | 311.1 ∓ 6.48       | 8.74         | 108.92 ∓ 9.71      |
| CLEVr - Two Composite       | 37.12 ∓ 1.16 | 235.66 ∓ 2.77      | 68.45 ∓ 2.32 | 191.72 ∓ 3.85      |
| CLEVr - Three Composite     | 62.26 ∓ 1.41 | 285.76 ∓ 2.43      | 58.5 ∓ 1.02  | 183.94 ∓ 4.27      |


Examples of the generated outputs have been shown in the attached pdf. We observe that in-painting can extend composition rules to in-distribution primitives but produces incoherent outputs when tested on both primitive rule extrapolation (Sec. 6.2) and composite rule extrapolation (Sec. 6.3). It performs the worst out of all the agents. The technical details, results, and code for the In-Painting will be added to the updated manuscript.

[1] Brenden Lake et al., “Building machines that learn and think like people,” Behavioral and Brain Sciences, 2016

[2] Brenden Lake et al., “One shot learning of simple visual concepts,” Cognitive Science, 2011 

[3] Amir Bar et al., "Visual prompting via image inpainting," Advances in Neural Information Processing Systems 35 (2022): 25005-25017






Meta-review: This work receives mixed reviews. Three expert reviewers think the in-context (composition) learning in vision modality is important yet highly under-explored, and the contributions (synthetic benchmark and datasets, problem formulation and the meta-learning framework, evaluation and analysis of compositionality of various degrees) in this paper are significant, while the other two expert reviewers concern about its novelty against visual analogy-making and its use in realistic setting. Overall, AC agrees this work is interesting towards visual in-context learning, emphasizes more on the (high-order) compositional generalization compared to previous visual analogy-making, and can be much strenghened by  evaluating on real-world problems (e.g. [1]).

[1] Amir Bar et al., "Visual prompting via image inpainting," Advances in Neural Information Processing Systems 35 (2022): 25005-25017
Decision: Accept (poster)