Spatial Reasoning Network for Zero-shot Constrained Scene Generation
Authors: Maxwell J Jacobson
Yexiang Xue
Venue: Submitted to ICLR 2023 (2023)
Keywords: Spatial Reasoning Network
Constrained Scene Generation

Abstract: Constrained scene generation (CSG) generates images satisfying a given set of constraints. Zero-shot CSG generates images satisfying constraints not presented in the training set without retraining. Recent neural-based models generate images with excellent details, but largely cannot satisfy constraints, especially in complex scenes involving multiple objects. Such difficulty is due to the lack of effective approaches combining low-level visual element generation with high-level spatial reasoning. We introduce a Spatial Reasoning Network for constrained scene generation (SPREN). SPREN adds to the state-of-the-art image generation networks (for low-level visual element generation) a spatial reasoning module (for high-level spatial reasoning). The spatial reasoning module decides objects' positions following the output of a Recursive Neural Network (RNN), which is trained to learn implicit spatial knowledge (such as trees growing from the ground) from an image dataset. During inference, explicit constraints can be enforced by a forward-checking algorithm, which blocks invalid decisions from the RNN in a zero-shot manner. In experiments, we demonstrate SPREN is able to generate images with excellent detail while satisfying complex spatial constraints. SPREN also transfers good quality scene generation to unseen constraints without retraining. 

Under review as a conference paper at ICLR 2023
SPATIAL REASONING NETWORK FOR ZERO-SHOT CONSTRAINED SCENE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Constrained scene generation (CSG) generates images satisfying given set of constraints. Zero-shot CSG generates images satisfying constraints not presented in the training set without retraining. Recent neural- models generate images with excellent details, but largely cannot satisfy constraints, especially in complex scenes involving multiple objects. Such difficulty is due to the lack of effective approaches combining low-level visual element generation with high-level spa- tial reasoning. We introduce Spatial Reasoning Network for constrained scene generation (SPREN). SPREN adds to the state-of-the-art image generation net- works (for low-level visual element generation) spatial reasoning module (for high-level spatial reasoning). The spatial reasoning module decides objects' posi- tions following the output of Recursive Neural Network (RNN), which is trained to learn implicit spatial knowledge, such as trees grow from the ground. During inference, explicit constraints can be enforced by a forward-checking algorithm. which blocks invalid decisions from the RNN in zero-shot manner. In experi- ments, we demonstrate SPREN is able to generate images with excellent details while satisfying complex spatial constraints. SPREN also transfers good quality scene generation to unseen constraints without retraining.
INTRODUCTION
Constrained content generation has long been an important task in artificial intelligence and has many implications across domains Nauata et al. (020)); Jiang et al. (222) ) Ma et al. (2021). This paper focuses on constrained scene generation generating a realistic scene image containing mul- tiple objects satisfying given set of constraints. While there has been exciting progress in scene generation, especially using deep generative models Deng et al. (2021) Liu et al. 2002 ; Arad Hud- son & Zitnick (202I), generating scenes involving multiple objects satislying complex spatial re- lationships remains a challenging task. Existing approaches often cannot generate scenes which contain the right number of objects the correct spatial relationship between the objects (according to user-defined constraints).
We hypothesize such difficulty is due the lack of effective approaches combining low-level visual element generation with high-level spatial reasoning. From psychophysiological studies Kahne- man [201 (Swwen et al. (2015); Lin Lien] (2013), human beings require multiple systems of reasoning and memory (including systems and 2) to complete complex content generation tasks. Procedural (P) cognition retains the skill to generate the texture/shape of standalone objects. System 1 (S1) cognition captures "common-sense knowledge and patterns. For example, trees are on the ground, but birds are in the sky. System (S2) cognition embodies reasoning about the high level task, planning the content of the image and enforcing explicit constraints at an abstract level. Over the years, neural generative models have been very successful in learning tasks associated with and SI -coogniii,,,, but fail consistently on S2-cognition especially enforcing complex constraints. Traditional constraint reasoning methods are able to provide the S2-cognition necessary for our task, but they too rigid to handle P and $1-cognition.
We introduce a Spatial Reasoning Network for constrained scene generation (SPREN). The key idea is to add to the state-of-the-art neural generative models responsible for low-level visual ele- ment generation, or P/SI cognition, a spatial reasoning module, which handles high-level spatial reasoning, or S2 cognition. The input of constrained scene generation is background image and

---

Under review as a conference paper at ICLR 2023
Input (constraints)
Input (background)
Pipeline Spatial Reasoning Module den sian szoring
lterative
Forward Checking (S2 (52-cognition)
Refinement RNN
(S1-cogritlon)
Above
Visual Element
Right
Left of
blueprint
Generation Module
(P-cognition)
Below
Blueprint
Output
Baseline Comparison (GLIDE)
2
Figure I: A scene generated by SPREN with a given background and six animals subject to complex positional constraints. The animals and the spatial constraints governing their spatial locations shown in the upper left panel. The upper center panel shows the background image on which the scene must be placed. The upper right panel shows the high-level pipeline of our approach. The lower left panel shows the locations of these animals generated by SPREN (blueprints). The lower center panel shows realistic image SPREN produces which satisfies constraints. The lower right panel shows a baseline scene for comparison, where constraints cannot be properly enforced.
set of spatial constraints represented in propositional logic. and the output is the generated im- age containing objects satisfying constraints. The spatial reasoning module decides the objects' positions following the output of Recursive Neural Network (RNN) in a process of iterative re- finement. The RNN outputs the bounding boxes (we call them blueprints) for each object to be generated. When determining one coordinate of the bounding box, the RNN iteratively halves the range of the coordinate until it is sufficiently small. During learning, the RNN is trained to under- stand implicit spatial knowledge, such as trees grow from the ground and birds fly in the sky. This is done by teacher forcing procedure which matches the bounding boxes predicted by the RNN and the ones containing the objects in the original image. During inference, explicit constraints can be enforced by forward-checking algorithm, which blocks the decisions leading to constraint viola- tions. The forward checking procedure also allows us to handle constraints zero-shot manner. During time when novel constraints are present, the forward checking procedure blocks the out- put of the RNNs following the same procedure handling familiar constraints, without any retraining or fine-tune tinettning. Figure demonstrates the generative procedure of SPREN. Here, the colored arrows in the upper left panel represent the spatial constraints restricting the animals to be generated. The upper center panel represents the input background image. The lower-left panel shows the blueprint (bounding boxes) output by the spatial reasoning module. The lower-middle panel shows the final output of SPREN and the lower-right comparison with the previous state-of-the-ar GLIDE model.
In experiments, we demonstrate SPREN is able to generate images with excellent details while satisfying complex spatial constraints. We also show that SPREN works well for object aware scene generation, which is an inpainting task involving adding additional objects to image containing existing ones satisfying constraints involving both the existing and newly added objects. SPREN also works well in zero-shot transfer learning: it generates good-quality scenes involving constraints unseen from the training set without retraining or fine-tuning. Overall, our contributions are:
We introduce the SPREN framework for constrained scene generation, combining low-level visual element generation with high-level spatial reasoning.
The objects are positioned satisfying explicit constraints and fit into the visual context of the image well, thanks to the spatial reasoning module.
2

---

Under review as a conference paper at ICLR 2023
The final generated images contain excellent details thanks to the deep generative networks for low-level visual element generation.
SPREN can handle zero-shot scene generation new constraints unseen during training can be added at any time without retraining or fine-tuning.
SPREN can successfully handle object-aware scene generation, involving handling constraints which constrain the positions of the known objects present in the background image and the newly added objects.
SPREN admits a rich set of spatial constraints represented in propositional logic.
PROBLEM DEFINITION
2.1 ZERO-SHOT CONSTRAINED SCENE GENERATION (ZS-CSG)
Standard scene generation the problem of generating a set of objects onto a given (or externally generated) background image to create realistic scene. Constrained scene generation requires the same, but with the additional demand that objects placed in the scene satisfy a set of positional constraints. These constraints may change from scene to scene. Moreover, certain constraints may not present in the training data. At test time, the neural network needs to generate images satisfying such constraints without any fine-tuning or retraining. We refer to such use cases as zero-shot constrained scene generation. Formally, the problem can be defined as:
Problem 1: (ZS-CSG) Given: let B be background image, C be set of positional constraints, and T be the set of "natural" images. Find: scene image S with B as back such that all constraints in Care satisfied. Moreover, S looks "realistic"; ; ie., it is visually close to the images in T.
We use propositional logic to represent the set of positional constraints. As future work, will attack the scene generation problem with natural language as the input. Propositional logic uses logical operators (A for "and", V for "or", for "not") connect purposes, predicates define the names and types of objects and the spatial relations between objects. For example, "prompr(O, a green sheep)" means object 0 is green sheep. "type(0, sheep)" defines the type of object 0 to be sheep. "left(O, 1, 200)" means object O needs to be positioned to the left-hand side of object by at least 200 pixels (the last argument is often left out, meaning object O is just left of I). The semantic meaning of logic connectors remains the same as in the standard propositional logic. As an example presented Figure the constraints are shown in the upper-left panel while the input background image is in the upper-center panel. Here the colors of the arrows represent different predicates and all the predicates are connected with logic "and" (A). The spatial reasoning module places the animals in the gray boxes in the lower-left panel, satisfying all the constraints. The final output image is shown in the lower-center panel of Figure T See the full list of predicates and logic formulations in Appendix A
of predicates. For our
2.2 OBJECT-AWARE CONSTRAINED SCENE GENERATION (OA-CSG)
Aside from the ZS-CSG problem where all the objects generated empty background, also consider the more challenging object-aware constrained scene generation problem (OA-CSG). In this problem, the given background may already contain objects. There are logic constraints which specify the spatial relationships between the objects to be generated and these existing objects. This version of the problem requires that the algorithm to be aware of pre-existing objects and can take them into account during the decision cecision-makkee making process both in relation to constraints and the realism of the scene.
let B be a background image which contains initial objects W. and C be a of positional constraints. C may reference objects in W. T is the set of "natural" images. Find a scene image S, with B as a background, such that C is satisfied and S "realistic"; ;; i.e. it is visually close to the images in T.
Problem 2: (OA-CSG): Given:
3

---

Under review as a conference paper at ICLR 2023
2.3 CHALLENGES
Previous approaches face major difficulties handling constrained scene generation. The ability for large diffusion models like DALL-E|Ramesh et al. (2021) and GLIDE et al. (202T) to inpaint objects into scenes given natural language prompt has made them quite attractive for constrained scene generation. A prompt can describe the objects to be generated and the constraints to be applied between those objects. However, these models consistently fail on any non-trivial instructions (such as constraining the number of objects, their types, and/or spatial relationships). Recently, a line of research encodes requirements (such as content of the images) into hidden vectors via graph or recursive neural nets Deng et al. (02T); Xu et al. (2018): (Engelcke et al. (2020), and then condition image generation on these hidden vectors. These approaches provide way to condition image generation on certain requirements, but still cannot enforce complex constraints explicitly. Further, all the conditions must be present in the training set. Otherwise, it is difficult to predict what the hidden vectors will be on an unseen new condition. Our SPREN approach, on the other hand, handles complex constraints that are unseen in the training set without retraining or fine-tuning.
2.4 ASPECTS OF COGNITION
Taking inspiration from the concept of slow and fast thinking Kahneman (201T). we base our ap- proach on the idea of integrating multiple levels of reasoning for the completion of this complex task. Under this paradigm, cognition is split into system 1 "fast" thinking, composed of learned behavior that can be accessed efficiently, and system 2 "slow" thinking, which is more intentional form of thought used in problem-solving and reasoning about abstract concepts. Our work draws the S1 and S2 cognitive aspects from this area of psychology. We also define P-cognition from the concept of procedural memory, colloquially known as muscle memory. Procedural memory requires great deal of practice to build, but allows to learn difficult skills through extreme repetition.
Constrained scene generation requires a synthesis of these three aspects of cognition. P-cognition involves learning the basic skills of larger task in this case, paining objects into a scene at a given location. SI S1-cognition captures fuzzy knowledge and data-driven patterns. For CSG, these are the locations of objects relative the background and the other objects in a non-constrained case. S1 learns how to place objects "realistically". 52-cognition handles the more abstract procedures of reasoning, specifically constraint satisfaction. 52-cognition may ensure that an image "has a dog to the left and cat to the right" but is not so concerned on the precise location of the cat and dog. In this work, S2-cognition regulates the decisions of Sl-cognition, and both work together to expertly dispatch P-cognition and accomplish the scene generation.
SPATIAL REASONING NETWORK (SPREN)
SPREN composed of one spatial reasoning module and one visual element generation module. The spatial reasoning module takes the input, and decides the spatial position of each object. In our setting, the spatial position is represented as a bounding box with parameters (x, y, width, height). Here, x, y is the coordinate of the upper-left corner and width, height represent the width and height of the bounding box. We call these bounding boxes "blueprints". See the lower-left panel of Figure [ for the generated blueprints. During learning, the spatial reasoning module is presented with set of background images and the corresponding bounding boxes of various objects in the images. The module is asked to generate bounding boxes which matches those given in the training data. This corresponds to learning implicit spatial preferences. e.g., the relative sizes of tree compared to house, and/or trees grow on the ground while birds flying in the sky. We implement the spatial reasoning module using an RNN-type structure, which iteratively refines the location of each object. During inference, the spatial reasoning module needs to decide the locations of each object given the background image and set of constraints. Here, propose a forward checking algorithm which rules out invalid outputs of the spatial reasoning module. Notice that the forward checking algorithm fits seamlessly with the RNN component and does not need to be trained. This setup allows SPREN to be transferred to new constraints not presented in the training set in a zero-shot learning setting. See Appendix for more on our architecture and hyperparameters.
The visual element generation module takes as input the background, the prompt, and location of each object, and outputs an image patch which contains the object and can be merged seamlessly
4

---

Under review as a conference paper at ICLR 2023
0,100
0.100
0,50
50,100
0,50
50,100
0,25
25,50
50,75
75, 100
0,12
12,25
50,62
62,75
X1 18
Y1 = 68
<left>
<left>
<right>
<topp>
<right>
<left>
<right>
<stop>
<right> <stop=
t
J
t
Constraint Reasoning (during Inference)
<start>
Scene
Var
Var
Encoder
Encoder
Encoder
bg img (obj x)
(obj y)
Figure 2: Rollout of the spatial reasoning module. The position of each variable is narrowed down until the stop token is selected. In the lower panel, the spatial reasoning network assigns probability scores each decision. At the start of the scene, the hidden state is initialized from resnet-based scene encoder encoding the background image. At the start of generating each position variable, the hidden state is added by a vector generated by the variable encoder. Constraint reasoning is applied during inference to choose final decision which satisfies constraints. Above, the decision-making process is visualized as a tree of value ranges (to generate the x and y positions of the horse).
into the background image. We use state-of-ther tfrt neural generative models the visual element generation module. While we can train a visual element generation module ourselves, we recognize the need for huge datasets and manual effort to train these models successfully. We also would like to be able to re-use and adapt to as many state-of-the-art neural models as possible. The lower- center panel of Figure [ demonstrates the generated image, with each bounding box filled by the visual element generation module.
3.1 ARCHITECTURE & SEMANTICS THE SPATIAL REASONING MODULE
The spatial reasoning module takes the input of the background image well as the object identifier, and outputs the position in the form of (x, y, width, height) for each object. We use encoder networks to encode both the background image and the object identifier (e.g., object 0) into high-dimensional vectors. The spatial reasoning module harnesses a Recursive Neural Net (RNN) structure and out- puts the four coordinates sequentially. The key idea in generating each coordinate is through ilerative refinement. The basic unit of the RNN output is the decision token a size vector representing a score distribution of four possible decisions special start token. The input at each timestep is the previous timestep's 's decision as a one-hot vec- tor. The network outputs at each step softmax vector assign the probability to each token. At the start of calculation, the selected variable is presumed to be within a given range (e.g. the first object's x value is between 0 and 100). The actions the range (e.g. left: 0 < x < 50), or in the second half of the range (e.g. right: 50 < x < 100), or to terminate and select the middle value between the current minimum and maximum as the final location (c.g. stop: = 50). This produces sequence of decisions leading to the assignment to the corresponding variable. The sequence embodied as a decision string composed of decision to- kens. For example, the variable x between 0 and 100 may be assigned the decision string "left, right, stop", refining the value choose the right of the range, then network is implemented in the form of GRUs (Cho et al. 2014 ). All contextual data including the background image encoding and the variable encoding introduced into the GRU state vector (size 500) by adding to it. Figure 2 shows the processes of generating two coordinates 2; and y1 using the proposed neural network module, including the encoding at the start of each variable.
more accurately three possible decisions and one
limit the position to be in the first half of
x < 50 as we choose the left of the range, then 25 < x < 50 as we
37 as we choose the center of the range. Our recursive neural
5

---

Under review as a conference paper at ICLR 2023
decision scoring
range table
<left>
X1
100]
<left>
X1
[0. [0]50]
<left>
X1
[0. 25
<right>
X2
[20. 50)
<right>
X2
[20, 50]
<right>
X2
[20. 50]
<stop>
<stop>
<stop>
Attempt left.
Attempt right
Attempt stop
CCheck constraints for 50)
Check constraints for [25, 50].
Check constraints for [12, 12]
-0
C₁ salisfiable.
25
C₁ not satisfiable.
-0 10x22 C₁ satisfiable.
Go left
Attempt left Check constralint for [0, 25 -0 20 - C₁ satistable.
stop.
Constraints
= 12.
C1: X1 X2.
Go left.
C3:
X2 > 19.
Figure 3: Example of forward checking during inference. Depth first search is applied in an order determined by sampling from multinomial distribution of the decision scores generated by the RNN (shown here as a 3-option bar graph). The DFS is augmented with a range table, allowing for quick reference between variables and efficient recognition of un-satisfiable paths.
3.2 TRAINING OF THE SPATIAL REASONING MODULE
The training process uses a dataset containing background images with bounding box for each object be generated. Each object location defined by the upper left point, as well as the width and height of the bounding box. These variables are converted to decision strings as described in the previous section. The training process a supervised procedure it trains the spatial reasoning module to generate bounding boxes (in the form of decision strings) which match those in the train- ing dataset. This procedure teaches the neural network implicit spatial knowledge (or S -cogniition including the relative sizes of objects (horses, dogs, trees, etc) and their common spatial relation- ships (birds fly in the sky, etc). We do not enforce explicit constraints during the training phase. The training of RNN is accomplished through teacher forcing (Lamb al. 2016); Williams & Zipser 1989 ) In teacher forcing, the ground truth decision token from the previous step drawn from the training set is used the input train the RNN to predict the decision of the current step. Teacher forcing can drastically speed up RNN training, but can also result in lower resilience to uncommon sequences. For this reason, a mixture of strategies with and without teacher forcing - is used to train the model. 50% of the time the input of the RNN is set by teacher forcing and 50% set by the predictions of the RNN itself. We also randomize the order of the objects the decision string during training. Finally, the spatial reasoning module also needs to learn to encode the background image - extracting important features like the horizon of the scene, or major obstacles. This ac- complished by back-propagee propagating the gradients through the scene encoder network. It is diffficult to find a real-world datasets of background images (containing no or few objects) and bounding boxes of the objects to be generated. We instead use real-world datasets of images containing objects and their corresponding bounding boxes (such as COCO). We remove these objects from the original images by painting them using GLIDE. This ensures the bounding boxes of the objects placed in their most natural positions. This step is discussed more in Appendix G
3.3 INFERENCE USING THE SPATIAL REASONING MODULE
In the absence of constraints, the network can simply output a decision string for each positional variable by sampling tokens from the probability distribution outputs until conclusive value is reached. At the beginming of a string, the start token is given as input and the variable encoder adds information to the GRU state vector. The decisions are sampled based the probabilities output by the spatial reasoning module. For example, if the module outputs the scores [left: 75%, right 5%, stop 20%). the spatial reasoning module will select "left" 75% of the time, "right" 5% of the time, and will choose to stop 20% of the time and to choose the middle value of the current range.
If constraints are introduced, they can be enforced by limiting the decisions considered in the sam- pling process. A forward checking algorithm is used for this purpose. For each decision in the iterative refinement, we check that the current value range contains at least one value assignment which satisfies all constraints. If at least one constraint must be violated, we can rewind the decision string one token and resample without the option of the original choice. Our forward checking algo- rithm is implemented as a depth first search exploring all future expansions of the decision strings though many algorithms could fit this role. This forward checking process is conducted after each decision made by the recursive neural network to see if constraint satisfaction is still possible. In our

---

Under review as a conference paper at ICLR 2023
previous example, we may have selected the "left" token first, but then found that value in the remaining range for that variable can satisfy all the constraints. We resample this token with new scores [left: 0%, right: 20%, stop: 80%]. The process is visualized in Figure 3
The addition of the forward checking algorithm applies much-needed S2-cognition to the S1- cognition encoded within the neural net, resulting in two part reasoning system which in- telligently plan for object placement within positional constraints.
3.4
THE VISUAL ELEMENT GENERATION MODULE
The blueprint from the spatial reasoning module is used to create masking boxes, which let us generate each object into the scene using the visual element generator. The result is completed scene including all objects and adhering to all constraints. The visual element generation module is sequentially called on each box from the blueprint created by the spatial reasoning module. For the purposes of this paper, we use the reduced and filtered GLIDE diffusion model released by OpenAl. Given masked image and a prompt, this network inpaints the masked areas accordingly. To fit the limitations of this model, all inpaint boxes must be less than 256 pixels in any dimension (this can be ensured with an extra constraint in the spatial reasoning module). The 256x256 pixel area including the box is cropped and used to inpaint the image only within the masked space. In most cases, this will maintain perfect constraint satisfaction, but sometimes GLIDE will output something that does not fit the prompt, or it will simply fail to inpaint anything except background. These inpainting failures are considered an inpaint stage constraint breach.
4 RELATED WORK
Image Generation. Realistic image generation became a feasible task along with the rise of deep neural networks. Generative autoencoders Kingma & Welling (2013); Makhzani al. (0016), gen- erative adversarial nets (GANs) Goodfflloo et al. (2014), flow models Rezende & Mohamed (2015). and diffusion models Sohl-Dickstein et al. (2015) all do this by learning function that transforms noise from a known distribution to a new distribution from dataset. For example, diffusion mod- els train the network in steps to reconstruct partial images from noise. Deep generative models are capable of producing extremely realistic images Brock et al. (2019)/Karras Karras et al. (2018) Zhu et al.) (2017) Karras al.) (2019) 2020), but are limited on how controlled the generation is from image to image. For some models, conditioning MMrza & Osindero (2014) is a good strategy to increase control over generation, but this requires very structured data source and does not guarantee the condition will be upheld.
Image Generation from Natural Language. A greater degree of control in image generation can be achieved by generating images from natural language prompt. Transformers Vaswani et al. (017)) are neural nets based around multi-headed attention mechanisms. While they are most known for their ability generate text language model, they have shown competence in image Chen et al. (2020) and audio Dhariwal et al. (2020) tasks as well. There now exist several deep generative models which create images from natural language interpreted by transformers Ramesh et al.) (2021 2022]: Nichol et al. (2021). This is possible in part due to massive datasets extracted from the internet. While transformer-basee models add control to the generation process, they often fail given complicated prompts or non-trivial constraints.
Scene Generation. Generating entire scenes is a difficult task even with current tools. This difficulty comes from finding realistic locations for objects within the scene which synergize with all other objects, and the internal representations that go along with that. Large transformer-basee generators are capable of generating whole scenes instead of single-object images, but single latent space limits the complexity of scenes that can be produced alone. Recurrent neural networks Ennellcee Hudson & Zitnick (2021),
et al. (2020), graph neural networks Deng et al. I). transformers and energy based models Du et al. (2020); Liu et al. (202I) have had some success in filling this gap. However, standalone GAN models have been able to accomplish good scene generation well with some modification Liu et al. (019 Xu et al. (2018).
Neural Networks & Reasoning. Adding various types of reasoning neural algorithms has been an important part of the Al research for quite some time. Much of it has focused on attempting to add "common-sense" reasoning to natural language models He et al. (2019): Trinh & Le (2018);
7

---

Under review as a conference paper at ICLR 2023
Mean Pref
Constraint
Training
Training
Dataset
Method
Score %
Sat %
Steps
Time (min)
SPREN (ours)
100
100
1600
90
Basic
GAN + CVX
100
100
673
4
SPREN (ours)
91.7
100
2600
207
Tight
GAN + CVX
81.3
100
2600
SPREN (ours)
87.5
100
2000
217
Complex
GAN + CVX
72.3
100
2600
16
Table 1: Results for evaluating the spatial reasoning module. Our SPREN is able to generate object
locations better satisfying the implicit preferences of object positioning in the training set (measured by the pref score) compared to baselines. Note, SPREN can also handle non-convex constraints, while "GAN+CVX" cannot. Details of this experiment are in the Appendix
BP
2 Objects, Constraints
3 Objects, Constraints
Method
Sat %
Overall
Obj
Pos
Overall
Obj
Pos
Sat %
Sat %
Sat
Sat %
Sat %
Sat %
SPREN (ours)
100
80
80
100
65
65
100
GLIDE (center)
N/A
5
5
100
0
GLIDE (unbound)
N/A
10
25
40
0
1.6
0
Table 2:
Constraint satisfaction results for the full CSG experiment. SPREN outperforms GLIDE
both in the blueprinting phase (generating locations) and in the final generated images when generat- ing different number of objects subject to constraints (manually inspected). Details in the Appendix
Peng et al. (015)) though progress has also been made in integrating case-based reasoning once common in expert systems Im & Park (2007). as well as relational reasoning between objects or terms Centoro et al. 2017); Lamb et al. (2020); Pise et (202I). Very recently, constraint reasoning has been considered fruitful domain for integration with neural nets convex optimization Boyd Vandenberghe (2004) as a neural net layer being a recent and applicable innovation (Agrwaal et al. [2019] 2020: 2021). and other constraint reasoning methods have also been successfully integrated into neural networks for variety of applications Xue & Hoeve (2019); Bai et al. (202T): Khalil et al. 2017 Chen al. (2021).
EXPERIMENTS
We evaluate both the spatial reasoning module alone in its ability to generate blueprints (bounding boxes for a scene), and our full SPREN model in generating constrained scenes.
Evaluating the Spatial Reasoning Module. We find that our approach generates blueprints that 100% satisfy spatial constraints and hetter match implicit preferences of object positioning com- pared to baselines. See table [ Due to space restrictions, we leave all the discussions the experi- mental setup, baseline description, etc, to the appendix. Note, problem setup requires constraint satisfaction tools that can be embedded into learning the implicit preferences of object positioning (or S1 cognition). Few constraint programming tools can handle this task, except for the "GAN CVX" approach we found Amos & Kolter (2017). However, "GAN + CVX" only handles convex objective functions and constraints, while our SPREN handles arbitrary constraints in propositional logic.
Evaluating the Full SPREN Algorithm. When evaluating our full CSG algorithm, SPREN sur- passes the baseline both quantitatively (high percentage of constraints satisfied, see table 2 and qualitatively (high quality images, see figure 4.. Due to space restrictions, we again leave the exper- iment setup, dataset generation, baseline descriptions, etc, to the appendix.
6 CONCLUSION
In this work, we considered the problem of Constrained Scene Generation (CSG), well as the zero- shot and abject-aware variants of this task. CSG represents class of generative problems which require tight coordination between automated reasoning and machine learning and is beyond the
8

---

Under review as a conference paper at ICLR 2023
SPREN (ours)
SPREN (ours)
GLIDE (center)
The dog above horse is the left OR dog is below and the horse is to the right
neon red sheep left white sheep a neon
blue sheep. All are equally wide and lined up.
gray wolf golden wolf. gray wolf brown dog left green
half of image width to the left of the golden wolf OR half above the horse. All in the bottom 2/3rds of the image. image width to the right.
A purple dog, a red dog, green dog, and a yellow dog in square. They are ordered clockwise, starting with upper
sheep, dog, and a giraffe. The sheep is left the dog. the dog narrower than the giraffe. The dog below sheep.
Two people flying kite already the image. An elephant near the bottom of the image and pterodactyl flying above.
Two Giraffes were already image. Another were placed between and left of them. OA-CSG
A dog and horse were already in the image. A sheep and
added above the horse. sheep is left of the horse and the brown dog is right of the horse. OA-CSG
Note that GLIDE removed the horse dog.
Figure 4: Selected examples of scene generation using SPREN, On the right side is a comparison between method and GLIDE (reduced and filtered). Images for object aware constrained scene generation (OA-CSG) are noted. Constraints are described in natural language to aid readability. Full constraints in propositional logic listed in the Appendix C
capabilities of most neural generative models. Our proposed Spatial Reasoning Network (SPREN) combines the state-of-the-art neural generative models for low-level visual element generation with spatial reasoning module for high-level spatial reasoning. Experiments show that SPREN is able to generate images with excellent details while satisfying complex spatial constraints. SPREN also transfers good quality scene generation to unseen constraints without retraining. Future directions of this research including expanding to other generation tasks (such as generating programs, music, etc) and taking in richer class of inputs characterizing constraints, such as the natural language.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J. Zico Kolter. Differentiable convex optimization layers. In Advances in Neural Information Processing Sys- tems, pp. 3558-9570, 2019.
Akshay Agrawal, Shane Barratt, Stephen Boyd, and Bartolomeo Stellato. Learning convex optimiza- tion control policies. In Alcxandre M. Bayen, Ali Jadbabaic, George Pappas, Pablo A. Parrilo, Benjamin Recht, Claire Tomlin, and Melanie Zeilinger (eds.), Proceedings of the 2nd Confer- ence on Learning for Dynamics and Control, volume 120 of Proceedings of Machine Learning Research, pp. 361-373. PMLR, 10-11 Jun 2020. URL h procceddios.ss..... mlr. cilii.pressvvvlv /220
aggrawal2aa.htun
Akshay Agrawal, Shane Barratt, and Stephen Boyd. Learning convex optimization models. IEEE/CAA Journal of Automatica Sinica, 885):1355-1364, 2021. doi: 10.1109/JAS.022. 1004075.
Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization layer in neural networks. In International Conference on Machine Learning, pp. 136-145. PMLR, 2017.
Dor Arad Hudson and Larry Zitnick. Compositional transformers for scene generation. Advances in Neural Information Processing Systems, 34:9506-9520, 2021.
Yiwei Bai, Di Chen, and Carla P. Gomes. CLR-DRNets: Curriculum Learning with Restarts to Solve Visual Combinatorial Games. In Laurent D. Michel (ed.), 27th International Conference Prin- ciples and Practice of Constraint Programming (CP 2021), volume 210 of Leibniz International Proceedings in Informatics (LIPles), pp. 17:141 Dagstuhl, Germany, 2021. Schloss Dagstuhl -Leibniz-Zentrum für Informatik. ISBN 978-3-95977-211-22 doi: 11A.20//.PPcc..CP2222......
URL https:/ hatp:://drrpssss httpag/ttuuh..eeoppuss/lleeeeeee
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA. May 6-9, 2019. OpenReview.net, 2019. /Rtpsstt///ppeerrreeee net -B
Di Chen, Yiwei Bai, Sebastian Ament, Wenting Zhao, Dan Guevarra, Lan Zhou, Bart Selman, R Bruce van Dover, John M Gregoire, and Carla P Gomes. Automating crystal-structur phase mapping by combining deep learning with constraint reasoning. Nature Machine Intelligence, (9):812-822, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1691-1703. PMLR, 13-18 Jul 2020. URL https:/ /tt:://prooc pwwppedinoc.g.... mlr. press hwwp/11111/h//pe chen20s.htmml
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Sincode-decodde approaches. In SSST EMNLP, 2014.
Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. In Inter- national Conference on Learning Representations, 2021. URL httL:tttss//oopeeeviieee .ee, /forumm
RmcPmmmm3tmk
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint 0rxivv2005.00341, 2020.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6637-6647. Curran Associates, Inc., 2020. URL https:/ http://pppooodddeneeerr.s..... (aappperr220/fflle file/
10

---

Under review as a conference paper at ICLR 2023
Martin Engeleke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Gener- ative scene inference and sampling with object-centric latent representations. In International Conference Learning Representations, 2020. URL httpss//openneeieww net htre/formm?ihi BVw
Jan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
Pengcheng He, Xiaodong Liu, Weizhu Chen, and Jianfeng Gao. A hybrid neural network model for commonsense reasoning. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pp. 13-21, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.1 1t.18653/v//D1--66
URL httpp::t////// httpp//////laaaaololooo..oooo /aaaaaaanooooooooorr.
Kwang Hyuk Im and Sang Chan Park. Case-based reasoning and neural network based expert system for personalization. Expert Systems with Applications, 32(1):77-85, 2007. ISSN 0957- 4174. doi: https://doi.org http:////oooorr/////o 10160.001............ 1..00. URL https: http///wcciiceccrreeccoom
aitence/article///e pitpcclee/pSS5574111111 pttp://00000110000000000000
Songyao Jiang. Hongfu Liu, Yue Wu, and Yun Fu. Spatially constrained gan for face and fashion synthesis. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recogni- tion (FG 2021), pp. 01-08. IEEE, 2021.
Daniel
Kahneman.
Thinking, fast and slow.
Farrar, Straus and Giroux, New
York, 2011. ISBN 9780374275631 0374275637.
URL https: www.wawmazo..d......
Thinking- Fast-Sio- Kahneman wwwpaaahnmaaa/////p//// /dp0074275633 ref-
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im-
proved quality, stability, and variation. In International Conference on Learning Representations, 2018. URL httpss///ppeeureeeeeee net hel..
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8110-8119, 2020.
Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial opti- mization algorithms over graphs. Advances in neural information processing systems, 30, 2017.
Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2013.
Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent net- works. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL (ttpss/ www.eediinnnn..uuippoe
paper//pppereeeee httpa//e/00000//////////00o00110 50066666088554111011144414313223 Pap.:aaaa...222--p.ppppppp. Paper.ddl
Luis C. Lamb, Artur d' Avila Garcez, Marco Gori, Marcelo O.R. Prates, Pedro H.C. Avelar, and Moshe Y. Vardi. Graph neural networks meet neural-symbolic computing: A survey and perspective. In Christian Bessiere (ed.). Proceedings of the Twenry-Nnnth International Joint Conference Artificial Intelligence, IJCAI-20, pp. 4877-4894 International Joint Con- ferences on Artificial Intelligence Organization, 7 2020. doi: 10.24963/jccai URL https:/ /doi.or/ 10. 24963 (0caai.20000.1 667 Survey track.
Wei-Lun Lin and Yunn- Kunn-Wen Lien. The different role of working memory in open-ended versus theory account. Creativity Research Jour-
closed-ended creative problem solving: A nal, 96, 2013. doi: 10. 00.1080/1040441..2111...3
URL http::: https:///doiorgg/// oot.d.oooooo0010000//
10400419.2013.72 752249

---

Under review as a conference paper at ICLR 2023
Nan Liu, Shuang Li, Yilun Du, Joshua B. Tenenbaum, and Antonio Torralba. Learning to compose visual relations. In NewrIPS, 2021.
Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, and Hongsheng Li. Learning to predict layout- to-image conditional convolutions for semantic image synthesis. In Advances in Neural Informa- tion Processing Systems, 2019.
Yuhui Ma, Jiang Liu, Yonghuai Liu, Huazhu Fu, Yan Hu, Jun Cheng, Hong Qi, Yufei Wu, Jiong Zhang, and Yitian Zhao. Structure and illumination constrained gan for medical image enhance- ment. IEEE Transactions on Medical Imaging, (0(13):3955 -0663,977222 2021.
Alireza Makhzami. Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders, 2016.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 2014.
Nelson Nauata, Kai-Hung Chang, Chin-Yi Yi Cheng, Greg Mori, and Yasutaka Furukawa. House- gan: Relational generative adversarial networks for graph-constraindd house layout generation. In European Conference on Computer Vision, pp. 162-177. Springer, 2020.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2021. URL https:/ wwwpaaorwww.aaaaaaa/a//aaaaaaaaa /ttp://1111111 2112.10741
Baolin Peng. Zhengdong Lu, Hang Li, and Kam-Fai Wong. Towards neural network-based reason- ing. ArXiv, abs//508 0555080 2015.
Anil Audumbar Pise, Hima Vadapalli, and Ian Sanders. Relational reasoning using neural networks: A survey. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 29 (Supp02):237-258, 2021. doi: 10. 1 1422/5021848852140001 URL httpps:/ /doi.org/ 10. 1142 502184885214401133
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831. PMLR, 18-24 Jul 2021. URL
htttps:/pproceedingg: wwwpeediinnnn...r.reeemm mil.ppress/v 1339//amss2222 harr//mmmmaaaaammmmm
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2XXi:2204.00625 2022.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna- tional conference on machine learning, pp. 1530-1538. PMLR, 2015.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Pe- Battaglia, and Timothy Lillicrap. A simple neural network module for relational rea- soning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 30, Curran Associates, Inc., 2017. URL lhtpp::/ prbooceedingss.uuuips..c//ppoeed file/ http://eeeebbbbiffffffffeeeessss lff- wwtpsSaaaaal--PPPP.Pppppp Paper.pdi
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.). Pro- ceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256 2265, Lille, France, 07-09 Jul 2015. PMLR. URL https:// httpss//ppooccciigg.....t www.ll.reess/ooo v37
h5.mmll
Paul T. Sowden, Andrew Pringle, and Liane Gabora. The shifting sands of creative think-
ing: Connections dual-process theory. Thinking & Reasoning, 21(1):40 60, 2015. doi: 10088//15466888.01448888
URL Inttps:///oi..ni 11.1100.00)) 111111133477711114411
Trieu H. Trinh and Quoc V. Le. A simple method for commonsense reasoning, 2018. URL https:
//arxiv.org///e wowsvv.org/bs//// wtwgg//a/////.././///1111 1806.02847
12

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent
neural networks. Neural Computation, 1(2):270-280, 1989, doi: (1162//eco..9991..222
Kun Xu, Haoyu Liang, Jun Zhu, Hang Su, and Bo Zhang. Deep structured generative models. arXiv 008777 2018.
preprint
Yexiang Xue and Willem--an van Hoeve. Embedding decision diagrams into generative adversar- ial networks. In International Conference Integration of Constraint Programming. Artificial Intelligence, and Operations Research, pp. 616-632. Springer, 2019.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
13

---

Under review as a conference paper at ICLR 2023
A
PROPOSITIONAL LOGIC LANGUAGE
above
value
<object id>
<object id>
<offset>
below
Language obb)-ob))
Constraint
Language (obj-val)
Constraint
right
(ob) obj)
(obj - val)
left
above (a)
v. Ve
above_value (a,
b
wider
below
V. V.
below_value (a,bb)
Va > b
narrower
left (a,
X2 < %
left_value
< b
shorter
right
x2 %
#ight_value b)
X2 b
taller
shorter
h2 < hg
shorter_value (a,
h2 < b
(a,
n2 hgg
taller_value (a,
h2 b
narrower
H2 < 1%
narrower_value (a, (a,b) b)
Ma < b
yeq
(ider(a,
M1 W3
wider_value b)
b
weq
(e)
x, X
xeg_value ()
x2 b
heq
(e)
y. Vo
yeg_value (a,
b
(a,
M. W6
weq_value (a
M_ b
(a,
h. hg
heq_value ba
h.
or
<propostion
<propostion
prompt
<object id
<string>
and
<propostion>
<propostion>
type
<object id>
<cata id>
not
<propostion
Figure 5: Definition of the propositional logic constraint language. At the top are a list of positional logic propositions on the bounding box of objects and their mathematic definitions. The point (x, y) defines the upper left corner of the box, and (x + + h)) the lower right corner. Below, "Prompt" and "Type" constraints are assignment operators specifying the prompt (given to the visual element generator) and the type (given the spatial reasoning module). Logical operators and (A), or (V), and not (-) are also defined.
Prompts and constraints delivered in the form of a structured logical language. This language is visualized in 5 Features of this language include object-to-object constraints. object-to-value constraints, assignment constraints, and set of logical operators. Many more constraints could be implemented using this approach, but even the current ones can be used to compose complex scene structures by combining the logical operators with the positional logic propositions.
B SPATIAL REASONING MODULE EXPERIMENTAL SETUP
It is our aim that the spatial reasoning module be the center of integrating S1 and S2 cognition within the SPREN system, manifesting intelligent decisions based on dataset preferences and specified con- straints. However, dataset preferences regarding object location are difficult to measure precisely in real world datasets like COCO. To get around this, we define 3 synthetic datasets which are gener- ated from known distributions, allowing for evaluation against ground truth locations. underlying distribution of object locations is known to the programmer (not to the algorithm). This allows for the creation of a metric for blueprint generation the preference score. For each gener- ated location, if the generated value is within range of common values, it is considered satisfied preference. The mean number of satisfied preferences is a metric for how well the algorithm has leamed the dataset preferences. See Appendix F for full specification of the datasets and preference scores.
each, the
There very few current generation algorithms capable of zero-shot scene generation, which makes evaluation difficult. Despite this, SPREN planning is compared with a similar technique to ours, which also combines reasoning and deep learning, but offers less rich set of constraints.
14

---

Under review as a conference paper at ICLR 2023
Specifically. compare with GAN augmented with convex optimization layer Agrawal et al. (2019). Embedded CVX solvers in neural nets have been used successfully in several domains, including learning control Agrawal et al. (2020) and building convex optimization models Agrawal et al. (202)) Like ours, this approach can produce zero-shot constrained scenes, though the constraint is limited convex functions while ours not. See Appendix E for description of the GAN + CVX approach.
Each of the methods trained on the three datasets until convergence. Each dataset is also matched with a set of constraints during test time. As both methods use constraint reasoning, they are both able to achieve a perfect score on this limited constraint library. Both guarantee constraint satis- faction. However, as shown in Table I our algorithm outperforms or matched GAN + CVX in the satisfaction score metric for every dataset.
C FULL SPREN ALGORITHM EXPERIMENTAL SETUP
For more realistic test, SPREN must learn from a subset of the COCO dataset. 10,674 images each including some combination of dogs, horses, sheep, cows, elephants, bears, and giraffes. For training. foreground objects must be removed. This done using the GLIDE (reduced and filtered) network shared by our visual element generation module remove the known objects. This COCO background dataset and the code to reproduce it will be included supplementary materials.
Two tests of 60 random specifications were conducted. The first randomly selects two animals from the learned set (without replacement) and generates two random constraints. The second does much the same, but with three animals and four constraints. Specifications that were not satisfiable were thrown out and re-generated (e.g. "the dog is right of the cow and the dog is left of the cow"). Random constraints included left, right, above, below, narrower, wider, taller, and shorter.
Predietably, SPREN produced perfectly satisfying outputs at the blueprint phase. However, the goal of this test is to also test the visual element generation module within SPREN. The results of these final scenes were manually checked for object satisfaction (correct number and type of objects painted in), positional satisfaction (positional constraints satisfied among scenes that have achieved object satisfaction), and overall satisfaction (correct number and type of objects painted in with all positional constraints satisfied among all scenes). We observe from Table 2 that SPREN vastly outperforms GLIDE across all metrics, and particularly when scenes become complicated (3 objects, 4 constraints).
Figure shows some examples of our COCO-trained algorithm applied to custom background im- ages. We compare this part the algorithm with a GLIDE model inpainting the center the given background with a text prompt similar to the structured language constraint set. GLIDE CVX is not tested as it uses the same visual element generation module, and is not capable of many of the provided constraints.
The following tables specifies the precise constraint language inputs used to produce each SPREN image.
15

---

Under review as a conference paper at ICLR 2023
Constraint Language
Natural Language
prompt(O, cow); type(0,cow); prompt(1, dog); type(1,cd) ::po): prompt(2, horse); type(O,horse); below 40000,400.0); helow val(1,400,0)) helow_val(24400.0), or(
The dog is above and the horse to the left OR the dog below and the horse is to the right
teft2200,600,.eft221..600))
1ight22.0.600)rrihh(2.....)))
prompt(O,a golden wolf)
A gray wolf and a golden wolf.
type(0,dog)
The gray wolf is either half of image width to the left of the golden wolf
prompt(1, I,a gray wolf) (ype(1.dog)
OR half of the image width to the right.
or(leñ(0,1,5000, right(O.1,500))
prompt(O,a bright purple dog) type(0,dog)
prompt(1,a bright red dog)
lype(1,dog)
prompt(2,a bright yellow dog)
type(2,dog)
A purple dog, a red dog, green dog, and a yellow dog in square. They are ordered clockwise, starting with the upper left.
prompt(3,a bright green dog)
lype(3.dog)
1)q(0.1)
yeq(2.3)
x0q(0,2)
xe3(1,3)
1e4400,),400)
above(0,2,200)
prompt(O,, neon red sheep);
(ype(0,sheep)
prompt(1, white sheep)
type(1,sheep)
prompt(2, neon blue sheep)
A neon red sheep left of white sheep left of a neon blue sheep. All equally wide and lined up.
type(2,shecp)
1)q(0,1)
yeq(0.2)
weq(0,1)
weq(0,2)
lef((0,1,,00)
left(1,2.200)
prompt(O, a brown dog)
yype(0,dog)
prompt(1, green horse)
type(1,horse)
prompt(2, white sheep)
A brown dog left of green horse and a white sheep above the horse. All in the bottom 2/3rds of the image.
(ype(2,sheep)
left(0,1,,00)
above(2,1,300)
below_value((
below value(1,300
below value(2.300)
16

---

Under review as a conference paper at ICLR 2023
Constraint Language
Natural Language
prompt(O, sheep)
yype(0,hheep)
prompt(1, a dog) type(1,dog)
A sheep, dog, and giraffe. The sheep is left of the dog. The dog narrower than the giraffe. The dog is below the sheep.
prompt(2, giraffe) lype(2.giraffe)
left(0,1,200)
(1..00er(1,2.100)
(10,001,
prompt(O, flying pterodactyl) (ype(0,giraffe)
Two people flying kite were already the image. An elephant near the bottom of the image and a pterodactyl flying above.
prompt(1, an elephant) (ype(1,elephant) bcl0w(1,0,500))
Detected obj 0 (giraffe)
Detected obj (giraffe)
prompt(2, giraffe) lype(2.giraffee prompt(3, giraffe) ype(3-giraffe))
Two Giraffes were already in the image. Another were placed between and left of them.
(2,0
(3,2,5)
narrower(2.0)
narrower(3.0)
left(2,0,80)
right(3,0,80)
Detected obj (dog)
Detected obj (horse)
prompt(2, sheep)
A dog and horse were already in the image. A sheep and brown dog added above the horse. The sheep is left of the horse and the brown dog is right of the horse.
type(2, sheep)
prompt(3, brown dog)
type(3, dog)
1,00
above(3,1,100) left(2,0,50) right(3,1,200)
17

---

Under review as a conference paper at ICLR 2023
D ARCHITECTURES AND HYPERPARAMETERS
<left> <ight> <topp
Softmax
Linear (3)
Dropout (0.2)
ReLU
Linear (512)
GRU (500)
Linear (500)
GRU (500)
ReLU
ResNet18
GRU (500)
Linear (500)
hidden state
prev token
bg img
(obj label, var label)
Figure 6: The recurrent network (left), scene encoder (center), and variable encoder (right) of spatial reasoning net. These networks together allow our method to make sequential decisions on the position of objects in the scene.
E GAN + CVX BASELINE
The GAN + CVX approach relies on the recent ability to embed convex optimization solvers into neural layers in such a way that gradients can flow through. In short, this method accomplished by creating a GAN which outputs a set of parabolas one for each positional variable such that the neural net's "best guess" for location is at the vertex. Then the convex optimizer finds the optimal value of the sum of parabolas subject to constraints.
Background
pass
Inpaint Model
Backward pass
Oojact Data
FC or conv layers
Reasoning Engine
Diffusion layers
Convex opt layers
List
Other operation
extracted data
Scona Dataset
N(O.1)
CVX Layer
Location List
Lout
Location List (x111)....
Background
Mask Location
Pre-trained
Dbjjctt
Composed Image
pata
Object Data
Map Object
Solve for
Prompt
Constraint List
Minimize 1(x by) 1ly1 b2) ....
s.t. Constraint List
B
C
Figure 7: GAN CVX pipeline. A) The full pipeline as a unified system. Three subnetworks are used. A GAN which proposes list of positions given object data and constraint list, pre-trained inpainting model capable of adding the given objects to background scene, and an adversarial discriminator. B) The architecture the GAN. C) The inpainting model.
18

---

Under review as a conference paper at ICLR 2023
F SYNTHETIC DATASETS
Two random functions were used to compose the synthetic datasets. rnd(j, k) randomly generates an integer between and from normal distribution with mean + k= and standard deviation 2
1 Selected values rounded, and constrained between and k. For example, if object I's value is drawn from rnd(1, 500), then the mean of x will be 250.5 and the standard deviation will be 41.583. uni(j, k) does the same with uniform distribution between j and
F.1 DS_BASIC
example prompt "a white wolf" "a black wolf"
y
width (w)
height (h)
rnd(1 1,500)
uni 4000, 550)
rnd 292,256)
md(,12,250
rnd 1500,1000)
uni 400, 550)
rnd (192)556)
md(128,
Checked preferences:
1 S 0ol S 500
500 < xo2 2 1000
F.2 DS TIGHT
example prompt
width (w)
height (h)
"a pyramid
(nd(1,1000)
[300, 700)
(220,
(120, 150)
"a cube"
(nd11,000)
(300,70700)
md(120,150)
mdd1200150)
"a sphere"
rnd(1,1000)
rn00
m10110,)50)
mnd222,256)
Checked preferences:
220 S Wol S 256
120 < w,2 S 150
120 S wa & 150
120 S hot I 150
120 < ho2 I 150
220 < hos & 256
19

---

Under review as a conference paper at ICLR 2023
F.3 DS.COMPLEX
example prompt
x
y mdd(375,55)
white horse"
(1d01,000
"a brown horse"
rnd(1,1050)
rdd(ya1 10,3.1 + 10)
fantasy tower"
(nd(1,900)
rnd(1,144)
"A pile of colorful cubes"
(nd(1,1050)
rnd(400,665)
width (w)
height (h)
white horse"
rnd 128)
rnd(wa1 1.5,200)
"a brown horse"
rnd(64, 128
rnd(w x 1.5,200)
fantasy tower"
rnd (64, 128)
rnd 2,W 2)
"A pile of colorful cubes"
rnd(64, 128)
rnd(wo4 10,w,4 + 10)
Checked preferences:
S Zol S 1050
1 S xo2 1050
1 S 4o3 900
1 1 To4 1050
375 S Vel S 565
Yo1 10 & yo2 S yo1 + 10
1 S yos $ 144
400 S yo4 665
64 S wal < 128
64 S wa2 S 128
64 S was I 128
64 S wo4 I 128
Wol 1.5 hel S 200
Wo2 x 1.5 S ha2 S 200
x 2 S h13 & was x
10 S hot S Vos + 10
20

---

Under review as a conference paper at ICLR 2023
G
COCO DATASET
Figure 8: Six examples from the object-removed background testing Images like these fed to the scene encoder for feature extraction before iterative refinement begins.
We use the COCO 2017 Detection dataset for training and testing SPREN. During training, SPREN requires access background images with known ground truth object locations. As COCO does not supply background images without objects in the way. We get around this by utilizing our visual element generator to "paint over" objects to the background. An empty prompt is given to generator, and the object bounding box is covered by a mask, the generator has context to what the object is. This produces object-free backgrounds, or backgrounds that are only stripped of certain objects. We create background dataset from COCO training data composed of the 10,674 images which contain dog, horse, sheep, cow, elephant, bear, or giraffe. This dataset is checked manually, but due to its size, minor mistakes may be found in it. For testing, background dataset of size 20 was acquired in the same way, but was checked more carefully due to its manageable size. Both datasets will be available the time of publication.
21


Reviews:
[{"review_id": "qph2dcdCRne", "reviewer": "Reviewer_wJp9", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a model for spatial reasoning using RNNs for constrained image generation. They compare their method to two baselines and show improved performance on some metrics.", "summary_of_the_review": "Overall, the paper is low on standards such as evaluation and novelty.", "strength_and_weaknesses": "Strengths:\n1. The paper is well motivated.\n2. The idea of constrained image generation with spatial reasoning is interesting.\n3. The paper is easy to follow.\n\nWeaknesses:\n1. The experiments are very limited and use uncommon metrics to measure the image generation quality (there is no FID, Inception Score, Precision and Recall, or user study).\n\n2. The literature review is very limited and mostly covers the earlier works on the topic and very few recent works.\n\n3. The novelty is limited. The paper re-introduces existing concepts without referring to them. The proposed blueprints are the same as scene layouts commonly used in image generation literature. Also, the bounding box prediction network is not new. There are a lot of works on the topic of image generation and manipulation using scene graphs and semantic scene generation that are not covered in this work. E.g. sg2im [1] already predicts bounding boxes from a scene graph (similar to the constraints in this work), constructs a scene layout (blueprints), and generates an image conditioned on this. The image manipulation part, which performs addition / deletion / changes to the objects has been explored in SIMSG [2]. [3,4] already learn spatial reasoning based on scene graphs. [5] generates images from scene layouts.\n\n[1] Johnson et al. \"Image generation from scene graphs.\" CVPR 2018.\n[2] Dhamo et al. \"Semantic image manipulation using scene graphs.\" CVPR 2020.\n[3] Zareian, Alireza, et al. \"Learning visual commonsense for robust scene graph generation.\" ECCV 2020.\n[4] Garg et al. \"Unconditional scene graph generation.\" ICCV 2021.\n[5] Zhao et al. \"Image generation from layout.\" CVPR 2019.\n\n4. The paper needs better organization. There is too much repetition of text in introduction and methodology that leaves a small space to the experiments which are an essential part of a research paper.\n\n5. The evaluation metrics are not explained. What are pref score, constraint sat, obj sat, pos sat?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "j4x55jGMPU0", "reviewer": "Reviewer_4LXR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focus on a generative problem of Constrained Scene Generation (CSG) under the condition of zero-shot and object-aware. Inspired by psychophysiological studies, author propose a Spatial Reasoning Network (SPREN) to imitate the multiple systems of reasoning and memory in human beings. Specifically, the framework integrated the state-of-the-art image generation network with the proposed spatial reasoning module to generate image satisfying complex spatial constraints. Besides, they design a forward-checking strategy to ensure the model could handle the zero-shot and object-aware CSG without any fine-tuning. Experiments indicate that SPREN could generate images specifying constrained conditions and also verify the effectiveness in zero-shot and object-aware variants of this task.", "summary_of_the_review": "As mentioned above, the highlight of this paper is that author proposed a a plug and play module to enhance the off-the-shelf image generation networks with spatial reasoning ability, which meanwhile equipped with a good interpretation from the view of psychophysiological studies. But this work pointed out the key challenge of the spatial reasoning in CSG is vital for future studies. The whole framework is easy to implement and has well practicability, but the experiments is not enough to support its generalizability. Some experiments result still need further analysis. ", "strength_and_weaknesses": "Strength\n1)The greatest strength of this paper is that author propose a plug and play module to enhance the off-the-shelf image generation networks with spatial reasoning ability. Meanwhile, it also could handle zero-shot and object-aware CSG without any fine-tuning, which cost little to increase the quality of generated images. Thus, this work has well practicability and worth to generalize.\n2)Besides, author refer to the psychophysiological studies and provide a theoretical explanation that why the cognition needs to combine the spatial constraints, which represented in propositional logic to perform the iterative decision. In this case, the model could explicitly show the reasoning process, which has good interpretability.\n3)In experiments, author also verified the effectiveness of spatial reasoning module to stepwise discriminant analysis and evaluated the quality of generated constrained scenes.\n\nWeaknesses\n1)Although the proposed spatial reasoning network is effective alone, the quality of generated images still relies heavily on the image generation network, which is implemented by off-the-shelf framework GLIDE. To verity the generalization of spatial reasoning network, I think author should further explore different frameworks of image generation.\n2)According to my understanding, the visual element generation module needs to be sequentially called on each box from the blueprint. When the spatial reasoning module provide all the blueprint, image generation network could also directly inpaints all the masked area while keep the global semantic and spatial information. So, I think this sequential manner maybe a little redundant.\n3)In section 3.3, author choose 20% as the stop time, but it is not discussed in the experiment. I suggest to add relevant comparative experiments.\n4)In practical application, some large-scale pre-trained models could directly realize the similar function for the task of CSG, even under the condition of zero-shot. I think the quality of generated images maybe better than the cases you present in experiments. Meanwhile, the propositional logic mentioned in this paper also requires additional manage. So, how do you view this problem? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Biru_8BreF", "reviewer": "Reviewer_9XhE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method that composites objects onto a background image in a way that satisfies a set of spatial constraints. A neurally-guided search approach is taken, where a recurrent neural network predicts, via binary space partition, a distribution over possible locations of objects, which is used to guide the solution of a constraint satisfaction problem via backtracking. It is shown that the generated composite matches the constraints better than other general purpose image editing methods. No evaluation on image quality/realism is conducted.", "summary_of_the_review": "While I find the idea of the paper interesting and that there can be some potentials down this direction, ultimately I am underwhelmed by the justification of the design choices, and more importantly, but the low quality of the results and the lack of appropriate evaluations, particularly on image realism.", "strength_and_weaknesses": "# Strength:\n- Using neural nets to guide the solution of CSP is an appropriate solution for this task.\n- Output satisfy the constraints well.\n\n# Weaknesses:\n### Design choices\nIn general, there is very little justification for the design choices, to list a few:\n- What's the motivation for using this binary partitioning scheme to infer the spatial location of objects. What's the advantage of this over, say, predicting a mixture of gaussians / a distribution over discretized coordinates / etc.?\n- Is it reasonable to predict x and y separately? I assume they are highly correlated.\n- Is it sufficient to condition the the spatial reasoning module on the \"object identifier\"? Does this identifier provide enough information to determine the location of objects?\n- How is the visual element generation module implemented and how to ensure the generated image patch blends well into the background image?\n### Evaluation & Quality of results\nThe evaluation performed in this submission is largely insufficient:\n- Evaluation focused on the amount of constraints satisfied. However, amount of constraints satisfied is not the only thing one cares about when evaluating these images. As the authors acknowledge themselves, visual quality is also very important. No evaluation on the visual quality of the outputs is provided, and, judging from the few qualitative examples provided, these image look appear to be very unrealistic , with a lot of problem with perspectives, illumination, and plausibility of layout in general.\n- The comparison conducted is not fair. The proposed method is designed solely around constraint satisfaction and has access to constraint formulations which are not available to the baseline. The baseline is a general purpose framework that accounts for much more than spatial constraints. I expect to see baselines that specifically focuses on constraints, this can be simple baselines such as a pure CSP-based algorithm without the neural guide part.\n- I think it's probably more appropriate to evaluate the proposed method on datasets that focused more on object relations and layouts, since the focus of this work does not seem to be image realism anyways. Datasets that target compositionality e.g. CLEVR, or more real-world scene datasets e.g. 3D-FRONT, 3DSSG, can be more appropriate here.\n- No ablations at all, which leaves me unsure about a lot of things e.g. how much does the RNN help with solving the CSP? Do the RNN really learn proper location priors e.g. lambs are never placed in the sky?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject