Image Manipulation via Multi-Hop Instructions - A New Dataset and Weakly-Supervised Neuro-Symbolic Approach
Authors: Harman Singh
Poorva Garg
Mohit Gupta
Kevin Shah
Ashish Goswami
Satyam Modi
Arnab Kumar Mondal
Dinesh Khandelwal
Dinesh Garg
Parag Singla
Venue: EMNLP 2023 Main (2023)
Keywords: Neuro-Symbolic Reasoning
Natural Language Guided Image Manipulation
Visual Question Answering
Weakly Supervised Learning
TL;DR: We propose a weakly supervised neuro-symbolic approach for the problem of image manipulation using text instructions.

Abstract: We are interested in image manipulation via natural language text -- a task that is useful for multiple AI applications but requires complex reasoning over multi-modal spaces. We extend recently proposed Neuro Symbolic Concept Learning (NSCL), which has been quite effective for the task of Visual Question Answering (VQA), for the task of image manipulation. Our system referred to as NeuroSIM can perform complex multi-hop reasoning over multi-object scenes and only requires weak supervision in the form of annotated data for VQA. NeuroSIM parses an instruction into a symbolic program, based on a Domain Specific Language (DSL) comprising of object attributes and manipulation operations, that guides its execution. We create a new dataset for the task, and extensive experiments demonstrate that NeuroSIM is highly competitive with or beats SOTA baselines that make use of supervised data for manipulation.

Image Manipulation via Multi-Hop Instructions A New Dataset and Weakly-Supervised Neuro-Symbolic Approach
Harman Singh!
Poorva Garg
Mohit Cupta 1*
Kevin Shah Ashish Goswami
Satyam Modi
Arnab Kumar Mondal
Dinesh Khandelwal
Dinesh Garg² 2 IBM Research AI
Parag Singla
Indian Institute of Technology Delhi
Abstract
et al., 2019; Mao et al., 2019; Han et al.. 2019) over- come these limitations by combining the power of (purely) neural with (purely) symbolic representa- tions. Studies (Andreas et al., 2016; Hu et al., 2017; Johnson et al., 2017a; Mao et al.. 2019) have shown that neuro-symbolic models have several desirable properties such as modularity, interpretability and improved generalizability.
We are interested in image manipulation natural language lext -ar-aa task that is useful for multiple Al applications but requires complex reasoning over multi-moda spaces. We extend recently proposed Neuro Symbolic Concept Learning (NSCL) (Mao al., 2019), which has been quite effective for the task of Visual Question Answering (V (VQA), for the task of im- age manipulation. Our system referred as NEUROSIM can perform complex multi-hop reasoning over multi-objec scenes and only requires weak supervision in the form of an- notated data for VQA. NEUROSIM parses an instruction into a symbolic program, based on a Domain Specitic Language (DSL) compris- ing of object attributes and manipulation oper- ations, that guides its execution. We create new dataset for the task, and extensive experi- ments demonstrate that NEUROSIM is highly competitive with or beats SOTA baselines that make use of supervised data for manipulation.
Our aim in this work is to build neuro-symbolic models for the task of weakly supervised manipu- lation of images comprising multiple objects, via complex multi-hop natural language instructions. Specifically, we are interested in weak supervision that only uses the data annotated for VQA tasks, avoiding the high cost of getting supervised anno- tations in the form of target manipulated images. Our key intuition here is that this task can be solved simply by querying the manipulated representation without ever explicitly looking at the target im- age. The prior work includes weakly supervised approaches (Nam et al., 2018; Li et al., 2020) that require textual descriptions of images during train- ing and are limited to very simple scenes (or in- structions). (See Section 2 for a survey).
1 Introduction
The last decade has seen significant growth in the application of neural models to a variety of tasks including those in computer vision (Chen et al., 2017; Krizhevsky et al., 2012), NLP (Wu al., 2016), robotics and speech (Yu and Deng, 2016). It has been observed that these models often lack interpretability (Fan et al., 2021), and may not al- ways be well suited to handle complex reasoning tasks (Dai et al., 2019). On the other hand, clas- sical Al systems can seamlessly perform complex reasoning in an interpretable manner due to their symbolic representation (Pham et al., 2007; Cai and Su, 2012). But these models often lack in their ability to handle low-level representations and be robust to noise. Neuro-Symbole models (Dong
Our solution builds on Neuro-Symbolic Concept Learner (NSCL) proposed by (Mao et al., 2019) for solving VQA. We extend this work to incorpo- rate the notion of manipulation operations such change, add, and remove objects in a given image. As one of our main contributions, we design novel neural modules and a training strategy that just uses VQA annotations as weakly supervised data for the task of image manipulation. The neural modules are trained with the help of novel loss functions that measure the faithfulness of the manipulated scene and object representations by accessing separate set of query networks, interchangeably referred as quantization networks, trained just using VQA data. The manipulation takes place through inter- pretable programs created using primitive neural and symbolic operations from a Domain Specific
Equal Contribution, 'Equal Contribution. First four authors did this work while at IIT Delhi. Correspondence: harmansings i td0gmaal halmalll.co com, parags@iito ac. ac.

---

Output Manipulation program
Input: Source image
O scenc)
Change the thing large ball to big
symbalic
argel
Image Manipulator
O fitentO, sphers)
(NeURoSIM)
O c re reattrr. benind
Input: Instruction
chaneesbasp
Output: Manipulated image!
Figure 1: The problem setup. See Section for more details.
Language (DSL). Separately, a network is trained to render the image from a scene graph represen- tation using a combination of L1 and adversarial losses as done by (Johnson et al., 2018). The entire pipeline is trained without any intermediate super- vision. We refer to our system Image Manipulator (NEUROSIM), Figure shows example of I/O pair for our approach. Contribu- tions of our work are as follows:
multi-hop instructions, iii) kinds of manipulations allowed add, remove, or change. Finally, the ap- proach consists of the following sub-dimensions: i) model neural or neuro-symboll and ii) whether symbolic program is generated the way not. Dong al. (2017), TAGAN (Nam et al., 2018), and ManiGAN (Li et al., 2020) are close to in terms of the problem setting, These manipulate the source image using GAN-based encoder-decoder architecture. Their weak supervision differs from ours We need VQA annotation, they need cap- tions or textual descriptions. The complexity of their natural language instructions is restricted to O-hop. Most of their experimentation is limited to single (salient) object scenes.
Veuro-Symbolit
We create NEUROSIM, the first neuro-symbolic, weakly supervised, and interpretable model for the task of text-guided image manipulation, that does not require output images for training. 2. We extend CLEVR (Johnson et al., (b)) benchmark dataset for VQA, to incorporate ma- dataset
In terms of task complexity, the closest to approaches such TIM-GAN (Zhang et al., 2021). Ge (EI-Nouby et al., 2019), which build an encoder-decoder architecture and work with a latent representation of the image as well as the manipulation instruction. They require a large number manipulated images as explicit annotations for training.
nipulation instructions and create called as Complex Image Manipulation via Nat- ural Language Instructions (CIM-NLI). We also create CIM-NLI-LARGI dataset to test zero-shot generalization.
3. We provide extensive quantitative experi- ments on newly created CIM-NLI, CIM-NLI- LARGE datasets along with qualitative exper- iments Minecraft (Yi et al., 2018). Despite being weakly supervised, NEUROSIM is highly competitive to supervised SOTA approaches including recently proposed diffusion based model (Brooks et al., 2023). NEUROSIM also performs well on instructions requiring multi- hop reasoning, all while being interpretable. We publicly release our code and data !
In terms of technique, the closest to our work are neuro-symbolic approaches for VQA such as NSVQA (Yi et al., 2018). NSCL (Mao et al., 2019), Neural Module Networks (Andreas et al., 2016) and its extensions (Hu al., 2017; Johnson et al., 2017a). Clearly, while the modeling approach is similar and consists of constructing latent pro- grams, the desired tasks are different in the two cases. Our work extends the NSCL approach for the task of automated image manipulation.
2
Related Work
Jiang et al. (2021),Shi et al. (2021) deal with editing global features, such as brightness, con- trast, etc., instead of object-level manipulations like in our case. Recent models such as Instruct- Pix2Pix (Brooks et al., 2023), DALL-E (Ramesh et al., 2022) and Imagen (Saharia al., 2022) on text-to-image generation using diffusion models are capable of editing images but require captions for input images; preliminary studies (Mareus et al., 2022) highlight their shortcomings in composi-
Table categorizes the related work across three broad dimensions problem setting, task complex- ity, and approach. The problem setting comprises sub-dimensions: i) supervision type self, di- rect, weak, ii) instruction format text or UI- based. The task complexity comprises of following sub-dimensions: ii) scene complexity - single or multiple objects, ii) instruction complexity zero or
chttps://gittee

---

Problem Setting
Task Complexity
Approach
Prior Work
Supervision Type
Instruction Format
SC
IC
Operations
Model Program
SIMSG
Self Supervision
UI
MO
N/A
change, remove, add
X
PGIM
Direct Supervision
N/A
MO
N/A
change (image level)
VeNeVA
Direct Supervision
Text
MO
Multi-Hop add
N
X
TIM-GAN
Direct Supervision
Text
MO
Zcro-Hop change, remove, add
X
Dong al
Weak Supervision
Text
SO
Zero-Hop change
N
X
TAGAN
Weak Supervision
Text
SO
Zero-Hop change
N
X
ManiGAN
Weak Supervision
Text
SO
Zero-Hop change
InstructPix2Pix
Pre-training Supervision
Text MO Multi-Hop change, remove, add
N
X
NEUROSIM (ours)
Weak Supervision
Text MO Multi-Hop change, remove, add
NS
Table I: Comparison of Prior Work. Abbreviations (column titles) SC:= Scene Complexity, IC==Instruction Complexity. Abbreviations (column values) MO:= Multiple Objects, MO" Multiple Objects with Regular Patterns, sO:= Single Object, N:= Neural, NS:= Neuro-Symholic, N/A:= Not applicable, /:= Yes, X: No. See Section 2 for more details.
tional reasoning and handling relations.
Figure 3 captures a high-level architecture of the proposed NEUROSIM pipeline. NEUROSIM allows manipulating images containing multiple objects, via complex natural language instructions. Similar to Mao al. (2019), NEUROSIM sumes the availability of domain-specife lan- guage (DSL) for parsing the instruction text T into an executable program P. NEUROSIM is capable of handling addition, removal, and change opera- tions over image objects. It reasons over the image for locating where the manipulation needs to take place followed by carrying out the manipulation operation. The first three modules, namely i) vi- sual representation network, ii) semantic parser, and iii) concept quantization network suitably customized from the NSCL and trained as required for our purpose. what follows, we describe the design and training mechanism of NEUROSIM.
3 Neuro-Symbolic Image Manipulator
3.1 Motivation and Architecture Overview The key motivation behind our approach comes from the following hypothesis: consider learner L (e.g., a neural network or the student in Fig 2) with sufficient capacity trying to achieve the task of manipulation over Images Further, let each image be represented in terms of its properties, or properties of its constituents (e.g. objects like ap- ple, leaf, tree, etc. in Fig 2), where each property comes from finite set S c.g. attributes of objects in image. Let the learner be provided with the prior knowledge (for e.g. through Question Answering as in Fig 2) about properties (e.g., color) and their possible values (c.g., red). Then, in order to learn the task of manipulation, it suffices to provide the learner with a query network, which given a ma- nipulated image i constructed by the learner via command C, can correctly answer questions (i.e. query) about the desired state of various properties of the constituents of the image j. The query net- work can be internll the learner (e.g., the student in Fig 2 can query himself for checking the color of apples in the manipulated image). The learner can query repeatedly until it learns to perform the ma- nipulation task correctly. Note, the learner does not have access to the supervised data corresponding to triplets of the form (L,, CyJ,), where I, is the starting image, C is the manipulation command, and I the target manipulated image. Inspired by this, we set out to test this hypothesis by building a model capable of manipulating images, without target images as supervision.
3.2 Modules Inherited from NSCL
1] Visual Representation Network: image this network converts it into scene graph G₁ (N, E). The nodes N of this scene graph are object embeddings and the edges E are embed- dings capturing the relationship between pair of objects (nodes). Node embeddings are obtained by passing the bounding box of each object (along with the full image) through ResNet-34 (He et al., 2016). Edge embeddings are obtained by concate- nating the corresponding object embeddings. 2] Semantic Parsing Module: The input to this module a manipulation instruction text T in nat- ural language. Output
Given input
symbolic program P generated by parsing the input text. The symbolic programs are made of operators, that are part

---

Part
applac thee
apples
Red Green
Leuriig Innsehlgge of conotsts cirouass coleoon concepts eg color, apple GrA pare
Loariig 1o munixuatto imsage self notionioo using krowladge conceple atricaee sogaiee
Figure 2: Motivating example NEUROSIM. Best viewed under magnification. See Section 3.1 for more details
Program P
Instruction Text
O soord
Change wize
G lirgel
Manipulation
0 benn sphwei
Manipuleed image
Pi Sepeorii
imagee/
11] Vaual
Doncept Quantization Netwark
Reprisontation Network
Score graph Gj for target image
Program Executor
Scane graph G for source image
Figure 3: High level architecture of NEUROSIM. See Section 3 for more details.
our DSL (Specified in Appendix Section A). 3] Concept Quantization Network: Any object in an image is defined by the set of visual at- tributes (A), and set of symbolic values (S,) for each attribute a A. E.g., attributes can be shape, size, etc. Different symbolic values allowed for an attribute are also known as concepts. E.g.. Scolor = (red, blue, green....). Each visual at- tribute a E A is implemented via a separate neu- ral network fa(-) which takes the object embed- ding as input and outputs the attribute value for the object in continuous (not symbolic) space. Let feotor Rdjjj R dai represent neural network for the color attribute and consider o Bd as the object embedding. Then, Ccolor = fcotor(o) E RR is the embedding for the object o pertaining to the color attribute. Each symbolic concept E Sa for a particular attribute a (c.g., differrnt colors) is also assigned respective embedding in the same con- tinuous space dun, Such embedding is denoted by Cs- These concept embeddings are initialized random, and later on, fine-tuned during train- ing. An attribute embedding (e.g. Vealor) can be compared with the embeddings of the concepts (c.g., Ged, Cblue. etc.) using cosine similarity, for the purpose of concept quantization of objects. Training for VQA: As first step, we train the above three modules via curriculum learning pro- cess (Mao et al., 2019). The semantic parser is trained jointly with the concept quantization net- works for generating programs for the question texts coming from the VQA dataset. The corre- sponding output programs are composed of prim- itive operations coming from the DSL (e.g. filter,
count, etc.) and do not include constructs related to manipulation operations. This trains the first three modules with high accuracy on the VQA task.
3.3 Novel Modules and Training NEUROSIM
NEUROSIM training starts with three mad-modules trained on the VQA task as described in Section 3.2. Next, we extend the original DSL. to include three additional functional sub-modules within the se- mantic parsing module, namely add, remove, and change. Refer to appendix section A for details the DSL. We now reset the semantic parsing module and train it again from scratch for generat- ing programs corresponding to image manipulation instruction text T. Such program is subsequently used by the downstream pipeline to reason over the scene graph Gj and manipulate the image. this step, the semantic parser is trained using an off- policy program search based REINFORCE (Williams, 1992) algorithm. Unlike the training of semantic parser for the VQA task, in this step, do not have any final answer like reward supervision for training. Hence, we resort weaker form of su- pervision. particular, consider an input instruc- tion text T and set all possible manipulation pro- gram templates P. from which one can create any actual program P that is executable over the scene graph of the input image. For program P E Pt, our reward is positive if this program P selects any object (or part of the scene graph) be sent to the manipulation networks (change/add/removv)) See A Appendix C for more details. Once the semantic parser is retrained, we clamp the first three modules and continue using them for the purpose of parsing

---

instructions and converting images into their scene graph representations. Scene graphs are manipu- lated using our novel module called manipulation network which is described next.
the (symbolic) value of the attribute a for the manipulated object is different from the desired value 8 in terms of probabilities. The second term la. on the other hand, penalizes the model if the values of any of the other attributes a', deviate from their original values. Apart from these losses, we also include following additional losses.
4] Manipulation Network: This is our key module responsible for carrying out the manipulation op- erations. We allow three kinds of manipulation op- erations addd, remove, and change. Each of these operations is composition of a quasi-symbolic and symbolic operation. A symbolic operation cor- responds to a function that performs the required structural changes (i.e. addition/delltion or an edge) in the scene graph G against given instruction. A quasi-symbolic operation is a ded- icated neural network that takes the relevant part of 12 as input and computes new representations of nodes and edges that are compatible with the changes described in the parsed instruction.
Laycle = 10 90 (0; cata)ll;;
(3)
Econsistency = 10 Io oa Cota)]l2
(4)
node
fobjGAN = Evco Dog D((o')
(5)
+ log D (ga(d':e)))
where Cold is the original value of the attribute of object o, before undergoing change. Intuitively the first loss term Leycle says that, changing an object and then changing it back should result in the same object. The second loss term fconsiscency intuitively means that changing an object that has value Cold for attribute a, into new object with the same value Cold, should not result in any change. These additional losses prevent the change network from changing attributes which are not explicitly taken care of in earlier losses (1) and (2). For e.g., ro- tation or location attributes of the objects that are not part of our DSL. We also impose an adversarial loss LubjAAN to ensure that the new object embed- ding is from the same distribution real object embeddings. See Appendix C for more details. (b) Remove Network: This network takes the scene graph Gy of the input image and removes the subgraph from G2 that contains the nodes (and incident edges) corresponding to the object(s) that need to be removed, and returns a new scene graph Gj which is reduced in size. The quasi-symbolic function for the remove network is identity.
(a) Change Network: For each visual attribute a E A (e.g. shape, size, ...). ), we have a separate change neural network that takes the pair of (object embedding, embedding of the changed concept) as input and outputs the embedding of the changed ob- ject. This is the quasi-symbolic part of the change function, while the symbolic part is identity map- ping. For e.g., let Gcolor Rayay datrr -> Rdd repre- sent the neural network that changes the color of an object. Consider o E Ra as the object embed- ding and Grod E Rd as the concept embedding for the red color, then ő - Icolor (0; Crea) E aay repre- sents the changed object embedding, whose color would be red. After applying the change neural network, we obtain the changed representation of the object ō ga (o; csz), where st is the desired changed value for the attribute a. This network is trained using the following losses.
ta- -- Σ -==s;
log (ha (ō) = =]]
(1)
(c) Add Network: For adding a new object into the scene, add network requires the symbolic val- ues of different attributes, say {Say Sazs.... Saz }. for the new object, e.g., {red, cylinder. It also requires the spatial relation r (e.g. RightOf) of the new object with respect to an existing object in the scene. The add function first predicts the object (node) embedding onew for the object to be added, followed by predicting edge embeddings for new edges incident the new node. New object embedding is obtained as follows: Ōnew JaddObj ( {Ce., Csay: ... Cs.y J. Oreis C7) where, Orel is the object embedding of an existing object, rela- tive to which the new object's position r is speci- fied. For each existing objects 0 in the scene, an
VsESa
f - Σ
Σ (0) = s)+
(2)
V,' to V. log(p(ha (ō) = s)]
where, ha(x) gives the concept value of the attribute a (in symbolic form s E Sa) for the object x. The quantity p (ha((x) = 8) de- notes the probability that the concept value of the attribute a for the object x is equal to s and is given as follows p (h)(x)) = s)
exprist(fa(())e.)) / dist(a, b) Ta 12)/tn the shifted and scaled cosine similarity, t1, f2 being constants. The first loss tern la penalizes the model
Eies, (xpdist(fa(a).c)))
where,

---

edge Enew, is predicted between the newly added object Onew and existing object o, in following manner: Cnewsi = GaddEdge (Onew, 0z). Functions (:ddObj(.) and (.)) are guasi-symbolic oper- ations. Symbolic operations in add network com- prise adding the above node and the incident edges into the scene graph.
which is the processed by GCN and passed through a mask regression network followed by a box regression network to generate a coarse 2- dimensional structure (scene layout). A Cascaded Refinement Network (Chen and Koltun, 2017) is then employed to generate an image from the scene layout. A min-max adversarial training procedure is used to generate realistic images, using patch- based and object-based discriminator.
The add network is trained in self-supervised manner. For this, we pick training image and create its scene graph. Next, we randomly select an object o from this image and quantize its concepts, along with a relation with any other object o; in the same image. We then use our remove network to remove this object o from the scene. Finally, we use the quantized concepts and the relation that were gathered above and add this object back into the scene graph using 9mdatbn(') and IuddFigee (-). Let the embedding of the object after adding back is Onew- The training losses are as follows:
Experiments
Datasets: Among the existing datasets, CSS (Vo et al., 2019) contains simple O-hop instructions and is primarily designed for the text-guided image retrieval task. Other datasets such as i-CLEVR (El- Nouby et al., 2019) and CoDraw are designed for iterative image editing. i-CLEVR contains only "add" instructions and CoDraw doesn't contain multi-hop instructions. Hence we created our own multi-object multi-hop instruction based image ma- nipulation dataset, referred to as CIM-NLI. This dataset was generated with the help of CLEVR (8) toolkit (Johnson et al., 2017b). CIM-NLI con- sists of (Source image 7, Instruction text T, Target image I*) triplets. The dataset contains total of 18K, 5K, 5K unique images and 54K, 14K, 14K instructions the train, validation and test splits respectively. Refer to Appendix B for more details about the dataset generation and dataset splits.
Ceoncepts
Σ j=1
log (p (Ōnew) = $az))
=8a3))
(6)
relatioon
(p(hh((((ewwwr) 0i) r))
fobjSup
110 Tnewllz
Sup
Σ iEO /luuaai Enew,ele 12
BedgeGAN
Σ [log D(fo; eold,; oi))+
viEo
log(1 D ((ōnew; Enew,a; o;)))]
(10)
where Say is the required (symbolic) value of the attribute ay for the original object o, and r is the re- quired relational concept. O is the set of the objects in the image, Cold, is the edge embedding for the edge between original object and its neighboring object og. Similarly, Enew,e is the corresponding em- bedding of the same edge but after when we have (removed + added back) the original object. The loss terms fconcepes and érelation ensure that the added object comprises desired values of attributes and relation, respectively. Since we had first removed and then added the object back, we already have the original edge and object representation, and hence we use them in loss terms given in equation 9, We use adversarial loss equation 10 for generating real (object, edge, object) triples and also a loss similar to equation for generating real objects.
Baselines: We compare our model with purely su- pervised approaches such TIM-GAN (Zhang et al., 2021), GeNeVA (EI-Nouby et al., 2019) and InstructPix2Pix (Brooks et al., 2023). In or- der to make a fair and meaningful comparison be- tween the two kinds (supervised and our, weakly- supervised) approaches, we carve out the following set-up. Assume the cost required to create one sin- gle annotated example for image manipulation task be am while the corresponding cost for the VQA task be o,,, Let a a..//a.. Let 8., be the num- ber of annotated examples required by a supervised baseline for reaching performance level of 7m on the image manipulation task. Similarly, let 33 be the number of annotated VQA examples required to train NEUROSIM to reach the performance level of n/e Let 3 B S... We are interested in figur- ing out the range of 3 for which performance of our system (n,) at least as good the baseline (nm) Correspondingly we can compute the ratio of the labeling effort required, i.c., a * 3, to reach these performance levels.
3.4 Image Rendering from Scene Graph
5] Rendering Network: Following Johnson et al. (2018), the scene graph for an image is first generated using the visual representation network,

---

Input Image
Instruction
Ground Truth
NEUROSIM
TIM-GAN
GeNeVA
IP2P
There is rubber thing front of the red matte ball; change the cylinder.
Remove matte cylinder left of the small object that behind the tiny cube.
There & large sphere, add small purple matte ball behind
Figure 4: Visual comparison of NEUROSTM with various baselines. See Section 4.4 for more details.
If 3 1, system achieves the same better performance, with lower annotation cost, Weakly supervised models (Li et al., 2020; Nam et al., 2018) are designed for a problem setting different from ours single salient object scenes, simple O-hop instructions (Refer Section 2 for de- tails). Further, they require paired images and their textual descriptions annotations. We, therefore, do not compare with them in our experiments. See Appendix G, H for computational resources and hyperparameters respectively.
8
0.054
8 0.54
Method
FID
R11
R3
rsim
FID
R1
R3
rsim
GeNeVA TIM-GAN
42.6 6.6
58.7 80.1
28.5
4.6
64.4 84.9
24.2
31.9
74.2
88.4
22.7
58.1
90.2 91.0
IP2P
3.4
40.6
77.0
88.8
2.2
49.2
81.8
94.5
NEUROSIM 35.0 45.3
65.5 91.3
35.1
45.5
66.7
91.5
Table 2: Performance comparison of NEUROSIM with TIM-GAN and GeN and InstructPix2Pix (IP2P) with 10% data (3 0.054) and full data (B 0.54). We always use 100K VQA examples (5K Images. 20 questions per image) for our weakly supervised training R1, R3 correspond to Recall @ 1,3 respectively. FID: lower is better; Recall/rsim: higher is better. See Section 4.1 for more details.
Evaluation Metrics: For evaluation on im- age manipulation task, we use three metrics - i) FID, ii) Recall@k; k, and iii) Relational-similarii (rsim). FID (Heusel et al., 2017) measures the realism of the generated images. We use the implementation proposed in Parmar al. (2022) compute FID. @ecall@k: measures the semantic similarity of gold manipulated image I" and system produced manipulated image T. For computing Recall@k, k. we follow Zhang et al. (2021), i.e. we use I as query and retrieve images from a corpus comprising the entire test set. rsim measures how many of the ground truth relations between the objects image. We follow (El-Nouby et al., 2019) to implement rsim metric that uses predictions from trained object-detector (Faster-RCNN) perform relation matching between the scene-grape gene-graphs of ground-truth and generated images.
based IP2P in full data setting, using the R@1 performance metric. This clearly demonstrates the strength of our approach in learning to manipulate while only making use of VQA annotations. We hypothesize that, in most cases, NEUROSIM will be preferable since we expect the cost of annotating an output image for manipulation be significantly higher than the cost of annotating a VQA example. To reach the performance the NEUROSIM in low data regime, TIM-GAN requires a larger number of expensive annotated examples (ref. Table 13 in Appendix). The FID metric shows similar trend across dataset sizes and across models. The FID scores for NEUROSIM could potentially be improved by jointly training VQA module along with image decoder and is a future direction.
present in the generated
4.1 Performance with varying Dataset Size
We evaluate InstructPix2Pix (IP2P) (Brooks et al., 2023), state-of-the the-arrttttt pre-trained diffusion model for image editing, in zero-shot manner the CIM-NLI dataset. Considering its extensive pre-training, expect IP2P have learned the concepts present in the CIM-NLI dataset. In this setting IP2P achieves a FID score of 33.07 and R01 score of 7.48 illustrating the limitations
Table 2 compares the performance of NEUROSIM other SoTA methods two level of 3 0.054 and 0.54 representing use of 10% and 100% samples from CIM-NLI. Despite being weakly supervised, NEUROSTM performs significantly better than the baselines with just 10k data samples (especially TIM-GAN) and not too far from diffusion model

---

large-scale models in effectively executing com- plex instruction- editing lasks without full dataset fine-tuning. Table 2 contains the results ob- tained by IP2P after tee-ttning for 16k iterations on CIM-NLI dataset.
model for zero-shot generalization, despite being weakly sueprvised.
4.4 Qualitative Analysis and Interpretability
Figure shows anecdotal examples for visually comparing NEUROSIM with baselines. Note, GeNeVA either performs the wrong operation on the image (row #1, 2, 3) or simply copies the input image to output without any modifications. TIM- GAN often makes semantic errors which show its lack of reasoning (row #3) or make partial edits (row #1). IP2P also suffers from this where it edits incorrect object (row #1,2). Compared to baselines, NEUROSIM produces semantically more mean- ingful image manipulation. NEUROSIM can also easily recover occluded objects (row #4). For more results, see Appendix I, J. NEUROSIM produces interpretable output programs, showing the steps taken by the model to edit the images, which also helps in detecting errors (ref. Appendix L).
Larger Scenes
Hops
Method
R1
R3
ZH
MH
A
Ge 54K
5.0
65.8
6.3
6.4
(+0.1)
GeNeVA 5.4K
8.2
64.6
8.5
9.9
(-1.4)
TIM-GAN 54K
66.3
92.4
84.0
76.2
(-78)
TIM-GAN 5.1K
30.2
80.7
56.4
41.6
(-148)
IP2P 54/
69.2
99.8
72.5
67.4
(-5.0)
IP2P 5.4K
64.9
99.4
69.3
54.8
(-14.5)
NEUROSIM 5.4K
63.7
89.1
64.5
63.0
(-1.5)
Table 3: (Left) Performance on generalization to Larger Scenes. (Right) RI results for 0-hop (ZH) vs multi- hop (MH) instruction-guided image manipulation. See Sections 4.2 and 4.3 for more details.
4.2 Performance versus Reasoning Hops
4.5 Evaluating Manipulated Scene Graph
Table 3 (right) compares baselines with NEU- ROSIM for performance over instructions requiring zero-hop (ZH) versus multi-hop (1 hops) (MH) reasoning. Since there are no Add instructions with ZH, we exclude them from this experiment for the comparison to be mcaningful. Ge performs abysmally on both ZH as well as MH. We see sig- nificant drop in the performance of both TIM-GAN and IP2P when going from ZH to MH instructions, both for training on 5.1K, as well as, 54K data- points. In contrast, NEUROSIM trained on 10% data, sees a performance drop of only 1.5 points showing its robustness for complex reasoming tasks.
We strongly believe image rendering module of NEUROSIM pipeline and encoder modules used for computing Recall@k add some amount of inefficiencies resulting in lower R1 and R3 scores for us. Therefore, we decide to as- sess the quality of manipulated scene graph Gf. For this, we consider the text guided image retrieval task proposed by (Vo et al., 2019). In this task, an im- age from the database has to be retrieved which would be the closest match to the desired manipulated image. Therefore, we use our ma- nipulated scene graph G the latent representation of the input instruction and image for image retrieval. We retrieve images from the database based on novel graph edit distance between NEUROSIM generated Gj of the desired manipulated images, and scene graphs of the im- ages in the database. This distance is defined using the Hungarian algorithm (Kuhn, 1955) with a sim- ple cost defined between any 2 nodes of the graph (ref. Appendix D for details). Table 4 captures the performance of NEUROSIM and other popular baselines for the image retrieval task. NEUROSIM significantly outperforms supervised learning base- lines by a margin of ~ 50% without using output image supervision, demonstrating that NEUROSIM
Method
R1 R3
Text-Only
0.2 0.4
Image-Only
34.1 83.6
Concat
39.5 86,9
TIRG
34.8 84.6
NEUROSIM 85.8 92.9
4.3 Zero-shot Generalization to Larger Scenes
Table 4: Gy Quality via image retrieval.
We developed another dataset called CIM-NLI- LARGE, consisting of scenes having 10- 13 objects (See Appendix B for details). We study the combinatorial generalization ability of NEU- ROSIM and the baselines when the models are trained on CIM-NLI containing scenes with 3- - 8 objects only and evaluated on CIM-NLI-LARGE. Table 3 captures such a comparison. NEUROSIM does significantly better, i.e., 33 pts (R1) than TIM- GAN and is competitive with IP2P when trained on 10% (5.4K data points) of CIM-NLI. We do see drop in performance relative baselines when they are trained on full (54K) data, but this is ex- pected as effect of supervision takes over, and ours is a weakly supervised model. Nevertheless, this experiments demonstrates the effectiveness of our

---

meaningfully edits the scene graph. Refer to Sec- tion 4.7 for human evaluation results and Appendix Section D-E, K, for more results including results on Minecraft dataset and ablations.
Qn. NEUROSIM TIM-GAN
IP2P
5.4K
54K
54K
Q1
0.41
0.27
0.25
4.6 A Hybrid Approach using NEUROSIM
Q2
0.33
0.84
0.78
From Table 3, we observe that both TIM-GAN and IP2P suffer significant drop in performance when moving from ZH to MH instructions, whereas NEUROSIM is fairly robust to this change. Further, we note that the manipulation instructions in our dataset are multi-hop in terms of reasoning, but once an object of interest identified, the actual manipulation operation can be seen as single hop. We use this observation to design a hybrid super- vised baseline that utilizes the superior reasoning capability of NEUROSIM IM and high quality editing and generation capabilities of IP2P.
Table 6: Human evaluation comparing various models.
4.7 Human Evaluation
For the human evaluation study, we presented 10 evaluators with a set of five images cach, including: The input image, the ground-truth image and ma- nipulated images generated by NEUROSIM 5.4K, TIM-GAI 54K, and IP2P 54K. Images generated by the candidate models were randomly shuffled to prevent any bias. Evaluators were asked two binary questions, each requiring 'yes' (1) or 'no' (0) sponse, assess the models: (QI) Does the model perform the desired change mentioned in the input instruction?, (Q2) Does the model not introduce any undesired change elsewhere in the image? Re- fer to Appendix Section M for more details about exact questions and the human evaluation process. The average scores from the evaluators across different questions can be found in Table 6. The study achieved high average Fleiss' kappa score (Fleiss et al., 2013) of 0.646, indicating strong inter-evaluator agreement. Notably, NEU- ROSIM (5.4K) outperforms TIM-GAN and IP2P (54K) in QI suggesting its superior ability to do rea- soning, and identify the relevant object as well as af- fect the desired change. In contrast, TIM-GAN and IP2P score significantly better in Q2, demonstrat- ing their ability not to introduce unwanted changes elsewhere in the image, possibly due to better gen- eration quality compared to NEUROSIM.
We take the CIM-NLI test set and parse the text- instructions through our trained semantic-parser to obtain the object embeddings over which the ma- nipulation operation is to be performed. We utilize our trained query networks to obtain the symbolic attributes such as color, shape, size and material of the identified object. Using these attributes we simplify complex multi-hop instruction into simple instruction with hops using a simple template based approach (see Appendix Section N for details). These simplified instructions are fed to the fine-tuned IP2P model to generate the edited images. We refer to our hybrid approach as IP2P- NS where NS refers to Seuro-Symbolic Table 5 presents the results. We find that there is clear advantage of using hybrid seuro-symbolii model integrating NEUROSIM with IP2P. We see a sig- nificant gain on FID, recall, rsim when we use the hybrid approach, especially in the low resource set- ting (5 = 0.054). Compared to IP2P, the hybrid neuro-symbolic approach results in better FID, re- call and rsim scores, except a small drop in RI for
Conclusion
We present neuro-symbolic. interpretable ap- proach NEUROSIM to solve image manipulation task using weak supervision in the form of VQA annotations. Our approach can handle multi-object scenes with complex instructions requiring multi- hop reasoning, and solve the task without any out- put image supervision. We also curate dataset of image manipulation and demonstrate the potential of our approach compared supervised baselines. Future work includes understanding the nature of errors made by NEUROSIM, having human in the loop to provide feedback to the system for correc- tion, and experimenting with real image datasets.
3 = 0.51 setting. This opens up the possibility of further exploring such hybrid models in future for improved performance (in the supervised setting).
8 = 0.054
9 = 0.54
Method
FID
R1
R3 rsim
FID
R1
R3 rsim
IP2P
3.4 40.6 77.0 88.8
2.2 49.2
84.8 94.5
NEUROSIM
35.0 45.3
65.5
91.3
35.1
45.5
66.7 91.5
IP2P-NS
1.96 45.5 83.2 94.0
1.8
48.0
85.5 95.6
Table 5: Comparison between IP2P-NS and IP2P.

---

6 Ethics Statement
or official policies, either expressed or implied, of the funding agencies.
All the datasets used in this paper were syntheti- cally generated and do not contain any personally identifiable information or offensive content. The ideas and techniques proposed in this paper are useful in designing interpretable natural language- guided tools for image editing, computer-aided de- sign, and video games. One of the possible ad- verse impacts of Al-based image manipulation is the creation of deepfakes (Vaccari and Chadwick, 2020) (using deep learning to create fake images). To counter deepfakes, several researchers (Dolhan- sky et al., 2020; Mirsky and Lee, 20 have also looked into the problem of detecting real vs. fake images.
References
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural net- works for question answering. In Proceedings of NAACL-HLLT, pages 1545-1554.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. 2023. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of CVPR, pages 18392-18402.
Shaowei Cai and Kaile Su. 2012. Configuration check- ing with aspiration in local search for SAT. In Pro- ceedings AAAI, pages 434
Liang-Chieh Chen, George Papandreou, Jasonas Kokki- nos, Kevin Murphy, and Alan I. Yuille. 2017. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully con- nected erfs. IEEE transactions pattern analysis and machine intelligence, 40(4):834 848.
7 Limitations
A limitation of our approach is that when trans- ferring to a new domain, having different visual concepts requires not only learning new visual con- cepts but also the DSL needs to be redefined. Auto- matic learning of DSL from data has been explored in some prior works (Eilis et al., 2021, 2018), and improving our model using these techniques are future work for us. We can also use more power- ful graph decoders for image generation, for im- proved image quality, which would naturally result stronger results on image manipulation.
Lichang Chen, Guosheng Lin, Shijie Wang, and Qingyao Wu. 2020. Graph edit distance reward: Learning to edit scene graph. In Proceedings of ECCV, pages 534.554.
Qifeng Chen and Vladlen Koltun. 2017, Photographic image synthesis with cascaded refinement networks. In Proceedings of ICCV, pages 1520-1529.
Blender Online Community, 2018. Blender - a 3D mod- elling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam.
Acknowledgements
We thank anonymous reviewers for their insight- ful suggestions that helped in greatly improving our paper. We also thank Rushil Gupta and other members of IT Delhi DAIR group for their helpful comments and suggestions on this work. This work was supported by an IBM AI Horizons Network (AIHN) grant. We thank IIT Delhi HPC facility? IBM cloud facility, and IBM Cognitive Computing Cluster (CCC) for computational resources. Ashish Goswami is a PhD student at Yardi-ScAl @ IIT- Delhi and is supported by Yardi School of AI Pub- lication Grant. Parag Singla was supported by the DARPA Explainable Artificial Intelligence (XAI) Program with number N66001-17-2-403: IBM AI Horizon Networks (AIHN) grant and IBM SUR awards. Any opinions, findings, conclusions or rec- ommendations expressed in this paper are those of the authors and do not necessarily reflect the views
Wang-Zhou Dai, Qiu-Ling Xu, Yang Yu, and Zhi- Zhou. 2019. Bridging machine learning and logical reasoning by abductive learning, Proceedings of NeurlPS, pages 281 1-2222.
Brian Dolhansky, Joanna Bitton, Ben Pilaum, Jikuo Lu, Russ Howes, Menglin Wang. and Cristian Can- ton Ferrer. 2020. The deepfake detection challenge (dfde) dataset. ArXiv preprint, abs/2006.07397.
Hao Dong, Simiao Yu, Chao Wu, and Yike Guo. 2017. Semantic image synthesis via adversarial learning. In Proceedings of ICCV, pages 5707-5715.
Honghua Dong. Jiayuan Mao, Tian Lin, Chong Wang. Lihong Li, and Denny 7hou. 2019. Neural logic machines. Proceedings of ICLR.
Alaacidin El-Nouby. Shikhar Sharma. Hannes Schulz, R. Devon Hjelm, Layla El Asti, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W. Taylor. 2019. Tell, draw, and repeat: Generating and modifying images based on continual linguistic instruction. Proceedings of ICCV, pages 10303-10311.
httpp:/suuprroompuuttiiiiiaddadt iitd.ac..

---

Kevin Ellis, Lucas Morales, Mathias Sablé-Meyer, Ar- mando Volar-Lezama, and Josh Tenenbaum. 2018. Learning libraries of subroutines for neurally-guided bayesian program induction. In Proceedings of NeurIPS.
compositional language and elementary visual rea- soning. In Proceedings of CVPR. pages 1988-1997.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of TCLR.
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Mey Meyer, Lucas Morales, Luke Hewitt, Luc Cary,
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hin- ton. 2012. Imagenet classification with deep convo- lutional neural networks, In Proceedings of NeurlPS. pages
Armando Solar-Lezama, and Joshua B Tenenbaum. 2021. Dreamcoder: Bootstrapping inductive pro- gram synthesis with wake-sleep library learning. In Proceedings of PLDI, pages 835-850.
H. W. Kuhn. 1955, The hungarian method for the assign- ment problem. Naval Research Logistics Quarterly, 2(1-2) 211-2):83-97.
Feng-Lei Fan, Jinjun Xiong, Mengzhou Li, and Ge Wang. 2021. On interpretability of artificial neu- ral networks: A survey. IEEE Transactions on Radi- ation and Plasma Medical Sciences, 5(6):741-760.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip H. S. Torr. 2020. Manigan: Text-guided im- age manipulation. In Proceedings of CVPR, pages 7877-7886. IEEE
Joseph L Fleiss, Bruce Levin, and Myunghee Cho Paik. 2013, Statistical methods for rates and proportions. john wiley & sons.
Ilya Loshchilov and Frank Hutter, 2019, Decoupled weight decay regularization. In Proceedings of ICLR.
Tan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of NeurlPS, pages 2672-2680.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. 2019. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In Proceed- ings of ICLR.
Chi Han, Jiayuan Mao, Chuang Gan, Josh Tenenbaum, and Jiajun Wu. 2019. Visual concept-metaconcept learning. In Proceedings of NeurIPS, pages 5002- 5013.
Gary Marcus, Emnest Davis, and Scott Aaronson. 2022.
A very preliminary analysis of dall-e 2. ArXiv preprint, abs/2204. 13807.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In Proceedings of CVPR, pages 770-778.
Yisroel Mirsky and Wenke Lee. 2021. The creation and detection of deepfakes: A survey, ACM Computing Surveys (CSUR), 54(1):1-41.
Seonghyeon Nam, Yunji Kim, and Scon Joo Kim. 2018. Text-adaptive generative adversarial networks: Ma- nipulating images with natural language. In Proceed- ings of NeurIPS, pages 42-51.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by two time-scale update rule converge to local nash equilibrium. In Proceedings of NeurIPS, pages 6626-6637.
Gaurav Parmar, Richard Zhang, and Jun- Yan Zhu. 2022. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of CVPR, pages 11400- 11410.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of ICCV.
Due Nghia Pham, John Thornton, and Abdul Sattar, 2007. Building structure into local search for sat. In Proceedings of IJCAI, pages 2359-2364
Wentao Jiang. Ning Xu, Jiayun Wang, Chen Gao, Jing Shi, Zhe Lin, and Si Liu. 2021. Language-guided global image editing via cross rooss-modal cyclic mecha- nism. In Proceedings of ICCV, pages
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text- conditional image generation with clip latents. ArXiv preprint, abs/2204.061
Justin Johnson, Agrim Gupta, and Li Fei-Fei. 2018. Image generation from scene graphs. In Proceedings of CVPR, pages 1228.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emiil Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding, ArXiv preprint, abs/2205.1 11487.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zit- nick, and Ross Girshick. 2017a. Inferring and execut- ing programs for visual reasoning. In Proceedings of ICCV, pages 3008-3017.
Jing Shi, Ning Xu, Yihang Xu, Trung Bui, Franck Der- noncourt, and Chenliang Xu. 2021. Learning by planning: Language-guice guided global image editing. Proceedings of CVPR, pages 13590-13599.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, Lawrence Zitnick, and Ross B. Girshick. 2017b. CLEVR: A diagnostic dataset for

---

Cristian Vaccari and Andrew Chadwick. 2020. Deep- fakes and disinformation: Exploring the impact of synthetic political video on deception, uncertainty, and trust in news. Social Media+ Sociery, 6(1).
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, 2019. Li Fei-Fei, and James 2019. Composing text and image for image retrieval empirical odyssey. Proceedings of CVPR, pages
Ronald Williams. 1992. Simple statistical gradient-
following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-256)
Yonghui Wu, Mike Schuster. Zhifeng Chen, Quee V Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. ArXiv preprint, htss1660,0881
Kexin Yi. Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. 2018. Neural- symbolic VQA: disentangling reasoning from vi- sion and language understanding. In Proceedings of NewrlPS. pages 1039-1050.
Dong Yu and Li Deng. 2016. Automatic speech recog- nition, volume I. Springer.
Tianhao Zhang, Hung-Yo Yu Tseng, Lu Jiang, Weilong Yang, Honglak Lee, and Irfan Essa. 2021. Text as neural operator: Image manipulation by text instruc- tion. In Proceedings of ACM MM, pages 1893-1902.

---

Appendix
Each instructions text T comprises one of the following three kinds of manipulation operations add, remove, and change.
A Domain Specific Language (DSL)
Table 7 captures the DSL used by our NEUROSIM
An add instruction specifies color, shape, size, and material of the object that needs to be added. It also specifies direct (or indirect) relation with one or more existing objects (called reference ob- jec The number of relations that are required traverse for nailing down the target object is referred to as # of reasoning hops and we have allowed instructions with up to 3-hops reasoning. We do not generate any O-hop instruction for add due to ambiguity where to place the object inside the scene.
pipeline. The first 5 constructs in this table are common with the DSL used in Mao et al. (2019). The last 3 operations (Change, Add, and Remove) were added by us to allow for the manipulation operations. Table shows the type system used by the DSL in this work. The first 5 types are inherited from (Mao et al., 2019) while the last one is an extension of the type system for handling the inputs to the Add operator.
B Dataset Details
A change instruction first specifies zero or more attributes to uniquely identify the object that needs to be changed. It may also specify a direct (or indirect) relation with one more existing reference objects. Lastly, it specifies the target values of attribute for the identified object which needs to be changed.
We CLEVR dataset and CLEVR toolkit (code to generate the dataset). These are public and are under CC and BSD licenses respectively, and are used by many works, including ours, for research purposes. We now give details of the datasets we create, building upon CLEVR.
B.1 CIM-NLI Dataset
remove instruction specifies zero or more at- tributes of the object(s) to be removed. Addition- ally. it may specify a direct (or indirect) relation with one or more existing reference objects.
This dataset was generated with the help of CLEVR toolkit (Johnson et al., 2017b) by using following recipe.
First, we create a source image / and the corre- sponding scene data by using Blender (Commu- nity, 2018) software.
Table 9 captures the fine grained statistics about the CIM-NLI dataset. Specifically, it further splits each of the train, validation, and test set across the instruction types add, remove, and change.
2. For each source image I created above, we gen- erate multiple instruction texts T's using its scene data. These generated using templates, similar to question templates proposed by (John- son et al., 2017b).
B.2 CIM-NLI-LARGE Dataset
We created another dataset called CIM-NLI- LARGE to test the generalization ability of NEU- ROSIM on images containing more number of ob- jects than training images. CIM-NLI-LARGE tests the zero-shot transfer ability of both NEU- ROSIM and baselines scenes containing more objects.
3. For each such (I, pair, we attach a corre- sponding symbolic program P (not used by NEUROSIM though) as well as scene data for the corresponding changed image.
Each image in CIM-NLI-LARGI dataset com- prises of 10 - 13 objects as opposed to 8 objects in CIM-NLI dataset which was used to train NEU- ROSIM. The CIM-NLI-LARGE dataset consists of K unique input images. We have created 3 instructions for cach image resulting in total of 3K instructions. The number of add instructions is significantly less since there is very little free space available in the scene to add new objects. To create scenes with 12 and 13 objects, we made all objects as small size and the minimum distance between
4. Finally, for each TT) pair, we generate the target gold image I* using Blender software and its scene data from the previous step.
Below are some of the important characteristics of the CIM-NLI dataset.
Each source image comprises several objects and each object comprises four visual attributes color, shape, size, and material.

---

Operation Signature [Output + Impurl)
Semantics
Scene
ObjSet + 0
Returns all objects in the scene.
Filter out set of objects from ObjSet that have a concept (e.g. red) specified Obj- Concept.
Filter
ObjSet (ObjSet, ObjConcept)
Filter out set of objects from ObjSet that have concept specified relation cept (e.g. RightOf) with object Obj.
ObjSet
(ObjSet, RelConcept,
Relate
Obj)
Returns the Attribute value for the object Obj.
Query
ObjConcept (Obj, Attribute)
Exist
Bool (ObjSet)
Checks if the ObjSet is empty.
Changes the attribute value of the input object (Obj), corresponding to the input concept, Concept
Change
Obj <- (Obj, Concept)
Adds an object to the input graph, gener- ating new graph having the object with attribute values ConceptSet, and present in relation RelConcept of the input Obj
Graph <
Add
(Graph, RelConcept, Obj. Concept- Set)
Removes the input objects and their edges from the input graph output graph
Remove
Graph <
(Graph, ObjSet)
Table 7: Extended Domain Specific Language (DSL) used by NECROSIM.
objects was reduced so that all objects could fit in the scene. Table 10 captures the statistics about this dataset.
C
Model Details
C.1 Semantic Parser
C.1.1 Details on Parsing
We begin by extending the type system of (Mao et al., 2019) and add ConceptSet because our add operation takes input set of concepts depict- ing attribute values of the new object being added (refer Table 8 for the details). Next, in a manner similar to (Mao et al., 2019), we use rule based system for extracting concept words from the input text. We, however, add an extra rule for extracting ConceptSet from the input sentence. Rest of the semantic parsing methodology remains the same as given in (Mao et al., 2019), with the difference being that our training is weakly supervised (refer Section 3.3 of the main paper).
B.3 Multi-hop Instructions
In what follows, we have given examples of the instructions that require multi-hop reasoning nail down the location/object to be manipulated in the image.
Remove the tiny green rubber ball. (0-hop)
There is block right of the tiny green rubber ball, remove it. (1-hop)
C.1.2 Training
Remove the shiny cube left of the block in front of the gray thing. (2-hop)
As explained in Section 3.3 of the main paper, for training with weaker form of supervision, we use an off-policy program search based REINFORCE (Williams, 1992) algorithm for calculating the ex- act gradient. For this, we define a set of all possible program templates IP, For a given input instruc-
Remove the small thing that left of the brown matte object behind the tiny cylinder that is be- hind the big yellow metal block. (3-hop)

---

Type
Remarks
ObjConcept
Concepts for any given object, such as blue, cylinder, etc.
Attribute
Attributes for any given object, such as color, shape, etc.
RelConcept
Relational concepts for any given object pair, such as RightOf. LeftOf, etc.
Object
Depicts single object
ObjectSet
Depicts multiple objects
ConceptSet
A set of elements of ObjConcept type
Table 8: Extended type system for the DSL used by NEUROSIM.
# reasoning hops
# objects
Operation
Split
#1,T,7*
min
mean
max
min
mean
max
train
17827
2.00
3
3
5.51
Add
valid
4459
2.00
3
3
5.50
8
test
4464
2.00
3
3
5.45
train
15999
1.50
3
3
5.50
8
Remove valid
5000
1.50
3
3
5.50
test
5000
1.50
3
3
5.48
train
19990
1.50
3
3
5.45
Change
valid
4996
1.50
3
3
5.56
test
4998
1.50
3
3
5.52
Table Statistics of CIM-NLI dataset introduced in this paper.
tion text T, we create a set of all possible pro- grams {Pr} from Pt, For e.g. given a template (remove(relate(, (illter(;,sceee())) filled in all possible ways, with concepts, concept- Set, attributes and relational concepts extracted from the input sentence to get programs for this par- ticular template. All such programs created using all templates form the set P,. All Py are executed over the scene graph of the input image. A typical program structure in our work is of the form ma- nip_or hip_op((eusonne reasoonnn)))))) 1), where manip op represents the manipulation operator, for example change, add, or remove; and reasoning() either selects ob- jects for change or remove, or selects reference object for adding another object relation to After hyperparameter search for the reward (refer
Section H of the appendix), we assign reward of +8 if the reasoning() part of the program leads to object being selected for change/remove in- struction
this is
related object being selected for add instruction. no such object is selected, we give a reward of +2. Reward values were decided on the basis validation set accuracy. We find that with this training strategy, we achieve the validation set accuracy of 95.64%, where this accuracy is calcu- lated based whether program lead to object being selected or not. Note, this is proxy the actual accuracy. For finding the actual accuracy, we would need validation of (instruction, ground truth output program) pairs, but we do not use this supervised data for training or validation.

---

# reasoning hops
objects
Operation
(1,T,I1)
min
mcan
max
min
mean
max
Add
393
2.0
3
10
11.53
13
Remove
524
0
1.50
3
10
11.48
13
Change
2083
0
1.51
3
10
11.50
13
Table 10: Statistics of CIM-NLI-LARGE dataset.
C.2
Manipulation Network
The remove network is a sym- bolic operation as described in Section 3.3 the main paper. That is, given an input set of objects, the remove operation deletes the subgraph of the scene graph that contains the nodes corresponding to removed objects and the edges incident those nodes. See Fig 5e for overview of the remove operator.
Remove Network:
In what follows, we provide finer details of manip- ulation network components.
Change Network: As described in Section 3.3 of the main paper, we have a change neural ner- work for each attribute. For changing the current attribute value of given object o, we use the fol- lowing neural network: Ō = (o; Csg), where s is the desired changed value for the attribute is the new representation of the object. We model (-(+) by single layer neural network without hav- ing any non-linearity. The input dimension of this neural network is (256- + 64) because we concate- nate the object representation o E R225 with the desired concept representation d E R64, We pass this concatenated vector through (.).) to get the revised representation of the object: ō E 2226
Add Network: The neural operation in the add operator comprises of predicting the object rep- resentation for the newly added object using a function IaddObi(-). This function is modeled as single layer neural network without any acti- vation. The input to this network is a concate- nated vector [[Ceo] Gouz ? !.. Coay. Orets G.). where [Cau] Goag?""" Coog represents the concatenation of all the concept vectors of the desired new ob- jects. The vector Orel is the representation of the object with whom the relation (i.e. position) of the new object has been specified and c, is the concept vector for that relationship. The input dimension of (addOby(-) is (I: + 64 + 256 + 64) and the output dimension is 256. For predicting representation of newly added edges in the scene graph, we use edge predictor IaddEdge The input to this edge pre- dictor function is the concatenated representation of the objects which are linked by the edge. The input dimension of gaddEdge (-) is (256 + 256) and the output dimension 256.
The loss used train the weights of the change network is weighted sum of losses equation! to equation given in the main paper. This leads to the overall loss function given below.
Loverall_change >1 la - la + A leycle
A4 Rconsistency + As LobjAAN
where, lobjGAN above is the modified GAN loss (Goodfellow al., 2014). Here A1 1, A2 1/((num_ _atrrs--------- (num_concepps)). Ag A4 103, and As 1/ (nnummnujeettt Here, suum_bjeect)) the number of objects in input image, (num_attrs) is the total number attributes for each object, and number of concepts in the NSCL (Mao et al., 2019) framework.
The loss used train the add network weights is a weighted sum of losses equation 6 to equation 10 along with an object discriminator loss. The overall loss is given by the following expression.
are the total
The object discriminator is neural network with input dimension 256 and single 300 dimensional hidden layer with ReLU activation function. This discriminator is trained using standard GAN ob- jective LobGAAN. See Fig 5a for an overview of the change operator
Loverall add = >1 + Agerelation
Ashotisup Aa
+ AsPedgeCAN A6GobGAAN
where, LobjGAN and lodgeGAN above denotes the modified GAN loss (Goodfellow et al., 2014). Here

---

Graph with changed object
Object Embedding
Original Graph
Changed Object Embedding
Icolor
Input concept Embedding: Red
(a) Change operator overview.
Graph with added object
Original Graph
Related Object
Added Object Embedding
JaddObj
Embedding
Relation Embedding
New edge embedding
JaddEdge (.)
Concept Set
(b) operator overview.
Graph with object removed
Object to remove
Original Graph
(c) Remove operator overview.
Figure 5: Overview of new operators (change, add and remove) added to the DSL..

---

A1 X2 1/(num attrs) X 24 10a.
Note, for each such (I, pair, some image from the database, say I D, is assumed to be the ideal image that should ideally be retrieved at rank-1. 1, This, so called desired gold retrieval image might even be an image which is the ideal manipulated version of the original images I in terms of satis- fying the instruction T perfectly. Or, image I may not be such an ideal manipulated image but it still may be the image in whole corpus D that comes closest to the ideal manipulated image.
A6- 1/(numoobeets).
The object discriminator is a neural network with input dimension as 256 and a single 300 dimen- sional hidden layer with ReLU activation function. This discriminator is trained using the standard GAN objective LuIJAAN. Note, LubjGAAN has 2 parts i) the loss for the generated (fake) object embed- ding using the add network, and ii) the loss for the real objects (all the unchanged object embeddings of the image). The former is unscaled but the latter one is scaled by a factor of 1/ (mum__bjjctaa
In practice, while measuring the performance of any such system for this task, the gold manipulated image for (T, T)) pair is typically inserted into the database D and such an image then serves as the desired gold retrieval image J.
The edge discriminator is neural network with input dimension (256- 3) and single 300 di- mensional hidden layer with ReLU activation func- tion. As input to this discriminator network, we pass the concatenation of the two objects and the edge connecting them. This discriminator is trained using the standard GAN objective fedgeciAN- See Fig 5b for an overview of the add operator
Our baselines includes popular super- vised learning systems designed for this task. The first baseline is TIRG proposed by Vo et al. (2019) where they combine image and text to get a joint embedding and train their model in supervised manner using embedding of the desired retrieved image as supervision. For completeness, we also include comparison with other baselines Concat, Image-Only and Text-Only - that were introduced by Vo et al. (2019).
Baselines:
Additional Results
D.1 Detailed Performance for Zero-Shot Generalization on Larger Scenes
Table 11 below is detailed version of the Table 3 in the main paper. This table compares the per- formance of NEUROSIM with baseline methods TIM-GAN, Ve and IP2P for the zero-shot generalization to larger scenes (with 2 10 objects), while the models were trained on images with 3 - 8 objects. Relative to the main paper's 's table 3, this table offers separate performance numbers for each of the add, remove and change instructions.
A recent model proposed by Chen et al. (2020) uses symbolic scene graphs (instead of embed- dings) to retrieve images from the database. Moti- vated by this, we also retrieve images via the scene graph that is generated by the manipulation mod- ule of NEUROSIM. However, unlike Chen et al. (2020), the nodes and edges in our scene graph have associated vectors and make a novel use of them while retrieving. We do not compare our per- formance with (Chen et al., 2020) since its code is unavailable and we haven't 't been able to repro- duce their numbers datasets used in their paper. Moreover, (Chen et al., 2020) uses full supervision of the desired output image (which is converted to a symbolic scene graph), while we do not.
D.2 Image Retrieval Task
A task that is closely related to the image manip- ulation task is the task of Text Guided Image Re- trieval, proposed by (Vo et al., 2019). Through this experiment, our is to demonstrate that NEU- ROSIM is highly effective in solving this task as well. In what follows, we provide details about this task, baselines, evaluation metric, how we adapted NEUROSIM for this task, and finally performance results in Table 12. This table is a detailed version of the Table in the main paper.
Evaluation Metric: We use Recall@k (and re- port results for k = 1, for evaluating the perfor- mance of text guided image retrieval algorithms which standard in the literature.
Retrieval using Scene Graphs: We use the scene graph generated by NEUROSIM as the latent representation to retrieve images from the database. We introduce novel yet simple method to retrieve images using scene graph representation. For con- verting an image into the scene graph, we use the vi-
Task Definition: Given an Image I, a text in- struction T. and database of images D, the task is to retrieve an image from the database that is se- mantically as close to the ground truth manipulated image as possible.

---

Train
Add
Change
Remove
Method
Data
Size
R1
R3
R1
R3
R1
R3
GeNeVA
54K
0.5
64.6
4.9
69.9
9.0
50.0
Ge
5.4K
0.0
60.1
8.2
69.2
14.3
49.6
TIMGAN
54K
12.3
77.4
73.4
95.2
78.2
92.2
TIMGAN
5.4K
1.0
70.0
32.1
84.4
44.7
74.0
IP2P
51K
38.2
100.0
72.7
100.0
78.6
98.9
IP2P
5.4K
34.1
100.0
68.8
100.0
72.5
96.4
NEUROSIM
5.1K
3.8
46.6
68.2
95.8
90.7
94.3
Table 11: Detailed performance scores for NEUROSIM, TIM-GAN, GeNeVA and IP2P for zero-shot generalization to larger scenes (with 2 10 objects) from CIM-NLI-LARGE dataset, while models are trained on images with 8 objects. Table has separate performance numbers for add, remove, and change instructions. Along with each method, we have also written the number of data points from CIM-NII dataset that were used for training. /?1 and R33 correspond to 1ecall@11 and Recall respectively.
sual representation network of NEUROSIM. Given the scene graph G for the input image and the manipulation instruction text T, NEUROSIM con- verts the scene graph into the changed scene graph G7. as described in Section C in Appendix. Now, we use this graph G7 as a query to retrieve im- ages from the database D. For retrieval, we use the novel graph edit distance (GED) between Gj and the scene graph representation of the database images. The scene graph for each database image is also obtained using the visual representation net- work of NEUROSIM. The graph edit distance is given below.
simple yet highly effective approach (See Table 4 in the main paper and Table 12 in the appendix), can be improved by more sophisticated techniques that include distance between edge embeddings and including notion of subgraphs in the GED. We leave this as future work. This result shows that our manipulation network edits the scene graph in desirable manner, per the input instruction.
D.3 Detailed Multi-hop Reasoning Performance
Table 14 below provides a detailed split of the performance numbers reported in Table 3 of the main paper across i) number of hops (0 3 3 hops) and ii) type of instructions (add/reeovve/canne)) We observe that for change and remove instruc- tions, NEUROSIM improves over TIM-GAN, GeNeVA and IP2P trained on 5.4K CIM-NLI data points by significant margin (~ 20% on 3- hop change/remove instructions). However, NEU- ROSIM lags behind TIM-GAN when the entire CIM-NLI labeled data is used to train TIM-GAN, We also observe that all the models perform poorly on the add instructions, as compared to change and remove instructions.
GED(G;, [mineen Eme(t cff...)) otherwise.
where, GJ (N7, Vi) and GD (Nd, VD). n; and y; are the node embeddings of the query graph Gj and scene graph GD of an image from the database. c(a, b) is the cosine similarities be- tween embeddings a and b. This GED much simpler than that defined in (Chen et al., 2020), since it does not need any hand designed cost for change, removal, or addition of nodes, or different attribute values. It can simply rely on the cosine similarities between node embeddings. We use the Hungarian algorithm (Kuhn, 1955) for calculating the optimal matching ग of the nodes, among all pos- sible matching TT. We use the negative the cosine similarity scores between nodes to create the cost matrix for the Hungarian algorithm to process. This
D.4 Detailed Performance for Different Cost Ratios 3
Table Section 4 of the main paper showed the performance of NEUROSIM compared with TIM- GAN and GeNe VA for various values of 3, where

---

Train
Add
Change
Remove
Method
Data
Size
1
2
3
0
2
3
2
3
Text-Only
54K
0.4
0.3
0.3
0.1
0.0
0.1
0.1
0.1
0.3
0.3
0.0
Image-Only
54K
35.3
33.4
32.3
20.1
23.2
16.9
19.8
46.3
41.3
53.1
57.8
Concat
54K
36.3
33.3
31.8
37.3
40.4
34.2
37.9
41.8
41.0
50.0
55.0
TIRG
54K
35.6
31.8
33.5
22.0
25.1
18.8
22.0
46.6
42.7
52.5
56.1
NEUROSIM
5.4K
96.2
95.3
95.3
83.3
82.9
81.3
78.7
79.6
77.4
86.4
82.2
Table 12: Performance scores (Recall 1) on the Image Retrieval task, comparing NEUROSIM with TIM-GAN and GeNeVA with increase in reasoning hops, add, remove, and change instructions. Along with each method, number of data points from CIM-NLI used for training are written.
3 the ratio of the number of annotated (with output image supervision) image manipulation ex- amples required by the supervised baselines, to the number of annotated VQA examples required to train NEUROSIM. In Table 13, we show detailed split of the performance, for the add, change, and remove operators, across the same values of B as taken before.
dataset used (Yi et al., 2018). Specifically, we create zero and one hop remove instructions
and one hop add instructions similar to the cre- ation of CIM-NLI. This dataset contains scenes and objects from the Minecraft video game and is used in prior works for testing Neuro- Syuuo-yy VQA systems like NSCL (Mao et al., 2019) and NS- -Q- (Yi al., 2018). The setting of the Minecraft worlds dataset is significantly different from CLEVR terms concepts and attributes of objects and visual appearance.
We find that for the change operator, NEU- ROSIM performs better than TIM-GAN by mar- gin of ~ 8% (considering Recall @ 1) for < 0.1. For the remove operator, NEUROSIM performs bet- ter than TIM-GAN by margin of ~ 1% (consider- ing Recall@) for / S 0.2. Overall, NEUROSIM performs similar to TIM-GAN, for 3 0.2, for remove and change operators. All models perform poorly on the add operator as compared to the change and remove operators, We find that having full output image supervision allows TIM-GAN to reconstruct (copy) the unchanged objects from the input the output for all the operators. This results in higher recall in general but its effect is most pronounced in the Recall@3. NEUROSIM, on the other hand, suffers from rendering errors which makes the overall recall score (especially @)call@3) lower. We believe that improving image rendering quality would significantly improve the performance of NEUROSIM and we leave this as future work.
Experiment: We use the above dataset for test- ing the addition and removal of objects using Neu- roSIM (See Fig 6). We train NeuroSIM's de- coder to generate images from scene graphs of the minecraft dataset. We assume access to a parser that gives us programs for an instruction. For re- moval, we use the same remove network as de- scribed above, while for addition, we assume ac- cess the features of object to be added, which is added to the scene graph of the image and the decoder decodes the final image. See Figure for of successful examples on the Minecraft dataset. We see that using our method, one can add and re- move objects from the scene successfully, without using any output image as supervision during train- ing. Though we have assumed the availability of a parser in the above set-up, training it jointly with other modules should be straightforward, and can be achieved using our general approach described in Section of the main paper.
D.5 Results on Datasets from different domains
E End-to-end Training
D.5.1
Minecraft Dataset
Dataset Creation: We create a new dataset having (Image, instruction) by building over the Minecraft
The main objective of this work is to make use weakly supervised VQA data for the image manipu-

---

3 = 0.054
B = 0.07
B = 0.1
8 = 0.2
[ = 0.54
Method
Instruction
R1
R3
R1
R3
R1
R3
R1
R3
R1
R3
add
0.0
57.3
0.7
63.6
Ve
change
5.9
36.3
4.1
39.4
remove
13.2
82.3
8.7
89.3
add
1.9
70.7
4.9
74.0
8.6
76.7
10.3
77.1
13.1
78.6
TIM-GAN
change
41.0
72.1
42.9
73.5
49.8
77.3
62.5
84.2
78.3
92.3
remove
49.6
79.5
47.0
91.9
53.9
93.1
65.3
96.8
78.0
98.5
add
0.7
70.5
5.4
78.6
IP2P
change
54.9
72.8
61.5
78.9
remove
62.0
87.2
76.0
96.3
add
4.9
30.9
6.4
34.8
5.7
34.7
5.9
38.9
5.6
35.0
NEUROSIM
change
57.2
79.4
57.3
79.3
57.2
79.3
57.2
79.4
57.1
79.3
remove
69.6
82.5
69.5
82.5
69.5
82.6
69.5
82.5
69.6
82.5
Table 13: Detailed performance comparison of NEUROSIM with TIM-GAN (Zhang et al., 2021), GeNeVA (El- Nouby al., 2019) and IP2P (Brooks al., 2023) with varying 3 levels, split across add, remove and change instructions. The '-' entries for GeNeVA and IP2P were not computed due to excessive training time (inference time as well in case IP2P); Geneva's 's performance is abysmal even when using data. TIM-GAN does the best among baselines in terms of its recall score at 3 0.54. We always 100K VQA examples (5K Images, 20 questions per image) for our weakly supervised training. It1 and It3 correspond to Recall and 3, respectively. For Recall, higher score is better.
lation task without using output image supervision. But natural extension of our work is to use output image supervision as well, to improve the perfor- mance of NEUROSIM. We devised experiment compare how much performance boost can be obtained by utilizing ground truth output (manipu- lated) images as the supervision for different mod- ules NEUROSIM. This experiment demonstrates the value of end-to-end training for NEUROSIM and how it can exploit the supervised data. We refer to this variant as NEUROSIM(e2e), We begin with pre-trained NEUROSIM model trained with VQA annotations and then fine-tune it using super- vised manipulation data. The detailed results are given in Table 15. This experiment demonstrates that with a small amount of supervised data, the performance of NEUROSIM can be significantly improved (e.g.. more than points increase for the change instruction with only 5.1K supervision examples)
Given the significant increase in performance of NEUROSIM when using supervised data, we also test it's generalization capability (Analogous to Section 4.2, 4.3), and quality of scene graph retrieval (Analogous to Section 4.5
From Table 16, we see that NEUROSIM(e2e)) shows improved zero-shot generalization to larger scenes. Even when trained on just 5.4k CIM-NLI data, NEUROSIM((2e) improves over TIM-GAN- 54k by 3.9 R @ points. 5.3 point improvement over TIM-GAN is observed when full CIM-NLI data is used.
Next, we measure drop in performance with in- creasing reasoning hops. From Table 17, we see that NEUROSIM((2e) achieves the lowest drop when compared to TIM-GAN. NEUROSIM(e2e) improves over weakly supervised NEUROSIM baseline by 6.6 R@ points.
Finally, we measure the quality of scene graphs via retrieval. From Table 18, we see that super-

---

Train
Add
Change
Remove
Method
Data
Size
1
2
0
2
3
0
1
3
GeNeVA
54K
1.1
0.5
0.5
3.6
4.2
4.7
3.9
9.0
8.0
8.3
9.4
VA
5.4K
0.0
0.0
0.0
4.7
7.1
5.6
6.1
12.3
11.3
15.5
13.5
TIM-GAN
54K
7.6
16.1
15.7
85.8
74.1
78.0
75.4
82.2
68.3
81.9
79.7
TIM-GAN
5.4K
1.4
2.3
1.9
54.5
36.4
38.7
34.5
58.3
40.9
50.9
48.2
IP2P
51K
6.6
5.5
4.2
67.9
60.6
59.1
58.5
77.1
71.9
78.1
76.8
IP2P
3.4K
0.5
0.7
0.9
64.0
54.8
50.6
50.3
74.5
55.3
60.3
58.0
NEUROSIM
5.1K
4.6
5.0
5.1
59.5
57.9
55.8
55.7
69.6
66.6
71.8
70.4
Table 14: Performance scores (Recall @ for NEUROSIM with TIM-GAN, GeNeVA and IP2P with increase in reasoning hops, for add, remove, and change instructions. Along with each method, number of data points from CIM-NLI used for training are written.
Input Image
Instruction
Program
Output
Remove the sheep
scene() fliter class(((shheep")) remove(]
Remove left chicken
scene() -> filteer (leeecssss((cccccken]]] -> relate(['leftt]] -> filter classs(trrsse'))) remove()
Add minecart to the front of tree
scene() filter fillte__classi/// (aree_ccass(((rree')) add/rel_concept-(fff concept=[ffrnttO") dlass ('minecart'])
Add a tree to the back cow
scene() >> filter_ class(((coo")))
addlrel_concept:[[an coceeettttepopcoo.......... class ('tree'))
Remove the cow to the left
scene() >> filter (iter_class("ttree]]
of tree
relate(['left') (ite__dass((ccoo]] remove[]
Figure 6: Results for addition and removal of objects from images of the minecraft dataset
vised training significantly improves the graph quality, thus improving retrieval perfor- mance. Supervised training improves retrieval by 7.3 R @@ points over weakly supervised NEU- ROSIM baseline. These findings suggest that NEU- RoSIM(e2e) significantly outperforms other super-
vised approaches in almost all settings. One fine-tune the image decoder and the visual repre- sentation network further enhance the findings, which should greatly enhance the outcomes.

---

of CIM-NLI examples used for training
Instruction
Model
5.4K
7K
10K
20K
54K
GeNeVA
0.0
0.7
TIM-GAN
1.9
4.9
8.6
10.3
13.1
Add
IP2P
0.7
5.4
NEUROSIM
4.9
6.4
5.7
5.9
5.6
NEUROSIM(e2e)
8.8
8.9
9.2
10.5
10.6
GeNeVA
5.9
4.1
TIM-GAN
41.0
42.9
49.8
62.5
78.3
Change
IP2P
54.9
61.5
NEUROSIM
57.2
57.3
57.2
57.2
57.1
NEUROSIM(e2e)
66.2
66.3
66.6
67.4
69.6
GeNeVA
13.2
8.7
TIM-GAN
49.6
47.0
53.9
65.3
78.0
Remove
IP2P
62.0
76.0
NEUROSIM
69.6
69.5
69.5
69.5
69.6
NEUROSIM((2e)
69.6
69.5
69.5
69.5
69.6
Table 15: Performance comparison of NEUROSIM((2e) with baselines using R NEUROSIM((2e) refers to NEUROSIM trained end-to-end by utilizing ground truth manipulated images the supervision for NEUROSIM modules.
Train Data Size
manipulation instruction text in natural language. generated the symbolic program by parsing the input text. To provide GPT-4 with context, we designed an extensive prompt that begins with our DSL followed by six different in-context examples representing various instruction types for few-shot learning. This prompt is then followed with the instruction text that want parse. We tested GPT-4 on randomly sampled subset of our dataset. For evaluation, we measured the accuracy of semantic parsing using an exact match between the generated symbolic program and the ground- truth symbolic program.
Model
R1
R3
TIM-GAN
5.4K
30.2
80.7
TIM-GAN
54K
66.3
92.4
NEUROSIM
5.4K
63.7
89.1
(EUROSIM(e2e)
5.4K
70.2
92.6
NEUROSIM(G2e)
54K
71.6
91.7
Table 16: Zero-shot generalization to larger scenes (Ex- tension of Table 3 of main paper).
The detailed results are given in Table 19. Inter- estingly, observed that GPT-4 performed poorly Add instructions, achieving less than 10% of parsing accuracy. To address this, we prompted GPT-4 separately with additional few-shot exam- ples for Add instructions, which led to the results displayed in the table. Even with the additional
F LLMs as few-shot parser
We also tested the semantic parsing ability of Large Language Models (LLMs), specifically GPT-4 for our task. The task of semantic parsing is given

---

Train
Hops
Method
Data
Size
2H
MII
TIM-GAN
5.4K
56.4
41.6
-4.8
TIM-GAN
54K
84.0
76.2
-7.8
NEUROSIM
5.4K
64.5
63.0
-1.5
NEUROSIM(E2e)
5.4K
69.4
67.3
-2.1
NEUROSIM(E2e)
54K
71.1
69.6
-1.5
Table 17: Performance with increasing reasoning hops (Extension of Table 3 of main paper).
G
Computational Resources
Model
R1
R3
We trained all our models and baselines on Nvidia Volta V100 GPU with 32GB memory and 512GB system RAM except IP2P which trained on 8-A100 80 GB GPUs. Our image decoder training takes about 4 days of training time. Training of the VQA task takes 7 days of training time and training the Manipulation networks take 5 hours of training time.
Text-Only
0.2
0.4
Image-Only
34.1 1
83.6
Concat
39.5
86.9
TIRG
34.8
84.6
NEUROSIM
85.8
92.9
NEUROSIM(62c) 93.1
96.7
H Hyperparameters and Validation Accuracies
Table 18: Quality of scene graph measured via retrieval (Extension of Table 4 of main paper)
H.1 Training for VQA Task
The hyperparameters for the VQA task are kept same as default values coming from the prior work (Mao et al., 2019). We refer the readers to (Mao et al., 2019) for more details. We obtained question answering accuracy of 99.3% after train- ing on the VQA task.
Instruction
Hops
Total
type
0
2
3
Add
NA 18.9
30.5 37.5
29.5
Remove
91.5 85.1
80.0 84.9
85.6
H.2 Training Semantic Parser
Change
70.7 81.7 76.2
70.2
74.6
The semantic parser is trained to parse instruc- tions. Learning of this module happens using the REINFORCE algorithm described in Section C of this appendix. During REINFORCE algorithm, we search for positive rewards from the set (7, 8, 10). and negative rewards from the set {0, 2, 3). We finally choose a positive reward of and negative reward 2. For making this decision, we first train the semantic parser for 20 epochs and then calcu- late its accuracy by running it the quantized scenes from the validation set. For a particular output program,
Total
81.5 55.6 55.9 58.1
60.5
Table 19: Few-shot parsing results of GPT-4
guidance, Add instructions remained significantly lower in accuracy compared to other instruction types. This analysis demonstrates that our rein- forcement learning-based instruction parser out- performs GPT-4, least on this dataset. It also highlights the need for more careful prompt engi- neering before LLMs like GPT-4 can be readily applied in our specific setting.
say it is correct if it leads an object being selected (see Section of the ap- pendix for more information) and this is how the accuracy the semantic parser is calculated. This

---

accuracy is proxy for the real accuracy. An alter- native is to use annotated ground truth programs for calculating accuracy and then selecting hyper- parameters. However, we do not use ground truth programs. All other hyperparameters are kept the same as used by (Mao et al., 2019) to train the parser on VQA task. We obtain a validation accu- racy of 95.64% after training the semantic parser for manipulation instructions.
change and remove instructions respectively. NEU- ROSIM's ' advantage lies in semantic correctness of manipulated images. For example, see Figure row #3,4; Figure 8 row #2; 9 all images. these im- ages, NEUROSIM was able to achieve semantically correct changes, while TIM-GAN, GeNeVA faced problems like blurry. smudged objects while adding them to the scene, removing incorrect objects from the scene, not changing/partially changing the object to be changed. Images generated by TIM- GAN are better in quality as compared to NEU- ROSIM. We believe the reason for this is that TIM- GAN, being fully supervised, only changes a small portion of the image and has learned to copy a sig- nificant portion of the input image directly to the output. However, this doesn't 't ensure the semantic correctness of TIM-GAN's manipulation, scribed above with examples where makes errors. The images generated by NEUROSIM look slightly worse since the entire image is generated from ob- ject based embeddings in the scene graph. Improv- ing neural image rendering from scene graphs can be a promising step to improve NEUROSIM.
H.3 Training Manipulation Networks
The architecture details of the manipulation net- work are present in Section of this appendix. We use batch size of 32, learning rate of 10 3, and optimize using A (Loshchilov and Hutter, 2019) with weight decay of 10 Rest of the hy- perparameters are kept the same as used in (Mao et al., 2019). During training, at every 5th epochs, we calculate the manipulation accuracy by using the query networks that were trained while training the NEUROSIM on VQA data. This serves as proxy to the validation accuracy.
de-
For the change network training, we use the query accuracy of whether the attribute that was supposed to change for particular object, has changed correctly not. Also, whether any other attribute has changed or not.
Errors
Figure 10 captures the images generated by our model where it has made errors. The kind of errors that NEUROSIM makes can be broadly classified into three categories.
For the add network training, we use the query accuracy of whether the attributes of the added object correct or not. Also, whether the added object is in a correct relation with reference ob- ject or not.
[Rendering Errors] This set includes images generated by our model which are semantically correct but suffer from rendering errors. The common rendering errors include malformed cubes, partial cubes, change position of ob- jects, and different lighting.
We obtained a validation accuracy (based on query- ing) of 95.9% for the add network and an accuracy of 99.19.... 19.1% for the change network.
H.4 Image Decoder Training
[Logical Errors] This set includes images gen- erated by our model which have logical errors. That is, manipulation instruction has been inter- preted incorrectly and a different manipulation has been performed. This happens mainly due an incorrect parse of the input instruction into the program, or manipulation network not trained to perfection. For example, change network chang- ing attributes which were supposed to remain unchanged.
The architecture of the image decoder is similar (Johnson al., 2018) but our input scene graph (having embeddings for nodes and edges) is di- rectly processed by the graph neural network. We use a batch size of 16, learning rate of 10-5, and optimize using Adam (Kingma and Ba, 2015) opti- mizer. The rest of the hyperparameters are same as (Johnson et al., 2018). We train the image decoder for fixed set of 1000 iterations.
[VQA Errors] The query networks are not ideal and have errors after they are trained on the VQA task. This in turn causes errors in supervision (ob- tained from query networks) while training the
I Qualitative Analysis
Figures 7, 8, 9 compare the images generated by NEUROSIM, TIM-GAN, and Ge on add,

---

Input Image
Instruction
Ground Truth
NEUROSIM
TIM-GAN
GeNeVA
Add rubber that is front of the big cube.
There a shiny thing that on the side side of shiny block, gray
Add large gray metallic cylinder object tiny green matte cylinder.
frant
purple shiny objject front the purple metal ball, large
the
There is rubber thing behind matte thing the tiny obyect, tiny shiry spterne behind
Figure 7: Visual comparison of NEUROSIM with TIM-GAN and Ge for the add operator. The red bounding boxes in the ground truth output image indicate the objects required to add to the input image.
Input Image
Instruction
NEUROSIM
TIM-GAN
GeNeVA
There a small matte thing: the purple.
There thing the red matte ball; change the shape it to cylinder.
cylinder the small metallic cylinder; change it to
the left of the small blue thing to the left of the big green metallic cylinder; change the material of
objoct front the small nubber thing that left of the matte sphere that yellow
shiny,
Figure 8: Visual comparison of NEUROSIM with TIM-GAN and GeNeVA for the change operator. The red bounding boxes in the input and ground truth output image indicate objects required changed.
manipulation networks and leads to a less than optimally trained manipulation network. Also, during inference, object embeddings may not be perfect due the imperfections in the visual rep- resentation network and that leads to incorrect rendering.
K
Ablations
Table 20 shows the performance of NEUROSIM when certain loss terms removed while learning of the networks. This depicts the importance of loss terms that we have considered. particular we test the performance the network by removing

---

Input Image
Instruction
Ground Truth
NEUROSIM
TIM-GAN
GeNeVA
There a shiny cube, remove it.
There metallic thing front gray thing, remove
Remove the large shiny object behind purple
bchind
thing. removellt it.
There is large metal object left metallic object that right large thing the spiwee, remove
Figure 9: Visual comparison of NEUROSIM with TIM-GAN and GeNeVA for the operator. bounding boxes in the input image indicate objects required to be removed.
The red
Input Image
Instruction
Ground Truth
NeuroSIM
Error Description
There is object on the right side red metallic thing that behind the red block; change the shape of ball.
Malformed cube
Change the shape of the black that on the right side of the large large blue ball ball.
Logical Incorrect object changed
There is red rubber object is left the green shiny cylinder, add purple shiny sphere in front it.
attributes, material matte instead of shiny
Remove the shiny sphere that is the right side of the thing that is on the left side of the tiny object in front the yellow object.
Logical error, incorrect object removed
Figure 10: Types of errors in NEUROSIM.
edge adversarial loss used by add network (row 2), object adversarial losses for both add and change networks (row 3, 5), self supervision losses used by add network (row 4), cyclic (row 6) and consistency (row 7) losses used by change network.
Interpretability of NEUROSIM
NEUROSIM allows for interpretable image manip- ulation through programs which generated as intermediate representation of the input instruction. This is one of the major strengths of NEUROSIM. since it allows humans detect where NEUROSTM failed. This not possible with purely neural mod- els, that behave as a black box. Knowing about the

---

templates one for each edit operation.
Loss
R1
R3
1. Change the [attribute] of [size] [color] [mate-
rial] [shape] to [attribute"]
8
45.3
65.5
2. Remove the [size] [color] [material] [shape]
padd
"edgeGAN
43.7
66.0
3. Add a [size] [color] [material] [shape] to the [relation] of [shape']. Next, given multi-hop instruction we parse it using our semantic parser which gives us the object's embedding on which either an operation is be executed (in case of change and remove operations) or new object has to be inserted in relation it (in case of add op- eration). The trained netwoss predicts the symbolic values the concepts the placeholders. Example, if the MH instruction is "Change the size of the big thing that is behind the metallic cylin- der behind the purple object that is to the right of the big brown shiny object tiny". find the placcholder attributes to be operation=change. tributc=size, color=ycllow, shape=cube, size=large, material=rubber, attribute' =tiny. Hence the sim- plified instruction becomes, "Change the size of the large yellow rubber cube to tiny". Add and Remove instructions follow similarly.
8
padd
44.3
60.2
"objGAN
padd
gadd odgeSup
44.1
57.9
"objSup
8
pehange
44.9
61.5
objGAAN
8
gehange
36.5
51.1
cyclee
gehange
31.0
44.8
"consisteccy
Table 20: Ablations conducted by removing some loss terms. is the total loss before any ablation. For each loss term being removed, the superscript denotes which network it belongs to (add or change). Ablations are conducted the setting where 3 0.054 (see main paper Section 4 for the definition of
failure cases of NEUROSIM also means that it can be selectively trained to improve certain parts of the network (for individually training on change instructions to improve the change command, if the model is performing poorly on change instruc- tions). We now assess the correctness of intermedi- ate programs using randomly selected qualitative examples present in Figure 11. Since no wrong program was obtained in the randomly selected set, we find 2 more data points manually, to show some wrong examples.
M Human Evaluation Details
See Table 21 for the questions (paraphrased) asked to the evaluators. Detailed instructions and an ex- ample of the questions provided the evaluators can be found in Figure 12. total of 10 evalu- ators, consisting of mix of undergraduate and post-graduate students, were involved in the study, The same set of 30 random images were given to each evaluator. They were compensated at rate three times the average hourly salary in the country of origin. Each evaluator was given upto 24 hours to complete the task.
Simplifying Multi-Hop Instructions using NeuroSTM Modules
In this section, we provide details on our method of utilizing the trained semantic parser to convert the complex multi-hop instruction into a simplified or hop instruction. We generate three simplified

---

Input Image
Instruction
Generated Program
Correctness
Add tiny purple metal ball front of the blue object that is behind the
scene() filter[[rubber, 'sphere]]] reline(( fitte[[aauue)))
V
add(rel, concept=['front]] concept
sei=['smalff settssp_
'purple', 'metall, 'sphere'])
brown object that is left of the blue matte block that left the side of large oyan metal cube.
scenel) filterl["arree', 'oyan', 'metal; 'cube']) relstel[\igght)) [itter[['rown'')) relate([]lfft]) > filterl[bluee, 'rubber', 'cube") relftel["leftt]]) filter[[brown', 'metal")) remove()
Change the shape of the big gray thing cube.
scene() fitterl[large', 'gray'l) changejattr=shape, cancept-[ccube]]
Remove the gray rubber thing front the gray matte sphere behind the large gray sphere.
scene() fitter[["arge', 'gray', 'rubber', 'sphere']) relate([ })>> filter('eray'' 'rubber', 'sphere reelete relste[["roon" filter[[igryy' 'rubber')) >> remove()
There matte block that front the gray rubber object; change the
scene() filtor[[largee, 'gray,, 'rubber]] ]] relate(("fonn"" filten[rrubbrr, 'cube"]] change(att--materia, concept-['metal]]
material shiny.
Remove the big ball that is large
scenel) pilterl"puuppee', 'metal'1) relotell"rront) filter([langee]]
right thing in front of the purple metallic object
relate([rightt)) fllten[(large,, 'sphere')) removel)
Change the material of the small block frent of the brown metal block to rubber.
scene() filterl['brown', 'metal", 'cube')) -> (elaat((ffrot"]]] [) filter([smalll,, 'metal", 'cube']) changelatt--material,, contapt-|ruubber]]
Remove the cylinder that is right of cylinder that is in front of
scene() fiterl['small] 'red', 'rubher)] -> relate([ fitter[[rubber, 'sylinder'[) relate([\right") (iterl('yylinder)) removel)
matto thing.
scene() >s filterl['salll,, 'gray']) melatel/bbehind]]] -s fittor[['matal,, 'cube"]) [) cube behind the tiny add(ral__oneae concept-["behing concept caaiipptssttllapppee set-l'large', 'purple', 'metal", 'sphero"]) objact.
Add large purple shiny sphere
shiny
Change the shape of the large object in front of the yallow ball.
scene() filter]'smalll, 'yellow', 'sphere"]) relate[[ (rontt]] filter([largee))
change(att-ssapp, concept-(sphere')
There blue thing behind the big blue block left of the thing
sonnel) > filter['bluee, 'rubber', 'cube']) o relate(("front)) filter["large', 'blue', 'cubn']]
relatel["lef()) -s filter["bluee]]] reiite""n"s change(ettrmcolor concept=['brownn
Corrected program:
blue matte block; change Remove brown object on the left side the object right of the brown object behind big block.
scene() filter(['luee blue', 'rubber', 'cube')) is relate((frontt)) >> relate(["left") -s filterl[largee',
'blue', "uube)) relate((behindd]] filter[[bluee]] change(sttr-coloff concept-['brown
scenel) filter[[ 'brown', 'cube']) [ relate([behind']]) -> filter(/brown"]) relate((righht)) filter("brown"]) relate(["lfft]) removel)
program:
scenel) o filter([arge),, 'brown', 'cube')) ("behind")]]) filter(/rrown"]]
relate([rightt)) relate[[[eett]) fitter('brown') > remove()
Figure 11: Qualitative examples of generated programs by NEUROSIM.

---

[Change] Are all the attributes (color, shape, size, material, and relative position) of the changed object mentioned in the instructions identical between the ground truth image and the system-generated image?
[Add] Are all the attributes (color, shape, size, material, and relative position) of the added object mentioned in the instructions identical between the ground truth image and the system-generated image?
Question 1:
[Remove] Are same objects removed in ground truth image and the system-generated image?
[Change] Are all the attributes (color, shape, size, material, and relative position) of the remaining objects identical between the ground truth image and the system- -gnerated image?
[Add] Are all the attributes (color, shape, size, material, and relative position) of the remaining objects identical between the ground truth image and the -enerrtee image?
Question 2:
[Remove] Are all the attributes (color, shape, size, material, and relative position) of the remaining objects identical between the ground truth image and the system -eeseateeeeetteed image?
Table 21: Questions asked to human evaluators for evaluating NEUROSIM and TIM-GAN, Note that there are some variations in the questions for Change, Add, and Remove instructions dues different semantic nature of the instructions.

---

Human Evaluation
There are 30 questions. In each question, there are images, an "Input" image, 'Reference Output" image (or ground image), randomly "Candidate Output" images generated by machine learning models.
object image properties:
Shape: cylinder, sphere, or cube
Size: small big
shiny "metallic" "rubber"
Color: green, red, purple, gray, blue, brown, yellow
Between each input and truth properties (or whether exists in the input or ground truth images or not). We call this the "changed object'. understand
different of
object" please following examples:
There are 3 types of changes:
Change property of one of the objects
objects
Addition of one object
Example
Here the "large gray cube" in the Input Image the "changed object", since it's shape has changed to "cylinder' in the Reference Output Image
Input
Reference output
Candidate
Candidate
Candidate

---

Example 2
Here "small pink sphere in the Reference output image the "changed object' since doesn't exist in the Input image and has been added in the Reference Output Image.
Input
Reference ourput
Candidate
Cardidate
Candidate
Example 3
"large green cylinder" in the Reference output image the "changed object", since exist in Input image but Reference Output Image.
Input
Fieloencc sutputt
Canddire
Cavdldae
Candioss

---

Question
Inp
Nefeerce rulpul
For the changed object in the Reference Output image, all the attributes (color shape, size, material, and relative position) same Candidate Output 1?
No
the remaining objects Reference image, attributes size, material, and relative position) same Candidate Output 1?
Yes
changed Reference Output image, attributes (color, shape. size, material, and relative position) same Candidate Output 2?
Yes
remaining objects image, attributes (color, shape, size, material, and relative position) same Candidate Output 27
Yes
For the changed Reference image, all attributes (color, shape, size, material, and relative position) same Candidate Output 3?
Yes
No
For the remaining objects the Reference Output image, are the attributes (color, shape, size, material, and relative position) same Candidate Output 3?
Yes
No
Figure 12: Screenshots for human evaluation study. See Section 4.7 for more details


Reviews:
[{"review_id": "BMhqsNrerM", "reviewer": "Reviewer_La5y", "summary": "", "questions": "A: Can the authors give more details about why this shallow RL reward leads to good results? Is due to the VQA pre-training prevent degenerate solutions or something else?\n\nB: IP2P has better R scores, but the FID is very low. What is the reason for this? How should we understand the trade-off between these systems are offering?\n\nC: Given the success of works like VisProg and ViperGPT, I am curious if you could integrating some of these image editing components with LLM to make a system that can follow more natural/general queries. Have the authors considered it?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for an in-depth review, for recognizing our contributions, and providing comments to further improve our paper.\n\nPlease see our response below:\n\n>#### **1. There is little intrinsic motivation to manipulate these kind of synthetic images, meanwhile it seems clear there are many hurdles to applying this system to natural images. Additionally the image editing operations themselves are relatively basic, most of the complexity/composition in the task comes from identifying what object the instructions is referring to.**\n\nWhile we agree that there are challenges in applying the technique to real-world datasets, foremost being (1) availability of a real-world dataset with complex multi-hop instructions (2) ability to deal with a large (potentially open) sets of concepts (3) availability of neuro-symbolic systems with high accuracy on these datasets, we would like to stress that there indeed a great deal of motivation in experimenting with a synthetic dataset, as a proof of concept, since none of the earlier works in the literature have tried these kind of manipulations as in our paper (refer Table 1 of related works). Further, we would argue that multiple works in neuro-symbolic image based reasoning and manipulation (or otherwise) have first demonstrated their capability on synthetic datasets such as CLEVR and Minecraft[1], and then made progress towards real datasets [2]. In that regard, our work should be seen as the first step towards building such a neuro-symbolic interpretable model for image manipulation, in a weakly supervised setting. We would also like to note that, while moving to the real-world datasets, the basic conceptual architecture (as depicted in Fig. 3) remains the same, and the difference lies in training of the concept and manipulation networks, which requires the availability of real-world manipulation dataset, along with a VQA system of high accuracy. As the field and these systems advance, this will become a realistic possibility, paving way for a wider applicability for our approach.\n\nRegarding Reasoning: It is true that much of complexity of the task is hidden in identifying the referential object, but we believe that is fair challenge to tackle, and that is where the strength of these neuro-symbolic models lie, which is evident from the relative drop in performance of the IP2P (5.4K) based models vs our approach when going from ZH to MH (20.3 points vs 1.5 points in Table 3).\n\nOverall, we believe that this is a challenging task, which merits attention in its own right.\n\nReferences:\n\n1. Yi et. al, Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. NeurIPS 2018\n\n2. Li et. al, Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images\n\n\n>#### **2. The system is very complex but still has mixed results compared to the end-to-end baselines.**\n\nWe note that reasoning and editing over images is a complex task. This means that a complex architecture is required to perform well on this task. While for purely neural models, the complexity is hidden in the design of the neural architecture (e.g. diffusion models), our complexity is expressed in terms of having multiple modules, which perform different tasks (e.g. parsing, language grounding etc), and integrating them together. As a result, we get interpretability, and the ability to do away with completely supervised data, as opposed to purely neural models, which require supervised training.\n\nRegarding results: while our model may seem comparable to existing systems in terms of its performance, we note that our performance is achieved using only VQA data, whereas all other existing approaches require supervised data, resulting in high human cost of annotation.\n\nIn low data settings, we are able to outperform existing approaches including some of the latest diffusion based models (see Table 2 of the paper).\n\n>#### **3. Can the authors give more details about why this shallow RL reward leads to good results?**\n\nWe understand your concern. We present a detailed explanation below for how the model can be trained using a simple reward based mechanism. We will be happy to incorporate this in the final version (either in the main paper, or in the supplement).\n\n\nRecall that the parser module is trained using an off-policy program search algorithm that searches over program templates. Parser fills these templates with concepts and attributes that are present in the input instruction. This creates a feasible set of programs for the parser to pick from. A +ve reward is assigned to a program as long as the execution of that program results in selecting an object.\nNow, coming to the question of how the parser differentiates between add/remove/change instructions. Note that the signature of the change, add, and remove commands are different in terms of the types of their respective inputs (for e.g., the remove command just takes an object to be removed as the input, whereas add command takes a concept_set of the object to be added as the input). Due to these differences in the input signature, there is no mistake made by the parser while differentiating among add/remove/change templates. For example, if the input is an add command, the parser will first try extracting a concept_set from the instruction, then filling in the corresponding templates. No template from remove or change will get chosen here because none of their signatures require concept_set as an input. Similar is the case for the change command, and the add/remove command.\nNow, we come to the question about the parser selecting a program that selects an incorrect object for the manipulation. As you rightly understood, it is very much possible that there are multiple different programs such that each of them results in selecting the manipulation object (probably an incorrect one) upon its execution and thereby receiving a positive reward despite selecting an incorrect object. However, it is our hypothesis that such an event does not occur very often. Let's consider the following example to convince you and understand the real dynamics underneath.\n\nConsider the following instruction: “change the color of the red sphere to blue.” The correct program for this instruction would be as follows:\n\n**change(attr=color, concept=blue, obj = Unique(Filter(sphere, objs = Filter(red, scene())))).**\n\nAn alternative program with the same template could be:\n\n**change(attr=color, concept=red, obj = Unique(Filter(sphere, objs = Filter(blue, scene()))))**\n\nNote, for this second program to succeed in terms of selecting an object, a blue sphere needs to be present in the scene. The likelihood of the concepts “blue” and “sphere” being present in the instruction, and at the same time, a blue sphere being present in the same image should be relatively low because scenes will be very diverse.\n\nTo further support the above line of arguments, we calculated how often an incorrect program leads to selecting an object while training the parser using templates. We find that 40.30% of the training data is such that only one program can be compiled from a given instruction and a template, which leads to selecting the “correct” object. For the rest of the training data, more than one program can be compiled using the template, such that each program leads to selecting an object and at least one of these programs is correct. These multiple programs can either all be correct, or some of them may be wrong. Based on this, our conclusion is that at least  40.30% of examples are helping the parser to get bootstrapped such that the learning takes place correctly. These examples force the parser to be “correct.” Hence, it can avoid the learning route that would have generated the wrong programs just to “select any object.”\n\nIn our experiments, the parser reaches about 94% in accuracy, which confirms that learning is effective using this weak supervision. \nPlease also see our response to your last question below that shows accuracy of LLMs on the program parsing task.\n\n\n>#### **4. IP2P has better R scores, but the FID is very low. What is the reason for this? How should we understand the trade-off between these systems are offering?**\n\na] IP2P has lower R1 scores in the low data regime setting as compared to NeuroSIM, and a similar R1 score compared to NeuroSIM on the full data setting, which shows that NeuroSIM is able to follow the image editing instruction better with less training data, and this is reflected in the recall scores. \n\nb] IP2P is trained with a very large pre-training dataset (stable diffusion training data = 2.3 Billion samples + additional 450k samples) hence IP2P is able to match distributions very well after fine-tuning. FID measures the similarity between the generated and ground-truth images, lower the number means a higher distributional similarity.\n\nc] Tradeoff between systems: IP2P is good at editing images with open ended instructions, while NeuroSIM offers interpretability, sample efficient learning and largely reduced annotation and data collection cost with better logic/reasoning capabilities due to its neuro-symbolic nature which improves recall scores. NeuroSIM suffers on the FID score, likely because of the image generation module, but improves faithfulness on editing (measured by the R1 score), especially in the low data regime. Here lies the trade-off.\n\n\n>#### **5. Given the success of works like VisProg and ViperGPT, I am curious if you could integrating some of these image editing components with LLM to make a system that can follow more natural/general queries. Have the authors considered it?**\n\nThis is a very interesting suggestion. Using large pretrained LLMs can definitely be used for generating programs as done in VisProg and ViperGPT. For the instructions in our dataset, we did not need to do this since the RL approach gives ~94% accuracy, and does not need any pre-training on language tasks. On the other hand, for extending this approach to tasks with more complex natural language instructions, we can use in-context learning in LLMs . In order to allow for better grounding of text in the images, one can also explore integrating with approaches which work with image-conditioned LLMs, such as in BLIP/InstructBLIP [1,2]. We believe this integration is an important suggestion for future work aimed at improving interpretability, sample efficiency and reducing annotation cost  (as offered by our method) of instruction based image editing models while maintaining the open-ended capabilities of SOTA diffusion models, and we will be happy to add a discussion about these approaches in the final version.\n\nIn order to evaluate the efficacy of the pipeline as proposed above, as an experiment we test the performance for LLMs, specifically GPT-4 on semantic parsing for our task. We provide the model with an extensive prompt specifying the DSL and 6 examples (of different instruction types) for few-shot learning. We tested the GPT-4 on a randomly sampled subset of our test dataset and obtained the following scores.\nThe Accuracies are reported as below:\n\n| Instruction type |     Zero Hop | One Hop   | Two Hop  | Three Hop | Total             |\n| ---------------- | ------------ | --------- | -------- | --------- | ----------------- |\n| Add\\*            |       NA     |   18.9%   |   30.53% |   37.56%  | 29.54%            |\n| Remove           |       91.58% |   85.18%  |   80%    |   84.94%  | 85.60%            |\n| Change           |       70.7%  |   81.72%  |   76.28% |   70.29%  | 74.61%            |\n| Total            |       81.55% |   55.64 % |   55.98% |   58.13%  | 60.58% (overall) |\n\n\n\\*We find that using the prompt with multiple different kinds of instructions, GPT performs very poorly on Add instructions (<10% accuracy). We repeated the experiment for Add instructions with more few-shot examples, this leads to the numbers reported in the table, which are still significantly lower than the other instruction types.\n\nFor e.g. \nPrompt : Add a tiny yellow matte sphere that is to the left of the big brown rubber object.\n\nGround_Truth_Program :  ['scene()', 'filter(large, brown, rubber)', 'relate(left)', 'add(small, yellow, matte, sphere)']\n\nGPT-4: ['scene()', 'filter(large, brown, rubber)', 'relate(left)', 'filter(small, yellow, matte, sphere)']\n\nAs we can see that instead of putting the add operation in the last, it puts the filter operation and it repeats the same mistake for most of the add instructions. While such mistakes are not frequent in the **Remove** and **Change** operation. Different prompt tuning techniques can be explored to improve accuracy, especially for the **Add** operation\n\nThis shows that our RL-based parsing of the instruction performs better than GPT-4 at least on this dataset, and that some more careful prompt engineering may be needed before LLMs can be made to work off-the-shelf in our setting.\n\n**We thank you for appreciating our work. Please let us know if you have any further questions, and we will be happy to address those during the discussion phase.**\n\nReferences:\n\n1. Li et. al, BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. ICML 2022\n\n2. Dai et. al, InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. Arxiv 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Presents a unique approach to image editing by integrating Neuro-Symbolic approaches, it seems like a pretty novel method to me.\n- I found the idea of using RL to do image editing intriguing, especially since its very difficult to get image editing training data for complex instructions\n- Very detailed evaluation that contains lots of experiments, analysis and details about the system, including a human evaluation and studies on many different settings.\n- Positive results over the baselines at least for low data regimens\n- Releases a dataset that can be used in future work", "reasons_to_reject": "- There is little intrinsic motivation to manipulate these kind of synthetic images, meanwhile it seems clear there are many hurdles to applying this system to natural images. Additionally the image editing operations themselves are relatively basic, most of the complexity/composition in the task comes from identifying what object the instructions is referring.\n- The system if very complex but still has mixed results compared to the end-to-end baselines, although the authors have pointed out their system only requires VQA data to train.\n- I am struggling a bit to understand why the REINFORCE algorithm works. The reward seems to be very shallow, so I am failling to see how it could train the model to select the correct manipulation operator, or to prevent the model from simply selecting a random object in the scene. I think more discussion here is needed. The author rebuttal had a very helpful explanation of this, I would strongly encourage the authors to include it somewhere in the paper.\n\nDespite these concerns overall I think this is an interesting and novel system and should be accepted.", "questions_to_authors": "A: Can the authors give more details about why this shallow RL reward leads to good results? Is due to the VQA pre-training prevent degenerate solutions or something else?\n\nB: IP2P has better R scores, but the FID is very low. What is the reason for this? How should we understand the trade-off between these systems are offering?\n\nC: Given the success of works like VisProg and ViperGPT, I am curious if you could integrating some of these image editing components with LLM to make a system that can follow more natural/general queries. Have the authors considered it?", "paper_topic_and_main_contributions": "This paper presents a model that can do follow complex, compositional image-editing instructions for images from CLEVR. Their system combines Neuro-Symbolic approaches to CLEVR VQA to identify objects that should edited with editing moduales that can add, remove, or change objects in the scene graph and a rendering model that can draw the modified scene back as an output image. They additionally build a dataset of image editing instruction to evaluate their work on.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kSoFPqFR96", "reviewer": "Reviewer_4Vv2", "summary": "", "questions": "Also see previous section. \n\nA: In L550, it says IP2P zero-shot manner, but the table 2 shows trained on 10% and 100% data. Is this referring to some other table?\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for an in-depth review, for recognizing our dataset and modeling contributions, and providing comments to further improve our paper.\n\nPlease see our response below:\n\n>#### **1. My main concern is that there is limited application for CLEVER like scenes … As such, a model trained on in this dataset is likely to have limited real world use case**\n\nWe created the extension of CLEVER for the image manipulation task (and we will release it for future research), since we wanted to compare various image manipulation models in a controlled setting, especially with multi-hop instructions, and we are not aware of other image manipulation datasets with such complex instructions. In addition to developing a novel weakly supervised approach for image manipulation, one of the important take-aways from our experiments in CLEVER like setting is that, even in this synthetic/controlled setting, SOTA diffusion models like IP2P which is trained on a large dataset (2.3B training examples) and can supposedly handle a large variety of scenes and language instructions as input, performs poorly compared to our weakly supervised approach trained from scratch (especially in the low data setting), pointing to some of its inherent limitations. These findings raise some important questions:\n\na. Do neuro-symbolic models have fundamental advantages for complex reasoning based manipulations, compared to other baselines, including SOTA diffusion models?\n\nb. Do these findings carry over to more real world datasets?\n\nc. Diffusion based models (or even other supervised approaches) do somewhat better in terms of overall image quality (FID scores). Can one explore the possibility of developing a hybrid model, combining the reasoning capabilities of neuro-symbolic approach, along with generation capabilities of purely neural models?\n\nThese points clearly highlight the value of our artificially created dataset, in finding answers to some of these questions and formulating some of these interesting questions for future research. Further, we hope to carry forward these experiments to more realistic real world datasets, with availability of such manipulation datasets, and improved availability of high performance VQA systems of real world datasets (which is an essential component of our approach). Finally, we would like to highlight that research in neuro-symbolic VQA started with results on datasets like CLEVR and Minecraft [1] while later matured into real world questions answering systems [2] and we can expect the same for our methods.\n\nReferences:\n1. Yi et. al, Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. NeurIPS 2018\n2. Li et. al, Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images\n\n\n\n>#### **2. The authors introduce multi-hop reasoning, but these are very synthetic (App B.3). In real world use case, the instructions are very likely to be only zero-hop since the user knows what they want to edit. That is the dataset is artificially made tougher.**\n\nWe would like to respectfully disagree. There may not be real world datasets currently, however, \nproblems of multi-hop reasoning, and one can easily think of examples for co-reference resolution that exist in the visual domain on real datasets as well.\n\nFor example, consider the following instruction:\n“Change the color of the red fruit kept on top of the table which has a lamp on it and is behind the bed.”\n\nThere might be various tables in the scene as well as fruits in the scene, and hence multi hop reasoning might very well be required. It is also often easy for people to give these multi hop instructions, instead of analyzing the scene very closely and provide a precise single hop instruction (and often, there may not be a zero-hop instruction which identifies the reference object uniquely, i.e., only in terms of its attributes).\n\nWe believe one of the reasons is the lack of such datasets that has left this problem not getting sufficient attention from the community. It might also be the inherent hardness of this problem, in terms of identifying and resolving such object references, before manipulation can be done. For instance, it is quite well known that diffusion models perform quite poorly in terms of their spatial reasoning [1] and it has been an active area of research that some papers have now started to look at [2].\n\nAnd we argue that this is where the contribution of this paper lies. Even though we create a simple CLEVR based dataset, it is non trivial to solve it and it clearly demonstrates the relative merits of various approaches in performing the manipulation task. We argue that this paves way for creation of more such benchmark datasets in future which are more realistic, and also potentially developing hybrid solution approaches as described above.\n\nReferences\n\n1. Brooks et. al, InstructPix2Pix Learning to Follow Image Editing Instructions, CVPR 2023\n\n2. Bar-tal et. al, MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation. Arxiv 2023\n\n>#### **3. I feel the authors missed a simpler (but potentially stronger) baseline which uses a combination of neuro-symbolic reasoning and IP2P.**\n\nWe are not completely sure if we understood your suggestion. We note in our case, instructions are multi-hop in terms of reasoning, and following which there is a simple (single) manipulation step, once the reference object has been identified. \n\nTherefore, based on our closest interpretation of your comment, which we found very interesting, we can carry out the following experiment. We utilize the neuro-symbolic component of our pipeline to convert the complex multi-hop instructions into simpler ones (for eg. zero or one hop) and use these converted instructions to guide IP2P for editing once the reference object has been identified. We hypothesize that in this hybrid setting, there would be less performance drop in IP2P when comparing the ZeroHop vs MultiHop results in Table3, while maintaining the high generation quality (FID score) of IP2P. Currently we’re running this experiment and would present the results during the discussion phase.\n\n>#### **4. How does the model handle lighting conditions? In the qualitative example (fig 4, top row), it appears Neuro-SIM is failing to replicate the conditions and thus the generated image is inconsistent.**\n\nWe believe NeuroSIM’s image generation model can in principle reconstruct the lighting conditions if trained to minimize our losses, given that object representations have global image representations as well. However, it is possible that some errors exist in neural modules used in NeuroSIM, as well as baselines, and these lighting errors should be captured as part of the worsening of the FID scores of the models. In our set up, these errors can be reduced in many ways. For example, if we perform end2end training with pixel loss from the decoder (Appendix Section E), these errors should be reduced. We can also add additional cyclic losses in the pixel space in the weakly supervised setting which would include manipulating the scene graph, and bringing it back to the original state, and taking a loss between the images generated from the initial and final scene graph. We would be happy to experiment with such additional losses and report results in the final version, if so desired.\n\nWe looked at figure 4 and did not find any specific artifact but we are happy to receive more detailed comments on figure 4 or any other figure, and we will be happy to look into these.\n\n>#### **5. In Table 3., how does Neuro-SIM with 54k examples perform? Is the performance saturated?**\n\nScores for NeuroSIM 54K are shown in the following table:\n\n|               | Larger Scenes |      | Hops |      |\n| ------------- | ------------- | ---- | ---- | ---- |\n| Method        | R1            | R3   | ZH   | MH   |\n| NeuroSIM-5.4k | 63.7          | 89.1 | 64.5 | 63   |\n| NeuroSIM-54k  | 64.2          | 89.7 | 65.8 | 64.1 |\n\nWe see a consistent increase over NeuroSIM 5.4K, however, you’re right that performance is nearly saturated, just like we shown in Table 2 as well, showing that NeuroSIM is sample efficient and needs only a few examples to saturate to its maximum performance. \n\n\n >#### **6. In L550, it says IP2P zero-shot manner, but the table 2 shows trained on 10% and 100% data. Is this referring to some other table?**\n\nWe refer the reviewer to L551 where we’ve mentioned the results of IP2P in Zero-Shot settings. These numbers were not put in Table2 as the table shows the performance numbers of the models that were either trained or fine tuned(case of IP2P) with varying data sizes. We will clarify this in the final version of the paper.\n\n**Please let us know if you have any other questions or concerns and we would be happy to answer them and engage in discussions.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Pros:\n\n1. Neuro-symbolic methods have the advantage of being completely explainable and make very little errors. Given the rise of image generation methods, having a neuro-symbolic editor on top of it is an interesting setup, and thus the task is well motivated. \n\n2. The paper also makes dataset contribution which can be useful for future work.\n\n3. The proposed neuro-symbolic method is competitive with strong baseline like Instruct-Pix2Pix (IP2P) even though it is trained on significantly less data.", "reasons_to_reject": "Cons:\n\n1. My main concern is that there is limited application for CLEVER like scenes which are constrained to add/remove/change. Further, ``change'' attributes are also restricted. Compare this to dataset used in IP2P where there are no constraints and is free form. As such, a model trained on in this dataset is likely to have limited real world use case.\n\n2. The authors introduce multi-hop reasoning, but these are very synthetic (App B.3). In real world use case, the instructions are very likely to be only zero-hop since the user knows what they want to edit. That is the dataset is artificially made tougher.\n\n3. I feel the authors missed a simpler (but potentially stronger) baseline which uses a combination of neuro-symbolic reasoning and IP2P. For instance, a neuro-symbolic reasoner (language only, no vision component) could create single-step (zero-hop) instructions which could then be executed by IP2P in a sequential manner. This would also remove the issues of worse FID scores. \n\n4. (Minor) The authors remark that other datasets namely CSS, CoDraw and i-CLEVER exist. Some comparison on those dataset might be helpful for future work (I don't expect the conclusions to change though).\n\n5. How does the model handle lighting conditions? In the qualitative example (fig 4, top row), it appears Neuro-SIM is failing to replicate the conditions and thus the generated image is inconsistent. \n\n6. In Table 3., how does Neuro-SIM with 54k examples perform? Is the performance saturated?", "questions_to_authors": "Also see previous section. \n\nA: In L550, it says IP2P zero-shot manner, but the table 2 shows trained on 10% and 100% data. Is this referring to some other table?\n", "paper_topic_and_main_contributions": "Brief Summary: The paper tackles the task of natural language based image manipulation where given an image and a edit (in natural language), the aim is to create an edited image. The authors propose neuro-symbolic framework for this task and apply it on CLEVER-like images (synthetic images). The key idea is to have additional manipulation branch on previous work (NSCL), to handle the image manipulation task using object gan losses. \n\nThe authors create their own dataset CIM-NLI for this task (synthetically). Compared to baselines, the proposed method outperforms existing competitive baselines when available data is limited and shows a smaller drop going from zero-hop to multi-hop reasoning.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DVom2O5TkK", "reviewer": "Reviewer_Q5BV", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for an in depth review of our paper, for recognizing our dataset creation and modeling efforts as well as the benefits and competitiveness of our model in terms of weak supervision and interpretability.\n\nPlease see our response below:\n\n>#### **1. Limited Scope: The paper focuses on image manipulation tasks involving multiple objects and complex multi-hop natural language instructions. While this is an interesting and challenging problem, the scope of the paper might be limited for researchers working on other aspects of NLP or image manipulation**\n\nWe respectfully disagree on having limited scope, rather we really believe our work opens doors for newer techniques and further pushes the frontiers in image generation and editing, especially in terms of the kind of instructions that one can deal with (i.e., complex multi-hop instructions), and while achieving this only using weakly supervised VQA data. There is already a strong community within NLP working on vision-language representation learning and generation who might be interested in this work as part of the “Language Grounding to Vision, Robotics and Beyond” in EMNLP. Along with this, image editing has been mostly studied for single object images and/or in single hop simplistic settings as pointed above. We are the first to propose the multi-hop editing problem and we thank the reviewer for recognizing that it is challenging and interesting. In particular we believe our results that show how our neuro-symbolic system can match or improve performance of even SOTA diffusion models is surprising and would be of great interest to researchers in academia and even industries who work on image editing, and might promote them to think of neuro-symbolic systems as an alternate pathway to achieve interpretability, sample efficiency etc, instead of trying to brute force these desiderata through larger and larger models trained on billions of samples. We believe the eventual application of this line of research would be for controllable image editing softwares with free form natural language instructions.\n\n>#### **2. Generalizability: Although the paper introduces the CIM-NLI-LARGE dataset to test zero-shot generalization, it is unclear how well NEUROSIM would perform on real-world images or more diverse datasets. Further evaluation on a wider range of datasets would help establish the model's generalizability.**\n\nThis is a fair criticism. At the same time, we argue that there is a dearth of datasets that test for multi-hop editing and ours is the first work that establishes such datasets, including CIM-NLI and CIM-NLI-LARGE. Although our datasets are based on synthetic CLEVR scenes, we believe they are still very useful, given that it is a challenge for even SOTA diffusion models like IP2P. We would like to mention that research in neuro-symbolic VQA started with images like CLEVR and Minecraft [1] while later matured into real world editing systems [2] and we expect the same for image manipulation based research/methods. We would definitely like to extend our method and experiment on real-world datasets and expect future research to build over our work to create and annotate more real world datasets.\n\nReferences:\n1. Yi et. al, Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. NeurIPS 2018\n2. Li et. al, Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images\n\n\n>#### **3. Comparison with Other Models: The paper compares NEUROSIM with several other models in terms of problem setting, task complexity, and approach. However, it would be beneficial to see more detailed comparisons, including the performance of NEUROSIM on tasks that other models excel at or struggle with**\n\n\nThank you for this suggestion. We use standard baselines closest to our problem setting [1,2] and SOTA diffusion model baseline (IP2P)  [2] for comparisons in our work. We would be happy to know about more baselines that we might potentially compare to and might be relevant to scientifically study the hypothesis in the paper. \n\nIn this paper, our primary focus is on the multi-hop image editing task and test models on a variety of settings as described in our experimental set up. We realize the reviewer might be referring to different types of image editing works that have sprung up in the last 6-8 months in the vision-language and image editing and generation communities. We acknowledge that our models might need modifications to work in all these settings, and we will be happy to cite/add a discussion with some of these works that have come about during the last few months [1]. With a very fast pace of research in this area, and moving goal posts, we believe it is plausible that newer methods have been proposed while we were preparing this manuscript. \nAt the same time, our focus in this work has been to develop a challenging and interesting task, and to show that the best image editing models also struggle on our novel and challenging task/datasets, and using a hybrid neuro-symbolic approach, we can achieve significant gains in interpretability, sample efficiency with reduced annotation costs. In the era of the largest LLMs and diffusion models, our paper also serves as a reality check for practitioners in these fields.\n\nReferences\n\n1. Bar-tal et. al, MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation. Arxiv 2023\n\n**Please let us know if you have any other questions or concerns and we would be happy to answer them and engage in discussions.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper and the main benefits to the NLP community are as follows:\n- Novel Model: The paper introduces NEUROSIM, the first neuro-symbolic, weakly supervised, and interpretable model for text-guided image manipulation that does not require output images for training. This model combines the strengths of neural and symbolic representations, offering modularity, interpretability, and improved generalizability.\n- New Dataset: The authors extend the CLEVR dataset to create a new dataset called Complex Image Manipulation via Natural Language Instructions (CIM-NLI) and CIM-NLI-LARGE for testing zero-shot generalization. These datasets provide a benchmark for evaluating models on the task of image manipulation using natural language instructions.\n- Competitive Performance: Despite being weakly supervised, NEUROSIM is highly competitive with or beats state-of-the-art baselines that make use of supervised data for manipulation. This demonstrates the effectiveness of the proposed approach in handling complex reasoning tasks.\n- Interpretability: The model is interpretable, as it parses instructions into a symbolic program based on a Domain Specific Language (DSL) comprising object attributes and manipulation operations. This allows for better understanding and debugging of the model's behavior.\n\n", "reasons_to_reject": "1. Limited Scope: The paper focuses on image manipulation tasks involving multiple objects and complex multi-hop natural language instructions. While this is an interesting and challenging problem, the scope of the paper might be limited for researchers working on other aspects of NLP or image manipulation\n2.  Generalizability: Although the paper introduces the CIM-NLI-LARGE dataset to test zero-shot generalization, it is unclear how well NEUROSIM would perform on real-world images or more diverse datasets. Further evaluation on a wider range of datasets would help establish the model's generalizability.\n3.  Comparison with Other Models: The paper compares NEUROSIM with several other models in terms of problem setting, task complexity, and approach. However, it would be beneficial to see more detailed comparisons, including the performance of NEUROSIM on tasks that other models excel at or struggle with.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper, titled \"Image Manipulation via Multi-Hop Instructions - A New Dataset and Weakly-Supervised Neuro-Symbolic Approach,\" focuses on the task of image manipulation using natural language text, which is useful for various AI applications but requires complex reasoning over multi-modal spaces\n\nThe main contributions of the paper are:\n1. The creation of NEUROSIM, the first neuro-symbolic, weakly supervised, and interpretable model for text-guided image manipulation that does not require output images for training.\n\n2. The extension of the CLEVR dataset, a benchmark dataset for Visual Question Answering (VQA), to incorporate manipulation instructions and create a new dataset called Complex Image Manipulation via Natural Language Instructions (CIM-NLI). The authors also create the CIM-NLI-LARGE dataset to test zero-shot generalization.\n\n3. Extensive quantitative experiments on the newly created CIM-NLI and CIM-NLI-LARGE datasets, along with qualitative experiments on Minecraft. Despite being weakly supervised, NEUROSIM is highly competitive with supervised state-of-the-art approaches, including a recently proposed diffusion-based model. NEUROSIM also performs well on instructions requiring multi-hop reasoning while being interpretable.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach for compositional edits to images in the CLEVR dataset. They also provide a dataset of editing instructions. Three reviewers provided reviews for this work that were in consensus. They found the paper to present a novel and interesting technique, they valued the dataset and found the results compelling. There were a few concerns noted in the reviews such as: the limited scope and generalizability to real world images. However, I must note that reviewers were still excited about the paper, and I consider these concerns to be addressable in future work. Based on the paper, reviews and discussion, I find this work sound and interesting for readers.
Decision: Accept-Main