GENOME: Generative Neuro-Symbolic Visual Reasoning by Growing and Reusing Modules
Authors: Zhenfang Chen
Rui Sun
Wenjun Liu
Yining Hong
Chuang Gan
Venue: ICLR 2024 poster (2024)
Keywords: Large language models
Neuro-symbolic Visual Reasoning
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)

Abstract: Recent works have shown that Large Language Models (LLMs) could empower traditional neuro-symbolic models via programming capabilities to translate languages into module descriptions, thus achieving strong visual reasoning results while maintaining the model’s transparency and efficiency. However, these models usually exhaustively generate the entire code snippet given each new instance of a task, which is extremely ineffective. On the contrary, human beings gradually acquire knowledge that can be reused and grow into more profound skills for fast generalization to new tasks since we are an infant. Inspired by this, we propose generative neuro-symbolic visual reasoning by growing and reusing modules. Specifically, our model consists of three unique stages, module initialization, module generation, and module execution. First, given a vision-language task, we adopt LLMs to examine whether we could reuse and grow over established modules to handle this new task. If not, we initialize a new module needed by the task and specify the inputs and outputs of this new module. After that, the new module is created by querying LLMs to generate corresponding code snippets that match the requirements. In order to get a better sense of the new module’s ability, we treat few-shot training examples as test cases to see if our new module could pass these cases. If yes, the new module is added to the module library for future reuse. Finally, we evaluate the performance of our model on the testing set by executing the parsed programs with the newly made visual modules to get the results. We find the proposed GENOME model possesses several advantages. First, it performs competitively on standard tasks like visual question answering and referring expression comprehension; Second, the visual modules learned from one task can be seamlessly transferred to new tasks; Last but not least, it is able to adapt to new visual reasoning tasks by observing a few training examples and reusing modules.

Published as a conference paper at ICLR 2024
GENOME: GENERATIVE NEURO-SYMBOL OLIC VISUAL REASONING BY GROWING AND REUSING MODULES
Zhenfang Chen
Rui Sun"
Wenjun Liu' Tsinghua University
MIT-IBM Watson Al Lab
Columbia University
Yining Hong
Chuang Gan
University of California, Los Angeles
MIT-IBM Watson AI Lab and UMass Amherst
ABSTRACT
Recent works have shown that Large Language Models (LLMs) could empower traditional neuro-symbolic models via programming capabilities to translate lan- guage into module descriptions, thus achieving strong visual reasoning results while maintaining the model's transparency and efficiency. However, these mod- els usually exhaustively generate the entire code snippet given each new instance of a task, which extremely ineffective. On the contrary, human beings grad- ually acquire knowledge that be reused and grow into more profound skills for fast generalization to new tasks since we are an infant. Inspired by this, we propose generative neuro-symbolic visual reasoning by growing and reusing mod- ules. Specifically, our model consists of three unique stages, module initialization, module generation, and module execution. First, given lisounlanguage task, we adopt LLMs to examine whether we could reuse and grow over established mod- ules to handle this new task. If not, we initialize a new module needed by the task and specify the inputs and outputs of this new module. After that, the new module is created by querying LLMs to generate corresponding code snippets that match the requirements. In order to get a better sense of the new module's ability, we treat few-shot training examples as test cases to see if our new module could pass these cases. If yes, the new module is added to the module library for future reuse. Finally, we evaluate the performance of our model on the testing set by executing the parsed programs with the newly made visual modules get the results. We find the proposed model possesses several advantages. First, it performs compet- itively on standard tasks like visual question answering and referring expression comprehension; Second, the modules learned from one task can be seamlessly transferred to new tasks; Last but not least, able to adapt to new visual rea- soning tasks by observing a few training examples and reusing modules!.
INTRODUCTION
Neuro-symbolic visual reasoning models (Andreas et al., 2016b; Mao et al., 2019b) refer to the al- gorithm family that combines deep neural networks (lec, 1998; Hochreiter & Schmidhuber, 1997) for learning correlations among the training data and symbolic methods (Yi et al., 2018; Andreas et al., 2016a) perform explicit and transparent multi-step reasoning. In contrast to pure ral network-bass models (Hudson & Manning, 2018; Li et al., 2023), heuro-symbolic approaches achieve strong performance in visual reasoning tasks, simultaneously offering superior model trans- parency and data efficiency.
Nevertheless, such models suffer from several inherent limitations. Firstly, their language parsers (Yi et al., 2018; Andreas et al., 2016b), employed for the conversion from natural language into sym- bolic programs, typically demand extensive domain-specific language-program pairs to train on, and struggle to generalize effectively to unconstrained natural language instructions. Additionally, these models necessitate a custom design for every module, rendering the process labor-intensive and lacking scalability.
indicates equal contributions
Project page: https: /vi /vis-www www. cs. umass. www.ss.edeggennoneee......

---

Published as a conference paper at ICLR 2024
Existing Methods
Neuro-symbolic Visual Reasoning
Library Modila LOCI... Modile 2:
def ocecate, command
BOXD MAAE objectt feood")
image jetzt lmnggFachitzagg
INAGE Piizae-DDA box--DO00)
IMAGEI= Pinagge box:
10 image-MMa E object 'ike'
colori couch patchuimphe mippe query WWaar's' the couch's coloee?
touct prichs
pash find""oosch
Q: Are the couch and sky same color? A: No.
Modila VQAL...
CROIN...
ANSW ERI-V ERL--RRAIlmaeee Q0 What color
ANSW ERO- QA (imigeeMAAEEn MP0000p coucat")
teloct tuuch patchstmple Wrat''''aarr's sky's color!")
We patihes image patch fredk "sky")
ANSW Laspp yos (ANSWERO)
(ANSWER.]
(4) VisProg
zlac rourr "No"
ooloerr emodlo?
"yre"
(B) ViperGPT
Our Method
Instruction: replare the apple with the same
I.M
Generato modiles
New Madale
PALL ATTRIBUTRCF
color is the grape with peach
4iobele
bacl wlan
Q: Does zicroware have diffecent color than the sosster?
Update
TresPale
No.
(C) G enerative Module Learning
(D) Applicatoon
The motivation of the GENOME. Compared with VisProg and ViperGPT which exhaus- tively generate a code snippet for each imput case, our GENOME is able to generate new modules and reuse old modules to handle the query. The module generated by GENOME can be used to handle other instances of the task for better performance. Second, the generated module can be transferred different tasks like image editing. Finally, it can learn to handle 1985; Zhang et al., 2019a) by learning modules from only a few training samples. The edited region and the correct answer for the Raven task labeled with red boxes for better visualization.
Figure 1:
tasks like Raven (Burke,
Recent advancements in large language models (LLMs) (Brown et al., 2020; Ouyang et al., 2022) have ushered in new with its remarkable performances across various applications, including chatbots (Shuster et al., 2022), virtual assistants (Dong et al., 2023), and programming assistants (Chen al., 2021a). Riding this unprecedented wave, researchers reformulate the old wisdom by incorporating LLMs into neuro- symuolcc reasoning, bypassing the inflexibility and ineffectiveness of domain- specifcc manguage-t--prggenn
parsers. Specifically, VisProg (Gupta & Kembhavi, 2022) pre-defines a set of visual modules, and uses L.MMs to transform language instructions into symbolic programs consisting of the pre-defined visual modules. Taking a step forward, ViperGPT (Surís et al., 2023) releases the burden on manually-define visual modules by introducing code generator that could produce a code snippet based on each input instance a new task.
Promising as these LLM-based neuro-symbolic models can be, they inevitably bear several weak- nesses compared to the learning and reasoning processes of human beings. First, both VisProg and ViperGPT exhaustively produce code snippet for each new instance of a task, which is extremely ineffective. This is in stark contrast with the human learning process: from an early age, we organ- ically accumulate knowledge from particular experiences. Such knowledge acquired from specific cases could be reused and reconfigured, enabling us to quickly adapt to new tasks and new demands (Harlow, 1949; Mitchell al., 1986; Lake al., 2016; Ellis et al., 2023). The knowledge blocks grow progressively over time, gradually into a library with extraordinary richness and flexibility any unseen task the knowledge library that these models fall short of. Second, both models do not verify and examine the codes they generate. It seems that when the models generate bad code snippet that cannot solve the input case, they just "let it go" without taking another stab for larger chance towards success. And of course, when they encounter similar cases again, they keep "stepping on the same rake". Human beings, on the other hand, would verify and examine the acquired knowledge by proposing set of test scenarios before storing them in the library (Brulé & Blount, 1989). It's crucial that neuro-symbolic reasoning model is equipped with the same abilities to verify the codes it produces, stores them in library if satisfactory, and makes another attempt when the codes fail.
for fast generalization
To this end, introduce novel Generative Neuro- Neuro-symbolic Visual Reasoning Model (GENOME), proficient in assimilating new neural modules from a limited set of training examples. This model excels in handling standard visual reasoning tasks such visual question answering. Addition-
2

---

Published as a conference paper at ICLR 2024
ally, it demonstrates outstanding module transfer capabilities for tasks like image editing, and ex- hibits an exceptional ability to generalize to new reasoning tasks with limited training examples. As illustrated in Figure 2, GENOME comprises three stages: 1) module initialization, 2) mod- ule generation, and 3) module execution. From the initial training set examples, an LLM discerns the necessity for new module tackle the task and, if required, produces the respective input and output. In the second stage, LLMs implement and refine the new module, ensuring seamless integration with existing modules and resulting in accurate responses to the training queries. Dur- ing testing, the LLM first converts language instructions into executable high-level programs like BOXOXO,BOXI,ATOO ATTR) for comparing attributes of different bound- ing boxes. The program will be run with the new module sets, producing the desired outputs.
responses
We assessed the performance of GENOME across six visual reasoning tasks, spanning from vi- sual question answering (Hudson & Manning, 2019) to Raven's 's Progressive Matrices (Zhang et al., 2019a). The experimental findings reveal that GENOME delivers competitive results on standard benchmarks, ensuring both transparency and interoperability. Notably, modules honed on these stan- dard tasks can be adeptly adapted to diverse domains, including image editing and knowledge tag- ging (Gupta & Kembhavi, 2022). Additionally, with minimal training examples, GENOME demon- strates the capability to manage new visual reasoning tasks (Burke, 1985; Jiang et al., 2023a) by repurposing modules.
RELATED WORK
Visual Reasoning. Our work aims to handle visual reasoning tasks, which require a model to draw new inferences based on the acquired visual cues in images videos (Hudson & Manning, 2019; Kazemzadeh et al., 2014; Goyal et al., 2017; Zhang et al., 2019a; Jiang et al., 2023a). Typical tasks for visual reasoning include visible question answering (Goyal et al., 2017; Hudson & Manning, 2019), visual grounding (Kazemzadeh et al., 2014; Yu et al., 2016; Chen et al., 2020) and Raven's Progressive Matrices (Burke, 1985; Zhang et al., 2019a). Various models (Hudson & Manning, 2018; Yu et al., 2018; Zhang et al., 2021; Ding et al., 2023) have been developed to handle these tasks but most of them are ad-hoc and carefully designed for a specific task, leaving it an open research question on how to build general model that can handle different kinds of visual reasoning problems by only showing a few examples.
Neuro-symbolic Visual Reasoning, Our work is also closely related to neuro-symbolic visual rea- soning models (Andreas et al., 2016b; Mao et al., 2019a; Chen et al., 2021c; 2022), where the models decompose the query of the visual reasoning tasks into a series of reasoning steps and repre- sent each reasoning step with a neural module (i.e., a code snippet for achieving specific functions like localizing objects and recognizing object categories). While these models have better model interoperability and data efficiency than previous connectionist models (Hudson & Manning, 2018; Anderson et al., 2018), they often show their limitations in representing natural language instruc- tions in the wild with the limited pre-defined reasoning steps (Yang et al., 2020; Chen et al., 2021b). Moreover, they need to manually define and implement each neural module one by one, making it hard to scale up and handle multiple tasks within a single model.
Foundation Models for Reasoning. Recently, large language models (LLMs) (Brown al., 2020; Ouyang et al., 2022) have been widely used in language understanding (Hendrycks et al., 2020) and reasoning (Cobbe et al., 2021; Amini al., 2019). Schick et al. (2023) develop the toolformer to show that LLMS can use external tools to better handle language tasks. Cai et al. (2023) shows that LLMs can make simple tools for natural language tasks by writing code snippets. LLMS have also been used in dision-languee nect LLMs with additional vision encoders and fine-tune them with massive vision- languale pairs. As evaluated by Xu et al. (2023b), while these models show great performance on in-domain tasks, they also perform poorly on tasks out of the training domains. They are also extremely computation- expensive. For example, it takes 15 days to use 1536 TPUv4 for training Flamingo (Alayrac et al., 2022).
tasks. Most of these works (Li et al., 2023; Alayrac et al., 2022) con-
LLMs for Programming. There are some works that use LLMs to write codes to handle tasks. Pereira & Hartmann used LLMs to progressively enhance and specify system subcomponents, em- powering users to develop versatile programs through systematic iterative disambiguation method. Jiang et al. (2023b) learned to generate code with LLMs, which involves a planning phase for outlin- ing solution steps and an implementation phase for generating code. Besides the dense engagement
3

---

Published as a conference paper at ICLR 2024
Qli dek made of differen QR Whe''s chair? Pantio....
QE: Dec dadio hune the sone materill th lidge?
-
Training ssamples
Testing csamples
Dupecc
Modele
Test
Sggnauures
cases
Generate CUNPARE AFTRIBCTRES
medilles
BOOD 4 MACIAAGGEaaaaa-- Vabbel
banda
ROL Loc LOC.AAnRi Hiile's TEINAGEL DESPURIIII
LLM
No
Train
Scalable Module
caddac dovid !ile??
image
Pase de
Fxnlisted coscss
Library
Cude Execution
lapec
BOOD LACIRANNE diout ? ANCCIERI CONPARRE
Medkk
Midkle
Anower No
Ourpu
Test image
Scalable Module Library
Module iignatures
Operation steps
(1). Module Initialization
(2). Generation
(3). Module Execution
The framework of our GENOME, which contains three stages, module initialization, module generation, and module execution. In stage 1, we feed the questions and the signature of the existing modules to the LLM and ask it to identify whether operations of the existing modules. If not, we ask the LLM to generate the signature of the new module (i.e. the input and output) and predict the reasoning steps handle the query task. In stage 2, we feed the module signature and the testing cases to the LLM and ask the LLM to implement the module and test its pass rate on the training examples. We only accept the modules that successfully handle the query. In stage 3, we first use the LLM to parse the query into symbolic operations and then execute these operations on the test images with the help of the scalable module library. We take VQA as an example and such framework can also be expanded to other tasks like referring expression comprehension and Raven.
Figure 2:
can handle the query within the
with the visual modalities as input, our GENOME differs from them in modularizing code snippets for better module expansion and reuse. These unique differences make our GENOME model have new capabilities like growing new modules to handle the visual reasoning tasks and transferring the modules into new domains. Some research works (Vendrow et al., 2023; Gao et al., 2023) also used LLMs and few-shot examples to improve Al models' performance. However, they only focused on improving the pure neural network model's performance and automatically discovered groups to use them for model design.
Visual Programming by LLMs. Another line of research has been combining vision models (Li* et al., 2022; Kirillov et al., 2023; Radford et al., 2021) with LLMs in an off-shelf manner. Early models (Yang et al., 2022: Chen al., 2023b) transformed images into captions and append the cap- tions into the LLMs' prompt to handle vision-language tasks. While the method is simple, they also perform inferior and lack transparency. Recently, VisPROG (Gupta & Kembhavi, 2022) uses LLMs to transform language instructions into pre-defined modular operations for step-by-step reasoning. However, still requires manually implementing each module one by one. Later, ViperGPT (Suris et al., 2023) shows that the LLMS can be used to write code snippet for each query instance inde- pendently to handle vision tasks. However, the code it writes has not been examined and tested by any training examples and there is no guarantee about the performance and code safety. Instead, we propose GENOME that ask LLMs to create new neural models (i.e. general code snippets to achieve specific functions) and handle the given tasks through only few training examples. Our GENOME has the reliance on a few training examples to learn new modules. However, such newly generated modules cooperate with each other and be reused for other tasks for better performance. There is also research like (Rahaman et al., 2021) which adopts pure neural network architecture to ab- stract the reasoning problem by generating the script and dynamically executing it. Differently, our GENOME is a neuro-symbolic method that provides better model transparency in explicit Python scripts and is able make use of existing large pre-trained models to make and reuse new modules.
METHOD
3.1 OVERALL
In this section, we present novel framework named Generative Neuro-symbolic Visual Rea- soning Model (GENOME) for the acquisition of neural modules and solutions of visual reasoning tasks with only limited set of training examples. GENOME comprises several pre-defined opera-
4

---

Published as a conference paper at ICLR 2024
tors that serve as the initial building blocks. Each neural operator corresponds to a neural module and is implemented using a Python code snippet, thereby enabling specifie functions such as object localization within an image. Nevertheless, it is not possible to pre-define all the necessary neural modules prior to addressing the visual reasoning tasks. Consequently, there arises a need to generate new modules based on a limited number of visual reasoning task examples.
Figure 2 illustrates that GENOME consists of three distinct stages: 1) module initialization, 2) mod- ule generation, and 3) module execution. In the module initialization stage, when provided with a small set of training examples from the visual reasoning dataset, the primary objective is to de- termine whether the existing neural modules are sufficient to address the query examples. If the existing modules are inadequate for the query instance, GENOME will identify the requirements for creating new modules, including defining the modules' input and output specifications. Dur- ing the module generation stage, GENOME leverages the LLM to implement the neural module based on the provided training examples and the specified input and output format (i.e., function signature) and add the module to the module library only when it passes the test cases. Once the new module is successfully implemented, the module execution orchestrates the transformation of input queries into sequence of neural operations. Subsequently, these operations the neural modules to obtain the correct output. All these three stages are powered by robust code generation capabilities and the in-context learning technique. Prompts for each stage can be found at Figure 15-17
applied to
3.2 MODEL DETAILS
Utilizing limited number of examples from the training set of visual reasoning tasks, we employ the GENOME framework, comprising three distinctive stages: module initialization, module gener- ation, and module execution.
The initial phase within our GENOME framework is module initializa- tion, dedicated determining the set of new modules required address the visual reasoning task. As depicted in Figure 2-1, we employ an LLM to assess the feasibility of handling these training instances using existing neural modules. Should this not be achievable, we task the LLM with specifying the necessary new modules (e.g. COMPARE ATTRIBUTE in Figure 2) for an accurate response to the query. The outcome of this stage comprises function signatures detailing the input and output formats for these new modules. Furthermore, it facilitates the transformation of the input query into a sequence of reasoning steps, which function as test cases to validate the correctness of the generated program within module generation. The prompt for the LLM in this stage is shown in Figure 15.
Module Initialization.
The second phase of our GENOME framework is module generation, which focuses on implementing the correct new modules proposed during the module initialization stage. Specifically, after receiving the signature of a new module, we incorporate all the corresponding test cases that call the new module into the prompt and employ Geaming-in-conter generate multiple program candidates. Note that a new module usually paired with multiple test cases. These program candidates are subsequently executed using the provided training examples. If a program encounters errors during execution, we incorporate the error information into the LLM's prompt and instruct it to rectify these issues. We only accept program candidates that achieve pass rate surpassing a predefined threshold (7). This procedure bears resemblance to the code transla- tion of LLMs discussed in (Chen et al., 2023a), but we extend it to accommodate more intricate multi-modal input types and instructions from natural language and raw images. The inclusion of module generation in the context of visual reasoning tasks offers two principal advantages. Firstly, it upholds the transparency and interpretability of neuro-symbolic models while preserving compet- itive performance. Secondly, it exhibits generative capabilities and scalability as our GENOME can autonomously generate new modules tailored to specific tasks.
Module Generation.
techniques to
Module Execution. Given the integration of newly-generane modules with existing neural mod- ules tailored for visual reasoning, the GENOME framework initiates query parsing from the testing dataset, transforming them into executable operations through in-context learning. An illustrative prompt for this stage is depicted in Figure 17. Notably, although various visual reasoning tasks may possess distinct inputs and outputs, they can re- re-purpose these intermediary modules designed for other tasks to enhance overall performance. This feature represents unique capability for code generation at the module level, an aspect hitherto unexplored by prior methods(Suris et al., 2023; Gupta& & Kembhavi, 2022).
5

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
In this section, we present a comprehensive series of experiments to evaluate the performance of our models. Initially, we demonstrate our models' effectiveness in learning neural modules on two estab- lished benchmarks: GQA (Hudson & Manning, 2019), focusing on compositional visual question answering, and RefCOCO (Kazemzadeh prehension. Subsequently, we illustrate how the modules acquired from these two datasets can be successfully applied to novel tasks such as image editing and knowledge tagging. Moreover, we highlight the adaptability of our framework to address novel visual reasoning tasks (Raven (Zhang et al., 2019a) and MEWL (Jiang et al., 2023a)), even with limited training examples. Before delving into these experiments, we provide an overview of the experimental settings.
al., 2014), which assesses referring expression com-
Experimental Details. The success of our GENOME relies on a set of pre-defined modules and APIs the starting point. We utilize handcrafted modules from VisProg (Gupta & Kembhavi, 2022) as our initial components, Additionally, we incorporate several new APIs from ViperGPT to enhance module creation. We also include some new APIs from ViperGPT (Surís et al., 2023) for making new modules. In Section 4.4, we also include results parsed by the open-source LLM from WLM (Xu et al., 2023a) to investigate the influence of different LLM models. A comprehensive list of the pretrained modules employed in our approach can be found in Section A. of the Appendix. We extract training examples acquire new modules. More precisely, we extracted 300 examples from GQA, 100 from RefCOCo, 10 from Raven, and 10 from MEWL.
Datasets and Evaluation Metric. We show experiments of our GENOME on standard vision- language benchmarks, GQA (Hudson & Manning, 2019) and RefCOcO Kazemzadeh et al. (2014), GQA is a popular compositional visual reasoning dataset with synthesis multi-hop questions, mak- ing it suitable for multi-step reasoming. RefCOCO is typical visual grounding dataset, evaluating models' ability localize objects and understand fine- -reaggrriddd spatial and semantic relationships. Following ViperGPT, we evaluate GQA on test-dev split and RefCOCO on the testA split. Then, we show GENOME's abilities on the other transferred tasks, image editing, and knowledge tagging and compare it with VisProg. Since the image editing and knowledge tagging datasets from VisProg not publicly available, we built two new datasets for evaluation. The new editing dataset contains 50 images and instruction pairs. The new knowledge tagging dataset contains 50 images with 50 referring expressions. We provide more details about the dataset in Appendix A.4. The datasets will be released for research purposes. Finally, we show that GENOME can learn to handle new visual reasoning tasks like Raven (Zhang et al., 2019a) and MEWL (Jiang et al., 2023a) by observing a few training examples and module learning. Raven is a task for relational and analogical visual reasoning of image sets and has been widely used for non- von-verbal intelligence tests. MEWL is recent benchmark proposed to assess how machines learn word meaning in grounded visual scenes. Examples of these tasks can be found at Figure 5 and Figure 6.
4.1 COMPARISON WITH BASELINES ON VISUAL REASONING.
We conducted analysis between our model and several baseline models us- ing the GQA and RefCOCO datasets. Due to the deprecation of the origi- nal professional Codex API (code-davinci-O00) currently available API ments with both our model and the baseline models to ensure fair compari- son. We did not carry out experiments with GPT44 due to the prohibitive cost.
we replaced it with the and conducted experi-
The results, as presented in Table 1, demonstrate that model achieves competitive performance in both visual question answering and referring expression comprehension, thus confirming its effectiveness. Fur- thermore, we provide an illustrative module from our model in Figure 1. This newly created module has the capability to utilize various available APIs to select at- tributes from the images. The step-by-step reasoning process of our model is detailed in Figure 3, offering greater transparency compared to end-to-end models.
Methods
GQA RefCOCO
BLIP-2
44.7
KOSMOS-2
57.4
ViperGPT-CodeX
48.1
72.0
VinPRoG-Instruct
45,4
ViperGPTTInstruct
38.2
62.4
Burs-Inntruct
45.6
69.2
Table 1: Evaluation on standard visual rea- soning benchmarks, GQA and RefCOCO.
4.2 GENOME FOR TRANSFER LEARNING In this section, we demonstrate our model's 's robust ca- pabilities in transfer learning- We augment the modular library by incorporating modules created

---

Published as a conference paper at ICLR 2024
GQA
prdtes decee
Ri
Questio: the sleveee long sleeved?
Generated Program:
aCotwiOC1 wagg-tnean www.ert (ckseet)
MISUERSHOROOEE Srg booe0088 satort aleeved")
3P, doort The thatter senver wrdrd mne 17 Maaven and prright noull Wwl Skordd
resel shatskevee
Refcoco
IMAGE:
Question: blue.
Program:
Ados lis uin lee recesrt((eenge taass, soc, Hete
0012-5011, SPWTTALLLCC Cww.aALcccagee Ceage-22GG bon, 2100-000..111 locar froor" tedee)))
Figure 3: Qualitative examples of GENOME's on GQA and RefCOCO. The query images, language instructions, and the parsed programs shown the left. The corresponding modules and the value of important variables are shown on the right.
Replace bottle
Select superhero
blue
Hide the second mon from the left with 5)
Create color pop of the third person from the right
Tag the secand bctor who played James Bond from the loft
Tag the Grammy winning muscian in the misdle
Tag the social media platforn logo the some color facebook
Tag the second gas planet our soler syste m
O
Figure 4: Qualitative examples of GENOME's on the image editing and knowledge tagging tasks. The language instructions of the tasks are shown the original images while the modified results of the images are shown below the original images. The emphasis of the instructions highlighted with red colors, which requires new modules handle. While the key regions of the output images are bounded with green colors.
from GQA and RefCOCO, employing in-context examples guide the Language Model (LLM) in generating step-by-step instructions for task execution. Qualitative results for this task are depicted in Figure 4. As illustrated in Figure 4, our model excels in generating semantically accurate images using the newly added module, whereas the baseline VisProg struggles to capture the required re- lationships with its fixed, pre-defined module library. To provide a more comprehensive evaluation of image editing, enlist annotators to manually assess the correctness of the generated images. The models' performance is compared in Table 2, where our model outperforms the bascline. In the context of knowledge tagging, task annotators with marking image regions referenced by

---

Published as a conference paper at ICLR 2024
Image Editing Accuracy
Tagging Recall
Localization
Methods
Precision
F1
Precision
Recall
F1
VisProg GENOME
16.7
18.4
21.7
19.9
32.8
35.3
34.0
55.3
67.1
52.3
58.8
76.9
57.9
66.0
Table 2: Evaluation of GENOME on Transfer Learning with image editing and knowledge tagging tasks. Our GENOME shows much better performance on all criteria, showing the effectiveness of the transferred modules. A qualitative comparison can be seen Figure in the Appendix.
Methods Center
L-R
U-D
Methods
shape
color material
ResNet+DRT
58.1
65.8
67.1
Aloe
34.2
33.2
31.0
ALANS-V
98.4
97.3
96.4
Flamingo-1.ID I.
49.3
35.3
48.5
GENOME
80.1
67.6
69.1
GENOME
43.7
45.3
41.0
Human
95.5
86.4
81.8
Human
92.4
87.2
72.7
Table 3:
Evaluation of GENOME on Raven (Zhang et al., 2019a). Compared MEWL (Jiang et al., 2023a). Compared with methods trained with massive in-domain with approaches trained on extensive in-domain data, our model performs competitively.
Evaluation of GENOME on Table 4:
data, our model shows competitive performance.
pressions and employ the same metrics as RefCOCO for evaluating the accuracy of bounding boxes and employ the BERT score to assess the correctness of labeled names. Our model demonstrates superior performance in both image editing and knowledge tagging. We show a typical example in Figure 7 of the Appendix to show how our GENOME make use of new modules to perform better knowledge tagging result the baseline.
4.3 GENOME ON FEW-SHOT TASK LEARNING.
As general module learning framework, our model is not only able to learn new modules to handle existing tasks but also can learn to handle new visual reasoning tasks from few training exam- ples. We evaluate such abilities on new tasks, Raven (Zhang et al., 2019a) and MEWL (Jiang et al., 2023a). Specifically,
first prompt the LLM learn pattern recognition modules for visual un- derstanding and then ask the LLM to generate solver module to handle the task. The instances of our model prediction are shown in Figure 5 and Figure 6. Note that visual reasoning from Raven is widely used in intelligent testing for humans, which shows our model's strong capabilities and poten- tial. We report the performance of our model and the baselines in Table and Table Our model is significantly better than previous fully-superviser methods like ResNet+DRT (Zhang et al., 2019a) and Aloe (Ding et al., 2021) 1), showing its effectiveness. Note that all these models ResNet+DST. ALANS-V (Zhang et al., 2022), Aloe (Ding et al., 2021) and Flamingo (Alayrac et al., 2022) are models fully- fillt-fieetuned on in-domain data, while our GENOME a general few-shot framework to observe the new compositionality and mod-
leamn modules for mooblem-solving Moreover, ule re-usage from Figure 8 of the Appendix. Although the SOLVER module was originally learned from center-type problems, it can be naturally transferred to other types like left-right and up-down.
4.4 ABLATIONS
Methods
RefCOCO
To gauge the efficacy of our model, we conducted a series of ablation studies addressing the following key inquiries: Q1 How effective is module learn- ing? Q2 What impact does the quantity of train- ing examples have on model performance? Q3 How crucial is the LLM's capability for optimal perfor- mance? In our experiments, GENOME w/o ML repre- sents configuration without any new module learn- ing but relies heavily on ViperGPT and VisProg-define point region matching the referring expression. On the other hand, GENOME-WLA, replaces the gpt-3. S-turbooiins
GENOME w/o ML
62.3
GENOME- WLM
64.4
GENOME (10)
49.4
GENOME (50)
67.0
GENOME (100)
67.1
Table 5: Ablation study of GENOME on RefCOCo.
modules, directing the LLM to pin-
API with Wi WizardCoder-Pythont 3eta-334B-..... 0 from Wiz- ardLM (Xu al., 2023a). The designations GEN
indicate models trained with
8

---

Published as a conference paper at ICLR 2024
RAVEN
GRddeEE TECT LO0Ed4 (Snage
SHRCHOCTEET (ongEenngee0
Piice Meng''
shapp, Jype: constant $20_3pc, progrossion
MOOUCI-OELVSTI 2MACE, 12200000101 shapeng (aAe-C22E) TuRPAR H(deadarate)
ANEWEEIS
A qualitative example from the Raven dataset (Zhang et al., 2019a) is provided. This task involves set of images with varying visual attributes, such as colors, shapes, and locations. Models are tasked with identifying the image that best matches the missing item in the Problem Matrix. GENOME exhibits the capability to compose modules (i.e. DETECT SHAPE and SOLVER) for detecting these attribute rules and constructing a solver module to address the task. The correct answer is indicated by a green box.
Figure 5:
MEWL
Content
Cuury
IMACOF::
oyon
gray
ojon
oyon gray green
oyan
ACHEE-SOLVERI ( c000,
WORD:
atiec
pono
Gange
sdacc
FINAL FTN (ostrtt
ROTO aioc
atlec encan encan
crean
Figure 6: A qualitative illustration from the MEWL dataset (Jiang et al., 2023a) is presented. This task entails a set of images featuring diverse visual attributes, such as material and shapes, and it necessitates models to determine the word that corresponds to the query image. GENOME demonstrates the capability to generate modules for identifying these attribute rules and composing solver module to address the task. The correct answer is indicated by a green box.
10, 50, and 100 examples, respectively. For resource constraints, limited our experimentation to 800 RefCOCO samples.
Table 5 presents the outcomes, leading to these insights: module learning, given sufficient test in- stances, can bolster task performance (Q1 addressed). A paucity training examples, such as for RefCOCO, might induce overfitting, but this diminishes with increased training data (50 examples), improving overall performance (Q2 addressed). Finally, model performance appears intrinsically tied the LLM's capacity, with superior LLMs delivering enhanced results (Q3 addressed).
5 CONCLUSION
In this study, we introduce GENOME, which is designed to tackle visual reasoning tasks when confronted with limited training data. This approach combines language models to parse natural language into executable operations and create specialized visual modules tailored to the given task. Our model exhibits competitive performance on conventional tasks, effortless transfer of acquired modules novel tasks, and the capability to adapt
tasks with limited training data. Our GENOME also proposes numerous avenues for future research. Firstly, it still necessitates task- specific prompts for each distinct reasoning task, and it would be intriguing explore the of universal prompt for all tasks. Secondly, the framework can be extended to encompass a broader range of multi-mode modal reasoning tasks, incorporating diverse inputs such as audio, video, and tactile information.
ACKNOWLEDGEMEN
This work was supported by DSO grant DSOCO21072. We would also like to thank the computation support from AiMOS, server cluster for the IBM Research Al Hardware Center.

---

Published as a conference paper at ICLR 2024
REFERENCES
Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11) 1):2278- 2324, 1998.
Lean-Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv, 2022.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Ha- jishirzi. Mathqa: Towards interpretable math word problem solving with operation-basee for- malisms. arXiv preprint arXiv: 13319, 2019.
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 6077-6086, 2018.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. arXiv preprint arXiv:1601.01700 2016a.
Jacob Andreas, Mareus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In CVPR, pp. 39-48, 2016b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
James F. Brulé and Alexander Blount. Knowledge acquisition. 1989. URL https:://ap)
somanticscholin www. Cerpg//oopsss::::: 11D66869699.
Jannis Bulian, Christian Buck, Wojeiech Gajewski, Benjamin Boerschinger, and Tal Schuster. Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation, 2022.
Henry Burke. Raven's progressive matrices (1938): More on norms, reliability, and validity. Journal of Clinical Psychology, 41(2):231-235, 1985.
Tianle Cai, Xuezhi Wang. Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021a.
Wenhu Chen, Zhe Gan, Linjie Li, Yu Cheng, William Wang, and Jingjing Liu. Meta module network for compositional visual reasoning. Proceedings of WACV, 202
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:23004.55128, 2023a.
Zhenfang Chen, Peng Wang. Lin Ma, Kwan-Yee K Wong, and Qi Wu. Cops-ref: A new dataset and task on compositional referring expression comprehension. In CVPR, 2020.
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Kenneth Wong, Joshua B Tenenbaum, and Chuang Gan. Grounding physical concepts of objects and events through dynamic visual reason- ing. In ICLR, 2021c.
Zhenfang Chen. Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B Tenenbaum, and Chuang Gan. Comphy: Compositional physical reasoning of objects and events from videos. arXiv, 2022.
Zhenfang Chen, Qinhong Zhou, Yikang Shen. Yining Hong, Hao Zhang, and Chuang Gan. See, think, confirm: Interactive prompting between vision and language models for knowledge-basee visual reasoning. arXiv, 2023b.
10

---

Published as a conference paper at ICLR 2024
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt Botvinick. Attention over learned object embeddings enables complex visual reasoning. Advances neural information processing systems, 04:9112-9120 2021.
Mingyu Ding, Yan Xu, Zhenfang Chen, David Daniel Cox, Ping Luo, Joshua B Tenenbaum, and Chuang Gan. Embodied concept learner: Self-supervised learning of concepts and mapping through instruction following. In CoRL, 2023.
Xin Luna Dong, Seungwhan Moon, Yifan Ethan Xu, Kshitiz Malik, and Zhou Yu. Towards next- generation intelligent assistants leveraging Ilm techniques. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5792-5793, 2023.
Kevin Ellis, Lionel Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lore Anaya Pozo, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: growing generalizable, interpretable knowledge with wake-sleep sleep bayesian program learning. Philosophical Transactions of the Royal Society A, 381 381(2251):2020055 2023.
Irena Gao, Gabriel Ilharco, Scott Lundberg, and Marco Tulio Ribeiro. Adaptive testing of computer vision models. In CVPR, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stur Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. ArXiv, abs/2211. 2022.
Harry Frederick Harlow. The formation of learning sets. Psychological review, 56 1:51-65, 1949.
URL httph:ttp/// /api..p...// wwm.nticsccootoi
orgg Corp//Crp : 22804426.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 00X0::0009003300, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Sheng Hu, Yuqing Ma. Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. In AAAI, 2021.
Drew Hudson and Christopher Manning. Compositional attention networks for machine rea- soning, arXiv preprint Ttt:::1803300000 2018.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Guangyuan Jiang, Manjie Xu, Shiji Xin, Wei Liang, Yujia Peng, Chi Zhang, and Yixin Zhu. Mewl: Few-shot multimodal word learning with referential uncertainty. In ICML, 2023a.
Xue Jiang. Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. Self-planning code generation with large language model. arXiv, 2023b.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023.
11

---

Published as a conference paper at ICLR 2024
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people, 2016.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv, 2023.
Liunian Harold Li Pengchuan Zhang*, Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The Neuro- Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervi- sion. In ICLR, 2019a.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In ICLR, 2019b.
Tom Michael Mitchell, Richard M. Keller, and Smadar T. Kedar-Cabellly Explamation-based gen- eralization: A unifying view. Machine Learning, 1:47-80, 1986. URL https://apa /apii. ieraaticschoo ar wtw..or//ooopsssi::11..
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Zamfirescu JD Pereira and Bjoern Hartmann. Iterative disambiguation: Towards Ilm-supported programming and system design.
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision. In ICML, 2021.
Nasim Rahaman, Muhammad Walced Gondal, Shruti Joshi, Peter Gehler, Yoshua Bengio, Francesco Locatello, and Bernhard Schölkopf. Dynamic inference with neural interpreters. 2021.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. ICCV, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv, 2023.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, et al. Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv, 2022.
Steven Spratley, Krista Ehinger, and Tim Miller. A closer look at generalisation in raven. In ECCV, 2020.
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv, 2023.
Joshua Vendrow, Saachi Jain, Logan Engstrom, and Aleksander Madry. Dataset interfaces: Diag- nosing model failures using controllable counterfactual generation. arXiv, 2023,
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint ar4iv 2023a.
12

---

Published as a conference paper at ICLR 2024
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang. Yu Qiao, and Ping Luo. Lvlm-chub: A comprehensive evaluation benchmark for large vision-language models. arXiv, 2023b.
Jianwei Yang, Jiayuan Mao, Jiajun Wu, Devi Parikh, David D Cox, Joshua B Tenenbaum, and Chuang Gan. Object-centric diagnosis of visual reasoning. arXiv preprint arXiv:2012.11580 2020.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang- An empirical study of gpt-3 for few-shot knowledge-based vqa. In AAAI, 2022.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. In NeurlPS, 2018.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amster- dam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pp. 69-85. Springer, 2016.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara I. Berg. Mat- tnet: Modular attention network for referring expression comprchension. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1307-1315, 2018.
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprini arXiv:2 1.08276, 2021.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song- Caug--huunnu Zhu. Raven: dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5317-5327, 2019a.
Chi Zhang, Baoxiong Jia, Feng Gao, Yixin Zhu, Hongjing Lu, and Song-Chun Zhu. Learning perceptual inference by contrasting, NeurlPS, 2019b.
Chi Zhang, Baoxiong Jia, Song-Chul Chun Zhu, and Yixin Zhu. Abstract spatial-temporal reasoning via probabilistic abduction and execution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Chi Zhang, Sirui Xie, Baoxiong Jia, Ying Nian Wu, Song-Chun Zhu, and Yixin Zhu. Learning alge- braic representation for systematic generalization in abstract reasoning. In European Conference on Computer Vision, pp. 692-709. Springer, 2022.
Kecheng Zheng, Zheng jun Zha, and Wei Wei. Abstract reasoning with distracting features. In NeurIPS, 2019.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
In this section, we substantiate claims in the paper by providing additional implementation de- tails (Section A.I), more experimental analysis (Section A.2), exemplar prompts for each stage (Sec- tion A.3), details on dataset collection (Section A.4), qualitative examples of new learned modules (Section A.5).
A.1 IMPLEMENATION DETAILS
Pre-defined Modules and API models. The success of our model still requires set of pre- defined APIs. Following the modules in VisProg and ViperGPT, we adopt the following APIs. We adopt GLIP (Li* al., 2022) for object localization. We adopt from WizardLM for code generation. We use BLIP (Li et al., 2023) for answering simple questions about images. We use CLIP (Radford et al., 2021) 1) and X-VLM (Zeng et al., 2021) for image-text classification. We use MiDaS (Ranftl et al., 2021) for estimating depths in images. We use stable diffusion (Rombach et al., 2022) for modifying image patches. Based on these APIs, we construct a set of pre-defined modules following VisProg. These pre-defined modules will be used cooperate with the learned modules for visual reasoning.
-tst--ruur from
OpenAl and SeeaariCoddr-Python---33
Descriptions
Modules
Loc for object location, FaceDet for face detection, Select and Filter Proterty for image-text classification. Filter Spatial for selecting image regions; Replace for image editing, colorPop for changing images colors, BgBlur for blurring background, Tag for annotating box regions and Emoji for face tagging. Crop and its variants for cropping patches from the images.
Image Understanding
Image Manipulation
List for retrieving factual knowledge, Count for counting object numbers, Eval, Result. BOX2MASK and MASK2BOX for formatting outputs.
Others
Table 6: Pre-defined Modules used GENOME.
Details Raven and MEWL. Note that visual reasoning task does not necessarily use lan- guage as input. All we need is to prompt the LLMs to generate modules that recognize the patterns and solve the problem. In RAVEN, by prompting LLM,, we can obtain DETECT_COM CCOLOR, DE- TECT_SHAPP and DETECT
The image is fed into these modules and the output is the color, shape, and size of the image. In this way, the input image is converted into a (color, shape, size) triplet. We provide LLM with ten examples from the RAVEN train split to demonstrate how to deduce the pattern of these triplets. By observing few-shot demonstrations, we let LLM generate the SOLVER( module, which detects the pattern of input triplets from the Problem Matrix and chooses the most appropriate answer from the Answer Set. Therefore, the internal of the SOLVERO module is primarily based on judgment, used to identify the patterns of the input triples in the Problem Ma- trix, thereby finding the answer in the Answer Set. The workflow of RAVEN is shown in Figure 5. As for MEWL, employ similar approach to handle it. One example is provided Figure 6. Since MEWL and RAVEN have different patterns, the SOLVER() module is not shared between RAVEN and MEWL. Thus, it utilizes distinct logic.
A.2 MORE EXPERIMENTAL ANALYSIS.
Computational Efficiency of Module Reuse. Our modularized design and module reuse strategy offer higher computational efficiency and generate shorter code compared to baseline approaches like ViperGPT, which creates solutions from scratch without modular abstraction and reuse. We calculated the average token count for both GENOME model and ViperGPT, which lacks module reuse mechanism, in their interactions with LLMs. As shown in table 7, our GENOME model's solutions are demonstrably shorter and more efficient.
14

---

Published as a conference paper at ICLR 2024
Methods
GQA
RefCOCO
ViperGPT GENOME
153.7
109.1
62.3
54.4
Table 7: Comparison of average token number for the generated solutions on GQA and RefCOCO.
More Ablation Study on Different Components. To dissect different components of our pro- posed method, we showcase more comprehensive and detailed ablation study here. We randomly select 800 samples from GQA test-dev split to further investigate the effectiveness of different com- ponents of GENOME. Moreover, we add baseline experiments of RAVEN and MEWL for better comparison. To better present all experimental results, all the ablation studies are organized into the following sections.
Methods
GQA
GENOME
45.9
GENOME w/o input and output format
43.2
GENOME w/o good initialization
41.8
GENOME w/o existing modules in prompt for module making
45.0
GENOME w/o creating new modules
44.7
GENOME w/ random sampling
44.3
GENOME (60)
44.5
GENOME (120)
45.3
GENOME (300)
45,9
GENOME w/ different LLM
44.3
GENOME w/o debugging
44.9
Table 8: More ablation GQA dataset.
Ablation on Prompt Design. We conducted a series of experiments to observe the impact of prompt design on the overall performance of GENOME. Firstly, we removed the descriptions of input and output formats from the prompt. After removing these descriptions, the performance dropped by 2.7%. This is because, without clear guidance on input and output formats, the modules might output the wrong format, leading to errors in subsequent parsing of the results. Further- more, top of removing the input and output format, also removed some of the in-context examples and descriptions about module signatures from the prompt. The performance further de- clined. Since method consists of three stages: module initialization, module generation, and module execution, where module initialization is the first step of our method. Without adequate module initialization a foundation, the subsequent results largely impacted. Therefore, can see that without good initialization, our performance drops by 4.1%.
GENOME
Regarding the use of existing modules and creating new ones, from Table 8, not using the predefined modules from VisProg results in 0.9% decrease in our performance. This demonstrates the robust module generation capability of GENOME. Even without a series of predefined modules, our method can still build modules from scratch, solve problems, and the performance does not drop significantly. the predefined modules. We can see that the result is 44.7%, which is 1.2% lower than our result of 45.9%. This performance gap highlights the effectiveness of the newly generated modules. By generating and using new modules, we can achieve better results.
observe that
don't create new modules, then we are merely using
Ablation on Sampling. In this section, we introduce our sampling strategy at first. Then, we conduct experiment showcase how the sampling methods will impact GENOME performance. Subsequently, we investigate how the number of training samples affects our results of different tasks.
Sampling Strategy. Our sampling strategy: the GQA dataset contains five structural types: choose, logical, compare, verify, and query. These structural types inspired the idea of generating our modules. Taking COMPARE COLOR as an example, this newly generated module is generated to address questions related color within the compare structural type. From the visualization
15

---

Published as a conference paper at ICLR 2024
RAVEN
MEWL
# of Samples
Center
L-R
U-D
shape
color
material
46.5
37.2
39.8
38.9
39.6
37.9
10
80.1
67.6
69.1
43.7
45.3
41.0
20
80.1
67.6
69.1
43.7
45.3
41.0
Table 9; Number of Sampling Examples on RAVEN and MEWL.
of GQA, it is apparent that the query type can be addressed using the existing VQA module from VisProg, and problems in the logical type can be decomposed into sub-problems of choose, compare, and verify types. Therefore, when selecting training samples, we randomly chose 100 samples each from the choose, compare, and verify types. Altogether, these three types comprise 300 samples, all sourced from the GQA train split. Hence, we are not are selecting training samples based on the structural types of GQA.
our training samples; rather, we
To explore the impact of sampling strategies on our experiment, we conducted an additional experi- ment with a random sampling of 300 samples, beyond our initial sampling strategy. In this setting, we randomly sampled 300 examples from the GQA train split. The performance was observed to be 44.3%, decrease of 1.6% compared to 45.9%. This result suggests that strategic sampling method can more effectively guide the LLM in generating more efficient modules for a given task. Relatively speaking,
method is robust in the face of choices in sampling strategies.
Number of Sampling Examples. We conduct a series of experiments to illustrate how the number of training samples influences the performance.
In the GQA and RefCOCO datasets, if a small number of training samples are used, it's possible for the generated modules to overfit certain samples, thereby reducing the generalization capability of the newly generated modules. Such overfitting in new modules can negatively impact the final results. Therefore. we can observe that when the number of samples small, the performance of GENOME is poorer. As the number of samples increases, the effectiveness of GENOME improves. However, with further increase in the number of samples, the performance gains of GENOME tend to saturate.
Regarding RAVEN and MEWL, since their patterns of change are limited, the number of few-shot samples selected is sufficient if it already covers all the variation patterns in RAVEN and MEWL. In other words, if the number of samples exceeds this threshold, there won't any further im- provement in the results; if it's below this threshold, the performance will decline. We selected 10 few-shot samples each in RAVEN and MEWL. As can be seen from the results in the table above, if the number of samples is equal to 5, there is a noticeable decrease in performance. This is be- cause 5 few-shot samples are not enough to cover all the variation patterns of RAVEN or MEWL. If the number of samples is equal to 10 or 20, at this point, the few-shot samples are sufficient to encompass all possible variations. In this case, the same results are obtained.
Ablation LLM's capability. By using better LLM,, our prompts can be better understood, and the LLM will generate higher-quality modules. In this experiment, we compared the results of using gpt- 3. wr.-tuuroo--o wwwuuuoooinnstuuu instruct (i.e., GENOME) and gpt-3. 1.c-3355turbb LLM). Our experimental results show that better outcomes are achieved when using the more ef- fective gpt 3.0t-3.25 5- turbo- instt It is evident that the capabilities of the LLM influence the performance of GENOME. As the abilities of LLMs continue to improve, so will the performance of GENOME. Thanks to the flexibility of GENOME, once better LLM is available, we can easily switch to the latest LLM to achieve better results.
(i.e., different
Ablation on Debug Mechanism. The error-correction prompt contains the error message from Python interpreter and wrong code snippet. We prompt the LLM to correct the wrong code based on the error message from Python interpreter. We heuristically set the maximal number of debug iterations as 5. If the wrong code can be corrected within 5 iterations, we will keep it. Otherwise, it will be abandoned. (Details can be found in the Module Generation section of Figure 2) The errors mainly stem from two sources: one is basic syntax errors in Python code, such as indentation and variable name errors. The other source is some fundamental logical errors, such as mistakes made when setting variable types, like treating variable that should be of the bool type the string
16

---

Published as a conference paper at ICLR 2024
RAVEN
MEWI
Methods
Center
L-R
U-D
shape
color
material
VisProg variant
36.8
26.1
27.8
35.2
35.9
34.9
ViperGPT variant
40.6
30.7
32.4
37.8
38.2
36.7
Ours
80.1
67.6
69.1
43.7
45.3
41.0
Table 10: Compare our GENOME model with baselines, VisProg, and ViperGPT on RAVEN and MEWI..
type. By observing the table above, we can conclude that the debug process can assist GENOME with generating more useful modules to elevate performance and prevent elementary programming mistakes.
Additional Baseline Experiments of RAVEN and MEWL. For RAVEN and MEWL, we have implemented the ViperGPT and VisProg baseline experiments in the following way. For VisProg, requires manual implementation of all modules by making use of the provided APIs. Thus, to enable VisProg to handle Raven and MEWL tasks, we manually implement and debug new hand- crafted modules for VisProg to recognize and discover patterns to handle the task. We call this baseline VisProg variant. We also put the training examples in GENOME' stage into the prompt of VisProg variant for better performance. For ViperGPT, it has no manual modules and ask the LLMs to make use of the APIs to handle the instances. Thus, we manually write solutions for the training examples into the prompt of the ViperGPT to teach ViperGPT to handle the task. We call this approach ViperGPT variant. We have added such analysis into the revised paper. VisProg by itself needs a handcrafted solver module to find the target solution and it would be extremely difficult for ViperGPT to generate solver from scratch. Thus, add the solver module learnt from our GENOME model to pre-defined API pool of VisProg and ViperGPT. As shown in table 10, our GENOME model achieves better performance than these two baselines, showing the great value of module learning for handling new tasks from only few examples.
New Modules
GQA
VERIFY ATTRIBUTE
14.1
ATOOSBATTRIBUTE
10.8
VERIFY-COLOR VERIFY .COLRR
6.9
COMPARE.ATTRIBUTE
5.9
VERIFY .MAEEIIAL
3.6
Table Percentage of top-5 most-used new modules GQA.
More Details about New Modules. We further take GQA as an example to exhibit the percentage of top-5 most-used new modules in GQA in the Table 11. The data in Table 11 shows the proportion of the five most common new modules appearing in the generated high-level programs. Overall, 38.7% of all generated high-level programs the newly generated modules (this 38.7% calculation includes other less common modules and excludes duplicate samples, such as a single high-level program containing multiple new modules). From these results, it can be seen that these newly learned modules can be widely applied to GQA, thereby helping GENOME achieve good results on GQA.
Additional Experiment on I-RAVEN. As independently demonstrated in (Hu et al., 2021) and (Spratley et al., 2020), the Raven dataset (Zhang et al., 2019a) exhibits flaws in its choice design, enabling models to learn shortcuts for solving the RPM reasoning task. Thus, we additionally con- duct the balanced I-RAVEN dataset (Hu et al., 2021) for further analysis. Following our setting in RAVEN (Zhang al., 2019a), we use 10 training samples for learning new modules to handle the task and test the model on the testing set. As shown table 12, our GENOME is still able to handle the abstract reasoning task with high accuracy and data efficiency.
17

---

Published as a conference paper at ICLR 2024
Methods
Center
L-R
U-D
LEN (Zheng al., 2019)
56.4
44.2
44.2
CoPINet (Zhang et al., 2019b)
54.4
51.9
52.5
SRAN (Hu et al., 2021)
78.2
70.1
70.3
GENOME
85.2
74.6
75.4
Table 12: Experiments for Raven's 's Progressive Matrices on I-RAVEN dataset (Hu et al., 2021).
Effectiveness of Module Learning. To better investigate the effectiveness of our GENOME, further two variants of our GENOME to show the effectiveness of module learning. GENOME w/o ML represents configuration without any new module learning but relies heavily on ViperGPT and VisProg-defines modules, directing the LLM to answer the question related to the image content and pinpoint region matching the referring expression. It strictly follows the function call style like VisProg and our GENOME. We also develop a variant of GENOME w/o ML v2 that allows LLM to call functions like ViperGPT with standard control flow and arbitrary Python logic. For resource constraints, we limited our experimentation to the randomly selected 800 examples from GQA and RefCOCO with gpt-3. 5-turbo-instruct API. As shown in table 13, our GENOME performs better than both two baselines across tasks, showing the effectiveness of the module learning.
Methods
GQA
RefCOCO
GENOME w/o
43.3
62.3
GENOME w/o ML v2
40.9
65.5
GENOME 45.9
67.1
Table 13: Ablation of module learning GQA and RefCOCO.
A.3 PROMPTS FOR EACH STAGE.
The ability of Our GENOME is from in-context learning of LLMs (Brown et al., 2020), when the prompts are keys tell what the LLM should generate. We show the exemplar prompts of our models to learn the VQA task in Figure 15-17.
A.4 DETAILS AND EXAMPLES OF THE NEW DATASETS.
To evaluate Knowledge Tagging, 50 tagging instructions are annotated on 50 internet images in- cluding personalities and a variety of objects such as logos, flowers, buildings, fruits and sports, among others. For each instruction, we manually annotated the ground truth bounding box and the associated tag. For image editing assessment, we collected 50 editing instructions on 50 images including personalities and various objects like foods, furniture, animals, utensils etc. 25 images are from the COCO dataset and the other 25 images are from the internet. For the image editing tasks, we ask three annotators to estimate whether the editing is correct or not. For the knowledge tagging task, consider the localization is correct if the detected region has IoU higher 0.5 with the ground-truth annotation. For text tagging, we compare the prediction with the annotated text with BERT matching (BEM) (Bulian et al., 2022). If the matching score higher than 0.5, we consider it a successful matching. More examples of the two datasets can be found at Figure 9 and Figure 10.
A.5 QUALITATIVE EXAMPLES.
In this subsection, we show the qualitative examples of the learned modules and qualitative cases of how they handle different tasks. We show an example of GENOME performs better than VisProg in Figure 7. At the top of Figure 7, our model effectively utilizes the COMPARE.COM COLOR module acquired from GQA to pinpoint the correct region, whereas VisProg fails to generate the correct program due to its rigid module library. Figure highlights emerging forms of compositionality and module re-usage. Notably, although the SOLVER module originally trained center-type
18

---

Published as a conference paper at ICLR 2024
Knowledge Tagging
Ours
CBUU
Program:
COMPARC COLOR
vagelaabes
Quesiion:
0620HLOC1 Lmugen Dttpp- codants vegatable")
(angeeraaanaeaa prAce, ok3ecb-0880, rategorleanstttee
vegetsblles
grapet)
IAAG::
0833 0833-820000 (esct 22ata-5823, 34ar2add022)
0824-NETT, of anccttoon rana- COMPARE, couoR!
segetabile
FINAL RESIIT
attiribute same")
reporee nat Lmagen (wweteakeeectooeeie
place
VisProg
Genarated Program:
canoo
hoer
rste- (aarge caanen incitss and vegetablles of the care colie 26 the grope (0se3k)
naseri bal leage feade rkderniden1)
FDN., NEaUL (iEEE-ETTTTTTtee-eeee
Figure 7: typical example of how our GENOME outperforms VisProg on knowledge tagging. In the top, model is able to make of the COMPARE.COLE module learned from GQA to localize the correct region while VisProg fail to generate the correct program with its fixed module library.
problems within the Raven dataset, it demonstrates inherent adaptability to other problem types, including left-right and up-down orientations.
New Learned Modules. We show the examplar new learned modules from the GQA and Ref- COCO in Figure 11-14. As shown in Figure 11, the new learned module THOOSE_ATTRIBBUT is able to use the LLM to retrieve relevant knowledge first and then adopt the image-text classifier match the attributes. In Figure 13-14, localize objects with spatial index.
that the new module SORT_ SPATIAL is able to
19

---

Published as a conference paper at ICLR 2024
Center
Up Down
O
deswor
Answor
Anseor
Generated Program:
Programt
Generated Program:
COLOGIOE TOCT LOUDH imagen
SPADED Ci-------PPP 2IACE, box-)009)
(naSetii ROP pitet bosesttont
bilcewot tect Jutc siger/AAtE
aoxa sLCCLLECE mage 29660, object- (1101"))
prAdesec 2aag--2-2 boo-80001)
#0R3+LOX Oc Cimoge Maae, dbjeect rOr")
ARDIS kedoo VERl image (800s, oolors a 2833
FIIAL REGULT RESUL RES.EEISSaa--a.... (addaa))
Soare and ECT SHaPe trades, inaget (MOSE1)
st2e-0te ic taagets saac leage 1-1a-661111
51zt DE2E-0ETECT imagem--eee Diocoo tmg.sa-----.
Suart-DSTeCT Suaae paaurr---nsseee inesee, traget- (mtre1)
naget Inaget
uddi
-OOLoR, ahapend sieansizz)
12 Resuk RES.LTHEESUUTTTE twwtssuTTyaarreeee..
FDMAL Jis. wwn.su (
Figure 8: New compositionality and module re-usage in the Raven dataset. While the SOLVER module was initially trained center-type problems in the Raven dataset, it exhibits natural transferability to other types, such as left-right and up-down problems.
Replace the spoon of the same material the spatula with knife
Create color pop of the first boat from front
Create color pop of the first child from the right
Replace the second pizza from the top with hamburger
Select the lamp with the same color as the one at the bottom and create color pop
Hide Tim Robbins with :) and Morgan Freeman with 8)
Figure 9: More examples of the new image edit dataset. The dataset asks models to edit images' fine-grained and regional details according diverse language instructions.
20

---

Published as a conference paper at ICLR 2024
Tag the famous landmark Europe in the bottom right
Tag the common bird of the same color as the ibis
Tag the second famous film director from the left
Tag the famous painting of the Louvre in the left
Tag the second dog from the right
Tag the second Nobel Laureate Physics from the left
Famous Scientists
Figure 10: More examples of the new knowledge tagging dataset. The dataset requires models to localize the target region and tag the region with the desired information.
21

---

Published as a conference paper at ICLR 2024
class CHOOSE ATTRIBUTE ()
Input:
image: image object
box: list of bounding boxes
object: a string
attributel: a string
attribute2: a string
Output:
10
result: a string
11
examples:
12
Question: Is the coat thick or thin?
(image- jbject/ccottt)
ANGWERO=CHOOSE ATTRIBUTE A (Ame box-8000 object='coat
attr:bbteel=tthii"""'tttetiii btt. att FINAL Srraa__ESSUL--RRSU1 (Yar-ANWEE
15
16
17
step_name CHOOSE
19
20
def
iiinii iiiii_nttllll (self):
21
print RegR stering Rnggiisterin
step')
def
expand box (self, img size, factoooo factor=1.5)
24
H img_size size
25
x1, y1, x2, = box
26
dw int (factor= (x2-x1) /2)
dh int (factor= [y2 y1) /2)
28
cx int ((x1 x2) 2)
29
cy int ((yl y2) 2)
30
xl max (0, cx dw)
x2 min (ex + dw, W)
y1 (0, cy dh)
33
y2 min (cy dh,
34
return [xl, yl, x2, x2,
35
def predict (self, i img, boxes, attr, attr2)
if 1en (boxes) box boxes
38
39
box = self. expand box (box, img.size)
out_img img = ing crop (box)
else:
42
out_img ing
43
prompt1 = f'Tell me the actributes when the (obj} is (attrl) in one sentence.
44
prompt2 = f'Tell the attributes when the (obj) is (attr2) in one sentence."
47
obi_descl API. (prompt1, "ept__general'
48
obj_desc2 3PI.gpt3 (prompt2, 'gpt3_generan
49
result1 = API.ciip (out iooottmnn___a_aaao obj_descil
result2 AFI.clip (out (out img, obi obs_desc2)
if result1 result2:
52
result attrl else:
53
result = attr2
return result
56
57
def exccuto (sclf, img, boxes, obj, attri,,trr(] result solf predict (img, boxes, obj, attrl, attr2) return result
58
Figure/ 11:
Exemplar generated module from the GQA dataset. This automatically constructed module can make use different APIs to compare attributes of an image region.
22

---

Published as a conference paper at ICLR 2024
class COMPARE COIORE
Input:
image: Image object
boxl: list of bounding boxes
box2: list of bounding boxes
objectl: a string
object2: a string
compare_type string
10
Output:
11
result: a string
12
def
expand_box (self, seelf,box, box, img_size, factor=1.5):
W, img_ size
15
x1, 14,1,,2 x2, box
16
dw int (factorv (x2-x1) /2)
17
dh int (factor> (y2 1yl-y111 /2)
int ((x1 x2) 2)
19
cy int ((yl y2) 2)
20
x1 max (0, cx dw)
21
x2 min (cx + dw, W)
yl max (0, сy dh)
y2 min (ey H)
24
return [xl, y2, def predict (self, img,
25
objl, obj2, sampare_types
26
if len (boxes1) > 0: boxl [0] box1 sel wxw.eeppdddbe.ee (box1, size)
29
out__mg: img. crop (boxl)
30
else:
out imm1 = img
if len (boxes2) box2 boxes2
33
34
box2 = self. cwa.ffeepnnddee box (box2, img. size)
35
img_ ing. crop (box2)
else:
out_ out_img? mg
38
colorl API vga (out imgl, E'What color the (objl)?")
39
color2 API vqa (out img2, E E'What color is the (obj2)?')
API.vqa
prompt f CaC the (colorl) be regarded the same color as' f' (coolr))) You should just reply yes or without any other words.
42
43
temp = API 3 (prompt, 'gpt3_gonoral")
44
if 'same' == compare_type:
if 'yes' in temp.lower
result 'yes'
47
elif 'no' result = 'no"
():
48
elif 'different' == compare_ttye type:
if 'yes" in temp. Lemer result 'no'
52
elif 'no' in temp. lower ():
53
result 'yes'
else:
if 'yes' in temp.lowed result 'yes'
56
S7
elif 'no' in lower. ():
58
result 'no'
return result
def execute (self,, img, boxesl, boxes2, objl, obj2, compare_typer
61
result self f. f...iriecctt (img, boxes1, 1, boxes2, objl, abj2, compare_type)
62
return result
Figure 12: Exemplar generated module from the GQA dataset.
23

---

Published as a conference paper at ICLR 2024
class (ERT_SPATAAL() ***
Select objects from the image that match the spatial location,
Objects are represented by the bounding boxes. Returns the bounding boxes that satisfie the condition.
Input:
image: raw PIL image
box list of unormalized bounding boxes
Location: the location can only be left, middle, right, top, bottom, front and behind
10
11
index: number for the rank the object
12
Output:
box: bounding box
Examples:
15
Question: second sandwich from the right on the bottom BCXLISTO=LO o00.:/11...00...aaaaa1////.11112
16
17
Loc. location-
'right', 1ndcx=2)
19
(XXL=SST==0RR__PAA111 otg.a//aar...aaannnooooooaaaaaar iocation= 'bottom! inndx=1)) PINAL_RRSUL.T-RESI (1 (var-8XX111)))
20
21
step def predict (self, img, box box list, Location, index)
'SORT "SATIAA SPATIAL
24
25
if index < C or index len
26
return []
if index =- C: return [box_list(o]]
29
if "front" in location or "behind" in location:
30
box box_depth_list. box _ex_list_sorted out_box_list [box_i[0] for box_ in box_list_sorted if "behind" in Location: out
(img, box_list:
sorted = sorted (box boox_depth___ee pipth_list, key=lambda x: x(1])
33
34
35
else:
if "left" in location;
box_list sorted (box_list, key-lambda x[0])
38
elif "right" location:
39
box_lis = sorted list, key=lambda x: x|2], reverse
=True)
elif "top" in location:
42
box_list sorted (box_list, key-lambda x[1])
43
elif "bottom" in location:
44
box_list = sorted (box list, key=lambda x: x[3], reverse -True)
else:
47
return []
48
if index > len (box_list):
return !]
out_box_ list bist [box List [index-1]
return out_box_list
52
def check_location (self, img, box, location) :
53
w, h = img.size
xl, yl, x2, y2 box
(xl
56
cy (yl + y2)
57
if 'loft' in location:
58
if ex w / 2: return False
Figure 13: Exemplar generated module from the RefCOCO dataset. The rest part of the code is in
Figure 14.
24

---

Published as a conference paper at ICLR 2024
elif 'right' in location:
if cx < W 2: return False
if 'top' location: if cy h 2: return False
elif 'bottom' in location:
if cy h 2:
return False return True
10
def parse depth (self, img, bis_) box box_depth_iist compute depths for front or background depth_map API ddepth for box in box_list:
parse_depth
13
14
15
yl, x2, y2 box
18
Septh_map np.array (depth map)
19
avg deg_depth np mpp.median (depth pappph_mppppl :x2, yl:y21) box dww.ppepp____.. hisehh_iisttto append (box, ) return list
22
23
def (sclf, img, box_list, location, index) return self, predict (img, box__iis,__ttobbbbll list, locat ion, index)
24
Figure 14: Exemplar generated module from the RefCOCO dataset. The former part of the code is in Figure 13. This generated module is able to localize objects based on their location in images and the depth of images.
25

---

Published as a conference paper at ICLR 2024
Pre- ere-defined Modules: class LOC :
Generate boxes of the object on the image.
Input:
image: an image object
object: an object string
Output:
box:
list of bounding boxes
Examples:
11
BOXC=1OC (image= objoct"''aacc
12
13
class COUNT ()
Count the number of boxes the list. Input:
16
17
box: a list of bounding boxes
Output:
number: number of boxes
20
Examples:
21
ANSWERO=COUNT (box=BOX1)
22
Suppose you are program expert. Given set of pre-defined modules, could you identify whether it is possible write program to get the answer to the question] If not, what new modules do we need?
25
26 Note that you can only use the below pre-defined modules: LOC, COUNT, CRCP
27
the purse to the left or to the right of the person? 30 Yes. The program is: BOXU=LOC (IA objjtt//roottttepeeeeoe IMAGE,
Questions
31
IMAGEO=CROP LEFTOF MMwaaage--MMMAAAA box-
(ix object= 'purse')
34
ANGWER0-COUNT (box- if (ANSWERO) 0 else 'right"")
35
36
P_NAL_REBULT-RESUUN (var-
Question: Which object is larger, the sphere the blue cube?
39 No. We need to make a new module "COMPARE_SI2A" SI2E" first. Here the header
40
of the class:
41
class COMPARE SIZE ():
***
43
Compare the size of two objects in the image. One object is identified by the first bounding box of box0 Another object is identified by the first bounding box of boxl
44
45
Input:
image: image object
48
box0: list of bounding boxes
49
boxl: a list of bounding boxes
50
Output:
flag: return Truc if first object is larger else False
52
Examples:
53
Question: Which object is larger, the sphere or the blue cube?
54
80X0=1.00 (image- "baaGG/tteeetttphhepsseee
80X1=1.00 www.ge-mmaaa.aaa.aa.e... obfeett bblue cube')
FLACO=COMPARE SIZE http:://wwM......../e/e///eeeaaa DOx0E0000 box0=BOXO, box1=BOX1)
57
"rS3VEE2=2VVVLLeexx====""""1" FINAL RESULT-RESUL (var=ANNWEE
(FLAGO) eise 'blue cube *))
58
59
Question: INSERT_MEW_QUESTTI
Figure 15: Prompt of module initialization (Stage 1) to propose the module candidates.
26

---

Published as a conference paper at ICLR 2024
Pre-defined APIS:
class API ():
def loc (cls, image: Image Image, obj_name: str) List [List [int]]: **"
Returns boxes of the specific object on the image.
Input:
image: image object
obj_namer text string
Output:
10
boxes: list of bounding boxes
Example:
[1261, 160, 525, 2991] API, 1cc (image, "camel
13
14
return boxes
15
def 3pt3 (cls, prompt: str) -> str:
17
Returns response from gpt3
18
Input:
19
prompt: prompt to gpt.3
Output:
response: gpt3 response
22
Example:
23
"joe biden" API. ("Who is the US President?")
24
return response
26
27
28
Given the above APIs, could you write a new module, Loc ()
class Loc ():
#**
Generate boxes of the object on the image.
32
Input:
33
img: an image object
obj_name: object string
Output:
36
selected boxes: list of bounding boxes
37
Examples:
88
BOX1=10C (1mage=1MAGE0, object=
40
Response:
41
class Loc ():
42
step_name '10C'
def execute (self, img, obj_name)
if "bi_name=="70P'
45
bboxes [self. [self.top_bbx (img)
46
elif obj_name== 'BOTT0M='''
47
bboxes [self.botton box (img))
elif obj 1
bboxes [self left_effoboox (img)
50
elif obj_name-- GHGHT
51
bboxes [self ftelf.right_bor box (img) ]
else:
bboxes self celf.predict (img, obj name)
54
return bboxes
55
56
def predict (solf, img, obj_name)
return API loc (img, obj_name)
59
vev the above APIs, could you write a new module, MODULE, NAME NODULR_HRADD_..
60
Figure 16: Prompt of module generation (Stage 2) to make a module based on the module's 's input and output.
27

---

Published as a conference paper at ICLR 2024
Think step by step to answer the question.
You can only use modules below:
LOC
COUNT
RVAL
RESULT
VER CORLPY_ATIN AYLTTRIIBUTE
VERIFY_COLD
VERIFY M
12
13
Question: Is the vehicle in the top of the image?
14
Program:
BOXO-LOC IMAGE, objjcttt
IMAGE.0-CROP (
17
BOX:-I.0 (1mamee http://AAtccjjjctaa_a_or.r objectt (vehicct"vehicee"))
18
ANSWERRO=COUNT (box-
it (ANSWERO) > else 'no"")
FINAL RESLLLETTTT---SSTTTOOA (Ya
21
22
Question: who carrying the umbrella7
23 Program:
1BX0=1OC (1000 (1mage---M,,,,,,,,,, bbap:///aoa..occcc-mmmmmmmaaaaaa
IMAGEO-CROP ( Box-BOX0)
26
(NNSEEx0-VVAAAimmmeemM........ ouestiiiiiiiccciiiiooooo Who is carrying the unbrella?')
27
28
Question: the towel and the box have different colors?
30
Program:
31
BOX0-1.0C (image- object="rowel"""
32
m1x:-1.000 (image- MAGG, (box))
ANSW:::0-CCOMPPRK___AA wtw.:///////tmeiiimmmaa
mage- MAGK, box box2-BOX1, objesttt'teee. towee
objject=/bbxx' http:///oo..attrrrrte==eooooooe question=QUESTION)
35
FINAL BEW (va
36
37 Question: Is the knife made of ceramic7
Program:
39
BOX0=LOC (image-IMAGB, objoct=
40
(irp:///EEEEEELAAAEAAaaaaaaaaaae a 'knife', "mmiffee.cuee:iion-0EEE:1111
box=BOXC, material-aarrrrreeaeaaae object=
41
ANSWER1-=SAAL
if (ANSWERO) else 'no"")
F_NAL_RESU1T-RESUUT1W (va-
44
45 Question: Is the coat thick or thing
46 Program:
HOX0-1.00 mimmageeMMGA.. "ojjcct''caat/
ANSW::0-CHOOSBL_ATTR BUTE (image-milimm MAGE, box-BOXO, object- 'coat 'coat/ attributel-
49
'thick', "hiiaa=*aazziioot2=="''1'''''''
50
(var-ANWEEE
53
Question: INSERTTTEBLOOESSTTTTT
54
Program:
Figure 17: Prompt of module execution (Stage 3) to parse programs for a new test case.
28


Reviews:
[{"review_id": "3Js79px0vh", "reviewer": "Reviewer_CDRY", "summary": "This paper relies on the abilities of Large Language Models, but in effect extends them by attempting to enable reuse of code, as opposed to generating code from scratch.  They focus on the visual domain, where tasks include asking questions which require both visual analysis and symbolic reasoning.  They demonstrate state-of-the-art performance on several tasks", "questions": "Please clarify exactly where you created novel algorithms or ideas.  If any of those ideas require more than iterative prompting, please state so explicitly.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "This paper makes a clear case for it's own contribution, and that contribution does appear to be valuable - visual learning is an important task, and the prohibitive cost of using SOTA large language models makes reuse of code appealing (although they explicitly don't use ChatGPT4 \"due to the prohibitive cost\", so maybe this is less of an argument than it would be otherwise).  The fact that they can show it's use on several domains and types of tasks is also appealing.", "weaknesses": "My main concern is that, based on the presentation, it seems that the authors took a lot of highly intricate API's for LLM's that large teams may have worked on and cobbled them together to solve a new task.  I refer to this section: \"The success of our GNSVR relies on a set of pre-defined modules and APIs as the starting point. We utilize handcrafted modules from VisProg (Gupta & Kembhavi, 2022) as our initial components. Additionally, we incorporate several new APIs from ViperGPT to enhance module creation. We also include some new APIs from ViperGPT (Sur´ıs et al., 2023) for making new modules.\"  I appreciate their novelty in how they use these API's, but the ratio of insights of these authors vs of the authors of the API's appears insignificant.\n\nI'm also not entirely convinced of the novelty of this paper.  I refer to \"Iterative Disambiguation: Towards LLM-Supported Programming and System Design\" (Pereira and Hartmann) and \"Self-planning Code Generation with Large Language Models\" (Jiang et al).  I don't think the fact that this is in the visual domain is enough to call it \"novel\", because there is virtually no engagement with visual modalities by the authors - as stated above, according to my understanding, they are using predefined modules which handle the interface between vision and language.\n\nIf I was given evidence against either of the two above claims - that all of the work (particularly the \"visual reasoning\" work) is being done by existing tools, or that the paper is not fundamentally using LLM's in a novel way - I would be happy to increase my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vlzGzJslie", "reviewer": "Reviewer_QVVg", "summary": "The authors introduce a Generative Neuro-symbolic Visual Reasoning Model (GNSVR) for vision-language tasks that involves creating, testing, and re-using neural \"modules\" i.e. neural network components that are steerable and solve well-defined abstraction and reasoning tasks. To evaluate these modules, a pre-trained LLM model is used to first determine whethere a new module is needed given a reasoning training set. Then, the LLM is asked to generate the function signature of the new module as well as an execution (reasoning) script to solve the given reasoning problem. The signature is used to generate a new module, which is evaluated against few samples to ascertain whether it is the right component. Once a generated module satisfies this criteria, it is added to the library of modules and the process is repeated. The empirical results show that GNSVR is able to develop novel reasoning modules, and utilize them successfully for a variety of vision language reasoning tasks.", "questions": "* Could the authors comment on why they chose to go with a neuro-symbolic approach versus a purely neural approach for defining the neural module scripts? Is it only for controllability, and if so can they comment on how a neural script (e.g. an LLM that abstracts the problem of generating the script and executing it) would compare? There have been approaches towards learning neural scripts and executing them dynamically at inference time for reasoning tasks in prior work e.g. [1]\n\n* I think the module initialization and evaluation during generation in GNSVR is closely related to the research on automatic group discovery and using it for model design paradigm introduced in computer vision previously e.g. [2, 3]. It would make the related works section more comprehensive to include discussion on how GNSVR relates to this growing subfield of automatic evaluation and model design.\n\n\n**References**  \n\n1. Rahaman, N., Gondal, M.W., Joshi, S., Gehler, P., Bengio, Y., Locatello, F. and Schölkopf, B., 2021. Dynamic inference with neural interpreters. Advances in Neural Information Processing Systems, 34, pp.10985-10998.\n2. Vendrow, J., Jain, S., Engstrom, L. and Madry, A., 2023. Dataset interfaces: Diagnosing model failures using controllable counterfactual generation. arXiv preprint arXiv:2302.07865.\n3. Gao, I., Ilharco, G., Lundberg, S. and Ribeiro, M.T., 2023. Adaptive testing of computer vision models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4003-4014).", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Firstly, the paper is very well written and the GNSVR method is very well explained, with a healthy split between using the main paper and supplementary for splitting the key information versus additional information.\n\nThe empirical performance of GNSVR for transfer tasks, as well as the examples of the modules generated is quite impressive - it would justify the overall approach to growing modules for reasoning and their effectiveness for few-shot  generalization.", "weaknesses": "There are several components to the GNSVR framework but the paper provides no detailed analysis in the main paper of the importance of each of the components of the GNSVR framework.  While I think the framework overall is useful, the components are not all equally important to solve the reasoning problem and hence it is important to understand for future research on modular reasoning to understand what works and what doesn't, and if so why not.\n\nHow big of a role does \"good initlialization\" of the neural module operators plays?How important is defining the correct input and output format for a new module? How important is the selection of the few shot samples to evaluate a new module? (the authors say \"We extracted 300 examples from GQA, 100 from RefCOCO, 10 from Raven, and 10 from MEWL based on experimental experience.\" - is the experience here just cherry picking for results or something else?) How important is the LLM capability to learn new modules? What role does the prompt play for the LLM in evaluating existing modules and creating new ones? Without detailed analysis to support answers to all these questions, the paper is limited in terms of explaining the method beyong just presenting a new method and showing empirical results.\n\nLastly, I would not suggest the authors not report results on the RAVEN dataset. As shown independently in [1] and [2] the dataset contains flaws in choice design which enables models to learn shortcuts to solve the RPM reasoning task. I would recommend the authors use the i-RAVEN dataset inroduced in [1] instead.\n\n**References** \n\n1. Hu, S., Ma, Y., Liu, X., Wei, Y. and Bai, S., 2021, May. Stratified rule-aware network for abstract visual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 2, pp. 1567-1574).\n2. Spratley, S., Ehinger, K. and Miller, T., 2020. A closer look at generalisation in raven. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16 (pp. 601-616). Springer International Publishing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4iIhr5iJO1", "reviewer": "Reviewer_vHZ5", "summary": "This paper proposes GNSVR, a neuro-symbolic system that grows a library of modules for visual reasoning tasks through the use of LLMs. It consumes a small set of example training tasks, and generates new modules with a two-step process. In step 1, a LLM is queried to see if an example task can be solved given functions from a base API, if the LLM responds negatively, then it is instructed to propose a new module to add into the API that would help solve this task, as well as to produce an I/O example for how this module would be used to solve the task. In step 2, given doc-string descriptions of these would-be useful modules, an LLM is prompted to author a python function, potentially referencing base-API logic, that matches the description and corresponds to its previously generated I/O example. At test-time, these modules are integrated into the API, and a LLM uses API calls to produce a program that can reason over visual input matching the logic of the input question. The system is primarily evaluated against other neuro-symbolic systems for visual reasoning that operate with a fixed API (VisProg, ViperGPT). Compared to the previous systems, GNSVR offers competitive performance on standard benchmarks (GQA, RefCOCO), and demonstrates promising results that its discovered modules can help generalize to related tasks, like image editing and knowledge tagging.", "questions": "While the broad-stroke description of the method is clearly presented, many of the important details were left unspecified (making reproducibility challenging without a reference implementation). To improve clarity, and help understanding of the results, I think the below questions would be important to answer in future revisions:\n\n# 1. Module Generation \n\n(a) Can new modules be hierarchical (e.g. can one proposed module referenced a previously proposed module), or can they only call out to the original API functions?\n\n(b) The error-correction and pass-rate logic for the module generation step are not clear. What is the \"pass-rate\" at which a function is accepted as a new module, from the listed examples it seems like a single input/output pair is used, so is the pass rate 1? \n\n(c) What exactly is the error-correcting mechanism when one of the authored functions produces an error -- is it sent back through the LLM with a modified prompt? How many times? What constitutes an error? This seems like a potentially important part of the contribution, so I would encourage the authors to even consider adding an ablation condition demonstrating this is helpful for the performance of the system.\n\n(d) It would be informative to provide additional details on the generated modules for each task: how many are created for each task? How often does each one get used in solving test-cases? Do they each capture distinct concepts, or are some modules duplicates?\n\n# 2. In-context examples\n\nI don't believe there is any information into how in-context examples are chosen, which can greatly affect LLM performance.\n\n(a)  From the example prompts, it looks like a single function is given as an in-context example for step 1 and step 2, are these always COMPARE_SIZE and LOC, as shown in figures 15 and 16? \n\n(b) For the inference in-context examples are these chosen randomly from the training tasks that found a successful program? \n\n# 3. MISC \n\n(a) For the Raven task, my understanding is that the input is just a series of images. If this understanding is correct, how do you turn these images into a question? I am also quite curious to see the internals of the SOLVER generated module, is this module shared between Raven and MEWL, or does it employ distinct logic?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In general, I think the direction of the proposed method is sound and compelling; systems like VisProg and ViperGPT provide promising ways to push neuro-symbolic reasoning past toy-domains, but they are limited by fixed APIs. Using LLMs to augment these APIs in a task-specific manner is an interesting idea with many potential avenues for future exploration.  \n\nFrom a methodological stand-point, I think the division of labor between the module initialization step and the module generation step is a neat insight. This design decision allows the LLM to create it's own tests, that can then be used to help guarantee that LLM generated functions 'do what they are supposed to'. \n\nThe experimental results are largely positive for the proposed method, although I have some concerns about their design that I'll detail below. That said, I think the experimental results do support the claim that the proposed system offers improvements over both VisProg and ViperGPT across tasks. Perhaps the most compelling claim is that the modules that GNSVR finds from the GQA and RefCOCO tasks can \"transfer\" for related tasks of image editing and knowledge tagging, supported by the fact that GNSVR significantly outperforms VisProg in this setting. The evaluations against fully-supervised methods on the Raven and MEWL tasks is also fairly impressive, considering GNSVR is operating in a few-shot learning paradigm.", "weaknesses": "# Main Concern\n\nFrom my perspective, the biggest current weakness of the paper is that from the experimental design its hard to parse out exactly how the discovered modules affect the system's performance. Ostensibly, this can be gleaned from comparisons between GNSVR and VisProg/ViperGPT, but there are more differences between these systems beyond merging in discovered modules. Specifically, GNSVR uses a \"base API\" that is a combination of VisProg and ViperGPT, so the \"fair\" comparison would be against an ablated version of GNSVR that removes steps 1 and 2, and just tries to solve test-problems with the original API functions. This condition is considered in the ablation experiment (GNSVR w/o ML), but only a subset of the RefCOCO test-set. To solidify the claim that the improvement GNSVR observes stems from its discovered modules, this base condition should be added to all of the experimental set-ups (tables 1-5), for example, from Table 2 its unclear how much of the delta improvement between VisProg and GNSVR can be attributed to improvements in the base API versus improvements to the API from the new modules. \n\nBeyond this, I'm also slightly concerned about the design of the GNSVR w/o ML baseline. At inference time, is this baseline allowed to invoke arbitrary python logic in the style of ViperGPT (e.g. standard control flow constructs) or is it restricted to *only* using API function calls in the style of VisProg. I would imagine that the first condition would be more fair to evaluate GNSVR. Solving some tasks might require simple logic that the LLM knows how to express in python, but might not be directly expressible with a series of API calls. In GNSVR, this logic is incorporated into modules, but in the baseline the LLM should still have the opportunity to invoke similar logic in its test-time solutions (otherwise its impossible to properly evaluate the usefulness of the discovered modules). Please clarify which of these modes the baseline is operating in. \n\n# Minor\n\nCompared to VisProg and ViperGPT this system seems to require more training data, as the I/O pairs are not only used to populate in-context examples, but also impact (i) what module concepts are proposed and (ii) how the correctness of each module concept is evaluated. This point about reliance on training data is touched on in the ablation section, but it would be good to make this distinction explicit when comparing the pros/cons of the proposed system against past work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lSWl1eYZmF", "reviewer": "Reviewer_JpcA", "summary": "The authors tackle the problem of prior visual programming methods exhaustively generating entire code snippets for each new task. Instead, they proposed GNSVR to grow and re-use a library of modules. GNSVR first initializes new modules based on a train set, then evaluates the new modules based on additional few-shot training examples, and finally it executes the modules on the full test set. It shows comparable performance on visual reasoning tasks, can transfer modules to new tasks, and adapt to new visual reasoning tasks with few training examples.", "questions": "Q1. Please clarify the selection of training examples used in stage 1 and 2. \n\nQ2.  “If a program encounters errors during execution, we incorporate the error information into the LLM’s prompt and instruct it to rectify these issues.” How is this done?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I am appreciative of the idea of generating more modular and composable modules that are verified, to be used in a library of skills for future tasks. I think the idea of continuously growing this library is innovative, and the method simple and elegant.", "weaknesses": "W1. Is there anything to prevent overfitting to the small train set of a given task, and only creating specific modules tailored for that domain? I can imagine that these overfitted modules may not be very useful in new tasks. \n\nW2. Isn’t it possible that the method creates a broad function signature, but during stage 2 verification with the small set of train examples, it overfits to a bad implementation that only works for those examples, and therefore actually harms performance from there on out? I’m mainly concerned about the above two overfitting challenges.\n\nW3. There seems to be an assumption that the queries in the train set actually all require similar modules, for example, to verify the new modules, the selected set of test cases from the train set must actually use those modules. I’m not sure if this is a reasonable assumption, and also, how is this sampling of train examples (both in stage 1 and stage 2) done?\n\nW4. In the experiments, are Visprog/Viper also given the same few-shot training set? For example, I believe you can use the same method of error correction and correct both Visprog/Viper when it is incorrect. In this way, you can include Visprog/Viper comparison in Tables 3 and 4. Would be great to better disentangle what drives this improvement of performance -- modularity in the proposed modules, or the training examples given in the loop, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)