Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmers
Authors: Aleksandar Stanić
Sergi Caelles
Michael Tschannen
Venue: Accepted by TMLR ()

Abstract: Visual reasoning is dominated by end-to-end neural networks scaled to billions of model parameters and training examples. However, even the largest models struggle with compositional reasoning, generalization, fine-grained spatial and temporal reasoning, and counting. Visual reasoning with large language models (LLMs) as controllers can, in principle, address these limitations by decomposing the task and solving subtasks by orchestrating a set of (visual) tools. Recently, these models achieved great performance on tasks such as compositional visual question answering, visual grounding, and video temporal reasoning. Nevertheless, in their current form, these models heavily rely on human engineering of in-context examples in the prompt, which are often dataset- and task-specific and require significant labor by highly skilled programmers. In this work, we present a framework that mitigates these issues by introducing spatially and temporally abstract routines and by leveraging a small number of labeled examples to automatically generate in-context examples, thereby avoiding human-created in-context examples. On a number of visual reasoning tasks, we show that our framework leads to consistent gains in performance, makes LLMs as controllers setup more robust, and removes the need for human engineering of in-context examples.

Published in Transactions on Machine Learning Research (04/2024)
Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmers
Aleksandar Stanić Google Mind
alenstanic
Sergi Caelles Google Research
scnelles@ggoogle.cmm
Michael Tschannen Google Mend
tschannen wwg.oell..ggg.e....ooo com com
Reviewed on OpenReview: https: // openreview. net/ forum? id= gVVs tX
Abstract
Visual reasoning dominated by end-to-end neural nelworks scaled to billions of model parameters and training examples. However, even the largest models struggle with composi- tional reasoning, generalization, fine-grained spatial and temporal reasoning, and counting. Visual reasoning with large language models (LLMs) as controllers can, in principle, address these limitations by decomposing the task and solving subtasks by orchestrating a set of (vi- sual) tools. Recently, these models achieved great performance on tasks such as compositional visual question answering, visual grounding, and video temporal reasoning. Nevertheless, in their current form, these models heavily rely on human engineering of in-context examples in the prompt, which are often dataset- and task-specific and require significant labor by highly skilled programmers. In this work, we present framework that mitágates these issues by introducing spatially and temporally abstract routines and by leveraging small number of labeled examples to automatically generate in-context examples, thereby avoiding human-created in-context examples. On a number of visual reasoning tasks, we show that our framework leads to consistent gains in performance, makes LLMs as controllers setup more robust, and removes the need for human engineering of in-context examples.
1 Introduction
Compositional visual question answering requires model to answer questions about visual content in a compositional manner, involving multiple steps of reasoning or considering relationships between different objects or entities within an image. Tt is a complex task as requires understanding both the visual information in an image and the structure of the question, and reasoning about how different visual elements relate to one another generate the correct answer. Recently, large progress has been made on many such vision and language tasks by scaling end-to-end neural networks models in terms of size, training data, and compute (Alayrac et al.)
2022 Chen et al., 2022b Yu et al.; 2022 Wang al. 2022a Gan et al. 2023b Driess et al. 2023 Chen et al. 2023d c). However, even the largest
2022 Lu et al. 2022 Li et al. state-of- ofet---rtttttttttto (SotA) models struggle in tasks that require compositional reasoning, ability to generalize,
fine-grained spatial reasoning capabilities, and counting (Bugliarello et al. 2023 Paiss et al. 2023 Hsieh
ct al. 2023 Yuksokgonul et al. 2022 Zhao et al.) 2022 Hendricks & Nematzadch 2021). An example of such Lask the following query: "Could the cookies on the table be equally distributed among children?" [Suris et al.) 2023). To solve this, the model needs to detect the cookies in the image, filter out the ones that are not the table, detect children, count cookies and children, and check if the cookics count is divisible
* Work completed during an internship Google Research.

---

Published in Transactions on Machine Learning Research (04/2024)
by the children count. Questions like these are difficult for current end-to-end vision and language models (VLMs). Scaling VLMs further makes them even more data- and compute-hungry Saatunggyyiiillloboon (Villalobos et al. 2022). so the scale alone seemns unlikely to solve these tasks, especially due to the exponentially-large long tail of compositional tasks.
On the other hand, it questionable whether solving compositional tasks with a single monolithic end-to-end neural network is the optimal approach. Intuitively, it might be easier to first decompose the task into several subtasks, individually solve the subtasks, and then use the intermediate results to solve the original task. This is reminiscent of the way humans approach compositional problems. According to Daniel Kahneman's framework (Kahneman 2017), our thought process can be thought of as consisting of two mechanisms: System and System 2. System 2 is the "slow", "analytical" system that can decompose the task into subtasks, while System 1 is the "fast", "reactive" system that solves individual tasks such as recognizing patterns. In machine learning, the early work on task decomposition was pioneered by Neural Module Networks (NMNs) [Andreas et al.) 2016 Johnson et al.) 2017 Hu et al. 2017). NMNs are trained end-to-end in the hope that every module will learn separate function that will be reusable across lasks. However, these models have a number of drawbacks, namely that the program generation requires hand-tuned parsers, they are difficult to optimize (sometimes requiring reinforcement learning), and they have the issuc of modulc "collapse", where some modules are never activated and others take over all the work, contrary to the design intentions.
Recently, an alternative approach based on "tool use" gained popularity (Cobbe et al. 2021 Komeili et al. 2021 Thoppilan et al. 2022 Parisi et al.] 2022; Zeng et al.j 2022 Gao et al. 2023a Qin et al.
2023 Zhuge
et al. 2023). In "tool use", an LLM solves a task by controlling (akin to System 2) a set of tools (such as an object detector, akin to System 1) (Zeng et al. 2022 Shen et al. 2023 Zhuge et al. 2023 In particular,
VisProg (Gupta & Kembhavi] 2023), ViperGPT (Surís et al., 2023). and CodeVQA (Subramanian et al., 2023) show great promise in solving visual question answering by using an LLM to generate a program (in Python or a custom scripting language). During execution, the program calls individual vision models (such as object detector, depth estimator) through an API that is provided in the prompt. For example, to answer "Which color is the jacket of the second person from the left?" (Figure la); the program needs to detect. people, sort them from left to right, select the second, detect their jacket, and query its color. These models achieved S on compositional visual question answering, visual grounding, and video temporal reasoning tasks. By their construction, they are interpretable, compositional, adaptable (tools can be upgraded on the fly), offer strong generalization mathematical, and reasoning skills, and do not require gradient-baser training. However, in their current form, they heavily rely on human engineering of in-context (program) examples (ICEs) in the prompt. Moreover, ICEs are datasct- and task-specific. To generate them, significant labor by highly skilled programmers required. For this reason, we argue that these methods should not be called zero-shot their current form.
In this work, we present framework that mitigates these issues, makes LLMs-as- proorammmrss setup more robust, and removes the need for human engineering of ICEs. Our framework, whose effectiveness we show across a number of compositional question-answering and vidco temporal reasoning tasks with ViperGPT (but is universally applicable to other approaches), consists of the following:
Firstly, instead of using a simple API with only basic routines that call individual tools, we introduce an "Abstract API". Abstract API consists of spatially and temporally abstract routines that remove the large burden on the LLM to have strong spatial and temporal reasoning.
Second, instcad of relying on a large number of dataset-specific (question, code)-pairs as ICEs, we introduce setup that generates ICEs automatically. Using a few labeled examples (that are significantly cheaper to obtain, e.g. via crowd-sourcingg, we generate query-code examples in a zero-shot manner and use these as ICEs. This mitigates the need for human engineering of ICEs.
Third, we show how LLMs as controllers for visual reasoning can (to some extent) perform "self- correction" through "self-debugging" and "self-tuning" without any ground truth labels. In "self- debugging", we generate new code when the previous fails to execute, either by providing the LLM previous query-code pair and execution error as feedback or from scratch. In "self-tuning", we show how the tool hyperparameters can be tuned automatically the execution fails due to a module.
2

---

Published in Transactions on Machine Learning Research (04/2024)
Query
Secoad
Visual code
Python API
Visual tools
Saxtore 2 Soet eccace)
Abstract routines
Code LLM
ICEs
Few-shot Q, A
Self-tuning
Answer XXL,X2
ACEs
(a)
(b)
Figure 1: (a) RefCOCO Yu et al. 2016) example image. (b) A code-generating LLM takes as input the query, the Python API (functions for "tool use" and Abstract API routines (functions) we introduce Section 2.2h and a number of TCEs (we replace human-engineered TCEs by automaticalll--eeneaaa ACFs
Soction 2.3). The LLM generates code that takes as input the image and outputs an answer (here bounding box). If code fails to run, "self-tuning" (Section 2.4) can adjust parameters and generate new code.
2 LLMs programmers for visual reasoning framework
In this section, we first provide brief overview of the ViperGPT approach (Surís et al. 2023) on top of which we show the utility of our framework. We then describe each component our framework, namely the Abstract API, automatic generation of ICEs, and self-correction.
2.1 Background
ViperGPP takes as input an image or a video and a textual query, The textual query is fed into an LLM (Codex (Chen et al.) 2021)), together with the tools API and ICEs. The LLM generates program that solves the given query using the tools without further training. The information in the prompt is crucial for good ViperGPT performance, as it is the only task-specific information provided. The prompt consists of a Python AP1 with the necessary functions to solve the visual query, such as object detection, depth estimation, and language model qucrics. Additionally, ViperGPT uses several datasct-specific ICEs the prompt. As we show (Section 3, performance depends heavily on these human-engineered examples.
ViperGPT AP1 defines an ImagePatch and a VideoSegment class that contain image and video processing functions. Each function calls pretrained model to compute the result. The API in the prompt does not contain function implementations, but it contains docstrings and query-code examples of their use. The ViperGPT API defines the following functions: find takes as input an image and a textual query, calls an open vocabulary detector and returns a list of image patches of detected objects; exists takes as input an image and a textual query and returns true if the query object exists the image, otherwise false; perefftyrropety takes as input an image, a noun representing an object and an attribute property to verify and returns a boolean whether the object has this property; best image match that takes as input list of image patches and textual query and returns the image patch that best matches the query; best_ past_text_mt teteteeatcchheeeeeeetttttttttete match that takes input a list of queries and one image, and returns the query that best matches the image; compute_depe that computes the median depth of an image or image patch; distance which computes the pixcl-distance between two images; simple_quest query which handles ual queries that are not. decomposable, by calling an image captioning model; relecc_ansvee that given a context text describing a scene and a list possible answers queries an LLM select the correct answer. The VideoSegment class does not contain any functions that call individual models, but only the start and end point of the video segment and an iterator over the frames, which returns ImagePatch object. For the full ViperGPT API, Appendix A.4
3

---

Published in Transactions on Machine Learning Research (04/2024)
The codo-generating LLM outputs code that attempts to solve the query. This code is executed, taking input an image or a video (and optionally a list of possible answers) and outputs result (e.g. a bounding box or a string). Due to generating programs in native Python code, ViperGPT avoids the need for custom interpreters and can leverage Python built-in functions (e.g. sort, if/else control flows, math functions, etc.).
2.2 Abstract API through visual routines
When programming, we continuously build new layers of abstraction. grouping them together into new functions. By abstracting away the implementation details, we reduce the cognitive load on the programmer which is then able to build systems of increased complexity. Motivated by this, we introduco a set of spatially and temporally abstract functions (routines D that abstract away a number of lines of code for the same functionality and together make the Abstract API. From a practical perspective, we are motivated by a number of failure cases observed in the experiments (see Seccion 3/. As presented in Secctio 2.1 ViperGPT's API is fairly simple (contains almost exclusively functions to call pretrained models). Although simplicity is good and often desirable, in the case of visual reasoning with LLMs as programmers, the lack of expressive visual routines requires the code-generating LLM to have strong spatial and temporal reasoning capabilities. Qualitative analysis showed that this is often not the case and that the current LLMs are not yet perfect in these terms (e.g. they confuse left and right, lop and bollom). For example, for the query "return the second person from the right", the program generated by the LLM correctly sorts the persons along the horizontal axis but then wrongly takes the second index in the array (instead of the second last). Similarly, they sometimes "confuse" temporal order, e.g., if "before" event means smaller or a larger time index.
For these reasons, we introduce a set of spatially and temporally abstract routines. We add the following spatial routines: get_patch_leff of, hts__atcc_rright____ of, get_patch_abon of, get_patch, below of for relational retrieval relative to a patch; get__pacch_cooeest_teee ttt__nnhhr_ooojeee that sorts patches by their distance to an anchor object and returns the one with the smallest distance; sort_patches, to sort the list of patches along hor-
Lofft_to_rrr torright,
sort_ sort_patches_ satrs_bottttttttte cooo_mttttop.._ and triidde_ppateee to get the middle patch from given list of image
izontal, vertical or depth axis; patches; For videos, add temporal routines for event "localization": before__veere
viiiioieeeeeeee of _eveentt_o_event,, event, get_vides wwwpssssegaeaeeeeeeeeeeee after__eeettttt__ee____aaesaaaae event, and routines to cither caption a get_ caption_video video or answer a simple question about the video: simple_ _uuurruerr..... The routines that we video: caption_ introduce are general in the sense that they are not specific to any individual task or dataset. This facilitates their reuse across tasks and avoids engineering task and dataset-specific routines. It is an open research question what the "optimal" set of primitive routines is. Another exciting research direction is using LLM with their own abstract routines, then reusing those to come up with even more abstract routines and so on. We leave these for future work.
2.3 Automatic generation of in-context examples
In-context examples (ICEs) greatly influence the performance of LLMs (Brown et al.] 2020 Chen al.) 2023b). For example, ViperGPT (Surís et al. 2023) uses between 10 and 16 hand-engineered dataset-specifie query-code ICEs per dataset. Similarly, VisProg (Gupta & Kembhavi 2023 uses between 16 and 31 ICEs and CodeVQA (Subramanian et al. 2023) about 50 ICEs. However, constructing these ICEs requires heavy human engineering, as they might need to be rewritten in a way that the LLM can use them to "generalize" to other examples across the dataset. Furthermore, the constructed examples are specific not only to the dataset but also to the LLM and the AP1, If any of those changes, they need to be written from scratch. Finally, to write good query-code ICEs, highly skilled labor is required, idcally somcone familiar with the workings of LLMs and good grasp of Python programming.
In our work, we move beyond this need for human engineering of query-code ICEs. We start from a small set of labeled examples (c.g. 16 saang--uestion-answwe tuples), as is common in few-shot transfer learning (Zhai et al. 2019 Kolesnikov et al. 2020). We run our framework in a zero-shot manner (without any ICFs) on these few-shot examples, sort the results by accuracy, select the best-performing programs, pair them with the corresponding queries, and use them as ICEs during test time. We call such ICEs automatically generated
1 Note that our "routines" do not correspond to the visual routines of Ulman (1987 such as tracing scanning.
4

---

Published in Transactions on Machine Learning Research (04/2024)
in-contert examples (ACEs). Importantly, no gradient-bused optimization is performed the few-shot ezamples. Intuitively, this works since even if the L.M does not always generate a program that solves the task correctly, it might sometimes come up with a correct program. Since retrieval is often easier than generating programs from scratch, the rcuse of the correct programs improves performance on the test set.
ACEs provide a number of benefits over manually writing TCFs. First of all, ACFs are much cheaper to obtain as they do not require highly skilled labor to write them. Second, the algorithm that generates ACEs is general: it is neither specific to the API nor the LLM. If any of these changes, ACEs can be casily generated by re-rumning the algorithm. Furthermore, they can be seen as first step of the LLM "coming up" with its own abstract rules and thus creating a "rulebook" (discussed in Section 2.2). Finally, few-shot (image, question, answer)-labeled examples are often available in datasets typically used in machine learning. If not available, they are cheap to obtain via crowd-sourcing and can be reused for further studies as a benchmark.
2.4 Self-correction
One of the advantages of solving visual reasoning tasks with LLMs as programmers is that we know when code fails to exccute. The failure can happen, c.g. due to compilation error (e.g. due to hallucination), some of the models failing, wrong return type (e.g. a bounding- box is expected, but code returns a string). Note that to detect these types of errors, no ground-truth labels are needed.
Self-debugging. If the code execution fails, we can query the code geteeeeenn LLM to correct the previously generated code. We do this by feeding back the query, the previously generated code, and the resulting error in the prompt (see the feedback template in Appendix A.3 Moreover, if the LLM's sampling temperature is higher than zero, we can query the model with a different random seed to generate new code from scratch. There are advantages to both of these approaches. If the rode-genng LLM has good "self-correction" abilities, then it should be able to correct its own mistakes based on the feedback, as we humans could. However, if the LLM is not good at self-correctoon does not know how to incorporate such feedback (e.g. if the LLM is not trained to be conversational), then feeding back the previous query and code will only "bias" the model to output the same solution. In that case, generating new code from scratch could work better.
Self-tuning. In some cases, we know that code exccution failed due to some components of a particular module. For example, the open vocabulary detector fails due to too high threshold hyperparameter. When the threshold is high, we have higher number of false negatives. For such cases, we propose to automatically change the hyperparameter of the module (e.g. reduce the threshold) and execute code again. Although here we experiment with only a single option of tuning such hyperparameters, the idea is applicable to any model that involves such "sensitivity" hyperparameters. For example, another such candidate could be the threshold that determines the "similarity" of the CLIP-style imago-text embedding model. The task of this module is to filter out a particular instance (e-g. "blue ball") from a set of instances ("balls"), e-g., based on attribute ("blue"). It does so by comparing the text embedding of "blue" with the image embeddings of all "ball" image patches in the image. Then it returns all image patches whose embeddings have similarity with the text embedding higher than the threshold value. Here we would similarly tune the threshold in case this module fails.
3 Experiments
Tasks. We evaluate our method on four datasets: RefCOCO, ReiCOCO+ Yu et al., 2016), & MManiing 2019 and NExT-QA (Xiao et al. 2021 used in previous work (urrs et al. 2023 These datasets evaluate diverse set of capabilities, namely visual grounding (RefCOCO, RefCOCO+), compositional image question answering (GQA), and video temporal rcasoning (NEXT-QA). In RefCOCO (example in Figure Ia), the task is lo detect a bounding box of an object given its natural language description ("referring expression"). In compositional question answering in GQA, the task is to answer in natural language a compositional natural language query. We use the "test-dev" split of the GQA datasot, as in ViperGPT. In NEXT-QA, the task is to answer a temporally compositional question by selecting one of the given multiple choice options. As in ViperGPT, we use NExT-QA "hard" split Buch et al. (2022). For RefCOCO and
GQA (Hudson
5

---

Published in Transactions on Machine Learning Research (04/2024)
Table 1: Comparison of our method against previous end-to-end and "LLMs as controllers" SotA methods.
For "Ours (code=bison)", we report mean scores = standard deviation across three random sceds. The
reference numbers for SotA on each dataset are taken from the following publications: RefCOCO: ZS (Yang
ct a
2023b), FS Yao et al.) 2021 Sup WWang et al. 2022b):
RefCOCO+: ZS (Yang et al. 2023b), FS
Yao et al. 2021 1 Sup (Wang et al.
2022b;; ; GQA: ZS (Li et al.
2023b FS (Jin et al. 2021] 1, Sup (gguyen
et al. 2022 NExT-QA: ZS
(Chen et al. 2023d), FS (Chen et al.
2023d), Sup (ee et al.
2023a
Model
RefCOCO (loU)
RefCOCO+ (IoU)
GQA (acc.)
NExT-Q2 QAA (acc.)
Zero-Shot (ZS) SotA
53.0
57.5
44.7
38.3
Few-Shot (FS) SotA
53.3
52.5
35.7
38.3
Supervised (Sup) SotA
94.0
91.7
72.1
63.1
ViperGPT (paper)
72.0
67.0
48.1
60.0
ViperGPT (GitHub (GH) ZS)
46.7
ViperGPT (GH w/ DS-ICEs)
60.5
E2E (SS OWT.w//Pa.I-)
33.5
31.7
40.1
58.9
E2E LLM-only baseline
53.3
Ours (code-bison, Zero-Shot)
44.4 0.9
38.2 + 0.0
32.1 + 0.4
60.2 0.3
Ours (code-bison)
51.2 0.2
45.7 + 0.1
33.4 # 0.2
61.0 0.1
RefCOCO+, methods are evaluated in terms of Intersection over Union (IoU) between the detected and the ground truth bounding box and for GQA and NExT-QA in terms of accuracy of the predicted answer.
Vision and Language Models. For code generation, use code instruction-tuned version of PaLM 2 Anil et al. 2023) code-bison accessible via the Google Cloud APT (Google 2023). We use the same model to select an answer for the multichoice questions in the NExT-QA dataset. Vision models we use are OWLv2 2023) for lext-image comparison, MiDaS 2023d) for image captioning and answering visual queries. Note that all models are different from the models that ViperGPT used (see Appendix A.2.
Minderer el al. 2023) for object detection, SigLiT (Zhai et al.
Ranftl et al. 2020) for depth estimation, and PaLI-3 (Chen et al.
Baselines. Strong baselines are essential for correctly measuring progress in machine learning. This is especially true in the emerging area of "tool use" (Cobbe et al.] 2021 Komeili et al.] 2021 Thoppilan et al.)
2022 Parisi et al. 2022 Zeng et al. 2022: Gao et al. 2023a Qin et al. 2023 Zhuge et al. 2023). When using an LLM and other pre-trained models, we must be careful to report the exact LLM version and/or API when it was accessed, and ideally report results over several random seeds to measure the statistical significance. In Table 1. we provide an overview of the previous Zero-Shot (ZS), Few-Shot (FS), and Supervised (Sup) SotA methods, ViperGPT, end-to-end (E2E) baselines, and our results all datasets we used for evaluation.
Early in the project, we found it difficult to reproduce the results reported in the ViperGPTT paper. Our first hypothesis is that this is due to differences in the vision and language models we use compared to the ViperGPT paper. However, when running the original ViperGPT code from the official GitHub repository on RefCOCO, we were only able achieve an ToU of 60.5 as opposed to 72.0 reported the paper. Note, however, that ViperGPT uses Codex, which is discontinued, so we use GPT-3.5-turbo (OpenAl 2023). Also note that this score was obtained using 16 dataset-specife ICEs (DS-ICEs). These examples contain large amounts of dataset-specific human-engineered information, such as "clothing requires returning the person". In the case of truly Zero-Shot learning (without any human-cngincerdd ICEs), the IoU score of ViperGPT's official GitHub code drops by 14 points to 46.7. Moreover, in their GitHub code we found engineengee improvements: if returning a bounding box fails, then return an "average" bounding box, and code exccution fails on GQA, then query the image captioner (BLIP-2 (Li al.) (023b)). These code changes lead to improved results, but make it hard to quantify the true power of LLMs as controllers approach.
6

---

Published in Transactions on Machine Learning Research (04/2024)
Ref00C0
RefCoCO+
OQA
NEXT-QA
пo ACE
no ACE
60
PO
48
42
34
Aceuracy
32
Annuracy
50
44
38
30
40
28
40
34
26
VparGPP
Abstract
VpprGPT
Abstract
VpeGGPT
VppeGPPT
Abstract
API
API
API
API
Figure 2: Using our Abstract APT improves performance over the ViperGPT API across all datasets. Similarly, ACEs consistently improve performance, and these gains compound with the gains from the Abstract API. Uncertainty bars represent standard deviations computed over three random seeds.
we also provide Zero-Shot end-to-end baselines. On RefCOCO and RefCOCO+, we feed the query as input the OWLv2 model and return its output. For the GQA end-to-end bascline, we qucry the PaLI-3 model (which was fine-tuned for multi-task inference on different captioning and VQA data sets, but not on GQA). For NExT-QA, provide two baselines. For the first baseline, we subsample and caption with PaLI-3 one frame per second, and then feed all these captions to LLM (code-bi bio)o together with the question and multiple choice answers. As the second baseline, we simply feed the LLM the question and the possible answers and ask it to choose one. LLM-only baseline achieves an accuracy of 53.3%, which is only 5.4% lower than the baseline that also gets vidcos as imput and significantly above the chance level accuracy of 20% (since there are possible inultcchoice answers to each question). This suggests that NExT-QA TAQ might. not fully evaluate the vision properties of the model and that new datasets are needed; for example, the Perception Test (Päträuccan al. 2023)
In Table I
created specifically to avoid such problems.
Lastly, in Table we report the Zero-Shot performance in selup, well results for our best performing model variant (averaged over three random seeds). In the following sections, we evaluate each component, as well as their combinations, to obtain the reported results. Using resulted in a slight drop in performance, but as we shall see below, the same conclusions hold for both code-bison and GPT-3.5-turbo for all our suggested improvements. In the following, we present the components of our framework. For an ablation study, see Table that shows that each component contributes positively to the final scores on each dataset.
instead of code-bison
3.1 Zero-Shot through spatially and temporally Abstract API
In Figure 2 we show the effect of using our Abstract API instead the API used ViperGPT. The API for RefCOCO, RefCOCO+, and GQA uses the same set of image routines (see Appendix A.4,, whereas the API for NExT-QA uses only video-specific (temporal) routines. For now, focus the brown bars in Figure 2 and compare the ViperGPT API and our Abstract API. We both with and without ACEs across all datasets. The performance gain is most notable for the NExT-QA dataset when ACEs are not used. We suspect that this is due to the LLM's difficulty in reasoning about the temporal order events. This confirms our hypothesis that building a more abstract API such that the LLM does not need to use low-level Python routines is a promising direction.
that our Abstract API leads to gains
Finally, we investigate whether our conclusions also hold for other LLMs, namely OpenAl's On RefCOCO (PT-3.5-turrl achieves an loU of 28.9 with the ViperGPT API and 39.8 with our Abstract API and an accuracy of 9.4 and 42.9 for the ViperGPT API and our Abstract AP1, respectively, This confirms that our Abstract API brings gains not only for code-bison, but also for other LLMs. For cach sample in the evaluation, we allow only one trial of code generation (no self-correction). Compared to the results with code-bison, IoU 37.7 and 40.5 on RefCOCO and accuracy 11.5 and 46.1 on NExT-QA for the ViperGPT API and our Abstract API, respectively, the results with GPT-3. 5-turbo are slightly worse. We suspect that the reason for this could be that GPT-3. 5-turbo is mainly built be conversational agent, while code-bison is specifically trained to have good coding capabilities.
NExT-QA.
7

---

Published in Transactions on Machine Learning Research (04/2024)
ReCCOC
RefcOCo+
GQA
NEXT-QA
32.6
48
42
32.0
60
46
40
Aocuracy
31.5
Accuracy
50
$1.0
38
30.5
40
42
ViperPPT API
Abelract
36
VperGPT API
Abetract
30.0
ViperPPT API
Abstract API
30
GP API
Abstracl
34
29,5
Number of ACEs
Numbar of ACEs
Number of ACEs
Number ACES
Figure 3: Incrcasing the number of ACEs in the prompt improves performance. Note that using the ViperGPT API on NExT-QA results in only three correct ACEs, so the performance plateaus after four ACEs.
3.2 Few-shot boostrapping via automatically generated in-context examples (ACEs)
We evaluate the effect of using automatically generated in-context examples (ACEs), described in Section 2.3 We can either sample few-shot examples manually or pick them at random. Both of these variants lead to good results, we show in the following experiments. However, selecting examples manually allows for a better "quality" (in terms of diversity) given a small number of few-shot examples, so by default we use these for all experiments. For the first set of experiments, we manually pick 16 few-shot examples from the training set: umage/vided question, and ground- truth answer. We try to make examples diverse to cover question "types" (e.g. left-right, front-back, closest-to, etc.).
In Figure 2 we show the effect of ACEs. For each dataset and for each API, the performance without ACEs is shown with the brown bar, and the performance with ACEs corresponds to the blue bar. We can see that for all datasets and all APIs, ACEs improve performance. The largest gains when using ACEs are for RefCOCO, RefCOCO+, and NEXT-QA datasets when using the ViperGPT API. This indicates that ACEs are effective in dealing with complex spatial and temporal reasoning. More importantly, it can be seen in Figure 2) that the gains from both the Abstract API and ACEs compound for all tasks, indicating that they provide complementary strengths. Figure 3 shows how the performance in terms of IoU and accuracy scales with the number of few-shot examples used to generate ACEs. As expected, increasing the number of few-shot examples leads to improved performance. Note that the ViperGPT API on NExT-QA is able to correctly "solve" only 3 few-shot examples, so there are no gains beyond using 4 few-shot examples.
We evaluate the effect of using randomly sampled few-shot examples instead of manually selecting them. On RefCOCO, we sample 100 few-shot random samples from the training set, run Zero-Shot framework on them, sort the resulting programs by their IoU, and select the top 16 programs. Therefore, we end up with the same number of ACEs as with manual selection. On ReICOCO, we achieve ToU of 47.9 and 49.1 with the ViperGPT API and our Abstract API respectively. These results are slightly better than those with manually selected few-shot examples (46.9 and 48.2 IoU). This shows that the manual labor for generating ACEs can be removed hetoget if we already have some labeled examples. With 50 few-shot random samples we obtain similar performance, and for 16 such samples we observe a small drop in performance (see Tables 5 and G in the Appendix A.1 for detailed results).
As in the previous section, we test whether our findings are consistent with and NEXT-QA. On RefCOCO, when using (PT-3.5-turbo, ACEs improve IoU from 28.9 to 39.8 with the ViperGPT API and from 38.1 to 41.6 with our Abstract API. Similarly, for
on RefCOCO
on NExT-QA, ACEs improve accuracy from 9.4 to 42.9 the ViperGPT API and from 56.7 to 58.8 with our Abstract API. This confirms that the benefit of ACEs is not only limited to code-bison but also holds for as well.
Another benefit of the few-shot setup when generating ACE is that it allows us to "tune" hyperparameters (HPs). For example, when sweeping over LLM temperature and object detection threshold HPs, we observed that the relative performances on the sew-shot examples closely resemble the one when sweeping over the full validation dataset (the analysis is shown in Appendix A.I.
8

---

Published in Transactions on Machine Learning Research (04/2024)
ReCOCC
.p. AM
34
Aln. AP
Vp. AMM ACE
33
Ale. All
Dystdd11
Accuracy
40
20
26
Namber of AGE3
Namber of ACE3
Number NAnberddAAEa aCAGEs
Number AGEa
Figure 4: Increasing the number of "self-tuning" steps leads to improved performance. Our Abstract API (Abs. API) consistently outperforms the ViperGPT API (Vip. API). The best performance is achieved when using dynamic object detector threshold (Dyn.t) addition to the Abstract API with ACE.
Table 2:
Component-wisee ablations of our framework. Each component contributes positively to the final score. Their relative contributions vary for different tasks. We report mean scores across three random sceds.
Model
RefCOCO (IoU)
RefCOCO+ (IoU)
GQA (acc.)
NEXT-QA (acc.)
ViperGPT API
38.4
32.0
27.9
11.5
+ Abstract API
42.3 (+3.9)
34.0 (+2.0)
30.0 (+2.1)
57.6 (+46.1)
+ ACE
47.3 (+5.0)
41.7 (+7.7)
30.6 (+0.6)
60.7 (+3.1)
+ Self-debugging
48.2 (+0.9)
42.9 (+1.2)
32.4 (+1.8)
61.0 (+0.3)
+ Self-tuning
51.2 (+3.0)
45.7 (+2.8)
33.4 (+1.0)
3.3 Self-correction
In this section, we analyze the ability of the framework to "self-correct" itself without any external feedback when code execution fails.
Self-debugging. When the program execution fails (due to e.g. compilation errors), we can retry by generating a program (Chen et al 2021). When creating new program, generated code and the question part of the prompt (see [Appndixx A.4). variant that we call "self- debugging". Another option would be to simply repeat the exact same prompt as in the previous trial and roly on stochasticity in the LLM with a temperature greater than zero to generate a new correct solution. our experiments, the "self-debugging" variant did not lead to an improvement in performance. the performance plateaus after the first trial. This in line with other recent findings Huang al. (2023); Stechly al. (2023); Valmeekam et al. (2023). On the other hand, the variant without any feedback in the prompt led to an increasingly better performance as the number of "trials" increases (see Figure 4/.
also feed the previously
all cases,
Self-tuning. In some cases, we know that code fails due to some specific module. Therefore, we can then adjust the hyperparameters of the respective module and re-run code. For example, the open vocabulary detector we used (OWLv2) has a threshold hyperparameter that controls how sensitive is. The lower this threshold, the more false positives we will have, but also the fewer false negatives. There is no global "optimal" value for this threshold that performs best for all images. Our framework allows hyperparameter dynamically: if the open vocabulary detector fails, we lower the threshold and the visual program again. In Figure 4, we can see that variants with a dynamic object detection threshold outperform all other variants and achieve the best. performance. Note that the variant that achieves the highest performance after five trials has a lower performance for the first trial. This happens because we start with a higher object detection threshold value of 0.15 (by default, we use 0.1). In this case, initially there will be more false negatives, but also fewer false positives. As we decrease the threshold in subsequent trials, the previously false negatives detected and the queries are correctly answered.
adjust this
9

---

Published in Transactions on Machine Learning Research (04/2024)
50
ViperGPT API
Abe
ACE
AGE
40
ACE Dynsmic Threshold
40
ACE
Dynamic Threehold
30
30
20
20
10
loU/aa.
loV; ViperGPT
Erfoy.Det Erfest
Effoner
1o lollp..c.
loUp.. 231
lolig
Erfoyy
Effect
Effocter
AbstractAPI
Figure 5: Error diagrams for the ViperGPT API and our Abstract API. We visualize the percentages of samples with loU in certain ranges. "Err" classes are samples for which code execution failed due to either: object detection (Obj.Det), wrong return type (Ret.Typo) or some other error (Other) c.g. hallucination.
3.4 Error analysis
Another benefit of visual reasoning with LLMs as programmers interpretability. For example, we can get insights into the percentage of successful program exccutions, which can be further decomposed into the ones that resulted in correct or incorrect responses, and for the programs that failed to execute, we can provide further insights into why they failed, i.e. which module failed to return the correct result. Figure 5 shows one such error analysis on RefCOCO. Categories labeled with "Error" are the ones for which code failed to execute due to either object detection (Obj.Det), wrong return type [Ret.Type) or some other error (Other) e.g. hallucination. For all other cases, code executed correctly (it returned bounding box), but sometimes it failed to detect the object ("IoU = 0" casc). First, we notice that for both APIs the number of "correct" detections (IoU higher than 0.7) grows we include ACEs and "self-tuning" through the dynamic object detection threshold. We also that the percentages of samples with high IoU are always higher for our Abstract APT compared to the ViperGPT API. Finally, note that the percentage of error cases drops from 12.8% to 4.8% for the ViperGPT API and from 9.1% to 3.8% for our Abstract API.
4 Related work
Visual reasoning with end-to-end monolithic models. Recently, SotA on VQA has been largely obtained by scaling end-to-end vision and language models (VLMs) in terms of their size, training data, and compute (Alayrac et al.) 2022 Chen et al., 2022b Yu ct al. 2022 Wang et al.) 2022a Gan ct al., 2022 Lu et al.J 2022] Li et al. 2023b Driess et al. 2023 Chen et al. 2023d cp. One of the earliest VLMs
Flamingo (Alayrac et al.J (022)) used a frozen pretrained language model up to 70B parameters and a frozen pretrained image encoder with 435M parameters and trained only "cross-attention" module that served as interface between them. Since then, efforts have mainly gone into scaling both the image and the language models: GIT (Wang et al. 2022a used a 300M language and scaled the image encoder to 4.8B parameters; PaLl (Chen et al. 2022b] scaled both components jointly, language model to 17B and image encoder to 4B parameters; PaLI-X (Chen et al. 2023d) contimued this trend of scaling the total number of parameters to 55B by using an image encoder with 22B parameters; PaLM-E scaled the mumber of total parameters the VLM to 562B by integrating the 540B PaLM (Chowdhery et al.) 2022) and the 22B Vision Transformer @Dsovviskiiy et al. 2020 Dehghani et al.] 2023). On the other hand, BLIP-2 (Li et al. 2023b) achieved SotA performance on various tasks with 12B VLM and PaLI-3 Chen et al. (023d) introduced significantly smaller VLM with 5B total parameters that achieves competitive performance with SotA models on various VLM benchmarks. These models are typically pretrained on large amounts of data and then fine-tuned for the best performance on the rownstream Lasks. In contrast to this, visual reasoning with LLMs as programmers (Gupta Kembhavi] 2023 Surís et al. 2023 Subramanian et al. 2023) does not require any finc-tuning or gradient updates on the downstream tasks. Morcover, even the largest VLMs struggle on the tasks that require compositional reasoning, the ability generalize, fine- iiooggaaned spatial capabilities,
and counting (Bugliarello et al. 2023 Hsieh et al. 2023
Tschannen et. al. 2023). Further scaling makes
10

---

Published in Transactions on Machine Learning Research (04/2024)
them even more data- and compute-hungry therefore, it is unclear whether scaling alone can solve these tasks. Conversely, using LLMs as programmers enables task decomposition into subtasks and holds promise of strong generalization and compositional reasoning.
Visual reasoning with Modular Networks. Neural Modular Networks (NMNs) (Andreas et al. 2016
Johnson ct al. 2017 Hu ct al., 2017) are an alternative approach to monolithic end-to-end networks and offer a polential route to (compositional) generalization. They are also typically trained end-to-end, using supervised or reinforcement learning. NMNs are designed to have several modules, and the hope is that during training, cach module will Iearn a different functionality, which will then be reusable across tasks. However, these models have a number of drawbacks: the program generation requires hand-tuned parsers, and they require optimization through reinforcement learning (e.g., REINFORCE, Williams| (1992)), which is often unstable. Learning all modules end-to-end hinders their ability to generalize (Bahdanau et al., 2018) and sometimes leads to a mode "collapse", where some modules take over all the work and other modules are never activated, or modules that do not learn intended functionalities (Subramanian et al.) 2020j. Furthermore, they sometimes require supervision for program learning, which is difficult to obtain at scale. On the other hand, visual reasoning with LLMs as programmers mitigates many issues of NMNs: it does not require gradient-based training or finetuning, it is able to incorporate any modules or swap the existing ones, it leverages the strong generalization ability of LLMs, and it generates programs by utilizing the in-context learning ability of LLMs, thereby removing the need for training program generators. The programs generated by LLM do not have to be domain specifie, but can use common language such as Python that does not require custom interpreters.
Visual reasoning with LLMs as programmers. The field of LLMs as controllers for visual reasoning has received a great deal of interest recently. LLMs as controllers (also known as "tool usc") approach became prominent in the literature (Parisi et al. 2022; Schick et al.] 2023). in particular for structured reasoning
in the natural language domain (Madaan et al. 2022 Wang et al. 2023b Gao et al.) 2023a Chen et al.) 2022a In the domain of using L.MMs as controllers for visual reasoning, PICa (Yang et al. 2022) solves a knowledge-based VQA task by first extracting an object and captions from the image and then querying
GPT-3 with this information and in-context examples to answer a question. Socratic models (Zeng al.,
2022
HuggingGP'T (Shen et al. (022). Societies of Mind (Zhuge et al. 2023), and many other papers (Liu
ct al.
2023
Zhu et al. 2023b Zhang et al. 2023b Li ct al.
2023c
Yang et al.
2024
Xu et al.) 2022b Zhang
et al.
2023a
Gao et. al. 2023b Ye el. al., 2023b Yang et al. 2023c Lu et al.
2024 Wu et al.] 2023 Zhu
et al.
2023a
(for a comprehensive survey, see Yin et al. 2023) or Li et al.
[2023a) compose vision and
language models to "communicate" in a fixed "protocol" " and solve tasks such as image captioning, visual question answering, image generation, and robot planning. On the other hand, models such as VisProg (Gupta & Kembhavi] 2023 ViperGPT (Surís al. 2023) and CodeVQA (Subramanian
al. 2023) go beyond a fixed communication "protocol" by having LLM write (Python) programs. During execution, the program calls individual vision modules (such as the object detector and depth estimator) through an API that is provided in the prompt. Additionally, VisProg is also capable of generating images as program output. These models showed great performance and achicved SotAA on tasks such as compositional visual question answering, visual grounding, and video temporal reasoning. However, in their current form, these models rely on heavy human engineering of query-code examples in the prompt that are dataset- and task-specific and require significant labor by highly skilled workers. Our framework, on the other hand, is able to automatically generate in-context examples, removing the need for humans to write query-code examples, uses an Abstract API that puls less burden on the LLM to have strong spatial and temporal reasoning abilities, and proposes a way to "self-correct" the programs that failed to execute. Note that our framework is equally applicable to all of the above approachos.
Automatizing prompt engineering. Vast litcrature shows that prompt format and contents are often important for achieving good performance with an LLM (Reynolds & McDonell, 2021 Zhao el al. 2021 Lu et al.) 2021 Moradi & Samwald, 2021 Madaan & Yazdanbakhshh 2022; Wei et al. 2023). A prompt typically consists of a task description (in natural language), in-context examples (c.g. query-code in ViperGPT) and an API (in the case where LLMs write a program). Various prompting techniques have been engineered, such Chain-of-Thor Thoughht prompting (Wei et al. 2022), Self-Consistency (Wang et al. 2022c), Tree of Thoughts
11

---

Published in Transactions on Machine Learning Research (04/2024)
Yao et al. 2023), Graph of Thoughts Besta et al., 2023 Plan-and-Solve Prompting (Wang al.
2023a
Mass-tto-Most Prompting (Zhou et al. 2022a), etc. All these techniques rely on human prompt engineering, in particular, on in-context examples. On the other hand, some methods try to automate prompt engineering.
They sometimes use gradient- based optimization (Shin et al.] 2020 Gao et al. 2020; Wen et al.) 2023) and some approaches require only API access to the model (Xu et al. 2022a Prasad el al. 2022]. Other works use LLMs for prompt optimization. APE ZZouu et al.) 2022b) first generates instructions with an LLM, then selects instructions with the highest accuracy, and uses them for future LLM prompts. APO (Pryzant et al.,
2023 generates feedback with an LLM that informs how to update the previous instruction. OPRO
YYan et al. 2023a) uses an LLM to generate new instructions at each optimization step, asking the L.L. to improve task accuracy by changing task instructions, which requires determining the score on small set of labeled examples and providing it in the meta-prompt. Promptbreeder (Fernando et al., 2023) goes a step further and proposes a seff-referential self inffiooereeent LLM using a meta-prompt that controls the generation of the main (task) prompt and evolves both via mutation. More importantly, Promptbrooder shows some surprising results such that a simple prompt "SOLUTION" outperforms all previous approaches. This further demonstrates the sensitivity of LLMs and the importance of automatizing the prompt engineering process. Common to all above frameworks for automatizing prompting is that they automatize the "task description" " part of the prompt. On the other hand, in our framework, we automatize the generation of in-context examples, which might, have an even greater influence on the performance of the LLM.
LLMs and self-correction. In the LLM literature, there have been mixed findings on the ability of LLMs to critique and self-correct their own reasoning and outputs.
(Madaan et al. 2023) provides feedback to the I.I. of the previously generated output, which is then refined. Several other approaches show benefits of providing feedback to LLM in improving reasoning (Shinn et al. 2024 Madaan et al. 2023)
code generation (Chen al.] 2023e Olausson et al.] 2023 Chen et al. 2023a): improving LLM alignment [Bai et al., 2022 Ganguli et al. 2023), etc. On the other hand, there has been increasing evidence that,
LLMs cannot self-correct rcasoning yet (Huang et al., 2023 Stochly ct al. 2023 Valmoekam et al. 2023 unless they receive external feedback, which usually requires access to ground truth labels. In our work, we also found that providing the previous question and code as feedback to the model did not improve the results. However, we show that it is possible to improve performance by tuning hyperparameters on the fly, a direction that, to the best of our knowledge, has not been explored previously.
5 Discussion and future work
Although the LLMs as controllers framework is very promising for visual reasoning, there is much future work to be explored. First, the use of vidco-specific models (or tools) could greatly improve performance on video tasks compared to the image-specific models we used. Moreover, the code generating LLM currently only takes the question as the input, but for some questions the program that correctly solves the question can only be generated given the image or video as the input too.
The results with the Abstract API show that this is a promising path forward, but more research is needed to find the "optimal" set of visual and temporal routines. Starting from these primitive routines, the model should be able to build an ever-growing library of routines (e.g. through the ACE generation process) that it can later reuse. This growing library of routines will most likely grow larger than the size of the context window, so research needed on an APT "router" that can select routines that are relevant to a specific task at hand. Furthermore, it would be important to rescarch ways of climinating the need for few-shot examples when generating ACEs, e.g. by providing a natural language dataset specification (a datasheet).
Lastly, more effort should be put into creating better benchmarks for evaluating compositional visual reasoning, as current ones have a number of limitations. For example, not all samples in RefCOCO and RefCOCO+ require compositional reasoning, so the LLM should only query the open-vocabulary object delector. Similarly, many referring expressions in GQA are ambiguous in the sense that there is not a single unique answer. Finally, NExT-QA contains ambiguous questions (e.g. why somcone did certain actions) or questions that can be answered by looking at the multiple choice answers only and disregarding the visual input altogether. The Perception Test (Pätráucean et al. 2023) is a promising benchmark for future work, as it was specifically
12

---

Published in Transactions on Machine Learning Research (04/2024)
created to avoid such problems. We hope that our findings inform future research on LLMs controllers for visual reasoning and encourage systematic evaluation and benchmarking efforts in the future.
6 Conclusion
In this work, we present a framework that makes LLMs as programmers for visual reasoning more robust, removes the need for human engineering of in-context examples (ICEs), and thus brings them a step closer to truly zero-shot visual reasoners. We introduce "Abstract API* that consists of spatially and temporally abstract routines, which improves performance by reducing the burden on the code-generating LLM to have strong spatial and temporal reasoning. By using a few labeled examples, we show how one can generate query-code ICEs automatically (ACEs) in a zero-shot manner. When used as in-context examples, ACEs consistently improve performance, climinating the need for human engineering of ICEs. We demonstrate how LLMs as controllers for visual reasoning can (to certain extent) perform "self-correction" through "self-debugging" and "self-tuning" without any ground-truth labels. In self-debugging, generating new code from scratch led to consistently better results, but providing the previous query-code pair as feedback to the LLM did not improve performance. In self-tuning, we show that the object detector threshold hyperparameter can be tuned aulomatically code execution fails due to this module. Across number of compositional drestion-answerte and video temporal reasoning tasks, we demonstrate that each component of our framework consistently leads to improvement.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 23736,, 2022.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 39-48, 2016.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. ar preprinl 2rXiv:230.114003, 2023.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: what is required and can it be learned? arXiv preprint 1rXiv:1811.1288, 2018.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai foodback. arXiv preprint arXiv:2212. 2.X0:22212.08033, 2022.
Macicj Besta, Nils Blach, Ales Kubicck, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadonski, Piotr Nyezyk, et al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXin:2308. Xii:2308.00667, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurol information processing systems, 33:1877-1901, 2020.
Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the' video' in video-language understanding. In Proceedings of the TEEE/CVF conference on computer vision and pattern recognition, pp. 2917 2927, 2022.
Emanuele Bugliarello, Laurent Sartran, Aishwarya Agrawal, Lisa Anne Hendricks, and Aida Nematzadch. Measuring progress in fine-grained vision-and-language understanding. ar preprint Xxi:2335500555. 2023.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R. Bowman, Kyunghyun Cho, and Ethan Perez. Improving code generation by training with natural language leedback. arXiv preprint arXit:2303.16749, 2023a.
Jiuhai Chen, Lichang Chen, Chen Zhu, and Tianyi Zhou. How many demonstrations do you need for in-context learning?, 2023b.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.000 008.04777,,844 2021.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Dis- entangling computation from reasoning for numerical reasoning tasks. arXiv preprint ar arXiv:2211.1250 2022a.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-inage model. arviv preprint arXic:2209.007 2022b.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint Xrxi:22305. 18565, 2023c.
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sobastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint X10:29300009199, 2023d.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self- arXiv preprint ar Xiv:20004 05128, 2023c.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sabastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ariie preprint orXiv:2204.00 023111, 2022.
Karl Cobbo, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.112 14168, 2021.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, An- dreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers 22 billion parameters. In International Conference on Machine Learning, pp. 7480-7512. PMLR, 2023.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192) 2020.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.0S3 03378, 2023.
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Prompt- breoder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2300. 16797, 2023.
Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. Vision-language pre-training: Basics, recent advances, and future trends. Foundations and Trends in Computer Graphics and Vision, 14(3 4):163 -52, 2022.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Deep Ganguli, Amanda Askell, Nicholas Schicfer, Thomas Liao, Kamilé Lukoštūtě, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self-correction in large language models. arXie preprint wrXiw:2302.07555 2023.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764-10799. PMLR, 2023a.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. orXiv preprint arXiv:2304.155 15010, 2023b.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint 2Xx::201..57720 2020.
Google. Google cloud vertex ai api [code-bison], available at: http:///rrrrtttteeemmmmmeeeeerrr
2023.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953 14962, 2023.
Lisa Anne Hendricks and Aida Nematzadeh. Probing image-language transformers for verb understanding. arxiv preprinl arXiv:2106 09141, 2021.
Cheng- Yu Hsich, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. arXiv preprint or Xiv:2300 14610, 2023.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEFE international conference computer vision, pp. 804 813, 2017.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXio preprint arXiv:2310.01 1179,, 2023.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 6700-6700 2019.
Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. A good prompt is worth millions of pa- rameters: Low-resource prompt-based learning for vision- lisiuanlanguag models. arXiv preprint arXiv:2110008/50 084484 2021.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman,, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE international conference on computer vision, pp. 2989-9999222 1899-2998, 2017.
Daniel Kahneman. Thinking, fast and slow. 2017.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby, Big transfer (bit): General visual representation learning. In Computer Aision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28-28, 2020, Proceedings, Part V 16, pp. 491-507. Springer, 2020.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmmented dialogue generation. orXie preprint ar 2007022107.07766, 2021.
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multimodal foundation models: From specialists to general- purpose assistants. arXiv preprint 2000003300.0000, 1(2::2 2023a.
15

---

Published in Transactions on Machine Learning Research (04/2024)
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping Sanguage-magg pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597. 2023b.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arx:2:9905. 06355, 2023c.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded aanguag-inaae pre-training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10965-10975, 2022.
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. orXie preprint arXiv:2311.0544 1074337, 2023.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddaa Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08911 2022.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivily. arXiv preprinl arXiv:2104.087 2021.
Aman Madaan and Amir Yazdanbakhsh. Text and patterns: For effective chain of thought, it takes two to tango. arViv preprint arXiv:2209.074 076866 2022.
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. arXiv preprint orXiv:2210.07128, 2022.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dairi, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement. with self-feedback. arXiv preprint Xrx:223303. 7651, 2023.
Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection, 2023.
Milad Moradi and Matthias Samwald. Evaluating the robustness of neural language models input perturbations. arXie preprint arXiv:2108.122. 2021.
Binh X Nguyen, Tuong Do, Huy Tran, Erman Tjiputra, Quang D Tran, and Anh Nguyen. Coarso-to-fine reasoning for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. Demystifying gpt self-repair for code generation. arXiv preprint arXh::2366.0989 2023.
OpenAL. Openai chatgpt api (gpt-3.5-turbo), available at: https://plafforroopppa..oom 2023.
model-an
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip count to ten. arXiv preprint arXiv:2302.1000 2023.
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205.1220 12255, 2022.
Viorica Päträuccan, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markoeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, et al. Perception test: A diagnostic benchmark for multinodal video models. arxiv preprint 2xXi:22355.13780 2023.
16

---

Published in Transactions on Machine Learning Research (04/2024)
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-based instruction search for prompting large language models. arXiv preprinl or arXi::2003.072 07281, 2022.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with' gradient descent" and beam search. arXiv preprint or NxXi::3305.0445, 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint XrXiv:2377. 2023.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust. monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions pottern analysis and machine intelligence, 14(3):1623-1637 2020.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Exlended Abstracts of the 2021 CHI Conference on Human Factors in Compuling Systems, pp. 1-7, 2021.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileamu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint Xrx::22302. 761, 2023.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2903. 1 17580, 2023.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010150080 2020.
Noah Shimn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Newral Information Processing Systems, 36, 2024.
Kaya Stochly, Matthew Marquez, and Subbarao Kambhampati. Gpt-4 doesn't know it's wrong: An analysis of ilerative prompting for reasoning problems, 2023.
Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, and Matt Gardner. Obtaining faithful interpretations from compositional neural networks. ar preprint arXiv:2005.00724, 2020.
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein. Modular visual question answering via code generation. arXiv preprint Xrxiv:2306. 05392, 2023.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2908.08 08128, 2023.
Romal Thoppilan, Daniel De Freitas, Jamic Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Landa: Language models for dialog applications. arXiv preprint arXio:2201.088: 08239, 2022.
Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer Image captioners are scalable vision learners too. arXie preprinl arXiv:2306.07915, 2023.
Shimon Ullman. Visual routines. In Readings in computer vision, pp. 298-328. Elsevier, 1987.
Karthik Valmockam, Matthew Marquez, and Subbarao Kambhampati. Can large language models really improve by self-critiqinng their own plans?, 2023.
Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. Will we run out of data? an analysis of the limits of scaling datascts in machine learning. arXio preprint arXiv:2211.04322, 2022.
17

---

Published in Transactions on Machine Learning Research (04/2024)
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. Xin preprinl
2rXiv:2205.14100 14100, 2022a.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka- Wei Lee, and Pe-Peng Lim. Plan-and- solve prompting: Improving zero-shot chain-of-though reasoning by large language models. ar preprinl arXiv:2305.04092 2023a.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to- sequence learning framework. In International Conference on Machine Learning, pp. 23318-23340 PMLR, 2022b.
Xingyao Wang, Sha Li, and Heng Ji. Code4struct: Code generation for few-shot event structure prediction. In Proceedings of the 61st Annual Meeting of the Association for Computational Lònguistics (Volume 1: Long Papers), pp. 3640 3663, 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-cone celf-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.1111 2022c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 -4837,, 2022.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Demny Zhou, et al. Larger language models do in-context lcarning differently. arXiv preprint arXiw:2303.038 03846, 2023.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. arXiv preprint arXiv:2302.0366 03666, 2023.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, drawing and editing with visual foundation models. arXiv preprint 3rXiv:2303.04677 2023.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of explaining temporal actions. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 9777-9786, 2021.
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang. Gps: Genetic prompt search for efficient few-shot learning. arXiv preprint arXiv:2210. 00o:2210.1704. 1704 2022a.
Zhiyang Xu, Ying Shen, and Lifu Huang. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. arXiv preprint arXiv:2212.1077; 2022b.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. Large language models optimizers. arXiv preprint Sttt::2309.03/000 2023a.
Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. Fine-grained visual prompting. arXiv preprint Xrxv:22306. 04356, 2023b.
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Go, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model use tools via self-instructton. Advonces in Neural Information Processing Systems, 36, 2024.
18

---

Published in Transactions on Machine Learning Research (04/2024)
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAT Conference Artificial Intelligence, volume 36, pp. 3081 3089, 2022.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompling chatgpt for multimodal reasoning and action. arXiv preprint XrXtt:2303.11381 2023c.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint ar trXw:2335510600 2023.
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuming for pre-trained vision-language models. orXiv preprint Xiv: 11797, 2021.
Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian, Ji Zhang, and Fei Huang. Hitea: Hierarchical temporal-aware video-language pre-training. In Proceedings of the IEEE/CVP International Conference Computer Vision, pp. 15405 15416, 2023a.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2344.1417, 2023b.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. Xix preprind arXiv:2306.1 13549, 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.0190 2022.
Licheng Yu, Patrick Poirson, Shan Yang, Alexandor C Berg, and Tamara Berg. Modeling context in referring expressions. In Computer Vision ECCV 2016: 14th European Conference, Amslerdam, The Netherlands, October 11-14, 2016, Proceedings, Part I1 14, pp. 69 85. Springer, 2016.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-wordee words, and what to do about it? In The Eleventh International Conference Learning Representations, 2022.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysstof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXim:2244.00508, 2022.
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXi::211..08276 2021.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. The visual task adaptation benchmark. 2019.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesníkov, and Lucas Beyer Sigmoid loss for language image pre-training. orXiv preprint 2rvi:22303. 15343, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tun audio visual language model for video understanding. arXiv preprint ar xiv:22000.00 02858, 2023a.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongshong Li, Peng Gao, and Yu Qiao. Llaparadapter: Efficient fine-tuning of language models with zero-init attention. ar preprint arXi::2333.1611 2023b.
19

---

Published in Transactions on Machine Learning Research (04/2024)
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint w.Xic2227.00222, 2022.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference Machine Learning, pp. 12697-12706.
PMLR, 2021.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et. al. Least-to-most, prompting enables complex reasoning in large language models. arXiv preprint or arXiv:2205. 10625, 2022a.
Yongchao Zhou, Andrei loan Muresamu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint orXiv22211.01110, 2022b.
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny, Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXiv:2303.0663 2023a.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023b.
Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Trie, et al. Mindstorms in natural language-based societies of mind. arXiv preprint or 2rxi 2023.
20

---

Published in Transactions on Machine Learning Research (04/2024)
A Appendix
A.1 Ablations
Here we present hyperparameter ablations, namely over the odo-generating LLM (code-bison) temperature and the open-vocabulary object detector (OWLv2) threshold.
Table 3: Code-generating LLM (code-bison) temperature hyperparameter ablations (with ACEs). The scores are reported as mean + standard deviation across three random seeds.
Model
LLM temp.
RefCOCO (loU)
RefCOCO+ (IoU)
NExT-QA (acc.)
ViperGPT API
0.0
41.4 =
39.8
36.0 + 0.1
0.4
46.9 +
41.7
53.1
0.8
46.9 =
42,0 0.7
53.2 0.1
1.0
46.2 + 0.7
40.9 4: 0.6
50.3 + 0.5
Abstract API
0.0
47.0 + 0.1
41.9 0.3
59.1 + 0.0
0.4
48.2 1
44.7
61.0
0.8
48.2 1
43.0 0.2
60.6
1.0
48.8 0.1
42.8 0.3
59.9 0.7
In Table 3) we report the scores for different. code-bison LLM temperatures: 0, 0.4, 0.8 and 1.0. We found the deterministic case to underperform compared to the cases with a temperature higher than zero. This indicates that the solutions of which the models most. "confident" of are not necessarily always correct. On the other hand, when the temperature is too high, the model starts to hallucinate functions that do not exist in the API and the performance degrades. Early in our work, we settled on the code-bison LLM temperature of 0.4 and did not tune it further.
Table 4: Open-vocabulary object detector (OWT.v2) threshold hyperporameter ablations. The scores are reported as mean + standard deviation across three random seeds.
Model
OWLv2 thrs.
RefCOCO (ToU)
RefCOCO+ (IoU)
0.05
42.6 i
41.9
ViperGPT API
0.10
46.9 =
41.7 0.4
0.15
45.0 i 0.5
38.2 4 0.2
0.20
33.3 = 0.6
29.8 0.2
0.05
42.8 +
42.7
Abstract API
0.10
48.2 0.1
44.7
0.15
47.1 1 0.2
38.6 1 0.1
0.20
33.7 + 0.4
30.1 + 0.4
Table 4 shows the effect of using different thresholds for OWLv2 open vocabulary detector. This threshold controls the level of 'sensitivity' of the open vocabulary detector. If the threshold value set too high, will have fewer false positives, but also more false negatives. We perform this study on RefcOCO and RefCOCO+. On both datasets, the threshold of 0.1 achieves the best results, by default use this threshold in all our experiments.
In Table 5 and Table 6 we show results for RefCOCO and ReICOCO+ when using randomlly sampled samples for the generation of ACEs, For comparison purposes, the tables we also provide scores when not using any ACEs and with the default setting when generating ACEs with 16 manually selected few-shot examples. From the tables, we can see that randomly selected 100 or 50 samples perform similarly as when using 16 manual samples. With 16 random samples we observe small drop in performance, though still observe
21

---

Published in Transactions on Machine Learning Research (04/2024)
Table 5: Results on RefCOCO with randomly sampled few-shot samples for generating ACEs. Shown are IoU scores for zero-shot (w/o ACE) setting, with the default setting that we used throughout the paper (16 manually sampled few-shot samples from the dataset (16 manual)) and with 100, 50 and 16 samples (100, 50 and 16 random) from the datasct. The IoU scores reported mean + standard deviation across three random seeds.
API
w/o ACE
16 manual
100 random
50 random
16 random
ViperGPT
41.7 u.6
46.9 0.8
47.9 + 0.2
45.8 + 0.4
41.5 0.2
Abstract
44.4
48.2 0.1
49.1 4: 49.0 + 46.9 0.2
Table 6: Results on RefCOCO+ with randomly sampled few-shot samples for generating ACEs. Shown are IoU scores for zero-shot (w/o ACE) sctting, with the default setting that used throughout the paper (16 manually sampled few-shot samples from the dataset (16 manual)) and with 100, 50 and 16 randomly-sample samples (100, 50 and 16 random) from the dataset. The ToU scores are reported as mean + standard deviation across three random seeds.
API
w/o ACE
16 manual
100 random
50 random
16 random
ViperGPT
36.7 0.4
41.7 0.4
41.3 a
40.3
36.3
rast
38.2 0.0
42.9 0.2
42.8 + 0.6
41.8 0.0
40.2 0.4
improvement compared the setting without any ACEs. In summary, this shows that the manual labor for generating ACEs can be removed altogether if we already have some labeled examples.
22

---

Published in Transactions on Machine Learning Research (04/2024)
A.2 Pretrained models
Here specify the pretrained models we used, and compare them with the ones used in ViperGPT:
Open-vocabulary object detector:
Ours: OWLv2 (Minderer al. 2023).
ViperGPT: GLIP Li et al. 2022) from the official GitHub repository
Depth estimation model:
-uus: MiDaS (Ranftl et al. 2020 v2 "DPT "mall" from PyTorch hut
ViperGPT: MiDaS (Ranftl et al.
2020 v2 "DPT Large" version from the PyTorch hut
Vision-language captioning model:
Ours: PaLI-3 (Chen et al. 2023d).
ViperGPT: BLIP-2 [Li al. 2023b from the official repository
CLIP-style image-text embedding model:
Ours: SigLiT
Zhai et al. 2023).
ViperGPT: X-VLM (Zeng et al. 2021 version finetuned for retrieval on MSCOCO from the official repository
Code-gencrating LLM:
Ours: code-bison accessible via the Google Cloud Vertex AT API
(Google 2023).
ViperGPT: Codex (code-davinci-002)) via the official OpenAl Python AP
Answer selector (based context information) LLM for multiple choice questions in NExT-QA:
Ours: code-bison accessible via the Google Cloud Vertex AT API
Google
2023
ViperGPT: GPT-3 via the official OpenAl Python AP
A.3 Self-debugging prompt
prompt *.n
Previously, for the query:
[query)
you 've generated the folloving code: (code)
The execution of the above code failed and returned the folloving error message: [error).
Given this, correct the above function, such that it executes correctly and solves the same query.
htttps///gihuub.co///ncceee /GLIP
htttpss//pttorch.og///b/////// /intelisl midas v2/
4https: htttpps///yyorcchooo///////lll midas v2/
7 htttpp///ghhbbbooom///gga///////
8 hhttps: http:p////oppeaccco/////og//////
http://accooooooooo/opoaaaaa oppeni-app
23

---

Published in Transactions on Machine Learning Research (04/2024)
A.4 Prompt listings
A.4.1 RefCOCO and GQA ViperGPT API
import math
class ImagePatch:
*.nA Python class containing a crop of an image centered around particular object, as well as relevant information. Attributes
cropped_image array_like
array-like the cropped image taken from the original image. left, lover, right, upper describing the position (dett/lover/riihh//ppet border crop's bounding the original image.
11
12
Nethods
13
14
(objectt__ name: -tres [InagePatch]
15
Returns a list of new ImagePatch objects containing crops of the image centered around any objects found in the image matching the object_name. exists (objectt name: str)->bool Returns True if the object specified by object_ name is found in the image, and False otherwise.
18
17
18
perity_proporry(propere (propert:: str)->bool
Returns True if the property is net and False otherwise. ()nput__dpth((->xf]))
Returns the median depth the image crop.
(left: int, lover: int, right: int, upper: int) int)->Ing ImagePatch
Returns InagePatch object containing crop of the image the given coordinates.
25
...
36
27
def i_in (self, image, left: int None, lover: int None, right: int None, upper: int None):
28
"""Initializes an ImagePatch object by cropping the image at the given coordinates and stores the coordinates as
29
attributes. If no coordinates are provided, the image is left unmodified, and the coordinates are set to the dimensions of the image. Parameters
30
image array_like
An array-like of the original inage. left, lover, right, upper int
Ån int describing position of the (bstt/loreerrrihhhh/ppp)) border of the crop 's bounding box in original image.
37
38
left
right self , iropped_inage image
upper
None:
39
40
self left
41
self lower
42
self right image. shape [2] # vidth
43
self upper image shape [1] # height
44
elae:
45
self wwwppedpppaaaeeeee inage image lover:upper left:right]
48
self self.left left left
47
self upper upper
right right
lower lower lover
self width self croppeddinnneeeeee image. shape [2]
self height self cropped_image shape [1]
self. horizontal hottiotttttallceeterr center (self.left self.right)
55
self. vertical center (self lover self. upper)
24

---

Published in Transactions on Machine Learning Research (04/2024)
nu* Returns list of ImagePatch objects matching object_ contained the crop 57
def find (self, object_nane: str) List [ImagePaath:
58
if any found.
59
Otherwise, returns an empty list. Parameters
60
61
62
object_nee name str the name of the object to be found
68
64
86
Returns
66
List [ImagePatch]
list of ImagePatch objects matching object_ contained in the crop
Examples
72
the foo
73
def execute_ (imag) List [ImagePatch]:
image_patel ImagePateh (image) foo_patches image_patch. find (*foo")
75
70
return foo_patches
77
78
return innd_in_ (self propped_inage. object_name)
70
80
def exists (self, object_nane str) -> bool:
81
"""Beurns True if the object apecified by object_nans is found in the image, and False otherwise.
82
Parameters
object_name str string describing the name of the object to be found in the image.
Examples
there both and garply bara
photo?
def execute_ (onngee)))
91
image_patch ImagePatch (image)
92
is_foo image_patch. exists ("foo") exioos("foo"))
98
is_garply image_ exists (agats"""arppaa ("garply bar")
94
5>>
return i8_foo 1e-garply_bar
95
96
return (sels find (object__nn))
97
98
def veridy-property (self, object_nans str, visual_property: str) -> bool:
99
"""Returns True if the object possesses the visual property, and False othervise. Differs from 'exists" in that it presupposes the existence of the object specified by object_name, instead checking shether the object possesses the property. Paraneters
object_namn
string describing the of the object be found the image.
visual_property :
string describing the simple visual property (e.g., color, shape, material) checked.
107
108
Examples
100
110
>>> Do the letters have blue color?
111
>>> def execute_com execute_command (ammand(imaae)) -> str:
112
>>>
magePatch(imagg >>> Lettera_patches = image_patch. ("letters"" ("s"attttc
113
>>> # Question assumes only one letter patch
>>> return letters_patches [ verify-propertt ("letters", "blue")
return veridy_proportyysolf. croppr/cpdde....oodd__ooo cropped_inee inage, object_name, property)
def compute_depth depth (self):
25

---

Published in Transactions on Machine Learning Research (04/2024)
n.* Returns median depth the image crop Parameters
Returns
124
125
float
128
the median depth of the image crop
127
128
Examples
129
130
>>> the bar furthest away
>>> def cot_utt__ccaaadddaaaaaaaa -> -Tinggga>Ina
>>>
image_patch ImagePatch (image)
>>>
bar_patches image_pater find find ("bar")
>>>
bar_patches.sort (eyyyy return bar [ar_patches[-1]
(kmy=lamdda bar: car.conptte_depth ())
>>>
depth_map compute depth(sell compute_depth return depth_map. map. median))
image)
def (sel, left: int, lower: int, right: int, upper: int) ImagePatch: Returns a new ImagePatch cropped from the current ImagePatch. Parameters
141
142
143
144
left, lover, right, upper int
145
The Loft/louer/right/nue berggh//ppprrrmmee most pixel of the cropped image.
146
147
return ImagePatch (self. peapped_inagee left, lover, right, upper)
def overlaps_ wichs_wwthhhsttee (self, left, lower, right, upper):
..* Returns True if a crop with the given coordinstes overlaps with this else False.
Parameters
left, lover, right, upper the (Lett/louer/ziih///ppe))) border of the crop be checked
158
Returns
150
160
bool
161
True if crop with the given coordinates overlaps with this one, else False
162
163
Examples
184
>>>
black foo on top of the qux def execute_commet won.amnddd -> InagePatch:
>>>
>>>
qux_patches ImagePatch
>>>
mmage_patch. find (*qux")
>>>
qux_patch qux [ux_patches [0]
>>>
foo_patches image_patch. find ("black foo*)
>>>
for foo in foo_patches:
if ver.eeracallcenterr qux_patch vertical_center
foo
175
return belf.eeft right and self right left and self lover upper and self, upper >= lower
170
177
178
def Bast_inage_or Catt_iiage_matchhiinee match (lisst_pahhtt List [InagePatch], content: List[str], return_index=False)) -> Union [ImagePatch, int]:
179
*""Returns the patch nost likely to contain the content.
Paraneters
list_patchs List [InagePatch]
content List [isrrstr] the object of interest return, index bool
26

---

Published in Transactions on Machine Learning Research (04/2024)
True. index of the patch likely contain the object
Returns
100
int
191
Patch most likely to contain the object
192
198
return best_image www.ammmaaellaaa (List_patches, content, return_inder)
194
196
198
def Clatance(patth.... ImagePatch, patch_b: ImagePatch) -> float:
Returns the distance between the edges of two InagePatches. If the patches overlap, it returns negative distance
corresponding to the negative intersection union.
Paraneters
patch_a ImagePatch
patch_b ImagePatch
206
Examples
207
208
Return the qux that is closest to the foo
200
>>> def aeecute_command
210
>>>
image_patch = ImagePatch (inage)
211
>>>
qux_patches = image_patchs find (' qux!)
212
>>>
foo_patches = tnn ('foo'
>>>
foo_patch fao_patches [o]
>>>
qux_patches. sort (key- x: (xstance foo_patch))
>>>
return qux_patches [0]
distance (patch_a, patch_b)
INSERT INSERT_IN_CONEE IN_OTTTET_ EXANPLES HERE
Write function using Python the ImagePatch (above) could be executed provide an
query.
222
223
Consider the following guidelines:
224
Use base Python (comparison. sorting) for basic logical operations, left/right/up/dom.
math, etc.
225
Make sure to always return an ImagePatch object.
226
Make sure that for all possible control flows, the program always returns an InagePatch
object.
INSERT PREVIRREE CODE AND ERROR_ HERE
INSERT QUERY HERE
27

---

Published in Transactions on Machine Learning Research (04/2024)
A.4.2 RefCOCO and GQA Abstract API
import math
class ImagePatch:
*."A Python class containing a crop of an image centered around particular object, as well as relevant information. Attributes
cropped_image array_like
An array-like of the cropped image taken from the original image. left, lover, right, upper int describing the position the (lett/Lover/righh/ppee)) border the crop bounding box the original image.
11
12
Nethods
13
14
(object_nanmm::: str)->List [InagePatch]
15
Returne a list of new ImagePatch objects containing crope of the image centered around any objects found in the image matching the object_name. exiata (objectt_n atr)->bool Returns True if the object specified by object_ name is found in the image, and False otheraise.
16
17
18
19
verity_property Returns True if the property is net, and False otherwise. ()tre_ddeph()))xftt
prroperty(ppopertyy (property: -tro->bool
Returns the median depth of the image erop
(left: int, lover: int, right: int, upper: int) ImaaeePhataaach Returns InagePatch object containing crop of the image at the given coordinates.
...
27
__iiit___sell image, left: int None):
None, lover: None, right: None, upper:
28
"""Initializes an ImagePatch object by cropping the image at the given coordinates and stores the coordinates as
29
attributes. If no coordinates are provided, the image is left unmodified, and the coordinates are set to the dimensions of the image. Parameters
30
31
32
image array_ like
An array-like of the original inage. left, lower, right, upper int An int describing the position of the (lett/bowerrriihh//ppp))) border of the crop's bounding box in the original image.
left None and right None and upper None and lover None: self cropped_
39
image
40
self left
41
self lower
42
self right image. shape [2] # width
43
self upper image shape [i] # height
64
elae:
45
self. . croppeddiiaeeeeeeee inage image lover: upper, left:right]
48
self left left
47
self. upper upper
48
self right right
self lover lover
50
self width = self. inapped_inage shape [2] self height self cropped_image shape [1]
self horizontal_centt
(self left self.right) /
self vertical_center (self lover self upper)
56
57
def find (self, abject_nane, str) List [InagePatch]:
28

---

Published in Transactions on Machine Learning Research (04/2024)
"""Returas list of ImagePatch objects matching object_ contained in the crop 1f any found.
58
Otherwise,
empty list.
60
Parameters
61
62
object_ner atr
63
the name of the object to be found
64
6G
Returns
66
87
List [ImagePatch]
68
list of InagePatch objects matching object_nanc contained in the crop
69
Examples
>>> return the feo
>>> def execute execute_cosmand command
(image) List [InagePatch]:
image_patee ImagePatch (image)
75
Ioo_patches Amage_patell find (*foo")
return foo_patches
77
78
return
(self. cropped_inage. object_name)
70
80
def exists (self, object_nand name: str) bool:
81
"""Returns True if the object specified by object_nane is found in the image, and False otherwise.
82
Parameters
83
84
object_name str string describing the nane of the object to be found in the inage.
Examples
>>> Are there both foos and garply bars in the photo?
>>> def precuteecoomand (image)->str:
image_patch ImagePatch (image)
i2_foo image_patch. exists ("foo")
is.garply_bar image_patch exists ("garply bar")
94
return is_foo is-garply_bar
96
98
return len(self. (objecttt_
97
98
def verify_property (self, object_nans str, visual.property, str) -> bool:
90
"""Returns True if the object possesses the visual property, and Palse othervise. Differs from 'exista' in that it presupposes the existence of the object specified by object_name _ane, instead checking whether the object possesses the property. Parameters
100
object_nanc str
string describing the name of the object to be found in the image. visual_prat Spoperpyy
string describing the simple visual property (e.g., color, shape, material) checked.
Examples
100
110
Do the letters have blue color?
111
def execute_ comeute (amaadd(imaae) str:
112
image_patch (magePatch(1mmge) letters_patches = image_patch. finde >>> # Question assunes only one letter patch
113
>>>
114
115
>>>
return letters_patcher (0) www.iiy ("letters", "blue")
return verity_ creppcccpppe....cppe..... image, object_name, property)
def cmmpute_depthseet) (self);
""*Returns the median depth of the image crop
Paraneters
29

---

Published in Transactions on Machine Learning Research (04/2024)
Returns
float
128
the median depth of the image crop
127
128
Examples
129
130
>>> the bar furthest away
131
>>> def execute_ cantte__ooaanddaaaeeee (1naggg)>>naagaPPahhh
132
5>>
image_patch ImagePatch (image)
>>>
bar_patches image_patch, find ("bar")
>>>
bar_patches. sort (key==lambd bar: car.conputo_depth ())
>3>
return bar [ar_pasches--1]
septh_map compute_ depphddeephhppeeppppeee (self (roppedd_iagee return depth_maye machh_map.aaa. median mediia))
def (self, left: int, Lower: int, right: int, upper: int) -> ImagePatch: n** Returns
ImagePatch crepped from the InagePatch.
Parameters
143
144
left, lower, right, upper int
148
The (left/lover/rte
pizel of the cropped image.
146
147
148
return InagePatch (self, eropped_inage left, lover, right, upper)
149
def overlaps_ teaipp_wwttheee (self, (self, left, lover, right, upper):
""* Returns True if a crop with the given coordinates overlaps with this one, else False. Parameters
left, lover, right, upper int the [offt/loue//riht// border of the crop to be checked
Returns
100
bool
101
True crop with the given coordinates overlaps with this one, else False
182
143
Examples
164
160
>>>
black foo on top of the qux def execute__comandd (image) -> InagePatch: Smage_pated ImagePatch (image)
166
>>>
>>>
>>>
qux_patches image_patch, find ("qux")
>>>
qux_patch gux_patches[O)
>>>
foo_patches amage_patch..ing find ("black foo*)
>>>
for foo in foo_patches:
>>>
if foo. www.vertial_ceeee return foo
qux_patch. vertical_
>>>
68*
self left right and right left and self lover upper and self. upper lower
176
177
178
def best_image__a. match (1istt_pthht List [InagePatch), content: [ist(str), return index_.ede Union [ImagePatch, iat]: *""Returns the patch nost likely to contain the content.
170
180
Paraneters
181
list_ _ist_patches List [InagePaach content dist[str]
the object of interest return_index bool if True, returns the index of the patch most likely to contain the object
30

---

Published in Transactions on Machine Learning Research (04/2024)
Returns
Patch likely contain object
192
193
return inat__mage__mar match (list_ content, retura_index)
194
195
196
def (estaance(patc..oo ImagePatch, patch_b: ImagePatch) >> float:
197
198
Returns the distance between the edges of tuo ImagePatches. If the patches overlap, it returns negative distance
corresponding to the negative intersection over union.
Paraneters
patch_a ImagePatch
patch_b ImagePatch
Examplea
208
Return the qux that is closest to the foo def execute_comee connanddinaga)
200
210
>>>
image_patch ImagePatch (image)
211
>>>
qux_patches image_patch. find ('qux")
212
>>>
foo_patches = image_patch find ft ('foo')
218
5>>
foo_patch foo_patches [0]
214
5>>
qux_patches. sort (key- x: (istance foo_patch))
3>>
return qux_patches [o]
return distance (patch_a, patch_b)
def get_patch, left_ of ImagePatch) -> ImagePatch: left_patch get_patch_ left_of(patch) return left_patch
def get wwt.ppatc_righhtte of (patch: ImagePatch) ImagePatch:
right_patch pet_patchhrightt.rr (patch)
225
return right_patch
228
227
def get_patch__bove (patch: ImagePatch) ImagePatch:
228
above_patch get_patcch_abrer (patch) return above_patch
220
230
231 def het_paacc_bbelox.oom (patch: ImagePatch) -> InagePatch:
belov_patet - get_ get_patcch___peppeeeeeeeee batch_beloke of (patch) return belov_patch
def hww_patchharooudd... of (patch: InagePatch) >> ImagePatch: patch__round____ (patch) (patch))) return around_patch
around_patch
def aort_patchea_let- cottttooarrrrttsttaatttta..a (list_patches: List (ImagePatch]) >> List [ImagePatch]:
.*.
patches according their horizontal centers.
242
243
Paraneters
244
245
1ist_patches List [InagePatch]
246
247
Examples
248
# Right foo
>>> def
execute_ comnand (araaand(iagg (inage):
>>>
image_patch = InagePatch (inage)
>>>
foo_patches = image_patch. (?foo)
>>>
foo foo_patches sort_patches, beft_to_rrght (foo (cto_eetccee)
>>>
right_ foo foo_pach foo foo_patches[-1
31

---

Published in Transactions on Machine Learning Research (04/2024)
rigat_.fo_ppatch
sort_patches_.fff.too. (listpaahhh)
250
290
def
List [ImagePatch]) List [ImagePatch]]
281
262
Sorts patches according to their vertical centers.
263
294
Paraneters
285
list_patcher List [InagePatch]
Examples
Second bar from the top >>> def execute, (omaged(inage)
image_patch ImagePatch (image)
bar_patchea sort_patches, botton_ to_ttotttoooooooo top (bar_patches)
bar_patchea watpsttooooottprppppee bar_patches
276
return tecond_toppost_bbrr_ettt
277
278
return oort_patchee.botttoo.ooooooooot
270
280
281
def sort_patcches_front to httpcc/e__.aatttooooocccaaaaaeee tosttpcaccciisspttcteess List (ImagePatch)) -> List (ImagePatch);
282
Sorts patches according to how far from camera they are.
Paraneters
list_patches List [InagePatch]
Examples
Person back
def execute_ coneundeeoomnddeeeee :
208
image_patch ImagePatch (image)
204
person_patches image_patch. find person') (iredtperooooon
296
>>>
person_patches
paaathhe__font_tt (pereeo...atcchee
298
>>>
beraon_in_thh__bee back person_patcher [-1]
297
>>>
return person_in_the_bant
298
299
return
back (asttaacctttttttccceesss)
def get (listtcchhitttttcccccce.. List [InagePatch]) >> InagePatch:
Returns the middle patch.
Paraneters
litt_patches List [ImagePatch]
310
Examples
811
912
Middle
913
>>> def
(inage):
314
>>>
image_patch = ImagePatch (image)
815
>>>
ham ham_patches = inage_patch, find (
816
>>>
siddle_ham_patch = get_middle_pei (han_patches)
>>>
return siddle_ham_pater
return get_middle_pan patch List_pptches)
32

---

Published in Transactions on Machine Learning Research (04/2024)
def get_, www.ppaccccooet.t.....e. closest, anchor _objett caccor..bjeetttsssspaaaa..... (1ist. List [ImagePatch), anchor, aachor_object. InsgePatch) ImagePatch:: sun
Returns object list_patchhs closest anchor object.
826
920
Paraneters
927
828
list_patches List [InagePatch] anchor_object ImagePatch
829
830
831
Examples
# Foo next to bar
>>> def execute_ commanddddddnnneeea (inage):
>>>
image_patch - ImagePatch (inage)
>>>
foo_patches image_patch. ("
>>>
bar_patches image_pateh... find ("bar")
bar_patch bar_patchea [0]
foo_next_to ceapnext_to_bb_rre. bo_ get_patch, cet_patcc.ccooestt.oo closest_tttttttooooooooeeoo ooo.odooooeeeooooooooooooe.eee (tooppaaheee..e
bar_patch)
foo
841
842
return get_patchh_ get_patch.cloest.too htgpaaa.cc.o........ooooooooooaa
anchor_object)
943
344
845
CWBERTIIN_CNTEATT BEOONTEAT_EXAAPLSS.... EXANPLES_ HERE
846
847
Write fuaction using Python and the ImagePatch class (above) that could be executed to provide an ansver to the query.
Consider the following guidelines:
Use base Python (comparison, sorting) for basic logical operations, lett/right/up/donn math, etc.
Make to aluays return ImagePatch object.
Make that for all possible control flous, the program always returns ImagePatch object.
ImagePatch
left and right denote horizontal edges. vertical edges.
ImagePatch
top
835
958
AND ERROR_HERS HERE
357
358
INSERT_ QUERY HERE QURRY_HERE
33

---

Published in Transactions on Machine Learning Research (04/2024)
A.4.3 NEXT-QA
ViperGPT API
import math
class ImagePatch:
*."A Python class containing a crop of an image centered around particular object, as well as relevant information. Attributes
cropped_image array_like
An array-like of the cropped image taken from the original image. left, lover, right, upper int describing the position the (lett/Lover/right/upee) border the crop bounding box the original image.
11
12
Nethods
13
14
(object_nannm::: str)->List [InagePatch]
15
Returne a list of new ImagePatch objects containing crops of the image centered around any objects found in the image matching the object_name. exiata (objectt_t atr)->bool Returns True if the object specified by object_ name is found in the image, and False otheraise.
16
17
18
19
verity_property (rroperty(ppopertty (property: str)->bool
Returns True if the property is net, and False otherwise. best_text_match (optinn..... [str), prefix: str)->str Returns the string that best natches the image. simple_query (question: str=Kone)-xatr
Returns the provided, compute_ (ettte__dett((>>>1oo
basic question asked about the inage. If question is "What this?",
Returns median depth
image crop.
27
crop (1eft: int, lover: int, right: int, upper: int) -maagePatch Returns a new InagePatch object containing crop the image at the given coordinates.
28
29
30
31
def ...init_.... (seff, image, left: int None, lover: int None, right: int None, upper: int None):
32
""IInitaaizes an InagePatch object by cropping the image at the given coordinates and stores the coordinates as attributes. If no coordinates provided, the inage is left unmodified, and the coordinates set to the dimensions of the inage. Parameters
image array_like
An array-like of the original inage. left, lover, right, upper position
40
(lott//ooee//rihhh//ppe))) border of int describing original image.
crop 's bounding box
41
42
left is None and right is Mone and upper is None and lover None: self self.croppe crooppedddddd image
43
64
self.left self left
45
self lower self right image.ehape [2] # width
48
47
self upper image shape [1] # height
slse:
self. cropped_ingeeeeeeeeeeeeppppppppe _iageinagee-e image lover upper, left:right]
self left = left
self upper upper
self right right
self lover lover
55
self width self. www.eeeeppppeeeeee image...__e____ shape [2]
56
self height self.crepe cropped image...... shape [1]
34

---

string describing the of the object be found the inage. Published in Transactions on Machine Learning Research (04/2024)
58
self meaizontal_centter (self left self. rightt self vertical_center (self
58
self upper)
60
61
def (self, object_nane str) List [ImagePatch):
62
nu* Returns a list of ImagePatch objects matching object_name contained in the crop 1f any are found.
63
Otherwise, returns an empty list. Parameters
64
66
68
object_name atr the name of the object to be found
67
Returns
List [ImagePatch]
list of InagePatch objects matching object_name contained in the crоp
73
Examples
75
77
def execute_
execute_conmand command (image) List (InagePatch):
78
image_patel ImagePatch (image)
79
>>>
foo_patches amage_patch. find (*foo")
80
>>> return foo_patcher
81
82
return find_in_image (self... cropped_inage image, object_name)
88
84
def exists (self, object_nane str) -> bool: Retuurs True if the object specified by object_nane is found in the image, and False otherwise.
Paraneters
object_name
91
Examples
98
Are there both foos and garply bars the photo? def execute_ (image)
94
96
image_patch (magPattch(1mage) 18_foo image_patell existe
98
>>>
97
>>2
is.garply_ban bar = imago_patch. exists ("garply bar")
98
>>>
return bool_to_yesno (ia_foo and is-garply_bar)
99
return (anssff.fine (objectt__nneee
def verify_property (self, object_nane str, visual_properg, str) -> bool:
""*Returns True if the object possesses the visual property, and False othervise. Differs from 'exists' in that it presupposes the existence of the object specified by object_nane, instead checking whether the object possesses the property. Paraneters
object_nane
string describing the visual_ propperyy : str
object found image.
100
110
string describing the simple visual property (e.g., color, shape, material) to checked.
111
112
Examples
113
114
>>> Do the letters have blue color?
>>> def cacpuute_oocmaddeeeee
=> str:
>>>
Paggeaachh(image letters_patches = finge_patch.fnnd@Meeeeeet.... # Question assumes only one letter patch
image_patch
>>>
>>>
>>>
return bool_to, tol__to_essn (letters_patee [0]. veriey_property ("letters", "blue"))
35

---

Published in Transactions on Machine Learning Research (04/2024)
verity-properse (sel1 (seli pwoppeed.naaee... object_name, property)
def teettteetttteeeeeeeeeeeeeeeeeeee
aue mataa (self, option list: [ist[) string
matches the image.
128
Parameters
128
127
option_list list atr list with the names of the different options prefix str string with the prefixes to append to the options
128
129
130
131
Examples
>>> # Is the foo gold white?
>>> def execute_ (image
>>>
smage_patel ImagePatch (image)
>>>
foo_patches image_patch. (*foo")
Question foo_patches [0] best_ (["gold", "Ihite"])
foo patch
best_text, best_text_maacch (self. croppedd image, option_list)
142
143
def simple-query (self, question: str None) str: Retuurs the ansuer to a basic question asked about the image. If no question is provided, returns the ansver to "What is this?". The questions are about basic perception, and are not meant to be used for complex reasoning or external knowledge. Paraneters
144
14G
140
question str string describing the question to be asked.
Examples
Which kind
is fredding?
def execute_ cacttee_ccmaaddd (image) str:
157
image_patch ImagePatch baz_patches image_patch. find (*baz")
158
150
5>>
for baz_patch in baz_patchea: if not baz_patch. wet.beeerttypppppreeeeee ("baz", "fredding"):
140
3>>
161
>>>
return baz_patch. simple_query ("What is this baz?")
162
163
>>> What color is the foo?
>>> def Ceccutte_oomaaddaae
-> str:
>>>
image_patch ImagePatch (image)
>>> foo foo_patches image_patch. find ("foo")
>>> foo_patch feo_patches [0]
>>>
return foo_patch.einple,.e himp:eeepeeeeeeeeeeeeeeeeeeeeeee ("What is the color?")
>>> the second bar from the left quuxy?
def execute_ execute_command (image) amage_patel bar_patches image_patch. find (*bar")
str:
mmagPatcch(Image
174
bar_patche sort (key=lambda x: x. h.hrisoottal.centee
176
bar_patch bar bar_patches
176
return bar_patch. par_patch.simple wtw.cch.eeppeeeeeeeeeeeeeeeeeeee ("Is the bar quuxy?")
177
178
return simple_query (self gcropped_image question)
170
180
def Smmpute_depth(ael1f
""*Returns the median depth of the image crop
Paraneters
Returns
float
36

---

franes represented ImagePatch objects, well Published in Transactions on Machine Learning Research (04/2024)
the median depth of the image crop
Examples
191
the bar furthest away def execute_ command (taage))-nnaaaaeaahhh
192
193
>>>
image_patch ImagePatch (image)
194
>>>
bar_patches image_ praagepptcc..tt find ("bar")
195
5>2
bar_patches. sort (key=lambda bar: bar. car.coopute__eppe ())
196
>>>
return bar bar_patches(-1)
197
depth_map compute, _eptt (self www.:eseele.e..eeeeeeeeeeeeecece imageed_image) return depth_map.median median
def (sel,, left: int, lower: int, right: int, upper: int) -> ImagePatch: ""* Returus new ImagePatch cropped from the current ImagePatch.
Parameters
left, lover, right, upper The (eet//ooverrsght///pee) most pixel of the cropped image.
208
200
return ImagePatch (self sropped_image left, lower, right, upper)
210
211
def cistapsssttssel left, lover, right, upper):
212
Returns True if a crop with the given coordinates overlaps with this one, else False.
218
214
Parameters
left, lover, right, upper int the (loft/loxer/riihh//ppp border of the crop to be checked
Returns
bool
True crop with the given coordinates overlaps this else False
Examples
225
228
black foo on top of the qux def execute_ (amaanddiaage image_pater ImagePatch (image)
227
InagePatch:
228
>>>
220
>>>
qux_patches image_patch,. find ("qux")
230
>>>
qux_patch qux qux_patches [0]
231
>>>
foo_patches image_patch fandge_patch.iiddccccc find
("black finda"aack foo*)
>>>
for foo in foo_patches:
>>>
if foo. veotveiiical_cntt return foo
> we..aatch.errticl.cco.ne.e
>>>
return self left < right and self right >- left and self lover upper and self.
upper >- lower
class VideoSegment:
*.*A Python class containing as relevant information. Attributes
241
242
243
video : torch. Tensor
244
A tensor of the original video. start : int
245
246
An int describing the starting frane in this video segment with reapect to the original video.
end : int An int describing the ending frame in this video segment with respect to the original video.
nun. frames >>int
An int containing the number of frames in the video segnent.
37

---

Published in Transactions on Machine Learning Research (04/2024)
Nethods
frame_ iteratorrttrrtt pectrratto->Itereeee ->Iterator [ImagePatch]]
256
trim (starr, end) -VVieeSSegment
250
Returns new VideoSegment containing trimmed version of the original video at the [ start, end] segment.
257
select_ansver
question, options) ->str
258
Returns the answer to the question given the options and additional information.
239
280
def ___ini__ (self, video: torch. Tensor, start: int None, end: int None, parent_start =0, queues=None))
Initializes VideoSegment object by trimming the video the given (start, end] times and stores the start and end times attributes. If times provided, the video is left unmodified, and the times the beginning and of the video.
Parameters
268
video torch Tensor
200
A tensor of the original video. start int An int describing the starting frame in this video segment with respect to the original video. end int An int describing the ending frane in this video segment with reapect to the original video.
270
271
272
273
if start is None and end is None: self trimmed_vides video video self start 0 self video shape[0] # duration else:
self trimmed_ video video [start:end)
283
start
284
end is None:
285
end video. shape [0]
288
self start start parent_start
287
self end end parent_start
288
289
self num num_framss = self trinmed_videe shaeo [0]
def
(self) >> Iterator[Ines [ImagePatch]:
="* Returns iterator over the frames in the video segment. .u*
for in (self. franes)
yield ImsgePatch (self trimmed, video [i], self start 1)
def (sinf start: Union[int None] None, end: Union Cint, None] None)
VideoSegment:
an* Retuurss the [start, end] segment.
VideoSegment containing trimmed version the original video
208
200
300
Parameters
901
302
start Union lint, None]
303
An int describing the starting frane in this video segment with respect to the original video.
end Union [int, None]
An int describing the ending frane in this video segment with respect to the original video.
Examples
38

---

Published in Transactions on Machine Learning Research (04/2024)
Return the second half of the video def execute_commee cviden)(
video_segment VideoSegment (video)
video_secone saaondaaaffff video_ segment. (video semeeee num franes 2,
rideo_segment caattnuum__aaeeee >>> return video_ half
913
314
815
if start is not None: start nax (start, 0) if end is not None:
816
817
818
end min(end, aelf fum_fraess)
return VideoSegment (self trimmed_vides video, start, end, self. start) def
info: dict, question: str, options: List[str]) str:
return select_annee aneeetr trinmedd info, question, options)
def (seli): "VideoSegment (O,, ()) .fommattt (self start, self end)
def qimple_query (self, question)
829
.** simple question about the video.
330
931
Examples
332
833
# why does X happen?
834
#
Paanswerl', 'answer2', 'answer3', 'answer4', 5answer5 ?
835
(video, question, possible_ansverg (str, dict]: def Create a video segment object
dideoSegmentridee
# The question is simple, so just ask info rideo_segsent www.seeteeeipeeeeee.e.eee.ee simple_query ("why does x happen?")
Choose the ansuer among given possible ansvers ansver select_ansved (info, question, possible_ansvers
return
Simple_quezs (question)
846
947
948
def best beat_image_nn mates
List [InagePatch], content: [ist(str), teturn_
Union [ImagePatch, iat]: *""Returns the patch most likely to contain the content. Paraneters
349
850
851
list_ _lat__atches List [ImagePaccc content List[str]
the object of interest return_ index bool if True, returns the index of the patch most likely to contain the object
Returns
Patch likely contain the object
802
...
383
return best_ htcps_imaaemmmeee match (list_patches, content, return_index)
384
985
366
def (esttance(patch.gg ImagePatch, b: ImagePatch) >> float:
367
368
Returns the distance between the edges of two ImagePatches. If the patches overlap, it returns negative distance
corresponding to the negative intersection over union.
Paraneters
patch_a ImagePatch
39

---

Published in Transactions on Machine Learning Research (04/2024)
patch_b ImagePatch
Examples
878
Return the qux that is closest to the foo >>> def precute_commant (Imagee)
379
380
>>>
image_patch ImagePatch (image)
881
>>>
qux_patches = inage_patch. find (' ('qux')
882
>>>
too_patches = image_patch, find finoooooooo
888
552
foo_patch foo_patches [0]
>>>
aort (key=lambda x: distance foo_patch))
>>>
return qux_patches [o]
return distance (patch_a, patch_b)
def toosnno_ ansuer: boal) => "yes" answer "no"
select_ansves (info: str, question: question, possible,answerg answers: str) str: ... Given an info, question and possible answers, select the correct answer
896
890
997
Examples
398
390
# what does man do at the end of the video
400
# possible _anseers--seer: ('answerl', 'answer2', 'answer3', 'answer4', 'anawer5 ']
401
def execute_ command (video, question, aosibble_assvers -> [atr, dict]: # Create video segment object sideo_segment
AddeoSegment(ridee (video)
# Caption last frane of the video (end of video)
last_frane = InsgePatch (video segmentt, -1)
last_caption last last_frame frame.rr....... wwipeee query W"tt is this?")
last_ fert_frame.Fre find ("man") ("ma""
len (men) == 0:
[last_frame]
men
man_ man_action www.see_peeeeee. Whth
doing?")
412
Answer the question. Remember to create the info dictionary info
413
414
"Caption of last frame": last_caption "Man looks like he is doing": aan_action
415
416
417
anawer rideo_segnest
question, possible_ansxers
418
return answer, info
INSERT CONTEXT, EXANPLES, HERE
Write function using Python and the VideoSegment class (above) that could be executed to the query.
provide
Consider the following guidelines:
428
base Python (comparison, sorting) for basic logical operations, left/right/up/donn.
math, etc.
429
430
INSERT INSERT_PREVIONE PREVIOUSEEECEEEEEEEEEEEEEERRRREE CODE_CCOEEE___E AND ERROR HERE
431
432
INSERT QUERY HERE QUERY_HERE
40

---

Published in Transactions on Machine Learning Research (04/2024)
A.4.4 NEXT-QA Abstract API
import math
class VideoSegment:
*."A Python class containing a video, as well as relevant information. Attributes
video ndarray
tensor of the original video. int An int describing the starting frame in this video segment with reapect the original video.
12
13
An describing ending frame this video segment with respect to original video.
14
>>int
15
An int containing the number frames in the video segnent.
16
17
Nethods
18
trim (start, end) V>deeSSeegett Returns new VideaSegnent containing trimmed version of the original video at the [ start, end] segment.
select_ansver (info, question, a2t
Returns the answer to the question given the possible anewers and additional information
None, end: None, parent_start ou* Initiaalls VideoSegment object by trimming the video the given [start, end] times and stores the start and end times as attributes. If no times are provided, the video is left unmodified, and the times are
def __init__ (self, video: up ndarray,
qunees=None):
26
27
28
set to the beginning and end of the video.
20
30
Parameters
31
52
video ndarray A tensor of the original video. start int An int describing the starting frane in this video segment with reapect to the original video. end int int describing the ending frane in this video segment with respect the original video. an*
38
39
40
start None end None: self. triamed_vides video video gelf.stare
41
42
43
self. end video shape [0] # duration else:
64
45
self. rrtamed_vides video (start:t
48
1f start is None:
47
start =
if end is None:
end = aideo.shape [0]
self start = start + parent_start
self end end parent_start
self num_frames self, trimmed_vides shape [0]
55
def trimf start: [int, None, [int, None) VideoSegment:
41

---

Published in Transactions on Machine Learning Research (04/2024)
"** ""* Returns VideoSegment containing trimmed version the original video the [start, end]
57
segment.
58
59
Parameters
60
61
start Union [int, None]
62
An int describing the starting frame in this video segment with reapect to the original video.
68
end Union [int, None]
64
An int describing the ending frane in this video segment with reapect to the original video.
65
>>> Return the second half of the video 66
Examples
>>> def cattreecocaaaddddcceeace
video_segment VideoSeguent (video)
71
video_secoud_half segment trim (video_segnent segment nun_ uaaere // Lideo_segment num_iramese video_ seceo__eeond_hae half
72
78
74
start not None: start (start, 0)
75
76
if end is not None:
77
end = nin( self nun_framss)
78
79
return VideoSegment (self srimmed_vvide. start, end, self.starty def return get_ video of (event)
oeventttt event) -> VideoSegment:
def video segment reement_before_eeee beforee (self, event) -> VideoSegment
return get_vides get_video__egment bereees_seeeeeebbefrreeeee tergmaettaoforre_eeeteeeeeeeeee (event)
87
def gidev wew.sseementtaee (fe event) VideoSegment:
return sew.eeoo_eeeeettatteeeee
00
def caption_vides (self, question) str:
01
return caption_vides (quoo (question)
92
93
def Ask simple question about the video.
(self, question) -> str:
94
95
96 Eramples
97
# why does X happen?
# possible_answeeeeeeeeeeeeeeeesee P'ansver'', 'answer2', 'answer3', 'answer4', Pansver8 *] def peecute_commad
question, geeibble_nnvereee <>[srr, dict]: Create video segment object video_segment VideoSegnent (video) The question is simple, info seggent Choose the www.seegteeeeeeeee....eeee simple-quezy ("xhy does happen?")
just ask
among given possible
select_ (info, question, possible_anavery
107
return answer
108
100
answer simple-query (question)
110
return answer
111
112
113
def select_answer str, question: question, pasaible_anwwert str) -> str: Given an info, question and possible ansvers, select the correct ansver.
Examples
what does person do after event X? aneeeee [ ['ansveri', 'ansuer2', 'answer3', 'ansverd'. *ansver5 ']
42

---

Published in Transactions on Machine Learning Research (04/2024)
def execute_command (video, question, poaslble.ausserrs) dict]:
Create video segment object
video__egment VideoSegment (video)
segment degme_teement.oo get_ wtt......deeeeaaat..ttt........ ("event x*)
124
125
Ask what the person A is doing
128
info video_segrent__tte aattgrett_tter..... wteggeatt.tterr..pttoo...te caption_video W"What is person doing?")
127
# Choose the ansver among given possible ansvers
128
anawer select_anssen (info, question, possible_ansvers) return anawer
129
130
INSERT_IN_CONTENN CON.EENNEETTEEAAPLEEE.. EXAMPLES_ HERE
Write function using Python and the VideoSegment class (above) that could be executed to provide ansuer the query.
Consider the following guidelines:
base Python (comparison, sorting) basic logical operatione, lett/right/up//ewn. math, etc.
139
The input to your program video, question and possible answers.
140
Alvays start your function by creating a 'video_segment VideoSegment (video) object.
141
142
INSERT BEEETTTRRRVVUUSSEE PREVIOUS_CODE_AAEE CODE_ ERROR_HERE
143
144
INSERT_ QUERY QUERY_HERE
43


Reviews:
[{"review_id": "lYKSLKUWux", "reviewer": "Reviewer_dZpC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- The paper is clearly written and is easy to follow. The introduction thoroughly motivates the problem by identifying shortcomings for existing works like ViperGPT. The method is concisely presented with examples and intuitions. \n- The paper proposes three useful ideas (mentioned above). Each of these ideas will be potentially useful for researchers working on visual reasoning problems using LLMs. \n- The experimental section has been well thought out. The paper evaluates the overall system using the three proposed ideas on several visual reasoning benchmarks (RefCOCO, RefCOCO+, GQA, Next-QA). The experiments also include evaluation of each individual proposal separately to demonstrate its efficacy. These experiments provide interesting insights that might be useful for researchers. \n\n# Weaknesses:\n\nOverall, I think the paper overstates several contributions/claims. \n\n### Abstract API - how is it creating abstractions?\n\nThe intuition, motivation and justification for introducing an \"Abstract API\" doesn't seem to align with the actual implementation. More concretely, here is the justification for the Abstract API: \n```\nWhen programming, we continuously build new layers of abstraction. We start from a set of primitive\nfunctions and then abstract them away by adding new functions. These are then grouped into libraries, and\nadditional layers of abstraction are built on top. By abstracting away the implementation details, we are able\nto build systems with ever-increasing capabilities and complexity. \n\nMotivated by this, we introduce a set of\nspatially and temporally abstract functions hat abstract away a number of lines of code for the\nsame functionality and together make the Abstract API.\n```\n\nThe actual implementation of the Abstract API involves simply defining additional functions: `get_patch_left_of, get_patch_right_of, get_patch_above_of, get_patch_below_of, etc`. Its unclear to me how this is creating abstractions in the API. \n\nThese functions simply reduce the burden of the LLM. For example, the original ViperGPT would have to synthesize a couple of lines of code to find the \"patch left of\" but now the LLM can simply call this function. This is also clear from the following motivation presented:\n```\nFor example, for the query “return the second person from the right”, the program generated by the LLM correctly sorts the persons along the horizontal axis but then wrongly takes the second index in the array (instead of the second last).\n```\n\nThe additional functions seem to be merely identifying logical sequences that are difficult for the LLM and replacing them with hand-crafted functions in the API. \n\n\n### Issues with automatic generation of in-context examples\n\nI think the idea of ACE is generally useful and seems to work well. However, I think it requires further investigation for reliability. \nThe in-context samples come from LLMs trying to synthesize programs for an (image, question) pair without any context. The only quality control / filtering for these generations is the evaluation using the annotated answers. However, I think several faulty programs could lead to the same answer. \n\nFor example, an image where there are two black dogs and the question is \"what is the color of the second dog from the left?\". Several incorrect programs could lead to the answer \"black\". \nThe paper claims that the overall pipeline is more robust than using human crafted context. I'm not sure if this is true given the unreliability of the in-context samples. \n\nThis could be solved by performing additional analysis on the synthesized programs across several samples. \n\n\n### Applicability of Self-tuning\n```\nIn some cases, we know that code execution failed due to some components of a particular module. For example, the open vocabulary detector fails due to a too high threshold hyperparameter. When the threshold is high, we have a higher number of false negatives. For such cases, we propose to automatically change the hyperparameter of the module (e.g. reduce the threshold) and execute code again.\n```\nThe applicability of this seems to be overstated. The text of the paper only suggests and evaluates one very specific version of this self-tuning idea - **lowering the threshold of object detectors**. There is no discussion on other avenues where this can be used and how it would be applied. \n\nHere is another example where the paper seems to overstate the applicability:\n```\nIn self-tuning, we show that hyper-parameters of certain modules can be tuned automatically if code execution fails due to these modules.\n```\nBased on the paper, there is evidence for only one module that can be tuned. I'm not sure if the results generalize to other possible applications of this idea. At the very least, there needs to be a discussion on the intuition behind why this idea might be more generally applicable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work builds upon the recent success of using LLMs for writing programs to solving visual reasoning tasks. These synthesized programs leverage pre-written functions that allow calling individual pre-trained computer vision models (like object detection, depth estimation, etc). These existing methods query LLMs using expert written programs for solving a few example visual reasoning questions. \n\nThis work addresses the following shortcomings:\n1. The predefined functions in the existing methods focus on higher level semantic tasks like object localization, attribute classification, etc. This work introduces additional \"primitive\" functions. (referred to as Abstract API)\n2. Existing methods include expert written example programs for the LLMs query (for in-context learning). This work proposes a solution to avoid this requirement by automatically synthesizing a few programs using ground truth (image, question, answer) triplets. (referred to as ACE)\n3. Programs generated by existing methods sometimes do not compile/run due to errors. This work proposes to use the error as feedback to re-synthesize a better program. (referred to as self-correction)", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall the ideas presented are sound improvements over the implementation of the original ViperGPT. However, I think the paper claims a very broad significance of the ideas which is not necessarily backed by evidence / discussion. \n\nI believe that since the overall ideas are effective, modifying/rephrasing the claims in the text might be sufficient. Alternatively, if I have misunderstood the significance of these ideas, please provide clarifications.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dkWMfntwpt", "reviewer": "Reviewer_5AHJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The studied direction, visual reasoning, is important, and the authors did several good attempts. Overall the proposed components are reasonable and are composed in a sound way, including abstract API, ACE, and self-correction mechanism. The paper is easy to follow. \n\nThe author conducted extensive experiments in standard datasets including, RefCOCO, RefCOCO+, GQA, and NExT-QA covering three major visual reasoning tasks (visual grounding, compositional image question answering, and video temporal reasoning). Also, extensive quantitatively ablation studies are conducted to verify the effectiveness of the proposed modules.. \n\nI have several major concerns the below:\n- The current design of Abstract API including get_patch_left_of, get_patch_right_of, get_patch_above_of, get_patch_below_of, and more, seems pretty ad-hoc. I expect advanced LLM and VLM and learn to propose them naturally or by asking them firstly to propose those high-level spatial-temporal operators.\n- I am unsure the significance of proposing Automatic generation of in-context examples (Section 2.3). This seems like a trivial solution to me.\n- How does the self-correction (Section 2.4) different to previous literatures such as Reflexion and self-improve llm papers? Only can tune hyperparameters seems not a substantial difference.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a framework enhancing visual reasoning in large language models (LLMs) as controllers. Traditional end-to-end neural networks usually falter in compositional reasoning, generalization, and fine-grained spatial-temporal reasoning. Their approach addresses these challenges by decomposing tasks into subtasks with a set of visual tools for solutions. They propose an \"Abstract API\" for spatial and temporal abstraction, reducing reliance on the LLM's intrinsic reasoning capabilities. Additionally, Their framework automatically generates in-context examples using a small set of labeled instances, eliminating the need for labor-intensive, human-crafted examples. Addtionally, it enables models to perform self-correction through self-debugging and self-tuning mechanisms. In some common visual reasoning tasks, including visual grounding, compositional image question answering, and video temporal reasoning, the experimental results show the strength of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- For Automatic generation of in-context examples, the current way starts from a small set of labeled examples as shown in Section 2.3. Could the requirement for a small set of labeled examples be eliminated?\n- Can the author try GPT-4 to see if the proposed abstract API can still bring improvements?\n- Tuning hyper-parameter is not exactly self-tuning to me. Are there further powerful or fundamental way here for self-tuning? \n- For verifying the effectiveness of the proposed module, can the author provide some qualitative comparisons?", "broader_impact_concerns": "The proposed technicals in this paper have no significant concerns on the ethical aspects.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ij9PtAfbqn", "reviewer": "Reviewer_g5Cs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths of the Submission**\n\n- *Zero-Shot Learning Enhancement*: The paper addresses an important  problem in tool use of multimodal LLM, how to reduce the human burden in creating the in-context examples (ICEs). The elimination of the need for human-engineered  ICEs and the introduction of automatic generation of query-code ICEs (ACEs) is a substantial stride towards true zero-shot learning.\n\n- *The method is clearly described and well motivated*: The implementation of the \"Abstract API\" effectively reduces the complexity of spatial and temporal reasoning for LLMs, thereby improving the overall robustness of the model. The paper's exploration of self-correction through self-debugging and self-tuning is a novel concept, enhancing the model's reliability and reducing dependence on external inputs.\n\n- *Empirical Validation*: The comprehensive testing across various tasks demonstrates the practical effectiveness of the proposed framework in general and ablates the impact of each components.\n\n\n**Weaknesses That Require Attention**\n-  *Limited impact*: Though the proposed method improve ViperGPT, there are several more effective methods for LLM or large multimodal model (LMM) to leverage external tools/API, for example, chaining LLM with multimodal tools such as Visual ChatGPT (Wu et al., 2023), MM-ReAct (Yang et al., 2023b), and training LMM with multimodal tools such as LLaVA-Plus (Liu et al, 2023). It would great to study the effectiveness and generality of the the proposed method with some of these more recent methods.\n\n-  *Missing Literature*: There is an emerge of LMM research and papers in the late half of 2023. The paper's focus seems a bit outdated. Please see the some related survey papers [1,2], and describe the proposed work in the context of up-to-date LMM development.\n\n[1] A Survey on Multimodal Large Language Models\n\n[2] Multimodal foundation models: from specialists to general-purpose assistants\n\n-  *Technical limitations*: (1) Resource Intensity. The framework's complexity, particularly in terms of self-tuning and self-debugging processes, might entail significant computational resources, which could be a barrier in resource-constrained environments. (2) Scalability Issues. The scalability of the proposed approach, especially when dealing with a large number of tools or more complex reasoning tasks, is not clearly addressed, which could be a concern for practical applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a framework to enhance the robustness of Language Model-Led (LLM) programming for visual reasoning tasks.\n\n- **Automatic Generation of Query-Code ICEs (ACEs)**: It removes the necessity for manually crafted in-context examples (ICEs), advancing the models towards genuine zero-shot visual reasoning capabilities. It demonstrates the zero-shot generation of ACEs using a few labeled examples, which, when used as in-context examples, significantly improve performance and eliminate the need for human-designed ICEs.\n\n- **Introduction of \"Abstract API\"**: This API comprises spatially and temporally abstract routines, which ease the spatial and temporal reasoning load on the code-generating LLM, enhancing performance.\n\n- **Self-Correction Capabilities in LLMs**: Showcases the ability of LLMs to perform \"self-correction\" through \"self-debugging\" and \"self-tuning\" methods without requiring ground-truth labels. (1) Self-Debugging: Generating new code from scratch yields better results than using previous query-code pairs as feedback. (2) Self-Tuning: Highlights the automatic tuning of hyperparameters in certain modules if code execution fails.\n\n- **Performance Improvement Across Tasks**: Validates that each component of the proposed framework consistently enhances performance in various compositional question-answering and video temporal reasoning tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Addressing Limited Impact**\n\n- Comparative Analysis: Conduct a comparative study of the proposed method's effectiveness and generality against more recent methods like Visual ChatGPT (Wu et al., 2023), MM-ReAct (Yang et al., 2023b), and LLaVA-Plus (Liu et al, 2023). This will provide a clearer understanding of where your method stands in the current landscape of LLM and LMM advancements.\n\n- Method Integration: Explore the possibility of integrating or chaining your method with these recent multimodal tools. This could potentially enhance the capabilities and applicability of your framework in the broader context of LLM and LMM research.\n\n**Updating Missing Literature**\n\n- Literature Review Expansion: Update the literature review section to include recent developments in LMM research, particularly those outlined in \"A Survey on Multimodal Large Language Models\" and \"Multimodal foundation models: from specialists to general-purpose assistants.\" This will help position your work within the current state of the field.\n\n- Contextual Relevance: Re-evaluate and describe your work in the context of these latest developments. Highlight how your method contributes to or differs from the current trends in LMM.\n\n**Mitigating Technical Limitations**\n\n- Scalability Assessment: Conduct a thorough assessment of the scalability of your approach. Include tests or simulations that demonstrate how your framework performs with an increasing number of self-tuning and self-debugging calls, external tools or in more complex reasoning scenarios. Based on these assessments, provide recommendations or modifications to improve scalability.\n\n- Technical Documentation: Offer detailed technical documentation outlining the computational requirements and practical limitations of the framework. This will help users understand the necessary resources and potential constraints when implementing your method.\n\nThese changes aim to enhance the relevance, applicability, and practical utility of your research, ensuring it aligns with the latest developments in the field and addresses key technical challenges.", "broader_impact_concerns": "Accessibility and Inclusivity: It will increase the impact of this paper if the system's advanced capabilities can be accessible to a diverse user base, including those with limited resource. This work considers several private APIs or high cost models, including PaLM 2 (Anil et al., 2023), OWLv2 (Minderer et al., 2023), PaLI-3 (Chen et al., 2023d). It naturally raise the concerns how the cost is affordable for many university labs. Ensuring the technology is inclusive is an important ethical consideration.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission received three expert reviews.  Post rebuttal, all reviewers agreed that the submission has met the two TMLR acceptance criteria.  The reviewers suggested that three ideas---abstract APIs, the automatic generation of ICEs (ACEs), and self-tuning---could be interesting enough to part of the TMLR audience.  Reviewers also suggested that the submission may have limited significance and impact as there have been other similar works with stronger performance, though this does not impact the fact that the submission has met the criteria to be accepted by TMLR.
Decision: Accept (poster)