Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment

Authors: Chao Wen, Jacqueline Staub, Adish Singla
Primary area: datasets and benchmarks
Keywords: Program Synthesis, Visual Programming, Large Language Models, Multimodal Models, Spatial Reasoning

Abstract:
Large language and multimodal models have shown remarkable success on various benchmarks focused on specific skills such as general-purpose programming, natural language understanding, math word problem-solving, and visual question answering. However, it is unclear how well these models perform on tasks that require a combination of these skills. In this paper, we curate a novel program synthesis benchmark based on the real-world tasks in the XLogoOnline visual programming environment. Each task requires a combination of different skills such as spatial planning, basic programming, and logical reasoning. Our evaluation shows that current state-of-the-art models like GPT-4V and Llama3-70B struggle to solve these tasks, achieving only $20$\% and $2.35$\% success rates, respectively. Next, we develop a fine-tuning pipeline to boost the performance of models by leveraging a large-scale synthetic training dataset with over $80,000$ tasks. Moreover, we showcase how emulator-driven feedback can be used to design a curriculum over training data distribution, through which a fine-tuned Llama3-8B drastically outperforms GPT-4V and Llama3-70B models. Finally, we provide an in-depth failure analysis to understand the limitations of different models. We will publicly release the benchmark for future research on program synthesis in visual programming.

Decision: reject
Reviewer scores: 5, 5, 3, 8, 8

Reviews and discussion:
<thread>
creation date:
1730578787140

thread_type:
official_review

<thread_object>
creator: reviewer_kkkk

<content>
summary:
This paper introduces a new visual reasoning benchmark that requires a combination of skills to be solved (like logic, math, coding, visual understanding). Then, they benchmark current visual language models, and show that they do not perform well on this benchmark. Finally, they propose an approach to improve such models, by finetuning them on more data that follows the distribution of the benchmark, as well as by doing it following a curriculum.

strengths:
1 - This paper contributes a benchmark to study the capabilities of VLMs on program synthesis for visual tasks. The benchmark is challenging enough for current SOTA models to fail at it, but at the same time it is easy enough for humans to solve, making it a good contribution for the community. It also requires a variety of skills that previous benchmarks do not need, making it more general.

2 - The contributions are well motivated and well presented, making it an easy paper to read.

3 - The fine-tuning strategy using curriculum learning based on the harder and easier task is very sensible in this setting where multiple skills need to be learned. The specific implementation is reasonable, and provides very good results.

4 - The analysis (ablation study based on difficulty of the tasks, as well as failure analysis) is informative both in terms of understanding the benchmark better, but also in terms of understanding the models better.  Also, the "transferable skills" section is very pertinent. I was actually thinking about the zero-shot capabilities to unseen tasks before I got to that section.

weaknesses:
I would appreciate if the authors analyzed the following:

1 - What is the drop (or increase) in performance in other standard vision tasks (such as VQA, classification) or even language or coding tasks when fine-tuning on the SIM dataset? It seems like the tasks follow a very specific pattern that may not be very related to other tasks. The paper shows that training on the SIM dataset contributes to tasks that were not part of the dataset, but I wonder if there is any contribution (positive or negative) to other tasks that follow different distributions (even though they may require similar visual or reasoning skills)

2 - What is the role of the prompt engineering in solving these tasks? How much effort and what techniques were used to ensure that the input prompts are not a limitation for the different models that were tested?

questions:
- I would probably not call the REAL dataset "real-world" tasks. They are pretty much very synthetic and toy tasks, not real-world. Maybe you can refer to them as "existing" (or any other word). The term "real-world" was confusing at the beginning.  It would also help if you shoed an example of a SIM task in Figure 1, or in Section 4.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699122213

thread_type:
official_review

<thread_object>
creator: reviewer_nred

<content>
summary:
This paper proposes XLOGOMINIPROG, a program synthesis benchmark in visual programming that requires a combination of different skills. They evaluate the performance of various large models (GPT, Llama, LLaVA) which show a surprisingly low success rate. 
Additionally, they finetune a Llama3-8B model on a large-scale synthetic dataset of 80K visual programming tasks, outperforming GPT-4V and LLama3-70B. Finally, they propose emulator-driven fine-tuning, which designs curriculum over training data distribution by leveraging emulator feedback, i.e., assigning higher weights to tasks where the model struggles. The paper includes an analysis of failure cases, capabilities, and transfer learning skills.

strengths:
* This paper introduces a new program synthesis benchmark for the *visual programming domain*, which currently lacks benchmarking compared to program synthesis benchmarks from natural language or docstrings (e.g., HumanEval, MBPP, APPS).  
* Experiment results are comprehensive across diverse model types and scale, along with an in-depth analysis of various failure types and capabilities. It is interesting to see that current models dominantly lack spatial reasoning, especially GPT-4V.
* Emulator-driven fine-tuning could be extended to other program synthesis domains that contain a wide range of complexity in the training set.
* The paper is well-organized and written clearly with sufficient detail on the dataset statistics, generation process, and experimental setup.

weaknesses:
* The paper proposes a new benchmark focusing on *visual programming*, yet only provides results on two VLM families: GPT-4V and LLaVA-1.5. As there exists a plethora of open-sourced VLMs, I think it would only make the paper better by adding diverse state-of-the-art models (e.g., Llama-3.2-vision, LLaVA-onevision, Phi-3-Vision, Phi-3.5-Vision, QwenVL, Cameleon, LLaVA-next, Pixtral).
* Need clarity in the data generation process. Are there cases where the generated code is correct for the wrong reasons (e.g., logic incorrect, answer correct)? Is the emulator able to filter out these cases during the correctness check? If not, how does that affect the performance of fine-tuned models trained on the synthetic dataset?
* Lacks detailed explanation of the emulator.

questions:
* How do the benchmark results look like on other open-sourced VLMs?
* It's very interesting that fine-tuning a VLM (Llava1.5-13B-Uni) performs worse than fine-tuning an LLM (Llama2, Llama3), while the vision model (GPT-4V) outperforms GPT-4 in the base models experiment. Why do you think this is the case? It would be interesting to compare fine-tuning a more state-of-the-art VLM (e.g., Llama-3.2-vision, LLaVA-onevision).
* How does the emulator validate the soundness of the generated code? For instance, some code may be correct for the wrong reasons (e.g., logic incorrect, answer correct). Is the emulator properly filtering out these cases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720974722

thread_type:
official_review

<thread_object>
creator: reviewer_81e9

<content>
summary:
The paper presents a novel benchmark for evaluating the performance of large language and multimodal models on tasks that require a combination of skills such as spatial planning, basic programming, and logical reasoning within the XLogoOnline visual programming environment.

strengths:
1. Novel Programming Generation Evaluation: The evaluation of programming generation based on the XLogoOnline visual programming environment is somewhat novel, as it integrates vision, language, and coding skills to assess LLMs.

2. Well-Structured Paper Organization: The paper is well-organized, starting with an introduction to the benchmark, followed by a detailed presentation of the training data, and concluding with an evaluation of different performance metrics and analyses.

weaknesses:
1. Lack of Task Challenge: The task presented in this paper appears to lack sufficient challenge, as evidenced by Table 6. Although the zero-shot performance of both closed and open-source models is low, simply synthesizing a large amount of training data can significantly boost performance. This suggests that the task is not particularly challenging. The poor performance of current models can be attributed to their lack of training with this specific data. A more detailed analysis of task difficulty is needed to solve this issue.

2. Narrow Scope of Visual Programming: The use of the XLogoOnline environment limits the code action space, as illustrated in Figure 1, which further contributes to the task's lack of challenge. Additionally, other visual programming languages, such as Scratch, are not considered in this work. This narrow focus limits the domain of the study, making it insufficient for a top-tier conference like ICLR.

3. Insufficient Evaluation of Large Multimodal Models: The paper evaluates only three multimodal models, leaving unclear whether current state-of-the-art closed and open-source multimodal models, such as Claude3.5, Gemini, and Qwen2-VL, could achieve better performance. A more comprehensive evaluation with a broader range of models is necessary to draw meaningful conclusions.

questions:
Check Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730973950623

thread_type:
official_review

<thread_object>
creator: reviewer_znzb

<content>
summary:
The paper introduces a new benchmark, which is designed to evaluate the performance of large language models on program synthesis tasks within the XLogoOnline visual programming environment. The benchmark includes real-world and synthetic tasks that require a combination of skills such as spatial planning, basic programming, and logical reasoning. The study found that current state-of-the-art models like GPT-4V and Llama3-70B struggle with these tasks, achieving low success rates. To improve model performance, the authors developed a fine-tuning pipeline using a large-scale synthetic training dataset, which significantly boosted the performance of the Llama3-8B model. Additionally, they showcased how emulator-driven feedback can be used to design a curriculum over training data distribution, further enhancing the performance of fine-tuned models. The paper provides an in-depth failure analysis to understand model limitations and will publicly release the benchmark for future research on program synthesis in visual programming.

strengths:
It introduces XLOGOMINIPROG, a new benchmark that tests multiskill tasks in visual programming, an area where current models perform poorly.
It demonstrates the effectiveness of emulator-driven feedback for designing a dynamic training curriculum, further boosting model performance.
Detailed experiments were carried out

weaknesses:
The emulator-driven fine-tuning provides only binary correctness feedback on the predicted code, which might not be sufficient for identifying and correcting specific errors in the generated code. Can it be in natural language?
There's a risk that models could overfit to the synthetic data used in training, which might not perfectly mimic the complexity and variability of real-world programming challenges. I think this is the most serious disadvantage

questions:
no questions
</content>
</thread_object>

</thread>
<thread>
creation date:
1731072278057

thread_type:
official_review

<thread_object>
creator: reviewer_miwe

<content>
summary:
This paper
1. curates a small (85 tasks) program synthesis benchmark based on real-world tasks in the XLogoOnline visual programming environment, which requires a combination of different skills;
2. shows that current SotA models like GPT-4V and Llama3-70B struggle to solve such tasks;
3. develops a data synthesis pipeline with curriculum learning based on task difficulty for the model to significantly boost the model performance on the benchmark.

strengths:
1. This paper identifies a new task that current SotA models fail on.

weaknesses:
1. This paper provides few new insights about identifying models' limitations: This paper shows that existing models fail in visual programming in XLogoOnline. But this task is similar to some tasks in works like MMMU, which also combine different skills like vision and reasoning and have already shown that current models struggle in visual reasoning tasks
2. This paper also lacks novelty about improving models' performance: 
    1. The data synthesis method is based on sampling and filtration, which has been widely adopted in works like STaR, RaFT and Rest-EM.
    2. The curriculum learning design adaptive to task difficulty has been explored by works like DART-Math.

questions:
1. To the best of my knowledge, OpenAI has never disclosed the parameter sizes of their models since GPT-3.5. Where is the statement of "GPT3.5 model with 175B parameters (version gpt-3.5-turbo-0125)" (line 335-336) from?
2. Which version of Llama3 models are used? 3 or 3.1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732201888972

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers and AC,

We would like to express our sincere gratitude to all the reviewers and the AC for their feedback and constructive suggestions on our manuscript. During the rebuttal period, we have focused on addressing these valuable feedback, incorporating additional experiments, and refining our manuscript. We hope that our current revised manuscript can address all reviewers' concerns.

As the reviewers highlighted, we believe our paper introduces a novel benchmark for the visual programming domain (**Reviewer znzb, Reviewer nRED, Reviewer KKKK**), which is easy for humans yet challenging for current SOTA models (**Reviewer KKKK**).

We appreciate the reviewers' feedback that our manuscript is well-written and organized (**Reviewer miWe, Reviewer 81e9, Reviewer nRED, Reviewer KKKK**), provides sufficient details on datasets and experiments (**Reviewer nRED**), and offers analysis that provides new insights (**Reviewer nRED, Reviewer KKKK**).

We also thank the reviewers for raising concerns regarding the challenges posed by our benchmark (**Reviewer miWe, Reviewer 81e9**), the evaluation of additional multimodal models (**Reviewer 81e9, Reviewer nRED**), the potential overfitting on synthetic data (**Reviewer znzb**), the clarification of the data generation process and emulator (**Reviewer nRED**), prompt engineering (**Reviewer KKKK**), and the impact of fine-tuning on other benchmarks (**Reviewer KKKK**).

In response, these concerns have been addressed in the rebuttal. Additionally, we have also carefully revised and enhanced our manuscript with the following key updates:

- **[All Reviewers]** We moved the Appendix from the Supplementary Material (zip file) to the end of the references in the main paper to make it easier for reviewers to access.
- **[Reviewer miWe]** We have removed the mention of GPT-3.5 parameters from the paper (Section 5.1).
- **[Reviewer 81e9, Reviewer nRED]** We added the evaluation of 7 additional state-of-the-art multimodal models (Figure 6).
- **[Reviewer nRED]** We added details of the emulator (Appendix C).
- **[Reviewer KKKK]** We added examples of tasks from the synthetic SIM dataset (Figure 12).
- **[Reviewer KKKK]** We added additional experiments on the impact of fine-tuning on other benchmarks (Appendix D.4).

For easier review, these updates are temporarily highlighted in **orange** in our manuscript.

We hope our responses and revisions address all the reviewers' concerns, and we are eager to engage in further discussions regarding these updates.

Thank you,  
The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733155053070

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General reminder

comment:
Dear Reviewers,

We thank all the reviewers for their comments and feedback. As the discussion period ends in less than 24 hours, we hope our responses below have addressed the concerns raised in your reviews. If there are any remaining issues or points that need clarification, we would be happy to address them within the remaining time.

Thank you,
The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a new benchmark to evaluate the performance of LLM on program synthesis tasks within the XLogoOnline visual environment. The primary concerns raised include:

1. Limited technical novelty

2. Insufficient experimental evaluation

 While the paper has notable merit, it is not recommended for acceptance. The authors are advised to incorporate the reviewers' feedback when revising the paper for submission to other venues.