Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment

## Abstract

Abstract Large language and multimodal models have shown remarkable successes on various benchmarks focused on specific skills such as general-purpose programming, natural language understanding, math word problem-solving, and visual question answering. However, it is unclear how well these models perform on tasks that require a combination of these skills. In this paper, we curate a novel program synthesis benchmark based on the XLogoOnline visual programming environment. The benchmark comprises 85 85 85 85 real-world tasks from the Mini-level of the XLogoOnline environment, each requiring a combination of different skills such as spatial planning, basic programming, and logical reasoning. Our evaluation shows that current state-of-the-art models like GPT-4V and Llama3-70B struggle to solve these tasks, achieving only 20 20 20 20 % and 2.35 2.35 2.35 2.35 % success rates. Next, we develop a fine-tuning pipeline to boost the performance of models by leveraging a large-scale synthetic training dataset with over 80 , 000 80 000 80,000 80 , 000 tasks. Moreover, we showcase how emulator-driven feedback can be used to design a curriculum over training data distribution. We showcase that a fine-tuned Llama3-8B drastically outperforms GPT-4V and Llama3-70B models, and provide an in-depth analysis of the models’ expertise across different skill dimensions. We will publicly release the benchmark for future research on program synthesis in visual programming.

## 1 Introduction

In recent years, large models have shown remarkable performance in various domains, such as general-purpose programming and visual question answering []. For instance, in programming, numerous tools and models use large language models (LLMs) for code generation [, ] and programming feedback generation [, , ], revolutionizing how programmers write code and how teachers instruct programming [, ]. Beyond text-based tasks, the focus has expanded to multimodal models that process and generate not only text but also images, achieving significant success in domains such as visual question answering [] and text-to-image generation [].

Despite these successes, the performance of large models on tasks that require a combination of skills remains unclear. Real-world tasks often demand a blend of skills. For example, a typical task like “navigating to the kitchen to fetch ten apples” involves spatial reasoning to understand the environment and plan a path around obstacles, together with basic arithmetic to ensure that exactly ten apples are retrieved. This example illustrates the multifaceted nature of real-world tasks. While various benchmarks focus on specific skills [, , , ], there is a lack of benchmarks evaluating how large models perform on tasks that require a combination of different skills.

To bridge this gap, we introduce XLogoMiniProg, a benchmark for program synthesis in the visual programming domain. Our benchmark is constructed using the Mini-level of the XLogoOnline platform [], featuring $85$ real-world visual programming tasks, each demanding a blend of diverse skills. Figure 1 illustrates examples of these tasks. Each task includes a visual grid with a turtle that needs to be directed to complete a specific goal. For example, in Task 28, the goal is to direct the turtle to collect all red shapes without stepping on the color green, requiring logical reasoning, spatial reasoning, planning, and basic programming skills. Task 38 requires additional math word problem-solving to collect $10$ strawberries. These tasks provide a testbed for evaluating how large models perform on tasks that require a combination of skills, presenting a unique challenge to current large models.

Figure: Task 28: Collect all red shapes without standing on the color green.
Refer to caption: extracted/5672133/figs/1_intro/xlogo-exercise-28-finallforbid.png

## 2 Related Work

Program synthesis benchmarks for large models.
Program synthesis aims to automatically generate programs from specifications. Recently, numerous recent works have focused on training large models specifically for program synthesis [, , , ]. To evaluate these large models, many program synthesis benchmarks have been developed, such as HumanEval [], MBPP [], and APPS []. However, these benchmarks focus on generating code from natural language or docstrings for general programming languages such as Python [, , ]. Our benchmark focuses on program synthesis in the visual programming domain. While our benchmark covers basic programming like loops and variables, it requires models to combine spatial, logical, and programming skills, posing unique challenges not addressed by these program synthesis benchmarks.

Large models for visual programming.
Visual programming has been studied in various scenarios, such as task synthesis [, , , ], program synthesis [, ], and student modeling []. With the rise of large models, some initial works evaluate ChatGPT [] and GPT-4 [] in these scenarios, showing that large models struggle with visual programming tasks [, , ]. In contrast, we provide a comprehensive benchmark for evaluating large models for program synthesis in visual programming, considering a wider range of models and skills.

Spatial reasoning and planning benchmarks.
Existing benchmarks for spatial reasoning and planning are primarily designed for reinforcement learning agents to solve sequential decision-making tasks [, ]. Additionally, some benchmarks aim to evaluate models in domains where spatial reasoning and planning skills are essential, such as visual navigation and object interaction [, ]. With the advent of large models, recent works have also begun to evaluate LLMs’ capabilities in spatial reasoning and planning [, ]. Our benchmark, however, focuses on the visual programming domain, which requires a broader range of skills beyond spatial reasoning and planning, including logical reasoning, math word problem-solving, and programming skills.

## 3 Background and Synthesis Objective

In this section, we provide the background on the XLogoOnline visual programming platform and then introduce the program synthesis objective.

### 3.1 Background on XLogoOnline-Mini Programming

XLogoOnline [] is a visual programming platform based on Logo programming language [] and is widely used by tens of thousands of students every year [, ]. In this work, we focus on the Mini-level (XLogoOnline-Mini). In XLogoOnline-Mini, each task includes a text description of the goal and code constraints, along with a two-dimensional visual grid. The visual grid features a turtle and various elements such as fruits, shapes, colors, lines, walls, and forbidden areas. To solve the task, one needs to write a program to direct the turtle’s movement in the visual grid to achieve the specified goal. Figure 1 shows illustrative examples of tasks, the required skills, and solution codes.

Required skills for XLogoOnline-Mini.
We examine the skills required for solving visual programming tasks in XLogoOnline-Mini. Specifically, the visual programming tasks in our domain cover the following skills:
(i) Logic: Understand underlying logical relationships specified in the goal; (ii) Math: Apply basic arithmetic to solve the task; (iii) Draw: Identify patterns and generate the corresponding code; (iv) Basic actions: Move and change directions using only basic commands; (v) Loops: Utilize loops to repeat commands multiple times; (vi) Variables: Utilize variables to set and update colors to draw lines with a specific color; (vii) Loops and Variables: Integrate loops with variables to solve a task; (viii) Code Constraints: Adhere to specific code constraints such as maximum code length.

Figure: (a) Task distribution of XLogoMiniProg:Real.
Refer to caption: x2.png

### 3.2 Program Synthesis Objective

Next, we formally define task and code specifications, and introduce our synthesis objective.

Task specifications. In XLogoOnline-Mini, a task $\texttt{{T}}:=(\texttt{G},{\texttt{L}},\texttt{W})$ consists of a goal G, code constraints L, and a visual grid world W. The goal G defines the turtle’s objective. The code constraints L specify the requirements for a solution code. There are five types of constraints for code: None (no restrictions), AtMost (maximum number of commands), Exactly (exact number of commands), StartBy (initial command sequence), and Hybrid (combination of constraints). The visual grid world W is a 2-dimensional visual grid featuring a turtle and various elements. We define the grid size as the maximum dimension of the grid. For example, in Figure 1 (Task 87), the goal is “Find the strawberry”, the code constraint is “use just 6 commands” (AtMost), and the visual grid world depicts a $3\times 4$ grid ($size=4$) with a turtle, a strawberry, and forbidden areas marked by gray cells.

Code specifications. The code space of XLogoOnline-Mini is defined by the domain-specific language (DSL) depicted in Figure 2(c). Note that while the DSL defines the formal structure and syntax, we implement it using Python-style code representation to leverage the large models’ pre-trained knowledge on Python programming. A *solution code* for a given task is the code that meets the task’s code constraints and achieves the specified goal when executed in the visual grid world. In Figure 1, a solution code is provided below each task.

Program synthesis objective.
Our objective is to develop a synthesizer function, $f:\texttt{{T}}\rightarrow\texttt{{C}}$, which generates a solution code C for a given visual programming task T. To evaluate $f$ on a task T, we first use $f$ to synthesize a code $\hat{\texttt{{C}}}$ and then execute the synthesized code $\hat{\texttt{{C}}}$ using an emulator. The emulator outputs *success* if the synthesized code $\hat{\texttt{{C}}}$ successfully solves the task T and adheres to code constraints; otherwise, the emulator outputs *fail*. We use *success* as the main evaluation metric. Given a dataset $\mathcal{D}_{\text{eval}}=\{\texttt{{T}}_{i}\}_{i=1}^{N}$, we calculate the success rate of $f$ on this dataset as the overall performance.
We curate a dataset XLogoMiniProg:Real of $N=85$ real-world visual programming tasks from XLogoOnline-Mini and we use this as the main dataset for evaluation. In Figures 2(a) and 2(b), we show the overall distribution of this dataset and the number of tasks requiring specific skills, respectively.

## 4 Methodology for Synthetic Dataset Generation and Fine-tuning

As discussed in Section 1, existing large models such as GPT-4V and Llama3-70B struggle with visual programming tasks in XLogoOnline-Mini. To address this, we develop a two-stage fine-tuning pipeline consisting of synthetic dataset generation and supervised fine-tuning. This section details the dataset generation process and the methodology for fine-tuning large models on the synthetic dataset.

Figure: (a) Task distribution of XLogoMiniProg:Sim.
Refer to caption: x3.png

### 4.1 Synthetic Dataset Generation

Our goal is to develop a large synthetic dataset for training models []. To achieve this, we adopt the task synthesis techniques from [, ], which were developed to automatically generate high-quality tasks in visual programming domains. Instead of random task generation, these techniques allow us to perform more controlled and systematic task synthesis, such as specifying task types, code concepts, and code lengths, enabling us to generate tasks with different skills and difficulty levels.

Adapting task synthesis techniques.
Given a task-code pair as a reference input, the original task synthesis techniques can produce a small, predefined number of tasks and solution codes suited for educational purposes [, ]. Since our goal is to develop a large and diverse dataset for training, we make two key modifications: (i) we remove scoring functions used to filter out low-score tasks, enabling us to generate a large quantity of tasks instead of a limited selection for educational uses; (ii) we relax task synthesis parameters to enhance techniques’ ability to generate more tasks, including allowing larger grid sizes and longer code lengths. While the resulting tasks may not be ideal for educational purposes, they are diverse and challenging for training large models.

Dataset generation process and statistics.
We use the adapted task synthesis technique to generate a synthetic dataset as follows: (i) we manually craft a solution code for each task in the XLogoMiniProg:Real dataset, resulting in a set $\{(\texttt{{T}}_{i},\texttt{{C}}_{i})\}_{i=1}^{85}$; (ii) for each $(\texttt{{T}}_{i},\texttt{{C}}_{i})$, we generate up to $1,500$ synthesized tasks and their solution codes. To ensure the quality of the dataset, we take the following processing steps: we remove any duplicate task-code pairs to maintain diversity, conduct a correctness check on the generated solution codes using the emulator, and exclude any task-code pairs present in the real-world XLogoMiniProg:Real dataset from our synthetic dataset. This last processing step guarantees that the model has not seen any tasks from the evaluation dataset during training. We ultimately produce the synthetic dataset XLogoMiniProg:Sim with $89,053$ task-code pairs. The statistics of this dataset are detailed in Figure 3. Note that the distribution of this synthetic dataset slightly differs from the real-world dataset XLogoMiniProg:Real due to the aforementioned processing steps and the fact that not all reference tasks can generate the desired number of synthesized tasks. From this synthetic dataset, we randomly select $1,000$ samples for validation, $1,000$ samples for evaluation, and the remaining samples for training. We will provide full details of the dataset generation process in the supplementary material.

### 4.2 Methodology for Fine-tuning

Translating tasks and codes.
In our synthetic dataset, tasks and codes are represented in JSON format for ease of parsing and interpretation. However, directly using the JSON format can be challenging for training large models, which are typically pre-trained on natural language texts.
Therefore, we translate the JSON representations of each task and code into natural language descriptions and Python-style code, respectively, using a fixed template shown in Figure 4(a).

Supervised fine-tuning using synthetic dataset.
Fine-tuning can involve adjusting all model parameters, modifying only a few layers, or adding new layers []. However, fully fine-tuning all parameters can be computationally expensive and time-consuming. Therefore, we adopt Low-Rank Adaptation (LoRA) [], a parameter efficient fine-tuning technique which introduces trainable rank decomposition matrices into the model’s network weights.
We train models on the XLogoMiniProg:Sim dataset using LoRA in a standard supervised manner. The model receives a natural language task description as input and outputs Python-style code. The model is optimized to minimize the cross-entropy loss between the predicted code and the ground truth solution code.

Figure: (a) Prompt template.
Refer to caption: x4.png

Emulator-driven feedback for fine-tuning.
Standard supervised fine-tuning assigns equal weights to all samples in the training dataset. However, our domain presents a unique challenge where tasks vary widely in required skills and difficulty levels (see Figure 3). Additionally, some skills serve as prerequisites for mastering more advanced ones. For instance, a model typically needs to understand basic actions before mastering loops and variables, and it generally solves tasks with shorter code lengths before being able to tackle longer ones. Consequently, treating all tasks with equal importance can be suboptimal in our setting. To address this, we introduce emulator-driven fine-tuning, which designs a curriculum over training data distribution by leveraging emulator feedback. The key idea is to dynamically adjust the training data distribution based on the emulator’s evaluation of the model, assigning higher weights to tasks where the model struggles, thereby progressively guiding the model from simpler tasks that it can easily solve to more complex tasks.

The overall process is shown in Figure 4(b) and 4(c). More formally, given an initial model $f_{0}$ and the training dataset $\mathcal{D}$, our goal is to learn a final model $f_{K}$. To achieve this, at each training epoch $k$, we first perform the *emulator-driven resampling* step (see Figure 4(c)), where we use the model $f_{k}$ to infer on the training dataset $\mathcal{D}$ to obtain the predicted code $\hat{\texttt{{C}}}_{i}$ for each task $\texttt{{T}}_{i}$. We evaluate each predicted code using an emulator and update the weight $w_{i}$ for $(\texttt{{T}}_{i},\texttt{{C}}_{i})$ as follows:

$$ $w_{i}=\frac{1}{|\mathcal{D}|}\big{[}1+\beta\cdot\mathbb{I}\big{(}\text{ Emulator}(\texttt{{T}}_{i},\hat{\texttt{{C}}}_{i})=\emph{fail}\big{)}\big{]},$ (1) $$

where $\mathbb{I}(\cdot)$ is an indicator function that returns $1$ if the predicted code fails to solve $\texttt{{T}}_{i}$, and 0 0 otherwise. The hyperparameter $\beta$ is adjustable, with a larger $\beta$ encouraging the model to focus more on its mistakes and $\beta=0$ equivalent to fine-tuning without resampling. Then, we sample the training dataset $\mathcal{D}$ according to the categorical distribution $w_{i}^{\prime}=w_{i}/{{\scriptstyle\sum_{j=1}^{|\mathcal{D}|}}w_{j}}$, obtaining a resampled dataset $\mathcal{D}_{k}$. After resampling, we perform the *training* step, where we train the model $f_{k}$ using the resampled dataset $\mathcal{D}_{k}$ to obtain the model $f_{k+1}$. Finally, we repeat the resampling and training steps until the model converges or reaches a predefined number of training epochs, yielding the final model $f_{K}$. To reduce computational costs, resampling can be performed at fixed intervals (set to $3$ epochs in our experiments).

## 5 Experimental Evaluations

In this section, we evaluate the performance of large models and fine-tuned models on XLogoMiniProg benchmark. We first describe the experimental setup in Section 5.1, and then present the main results in Section 5.2, followed by additional results and analysis in Section 5.3.

### 5.1 Experimental Setup

Models evaluated. We compare a range of large models and their fine-tuned versions. All models are queried with temperature 0 0. We evaluate the following models:

- •
*GPT family base models.* We consider the following models from OpenAI’s GPT family: (i) GPT-3.5 model with $175$B parameters (version gpt-3.5-turbo-0125) []; (ii) GPT-4 model (version gpt-4-turbo-2024-04-09) []; and (iii) GPT-4V model, which is identical to GPT-4, but we incorporate a task image as additional input to leverage its vision capabilities (version gpt-4-turbo-2024-04-09) [].
- •
*Llama family base models.* We consider the following models from Meta’s Llama2 and Llama3 family: (i) Llama2-7B, Llama2-13B and Llama2-70B models with 7B, 13B, and 70B parameters, respectively (version instruction-tuned) []; (ii) Llama3-8B and Llama3-70B models with 8B and 70B parameters, respectively (version instruction-tuned) [].
- •
*Fine-tuned models.*
We fine-tune the Llama2-7B and Llama3-8B base models (non-instruction-tuned versions) using our synthetic dataset. We choose non-instruction-tuned versions because the base models will be fine-tuned to generate code, without requiring instruction-following capabilities.
For each base model, we use standard fine-tuning and emulator-driven fine-tuning. Llama3-8B-Uni is fine-tuned on our synthetic training dataset with uniform data distribution (i.e., standard fine-tuning). Llama3-8B-Emu is fine-tuned on the same dataset with emulator-driven resampling in Section 4. We apply the same fine-tuning procedures to Llama2-7B base models (non-instruction-tuned versions), yielding Llama2-7B-Uni and Llama2-7B-Emu. We will provide additional fine-tuning details in the supplementary material.

Evaluation procedure and metrics.
We evaluate models using two datasets: XLogoMiniProg:Real and the synthetic evaluation dataset (see Figure 3(c)), which we refer to as XLogoMiniProg:Sim-Eval. For each task in our evaluation datasets, we first convert the task from JSON format into natural language description using a fixed prompt template. For GPT-4V model, we also provide an image of the task as additional input to the model. Then, we use the model to generate the code in Python programming language. However, the model might produce the natural language explanation alongside code. We extract only the Python code from the models’ outputs. Finally, we run the extracted code using an emulator and evaluate the model. We use *success* as the main metric (see Section 3.2), and also consider two additional metrics: (i) Format, which evaluates if the model’s output adheres to the desired code format, and (ii) No-Crash, which evaluates if the code runs without crashing such as hitting walls, entering forbidden areas, or exceeding grid boundaries.
To evaluate a model across various dimensions, such as skills and task types, we automatically categorize each task-code pair according to different dimensions (e.g., task type). We determine the model’s capability in a specific aspect within a dimension (e.g., math in the task type dimension) by calculating the success rates for all tasks involving that aspect.

### 5.2 Main Results

The results are shown in Figure 5. For fine-tuned models, we report the mean and standard error of the performance across $5$ different random seeds.

Figure: Figure 5: Performance comparison of base and fine-tuned models on two evaluation datasets.

Base models’ performance. Among the base models evaluated, GPT-4V performs the best with a success rate of $20.00\%$ on the XLogoMiniProg:Real.
Notably, GPT-4V outperforms GPT-4, which has a success rate of $12.94\%$. This suggests that incorporating visual information can enhance the performance of large models on visual programming tasks. However, all other base models, including GPT-3.5 and Llama models, perform poorly on XLogoMiniProg:Real. Regarding the synthetic evaluation dataset XLogoMiniProg:Sim-Eval, we find that the performance of most base models declines. This is because the tasks in XLogoMiniProg:Sim-Eval are more challenging than those in XLogoMiniProg:Real in terms of code length and grid size (see Figure 2 and 3). For example, the calculated percentage of code with length “Long (11-17)” in XLogoMiniProg:Sim-Eval is $25.14\%$, compared to $17.65\%$ in XLogoMiniProg:Real.

Effectiveness of fine-tuning. Standard fine-tuning on a domain-specific dataset significantly enhances the performance of base models. As shown in Figure 5, after standard fine-tuning, the success rate for Llama3-8B-Uni is $54.12\%$ on XLogoMiniProg:Real and $62.72\%$ on XLogoMiniProg:Sim-Eval. Similar improvements are observed for Llama2-7B-Uni. We also note that the performance of fine-tuned models on XLogoMiniProg:Real generally lags behind their performance on XLogoMiniProg:Sim-Eval. This is because the task distribution of XLogoMiniProg:Sim-Eval more closely resembles the training dataset due to the dataset split. Our results also show that emulator-driven resampling effectively enhances fine-tuning performance.
Llama3-8B-Emu achieves a success rate of $60.23\%$ and $66.92\%$ on XLogoMiniProg:Real and XLogoMiniProg:Sim-Eval, respectively, outperforming Llama3-8B-Uni by $6.11\%$ and $4.20\%$. Similar improvements are observed when comparing Llama2-7B-Uni and Llama2-7B-Emu, indicating that our resampling technique consistently boosts fine-tuning performance across different base models.

### 5.3 Additional Results and Analysis

Comparative analysis of models’ capabilities.
In Figure 6, we present a comparative analysis of three representative models—GPT-4V, Llama3-70B, and Llama3-8B-Emu—across distinct dimensions using the XLogoMiniProg:Real dataset. Overall, Llama3-8B-Emu consistently outperforms other models across all dimensions and GPT-4V shows superior performance compared to Llama3-70B in most aspects. In Figure 6(a), 6(b), and 6(c), we find that the GPT-4V and Llama3-70B struggle most with tasks of type “logic” and with scenarios that impose code constraints such as AtMost and Exactly. In terms of the code concepts, Llama3-70B fails to solve any tasks that require variables, showing its limitations in handling complex programming concepts in visual programming. Notably, as shown in Figure 6(d), the performance of all models declines with the increasing difficulty of tasks, as indicated by longer code lengths and larger grid sizes. GPT-4V fails to solve tasks requiring long code sequences or grid sizes larger than $6$. Llama3-70B performs even more poorly, starting to fail on tasks requiring medium-length codes and grid sizes larger than $3$.

Figure: (a) Task Type
Refer to caption: x6.png

Figure: Figure 7: Success rates of fine-tuned models across different training epochs.
Refer to caption: x11.png

Fine-tuning performance across different epochs.
Figure 7 illustrates the performance of fine-tuned models across different epochs. For the emulator-driven fine-tuning (Emu), we adjust the resampling interval to every three epochs, specifically at epochs 3 and 6. At epoch 3, we reuse the checkpoint from the standard fine-tuning (Uni) to save time and resources. As a result, the performance of the emulator-driven fine-tuning (Emu) matches that of the corresponding standard fine-tuning (Uni) up until epoch 3. Then, an emulator-driven resampling is performed at epoch 3, leading to further performance improvements compared to models without resampling. Notably, at the end of training, Llama2-7B-Emu achieves performance close to that of Llama3-8B-Uni, despite the latter being fine-tuned on a more advanced base model. This demonstrates the effectiveness of the curriculum designed by emulator-driven resampling in enhancing the performance of standard fine-tuning.

## 6 Concluding Discussion

Summary.
In this paper, we introduced the XLogoMiniProg benchmark to evaluate the program synthesis capabilities of large models within the XLogoOnline visual programming environment. We found that large models struggle with visual programming tasks that require a combination of skills, despite our benchmark tasks only requiring basic programming skills. Our best evaluated base model, GPT-4V, only achieved a $20\%$ success rate. To improve performance, we developed a fine-tuning pipeline that involves synthetic dataset generation followed by supervised fine-tuning. This pipeline enabled the Llama3-8B model to achieve a success rate of $54.12\%$ on the benchmark tasks. Additionally, we demonstrated that leveraging emulator-driven feedback can further enhance standard fine-tuning performance by approximately $6\%$ in both Llama3-8B and Llama2-7B models.

Limitations and future work.
We discuss some limitations of our work and propose ideas for addressing them in the future. First, we observe that GPT-4V outperforms GPT-4, indicating that incorporating visual information can enhance the ability of large models to understand visual and spatial relationships in tasks. In the future, one can fine-tune large vision models and evaluate their performance on our benchmark. Second, our emulator-driven fine-tuning provides the model with only binary feedback on the correctness of the predicted code. In the future, it would be interesting to provide more detailed feedback, such as identifying specific errors in the generated code and then using this more informative feedback to guide the fine-tuning process.

## Acknowledgments and Disclosure of Funding

Funded/Co-funded by the European Union (ERC, TOPS, 101039090). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 4
questions: 1. To the best of my knowledge, OpenAI has never disclosed the parameter sizes of their models since GPT-3.5. Where is the statement of "GPT3.5 model with 175B parameters (version gpt-3.5-turbo-0125)" (line 335-336) from?
2. Which version of Llama3 models are used? 3 or 3.1?
soundness: 3
strengths: 1. This paper identifies a new task that current SotA models fail on.
summary: This paper
1. curates a small (85 tasks) program synthesis benchmark based on real-world tasks in the XLogoOnline visual programming environment, which requires a combination of different skills;
2. shows that current SotA models like GPT-4V and Llama3-70B struggle to solve such tasks;
3. develops a data synthesis pipeline with curriculum learning based on task difficulty for the model to significantly boost the model performance on the benchmark.
weaknesses: 1. This paper provides few new insights about identifying models' limitations: This paper shows that existing models fail in visual programming in XLogoOnline. But this task is similar to some tasks in works like MMMU, which also combine different skills like vision and reasoning and have already shown that current models struggle in visual reasoning tasks
2. This paper also lacks novelty about improving models' performance: 
    1. The data synthesis method is based on sampling and filtration, which has been widely adopted in works like STaR, RaFT and Rest-EM.
    2. The curriculum learning design adaptive to task difficulty has been explored by works like DART-Math.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: no questions
soundness: 2
strengths: It introduces XLOGOMINIPROG, a new benchmark that tests multiskill tasks in visual programming, an area where current models perform poorly.
It demonstrates the effectiveness of emulator-driven feedback for designing a dynamic training curriculum, further boosting model performance.
Detailed experiments were carried out
summary: The paper introduces a new benchmark, which is designed to evaluate the performance of large language models on program synthesis tasks within the XLogoOnline visual programming environment. The benchmark includes real-world and synthetic tasks that require a combination of skills such as spatial planning, basic programming, and logical reasoning. The study found that current state-of-the-art models like GPT-4V and Llama3-70B struggle with these tasks, achieving low success rates. To improve model performance, the authors developed a fine-tuning pipeline using a large-scale synthetic training dataset, which significantly boosted the performance of the Llama3-8B model. Additionally, they showcased how emulator-driven feedback can be used to design a curriculum over training data distribution, further enhancing the performance of fine-tuned models. The paper provides an in-depth failure analysis to understand model limitations and will publicly release the benchmark for future research on program synthesis in visual programming.
weaknesses: The emulator-driven fine-tuning provides only binary correctness feedback on the predicted code, which might not be sufficient for identifying and correcting specific errors in the generated code. Can it be in natural language?
There's a risk that models could overfit to the synthetic data used in training, which might not perfectly mimic the complexity and variability of real-world programming challenges. I think this is the most serious disadvantage

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Check Weaknesses.
soundness: 2
strengths: 1. Novel Programming Generation Evaluation: The evaluation of programming generation based on the XLogoOnline visual programming environment is somewhat novel, as it integrates vision, language, and coding skills to assess LLMs.

2. Well-Structured Paper Organization: The paper is well-organized, starting with an introduction to the benchmark, followed by a detailed presentation of the training data, and concluding with an evaluation of different performance metrics and analyses.
summary: The paper presents a novel benchmark for evaluating the performance of large language and multimodal models on tasks that require a combination of skills such as spatial planning, basic programming, and logical reasoning within the XLogoOnline visual programming environment.
weaknesses: 1. Lack of Task Challenge: The task presented in this paper appears to lack sufficient challenge, as evidenced by Table 6. Although the zero-shot performance of both closed and open-source models is low, simply synthesizing a large amount of training data can significantly boost performance. This suggests that the task is not particularly challenging. The poor performance of current models can be attributed to their lack of training with this specific data. A more detailed analysis of task difficulty is needed to solve this issue.

2. Narrow Scope of Visual Programming: The use of the XLogoOnline environment limits the code action space, as illustrated in Figure 1, which further contributes to the task's lack of challenge. Additionally, other visual programming languages, such as Scratch, are not considered in this work. This narrow focus limits the domain of the study, making it insufficient for a top-tier conference like ICLR.

3. Insufficient Evaluation of Large Multimodal Models: The paper evaluates only three multimodal models, leaving unclear whether current state-of-the-art closed and open-source multimodal models, such as Claude3.5, Gemini, and Qwen2-VL, could achieve better performance. A more comprehensive evaluation with a broader range of models is necessary to draw meaningful conclusions.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: * How do the benchmark results look like on other open-sourced VLMs?
* It's very interesting that fine-tuning a VLM (Llava1.5-13B-Uni) performs worse than fine-tuning an LLM (Llama2, Llama3), while the vision model (GPT-4V) outperforms GPT-4 in the base models experiment. Why do you think this is the case? It would be interesting to compare fine-tuning a more state-of-the-art VLM (e.g., Llama-3.2-vision, LLaVA-onevision).
* How does the emulator validate the soundness of the generated code? For instance, some code may be correct for the wrong reasons (e.g., logic incorrect, answer correct). Is the emulator properly filtering out these cases?
soundness: 4
strengths: * This paper introduces a new program synthesis benchmark for the *visual programming domain*, which currently lacks benchmarking compared to program synthesis benchmarks from natural language or docstrings (e.g., HumanEval, MBPP, APPS).  
* Experiment results are comprehensive across diverse model types and scale, along with an in-depth analysis of various failure types and capabilities. It is interesting to see that current models dominantly lack spatial reasoning, especially GPT-4V.
* Emulator-driven fine-tuning could be extended to other program synthesis domains that contain a wide range of complexity in the training set.
* The paper is well-organized and written clearly with sufficient detail on the dataset statistics, generation process, and experimental setup.
summary: This paper proposes XLOGOMINIPROG, a program synthesis benchmark in visual programming that requires a combination of different skills. They evaluate the performance of various large models (GPT, Llama, LLaVA) which show a surprisingly low success rate. 
Additionally, they finetune a Llama3-8B model on a large-scale synthetic dataset of 80K visual programming tasks, outperforming GPT-4V and LLama3-70B. Finally, they propose emulator-driven fine-tuning, which designs curriculum over training data distribution by leveraging emulator feedback, i.e., assigning higher weights to tasks where the model struggles. The paper includes an analysis of failure cases, capabilities, and transfer learning skills.
weaknesses: * The paper proposes a new benchmark focusing on *visual programming*, yet only provides results on two VLM families: GPT-4V and LLaVA-1.5. As there exists a plethora of open-sourced VLMs, I think it would only make the paper better by adding diverse state-of-the-art models (e.g., Llama-3.2-vision, LLaVA-onevision, Phi-3-Vision, Phi-3.5-Vision, QwenVL, Cameleon, LLaVA-next, Pixtral).
* Need clarity in the data generation process. Are there cases where the generated code is correct for the wrong reasons (e.g., logic incorrect, answer correct)? Is the emulator able to filter out these cases during the correctness check? If not, how does that affect the performance of fine-tuned models trained on the synthetic dataset?
* Lacks detailed explanation of the emulator.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - I would probably not call the REAL dataset "real-world" tasks. They are pretty much very synthetic and toy tasks, not real-world. Maybe you can refer to them as "existing" (or any other word). The term "real-world" was confusing at the beginning.  It would also help if you shoed an example of a SIM task in Figure 1, or in Section 4.
soundness: 3
strengths: 1 - This paper contributes a benchmark to study the capabilities of VLMs on program synthesis for visual tasks. The benchmark is challenging enough for current SOTA models to fail at it, but at the same time it is easy enough for humans to solve, making it a good contribution for the community. It also requires a variety of skills that previous benchmarks do not need, making it more general.

2 - The contributions are well motivated and well presented, making it an easy paper to read.

3 - The fine-tuning strategy using curriculum learning based on the harder and easier task is very sensible in this setting where multiple skills need to be learned. The specific implementation is reasonable, and provides very good results.

4 - The analysis (ablation study based on difficulty of the tasks, as well as failure analysis) is informative both in terms of understanding the benchmark better, but also in terms of understanding the models better.  Also, the "transferable skills" section is very pertinent. I was actually thinking about the zero-shot capabilities to unseen tasks before I got to that section.
summary: This paper introduces a new visual reasoning benchmark that requires a combination of skills to be solved (like logic, math, coding, visual understanding). Then, they benchmark current visual language models, and show that they do not perform well on this benchmark. Finally, they propose an approach to improve such models, by finetuning them on more data that follows the distribution of the benchmark, as well as by doing it following a curriculum.
weaknesses: I would appreciate if the authors analyzed the following:

1 - What is the drop (or increase) in performance in other standard vision tasks (such as VQA, classification) or even language or coding tasks when fine-tuning on the SIM dataset? It seems like the tasks follow a very specific pattern that may not be very related to other tasks. The paper shows that training on the SIM dataset contributes to tasks that were not part of the dataset, but I wonder if there is any contribution (positive or negative) to other tasks that follow different distributions (even though they may require similar visual or reasoning skills)

2 - What is the role of the prompt engineering in solving these tasks? How much effort and what techniques were used to ensure that the input prompts are not a limitation for the different models that were tested?

Meta-review: The paper introduces a new benchmark to evaluate the performance of LLM on program synthesis tasks within the XLogoOnline visual environment. The primary concerns raised include:

1. Limited technical novelty

2. Insufficient experimental evaluation

 While the paper has notable merit, it is not recommended for acceptance. The authors are advised to incorporate the reviewers' feedback when revising the paper for submission to other venues.

Decision: Reject