Feynman: Knowledge-Infused Diagramming Agent for Scaling Visual Reasoning Data

Authors: Zixin Wen, Yifu Cai, Kyle Lee, Sam Estep, Joshua Sunshine, Aarti Singh, Yuejie Chi, Wode Ni
Primary area: datasets and benchmarks
Keywords: Vision-Language Dataset, Synthetic Data, Visual Reasoning Benchmark

Abstract:
Visual reasoning is an essential ability of state-of-the-art multi-modal AI systems. Improving these systems requires high-quality vision-language data at scale. Despite the abundance of internet image and text data, knowledge-rich and well-aligned image-text pairs are rare. In this paper, we present a scalable data generation pipeline built with our diagramming agent, **Feynman**. To create diagrams, Feynman first enumerates domain-specific knowledge components ("ideas") and performs code planning based on the ideas. Given the plan, Feynman translates ideas into simple declarative programs and iterates to receives feedback and visually refine diagrams. Finally, the declarative programs are rendered by the Penrose diagramming system. The optimization-based rendering of Penrose preserves the visual semantics while injecting fresh randomness into the layout, thereby producing diagrams with visual consistency and diversity. As a result, Feynman can author diagrams along with grounded captions with very little cost and time. Using Feynman, we synthesized a dataset with more than 100$k$ well-aligned diagram-caption pairs. We also curate a visual-language benchmark, **Diagramma**, from freshly generated data.

Decision: reject
Reviewer scores: 3, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1730064471261

thread_type:
official_review

<thread_object>
creator: reviewer_jbwn

<content>
summary:
The authors introduce Feynman, a diagram-generation agent.  Feynman leverages Penrose, a diagramming tool that generates vector graphic diagrams from its own programming codes. Feynman’s diagram synthesis has four steps: idea, plan, iterate, and render.

- In idea step, an LLM is given a text description of target diagram and generates relevant ideas/concepts in plain text.
- In plan step, an LLM is given the documentation of Penrose and in-context examples and generates visual elements.
- In iterate step, a panel of visual judges (MLLMs) iteratively provide the feedback and the generation plan gets refined.
- Render step is not explicitly described, but perhaps Penrose tool would render the iteratively refined plan from the previous step into a diagram.

The authors also provide a benchmark called Diagramma, which contains 1058 multiple-choice QAs on the diagrams generated with Feynman.

The authors provide a brief qualitative comparison of Feynman and baselines (AutomaTikz and Flux-Pro). They also provide the evaluation of different MLLMs on Diagramm, and ablation of different planning components within Feynman.

strengths:
- Introduction of Feynman, an LLM-based diagram generation framework that uses Penrose as a backend.
- Introduction Diagramma, a VQA dataset to evaluate  diagram understanding capability of MLLMs where the diagram images are generated with Feynman

weaknesses:
- **Unclear discussion about Penrose vs. TikZ.** In the introduction, the authors mention that “AutomaTikz still exhibits efficiency overhead in synthesizing scientific diagrams at scale, due to the inherent complexity of both the TikZ language and visual design.” However, the authors do not compare using TikZ or Penrose as a rendering backbone. There must be different scenarios when Tikz and Penrose are more useful, and the authors should discuss them.
- **Weak comparison with baselines.** The authors only provide brief qualitative examples in Fig. 7, but it is unclear how the five input prompts are selected. In addition, compared to AutomaTikz, Feynman uses stronger LM (GPT-4) than Llama 2 used in AutomaTikz. This makes it hard to understand the fundamental advantage of Feynman compared to baselines such as AutomaTikz.
- **Missing related work.** DiagrammerGPT (COLM 2024; https://arxiv.org/abs/2310.12128) seems to be highly relevant, as it also uses an LLM for diagram layout planning (w/ iterative refinement) and uses diffusion or different vector graphic tools (e.g., PowerPoint/Inkscape/illustrator) for rendering. They also provide a training annotation and benchmark in the scientific domain - AI2D caption. I highly encourage authors to add discussion and comparison to this work.
- **Weak analysis of the correctness of the generated diagrams.** The highest performance among its ablation study (in Table 5) does not mean that the generated diagrams are correct. The authors should provide a more rigorous analysis of the correctness of the diagrams generated by Feynman (e.g., human evaluation). It is unclear whether Diagramma can be a meaningful evaluation benchmark of MLLMs without such analysis.

questions:
- The Penrose diagram tool should be cited (with URL) the first time it’s mentioned.
- In Sec. 2, the subsection names do not directly match with the four steps mentioned in the first paragraph of Sec 2: idea, plan, iterate, and render. For example, Sec 2.2 is ‘Knowledge Planning' instead of ‘idea'; there's no subsection for the last step - rendering. Explicitly matching the subsection names with the four steps would make the readers easily match which subsection describes which step.
- Is Sec 2.5 part of the ‘iterate’ step? it is not clear how the QA data is generated and used. The step needs to be explained in more detail.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730615877559

thread_type:
official_review

<thread_object>
creator: reviewer_aiem

<content>
summary:
This submission presents FEYNMAN, a knowledge-infused diagramming agent designed to generate high-quality, well-aligned diagram-caption pairs at scale. ​ The agent decouples knowledge elicitation from visual production, leveraging LLMs for domain-specific knowledge enumeration and the PENROSE diagramming system for rendering. ​ The authors synthesized a dataset of over 100k diagram-caption pairs and curated a visual-language benchmark, DIAGRAMMA, to evaluate the visual reasoning capabilities of vision-language models.

strengths:
There are several noticeable strengths in this paper:
* This paper presents a well-justified motivation for introducing LLMs into the pipeline of automatic diagram generation process.
* The manuscript is commendable for its clarity and structured writing style, which greatly facilitates reader comprehension.
* Additionally, the inclusion of clear and illustrative figures and tables is a notable strength, as it significantly aids in conveying the main claims of the paper to the audience.

weaknesses:
Since this submission is one of the first studies on MLLM-assisted diagram generation agents, it could offer even greater value to future research if the authors provided a more comprehensive error analysis. Such analysis would help identify bottlenecks in the current pipeline and suggest directions for improvement. 

Specifically, in the four steps of the FEYNMAN pipeline—idea, plan, iterate, and render—what types of errors occur? How are these errors distributed across the steps, and what might be their underlying causes? For example, are certain errors attributable to the limitations of the PENROSE program, or are others due to knowledge gaps or inaccuracies inherent in the LLMs? A detailed breakdown of these errors would provide valuable insights for future work.

questions:
Please refer to the Weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625209011

thread_type:
official_review

<thread_object>
creator: reviewer_rjpm

<content>
summary:
This paper presented a scalable data generation pipeline with a diagramming agent, FEYNMAN. Through four steps, idea, plan, iterate and render, FEYNMAN decouples knowledge elicitation and visual production to create programs, which are then rendered into different layout diagram variations using penrose.


FEYNMAN generated 10693 programs, further resulting in 106,930 diagram-caption pairs. Additionally, it curated a scientific benchmark, DIAGRAMMA,  which contains 1,058 multiple-choice questions focused on visual understanding and reasoning. 

The paper experimented with the generation capabilities of FEYNMAN pipeline across different subjects and the Appendix D.3 further discusses the diversity between programs and images under identical and distinct substances or domains. Furthermore, the paper evaluated DIAGRAMMA on 17 different MLLMs, confirming its validity for measuring the scientific visual reasoning capabilities of MLLMs.


Additionally, it qualitatively compared different diagramming approaches, discussing the relationship between the number of generate images and the total input/output tokens. At the same time, an ablation study was conducted to examine the impact of various pipeline components on the accuracy and quality of the generated programs.

strengths:
- The paper conducted comprehensive experiments and analyses. 

    - It studied the generation capabilities of the pipeline across different subjects, evaluated the performance of 17 MLLMs on DIAGRAMMA. 

    - Additionally, compared FEYNMAN with other methods for generating knowledge-rich images, and  an ablation study was performed to examine the impact of different pipeline components. 

     - The appendix provides detailed prompt designs for each component of the pipeline.

- The paper uses iterative visual refinement to obtain programs that better represent knowledge. 

    - During the iterations, VLMs act as judges, providing visual judgments and feedback to refine the program and produce better images with correct representation, proper relationships, legible text, simplicity, no redundancy and other essential qualities.

weaknesses:
- How many data were generated exactly? Is this in Section 3.1 mentions generating 10,693 unique substance programs across different subjects through the pipeline?

- To demonstrate the scalability of the pipeline, Section 4.2 discusses the relationship between the total input/output tokens and the number of generated images after de-duplication.  I don't quite understand the significance of this figure. The more samples are generated, the more tokens are naturally consumed. This seems like correct but useless talk.

- In Section 4.3, ablation experiments were conducted on 10 subdomains, but the experimental conditions and analysis are somewhat vague.
    - What exactly is the specific experimental design? How are these several pipelines compared? 
    - What is the specific number of samples in each sub-domain?
    - How exactly were KP, CP, and S ablated? If CP is missing, how is code generation achieved?
    - In Table 3, Why does the Yield and Compile of CP + S differ by about 10% when KP is absent?
    - Were these evaluation metrics designed by the author (PR, LR,...) all given by VLMs? Is this accurate enough?

- In the iterative process, feedback is provided by GPT-4o, Claude-3.5-sonnet, and Gemini-Pro-1.5. However, what I want to say is that this evaluation seems to be inaccurate. The benchmark generated in this paper evaluates the performance of VLMs on such math-related charts. Judging from Table 2, the highest performance is only 60%. It seems unreliable to use these VLMs for evaluation and provide feedback.

questions:
As I mentioned above.

- What exactly is the significance of Figure 8? 
    - Can the relationship between tokens and the number of programs demonstrate scalability?
    - There are too few data points shown in the figure. It seems that it is difficult to observe obvious patterns when fitting these lines.
    - It shows the relationship between total input/output tokens and the number of generated images after de-duplication. Does it consider the number of tokens consumed in the iterative process?

- In Algorithm 1 
    - Does setting different rendering seeds affect the scoring? 
    - Is it a binary score (good/bad), or is it a score given within a specific range?
 
- If we remove Iterative Visual-Refine based on VLMs and only reflect based on feedback by the compiler. What will be the effect? How many samples can it generate?

- Please elaborate in more detail on what the innovation of the paper is? It seems that generating charts through code as a medium has been explored by some works.

- FEYNMAN generated over 106,930 diagram-caption pairs, yet DIAGRAMMA only contains 1,058 multiple-choice questions？ How many testing set in total?

- Could you provide a complete example, including each step of the pipeline’s output and the iteration process?

- Since no anonymous open-source link has been provided, the quality of the data cannot be determined. Hope to see the provision of an anonymous link in the subsequent rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730629500743

thread_type:
official_review

<thread_object>
creator: reviewer_tw8f

<content>
summary:
This work proposes the Feyman framework, which is tackling the challenges of scalable generations of knowledge-grounded diagrams. Feyman follows a code-generation and diversification fashion. At first, an LLM is used to propose some “ideas”, and then these ideas will be translated into codes following the Penrose template. The Penrose code can be formulated as a constraint optimization problem to derive diagrams. Then equipped with a visual feedback mechanism, Feyman generates visual diagrams with the codes and refine upon satisfaction. Eventually Feyman is used to generate knowledge-grounded visual QAs.
The authors then conduct some extensive studies on their synthetically generated data with various open-source and proprietary visual-language models.

strengths:
- The whole approach of the proposed Feyman is sound.
- The work is tackling a challenging domain and indeed requires a much better scalable solution.
- The feedback mechanism and deduplication is interesting.

weaknesses:
- More detailed statistics are needed for the proposed dataset. It is clueless to judge its quality, complexity, as well as how it would contribute/drive the research community.
- While Table 2 is much appreciated (for its comprehensiveness), what is the current plausible upper bound? Of course 100% correctness would be the ultimate ideal performance, but what is the realistic upper bound for the community to judge the progress?
- More details of the visual judges are needed. For example, what exactly are they? Are they fine-tuned, are they evaluated with a certain type of performance measurement? How do we know if the judges are faithful enough?

questions:
- What are the visual judges?
- Why were there no human evaluations even at the very least for sanity checking conducted?
- Why choose the QA format after all the hard work? In addition, how do we envision the community would benefit from this dataset or any of the proposed methods?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents Feynman, a knowledge-infused diagramming agent for generating scientific diagrams and aligned captions at scale. While the approach of using LLMs for knowledge extraction and Penrose for rendering is interesting, several major concerns were raised by reviewers: (1) Insufficient evaluation of data quality and lack of human verification beyond the curated benchmark (Tw8f, rjPm), (2) Limited analysis of error patterns and bottlenecks in the pipeline (AiEM), (3) Weak baseline comparisons and missing discussion of key related work like DiagrammerGPT (JbwN). The primary contribution appears to be the pipeline itself rather than the benchmark, but more rigorous validation of the generated outputs is needed. Given these limitations in evaluation and analysis, I recommend rejection.