CodePlot-CoT: Mathematical Visual Reasoning by Thinking with Code-Driven
  Images
Authors: name: Chengqi Duan, name: Kaiyue Sun, name: Rongyao Fang, name: Manyuan Zhang, name: Yan Feng, name: Ying Luo, name: Yufang Liu, name: Ke Wang, name: Peng Pei, name: Xunliang Cai, name: Hongsheng Li, name: Yi Ma, name: Xihui Liu
Organization: The University of Hong Kong

Recent advances in Large Language Models (LLMs) and Vision Language Models
(VLMs) have shown significant progress in mathematical reasoning, yet they
still face a critical bottleneck with problems requiring visual assistance,
such as drawing auxiliary lines or plotting functions to solve the problems.
Most LLMs and VLMs are constrained to text-only reasoning chains, while
multimodal unified models that can generate interleaved text and images lack
the necessary precision and controllability for such tasks. To address this, we
propose CodePlot-CoT, a code-driven Chain-of-Thought paradigm for "thinking
with images" in mathematics. Our approach leverages the VLM to generate text
reasoning as well as executable plotting code, which is then rendered into
images as "visual thought", to solve mathematical problems. To achieve this, we
first construct Math-VR, the first large-scale, bilingual dataset and benchmark
for Mathematics problems with Visual Reasoning, comprising 178K samples.
Second, to create high-quality training data, we develop a state-of-the-art
image-to-code converter specialized for parsing complex mathematical figures
into codes. Finally, using these training data, we train the CodePlot-CoT model
for solving mathematical problems. Experimental results show that our model
achieves up to 21% increase over base model on our new benchmark, fully
validating the efficacy of our proposed code-driven reasoning paradigm. Our
work opens a new direction for multimodal mathematical reasoning and provides
the community with the first large-scale dataset, comprehensive benchmark, and
strong approach for such problems. To facilitate future research, we make our
datasets, code, and pretrained models publicly available at
https://github.com/HKU-MMLab/Math-VR-CodePlot-CoT.