GeoX: Geometric Problem Solving Through Unified Formalized
  Vision-Language Pre-training
Authors: name: Renqiu Xia; name: Mingsheng Li; name: Hancheng Ye; name: Wenjie Wu; name: Hongbin Zhou; name: Jiakang Yuan; name: Tianshuo Peng; name: Xinyu Cai; name: Xiangchao Yan; name: Bin Wang; name: Conghui He; name: Botian Shi; name: Tao Chen; name: Junchi Yan; name: Bo Zhang
Published: 2024-12-16 15:20:03

Abstract: Despite their proficiency in general tasks, Multi-modal Large Language Models
(MLLMs) struggle with automatic Geometry Problem Solving (GPS), which demands
understanding diagrams, interpreting symbols, and performing complex reasoning.
This limitation arises from their pre-training on natural images and texts,
along with the lack of automated verification in the problem-solving process.
Besides, current geometric specialists are limited by their task-specific
designs, making them less effective for broader geometric problems. To this
end, we present GeoX, a multi-modal large model focusing on geometric
understanding and reasoning tasks. Given the significant differences between
geometric diagram-symbol and natural image-text, we introduce unimodal
pre-training to develop a diagram encoder and symbol decoder, enhancing the
understanding of geometric images and corpora. Furthermore, we introduce
geometry-language alignment, an effective pre-training paradigm that bridges
the modality gap between unimodal geometric experts. We propose a
Generator-And-Sampler Transformer (GS-Former) to generate discriminative
queries and eliminate uninformative representations from unevenly distributed
geometric signals. Finally, GeoX benefits from visual instruction tuning,
empowering it to take geometric images and questions as input and generate
verifiable solutions. Experiments show that GeoX outperforms both generalists
and geometric specialists on publicly recognized benchmarks, such as GeoQA,
UniGeo, Geometry3K, and PGPS9k.