UniMath: A Foundational and Multimodal Mathematical Reasoner
Authors: Zhenwen Liang
Tianyu Yang
Jipeng Zhang
Xiangliang Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Multimodal Math Reasoning
Math Word Problem Solving
Geometry Problem Solving

Abstract: While significant progress has been made in natural language processing (NLP), existing methods exhibit limitations in effectively interpreting and processing diverse mathematical modalities. Therefore, we introduce UniMath, a versatile and unified system designed for multimodal mathematical reasoning tasks. Tackling complex problem-solving in arithmetic, geometry, and table-based math, UniMath utilizes a fine-tuned T5 model augmented with a variational autoencoder (VAE)-based image tokenizer. By jointly training and evaluating the model on three diverse datasets - SVAMP, GeoQA, and TableMWP, UniMath achieves state-of-the-art performance. The model's generalization ability is further demonstrated via fine-tuning on two additional datasets, MathQA and Geo-Proving. Through comprehensive evaluations, we showcase that joint training across diverse math tasks improves overall model performance and enhances its ability to generalize across different mathematical reasoning tasks. This pioneering approach provides a blueprint and inspires further efforts on unified mathematical reasoning with deep learning systems.

UniMath: A Foundational and Multimodal Mathematical Reasoner
Zhenwen Liang', Tianyu Yang'. Jipeng Zhang and Xiangliang Zhang
University of Notre Dame, {zliang6, tyang4, xzhang33) 0nd. Hong Kong University of Science and Technology, jehanggr@conect.us ust. hk
Abstract
tual information and execution of symbolic reason- ing (Hosseini et al., 2014; Kushman al., 20141 4). Another essential aspect of mathematical reason- ing is geometry problem-solving which demands the understanding of visual context and reasoning on spatial relations (Seo et al., 2015; Lu et al., 2021; Chen et al.. 2022a). Furthermore, table- based math mrobeem-solvigg presents a unique chal- lenge as it requires processing heterogeneous and structured table content extract relevant infor- mation for problem-solving (Pasupat and Liang. 2015; Lu et al., 2023a). Existing work often re- sorts to task-specific models, each fine-tuned for specific modality. While these models perform efficiently in their specific domains, they struggle to generalize across different modalities, which is an essential capability for advanced Al systems. Observing this limitation, we propose shift wards a unified model, applicable across all tasks and grounded formal symbolic language genera- tion. Our proposed model aims to address a wide range of tasks in the mathematical reasoning do- main with single adaptable model, improving its capacity for advanced reasoning similar to human cognition.
While significant progress has been made in natural language processing (NLP), existing methods exhibit limitations in effectively inter- preting and processing diverse mathematical modalities. Therefore, we introduce UniMath, a versatile and unified system designed for mul- timodal mathematical reasoning tasks. Tack- ling complex problem-solving in arithmetic,, geometry, and table-based math, UniMath uti- lizes fine-tuned T5 model augmented with a variational autoencoder (VAE)- based image tokenizer. By jointly training and evaluating the model on three diverse datasets SVAMP, GeoQA, and Tab UniMath achieves state-of-the-an performance. The model's gen- eralization ability is further demonstrated via fine-tuning on two additional datasets, MathQA and Geo-Proving. Through comprehensive evaluations, we show that joint training across diverse math tasks improves overall model per- formance and enhances its ability to generalize across different mathematical reasoming tasks. This pioneering approach provides a blueprint and inspires further efforts on unified mathe- matical reasoning with deep learning systems.
1 Introduction
In the pursuit of multimodal reasoning, existing approaches can be broadly categorized into two types: 1) Lu et al. (2022) leveraging pre-trained image captioning models to convert images into tex- tual descriptions, which are then combined with the text content, and 2) Chen et al. (2021a, 2022a) em- ploying pre-trained image feature extractors, such as ResNet He et al. (2016), to obtain latent repre- sentations of images and concatenate them with the latent features of the text component. How- ever, these approaches exhibit limitations when applied mathematical reasoning tasks. Both im- age captioning and ResNet models are primarily designed to comprehend and describe real-world images rather than mathematical diagrams contain- ing geometric shapes. As a result, these models
Mathematical reasoning, an essential aspect of hu- man intelligence, plays a pivotal role in our daily lives and decision-making processes (Dehaene and Sybesma, 1999). Despite the significant progress in natural language processing (NLP) and research, the development robust NLP system capable of handling multimodal input and accommodating di- verse downstream tasks remains an under-explored challenge. Mathematical reasoning naturally in- volves wide range of tasks and modalities, re- flecting the complexity and adaptability of human thinking in this area.
One common task used to test mathematical reasoning skills is solving math word problems (MWP), which necessitates comprehension of tex-

---

Transformer (T3) Encoder
Transformer (15) Decoder
Image Tolenizer
Table
Text
Goometry problom solving
Table problem solving
Math word problem solving
Tokenizer
Tokenizer
VQ-VAE
Reconstmccionn
Problem:
Teecher
Doga
Gata
There 732 and more girls and 8 more boys jorned the school. How many garls now?
"AB the diameter chord AB, A-C BD, "i' 2CAB-22 28 DD-2 40"
Find Mas. South row. in this row for boys and girli. Answer
Antwwer
1414 ginls
5
Bos
29
9
Figure 1: The overall workflow of our proposed UniMath.
may not be suitable for accurately interpreting ge- ometry images in mathematical context.
Our Approach
The architecture of our proposed model can be found in Fig. 1, detailed description of our frame- work is located in the appendix. We build UniMath on TS backbone (also extendable to any encoder- decoder LLMs). For the MWP solving and table MWP solving task, we train the model with textual problem description as input and the answer as out- put. For tables. we follow the approach in (Lu et al.. 2023a) transform tables into texts. Also, We sep- arate the explanation and answer of the TableMWP dataset into two targets during training controlled by different prefixes. For geometry problems, we integrate an external Vector Quantized Variational Autoencoders (VQ-VAE) as our image encoder to transform image patches to new tokens and con- catenate them with the textual tokens as the input. This framework is flexible to deal with text-only input and text-image mixed input. Also, the image tokenizer employed in our approach is flexible and compatible with various backbone models, allow- ing for the integration of more advanced geome- try feature extractors when they become available. Empirically, we use two 2-layer ResBlocks as our encoder and decoder in VQ-VAE Our UniMath is trained by simply adding the VAE reconstruction loss and the cross-entropy loss of our pre-trained language model backbone. During testing, the de- coder part of VQ-VAR discarded.
Another challenge in the development of a uni- fied model for mathematical reasoning is the issue data management, including data collection and tokenization. To achieve unified reasoning, We need to process geometric diagrams, natural lan- guages, and symbolic representations. Handling these various data modalities, which involve tasks such pre pre-processing and tokenization, emerges significant question of exploration. Ensuring an efficient and coherent process unify these divergent data types is critical to the successful im- plementation and performance of a singular model for mathematical reasoning.
In this paper, present UniMath, a unified mathematical reasoner capable of processing mul- timodal input, including math word problems (MWP), geometry problems, and table-based MWP. We achieve this by fine-tuning T5 (Chung et al., 2022) model and augmenting its vocabulary with tokenized image representations generated by train- able VQ-VAL (Van Den Oord et al., 2017; Razavi et al., 2019). We jointly train and evaluate our model on three held-in datasets, i.e.. SVAMP (Pa- tel al., 2021), GeoQA (Chen et al., 2021a) and TableMWP (Lu al., 2023a). Also, we fine- tume our UniMath towards two held-out datasets MathQA (Amini et al., 2019) and -ro-Proving (Chen et al., 2022a) to demonstrate its potential on external tasks. Furthermore, we conduct a com- prehensive evaluation of different data tokeniza- tion methods and examine the impact of chain- of-thought explanations in training such multi- modal reasoner. We believe that our work serves pioneering study in the development of unified mathematical reasoning systems, inspiring future researchers to devise more advanced approaches in this domain.
Experiments
3.1 Held-in Datasets
SVAMP The SVAMP dataset (Patel al., 2021) comprises 1,000 English math word problems, created by introducing challenging variations existing problems. We adopt the original eval- uation settings proposed in (Patel et al., 2021), where MAWPS
et al., 2016)

---

Held-in Tasks
Held-out Tasks
SVAMP
GeoQA
TableMWP
MathQA
UniGeo-Proving
Best Fine-tuned Baseline
47.3
46,8
58,5
78.6
80.6
Train Individually on T5-base
29.8
13.7
62.7
82.3
82.7
Train Individually Fl-a-TS-base
30.5
45.1
64.5
82.0
83.0
UniMath- ---bae UniMath-Fii -aann- -a-----base
37.3
49.6
65.4
83.3
82.9
41.8
50.0
66.5
82.7
83.0
Table I: The comparison between our unified model and baselines in terms of solution accuracy. a: Jie et al. (2022),
b: (Chen et al., 2022a), c: (Lu et al., 2023a). *: Our reproduced results based on the official codes from the authors.
and ASDiv-a (Miao et al., 2020) serve the train- ing set and SVAMP is used the testing set.
single NVIDIA RTX 3090 GPU. As for hyperpa-
rameters, We utilized batch size of 16, thereby suring an optimal balance between computational efficiency and model accuracy. To mitigate the feet of overfitting and help regularize the model, we implemented a dropout rate of 0.1. This al- lowed for a healthy proportion of neurons to be randomly ignored during training, promoting gen- eralization and reducing potential over-dependency certain features. In addition, used Adam W as our optimizer with learning rate of 0.0004
GeoQA GeoQA (Chen et al., 2021a) contains 4,998 diverse real-world geometry problems found Chinese middle school exams. Each problem is further annotated with specific programs that outline the problem-solving process. All problems in this dataset are of the calculation type, implying that their solutions are derived and calculated from the problem description. We utilize the English version of this dataset provided in (Chen et al., 2022a) ensure linguistic consistency with other datasets.
and weight decay of 0.01, aiding in optimizing our model's parameters while controlling for overfit- ting by preventing the weights from growing too large. To encourage the model to converge more reliably during the initial stages of training, we employed linear warm-up with ratio of 0.1. Fur- thermore, to avoid exploding gradients which result in destabilization of the training process, we employed gradient clipping with maximum gradi- ent norm 5.0. This selection of hyperparameters was strategic, balancing model performance with computational resource usage.
TWb The TabMWP dataset (Lu et al., 2023a) includes 38,431 tabular math word prob- lems consisting of free-text questions and multiple- choice questions. A distinguishing feature of TabMWP is that each problem is accompanied by tabular context, in both image format and texual format, which are essential to solve the problem.
3.2 Held-out Datasets
MathQA MathQA (Amini et al., 2019) contains 37,200 math problems collected from GRE exams, However, many problems are either unsolvable through equations or annotated in an incorrect for- mat. Consequently, we follow Jie et al. (2022) and select an MWP subset, comprising 16,191 prob- lems in the training set and 1,601 for testing.
3.4 Main Results
We evaluated the effectiveness of our UniMath model two backbones, T5 and Flan-T5, and benchmarked it against the best fine-tuned base- lines (we exclude those prompting- based baselines that use LLMs), as illustrated Table 1. First, the model was jointly trained and evaluated on three held-in tasks, the results of which are shown in the first three columns. Then, to test the model's gen- eralizability while training on dut----dist data, it was fine-tuned separately two external datasets, with the results in the last two columns.
BeiGeo-Reasoning al., 2022a) features 9,543 proving problems, with each entry consisting of a colored geometry diagram, description text, and a proof with reasons. The latter includes the reasoning skills or geometry theorems applied in each step.
UniGeo-Proving,
(Chen
To demonstrate the impact of our unified training approach, we also provide the accuracies when each dataset was separately and individually trained on the same backbone. Notably, UniMath outper-
3.3 Implementation Details
We implement our code with Pytorch framework, all the experimental results can be produced by a

---

forms the baseline in of the three held-in tasks and in both held-out tasks. While UniMath does not beat the DeductiveReasoner (Jie al., 2022) on the SVAMP dataset, it should be noted that the baseline model uses a specialized neural-symbolic decoder, which typically outperforms a general- purpose decoder like T5, albeit lacking generaliz- ability to non-MWPs. The main takeaways are: 1. We successfully obtain an effective unified math- ematical reasoner with very competitive accu- racy against state-of-the-ar baselines.
4) transform both. As evidenced by the results in Figure 2(a), arithmetic operators perform opti- mally when preserved their original form, while geometric symbols are better when transcribed into their natural language counterparts such as perpendicular to and congruent_ to like func- tion names. This discrepancy may stem from the higher frequency of arithmetic operators in the pre- training corpus, thereby enabling models like T5 to have more nuanced understanding of them com- pared to the less common geometric symbols. In our UniMath tokenizer, we only transform geomet- ric relations according to the results of our analysis in this section.
2. The mathematical reasoning ability derived from held-in tasks is able to generalize and help improve the fine-tuning on held-out tasks.
3.6 Analysis on Image Tokenizer
3.5 Analysis on Symbol Pre-processing
Geometry problem-solving necessitates models ca- pable of deciphering geometric diagrams. Exist- ing techniques, such as those referenced in (Chen et al., 2021a, a, 2022a), employ frozen ResNet- 101 to derive embeddings by treating diagrams as im- ages. However, given the significant differences between geometric diagrams and real-world im- ages, the suitability of ResNet is questionable. this study, documented in Figure 2(b), we carry out a thorough examination of various image to- kenizer designs. We considered five different set- tings: 1) Frozen ResNet, 2) Trainable ResNet, 3) Two trainable Reshlocks,
Mathematical groblem-solving often involves var- ious symbolic representations such as +, A, 1 _ =. Pre-processing these special tokens has been topic of interest in recent research (Wang al., 2017: Zhang et al., 2020b; Jin et al., 2021; Chen al., 2022a). We examined the pre- processing techniques of two categories of sym- bols: arithmetic operations (i.e., +, -) and geomet- ric relations (c.g., 1 and 2). In Table 2, we show a comprehensive list of symeol-to-name transfor- mations. We considered four different settings: 1)
Category
Symbol
Representation
Trainable ResNet + VQ- VAF, and 5) Two trainable Reshlocks VQ- VAF. Notably, while the ResNet was pre-trained on ImageNet, the Resblocks were randomly initial- ized. Additionally, it's worth mentioning that two Resblocks have significantly fewer parameters than ResNet. Our findings reveal that ResNet coupled with VQ-VAE yields the most impressive results, though two ResBlocks in conjunction with VQ- VAE can achieve comparable performance with much fewer parameters. This result underscores the beneficial impact of VQ-VAE reconstruction loss in enhancing image tokenizer efficacy. Also, we can conclude that the pre-trained ResNet does not contribute as effectively to geometric diagram comprehension.
Arithmetic
cal_add add
Arithmetic
val_minus
Arithmetic
cal _almuuliply
Arithmetic
aal_divide
Geometric
not not_equal
Geometric
approximate
Geometric
triangle
Geometric
angle
Geometric
parallel
Geometric
circle
Geometric
perpendicular
Geometric
congruent
Geometric
parallelogram
Geometric
similar
Geometric
arc
3.7 Analysis on Chain-of- Chain-of-Thought Explanations
Table 2: The symbol-to-namee transformations used in our paper. We transform all geometric relations during data pre-processing to help the language model under- stand them.
One of our held-in tasks, namely TableMWP, is originally annotated with Chain-of-Thought (CoT) (Wei et al., 2022) explanations (i.e., step-by-step descriptions of solutions). We evaluated perfor- mance across three different settings: 1) Omitting
no transformation, 2) transform arithmetic oper- ators only, 3) transform geometric relations only,

---

Analysis on Symbol Processing (Accuracy)
Analysis on Image Takenizer (Accuracy)
Analysis on CoT Explanations (Accuracy)
48
SWAMP GeoGA
48.4
80
82.3
65
SWAMP TableMWWP
46
48.2
70
55
55.6
45.0
66
Geo
UnGGw-PProving
45
42
41.9
66
9X.4
60.3
00.0
80
48.1
36
Swiling
Selling
Seiling
Switing
Swiing
Suiling
Salling
Swlling
Seiling
[4]
Figure 2: Detailed analysis on pre- prepprcesssng. tokenizers, and chain- of of-thhotht explanations. The descriptions of different settings are italics in Section 3.5, 3.6 and 3.7.
CoT and focusing solely on the final answer; 2) Combining CoT with the final answer to form joint target; and 3) Generating CoT and the final answer with different prefixes, essentially splitting them into two tasks.
For example, TAPAS (Herzig et al., 2020) intro- duced new way of parsing tabular data using pre-training large corpus of tables. Lu et al. (2023a) introduced a new dataset and employed re- inforcement learning to select in-context examples. For more detailed summary of mathematical problem solving, please refer to the survey (Lu et al.. 2023b).
Our experimental findings, detailed in Figure 2(c), indicate that incorporating the CoT method can considerably improve the problem-solving ac- curacy, not only the TableMWP dataset but also SVAMP, a dataset not originally annotated with COT. This suggests promising capacity for CoT to be generalized external datasets. Additionally, our results show that multi-task learning tends to be the most effective approach compared to the other two strategies we examined.
Multimodal Foundation Models: Pioneering works such BETT (Bao et al., 2022) and Uni- perceiver (Zhu et al., 2022) have been investigat- ing pre-training strategies for handling multimodal data. Further, all-in-one Transformer (Wang et al., 2023) and OFA (Wang et al., 2022) have shown promise in providing a unified approach to handling diverse modalities. ML-MFSL (Najdenkoska et al., 2023) introduced meta-learning that enables flexi- ble adaptation across vision and language tasks.
4 Related Work
Math Word Problem Solving: Math word prob- lem solving is prevalent evaluation method for NLP models (Amini et al., 2019; Patel al., 2021; Cobbe al., 2021). Early solutions relied on statis- tical and rule-based parsing (Hosseini et al., 2014; Koncel-Keddee Kedziorski et al., 2015), and have evolved into Seq2Seq- based neural networks (Xie and Sun, 2019; Zhang et al., 2020a; Liang et al., 2022a,b) and LLM-enhanced solvers (Liang et al., 2023a,b). Geometry Problem Solving: Geometry problem solving often necessitates the understanding of vi- sual diagrams and textual descriptions. State-of- the-art models and datasets, namely Gcometry3K (Lu et al., 2021), GeoQA (Chen et al., 2021 GeoQA+ (Cao and Xiao, 2022), and UniGeo (Chen et al., 2022a), are aiming to enhance both perfor- mance and explainability.
5 Conclusion
In this work, we introduce UniMath, a unified math- ematical reasoner, aiming to address diverse and multi modal mathematical tasks. Our extensive experiments demonstrate its superior performance and generalizability across various mathematical modalities, outperforming many task-specific base- lines. Moreover, we share key insights on data pre- processing and tokenization techniques to achieve our unification goal. We further discuss the impact of chain-of of-thought explanations in the training pro- cess of such unified reasoner. We hope that this work will encourage further exploration in NLP, and contribute to the broader goal of achieving human-like mathematical reasoning in AI systems. Structural Problem Solving: Structural math problems, especially those entailing tables, require an intricate blend of interpretation and reasoning.
Codes and data are available at hnttps://1tthi com/ Zhenwen-tee NeeeenNNLP//// hupp/nniii Math.

---

Limitations
abstractive summarization. Advances Neural
formation Processing Systems, 35:24516-2444 -45288.
Our UniMath model, while effective, does have some limitations that need to be addressed. First, we currently use convolutional neural networks, specifically ResNet or Resblocks, to process ge- ometry diagrams. Although this is common ap- proach in previous studies, it might not be the best solution for every situation. In other words, there might be more effective way handle geom- etry diagrams. In our future work, we plan to incorporate a more specialized tool, a geometric parser, into our model to improve how it handles these tasks. Secondly, UniMath is built around an encoder-decoder structure, similar to the TS model. The adaptability of our approach popu- lar decoder-only models, like BLOOM (Scao et al.. 2022) and LLaMA (Touvron al., 2023) models, is still unknown. In future work, we intend ex- plore how we can add more tokenizers to these decoder-only models.
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li. Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling enstruction-finetunee language models. arXiv preprint arXiv:2210.114::
Karl Cobbe, Vineet Kosaraju, Mohammad Bavar- ian, Jacob Hilton, Reicchiro Nakano, Christopher Hesse, and John Schulman. 2021. Training veri- fiers solve math word problems. arXiv preprint arXiv:21 14168.
S Dehaene and R Sybesma. 1999. The number How the mind creates mathematics. Glot interna- rional, 4(4).
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In CVPR, pages 770-778.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Mueller, Francesco Piccinno, and Julian Eisensch- los. 2020. Tapas: Weakly supervised table parsing via pre-training. In ACT,, pages 4320-4333.
References
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman, 2014. Learning to solve arithmetic word problems with verb categorization. In EMNLP. pages 523-533.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Kenzie-Kedziorski, Yejin Choi, and Hannaneh Ha- jishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based for- malisms. NAACI, pages 2357-2367
Zhanming Jie, Jierui Li, and Wei Lu. 2022. Learning to reason deductively: Math word problem solving as complex relation extraction. In ACL, pages 5944- 5955.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2022. Beit: Bert pre-training of image transformers. In ICLR.
Zhihua Jin, Xin Jiang, Xingbo Wang. Qun Liu, Yong Wang, Xiaozhe Ren, and Huamin Qu. 2021. Numgpt: Improving numeracy ability of generative pre-trained models. arXiv preprint arXiv:2109.031.7.
Jie Cao and Jing Xiao. 2022. An augmented benchmark dataset for geometrie question answering through dual parallel text encoding. In Proceedings of the 29th International Conference Computational Lin- guistics, pages 1511-1520
Rik Joncel-Kedziorske Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equa- tions. Transactions the Association for Computa- tional Linguistics, 3:585-597.
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin,
Chongyu Chen, and Xiaodan Liang. 2022a. Unigeo: Unifying geometry logical reasoning via reformulat- ing mathematical expression. EMNLP.
Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. Mawps: A math word problem repository. In NAACL, pages 1152 1157.
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. 2021a. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. In Findings of ACL, pages 513-523.
Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In ACL, pages 271- 281.
Xiuying Chen, Zhi Cui, Jiayi Zhang, Chen Wei, Jianwei Cui, Bin Wang, Dongyan Zhao, and Rui Yan. 2021b. Reasoning in dialog: Improving response generation by context reading comprehension. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 2688-12691..
Zhenwen Liang, Dian Yu. Xiaoman Pan, Wenlin Yao. Qingkai Zeng, Xiangliang Zhang, and Dong Yu. 2023a. Mint: Boosting generalization in mathe- matical reasoning via multi-view fine-tuning. arXiv preprimt arXivv2307.0795.
Xiuying Chen, Mingzhe Li, Xin Gao, and Xiangliang Zhang. 2022b. Towards improving faithfulness in

---

Zhenwen Liang. Wenhao Yu. Tanmay Rajpurohit, Pe- ter Clark, Xiangliang Zhang, and Ashwin Kaylan. 2023b. Let gpt be math tutor: Teaching math word problem solvers with customized exercise generation. EMNLP.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, al. 2022. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint (0t:::22211050000
Zhenwen Liang, Jipeng Zhang, Lei Wang, Wei Qin, Yunshi Lan, Jie Shao, and Xiangliang Zhang. 2022a. Mwp-bert: -augreeeeed pre-training for math word problem solving. In NAACL, pages 997- 1009.
Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. 2015. Solving geometry problems: Combining text and diagram interpretation. In EMNLP, pages 1466-1476.
Zhenwen Liang. Jipeng Zhang, and Xiangliang Zhang. 2022b. Analogical math word problems solving with enhanced problem-solution association. EMNLP.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 2.Xiv:2302.13977
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-chun Zhu. 2021. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In ACL, pages 6774-6786
Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. NeurlPS, 30.
Pan Lu,, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang. Song-Chun Zhu, Oyvind Tafjord, Peter Clark. and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. NerulPS, 35:2507-2521.
Jinpeng Wang. Yixiao Ge. Rui Yan, Yuying Ge. Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, et al. 2023. All in one: Exploring unified lideo-language pre- training. CVPR, pages 6598-6608.
Pan Lu, Liang Qiu, Kai- Kai-Wei Chang. Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023a. Dynamic prompt learn- ing via policy gradient for semi-structured mathemat- ical reasoning. ICLR.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. Ofa: Unifying chitectures, tasks, and modalities through a simple -equence-t--sqquence learning framework. In ICML, pages 23318-23340.
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai- Wei Chang, 2023b. A survey of deep learning for mathematical reasoning. ACI..
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. EMNLP, pages 845-854.
Shen-Yun Miao, Chao-Chun Liang, and Keh- Su. 2020. A diverse corpus for evaluating and developing english math word problem solvers. In ACL, pages 975-984.
Jason Wei, Xuezhi Wang, Dale Schuurmans. Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. NeurlPS.
Ivona Najdenkoska, Xiantong Zhen, and Marcel Wor- ring. 2023. Meta learning to bridge vision and lan- guage models for multimodal few-shot learning. In ICLR.
Zhipeng Xie and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In IJCAI, pages 5299-5305.
Panupong Pasupat and Percy Liang. 2015. Composi- tional semantic parsing stmi-strrctture tables. In ACL, pages 1470-1480.
Jipeng Zhang. Lei Wang, Roy Kei Lee, Yi Bin, Yan Wang, Jie Shao, and Ec-Peng Lim. 2020a. Graph- to-tree learning for solving math word problems. In ACL, pages 5928-3937
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? In NAACL, pages 2080-2094,
Xikun Zhang, Deepak Ramachandran, Tan Tenney, Yanai Elazar, and Dan Roth. 2020b. Do language embeddings capture scales? In Findings of EMNLP. pages 4889-4896.
Colin Raffel. Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research, 21:1- 67.
Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hong- sheng Li, Xiaohua Wang, and Jifeng Dai. 2022. Uni Unippeere Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In CVPR, pages 16783-16794.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. 2019. Generating diverse figh-fidelity images with vq-vae-2. NeurlPS, 32.

---

A Appendix
The loss function we use for training the VQ- VAE follows the one proposed in (Van Den Oord et al.. 2017). This loss function can be formulated as the addition of the following three terms:
A.1 Our Approach
Our proposed method builds upon the T5 model series (Raffel et al., 2020; Chung et al., 2022), an encoder-decoder transformer architecture that has shown promising results in various natural lan- guage processing tasks. The encoder receives in- puts and the decoder produces results. To enable the encoder being able to understand geometry diagrams, we integrate an external Vector Quan- tized Variational Autoencoder (VQ-VAE) (Van Den Oord et al., 2017; Razavi et al., 2019) as our image encoder. This allows us to transform image patches into tokens, which can then be concate- nated with the textual tokens to form the input for UniMath. Because VQ-VAE uses discrete latent space to encode the image patches, which can be efficiently processed by the T5 model.
reconstruction = || Decoder(22)]1
qquatiizatinn = ||8g[Ee - z0
Leommitment B 1|ze - sgi 2g1
where represents the input the model, which the geometry diagram in our model. 2e the out- put from the encoder, which is continuous latent representation. 2q is the selected quantized vec- tor that is closest to Ze in the codebook of vectors. The symbol sg denotes the stop-gradient opera- tion. This operation is used in the computation of the quantization loss and commitment loss. When the toop-gradeent operation is applied variable, during backpropagation, no gradient will be back- propagated through this variable. This means that the stop-gradd toodgrraient operation prevents its input from being updated by gradient descent during training. The reconstruction loss trains the image encoder and decoder to reconstruct the input image. The quantization loss encourages the quantized embed- ding to move towards the latent feature generated by the encoder. The commitment loss encourages the output of the encoder stay close the chosen embedding vector, preventing the encoder's output from fluctuating too frequently between different code vectors.
For the tabular input, we adapt the approach of (Lu et al., 2023a) to transform tables into tex- tual representations. Tables can be complex to parse and understand directly due to their multi- dimensional nature and various possible formats. By converting the tables into text, we can leverage the existing language understanding capabilities of the T5 model to solve these problems. Specifi- cally, we convert the raw table text into flattened token sequence, with each row separated by a new- line character 'In' and each column separated by 'l' Once the table is transformed into this textual format, it can be input into UniMath in the same manner regular problem descriptions. Also, in the TableMWP dataset, every problem is annotated with step-by-step chain-of-thought (CoT) expla- nation and the final answer. We apply multi-task learning way to separate the CoT and the answer, which is accomplished by using different prefixes to control the output of the model, enabling the model to generate both the problem solution and the explanation for the solution. An analysis of this can be found in Section 3.7.
A.1.2 Training and Testing
The training of UniMath is done by combining the VAE reconstruction loss with the cross-entropy loss of our T5 backbone. This results in hybrid loss function that optimizes both the language under- standing and image encoding capabilities of Uni- Math. During the testing phase, we only retain the encoder part of the VQ-VAE. The decoder part, which is used during training to reconstruct the original image, is discarded.
A.1.1 VQ-VAE Implementation
In our implementation of VQ- VAE, we use a se- ries of two 2-layer ResBlocks as our encoder and decoder, and each ResBlock is:
To sum up, our UniMath model is a comprehen- sive approach that handles various types of math problems, including text-based problems, table- based problems, and geometry problems, by com- bining the power of transformer-based language model with a VQ-VAF--base image encoder. This combination allows UniMath to effectively inter- pret and solve a wide range of math problems.
y ReLLU/CCnn///LLL///////// + x)
where x and are the input and output, respec- tively.


Reviews:
[{"review_id": "QA6I9rCo1v", "reviewer": "Reviewer_U2A1", "summary": "", "questions": "If a math problem requires more than one modality for its solution, can it be successfully solved by UniMath?\nCan UniMath be deployed on decoder-only LLMs?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for your insightful comments. We genuinely appreciate your feedback. We hope our responses can address your concerns so that you may consider increasing the scores of the review.\n\n**Reply to reason to reject**\n\nWe totally understand the concerns the reviewer has raised. In fact, we have provided an exhaustive description of our training process, configurations, and technical details in the appendix of the paper. Given the constraints of space, we had to put them in the appendix in our submission. However, we would be more than happy to relocate them to the main body of our paper in our camera-ready version.\n\nMoreover, to ensure reproducibility, we have submitted our code for reference during the review process. We would like to assure the reviewers and the community that we are committed to releasing our data and codes upon the paper's acceptance as promised in our submission.\n\n**Response to the Question:**\n\nThanks for the insightful questions. Our response:\n\n1. Our UniMath can only output text modality. However, it can be combined with additional generative models to output other modalities, in a post-processing way.\n\n2. It can theoretically adapt to the decoder-only LLMs by adding additional tokenizers, but it falls beyond the scope of the current paper. \n\nWhile we acknowledge the importance of these questions, they fall beyond the scope of our current paper. We will mention them in the limitation section of our paper and explore them in our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Solving math problems is an important task and I think that a multimodal math reasoning model can be useful as certain complex math problems demand the utilization of multiple modalities for their solution", "reasons_to_reject": "The method seems to be a basic combination of three individual modalities (geometry, table, text).\nThe technical details are unclear as the authors did not provide comprehensive information regarding the model architecture, parameter size, and the training process.", "questions_to_authors": "If a math problem requires more than one modality for its solution, can it be successfully solved by UniMath?\nCan UniMath be deployed on decoder-only LLMs?", "paper_topic_and_main_contributions": "This paper proposes UniMath, a multimodal mathematical problem-solving model that can solve arithmetic, geometry, and table-based math problems.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4OtTTujvl2", "reviewer": "Reviewer_FwBq", "summary": "", "questions": "Does the jointly trained UniMath model have some experimental evidence on zero-shot (or few-shot) math-related task transferring?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. \n\n**Case Study**\n\nWe will include a comprehensive case study section in the revised version of the paper to more effectively demonstrate the capabilities of UniMath in facilitating better mathematical reasoning through joint training.\n\n**Image Tokenizer Detail**\n\nWe follow the settings from [1]. Specifically, we fill geometric diagrams with a white background to equalize length and width, then resize them to 224×224 pixels. These images are further segmented into 49 patches, each of 32×32 pixels. More details on this will be added to the revised paper for clarity.\n\n[1] UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression. Chen et, al.\n\n**zero-shot or few-shot transferring**\n\nActually, we have conducted evaluations of UniMath trained on SVAMP, GeoQA, and TableMWP across MathQA and Geo-Proving datasets. It appears that our fine-tuned UniMath model is already overfitted on the training datasets, making it challenging to generalize to other tasks in zero/few-shot settings. Nonetheless, our results indicate that UniMath can be further trained on held-out datasets to unlock its potential for out-of-distribution mathematical reasoning.\n\nWe've also incorporated the suggested reference into our paper. Thank you for the recommendation!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research question is meaningful: it is important and needed for a math model to deal with different formats and modalities for the existing math-related tasks.\n2. The approach of UniMath is straightforward: it's natural to format these tasks into seq-to-seq paradigm and use appropriate encoder to encode each modality (leveraging pretrained encoders).\n3. The experimental result is encouraging: the joint training of multiple downstream tasks after unification is helpful for not only held-in but held-out math tasks.\n4. Comprehensive analysis over various implementation details are provided.", "reasons_to_reject": "1. More case studies are needed to demonstrate the effectiveness of UniMath, facilitating better math reasoning by jointly training.\n2. For the input image modality, more details about image resolution setting and its effect on model performance should be provided.", "questions_to_authors": "Does the jointly trained UniMath model have some experimental evidence on zero-shot (or few-shot) math-related task transferring?", "paper_topic_and_main_contributions": "This paper proposes a framework which unifies various types (different in domains and input modality) of math-related generation tasks into T5-based transformer encoder-decoder framework. With the help of unification, the framework called UniMath leverages pretrained T5/Flan-T5 and jointly trained with geometry problem solving, table problem solving and math word problem solving tasks. The experimental results show the effectiveness of UniMath framework, which surpasses baselines individually trained on each downstream task, and the model is capable to be further transferred to held-out math-related tasks. Comprehensive analysis on image encoding, CoT setting, math symbol processing are also provided. In general, it is a great work towards a unified solution to math-related tasks.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The below work is also about the unification of math reasoning tasks and should be cited:\n1. Matcha: enhancing visual language pretraining with math reasoning and chart derendering", "Typos_Grammar_Style_and_Presentation_Improvements": "Not found."}, "extra_text": {}}, {"review_id": "0WBNRMIDvh", "reviewer": "Reviewer_xULr", "summary": "", "questions": "Did the authors try using a large VQ-VAE pre-trained on natural images from prior work? Was is necessary to train a new (and much smaller) one? \n\nWhy is the GeoQA results so low?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and providing such detailed feedback. We are also very grateful for confirming that our paper has some interesting findings. We sincerely hope the reviewer can raise the assessment score of our paper if the following responses have successfully addressed the concerns.\n\n**Concerning GeoQA Comparisons:**  \nWe would like to clarify that our results we presented are based on the Top-1 accuracy, while baselines [1] and [2] reported their best results using the Top-10 accuracy, where [1] actually used a beam size of 10 as a representation of top-10 accuracy. We chose the Top-1 accuracy as we believe this metric better reflects the precision required in mathematical problem solving. Nevertheless, this feedback is very helpful and has prompted us to recognize the need for clearer clarification. In our revised version, we will ensure to add both the clarification regarding our choice of accuracy metric and the comparison with baselines while using top-10 accuracy.\n\n[1] GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning. Chen et, al.\n\n[2] UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression. Chen et, al.\n\n**Concerning TableMWP Results:**  \nWe totally agree with the reviewer. We will revise our expressions to avoid any over-claims and include the performance of LLMs and in-context learning methods for the readers' convenience of reference.\n\n**Regarding the use of VQ-VAE pre-trained on natural images:**  \n\nIf we understand correctly, we have conducted this exploration in Section 3.3: \n\n`Our findings reveal that ResNet coupled with VQ-VAE yields the most impressive results, though two ResBlocks in conjunction with VQ-VAE can achieve comparable performance with much fewer parameters. This result underscores the beneficial impact of VQ-VAE reconstruction loss in enhancing image tokenizer efficacy. Also, we can conclude that the pre-trained ResNet does not contribute as effectively to geometric diagram comprehension.`\n\nWe will further make our conclusions clearer and clarify our explanations in the updated version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Reasonable results on these benchmarks despite using a relatively small LM (T5-Base). \n- Shows the benefit of multi-tasking these tasks despite them being non-trivially different, and the ability to transfer for TableMWP CoT to other tasks.\n- I found use of the VQ-VAE interesting, while a lot of works uses various image backbones to embed images for LLMs using a VQ-VAE is less well studied.", "reasons_to_reject": "- I am a bit confused by their comparisons. Chen et al report 60% on GeoQA, why is number here so much lower? Because the authors were not able to reproduce their result? Le et al. report 68% on TableMWP, I assume that number has been not shown because its not considered a \"fine-tuned baseline\" but I think for honesty's sake it should be in the table.\n- The abstract and intro state \"UniMath achieves state-of-the-art performance\" but I as far as I can tell existing work has reported better scores on SVAMP, GeoQA, and TableMWP so this claim seems to be simply false.\n\nI feel like this paper could be accepted since I thought some of the scientific finding were interesting and being behind other works that use much larger LLMs is not unexpected. However the abstract and intro over-promises and the main results table is questionable in how to presents prior work to the point I would call this paper misleading.\n\nThe rebuttal has significantly mitigated this concern for me, so I have raised my score.", "questions_to_authors": "Did the authors try using a large VQ-VAE pre-trained on natural images from prior work? Was is necessary to train a new (and much smaller) one? \n\nWhy is the GeoQA results so low?", "paper_topic_and_main_contributions": "This paper constructs a math-problem solving model that can handle problems with both tables and geometric diagrams. They fine-tune T5 on the training set of several math datasets and use a VQ-VAE to embed in images as tokens for geometric problems.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work addresses the task of multi-modal math problem solving. They augment a T5 model with a VQ-VAE image tokenizer and train the model on 3 math datasets which also include images and tables. Their model shows good performance on these datasets and also performs well when fine-tuned and evaluated on 2 other datasets.

The reviewers mostly agree this work is exciting. The initial reviews noted that some of the claims made are not accurate - particularly the depiction of existing works and reporting of the results from prior works. The authors have clarified this in the subsequent rebuttal and have promised to tone down the claims of state of the art. The authors should also report results from other works including some references that should have been included in their tables and are missing. One reviewer has given a lower score stating that the technical details were lacking and the work may not be reproducible but the authors promise to make their code available and have included information in the supplement.

Overall the contribution is good for a short paper. The authors should incorporate the feedback from the reviewers' comments to improve the paper and in particular should tone down the claims and represent results from existing works clearly.
Decision: Accept-Main