Chain-of-Thought Reasoning in Tabular Language Models
Authors: Mingyu Zheng
Hao Yang
Wenbin Jiang
Zheng Lin
Yajuan Lyu
Qiaoqiao She
Weiping Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: Tabular mathematical reasoning
Chain-of-thought reasoning
Tabular language models
TL;DR: we propose a novel framework which extends the chain-of-thought reasoning into tabular language models for the first time.

Abstract: Tabular mathematical reasoning task requires models to perform multi-step operations including information look-up and numerical calculation, based on heterogeneous data from tables and questions. Existing solutions tend to extend chain-of-thought (CoT) reasoning into powerful large language models (LLMs) to promote multi-hop mathematical reasoning. However, such LLM-based approaches are not a viable solution in the scenario of privatization deployment or limited resources. To address this problem, we revisit small-scale tabular language models (TaLMs) and extend chain-of-thought reasoning into TaLMs for the first time. Specifically, we propose a novel framework, TaCo, which coordinates two TaLMs responsible for CoT generation and answer inference, respectively.  Besides, our framework can be combined with an external calculator to enhance accurate numerical calculation. On the TABMWP dataset, TaCo outperforms the state-of-the-art ChatGPT by 9.55\% (82.60\%$\rightarrow$92.15\% in accuracy) with much less parameters (0.8B). The code will be released along with the paper.

Chain-of-Thought Reasoning in Tabular Language Models
Mingyu Zheng .2, Yang Hao', Wenbin Jiang', Zheng Lin 1:2
Yajuan Lyu', Qiaoqiao She Weiping Wang' Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China "Baidu Inc, Beijing, China
(zhengmingyu, wetggy,,,iizenrn linzheng, wangweipingg@@iie.ee ong ac. cn
{haoyango3, laapraanggg..iagngnneeeeeaa jiangwenbin, lvyajuan, http:/eeiin...uuuuuuuiiooiaaiaal com
Abstract
Day
Namber of baskets
Thursday
Tabular mathematical reasoning task requires models to perform multi-step operations includ- ing information look-up and numerical calcula- tions, based heterogeneous data from tables and questions. Existing solutions tend to extend chain-of-thougle (CoT) reasoning into power- ful large language models (LLMs) promote multi-hop mathematical reasoning. However, it can extremely difficult to apply such LLM- based approaches under scenarios of privatiza- tion deployment or limited resources. To ad- dress this problem, we revisit small-scale tabu- lar language models (Tal and extend chain- of of-thought reasoning into TaLMs for the first time. Specifically,
Friday
48
Saturday
Sunday
54
Monday
Tuesday
49
Rengan jotted down how many baskets she made during busketball practice each day. What. the mean the numbers?
Question:
Solution:
Read the numbers from table.
49, 48, 51, 54, 49 First, count how many numbers the group.
There 6 numbers.
Now add all the numbers together:
49 + 48 +-1 54 37 49 - 288
Now divide the sum by the number of numbers:
propose novel frame- work, TaCo, which coordinates two TaLMs responsible for CoT generation and answer inference, respectively. Besides, frame- work can be combined with an external cal- culator to enhance accurate numerical calcula- tions. On the TABMWP dataset, TaCo outper- forms the state-of-the-a ChatGPT by 9.55% (82.60% +92.15 92215% in accuracy) with much less parameters (O.8B).
288 6 -88
The mean
Answer:
An example from the TABMWP dataset. To solve the problem, the model needs to perform multi- step mathematical reasoning based the table and the question.
Figure I:
Considering the inherent demand for multi-step operations, existing studies tend to extend chain- of-thought (CoT) reasoning (Wei et al., 2022; Wang et al., 2023a; Kojima et al., 2022: Zhang et al., 2022) into powerful Large Language Mod- els (LLMs) (Brown al., 2020; Chowdhery et al.. 2022: Thoppilan al., 2022: Chen al., 2021a) a) to promote multi-hop mathematical reasoning. As depicted in Figure 2 (b), this paradigm prompts LLMs with several in-context examples containing CoT demonstrations to elicit intermediate reason- ing steps before inferring the final answer.
1 Introduction
Tabular mathematical reasoning task aims at an- swering math questions based on heterogencous tabular and textual data, which can provide users with insights from tables containing valuable fig- ures (Lu et al., 2023b; Zhu et al., 2021; Chen et al., 2021b). This task highlights the demand for multi- step mathematical reasoning including information look-up and numerical calculations. For example, given the table and the question Figure 1, we firstly need to count how many numbers are in the table, then add all the numbers together to get the sum of baskets, and finally compute the mean of the sum.
Though the combo of LLM and CoT has achieved great performance, such LLM-based methods may not be feasible approach in some real-world scenarios. For instance, it is financially expensive to satisfy the high computational require- ments, the storage capacity and the desired band- width of LLMs, which makes it a challenge for
'The code will be released httpss//githee com/ SpursGoZmy/ /aaCo
'This work was done during an internship Baidu Inc. Corresponding author: Zheng Lin.

---

individual users or small organizations utilize LLMs in their applications (Strubell al., 2019; Bender et al., 2021). consideration of the data security, enterprises may also seek privatization deployments where private data is not allowed to be processed by third-party LLM APIs. What's more, despite the fact that many pre-trained tab- ular language models have been developed (Liu al., 2022; Herzig al., 2020; Wang al., 2021; Dong al., 2022), their CoT reasoning ability has not been thoroughly investigated and it could be inadequate for solving the tabular mathematical reasoning task. As a result, an alternative approach, with lower costs and competitive CoT reasoning ability, is needed.
follows:
To the best of our knowledge, we explore the chain-of-thought reasoning in TaLMs for the first time, and advocate a new and promising paradigm for tabular mathematical reasoning, especially under scenarios where LLM-based methods not feasible.
We propose a novel framework, TaCo, which coordinates two TaLMs responsible for CoT generation and answer inference, respectively. It is also integrated with a calculator to en- hance accurate numerical calculations.
Our method can boost the performance of small-scale TaLMs and surpasses the state- of-the-art ChatGPT by 9.55% on TABMWP benchmark with much less parameters (0.8B).
To accomplish this goal, we revisit small-scale tabular language models (TaLMs) and initiatively explore the chain-of of-thought reasoning in TaLMs. Specifically, we propose a novel framework named TaCo, which coordinates two TaL.ss that are re- sponsible for CoT generation and answer inference. respectively. Given the input table and question, the first TaLM is fine-tuned to generate interme- diate reasoning steps. Based on the original input and generated reasoning steps, the second TaLM is fine-tuned to infer the final answer. To alleviate the weakness of TaLMs in solving mathematical expressions, TaCo also combined with an ex- ternal calculator which is used to perform math calculations and fix incorrect results in the output reasoning steps.
Pilot Experiment
Before diving into the specific method, we present pilot experiment the TABMWP dataset to swer two important questions: (i) Do existing pre- trained generative TaLMs possess chain-of-thouge of-thut reasoning ability? (ii) Whether generative TaLMs can benefit from Chain-of-thoughhe when predict- ing the final answer. We select the state-of-the- art TAPEX model (Liu et al., 2022) for experi- ments, which is based on the encoder-decoder lan- guage model BART (Lewis al., 2020) and is additionally pre-trained the tabular data. We consider two model sizes: TAPEX-base (140M) and TAPEX-large (400M).
To verify the effectiveness of the proposed method, we conduct comprehensive experiments on the TABMWP (Lu et al., 2023b) dataset, which is the latest math word problem benchmark over tabular data and provides detailed chain-of- thoughts to solve the problem step by step. Ex- perimental results reveal that TaCo explores new and promising paradigm for tabular mathematical reasoning, which is illustrated in Figure 2 (c). Com- pared with traditional fine-tuned TaLMs, TaCo im- proves the accuracy of recent TAPEX model by 29.76%. Compared with LLM-based approaches. TaCo outperforms the state-of-theeart ChatGPT by 9.55% 892.60%-92.115 with much less param- eters (O.8B). Moreover, we conduct ablation stud- ies to analyse contributions of different parts in the framework. The detailed error analysis is also performed provide insights for future improve- ments.
Experiments are conducted three different set- tings, i.c., vanilla, zero-shot CoT and gold COT. For the "vanilla" setting, the pre-trained TAPEX model f(-) autoregressively generates the answer based the table t and the question g. i.e., f(t, q). For the "zero-shot CoT" setting, we follow Ko- jima et al. (2022) to evaluate the CoT reasoning of the TAPEX. Specifically, a trigger sentence pi is appended to the question in order to ask the TAPEX to output intermediate reasoning steps s, i.e., 8 = f(t, PI). Then, given the original input and the generated CoT, another trigger sentence P2 is appended to make the TAPEX output the final answer i.e., a ʃ(l, q, &,, For pl, we try various templates such as "Let's think step by step" and report best results. For P2, we intuitively select "As a result, the answer is" as the trigger sentence. For the "gold CoT" setting, we replace
To summarize, we conclude our contributions as

---

Answer
Answer
CoT
Answer Inference Model (Finetuned)
Tabular LM (Finetuned)
LLM (Fixed
CoT
CoT Generation Model (Finetuned)
Table
Question
Table
Question
Tabk
Table
Question
(a) Finetuning Tabular Language Models
(b) Prompting Large Language Models with Chain-of-Though (CoT)
(c) Coordinating Tabular Language Models for CoT Reasoning and Exploitation (Ours)
Figure Different paradigms for tabular mathematical reasoning.
generated reasoning steps with annotated ones and other procedures are as "zoro-shot CoT"
generation and (ii) answer inference, where two generative TaLMs with the same architecture fine-tuned independently with different inputs and outputs. In this section, introduce the frame- work with the TAPEX model as selected backbones, should be noted that TaCo is compatible with arbitrary generative TaLMs to boost their perfor- mance. The overview of TaCo framework is illus- trated in Figure 3.
Pre-trained TaLMs
Acc-Dev
Acc- Test
TAPEX-base (vanilla)
15.66
15.69
TAPEX-larg -lagge (vanilla)
18.41
18.59
TAPEX-buse (zero-shot CoT)
15.30
15.25
TAPEX- TAPEX-large (zero-shot CoT)
18.25
17.94
TAPEX- (gold CoT)
40.54
39.99
TAPEX-large (gold CoT)
47.48
48.01
Pilot experimental results of pre-trained TAPEX under different settings. "Acc-Dev" and "Acc- Test" represents accuracy on the development and the test set respectively.
Table 1:
3.1 CoT Generation
In the CoT generation stage, a TAPEX model is fine-tuned to generate a solution which consists of multiple reasoning steps to solve the problem. Given input table T with M rows (R.) = and N column headers {c;) the TAPEX will linearize the table into flattened text sequence T [HEAD]
From the results in Table 1, the TAPEX with "zero-shot CoT" setting performs even worse than the vanilla one, which shows that the small-scale TAPEX is not decent zero-shot reasoner like L.LMs and does not possess CoT rea- soning ability. This is also consistent with find- ings from previous CoT studies (Wei et al., 2022; Ho et al., 2023). After inspecting the model out- puts, find that the pre-trained TAPEX model cannot follow the instruction to generate reason- ing steps. In most cases, directly generates the answer or illogical texts. However, given the anno- tated "gold CoT", the model achieves a remarkable performance gain. For instance, the accuracy of TAPEX-large on test set increases from 18.59% to 48.01% %. This demonstrates that CoT reasoning steps are beneficial to TAPEX when inferring the correct answer and it encourages to further elicit CoT reasoning ability of TaLMs by finetuning.
that
CN [ROW] 1 C1 | [RM, where [HEAD] R1 [ROW] 2 Rg ...
and [ROW] are special tokens used to indicate the region of column headers and rows, respectively. The number after [ROW] represents different row index and the vertical bar "I" separates headers cells in different columns. For instance, the table in Figure I will be linearized into the following sequence:
col : Day Number of baskets row 1 Thursday I 49 row 2 Friday 48 row
6: Tuesday 49
The resulting sequence T will be concatenated with the textual context, which includes question Q and trigger sentence Based on the concate- nated input, the probability of generating the target solution is computed as follows:
3 Method
Based on observations in Section 2, we propose the TaCo framework for tabular mathematical rea- soning. It includes two training stages: (i) CoT

---

External Calculator
Thurddy
48
Fullay
Firz, court how mare aumbers the aroup
First count how many aumbers the gropp
Saertay
Koosly
Sumbby
54
CaT Model (Finetuned)
85 2--ft 5) -4 37 40 3313 Now divide the sun Bhe number ofnumbess: 313 4 5-57
Now add
Theie numbers.
number togetler
Now add alll number together 63 65- 54 - 57 2688 Now divade the sam the number of sumbers
There nambers.
Model [Finetuned]
Inference
Itaedoy
48
Finel Answer
318 -6 6-448
Quentron: Reagan jette drwn how badoms she tesketholl proctie each What the rumbere? Question
The nenn ia S7.
The monn
Generated Solution
Corrected Solution
Figure Overview of the TaCo framework, with the table and the question in Figure as running example.
3.2
Answer Inference
In answer inference stage, another TAPEX model is fine-tuned to generate the final ansewr based on the original input and the annotated solution S. Similar with the CoT generation stage, the probability of generating target answer A is computed by:
L P(STT", ,, = 11 Q.... (1)
where L is the length of target solution. We select "Let's think step by step" as the trigger sentence P since it gives the best performance pilot experi- ments.
N II pa ..P.S.AAAAA
P(ATT" Q.
(2) where N the length target answer. During the inference phase, the annotated solution is replaced with the corrected solution to output the pre- dicted answer A. Both CoT generation model and answer inference model are trained with standard language modeling objective.
After generating a potential solution S, find that 5 often contains some numerical calculation errors. This is often the case with language models because TaLMs and even L.Mss are not suitable for actually solving mathematical expressions (Chen et al., 2022). Take the generated solution in Figure 3 as an example. Though the model generates plau- sible reasoning steps, calculation results among these steps are all wrong (in red color), e.g.. "49 + 48 51 54 37 49 312". Such calculation errors will accumulate the last reasoning step and seriously mislead the answer inference model into predicting the false answer.
4 Experiments
4.1 Dataset and Evaluation Metric
Experiments are conducted on the TABMWP (Lu et al., 2023b) dataset, a recent large-scale bench- mark which is constructed from grade-level math curricula and contains 38,481 math word problems with the tabular context. Beside the gold answers, TABMWP also provides detailed step-by-step solu- tions to solve the problems, which can be utilized chain-of thoughts to finetuning TaLMs. There are two question-typer in the TABMWP: 28,719 free- text questions with integer answers (INT) and deci- mal answers (DEC), and 9,712 multi-choice ques- tions with extractive text answers (EXTR), boolean text answers (BOOL) and other text answers (OTH). Statistics of each split are shown in the Table The test set contains 7,686 questions in total. Among them, 74.08% are INT (4,529) and DEC (1 (1165) questions, and 25.92% are DEC (1,165), EXTR (987) and OTH (105) questions. Thus, INT and
To mitigate the influence of calculation mistakes, we introduce arithmetic calculator g(-) to solve mathematical expressions of "+,-xx,"" in the gen- erated solution S and output the corrected solu- tion S g()). Concretely, we extract equation strings in S using regular expressions and calculate their results using the Python eval function. Since multiple equations may exist in one solution and one equation could also refer to results of previous cquations, the caleulation result of each equation is propagated to the following equations by string replacing. As can see from Figure original wrong results 5 are successfully fixed and are replaced with correct results (in green color), c.g., "49 + 48 + 51 + 54 + 37 49 288

---

Train
1evv
Test
Total
lection strategies for few-shot prompting are listed in Table 8. (3) Large language models with CoT prompting: Beside standard prompting, we also consider above LLMs with the chain-of-thought prompting. PromptPG (Lu et al., 2023b) utilizes the policy gradient method to select in-context examples for test samples when constructing the prompt for LLMs. PoT (Chen et al., 2022) pro- poses the "program-of-thougghe Codex to generate the text and Python program for math computations. The generated program is exe- cuted by a program interpreter to output the final answer. The "Heuristic guess" is a baseline from the TABMWP paper. For multi-choice questions, it randomly selects one from the given options with even probabilities. For free-text questions, it ran- domly chooses number from the question the table as the prediction.
of questions
23,059
7.686
7.686
38.431
* of free-text
17.135
5,710
5,694
28,719
of multi-choice
5.744
1,976
1,992
9,712
of tables
22,620
7.546
7.549
37,644
# of solutions
21,623
7,365
7,378
35,442
Table Dataset statistics of TABMWP.
DEC questions are more essential for the overall ac- curacy. Given the predicted answer and the ground truth, we employ the exact match accuracy as the metric and use the official evaluation script eval- uale the model performance.
which exploits
4.2 Implementation Details
Implementations. Our framework is imple- mented with Pytorch (Paszke et al., 2019). We mainly employ the TAPEX (Liu et al., 2022) as the backbone TaLM in the proposed framework. We also replace TAPEX with UnifiedQA (Khashabi et al., 2020) for the ablation study. Various model sizes are included to present more valid evaluation across different model capacities. Both CoT gener- ation model and answer inference model are opti- mized by AdamW (Loshchilov and Hutter, 2019). We validation set for the model selection and manually tune hyper-parameters, and evaluate the best model on the test set. For CoT generation, we adopt the beam search decoding with the beam size of 3. For answer inference, we adopt the greedy decoding. Hyper-parameter configurations for best- performing models and more implementation de- tails are shown in the Table 6 and Table 7.
4.3 Main Results
Table 3 demonstrates main experimental results on the TABMWP dataset. For TAPEX, UnifiedQA and ChatGPT baselines, we report results based on our implementation. For other baselines, we report published results from original papers (Lu et al.. 2023b; Chen et 2022).
From the results in Table 3, we can find that: (1) With two TAPEX-large models as backbones, the TaCo framework establishes new ofatc-of-the-art accuracy of 92.15% on the TABMWP test set, out- performing the previous best model ChatGPT with CoT prompting by 9.55% which demonstrates the effectiveness of the proposed method. Notably, compared with LLMs such as GPT-3 and Codex,
Baselines. (1) Pre-trained and Fine-tuned lan- guage models: We develop TAPEX (Liu et al., 2022) and UnifiedQA (Khashabi et al., 2020) in both pre-trained and fine-tuned settings to predict the final answer. TAPEX is the state-of-the-art BART-based (Lewis al., 2020) TaLM which is pre-trained on the tabular data to mimic a SQL executor. UnifiedQA is TS-based (Raffel al.,
the parameters in TaCo framework are much less (0.8B), which brings lower costs for application deployments. (2) Compared with LLM-based ap- proaches with the standard few-shot prompting, fine-tuned TAPEX and UnifiedQA can achieve competitive results. For instance, the fine-tuned TAPEX-large even performs better than GPT-3 and Codex. However, when combined with the CoT prompting, LLM-based methods are significantly better than fine-tumed small-scale language models, which shows that the CoT prompting plays an im- portant role in the tabular mathematical reasoning task. By contrast, the TaCo framework extends the CoT reasoning into TaLMs for the first time, and improves the performance of 1APEX-base and TAPEX-large model by 29.19% and 29.76%, re- spectively.
2020) QA model which is pre-trained on QA datasets of multiple formats. We consider three model sizes for UnifiedQA: small (60M), base (220M) and large (770M). Given the flattened table and question, both TAPEX and UnifiedQA can gen- crate the answer text autoregressivelly (2) Large language models: We consider GPT-3 (Brown al., 2020), Codex (Chen et al., 2021a) and Chat- GPT with the standard few-shot and zero-shot prompting. ChatGPT is based on the 3.t-3.5-turbo engine. Numbers of in in-tontext examples and se-

---

Question Types
Answer Types
Grades
Model
Acc-Dev
Acc-Test
FREE
MC
INT
DEC
EXTR
BOOL
OTH
1-6
7-8
Heuristic baselines
Heuristic guess
15.29
6.71
39.81
8.37
0.26
30.80
51.22
26.67
17.55
12.27
Human performance
90.22
84.61
93.32
84,95
83.29
97.18
88.69
96.20
94.27
81.28
Pre-truined LM
TAPEX-base
15.66
15.69
7.29
39.71
8.63
2.06
34.95
47.11
20.95
18.6
11.81
TAPEX-large
18.41
18.59
8.80
46.59
10.62
1.72
46.91
48.11
30.48
22.65
13.18
UnsfiedQA-small
10.71
12.18
1.18
43.62
1.37
0.43
38.7
49.78
37.14
15.57
7.65
UnificdQA-base
12.10
14.56
4.60
43.02
5.28
1.97
37.08
50.11
38.1
17.14
11.11
UnifiedQA-large
14.00
14.06
3.37
44.63
4.02
0.86
40.53
50.22
35.24
17.21
9.87
Fine-tuned I.M
TAPEX-base
$7.10
56.39
48.33
79.42
56.33
17.25
90.37
67.78
76.19
65.17
44.67
TAPEX-large
62.28
62.39
55.50
82.08
64.21
21.63
96.47
65.78
77.14
71.32
50.47
UnifiedQA-smal
35.79
34.82
27.99
54.32
33,94
4.89
52.99
53.89
70.48
42.23
24.93
UnifiedQA-brn
51.89
51.08
42.10
76.76
49.83
12.02
89.16
63.33
75.24
59.03
40.48
UnifiedQA-lare
59.35
59.26
51.62
81.12
60.68
16.39
92.20
69.44
77.14
67.11
48.80
LLM
GPT-3 (zero-shot)
56.96
53.57
66.67
55.55
45.84
78.22
55.44
54.29
63.37
48.41
GPT-3
57.13
54.69
64.11
58.36
40.40
75.95
52.41
53.02
63.10
49.16
Codex
59.40
CharGPT
12
65.52
65.84
64.61
66.55
63.09
74.67
54.67
55.24
69.75
59.88
I.M+CoT
GPT-3 (zero-shot)
$7.61
54.36
66.92
55.82
48.67
78.82
55.67
51.43
63.62
49.59
GFT-3
62.92
60.76
69.09
60.04
63.58
76.49
61.19
67.30
68.62
55.31
Codex
65,20
PromptPG
68.23
66.17
74.11
64.12
74.16
76.19
72.81
65.71
71.20 64.27
Codex-SC
75.40
PoT
73.20
PoT-SC
81.80
ChatGPT
82.49
82.60
80.89
87.50
79.36
86.87
81.86
94.00
84.76
82.68
82.51
Ours
TaCo (TAPEX base)
12.0.13.
85.5840.14
85.53
85.74
85 35.29
86
93.31
77.89
81,90
87 /3
83.12
TaCo (TAPEX-large 92.91=0.17 92.15+0.13
91.69
93.47
92.54
88.41
96.05
91.44
86.67
92.37 91.86
Table 3: Accuracy (%) on the development set and test set of TABMWP. We also report detailed accuracy on different types of questions in test set. FREE: free-text questions; MC: multi-choice questions. INT: integer answers; DEC: decimal answers; EXTR: extractive text answers; BOOL: Boolean text answers; OTH: other text answers. The best results are marked in bold. stands for standard deviation over 3 repeated experiments. If otherwise specified, L.M baselines are in few-shot setting. "-SC" represents using self-consistency decoding strategy (Wang et al., 2023a).
(3) Among different baselines, the model per- formance on free-text questions is obviously worse than that multi-choice questions, with age difference of 21%. The reason is that, com- pared with multi-choice questions, free-text ques- tions usually require more complicated numerical calculations and also do not directly provide an- swer options in the input. The detailed evidence is presented in the Appendix B. Nevertheless, from pre-trained LM to LLM+CoT and to the proposed TaCo framework, the performance gap between two question types gradually decreases. For in- stance, the accuracy gap of TaCo (TAPEX-large) framework (1.78%) much lower than that of fine-tuned TAPEX-large (26.58%). This shows our method can obtain better generalization on two types of questions. (4) Considering questions of various answer types, the TaCo framework beats
other baselines questions with integer (INT) and decimal (DEC) answers, which may resulted from the utilization of the external calculator. ChatGPT with the CoT prompting outperforms other meth- ods including the human baseline on questions with Boolean text answer, which may contribute to its great general semantic understanding ability. For example, judging yes/no questions based on previ- ously generated reasoning steps. (5) Not surpris- ingly, all the models perform worse questions from the grade 7-8 than that from the grade 1-6 due to the increasing difficulty. Among them, the pro- posed framework achieves the best accuracy than other baselines on harder questions from grade 7-8,
4.4 Ablation Study
We conduct ablation experiments to systematically investigate the effeet of the external calculator, the progressive two-stage paradigm and the TaLM

---

Settings
Dev
Test
Average
Drop.
FREE
Question
Types MC
Model
Dev
Test
Question
FREE
Types
MC
OUl'S
w/ TAPEX
TaCo (hase)
86.12
85,58
85.53
85.74
TAPEX-base
86.12
85.58
85.53
85.74
TaCo (large)
92.91
92.15
91.69
93.47
w/o calculator
TAPEX-large
92.91
92.15
91.69
93.47
Q7 > A (base)
65.21
64.35
21.07
56.23
84.55
w/ UnifiedQA
QT + 8 + A (large)
75.60
74.58
17.44
67.77
93.03
BarfedQQA-saall
48.32
48.17
46.45
53.06
w/o two-stage paradigm
UnifiedQA-bass
66.32
65.46
60.70
79 07.07
QI SA (base)
78.22
77.66
7.91
77.15
79.12
UnifiedQA -large
77.44
76.96
73.50
86.85
QT + SA (large)
84.73
84,25
8.04
83.95
85.14
fine-tuned
QI + AS (base)
75.18
74.34
11.09
71.88
81.38
UniiedQA-small
35.79
34.82
27.99
54.32
AS (large)
81.45
81.41
11.10
80.21
84.84
UnifiedQA-basse $1.89
51.08
42.10
76,76
w/o two-stage paradigm and calculator
QI - S.A (base)
59.69 59.41
26.30
50.86
83.84
UnifiedQA-large 59.35
59.26
51.62
81.12
QT SA (large)
69.57 68.85
23.32
63.79
83.33
QT + AS (base)
56.43 54.85
30.21
45.64
81.17
Table 5: Experiment results of TaCo framework with
QT > AS (large)
63.80 63.41
28.
56.06
84,44
TAPEX and UnifiedQA as backbone, respectively.
wfo two-stage paradigm, calculator and solution
QT + A (base)
57.10 56.39
29.11
48.33
79.42
QT' > A (large)
62.28 62.39 30.20
55.50
82.08
two TaLMs, respectively, More importantly, one- stage paradigms cannot fully utilize the corrected CoT to change the original (wrong) answer. By contrast, the two-stage paradigm brings a second chance to re-contemplate the improved reasoning steps before making the final judgement. The simi- lar two-stage paradigm has also been explored in recent works (Press et al., 2023; Zhao et al., 2023), where they utilize one LLM to generate the CoT to be improved, and then ask the same LLM infer the final answer based on the improved CoT.
Table 4: Ablation study of the external calculator and proposed two-stage paradigm. "base" and "large" stands for model sizes of TAPEX backbone.
backbone in the TaCo framework. QT -> - represents the proposed two-stage paradigm, which firstly generates the solution S and then arrives at the final answer A based on the input question Q, table T and generated solution S. QT + SA and QT + AS represents one-stage paradigms, which generate the solution and the answer in different or- ders, respectively. QT + stands for the vanilla fine-tuning paradigm that directly predicts the an- swer.
Comparing two one-stage paradigms. we notice that QT + SA performs better than QT + AS. This shows that it may be more suitable for TaLMs to infer the final answer according to produced reasoning steps, rather than give explanations based on the predicted final answer. If we remove both the two-stage paradigm and the external calculator, the model performance would suffer a more steep decline. But it is still better than that of traditional fine-tuned models in QT A paradigm, which validates the value of intermediate reasoning steps for TaLMs.
Effect of External Calculator. As shown in Ta- ble 4, there is drastic performance drop for the TaCo framework (e.g., 92.15% + 74.58%) when removing the external calculator. With further ob- servations, we find that the performance decline mainly comes from free-text questions which de- mand more numerical calculations. For instance, the accuracy of TaCo (TAPEX-large) plummets from 91.69% to 67.77%. It demonstrates the great significance using the external calculator to re- duce calculation errors in the generated solutions. Otherwise, the answer inference model is likely to be misled by the incorrect solution and arrives at the wrong answer.
Effect of TaLM Backbone. To investigate the performance of TaCo with different backbones, we replace TAPEX with UnifiedQA as the backbone model. Related experimental results are presented in Table 5. When the backbone changes from TAPEX to UnifiedQA, the TaCo framework suf- fers a sharp performance drop on both free-text and multi-choice questions. For instance, even with more parameters (1.54B), the accuracy of TaCo with UnifiedQA-large on the test set (76.96%) is much lower than that with TAPEX-large (92.15%). which indicates the advantages pre-trained tabu- lar language models. Unlike UnifiedQA which is solely pre-trained on the unstructured textual data,
Effect of Two-stage Paradigm. When we
change the two-stage paradigm to one-stage ones, the model performance drops about 9.5%, which reveals the contribution of two-stage paradigm. We think is challenging for single small-scale TaLM to generate correct reasoning steps and the final answer simultancously. As a result, we delegate the CoT generation and the answer inference to

---

tion, e.g., compute the slope of the function based on the table data.
11%
Counting Operation
Fraction Calculation
36%
Wrong Formula
For multi-choice questions, error cases can be divided into the following five types. (1) Number comparison (44%): the model cannot determine which number is larger or smaller. (2) Time cal- culation (21%): the model needs to perform time caleulation such as compute the elapsed time be- tween 9:15 A.M. and 11:20 A.M.. (3) Max/Min operation (19%): the question demands finding the biggest or smallest number in a group. (4) False CoT (9%): the CoT generation model gives wrong or hallucinated reasoning steps, e.g., using numbers that do not exist in the table or the question when generating formulas. (5) Commonsense (7%): the commonsense knowledge is needed to answering the question, which is weakness of small-scale language models.
Function Problem
49%
(a) free-tex questions.
7%
9%
Number Comparison
Time Calculation
Max/Min Operation
Falso CoT
44%
Commonsense
21%
19%6
(b) multi-choice questions.
Figure Error distributions different question types.
Related Work
TAPEX is additionally pre-trained the tabular data and thus has better understanding of table structures. As more powerful generative TaLMs emerge, they can be integrated into the TaCo frame- work to improve their performance on the tabular mathematical reasoning task.
CoT prompting for LLMs. By providing few in-context examples (or demonstrations) which contain chain-of-thoughts, CoT prompting can courage LLMs output intermediate reasoning steps before predicting the final answer (Wei al., 2022). Existing CoT studies mainly focus on two directions. (1) Improving the quality of CoT demonstrations. For instance, selecting better in- context examples for CoT prompting according to the question diversity (Zhang et al., 2022), the so- lution complexity (Fu et al., 2023), the example similarity (Rubin et al., 2022). (2) Exploring representations of CoT reasoning steps. Beside the typical natural language format, researchers also proposed chain ofain-of-thoughts in other formats. For instance, program-of-thouue of-uuhh (Chen et al., 2022), tree-of-thoughts (Yao et al., 2023a), and graph-of- thoughts (Yao et al., 2023b). Among them, the CoT in program languages has emerged as powerful approach for LLMs to invoking external tools (Qin al., 2023). Recently, Lu al. (2023a) proposed the Chameleon framework that augments LLMs with various tools like search engines and Python exccutors. We treat as a contemporary work our paper and list its results in the Appendix D.
4.5 Error Analysis and Case Study
As illustrated in Figure 6, for this problem that involves two multiplication and one addition oper- ations, the TaCo framework successfully generates correct intermediate reasoning chains and finally predicts the right answer.
There are 473 free-text questions (78%) and 130 multi-choice questions (22%) for which the TaCo (TAPEX-large) gives wrong predictions. We ran- domly selected 100 questions each type for error analyses. Figure 4 depicts error distributions by question types. More error instances are presented and discussed in Appendix C.
For free-text questions, error cases fall into the following four categories. (1) Counting operation (49%): the question requires the model to count numbers as the final answer, which is challenging for generative language models. (2) Fraction calcu- lation (36%): the model fails to conduct fraction- related caleulations such as reducing fraction, which may be alleviated with advanced calcu- lator. (3) Wrong formula (11%): the CoT genera- tion model outputs wrong formulas in the reason- ing steps. (4) Function-relatee problem (4%): the model fails to solve problems related to the func-
Pre-trained TaLMs. Inspired by the success of pre-training on the natural language text, various TaLMs are proposed for pre-training on the semi- structured tabular data (Dong et al., 2022). Ex- isting TaLMs mainly inherit the architectures traditional language models and can be classified

---

into three types. (1) Encoder-based TaLMs like TAPAS (Herzig et al.. 2020), MATE (Eisensch- los et al., 2021) and TUTA (Wang et al., 2021).
scenarios where LLMs are not available.
Ethics Statement
(2) Gmcoder-Decodee TaLMs such as TAPEX (Liu et al., 2022) and STTP (Xing and Wan, 2021). (3) Decoder-based TaLMs like TableGPT (Gong et al., 2020). In previous studies, TaLMs are usually fine- tuned to directly generate final answers simple formulas. By contrast, are the first to explore the combination of the CoT reasoning and pre-trained TaLMs.
This paper proposes a two-stage framework for the tabular mathematical reasoning task, and models are trained and evaluated on the public TABMWP dataset. Thus, the authors foresee no ethical con- cerns with the research in this paper.
A Acknowledgements
This work supported by the National Natural Science Foundation of China (No. 61976207) and the National Social Science Foundation of China (No. 21AZD145).
6 Conclusion
We extend the CoT reasoning into small-scale TaLMs for the first time, and provide an effective approach for tabular mathematical reasoning task, especially under scenarios where LLMs are not accessible. Specifically, we propose novel frame- work named TaCo, which coordinates two TaLMs responsible for CoT generation and answer infer- ence, respectively. By introducing an external cal- culator, we further augment TaCo with the accurate math computing ability. With two TAPEX-lare -large models as backbones, the TaCo outperforms the statc-of-the-art ChatGPT on the TABMWP dataset by 9.55% (82.60% >92.15%) with much less pa- rameters (0.8B).
References
Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language mod- els be too big? In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Trans- parency, FAccT 22,, page 610-623, New York, NY. USA. Association for Computing Machinery.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates. Inc.
Limitations
Though the proposed method achieves great perfor- mance with less parameters, the fine-tuning of the CoT generation model and the answer inference model depends on annotated chain-of-thoughts and th-thh
gold answers. As a result, the reasoning ability of TaCo could be limited to the tabular mathematical reasoning task. In the future research, one can utilize open-source LLMs to gen- erate chain-of-thoughts of more diversities and of more table-related tasks (Wang et al., 2023b; Ho al., 2023), which may further extend the gener- alization ability of TaLMs and reduce the cost of manual annotation.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Mathias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert-Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa. Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 202 la. Evaluat- ing large language models trained on code.
In the aspect of external tools, compared with frameworks which enable LLMs to access various tools (Shen et al., 2023; Lu et al., 2023a), TaCo only utilizes calculator to complete common arith- metic calculations, i.e., "+.-x,+"". More advanced external tools may be integrated enhance the capability of the framework. We believe that the tool learning with small-scale language models is a valuable future direction, especially for particular

---

Wenhu Chen, Xueguang Ma, Xinyi Wang. and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks.
Jonathan Herzig, Pawel Krzysztof Nowak. Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics, pages 5520-4333, Online. Association for Computa- tional Linguistics.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa,
Matt Beane, Ting-Hao Huang. Bryan Routledge, and William Yang Wang. 2021b. FinQA: A dataset of nu- merical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 3697-3711, Online and Punta Cana, Dominican Republic. Association for Computational I Linguistics.
Namgyu Ho, Laura Schmid, and Se- Young Yun. 2023. Large language models are reasoning teachers.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- nanch Hajishirzi. 2020. UNIFIEDQA: Crossing for- mat boundaries with a single QA system. In Find- ings of the Association for Computational Linguistics: FMNLP 2020, pages 1896-1907, Online, Association for Computational Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaram, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz. Orhan Firat, Michele Catasta. Jason Wei, Kathy Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- cling with pathways.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models zero-shot reasoners. In Advances in Neural Information Processing Systems.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zetllemoyer. 2020. BART: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 7871-7880. Online. Association for Computa- tional Linguistics.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin. Weizhu Chen, and Jian-Guang Lou. 2022. TAPEX: Table pre-training via learning a neural SQL executor. In International Conference on Learning Representations.
Haoyu Dong, Zhoujun Cheng, Xinyi Hc, Mengyu Zhou, Anda Zhou, Fan Zhou, Ao Liu, Shi Han, and Dong- mei Zhang. 2022. Table pre-training: A survey on model architectures, pre-training objectives, and downstream tasks. In Proceedings of the Thirty-First International Joiní Conference on Artificial Intel- ligence, IJCAI-22, pages 5426-5435 International Joint Conferences on Artificial Intelligence Organi- zation. Survey Track.
Ilya Loshchilov and Frank Hutter, 2019. Decoupled weight decay regularization.
Pan Lu, Baolin Peng, Hao Cheng. Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023a. Chameleon: Plug-and- play compositional reasoning with large language models.
Julian Martin Eisenschlos, Maharshi Gor, Thomas Müller, and William W. Cohen. 2021. Mate; Multi- view attention for table transformer efficiency.
Pan Lu, Liang Qiu. Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023b. Dynamic prompt learn- ing via policy gradient for semi-structured mathe- matical reasoning. In International Conference on Learning Representations (ICLR).
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-base prompting for multi-step step reasoning.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Yang. Zach De Vito, Martin Raison, Alykhan Tejani. Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Jun- jie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, perf-peeformance deep learning li- brary.
Heng Gong. Yawei Sun, Xiaocheng Feng. Bing Qin, Wei Bi, Xiaojiang Liu, and Ting Liu. 2020. TableGPT: Few-shot table-to-text generation with table structure reconstruction and content matching. In Proceedings of the 28th International Conference on Computational Linguistics. pages 1978-1988, Barcelona, Spain (Online). International Committee on Computational Linguistics.

---

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models.
Hajishirzi. 2023b. Self-instruct: Aligning language models with self gefereted instructions.
Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. 2021. Tuta: Tree- based transformers for generally structured table pre- training. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery amp; Data Min- ing, KDD '21. page 1780-1796 New York, NY, USA. Association for Computing Machinery.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding. Ganqu Cui, Zheni Zeng, Yufei Huang. Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang. Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang. Xingyu Shen, Bokai Xu, Zhen Zhang. Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan. Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang. Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023. Tool learning with foundation models.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi. Quoe V Le, and Denny Zhou 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Xinyu Xing and Xiaojun Wan. 2021. Structure-aware pre-training
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former.
tablett-ttext generation. In Find- ings of the Association for Computational Linguis- tics: ACL-IJCNLP 2021, pages Association for Computational Linguisties.
Online.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023a. Tree of thoughts: Deliberate problem solving with large language models.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in hugging face.
Yao Yao, Zuchao Li, and Hai Zhao. 2023b, Beyond chain-of thought effective graph-of- of-thhugttthhtttt reason- ing in large language models.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning nip.
Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. 2023. Verify-and-ediit A knowledge-enhanced ofain-of-thought framework.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yamping Huang, Maxim Krikun, Dmitry Lepikhin. James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung- Ching Chang, Igor Krivokon. Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-Johh Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoe Le. 2022. Lamda: Language models for dialog appli- cations.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang. Chao Wang. Shuo Zhang. Jiancheng Lv, Fuli Feng, and Tat- Seng Chua. 2021. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. CoRR, abs/2 105.07624.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023a. Self-consistency improves chain of thought reasoning in language models.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh

---

A More Implementation Details
B The complexity of CoT generation
In our experiments, we employ TAPEX and Uni- fiedQA as backbones of TaCo framework. When linearizing the table into flattened sequence, there exist no column headers in the original ta- ble, pseudo column headers will be inserted, e.g.. "Column header 1'. The hyper-parameter config- urations of TAPEX and UnifiedQA backbone and their model sizes are shown in Table 6 and Table 7, respectively. Our experiments are all performed on a 32G NVIDIA V100 GPU.
Table 3 reveals a significant performance difference between free-text questions and multi-choice ques- tions. To shed more light on the TABMWP dataset, we quantitatively analyze the complexity of the CoT generation for two question types. Specifi- cally, we compute the number required numer- ical calculations in the gold CoT (including +, x, +, counting, min, max), the number of reason- ing steps (we treat each line in the gold CoT one reasoning step for simplicity) and the length of the gold COT. The statistical results in the Table 9 demonstrate that, in the TABMWP dataset, the CoT generation from free-texi questions more com- plex than that from multi-choice questions. Based on our observations, at least 18% multi-choice ques- tions (mainly of EXTR and OTH answer types) do not need numerical calculations, but almost all free- text questions need numerical calculations.
For LLM-based baselines, we list numbers of few-shot examples and selection strategies in Table 8. For ChatGPT baseline, we randomly select 4 examples from train set for each question type. For fair comparison, we use the same prompt format as PromptPG (Lu et al., 2023b) to construct in-context examples, which is demonstrated in Figure 5.
TAPEX
Parameters
base (140M)
large (400M)
Learning Rate
3e-5
3e-5
Batch Size
16
32
Weight Decay
0.01
0.01
C Error Instances and More Analysis
Max Grad Norm
1.0
1.0
Warmup
Linear
Linear
Warmup Fraction
0.1
0.1
In this section, we present detailed error instances to analyze the weakness of TaCo framework, which is shown in Figure 7 to Figure 10. We find that most of errors are caused by the inability of used external tool and the representation of chain-of- thoughts. Take the error instance in Figure 7 an example. To correctly answer the question in Figure 7, the model should find numbers from the table which are greater than 53, and then count how many numbers are found. However, as the CoT generation model is fine-tuned to generate chain-of of-thoughts in simple natural language, it is difficult for the model to describe the above process in short and straightforward expression, which makes it hard to invoke external tools. If we could represent chain-of-thoughe in program languages like Python, the solution of this ques- tion would be much more clear. For instance, one can write a line of Python code: "Ans = C61 ,,,,6 65, 65, 66, 70, 66, 78)", and imple- ment a Python function "Count ()" as an externa tool to get the accurate result. The same method- ology could be applied to error instances which demand other abilities such as fraction calculation, min/max operation and time calculation. Besides. lacking commonsense knowledge also increases the difficulty for models to comprehend tables and questions, e.g., reading bus schedule in Figure 10.
Epochs Stage
20
25
Epochs for Stage
1.
20
Truining Time for Stage Training Time Stage
hours hours
8 hours
hours
Table 6: Hyper-panee parameter configurations for TAPEX backbone.
UnifieOOA (220K))
Paramederr
small ((IM)
large (770M))
Leaming Rate
5c-5
5o-5
5e-5
Batch, Size
16
16
48
Weight Decay
0.01
0.01
0.01
1.0
Warmup
Linear
Linear
Lineur
Warmup Fraction
0.1
0.1
0.1
Epochs for Stage
15
20
25
Fpochs Stage
15
15
Training Time for Stage
hours hours
hours
15 hours
Training Time for Stage
hours
12 hours
Table 7: flyeer-arameee configurations for UnifiedQA backbone.
# few-shot
Sclection strategy
Method
Acc-Test
examples
GPT-3
Random selection
57.13
Codex
Manual construction
59.40
GPT:3+CiT
Random selection
62.92
Codex+CoT
Manual construction
65.20
PremptPG
Policy Gradicat
68.23
Manual construction
73.20
ChaGGP
Random selection
65,52
CharGPTCCoT
Random selection
82.60
Table 8: Number of in coneott examples and selection strategies L.M baselines.

---

Table:
apron
$20.00
apron $20.00
soup pot
$45.00
soup pot $45.00
rolling pin
$12.00
rolling pin| S12.00
set of forks
$13.00
set of forks $13.00
serving platter
$10.00
serving platter $10.00
cookbook
$16.00
cookbook $16.00
Question: How much money does Noah need to buy apron and soup pot? (Unit: S)
Answer:
Add the price of an apron and the price of a soup pot:
$20.00 + $45.00 = $65.00
Noah needs $65.00.
The answer is 65.
Figure 5: The format of in-context examples for ChatGPT baseline (ID:19324).
# of numerical calculations (median/mean))
# of reasoning steps (median/mean)
the length of CoT (median/mean) 196.00/239.15 180.00/253.21
Question Types
free-text multi-choice
2.00/2. 15
4.00/5.18 18 2.00/3.84
1.00/1.78
Table 9: The quantitative analysis the complexity of the CoT generation for two question types.
D Results of Chameleon framework
Recently, Lu al. (2023a) proposed compo- sitional reasoning framework named Chameleon, which treats LLMs as a natural language planner to utilize a variety of tools including vision models, web search engines, Python functions and so on. As shown in Table 10, based on the powerful GPT- 4 and multiple external tools, Chameleon achieves the best accuracy of 98.78 78% TABMWP test However, the proposed TaCo framework still achieves a competitive result of 92.15% with less parameters.
Question
Types
Method
Acc-Test
FREE
MC
ChatGPT CoT
82,03
78.43
92.32
ChatGPT PoT
89.49
90.24
87.35
We also apply the same calculator to the output of ChatGPT and use regular expressions to extract the final answer from the output. There is slight performance increase from 82.60% to 83.07%. Af- found
GPT-4 CoT
90.81
88.48
97.49
GPT-4 PoT
96.93
97.40
95.58
Chameleon (ChatGPT)
93.28
93.13
93.72
Chameleon (GPT-4)
98.78
98,95
98.29
TaCo (Ours)
15+00.1
91.69
93.47
ter inspecting error cases of ChatGPT, that most emrors resulted from wrong reasoning steps rather than calculation mistakes. Compared with small-scale TaLMs, the numerical calculating ability ChatGPT is much more better, which may attribute to the potential use of more advanced external tools behind the ChatGPT system.
Table 10: Accuracy Chameleon on TABMWP test

---

caramel
$1.13 kg
Question: How much would cost to buy kilograms of kettle corn and 2 kilograms Parmesan popcorn? (Unit: S)
chocolate popcorn
$1.78 per kg
Generated Solution: Find the cost of the kettle comn. Multiply: $0.69 x 3 $2.07.
kettle corn
$0.69 per kg
Find the cost the Parmesan popcorn. Multiply:
ranch popcorn
$1.72 per kg
$1.41 x $2.82.
cheddar popcorn
$0.90 per kg
Now find total cost by adding:
$2.07 $2.82 $4.89. would $4.89,
Parmesan popcorn
$1.41 per kg
Predicted Answer: 4.89
Figure 6: A correct instance where TaCo generates right solution and answer. (ID:752).
Broken tiles per shipment
Question: Clarksville Flooring tracked number of broken tiles each shipment received last year. How many shipments had at least 53 broken tiles? (Unit: shipments)
Stem
Leaf
2
9
Solution of TaCo:
1689
Find the row with stem 5.
4
123799
Count all the leaves greater than or equal to 3. Count all the leaves in the rows with stems 6 and 7. You counted leaves.
5
6
11356
shipments had at lenst 53 broken tiles. Answer of TaCo: shipmemls Correct Answer:
7
068
Figure 7: An error instance of counting operation (ID:449), where TaCo cannot correctly count how many numbers satisfying requirements.
Drink preferences
Question:
Students Fairfix High School were polled regarding their favorite drinks. What lluction students preferred fruit punch? Simplify your mewee.
Drink
Number of students
Solution TaCo:
Find how many students preferred fruit punch: 680 many
Diet cola
450
were pulled in lotal:450 450
320 1580
Divide 680 by 5500
Fruit punch
680
1580
Reduce the fraction:
44
Lemon-lime soda
130
44 5a
students preferred punch...
Other
320
Answer of TaCo:
Correct Answer:
58
79
Figure 8: An error instance of fraction calculation (ID:1711), where TaCo makes mistakes when reducing fraction.
play ticket
$24.00
Question: Dominic has $72.00. Does he have enough to buy play ticket and
ticket?
Option: (A) (B)
comedy show ticket
$49.00
Solution of TaCo:
boat show ticket
$23.00
Add the price of play ticket and the price of soccer ticket:
$24.00 $50.00 $74.00
soccer ticket
$50.00
Since Dominic has $72.00, be has enough money.
Answer of TaCo: yes
Correct Answer: No
Figure 9: An error instance of number comparison (ID:1434), where TaCo cannot correctly judge which is the larger number between 72.00 and 74.00.
schoddde
Look the following schedule. Ruth just missed the 11.00 A.M. bus the mall. How long does she have to wait until the next bus?
Question:
666AM
$15AM
AM
AM
I0A5AM
Te Sirutty
500AM
$30AM
AM
AM
1130AM
Option:
(B) 30 minutes
(D)
Tie real
M
MM
PM
12:5RM
PM
Solution
Find 11:00 A. M. in the for the mall.
12:00 PM
12:30 194
3A
PM
Look for the next hus in that now.
decer's office
PM
PM
PM
PM
12:00 P. Find the elapsed time. The elagsed time is 30 minutes.
The
12:45
116
216PM
PM
Es SOWKE FRGIOU
1.00 PM
PM
PM
Answer TaCo: 30 minutes. Correct Answer: hour
Figure 10: An error instance time calculation (ID:2766). where TaCo fails to compute the elapsed time between 11:00 A.M. and 12:00 P.M.


Reviews:
[{"review_id": "5LjNcudJJv", "reviewer": "Reviewer_UoLv", "summary": "", "questions": "1. In (3) of subsection 4.3, I am not so sure why only free-text questions requires the model to perform complicated numerical calculation. As to answer the question, TaCo always generate the CoT first, then distill into the final answer. Thus, from my intuition, the complexity of the CoT generation in two setting free-text or CoT is roughly similar. Please elborate it.\n\n2. About the effect of two-stage paradigm, from my point of view, the role of Answer Inference Model is not significant. In fact, a piece of hard code using regex can cover almost all cases of output to extract the final answer. Thus, I am skeptical about the result of QT -> SA in Table 4. To make the claim about the importance of Answer Inference Model more convincing, some case studies are needed (better if the code/algo to extract final answer is provided).", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the thorough review and valuable comments. We are delighted to learn that our paper \"gives readers many insights\" together with \"an interesting organization\" which better equips readers with the research context. Moreover, we are pleased to get your constructive feedback which is very useful to further improve our work. We hope the following responses will adequately address your concerns.\n\n**Q1: Why only _free-text_ questions require the model to perform complicated numerical calculations?**\n\n**A1:** We are sorry that some expressions in Section 4.3 (3) result in some confusion and ambiguity. In fact, we intended to point out that, **compared with _multi-choice_ questions, _free-text_ questions in the TABMWP dataset usually require more complicated numerical calculations,** which makes it more difficult to obtain correct answers.\n\nThe evidence is as follows.\n\n1) _Free-text_ questions include two answer types, i.e., integer answers (INT) and decimal answers (DEC), and _multi-choice_ questions include three answer types, i.e., extractive text answers (EXTR), boolean text answers (BOOL) and other text answers (OTH). Based on our observations, at least 18% _multi-choice_ questions (mainly of EXTR and OTH answer types) do not need numerical calculations, but almost all _free-text_ questions need numerical calculations.\n\n2) **We quantitatively analyze the complexity of the CoT generation for two question types.** We compute the number of required numerical calculations in CoT (including +, -, ×, ÷, counting, min, max), the number of reasoning steps in CoT (we treat one line as one reasoning step for simplicity) and the length of CoT. The results are shown below. \n\n| Question Types | # of numerical calculations  (median/mean) | # of reasoning steps (median/mean) | the length of CoT  (median/mean)  |\n| :---: | :---: | :---: | :---: |\n| _free-text_ | 2.00/2.15 | 4.00/5.18 | 196.00/239.15 |\n| _multi-choice_ | 1.00/1.78 | 2.00/3.84 | 180.00/253.21 |\n\nStatistical results reveal that, in the TABMWP dataset, the CoT generation from _free-text_ questions is more complex than that from _multi-choice_ questions, which finally leads to different model performance. Related expressions in the paper will be revised to make them more clear and above evidence will be added to support our claim.\n\n**Q2: The role of Answer Inference Model is not significant. It can be replaced by regular expressions.**\n\n**A2:** Thank you for this valuable question. In the following analysis, we showcase two important effect of Answer Inference model.\n\n1) **Answer Inference model handles different CoTs with flexible answer locations and various answer types.** If we directly use regular expressions to extract the final answer from the generated CoT (we can denote this method as \"QT-->S\"), it is very difficult to cover different cases where multi-type answers may occur at arbitrary locations, which are demonstrated in the following CoTs with bold answer text.\n\n> … Estelle will get to Princeton at **10:30 P. M.** (ID: 1105)\n>\n> … the constant of proportionality is **2** pounds per plate. (ID: 1136)\n>\n> … **2** boxes had at least 44 broken crayons but fewer than 54 broken crayons. (ID: 1326)\n>\n> ... In 1995, **cantaloupes** cost more. (ID: 2394)\n>\n> ... An airplane calendar costs $**3.13** more than a map of Africa. (ID: 2643)\n\nThus, it is desirable to train an Answer Inference model which can automatically infer final answers based on different CoTs.\n\n2) **Answer Inference model brings a second chance to recontemplate improved reasoning steps before making the final judgement.** For the \"QT-->SA\" method, we construct a concluding sentence \"As a result, the final answer is {answer_text}\" and append it to the end of CoT, which is used to train the TaLM to generate reasoning steps and the final answer in one forward pass.  Then, the regular expression is adopted to extract the final answer from the model output:\n\n```python\nimport re\nret = re.match('.*the final answer is (.*)', model_output)\nfinal_answer = ret.group(1)\n```\n\nThe \"QT-->SA\" indeed reduces the difficulty to extract the final answer. However, **such one-stage paradigms cannot fully utilize the corrected CoT to change the original (wrong) answer.** In the following example (ID: 22198), though errors in the CoT are corrected by the external calculator, the original wrong answer is not changed. By contrast,  **the two-stage paradigm \"QT-->S-->A\" can take full advantage of the corrected CoT to infer the right answer instead of jumping to the conclusion directly.**\n\n>**Question:** Shelley has $272.00. Does she have enough to buy a lawn mower and an iron bench? (A) yes (B) no\n>\n>**Original Output of QT->SA:**\n>\n> Add the price of a lawn mower and the price of an iron bench:\n>\n> 145.00 + 140.00 = \\$265.00\n> \n> As a result, the final answer is yes.\n>\n>**Corrected Output of QT->SA:** \n>\n> Add the price of a lawn mower and the price of an iron bench:\n>\n> 145.00 + 140.00 = \\$285.00\n>\n> As a result, the final answer is yes.\n>\n>**Final Answer of QT->SA:** yes\n>\n>**Correct Answer:** no\n\nIn addition, **it is challenging for one small-scale TaLM to conduct both CoT reasoning and answer inferring**. For example (ID: 1937), the \"QT->SA\" chooses the wrong number as the final answer more often than \"QT->S->A\".\n\n> **Output of QT->SA:** ... 5 amusement parks have at least 10 roller coasters but fewer than 40 roller coasters.\nAs a result, the final answer is 10.\n>\n>**Final Answer of QT->SA:** 10\n>\n>**Correct Answer:** 5\n\n**The similar two-stage paradigm has also been adopted in recent work** like Verify-and-Edit[1] and Self-ask[2], where they first utilize one LLM to generate the CoT and then ask the same LLM to infer the final answer based on the improved CoT. By contrast, we delegate CoT generation and answer inference to two small TaLMs, respectively. In the next version of the paper, more case studies will be included to highlight the importance of Answer Inference model.\n\n**Q3: Reproducibility problem, e.g., The settings of parameters are underspecified or subjectively determined.**\n\n**A3:** Due to space limitation, implementation details including parameter settings are listed in the Section 4.2 and Appendix A. We will add more implementation details about ablation studies and the code will be released along with the paper.\n\n[1] Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework. ACL 2023.\n\n[2] Measuring and Narrowing the Compositionality Gap in Language Models. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The organization of this paper is quite interesting. It shows the pilot the study as the motivation to conduct a larger scale study on the tabular mathematical reasoning problem, which better equips readers with the context of this research.\n2. It conducts a very extensive evaluation, comparing its models with a lot of other settings. The paper also analyzes the result and ablates the model in different aspects, giving readers many insights about the model.", "reasons_to_reject": "Please refer to questions.", "questions_to_authors": "1. In (3) of subsection 4.3, I am not so sure why only free-text questions requires the model to perform complicated numerical calculation. As to answer the question, TaCo always generate the CoT first, then distill into the final answer. Thus, from my intuition, the complexity of the CoT generation in two setting free-text or CoT is roughly similar. Please elborate it.\n\n2. About the effect of two-stage paradigm, from my point of view, the role of Answer Inference Model is not significant. In fact, a piece of hard code using regex can cover almost all cases of output to extract the final answer. Thus, I am skeptical about the result of QT -> SA in Table 4. To make the claim about the importance of Answer Inference Model more convincing, some case studies are needed (better if the code/algo to extract final answer is provided).", "paper_topic_and_main_contributions": "Topic\nCorporate chain-of-thought reasoning in small-scale tabular language models for real use case (with resource limitation and data security)\n\nMain contribution\n1. Propose a new paradigm of tabular mathematical reasoning\n2. Propose a framework TaCo which uses 2 small-scale TaLMs for CoT generation and answer inference respectively and also has mechanism to deal with incorrect calculation by using string parsing and external calculator.\n3. Achieve SoTA result in the evaluation benchmark, along with an extensive analysis of result.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PlamPVrfeE", "reviewer": "Reviewer_xSdW", "summary": "", "questions": " A. In the zero-shot pilot experiments, was TAPEX fine-tuned any further on zero-shot examples, or did you use the existing fine-tuned model?\n\n B. TaCo outperforms baselines moreso on questions with integer or decimal answers. How much of this performance gap is accounted by the external calculator? Is it feasible to apply the external calculator correction to the outputs of the LLMs? If you've already tried this, is there a notable difference in accuracy?\n\n C. What is the \"heuristic guess\" method listed in Table 3?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable effort and the precious time spent for carefully reviewing our paper. We greatly appreciate your positive feedback and we are very glad to learn that you found our work \"interesting\" and \"well-thought\". Your thoughtful and valuable suggestions will be incorporated in the future version of our paper. The questions and problems will be properly addressed as we reply.\n\n**Q1: Including more non-GPT open LMs as baselines.**\n\n**A1:** Thank you for this constructive advice. The non-GPT open LMs like LLaMA[1] and Alpaca[2] have played an important role in the recent open-source AI community. We will consider evaluating their CoT reasoning ability on the tabular data in the future work.\n\n**Q2: In the zero-shot pilot experiments, was TAPEX fine-tuned any further on zero-shot examples, or did you use the existing fine-tuned model?**\n\n**A2:** We follow the TABMWP paper and use the fine-tuned TAPEX as a strong backbone to conduct pilot experiments. **It was pre-trained and then fine-tuned on the WTQ[3] dataset, with no further fine-tuning on zero-shot examples.** Model checkpoints were directly downloaded from the offical huggingface model hub (\"tapex-base-finetuned-wtq\" and \"tapex-large-finetuned-wtq\"). More details will be included in the future version.\n\n**Q3:  How much of the performance gap on questions with integer or decimal answers is accounted by the external calculator?**\n\n**A3:** The performance of TaCo without the external calculator is listed below, where accuracy scores on test questions with integer (Acc-INT) or decimal (Acc-DEC) answers are reported respectively. **The experimental results indicate that the external calculator is an essential contributor for INT and DEC questions, which accouts for an average performance gap of about 35%.** Compared with INT questions, the performance gap on DEC questions is more significant as they usually demand more intensive and accurate numerical calculations.\n\n| **Model**          | **Acc-INT** | **Acc-DEC** | **Ave. Acc Gap** |\n| :----------------: | :---------: | :---------: | :---------------: |\n| TaCo  (base)       | 85\\.29      | 86\\.44      | -                 |\n| TaCo  (large)      | 92\\.54      | 88\\.41      | -                 |\n| **w/o calculator** |             |             |                   |\n| QT → S → A (base)  | 62\\.95      | 30\\.13      | 39\\.325           |\n| QT → S → A (large) | 73\\.70      | 44\\.72      | 31\\.265           |\n\n**Q4: Is it feasible to apply the external calculator correction to the outputs of the LLMs?**\n\n**A4:** Yes. We applied the same calculator to the output of ChatGPT and use regular expressions to extract the final answer from the output. **There is a slight performance increase from 82.60% to 83.07%.** After inspecting error cases of ChatGPT, we found that most errors resulted from wrong reasoning steps rather than calculation mistakes. Compared with small-scale TaLMs, the numerical calculating ability of ChatGPT is much more better. Thus the effect of the external calculator is not significant.\n\nHowever, we believe that the \"Tool Learning with LLMs\"[4] is a very promising direction. Various advanced tools like programming languages and search engines have been combined with LLMs and have shown great performance on different tasks including the tabular mathematical reasoning[5-6].\n\n**Q5: What is the \"heuristic guess\" method listed in Table 3?**\n\n**A5:** We are sorry that related descriptions were missing and they will be added in the final version. The \"heuristic guess\" is a baseline from the TABMWP paper. For _multi-choice_ questions, it randomly selects one from the given options with even probabilities. For _free-text_ questions, it randomly chooses one number from the question or the table as the prediction.\n\n**Q6: Typos, Grammar, Style And Presentation Improvements**\n\n**A6:** Once again, we would like to express our sincere gratitude for your careful review and precious suggestions, which remarkably improve the quality of our paper. We will incorpate all these changes in the final revision and read through the entire manuscript to refine the presentation.\n\n[1] LLaMA: Open and Efficient Foundation Language Models. 2023.\n\n[2] Alpaca: A Strong, Replicable Instruction-Following Model. 2023.\n\n[3] Compositional Semantic Parsing on Semi-Structured Tables. ACL 2015.\n\n[4] Tool Learning with Foundation Models. 2023.\n\n[5] Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. 2022.\n\n[6] Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " - The proposed model is interesting and performs well empirically, and is much more parameter-efficient than the baseline LMs.\n - There are a number of well-thought ablation experiments that help to to highlight which components of the model are most instrumental.", "reasons_to_reject": " - Reproducibility would be improved if the experiments included non-GPT open LMs. But since the LMs were only used as baselines, this is not a huge concern.\n - The presentation could be made cleaner.", "questions_to_authors": " A. In the zero-shot pilot experiments, was TAPEX fine-tuned any further on zero-shot examples, or did you use the existing fine-tuned model?\n\n B. TaCo outperforms baselines moreso on questions with integer or decimal answers. How much of this performance gap is accounted by the external calculator? Is it feasible to apply the external calculator correction to the outputs of the LLMs? If you've already tried this, is there a notable difference in accuracy?\n\n C. What is the \"heuristic guess\" method listed in Table 3?", "paper_topic_and_main_contributions": "The authors present a new two-stage model for tabular QA where each stage is implemented with a small-scale tabular language model. The first stage produces a CoT and the second stage outputs the answer. The demonstrate that their proposed method achieves state-of-the-art performance on the TABMWP dataset.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are a small number of minor errors. I list some of the error I found in the first few pages, but I encourage the authors to read through the entire manuscript more carefully and correct all such errors.\n\nLines 57-58: Redundant use of \"requirements.\"\n\nLine 65: \"despite\" -> \"despite the fact that\"\n\nLine 93: \"Experiment\" -> \"Experimental\"\n\nLine 102: \"We\" -> \"we\"\n\nLine 151: \"golden CoT\" -> \"Gold CoT\" This sentence can also be rephrased to be clearer.\n\nFigure 2c: The verb \"exploiting\" is usually used with a direct object.\n\nLine 195: \"flatten\" -> \"flattened\"\n\nLine 207: \"resulted\" -> \"resulting\"\n\nThere are also a number of examples of noun phrases with missing determiners (e.g. \"the\", \"a\", etc).\n\nAlso, it seems that the term \"solution\" is used later in the paper to refer to CoTs. This may be confusing for readers since colloquially, \"solution\" is often synonymous with \"answer.\""}}, {"review_id": "uFpZeRnMhB", "reviewer": "Reviewer_Z9oN", "summary": "", "questions": "1. Could you explain the value of the answer inference model? Can we discard this model? Is this setting the same as QT -> SA in the paper? \n2. What if the training dataset does not contain detailed solutions? A more general method is highly expected since most datasets do not have detailed solutions. \n3. Could you add more benchmark datasets to cross-validate your claims?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the careful review and insightful feedback. Your constructive suggestions are crucial for us to refine our work. We are also delighted to learn that you regarded our work as \"A good initiative to start the research\" and considered experiments \"thorough and showing good results\". We hope our response could alleviate your concerns and your valuable advice will be fully incorporated in the final version.\n\n**Q1: Incremental work. Look like the same method as previous cot work applied to the table domain.**\n\n**A1:** Thank you for this constructive comment. Admittedly, LLMs and CoT reasoning have experienced rapid advancements since the last year. Different CoT prompting methods have been explored to elicit intermediate reasoning steps from LLMs to solve complex questions[1-5]. Nevertheless, most of previous CoT work is based on LLMs and mainly focuses on unstructured text data. This raises two natural questions: **(i) Does this mean the end of traditional small-scale LMs? (ii) On the structured and prevalent table data, how good are the CoT reasoning abilities of existing LMs?**\n\n**The contribution of our work lies exactly on the intersection of above two research scopes which have not been thoroughly investigated.** To summarize, the contribution of our work are threefold. (1) We are the first to extend the CoT reasoning ability into TaLMs and advocate a general two-stage paradigm, which could be combined with future TaLMs. (2) The proposed method TaCo outperforms powerful LLM baselines with much less parameters and can surve as an alternative approach for scenarios where LLM-based methods are not suitable. (3) Thorough experiments were conducted to present a comprehensive evaluation and meaningful insights. As a result, we respectfully argue that the contribution of our work is not incremental.\n\nMoreover, we believe that the CoT reasoning on the tabular data is a promising direction and it deserves further exploring. For example, designing in-context learning strategy which considers table structures, enhancing the TaLM's CoT reasoning ability across different table types. We hope our work could attract more interests into this field.\n\n**Q2: The value of the answer inference model.**\n\n**A2:** Thank you for this insightful question. In the following analysis, we demonstate two important effect of Answer Inference model with detailed case studies.\n\n1) **Answer Inference model handles different CoTs with flexible answer locations and various answer types.** If we directly use regular expressions to extract the final answer from the generated CoT (we can denote this method as \"QT-->S\"), it is very difficult to cover different cases where multi-type answers may occur at arbitrary locations, which are demonstrated in the following CoTs with bold answer text.\n\n> … Estelle will get to Princeton at **10:30 P. M.** (ID: 1105)\n>\n> … the constant of proportionality is **2** pounds per plate. (ID: 1136)\n>\n> … **2** boxes had at least 44 broken crayons but fewer than 54 broken crayons. (ID: 1326)\n>\n> ... In 1995, **cantaloupes** cost more. (ID: 2394)\n>\n> ... An airplane calendar costs $**3.13** more than a map of Africa. (ID: 2643)\n\nThus, it is desirable to train an Answer Inference model which can automatically infer final answers based on different CoTs.\n\n2) **Answer Inference model brings a second chance to recontemplate improved reasoning steps before making the final judgement.** For the \"QT-->SA\" method, we construct a concluding sentence \"As a result, the final answer is {answer_text}\" and append it to the end of CoT, which is used to train the TaLM to generate reasoning steps and the final answer in one forward pass.  Then, the regular expression is adopted to extract the final answer from the model output:\n\n```python\nimport re\nret = re.match('.*the final answer is (.*)', model_output)\nfinal_answer = ret.group(1)\n```\n\nThe \"QT-->SA\" indeed reduces the difficulty to extract the final answer. However, **such one-stage paradigms cannot fully utilize the corrected CoT to change the original (wrong) answer.** In the following example (ID: 22198), though errors in the CoT are corrected by the external calculator, the original wrong answer is not changed. By contrast,  **the two-stage paradigm \"QT-->S-->A\" can take full advantage of the corrected CoT to infer the right answer instead of jumping to the conclusion directly.**\n\n>**Question:** Shelley has $272.00. Does she have enough to buy a lawn mower and an iron bench? (A) yes (B) no\n>\n>**Original Output of QT->SA:**\n>\n> Add the price of a lawn mower and the price of an iron bench:\n>\n> 145.00 + 140.00 = \\$265.00\n>\n> As a result, the final answer is yes.\n>\n>**Corrected Output of QT->SA:**\n>\n> Add the price of a lawn mower and the price of an iron bench:\n>\n> 145.00 + 140.00 = \\$285.00\n>\n> As a result, the final answer is yes.\n>\n>**Final Answer of QT->SA:** yes\n>\n>**Correct Answer:** no\n\nIn addition, **it is challenging for one small-scale TaLM to conduct both CoT reasoning and answer inferring**. For example (ID: 1937), the \"QT->SA\" chooses the wrong number as the final answer more often than \"QT->S->A\".\n\n> **Output of QT->SA:** ... 5 amusement parks have at least 10 roller coasters but fewer than 40 roller coasters.\nAs a result, the final answer is 10.\n>\n>**Final Answer of QT->SA:** 10\n>\n>**Correct Answer:** 5\n\n**The similar two-stage paradigm has also been adopted in recent work** like Verify-and-Edit[1] and Self-ask[2], where they first utilize one LLM to generate the CoT and then ask the same LLM to infer the final answer based on the improved CoT. By contrast, we delegate CoT generation and answer inference to two small TaLMs, respectively. In the next version of the paper, more case studies will be included to highlight the importance of Answer Inference model.\n\n\n**Q3: A more general method is highly expected since most datasets do not have detailed solutions.**\n\n**A3:** We fully understand that a more general method which needs the fewest human-annotated CoTs is highly expected. **As mentioned in the Limitation Section, for datasets lacking annotated solutions, we can utilize LLMs to generate CoTs[8-9] and then use them to further enhance the generalization of TaCo**. This can be viewed as a kind of transfer learning from LLMs to TaLMs and it can greatly alleviate the dependency on manual annotations.\n\n**Q4: Adding more benchmark datasets to evaluate the proposed method.**\n\n**A4:** Thank you for this great advice. We select the TABMWP dataset to conduct experiments as it is the latest large-scale table dataset with high-quality gold solutions. More experiments on other datasets are promised. We will utilize the transfer learning method in A3 to train the TaCo on more datasets and evaluate its performance.\n\n\n[1] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NIPS 2022.\n\n[2] Large Language Models are Zero-Shot Reasoners. NIPS 2022.\n\n[3] Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.\n\n[4] Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. ICLR 2023.\n\n[5] Complexity-Based Prompting for Multi-Step Reasoning. ICLR 2023.\n\n[6] Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework. ACL 2023.\n\n[7] Measuring and Narrowing the Compositionality Gap in Language Models. 2022.\n\n[8] Large Language Models Are Reasoning Teachers. ACL 2023.\n\n[9] Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A good initiative to start the research: a. pre-trained small tabular LMs perform worse in cot reasoning ability. b. tabular LMs benefit from cot when inferencing the final answer. \n2. thorough experiments showing good results. \n", "reasons_to_reject": "1. incremental work. Look like the same method as previous cot work applied to the table domain. \n2. expect a more general method to adapt to those datasets that don't have detailed solutions (cot). ", "questions_to_authors": "1. Could you explain the value of the answer inference model? Can we discard this model? Is this setting the same as QT -> SA in the paper? \n2. What if the training dataset does not contain detailed solutions? A more general method is highly expected since most datasets do not have detailed solutions. \n3. Could you add more benchmark datasets to cross-validate your claims?", "paper_topic_and_main_contributions": "This paper proposes TaCo, which utilizes two tabular LMs, one for CoT generation, and another one for answer inference.  This can be regarded as a combination of the method of CoT with traditional tabular-related LMs. \n\nMain contributions include:\n1. the first to explore cot reasoning in tabular-related tasks. \n2. achieve significant performance improvement. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a two stage method for tabular reasoning and calculations. The two-stage involves using smaller language models to reduce the cost of running larger language models for this task. The reviewers highlighted the comprehensive list of experiments and ablations and novelty of exploring CoT for the task. Reviewers have suggested some revision to improve the presentation of the work which should be considered by the authors and community would benefit from more clear presentation. Also since one of the main contribution of the work is the computational cost it would be good to discuss this along the dimensions of accuracy. It is good to see the limitation section in the manuscript, where authors discussed the limitations including reliance on CoT training data which might not be available in more general settings.
Decision: Accept-Findings