Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data
Authors: Mubashara Akhtar
Abhilash Shankarampeta
Vivek Gupta
Arpit Patil
Oana Cocarascu
Elena Simperl
Venue: EMNLP 2023 Findings (2023)
Keywords: numerical reasoning
probing language models
numeracy
tables
tabular data
TL;DR: This paper studies the numerical reasoning skills of language models across ten reasoning types which we organise in a taxonomy and evaluate models on the table NLI task.

Abstract: Numerical data plays a crucial role in various real-world domains like finance, economics, and science. Thus, understanding and reasoning with numbers are essential in these fields. Recent benchmarks have assessed the numerical reasoning abilities of language models, revealing their limitations in limited and specific numerical aspects. In this paper, we propose a complete hierarchical taxonomy for numerical reasoning skills, encompassing over ten reasoning types across four levels: representation, number sense, manipulation, and complex reasoning. We conduct a comprehensive evaluation of state-of-the-art models on all reasoning types. To identify challenging reasoning types for different model types, we develop a diverse and extensive set of numerical probes and measure performance shifts. By employing a semi-automated approach, we focus on the tabular Natural Language Inference (TNLI) task as a case study. While no single model excels in all reasoning types, FlanT5 (few-/zero-shot) and GPT3.5 (few-shot) demonstrate strong overall numerical reasoning skills compared to other models in our probes.

Exploring the Numerical Reasoning Capabilities of Language Models:
A Comprehensive Analysis on Tabular Data
Mubashara Akhtar", Abhilash Shankarampeta", Vivek Gupta', Arpit Patil* Oana Cocaraseu' and Elena Simperl'
King's College London Meesho University of Pennsylvania *University of Utah
mubashara. akhtarkkcl.ccccmm ac. uk abhilash. haankarampetateeeeehtte com
Abstract
Hulk
Numbers are crucial for various real-world domains such as finance, economics, and science. Thus, understanding and reasoning with numbers are essential skills for language models to solve different tasks. While different numerical benchmarks have been introduced recent years, they are limited to specific numerical aspects mostly. In this paper, we propose a hierarchical taxonomy for numerical reasoning skills with more than ten reasoning types across four levels: representation, number sense, manipulation, and complex reasoning. We conduct a comprehensive evaluation of ofateoofthhe-art models to identify reasoning challenges specific to them. Hence- forth, we develop diverse set of numerical probes employing semi-automated approach. We focus on the tabular Natural Language Inference (TNLI) task a case study and measure models' performance shifts. Our results show that no model consistently excels across all numerical reasoning types. Among the probed models, FlanT55 (few-/zero-shot) GPT-3.5 (few-shot) demonstrate strong overall numerical reasoning skills compared to other models. Label-flipping probes indicate that models often exploit dataset artifacts to predict the correct labels.!
Directed by
Ang Lee
Release date
June 20, 2003
Running time
138 minutes
Budget
$137 million
Box office
$245.4 4 million
HI: Hulk was released on 20th June, 2003. (E)
Date: Hulk was released on 20-06-2003, (E)
Date Flip: Hulk was released on 12-08-2009, (C)
H2: The movie has a length of 138 minutes. (E)
Appr: The movie has length about 150 minutes. (C)
H3: The movic can he watched about two hours. (E)
Num: The movie can be watched in about 2 hours. (E)
Num Flip: The movie can be watched in hours. (C)
Arith: Hulk brought in $108. 4 million profit. (E)
Arith Flip: Hulk brought in $120.9 million profit. (C)
Table I:
Base hypotheses (HI, H2, H3) and (flipped)
prohes for heterogencous numbers (i.c. date), approximation, numeracy, and arithmetic, Labelled as Entail or Contradict.
Moreover, evaluating models on numerical benchmarks, it often remains unclear why models struggle with the tasks. For example, the issues can arise from models struggling to recognize numeri- cal representations in text, failing to compute arith- metic operations, predicting incorrect outputs due a lack of numerical commonsense knowl- edge. We aim to explore these questions in greater detail in this study. Limitations of language models' numerical abilities, as discussed in prior research, include tokenization and representation of numbers in text (Thawani et al., 2021b), hallucination (Ji et al., 2023; Chen et al., 2023; Ye et al., 2023), and generalizabilltyy/ouuttess issues (Razeghi et al., 2022; Geva et al., 2020; Xu et al., 2022).
1 Introduction
Numerical data is ubiquitous in the real-world. Many applications in domains such finance, eco- nomics and science require understanding and rea- soning with numbers. In recent years, benchmarks were introduced to study language models' numer- acy skills (Zhang et al., 2020; Wallace et al., 2019; Dua et al., 2019). However, these datasets mostly concentrate on few, specific numerical reasoning types (e.g. scales (Zhang et al., 2020)).
Successful numerical reasoning requires a com- bination of skillsets: understanding representa- tion of numbers (Thawani al., 2021a,b) and their meaning in given context (Loukas al., 2022). applying operations (Geva al., 2020; Pa- tel et al., 2021), and integrating factual and com- monsense numerical knowledge to solve numerical
"Equal contributions
'Data and code are available at https: /ttp:s///ttttt.... com/ mubasharaak/ wataassharraanmmmmrrraa

---

problems (Lin al., 2020; Park et al., 2022). For example, classifying the hypotheses "The movie can be watched in about 2 (or 'two') hours." from Table requires understanding that both "2" and "two" " depict the same numerical value, converting "2 hours" " to another unit (i.c. 120 minutes), and applying approximation to map "120 minutes" to "138 minutes" in the table.
Representation
Hetarogeneoca
Nugative
Compartsion
Numericcl Reasoning
Approxdmation
In this paper, we evaluate state-of saae--of-hher toefthh-artt models on various numerical reasoning types. assess which reasoning types are challenging for specific models, we create a diverse and large set numer- ical probes and measure shifts in models' perfor- mance. We organize all probed reasoning types in a hierarchical laxonomy. Inspired by how humans understand and reason with numbers, well as previous numerical benchmarks, we include eleven reasoning types across four level: representation, number sense, manipulation, and complex reason- ing (Figure 1). We apply a semi-automated ap- proaches for probe creation. We select tabular NLI (TNLI) case study task, given three criteria: (i) numerical data (numbers, percentages, dates, etc.) is prevalent in tables; (iz) tables are common in real-world data sources such in scientific publi- cations, database systems and financial documents; (iit) tables as structured data facilitate automated perturbations to create large-scale probing sets. See Table for some examples of prohes created from hypotheses (H1, H2, H3) and the given table.
Range
Sorting
Manipulation
Word Problems
Complex
Reasoning
Common Sersa
Figure Overview of numerical reasoning types.
create large-scale, numerical probe sets using table II. datasets.
We evaluate three different categories of lan- guage models (LMs) on our numerical probe sets: (i) numerical LMs; (it) LMs for tabular data; and (tit) zero-/few-shot LMs.
2 A Taxonomy for Numerical Reasoning
This section introduces a hierarchical taxonomy for numerical reasoning, inspired by previous works on numeracy in NLP (Thawani et al., 2021b; lb; Xu et al., 2022) and psychology (Barrouillet and Fayol, 1998a; Whyte and Bull, 2008; Bofferding, 2019).
We group numerical reasoning skills given their complexity level in four categories: R1 R4.
Our experiments conclude that large language models (LLMs) like FlanT55 and GPT3.5 perform better than other models on various numerical rea- soning tasks. Both table-based and numerical mod- els struggled to understand data with flipped la- bels and negative values. Moreover, we observe that some models' performance improves signif- icantly for counterfactual probes (e.g. NT5 and TAPAS) and label-flipping probes (e.g. FlanT5 zero-shot), which indicates that models might ploit dataset artifacts and are biased towards one label. These findings emphasize the importance further systematically investigating numerical reasoning capabilities across various NLP models. Our contributions are as follows:
2.1 Number Representation (R1)
This category includes skills for understanding the form numerical data. Similar to the notion of form in language (Bender and Koller, 2020), this is the realization of numbers in text; the way they are represented and expressed.
Numeration. Numeration studies language model's understanding of representation systems common for numbers in English: the Arabic ("2") and English ("two") numeration systems. Specifically, we probe LMs can link between distinct symbols used for the same number. For example in Figure 1, H3 contains "two" as a word, which can be also represented through "2"
We introduce a taxonomy for numeri- cal reasoning skills, including representa- tion/number sense/manipulation skills and complex reasoning with numbers.
Heterogeneous Number Types. Formatted num- bers (e.g. dates, times, and fractions) are frequently used to convey additional information associated with a numerical value. Numbers are formatted in a specific way given their context and purpose,
We propose a aeto-auuemated approach to

---

such as expressing times and dates using full-stop (*"",, using the "%" symbol to indicate fractions, and different currency symbols for money (i.e. "S"
table. As a reasoning skill, it allows make quick estimations and metric unit conversations, and un- derstand the approximate values of numbers with- out calculating them explicitly.
or "E"), e.g. H1 and "Arith" in Figure 1.
Negative Numbers. Early in their develop- ment, children develop some mental model for neg- ative numbers (see experiments with first-graders in Bofferding (2019)). Using negative numbers re- quires understanding the notation negatives (i.e. followed by a number). This also includes dis- tinguishing between minus in subtractions (1 3).
2.3 Manipulation (R3)
Manipulation reasoning types are used to apply basic operations on numbers such as addition. Suc- cessful manipulation of numbers requires under- standing their representations and meaning in the given context (i.e. number sense).
dates (12-12-2022), counts (i.e. from one to three) and in negative number (-2).
Sorting. The sentence "Out of all Ang Lee's 's di- rected movies, 'Hulk' was the one with the sec- ond highest box office income." requires sorting all movies according to their box office income in order to select the one with the second highest in- come. Sorting objects according to some criteria is basic milestone for developing cognitive skills, By age two, children already begin to understand the concept of sorting.
2.2 Number Sense (R2)
Number sense includes reasoning skills for con- ceptualizing number quantities and understanding their meaning in a given context.
Scale. In everyday communication, numbers commonly occur with measurement scales, e.g. weights, distances, heights. Understanding num- bers in context of scales is a basis for various ap- plications, c.g. question answering (e.g. "We are driving 80 km/ h, is this within the speed limit?"), commonsense (e.g. "Cats weight between four and five kilograms.") (Lin et al., 2020), and temporal reasoning (e.g. "She left the office thirty minutes ago.") (Zhou et al., 2020; Zhang et al., 2020).
Simple arithmetic.
Arithmetic reasoning is the ability of manipulating numbers with basic oper- ations (addition, subtraction, multiplication, divi- sion). While adults commonly retrieve results of simple calculations from memory, children use dif- ferent operations (Barrouillct and Fayol, 1998b).
2.4 Complex Reasoning (R4)
This category builds on all previous reasoning cat- egories (R1 R3) to solve numerical word prob- lems (NWP). NWP are expressed through natural language and require multistep reasoning. Extract- ing information from the problem description and applying numerical/maathmmaiicl reasoning using the retrieved information and world/commonsense knowledge is required (Upadhyay and Chang. 2017; Amini al., 2019; Huang et al., 2016).
Comparison. Comparing numbers allows under- standing numerical relationships. It involves iden- tifyngg which numbers are greater than, less than, or equal to others. For example, given the table in Figure 1, understanding "The ramning time of Hulk is longer than 120 minutes." requires comparison.
Range. The question "Was the budget of the movie between $130 and $215.17" about the table in Figure 1 requires understanding number ranges. Already at age between two and three years, children develop numerical abilities to understand sequences of numbers and start reciting numbers in an appropriate order (Fuson, 2012; Laski and Siegler, 2007). Models' that understand the nota- tion of ranges, can correctly answer the question by knowing that 137 is in the range 130 245.4.
Numerical Probing Framework
This section provides overview of the probing framework. We use tabular Natural Language In- ference (TNLI) for automated probe creation.
3.1 Preliminaries
Tables for numerical probing. Tables align well with our objectives given three key criteria: (i) numerical data is common in tables; (ii) tables are frequent in real-world data sources; (iil) ta- bles, due to their structured formats, facilitate au- tomated perturbations for probe creation. Tables'
Approximation. Humans commonly approxi- mate number values in everyday life (Odic and Starr, 2018: Bonny and Lourenco, 2013). H3 in Figure requires approximation among other skills to map "about two hours" to "138 minutes" in the

---

semi-structured format, the alignments available between table cells and column/row headers, and the frequency of numbers, make them well suitable for creating numerical probes automatically.
Dataset
Hypotheses
Tables
Num
Probes
TabFact
118,275
16,573
59.00%
214,440
InfoTubs
23.738
2,540
53.6%
19,779
TXTQA-NII
4,947
2,156
59.7%
15,139
ToTTo
1,000
892
45.7%
1,000
TabMWP
283
283
38.3%
238
Table NLI. Given a natural language sentence as hypothesis and a tabular premise, the aim of TNLI is to classify if the hypothesis entails or contradicts the table (Gupta et al., 2020). We use the table NLI datasets TabFact (Chen et al., 2020) and In- Tabs (Gupta et al., 2020), as well as recast the table QA datasets TAT-QA (Zhu et al., 2021) and TabMWP (Lu et al., 2023) to NLI (i.e. TATQA- NLJ, TabMWP-NLI). TAT-QA includes metadata, i.c. annotations of cells and operations per correct answer. This information is not available for any TNLI dataset and is crucial to create probes for specific reasoning types, e.g. arithmetic reasoning. Table 2 provides an overview of the TNLI datasets.
Table 2: TNLI probing datasets; num cells refers to the average ratio of numerical cells in tables.
substitute measurement units for a particular scale (e.g- "kilograms") by another unit (e.g. "meters") or introduce errors conversion of units (e.g. 3 kilometers replaced by 3, 000 meters).
3. Table Probes through Counterfactual Table Editing We also probe with counterfactual tables to evaluate if models rely spurious patterns in the premise table for label prediction. We filter the counterfactual datasets by Jena al. (2022) consist- ing of /hypothesis; original table; counterfactual table) for numerical hypotheses.
Preprocessing. For each of numerical reasoning type, we first identify base TNLI hypotheses and/or tables in the datasets that can be used for auto- mated probe creation. Hereby, we defined a list of reference tokens specific for each reasoning type and to filter relevant dataset samples. For exam- ple, used units of measurements such as "hour", "meter", "kilogram" filter hypotheses for scale probes (see $4 for more details). To recast the TAT-QA dataset, we follow the simple yet effective, rule-based approach proposed by Demszky et al. (2018) for QA to NLJ conversion.
4 Probing with TNLI Datasets
This section discussed probes in detail and how we created them for each reasoning type from $3.2
Numeration. To study models' understanding of string ("two") and numerical (e.g. "2") number representations, we create two types of numeration probes. One converting number representations from strings numeric, while the second category applies the conversion vice versa. We filter hy- potheses with numbers written strings ("two") and substitute them by their numeric counterpart (e.g. "2"). The label-presers tically equivalent to the base hypotheses and the la- bel (c.g. entailment) is not changed. Label-flippigg probes replace the converted number by random number in the range of [x x 0.5; + # 0.5). For example, the numeration flipping probe of H1 (Ta- ble 3) replaces 112 by one hundred and forty-four and flips the label from entailment to contradiction.
probes are seman-
3.2 Probes through Structural Perturbation
Overall, we our framework includes three types of probes, created through hypotheses perturbation and counterfactual tables.
1. Hypothesis label- labee-pres probes We cre- ate label- preservvin probes changing the base hy- pothesis such that its meaning is not changed and the initial label is preserved. They are used to evalu- ate model's ability to reason and predict the correct label given mmmantically-equivaai
Heterogeneous number types. We created het- erogeneous probes for the following categories fre- quent in the TNLI datasets: date formats, ordinals, percentage, currencies, and scientific notation. To filter base hypotheses, we applied simple, rule- based approach specific each category (i.e. dates formats, percentage, ordinals, etc.). To create label- preserving probes we applied representation changes which did not change the semantic mean- ing, For H3, we substituted 3rd June. 1986 by an- other English date format 03-06-1986. To flip the
changes.
2. Hypothesis lahel-ffippigg probes To generate label- -hel-ffipping probes, we modify the base hypothe- sis such that its meaning alters and the label of the probe flips, c.g. from entailment to contradiction. We aim to overcome potential dataset artefacts that might be exploited for label prediction instead of performing numerical reasoning.
These changes are specific to the reasoning types. For example, to flip labels for scale probes, we
FFind details on probe statistics in Appendix B.

---

Rafael Nadal
Plays
Left-handed
Born
June 1986 (age 37)
Height
1.85 m
Turned pro
2001
Prize money
US$116.111,566 (3rd all-time leader in earnings)
Base Hypothesis H; Numeration Probe Hi Nun Flip Probe H; Range Probe Hi
Born in
1986 Nadal is age 37 currently.
Born in nineteen eighty six, Nadal is age thirty seven currently. Born in nineteen ninety two, Nadal is age forty one currently. Born in 1986, Nadal is age between 31-43 currently.
Base Hypothesis H2
The player's birth date is on 3rd June, 1986 The player's 's birth date is on 03-06-1988. The player's birth date is on 15-01-1999.
Heterog Probe Hr Heterog Flip Probe H2
Base Hypothesis Hj
With $116, 111,661 prize money, he is the 3rd highest earning all-time player.
Heterog Probe Hs Approx Probe H2
With S116. 111561e prize money, be is the third highest earning all-time player. With about 5116, 000, 000 prize money, he is the 3rd highest carning all-time player.
Base Hypothesis Ha
Rafacl Nadal has a height of 1885 meters. Rafacl Nadal has a height of 185 centimeters. Rafael Nadal has a height of 5,2 ft.
Scale Probe H4 Scale Flip Probe H4
Base Hypothesis Hy
After the year 2000, the player Nadal turned pro.
Comparison Probe IIs Comparison Flip Probe Hs
After the year 1995, the player Nadal turned pro.
Before the 1990, the player Nadal turned pro.
Table 3: Exemplary hypotheses and non-/flipping probes evaluated reasoning types
label, we replaced the date in the adjusted format by a random date, i.c. 15-01-1999. We replaced per- centage signs by the token "percentages" and vice versa. Similarly, ordinals written words (first) were exchanged by numerical representations st) and the other way around. For hypotheses with large numbers (e.g. "S116,111,566 in H3), 6).
Mapping probes replace the number and measure- ment unit by an equivalent (c.g. 1.85m by 1.85 meters) for label-preserving probes and a random measurement unit e.g. 1.85m to 1.85 kilograms) to flip the base hypotheses.
Comparison. We first created list of sig- nal word-pairs by prompting GPT3.5. The list includes pairs such as ""bigger"="smaalle""" { ""aaller"""shorter"" and ("faster":"slower") Us- ing these pairs and their synonyms, we filtered base hypotheses and created three types of comparison probes. First, changing the signal word with its op- posite counterpart to flip labels (see H5 in Table 3 flipping "after" "before"). Second, altering the number such that the comparison and label do not change: replacing "after 2000" by "after 1995" (H5). Finally, we combine both prior approaches to create label-flipping probes, e.g. "Before the year 1990, the player Nadal turned pro."s
introduced scientific notations
Negative numbers. To create negative probes, we replaced negative numbers -n (c.g. -3) by string equivalents (e.g. minus 3; negative 3) and evaluated changes in model performances on these semantically same sentence pairs. For label- flipping probes, converted negative numbers into the positive counterpart n. For example, con- verting "The company's monthly closing resulted -5 million USD." to "The company's 's monthly closing resulted in 5 million USD." " flips the label.
Scale. We created two types of scale probes: (i) conversion; (ii) mapping. Conversion convert numbers within a measurement scale. For H4 in Table 3, converted the number and measure- ment unit 1185 meters) ) to the next smaller unit within the same scale (185 centimeters) for the label-preserving probe. For label-flip, we intro- duced an error in the converted number, i.c. con- verting 1.85 meters. to 5.2 ft instead of 6,07 ft.
Approximation. We first extract a number n from our base hypothesis and given the value of n, we decide the magnitude of rounding to ap- ply. While smaller numbers are rounded to tens, larger number are rounded to hundreds, thousands or larger decimal points. For example, we cre- ated the probe "With about $116, 000, 000 prize money, he is the 3rd highest earning all-time player"

---

by rounding n equal $116.111,561 $116, 000, 000" (H3 in Table 3).
"about
Experiments and Analysis
Next, we provide an overview of all models that were evaluated on the probes from $4. We also discuss the obtained results and insights.
Range. To create range probes, we substitute number in the base hypothesis by an appropri- ate range, e.g. 37 by "between 31-43" (H1). We define the radius of the range and its boundaries automatically given the value of n. For example, given 10, we randomly sample radius be- tween 1 5. For
5.1 Probed Models
We use state-of-the-art models which are divers in terms of architecture, size, and training setup, grouped into three categories:
and sampled radius of 2, the range will be [5 - 9]. We select decimal boundarics n is a decimal number.
(C1) Numerical LMs. This category includes LMs adapted for numerical reasoning. LUNA (Han et al., 2022) is recent transformer-based model with an adapted tokenization approach for numbers. The model encodes numbers as single tokens (e.g. 3, 201) instead of splitting them down into sub- words or binned tokens. NT5 (Yang et al., 2021) is variation of the T5 model. It has been mod- ified for numerical reasoning through additional pretraining objectives and fine-tuning using numer- ical datasets. PASTA (Gu al., 2022) is based on DeBERTa and is pretrained with objectives that use table-based numeric operations.
Sorting. We utilized table columns number sequences create sorting probes. We generated list of position indicators in number sequences (e.g. "top". "second" "3rd", "biggest", "lowest"), These words were used filter base hypotheses. To cre- ate label-flipping probes, we changed the position of the sequence to another one. For instance, we modified "in the first quarter of 2018" to "in the third quarter of 2018" by selecting the value from the third row instead of the first.
Simple arithmetic. Using TATQA-NLI its metadata indicating the involved numbers and op- erations for numerical reasoning, we created arith- metic probes. We extracted probes involving addi- tion, subtraction, multiplication, and division. Ad- ditionally, we generated label- fiippifii probes by replacing the operation output (e.g. result of sub- traction) in the hypothesis with different number. In Table 1, the "Arith" probe involves calculating the difference between the budget and box office values to determine the correctness of 108.4. The flipped arithmetic probe produces close but incor- rect subtraction output, 120.9.
(C2) LMs for tabular reasoning. TAPAS (Herzig et al., 2020) extends the BERT encoder with table-specific emheddings. We used TAPAS model trained with intermediate pretraining on synthetic and counterfactual data (Eisenschlos et al., 2020). We also probe TAPEX (Liu et al.. 2022), which uses BART (Lewis et al., 2020) as its base model and pretrains the model to mimic neural SQL executor over tables. Similarly, ReasTAP (Zhao al., 2022) is BART-based model pretrained on synthetically generated data requiring seven table reasoning skills, including a numerical task, temporal reasoning, and conjunc- tion. Previous works have also shown the success of the *BERT models on tabular NLI tasks (Herzig et al., 2020; Yin et al., 2020; Shankarampeta et al., 2022; Akhtar al., 2022). Tables either linearized processed into sentences structured formats. The transformed tables are then used as input to the models. We used a DeBERTa model (He al., 2021) trained on multiple NLI datasets for this setting.
Numerical word problems. We converted TabMWP questions and answers into declarative hypotheses. TabMWP is dataset of free-text math word problems that involve reasoning with tabu- lar data. For fabpping probes, we substituted numbers in the hypotheses with random numbers from the same column.
Counterfactual Table NLI Probes. We filtered the counterfactual ToTTo (Parikh et al., 2020) dataset by Jena al. (2022) for numerical hypothe- sis. To create counterfactual tables, they swap two more table cells modify the tables such that the label the respective hypothesis changes from entailment to contradiction and vice versa.
(C3) Large LMs. For few-/zero-shot evaluation. we selected FlanT5 (Shen al., 2023), GPT3.5, and PaLM 2 (Chowdhery et al., 2022) and probed them in both a fhw-shot and zero-shot setting.

---

Model
Table Specific
Numerical Specific
Large LMs
TAPAS
DeBERTa
TAPEX
NTS
LUNA
PASTA
ReasTAP
F5anT5
GPT3.5
PaLM
Reasoning
few
Few
few
Representation
Numeration
-0.32
-1.82
-7.84
4.18
-5.22
-7.7
-7.18
1.28
-8.84
-0.47
5.65
-1.35
-3.07
Heterogeneous
4.03
-2.36
-5.94
-3
-10.09
-7.76
-3.18
0.34
-5.49
6.8
6.65
0.44
-2.22
Negative
-46.11
-13.77
0.56
-94.48
-75.55
-10.68
2.65
19:21
42.3
8.24
2.3
-2.17
1.14
Label Flipped
Numeration
38.87
4.09
-43.3
48.53
-71.35
-2.855
-37.21
-78.37
33.38
-37.78
44.71
-37.29
-46.45
Heterogeneous
-9.57
-32.25
-1.97
.43.48
-23.59
-16.21
-53.44
86.6
-27.97
27.79
-20.65
-25.31
Negative
-4.81
-1.56
-97.01
-17.87
76.85
-70.58
-96.46
-83.92
173.14
2.2
-80.43
-78.41
Number Sense
Scale
0.03
-6.25
1.21
-11.43
-4.6
-9.45
-0.52
-17.58
Compurison
-1.88
18.118
-12.58
19.19
-30
-35.11
-0002
29.38
140.82
-9.39
9.13
16.91
-20.08
Approximation
-5.61
-6.65
-18.9
-9.55
-7.67
-27.44
-7.81
-9.66
-12.94
0.03
12.08
-10.44
-12.03
Range
-18.89
-33.77
-1.96
-20.43
86.77
-84.66
4.97
22.44
178.13
0.5
-1.07
14.37
4.41
Label Flipped
Scale
-23.73
64.58
-30.41
-39.08
68.44
-1.66
16.54
.69.56
93.77
-39.08
39.98
-17.85
-27.4
Comparison
57.67
-19366
-4,83
-29.62
-0.28
-19.1
-15.65
-8.47
-40.75
-20.81
14.67
-17.96
-16.09
Manlpulation
Sorting Arithmetic
-34.8
28.66
-91
-22.6
54.31
..99
-83.96
-86.67
25
-57.39
-5.62
-32.45
-39.59
-58.62
-95.33
7.07
49.00
-88.87
-71.53
265.07
-60.34
-67.67
Complex Reasoning
Complex
63.37
6.93
-80.18
-3,22
41.41
-50.84 84
116.17
-89.77
-60.22
-4.35
-73.4
-75.9
Counterfactual
44.5
55.54
-12.29
159.3
0.98
-6
-10.12
61.5
12.23
40.63
5.26
30.57
48.56
Table 4: Probing results given as accuracy difference (in %) between base hypotheses and probes.
5.2 Training and Evaluation
trained on only 78G of data (He et al., 2021).
Interestingly, both NT5 and FlanT5 use T5 as their base model. F5anT5 was instruction-fine-tue and outperforms NTS many probing categories.
To fine-tune models, we used the base hypotheses of the training datasets (e.g. InfoTabs) and eval- uated models only probes created with their testsets. The few-shot models were prompted with 2-shot extrapolation. We evaluated all models in 3-step process: (1) evaluation of base hypotheses H; (2) evaluation of probes P, created using H; (3) calculating changes in model performance by comparing accuracy of P to H. As our TNLI task is a binary classification task, we used accuracy for evaluation.
(Q2) What factors can contribute to high per- formance variations across certain reasoning types? Large performance variations mainly oc- cur due to inconsistent numerical reasoning of mod- els across tasks. For example, we observe that some models struggle with more basic reasoning (e.g.. FlanT5 zero on numeration) while perform- ing very well on more complex types. This be- havior might have different reasons. One potential reason memorization. Previous works (Petroni et al., 2019; Carlini et al., 2021; Ishihara, 2023) show that large pretrained language models store knowledge in their parameters, which they tend to retrieve instead of reasoning over the provided input (Gupta et al., 2022a). Hence, models can memorize common arithmetic operations they en- counter during training and perform well certain downstream tasks. For example, flipping numbers as words ("two") numerals ("2") might allow models to retrieve knowledge which they didn't consider for the initial hypothesis. Another reason for high-performance drops can be the hallucina- tion of models. While models initially perform well on hypotheses, adjusting the numbers can hinder models from relying on spurious patterns.
5.3 Results and Discussion
Table 4 gives on overview of all probing results. If available, we separately list scores for flipped probes, e.g. numeration and numeration flipped.
(Q1) Does any model exeel in all numerical reasoning types? While there is not best- performing model across all reasoning types and different models struggle with different types, FlanT5 and GPT3.5 show overall good perfor- mance in a zero-shot setting. While GPT3.5 (few- shot) performance drops by -60.22% for complex reasoning probes, the model's average accuracy change is around - 16.7% for other types. This can be related to (1) models pretraining data, and (2) training on chain-of-thoughh reasoning tasks (Wei al., 2022). GPT3.5 was trained on more than 300 TB data Common Crawl, allowing the model to memorize much more numerical data than other probed models. In comparison, DeBERTa was
(Q3) How do models perform on different types of numerical reasoning? Representation.

---

Table 4, comparing numeration probes, we find for all models performance drop of between [0; -0] percentages cept FāanTă (few). This drop strongly increases for almost all models evaluated numeration flipped probes. For example, F5anT5 (few) shows a per- formance drop of -8.33%. FlanT55 (few) also performs well on heterogeneous probes, followed by DeBERTa 2.4%) and NT5 (- 3%). Whereas LUNA performance drops significantly for hetero- geneous probes (flipped and non- non- non-flipped). TAPAS, NT5, and LUNA show significant performance drops (between -3%.87% and - -71.35%) on neg- ative number probes. This could be because the models exploit correlations between the "_" sign and labels for predicting base hypotheses. Interest- ingly, few- and zero-shot models like FlanT5 and GPT3.5 show improvements on negative number probes. This may be because the models under- stand probed versions of negative numbers (e.g. "minus 22") as a negative number but not the initial representation (e.g. "222")
(Q4) Do models perform similarly for flipped and non-slipped probes? We observe higher per- formance drops for label-flipingg probes compared to con-flipping probes across models. Models that struggle with flipping probes but perform well on their non- ann-ippping counterparts indicate reliance on spurious patterns for label prediction. The per- formance of TAPAS, TAPEX, PASTA, ReasTAP, FlanT5 T5 (few), and PaLM drops significantly for the representation reasoning category comparing non-flipping and flipping probes. For example, TAPAS performance drops by 2.28% on numer- ation probes, but show a drop of -45.98% on nu- meration flipping probes. Similarly, DeBERTa per- forms well on scale probes (-6.28 6.25%) compared to the flipping version (-61.58%) PaLM perfor- mance numeration, heterrogeneous, and negative probes drops by approximately -35%, 20%, and -00 on flipping counterparts. DeBERTa exhibits robust performance on number flipping probes for sorting and FlanT5 on negative numbers, as well as arithmetic probes.
numeration probes, ex-
(Q5) Are numerical and table-specific models better for numerical reasoning than LLMs? Numerical models. Our experiments do in- dicate any superiority of numerical models over others. LUNA, a transformer model that uses a spe- cific tokenization method for numbers, performs similarly to other models on many reasoning types. The only reasoning type where LUNA outperforms is comparison flipping probes, with small im- provement of 0.28%. PASTA is DeBERTa-basee model trained on numerical data and pretraining objectives. However, compared to DeBERTa, it only performs better on negative number and scale probes.
Number sense. Comparing models based on number sense probes, we observe different patterns for fine-tuned models and few-/zero-shi models. Fine-tuned models struggle especially with com- parison probes, with a -66.7% average perfor- mance drop. Scale probes show a 12.1% decrease on flipping probes, while approximation (flipping) probes report 12.0% decrease in model per- formance. In contrast, FlanT5 perform better on comparison and range probes, sometimes surpass- ing predictions on the base hypotheses. All mod- els demonstrate lower performance on approxima- tion probes compared to the base hypotheses, with PASTA performance dropping by 227.4%%.
Related Work
Manipulation and Complex Reasoning. Fine- tuned models exhibit an average accuracy drop of 57%% on arithmetic probes, except for LUNA with a slight performance increase. The perfor- mance of PaLM (zero) and FlanT5 (few) drops by 67.67% and - 71.53% respectively. All models' performance drops on sorting probes (avg. -27%), except for DeBERTa, LUNA, and FlanT5 (zero). Unlike most other reasoning types, fine-tuned mod- els outperform few-izero-shot models complex reasoning probes. ReasTAP achieves the highest accuracy, followed by TAPAS and LUNA. FlanT5, TAPEX, and PaLM have the largest performance drops on complex reasoning probes.
Numeracy Taxonomies in NLP. Prior works have introduced surveys and taxonomies to organ- ise numeracy in NLP research. Thawani al. (2021b) categorise NLP work on numeracy into seven subtasks along the dimensions granularity (i.e. exact and approximate numbers) and unit (i.e. abstract and grounded numbers). Xu et al. (2022) focus on the robustness of QA systems in handling numerical data and organize their numeracy prob- ing tasks in two broad categories: (i) numerical parsing, and (ii) semantic parsing. The DROP benchmark (Dua et al., 2019) is QA dataset that requires discrete operations (e.g. subtraction, count,

---

sort) to answer questions over text. While Thawani al. (2021b) concentrate on number representa- tions in NLP systems, our work includes three further numerical reasoning categories. Xu et al. (2022) focus on the robustness of NLP models in handling numerical data, Our probing study on the other side pivots towards the reasoning capabilities of models when dealing with numerical and tabu- lar data. Different to prior work, our study gives a broad and in-depth evaluation of ten different models from three different categories (numerical, tabular, large pretrained LMs) on more than ten different reasoning types (representation, number sense. manipulation, and complex reasoning).
table QA datasets and generate NLI data. For data recasting, they follow systemic approach simi- lar to ours. However, the focus on transforming QA datasets, emphasizing the end-result (i.e. the NLI data) through data augmentation. They do not evaluate the presence of numerical data in their ta- bles or consider numerical reasoning in the model evaluation phrase.
Comparison to Prior Work. All the above men- tioned prior works on numerical reasoning have provided motivation for our research. However, their evaluations have focused on narrow range of reasoning types and models. Most study only concentrated on one specific model such as T5 (Pal and Baral, 2021), GPT3 (Muffo et al., 2022), or BERT (Park et al., 2022). In contrast, our frame- work provides comprehensive evaluation of nu- merical reasoning skills. We cover wide spectrum of complexity levels, ranging from representation to complex reasoning. Moreover, assess vari- ety of models with diverse architectures, sizes, and training settings for numerical reasoning.
Language Model Numerical Skills. Var- ious studies have evaluated LMs' numeri- cal skills in recent years. Earlier works probed word embeddings for numeration (e.g. 1=four) (Naik et al., 2019), comparison (e.g. 3 < 4) (Wallace et al., 2019), scale (Zhang al., 2020), and superlatives (Wallace et al., 2019). More recent works evaluate LMs on out-of- distribution numbers (Kim al., 2021), numera- tio/mmggiind//sorttggsssprriiiie (Pal and Baral, 2021), and arithmetic (Muffo et al., 2022). Our work builds upon these previous evaluation studies and extends them with further numerical probing categories, e.g. heterogeneous numbers.
7 Conclusion
This paper presents framework for probing lan- guage models' numerical reasoning skills. We or- ganise skills in a taxonomy and generate large-scale sets of probes covering more than ten numerical reasoning types. Using table NLI as a case study, we evaluate the numerical reasoning abilities of ten models. These models belong to the categories numerical LMs, tabular LMs, and few-izero-shot LLMs. We discuss reasoning types that prove chal- lenging for the probed models and explore promis- ing directions for future research.
Numerically-tuned Language Models. Various numerical LMs have been developed in recent times. Geva et al. (2020) and Liang et al. (2022) in- ject numerical skills into BERT through numerical pretraining objectives. PASTA (Gu et al., 2022) and NT5 (Yang al., 2021), which are based on De- BERTa and T5 respectively, fall into the same cate- gory of models. Another line of work adjusts LMs" architectures for numerical reasoning through nu- merical tokenization (Han et al., 2022) or addi- tional, numerical embeddings (Jin et al., 2021).
Future Directions. For certain (numerical) tasks, tool-augmented LMs equipped with capa- bilities such as calculators or code execution have been proven valuable (Mialon et al., 2023). How- ever, certain tasks require implicit numerical rea- soning which might not necessarily involve direct calculations based on numbers. For instance, classi- fying sentences that incorporate numbers in varied settings, like time indications (Feng et al., 2023), currencies conversations (Macina et al., 2023). Such tasks demand a numerical interpretation be- yond mere arithmetic computations. Moreover, calling external tools using LMs requires basic nu- merical comprehension to invoke an external tool correctly (Chen et al., 2022).
Systematic Probes for Tables. Tables have been utilized previously used to create probes for ta- ble grounding (Gupta et al., 2022b) or recasting non-NLI datasets (e.g. question-answent answering)) to NLI (Jena et al., 2022). Unlike unstructured text data, tables have natural structure that allows crc- ating controlled experiments more easily (Gupta al., 2022a). We drew inspiration from prior tab- ular probing approaches and extended them for automating probing of numerical tabular data. Jena et al. (2022) introduce a generic approach to adjust

---

Limitations
number and alphabetic arithmetic in children and adults. Memory &amp Cognition, 26(2):355
This work proposes taxonomy and framework to probe numerical reasoning skills in LMs. It in- volves the creation of large-scale probing sets using an automated approach. However, the evaluation of this approach is currently limited to the task of table NLI For future research, it is interesting to extend this to include additional tasks and datasets. This extension serves two purposes: first, it allows eval- uating a more diverse range of datasets. Second, it enables including challenges specific to other tasks.
Pierre Barrouillet and Michel Fayol. 1998b. From al- gorithmic computing direct retrieval: Evidence from number and alphabetic arithmetic in children and adults. Memory & Cogmition, 26:355-368.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NL. meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guisties, pages $185-5198 Online. Association for Computational Linguistics.
In this paper, the evaluation of most reasoning types primarily involves structural changes at the hypotheses level. While we include counterfac- tual table probes, they are limited to one dataset and perturbations method only. Further research is needed to study models' performance on numeri- cal data in the premise data. Therefore, we need table-based probes for all reasoning types of the proposed laxonomy.
Laura Bofferding. 2019. Understanding Negative Num- bers, pages 251-277. Springer International Publish- ing. Cham.
Justin W. Bonny and Stella F. Lourenco. 2013. The ap- proximate number system and its relation to early math achievement: Evidence from the preschool years. Journal of Experimental Child Psychology, 114(3):375-385
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine
Ethics Statement
Lee, Adam Roberts, Tom B. Brown, Dawn Song. ÚI- far Erlingsson, Alina Oprea, and Colin Raffll. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium, USENIX Se- curity 2021, August 11-13, 2021, pages 2633-2650. USENIX Association.
In this paper, we study the numerical reasoning skills of different LMs. However, to deploy these systems in real-world upplications, further studies and evaluations specific the intended use cases are required. In order to support future research, we plan to release all the scripts and resources used for probe creation and model evaluation. This will facilitate and encourage further research in this field.
Anthony Chen, Panupong Pasupat, Sameer Singh, Hon- grae Lee, and Kelvin Guu. 2023, PURR: efficiently editing language model hallucinations by denoising language model corruptions. CoRR, abs/2305.14900
Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W. Cohen. 2022. Program of thoughts Disentangling computation from rea- soning for numerical reasoning tasks. CoRR.
prompting:
References
abs/2211.12588..
Mubashara Akhtar, Oana Cocarascu, and Elena Simperl. 2022. Tub A public health table-based dataset for sidencee-based fact checking. In Find- ings the Association for Computational Linguistics: NAACL 2022. pages 1-16. Seattle, United States. As- sociation for Computational Linguistics.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang. Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact: A large-scale dataset for bable-based fact verification. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Medzil-Kediiorsii Yejin Choi, and Hannanch Ha- jishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-based for- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), ), pages 2357-2367, Minneapolis, Minnesota. Association for Computational Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny
Pierre Barrouillet and Michel Fayol. 1998a. From algo- rithmic computing to direct retrieval: Evidence from

---

Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz. Orhan Firat, Michele Catasta. Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- cling with pathways. CoRR, abs/2204.0231 1..
reasoning. Transactions of the Association for Com- putational Linguistics, 10:659-677 679.
Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as semi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309-2324, Online. Association for Computational Linguistics.
Vivek Gupta, Shuo Zhang, Alakananda Vempala, Yu- jie He, Temma Choji. and Vivek Srikumar. 20226. Right for the right reason: Evidence extraction for trustworthy tabular reasoning. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers)), pages 1268-3283, Dublin, Ireland. Association for Compu- tational Linguistics.
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datascts. CoRR. abs/1809.02922.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019, DROP: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long and Short Pupers), pages 2368-2378, Min- neapolis, Minnesota. Association for Computational Linguistics.
Hongwei Han, Jialiang Xu, Mengyu Zhou, Yijia Shao. Shi Han, and Dongmei Zhang. 2022. LUNA: lan- guage understanding with number augmentations transformers via number plugins and pre-training. CoRR, abs/2212.02691.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net.
Julian Eisenschlos. Syrine Krichene, and Thomas Müller. 2020. Understanding tables with interme- diate pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 281-296, Online. Association for Computational Lin- guistics.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Procediiggs of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4320-4333. Online, Association for Computa- tional Linguistics.
Yu Feng, Ben Zhou, Haoyu Wang, Helen Jin, and Dan Roth. 2023. Generic temporal reasoning with dif- ferential analysis and explanation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1 12013-12029, Toronto, Canada. Association for Computational Linguistics.
Danqing Huang, Shuming Shi, Chin- Lin, Jian Yin, and Wei- Ying Ma. 2016 How well do computers solve math word problems? large-scale dataset con- struction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 887-896, Berlin, Germany, Association for Computational Lin- guistics.
Karen C Fuson. 2012. Children's counting and concepts of number. Springer Science& & Business Media.
Mor Geva, Ankit Gupta, and Jonathan Berant. 2020. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 946-958, Online. Association for Computa- tional Linguistics.
Shotaro Ishibara. 2023. Training data extraction from pre-trained language models: A survey. CoRR, 165/2305.16157.
Zihui Gu, Ju Fan, Nan Tang, Preslav Nakov, Xiao- man Zhao, and Xiaoyong Du. 2022. PASTA: Table- operations aware fact verification via sentence-table cloze pre- traitraining. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, pages 4971-4983, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Aashna Jena, Vivek Gupta, Manish Shrivastava, and Julian Eisenschlos. 2022. Leveraging data recasting to enhance tabular reasoning. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 4483 4496, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu. Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput. Surv, 55(12):248:1
Vivek Gupta, Riyaz A. Bhat, Atreya Ghosal, Manish Shrivastava, Maneesh Singh, and Vivek Srikumar. 2022a. Is my model using the right evidence? sys- tematic probes for examining evidence-base tabular

---

Zhihua Jin, Xin Jiang, Xingbo Wang, Qun Liu, Yong Wang, Xiaozhe Ren, and Huamin Qu. 2021. Numgpt: Improving numeracy ability of generative pre-trained models. CoRR, abs/2109.00 003377
and Ashwin Kalyan, 2023. Dynamic prompt learning via policy gradient for cami-structured mathematical reasoning. In International Conference Learning Representations (ICLR).
Jeonghwan Kim, Giwon Hong, Kyung-min Kim, Junmo Kang, and Sung-Hyon Myaeng. 2021. Have you seen that number? investigating extrapolation in question answering models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7031-7037, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023. Mathdial: A dialogue tutor- ing dataset with rich pedagogical properties grounded in math reasoning problems. CoRR, abs/2305. 14536.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ramakanth Pasumuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023. Augmented language models: survey. CoRR, abs/2302.07842.
Elida V. Laski and Robert S. Siegler. 2007. Is 27 big number? correlational and causal connections among numerical categorization, number line estima- tion, and numerical magnitude comparison. Child Development, 7836:1723-1
Matteo Muffo, Aldo Cocco, and Enrico Bertino. 2022. Evaluating transformer language models on arith- metic operations using number decomposition. In Proceedings of the Thirleenth Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20-25 June 2022. pages 291- 297. European Language Resources Association.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880 Online. Association for Computa- tional Linguisties.
Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. 2019. Exploring numeracy in word embeddings. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 3374-3380, Florence, Italy, Asso- ciation for Computational Linguistics.
Zhenwen Liang, Jipeng Zhang, Lei Wang. Wei Qin.
Yunshi Lan, Jie Shao, and Xiangliang Zhang. 2022.
WWP-BERT: Numeracy-angmeenee pre-training for math word problem solving. In Findings of the Asso- ciation for Computational Linguistics: NAACL 2022, pages 997-1009, Seattle, United States. Association for Computational Linguistics.
Darko Odic and Ariel Starr. 2018. An introduction to the approximate number system. Child Development Perspectives, 1244)2223-229
Kuntal Kumar Pal and Chitta Baral, 2021. Investigating numeracy learning ability of text-to-text transfer model. In Findings of the Association for Computa- tional Linguisties: EMNLP 2021. pages 3095 -301,, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre- Trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862-6868, Online. Association for Computational Linguistics.
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Man- aal Faruqui, Bhuwan Dhingra, Diyi Yang. and Dipan- jan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP),, pages 1173-1886, 186, Online. As- sociation for Computational Linguistics.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022. TAPEX: table pre-training via learning neural SQL executor. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Sungjin Park, Seungwoo Ryu, and Edward Choi. 2022. Do language models understand measurements? In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 1782-1792, Abu Dhabi, United Arah Emirates. Association for Com- putational Linguistics.
Lefteris Loukas, Manos Fergadiotis. Ilias Chalkidis, Eirini Spyropoulou, Prodromos Malakasiotis,, Ion Androutsopoulos, and Georgios Paliouras. 2022. FINER: Financial numeric entity recognition for XBRL tagging. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 4419-4431, Dublin, Ireland. Association for Computational Lin- guistics.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems?
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, Online. Association for Computational Linguistics.
Pan Lu, Liang Qiu, Sai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark,

---

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP--JCNNPP), pages 2463- 2433, Hong Kong. China. Association for Computational Linguistics.
in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNIP-IJCNLP)) pages 5307-5315, Hong Kong. China, Association for Computational Linguis- tics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 840-854, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Jemma Catherine Whyte and Rehecca Bull. 2008. Num- ber games, magnitude representation, and basic num- ber skills in preschoolers. Developmental Psychol- ogy, 44(2):888-59.
Jialiang Xu, Mengyu Zhou, Xinyi He, Shi Han, and Dongmei Zhang. 2022. Towards robust numerical question answering: Diagnosing numerical capabili- ties of NLP systems. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 7950-7966, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Abhilash Shankarampeta, Vivek Gupta, and Shuo Zhang. 2022. Enhancing tabular reasoning with pat- tern exploiting training. In Proceedings of the 2nd Conference of the
Chapter of the Asso- ciation for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 706-726. Online only. Association for Computational Linguis- tics.
Peng-Jian Yang. Ying-Ting Chen, Yuechan Chen, and Daniel Cer. 2021. N15?! training T5 to perform numerical reasoning. CoRR, abs/2104.07307.
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre. Jason Wei, Hyung Won Chung, Barret Zoph. William Fedus. Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vin- cent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Dar- rell, and Denny Zhou. 2023. Flan-moe: Scaling instruction finntuundd language models with sparse mixture of experts. CoRR, abs/2305. 14705.
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yonghin Li. 2023. Large language mod- els are versatile decomposers: Decompose evidence and questions for table-based reasoning. CoRR, abs/2301. 13808.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Se- bastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data, In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 9413-8420 On- line. Association for Computational Linguistics.
Avijit Thawani, Jay Pujara, and Filip Ilievski. 202 la. Numeracy enhances the literacy of language models. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 6960-6967, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth. 2020. Do language embeddings capture scales? In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020. pages 4889-4896, Online. Association for Computa- tional Linguistics.
Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. 2021b. Representing numbers in NLP: a survey and a vision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compitational Linguistics: Human Language Technologies, pages 644-656, Online. As- sociation for Computational Linguistics.
Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, and Dragomir Radev, 2022. ReasTAP: Injecting ta- ble reasoning skills during pre-training via synthetic reasoning examples. Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 9006-9018, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Shyam Upadhyay and Ming-Wer Wei Chang. 2017. An- notating derivations: new evaluation strategy and dataset for algebra word problems. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Vol- ume I, Long Papers, pages 494-504, Valencia, Spain. Association for Computational Linguistics.
Ben Zhou, Qiang Ning. Daniel Khashabi, and Dan Roth. 2020. Temporal common sense acquisition with min- imal supervision. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7579-7589, Online. Association for Computational Linguistics.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP models know num- bers? probing numeracy embeddings. In Proceed- ings of the 2019 Conference on Empirical Methods

---

Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang. Jiancheng Lv, Fuli Feng, and Tat- Seng Chua. 2021 1. TAT-QA: A question answering hybrid of tabular and textual con- tent in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 3277-3287, Online. Association for Computational Linguistics.
improved significantly, which leads the conclu- sion that models are biased toward label.
Overall no language model excels the nu- merical reasoning tasks. Surprisingly, models per- form relatively well in complex tasks like Numeri- cal Word Problems but struggle at simple reasoning tasks like numeration and comparison.
benchmark
A Insights
Main Insights. We investigated the language mod- els and found that LLMs like F5 and GPT3.5 perform better than other models on various numer- ical reasoning tasks. When the labels are switched around and when dealing with negative values, we found that both aablebbsed and numerical models had difficulty comprehending the data. In contrast, DeBERTa performs relatively well compared to models like LUNA and PASTA, which are tuned for improved numerical reasoning skills.
B Probe Statistics
Reasoning Type
Count
Word Problems
Sorting
379
Counterfactual
1,000
Currency
1,014
Negative
3,316
Range
4,208
Scientitic notation
6,274
Arithmetic
8,082
Ordinal
10,569
Percentage
16,851
Date
18,642
Approximation
20,440
Compurison
30,763
Numeration
166,319
Total
288,095
Flipped probes
77,687
Table 5:
Breakdown of probes per reasoning type.
Table gives an overview of probes per dataset. Most probes (i.e. 214, 440) are created from Tab- Fact hypotheses as this is also the biggest dataset available, followed by InfoTabs (19, 779). Table provides a breakdown of probes per reasoning type. In total, we have 286, 857 probes, of which 76, 101 are label-flipping probes.
In the ideal scenario with counterfactual tables. the models' performance should be similar to the performance on the original tables. However, we observed that TAPAS and DeBERTa's performance

---

Specilic
Numerical Specific
Large
TAPAS
DeBERTa
TAPEX
NTS
LUNA
PASTA
REaSTAP
HanTT5S
GPT335
PaLM
Reasoning
few
few
Representation
Numeration
19.02
61.69
74.83
64.19
63.57
85.43
75.35
64.59
67.58
77.6
61.83
76.38
70.76
Heterogeneous
79.25
56.61
81.17
66.85
70.71
89.11
76.49
70.17
71.04
78.82
61.26
76.75
69.85
Negative
59.38
69.81
97.01
51.77
24.66
70.44
81.04
70.8
20.37
85
87
92
88
Label Flipped
Numeration
23
61.92
75.79
71.98
76.34
86.96
73.64
88.93
66.93
90.57
54.1
85 97
87.91
Heteragenevous
67.66
53.12
82.96
62.94
77.57
83.75
68.37
87.48
51.55
89.18
65.32
86.55
79.14
Negative
69.81
97.01
51.77
24.66
70.44
72.53
70.8
20.37
85
Number Sense
78.32
72.24
61.25
69.59
69
78.04
71.15
Comparison
54,12
37.77
23.87
63.78
Approximation
75.49
64.65
63.3
67.09
86.08
58.66
62.28
78.0% 78.09
75.13
59.62
76.
75.35
Range
70.84
57.53
94.99
58.14
48.86
73.25
87.77
80.2
20.09
90
85.5
80
81.41
Label Flipped
Scale
72.33
76.74
71.78
65.17
78.8
87
95,92
85.71
59.6
92.62
60.71
79.58
80.21
Comparison
53.55
70.75
38.29
68.64
69.76
85.74
42.69
86.12
49.66 99.66
96.05
37.34
82
54.86
Manipulation
Sorting
68.71
49.98
94,65
69.83
51.11
69.87
83.33
75.44
61.4
91.44
77
68.94
70.5
Arihmetic
71.15
58.01
95.92
58.83
73.6
88.97
78.92
22.42
89.5
85.5
83.5
81.5
Complex Reasoning
Complex
52.65
82.69
52.94
39.22
81.93
91.69
69.03
56.1
Counterfactual
86.9
53.85
Table 6: Results original sets (average accuracy).
Model
Table Specific
Numerical Specific
Large LMs
TAPAS
DeBERTa
TAPEX
NTS
LUNA
PASTA
ReaSTAP
FlanT5
GPT3.5
PaLM
Reasoning
few
zero
few
zero
few
zero
Representation
Numeration
18.4
59.24
68.34
60.5
59.44
79.64
69.36
64.5
63.01
77.19
64.3
75.38
74
Heterogeneous
72.76
54.52
75.75
62.19
63.73
80.59
74.32
68, 98.92
07.52
77.36
65.78
75.5
70.22
Nogative
32
60.2
97.55
2.86
62.92
76.19
84.4
28.98
92
89
Label Flipped
Numeration
43.68
39.41
29.82
41.05
70.04
56.83
45.91
Heterogenevuus
63.1
42.45
49.44
57.08
47.97
68.75
56.58
40.57
80.34
16
78.83
70.12
62.18
Negative
20.9
40.8
2.9
42.52
43.61
20.72
41.28
11.38
55.63
32
86.87
18
Number Seuse
Scale
71.68
49.43
68.38
60.39
62.21
73.63
67.23
53.12
62
80.46
66
75.15
59.39
Comparison
42.78
54.91
45.32
45.94
54.79
60.04
40.73
48.84
55.33
73.74
58.69
64.97
$1.71
Approximation
62.94
58.65
56.01
56.75
61.56
61.55
51.26
56.23
67.66 07.65
74.96
66.1
69.25
66.4
Runge
57.73
38.09
93.13
46.28
6.46
11.24
92.12
98.2
$5.87
90.47
84.5
91.5
Label Flipped
44.24
38.91
44.55
28.63
23.69
77.67
56.3
66
60.93
52.57
Comparison
$7.28
37.15
66.75
66.43
50.29
44.98
Manipulation
Sorting
44.8
63.75
8.47
53,89
77.79
64.54
13.29
10.03
76.75
38.75
72.81
46.38
42.31
Ariihmetic
71.15
43.52
4.28
42,89
37.5
37.5
9.88
22.45
81.08
35.5
89.5
28.5
Complex Reasoning
Complex
89.22
56.3
16.39
51.24
55.46
40.28
9.38
41.42
37
72.78
25
Coumerfiactal
86.21
67.24
77.8
31.23
82.1
78.43
78.1
$7.27
70. 14
81.82
72.73
72.31
80
Table 7: Results on probed sets (average accuracy).


Reviews:
[{"review_id": "LDPP26gk2M", "reviewer": "Reviewer_9VRp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their detailed review, feedback, and questions, which we address below in more detail. Moreover, we are delighted they find our work focusing on important numerical reasoning skills and providing comprehensive experiments and insightful findings. \n\n**Comparison to previous work, specifically [1]**\n\nThe paper [1] is an important work on numerical reasoning with LMs. Hence, we will include the paper in our related work. We also plan to include a detailed comparison between their findings and ours. \n\nRobustness vs. reasoning: Xu et al. [1] focus on the robustness of NLP models in handling numerical data, which is a very important constitution of NLP and numeracy. On the other hand, our probing study pivots toward models' reasoning capabilities when dealing with numerical and tabular data. This means we delve deeper into understanding how well models can make sense of numerical relations in a structured format rather than just testing their resilience.\n\n*Comparing reasoning types:*\n- [1] gives interesting insights on models reasoning on two levels: a) number detection and extraction; b) semantic parsing of numbers\n- a) is comparable to our reasoning types \"numeration\". Similar to us, they evaluate number mapping between numerals and number tokens. They also include float numbers while studying number detection. We take a similar approach for reasoning type \"heterogeneous numbers\" but additionally include date, fraction, percentage, and scientific notation as heterogeneous number types in addition to floats. These number representations are commonly found in various real-world documents.\n- b) is similar to our reasoning type \"arithmetic.\" However, while this work gives interesting insights into number parsing and using operands for calculations based on parsed numbers, our probes give a broader number of reasoning types and related challenges. For example, understanding the value of numbers by means of comparison between them, sorting numbers to understand their relation to each other, and so on. \n- Additionally, our work includes further reasoning types, which we motivate and discuss in detail in our paper. \n\n**Regarding probing additional models**\n\nOur goal on the model level was to cover a broad range of models. Hence, we included at least two models from three different categories (see Sec 5.1 for more details):\n\n1. Numerical LMs: LUNA and PASTA (both are tabular reasoning models focusing on numerical data) NT5\n2. LMs for tabular reasoning: TAPAS and DeBERTa with a table linearization approach\n3. LLMs: FlanT5, GPT3.5 (to include both an open source model, i.e., FlanT5, and a closed model with high performance on various tasks, i.e., GPT3.5)\n\nThe author further asked to include TAPEX [2]. While we considered doing so in the beginning, we selected TAPAS over TAPEX due to the following reasons (which we will include and clarify in the final paper): \n- TAPEX is pretrained by learning a neural SQL executor over database-type tables. However, our probes are not restricted to these kinds of tables but also contain InfoTabs infoboxes. \n- Moreover, we selected the TAPAS version from Eisenschlos et al. (2020) [3], which has a comparable performance to TAPEX.\n- To allow comparison to TAPEX, we have started probing TAPEX and will include probing results for the datasets other than InfoTabs to the final paper. \n\nFor the final paper, we further plan to extend our three model categories with one additional model per category. Concretely, TAPEX and ReasTAP [4], which we have started probing over the last few days.\n\n**Typos, etc.**\n\nWe thank the reviewer for pointing this out. We will fix all points mentioned in the final paper version. \n\n**References**\n\n[1] Jialiang Xu, Mengyu Zhou, Xinyi He, Shi Han, and Dongmei Zhang. 2022. Towards robust numerical question answering: Diagnosing numerical capabilities of NLP systems. EMNLP 2022.\n\n[2] Liu Q, Chen B, Guo J, et al. TAPEX: Table pre-training via learning a neural SQL executor.\n\n[3] Eisenschlos, Julian, Syrine Krichene, and Thomas Mueller. \"Understanding tables with intermediate pre-training.\" Findings of the Association for Computational Linguistics: EMNLP 2020. 2020.\n\n[4] Zhao, Yilun, et al. \"ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) This paper focuses on the important numerical reasoning skills of LMs especially LLMs, which is a key necessity in many application scenarios.\n(2) The authors introduces a new taxonomy for numerical reasoning skills together with a probe dataset for TNLI task.\n(3) Comprehensive experiments are conducted across various LMs and several insightful findings are given. ", "reasons_to_reject": "(1) In consideration of previous work [1], the contribution of this paper seems to be incremental. [1] proposed a taxonomy for numerical reasoning skills and it also introduced pertubations to construct adversarial (probe) examples, which is very similar with this paper. For instance, changing \"192\" to \"one hundred and ninety-two\", which is the same as the \"Numeration\" probes. As a result, this paper looks like an imitation of [1] with incremental adaptation.\n\n(2) Some recent sota LMs are not included in the experiments. For example, the authors choose TAPAS and DeBERTa as representatives of \"LMs for tabular reasoning\". However, new Table pre-training models such as TAPEX[2] were proposed and can not be treated as contemporary works.\n\n[1] Jialiang Xu, Mengyu Zhou, Xinyi He, Shi Han, and Dongmei Zhang. 2022. Towards robust numerical question answering: Diagnosing numerical capabilities of NLP systems. EMNLP 2022.\n[2] Liu Q, Chen B, Guo J, et al. TAPEX: Table pre-training via learning a neural SQL executor. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "With the help of a specially curated probe dataset, this paper evaluates the numerical reasoning capabilities of language models (LMs) in the tabular Natural Language Inference (TNLI) task. Specifically, samples from five tabular datasets are converted to the <tabular premise,hypothesis> format and LMs are required to classify if the hypothesis entails or contradicts the table. Moreover, three perturbations are introduced to create adversarial examples or probe examples, such as altering the hypothesis while maintaining the label unchanged. Finally, the authors evaluate three types of LMs with the resulted probe dataset and give detailed experimental analyses. \n\nThe contributions are two-folded. First, the authors introduce a hierarchical taxonomy for numerical reasoning skills and construct a probe dataset based on the proposed taxonomy. Secondly, the authors conduct extensive experiments to evaluate several LMs including SOTA LLMs such as FlanT5 and GPT-3.5.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "(1) In the caption of Table 1, \"apprimation\" --> \" approximation\". The label of \"Date Flip\" probe example of H1 should be \"C\".\n(2) In section 2, \"Figure 1\" should be \"Table 1\" or \"Table 3\"?"}, "extra_text": {}}, {"review_id": "PBGvkUDiyw", "reviewer": "Reviewer_pyHU", "summary": "", "questions": "* How do you view tool-use based methods in the context of your proposed evaluation framework?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers and are encouraged they found our work to be an interesting perspective on how LMs represent numbers and reason about them.\n\n**Regarding accuracy tables.:**\nWe thank you for raising this point. While we initially included accuracy, we swapped it by percentage to allow better comparison of performance changes across reasoning types and models. We do understand and value the point raised by the review. Therefore, we will include accuracy tables in the final version of the paper. If space allows, we will include tables in the main paper. Otherwise, we will split the tables between the main and appendix.\n\nBelow, please find the full table for *numeration (category representation)* as an example, and similar tables will be added for other reasoning types as well.\n\n| dataset  | split       | type    | label flip | model         | original accuracy | probe accuracy | delta          |\n| -------- | ----------- | ------- | ---------- | ------------- | ----------------- | -------------- | -------------- |\n| InfoTabs | alpha 1     | int2str | no         | FlanT5        | 61.95             | 61.64          | \\-0.5004035513 |\n| InfoTabs | alpha 1     | int2str | yes        | FlanT5        | 98.13             | 3.75           | \\-96.17853867  |\n| InfoTabs | alpha 2     | int2str | no         | FlanT5        | 62.1              | 59.24          | \\-4.60547504   |\n| InfoTabs | alpha 2     | int2str | yes        | FlanT5        | 95.45             | 5.84           | \\-93.88161341  |\n| InfoTabs | alpha 3     | int2str | no         | FlanT5        | 70.07             | 66.67          | \\-4.852290567  |\n| InfoTabs | alpha 3     | int2str | yes        | FlanT5        | 96.51             | 8.85           | \\-90.82996581  |\n| infotabs | test_alpha2 | int2str | no         | TAPAS         | 48.73             | 47.77          | \\-1.97003899   |\n| infotabs | test_alpha3 | int2str | no         | TAPAS         | 68.27             | 64.93          | \\-4.892339241  |\n| infotabs | test_alpha1 | int2str | yes        | TAPAS         | 71.59             | 20.45          | \\-71.4345579   |\n| infotabs | test_alpha3 | int2str | yes        | TAPAS         | 62.79             | 38.76          | \\-38.27042523  |\n| infotabs | test_alpha1 | int2str | no         | TAPAS         | 68.34             | 66.46          | \\-2.750951127  |\n| infotabs | test_alpha2 | int2str | yes        | TAPAS         | 51.14             | 37.5           | \\-26.67188111  |\n| infotabs | dev         | int2str | yes        | TAPAS         | 84.21             | 21.93          | \\-73.95796224  |\n| infotabs | test_alpha1 | int2str | no         | deberta_large | 78.93             | 74.62          | \\-5.460534651  |\n| infotabs | dev         | int2str | yes        | deberta_large | 85.96             | 36.84          | \\-57.14285714  |\n| infotabs | test_alpha3 | int2str | no         | deberta_large | 69.28             | 67.62          | \\-2.396073903  |\n| infotabs | test_alpha3 | int2str | yes        | deberta_large | 59.69             | 46.51          | \\-22.08075054  |\n| infotabs | test_alpha1 | int2str | no         | GPT3.5        | 0.75              | 0.75           | 0              |\n| infotabs | test_alpha3 | int2str | no         | GPT3.5        | 0.76              | 0.75           | \\-1.315789474  |\n| infotabs | test_alpha2 | int2str | no         | GPT3.5        | 0.66              | 0.6767676768   | 2.540557086    |\n| infotabs | test_alpha2 | int2str | yes        | LUNA_TAPAS    | 67.05             | 26.14          | \\-61.01416853  |\n| infotabs | dev         | int2str | yes        | LUNA_TAPAS    | 91.23             | 11.4           | \\-87.50411049  |\n| infotabs | test_alpha3 | int2str | no         | LUNA_TAPAS    | 71.61             | 64.93          | \\-9.328306102  |\n| infotabs | test_alpha3 | int2str | yes        | LUNA_TAPAS    | 77.52             | 24.81          | \\-67.99535604  |\n| infotabs | test_alpha2 | int2str | no         | LUNA_TAPAS    | 55.41             | 48.41          | \\-12.63309872  |\n| infotabs | test_alpha1 | int2str | yes        | LUNA_TAPAS    | 85.23             | 6.82           | \\-91.99812273  |\n| infotabs | test_alpha1 | int2str | no         | LUNA_TAPAS    | 72.41             | 66.77          | \\-7.788979423  |\n| infotabs | test_alpha1 | int2str | yes        | nt5           | 76.14             | 27.27          | \\-64.18439716  |\n| infotabs | test_alpha2 | int2str | yes        | nt5           | 53.41             | 53.41          | 0              |\n| infotabs | test_alpha2 | int2str | no         | nt5           | 50                | 49.72          | \\-0.56         |\n| infotabs | test_alpha1 | int2str | no         | nt5           | 62.94             | 57.61          | \\-8.468382587  |\n| infotabs | dev         | int2str | yes        | nt5           | 73.68             | 34.21          | \\-53.56948969  |\n| infotabs | test_alpha3 | int2str | no         | nt5           | 51.81             | 49.4           | \\-4.651611658  |\n| infotabs | test_alpha3 | int2str | yes        | nt5           | 48.84             | 41.86          | \\-14.29156429  |\n| infotabs | test_alpha1 | int2str | yes        | PASTA_DeBERTa | 92.05             | 44.32          | \\-51.85225421  |\n| infotabs | test_alpha2 | int2str | yes        | PASTA_DeBERTa | 77.27             | 56.82          | \\-26.46563996  |\n| infotabs | test_alpha2 | int2str | no         | PASTA_DeBERTa | 76.43             | 73.57          | \\-3.741986131  |\n| infotabs | test_alpha1 | int2str | no         | PASTA_DeBERTa | 85.58             | 82.45          | \\-3.657396588  |\n| infotabs | dev         | int2str | yes        | PASTA_DeBERTa | 88.6              | 49.12          | \\-44.55981941  |\n| infotabs | test_alpha3 | int2str | no         | PASTA_DeBERTa | 83.09             | 81             | \\-2.515344807  |\n| infotabs | test_alpha3 | int2str | yes        | PASTA_DeBERTa | 75.97             | 48.84          | \\-35.71146505  |\n\n**Regarding models’ performance swing**\nThe large performance swings mainly occur due to inconsistent numerical reasoning of models across tasks. For example, we observe that some models struggle with more basic reasoning (e.g., FlanT5 zero on numeration) while performing very well on more complex types. \n\nThis behavior might have different reasons, which we will include and discuss in the final paper version to provide more insights (beyond what is addressed with Q1-Q4 currently). One potential reason is memorization. Previous work [1,2,3] on LMs and memorization shows that models tend to store knowledge in their parameters, which are retrieved during a task instead of reasoning with the provided input. Hence, models can memorize common arithmetic operations during their pretraining phase and, therefore, perform very well on certain tasks. For example, flipping numbers in words (“two”) to numerics (“2”) might allow models to retrieve knowledge which they didn't consider for the initial hypothesis. Therefore, we tried to design our probes as diverse as possible (e.g., “2” to “two” flipping and vice versa). \n\nA reason for very high-performance drops can be the hallucination of models, as discussed by Gupta et al. (2022) [4]. While models might have initially performed very well on the hypothesis, adjusting the numbers can hinder models from relying on spurious patterns. We will include this and further discussion on model performance swings in the final paper version in section 5 as a newly introduced question (i.e., Q5).\n\n**Regarding Q1 being negative and expanding the hypotheses studied in Section 5**\n\nThis is a correct observation. As discussed earlier, the results show that the studied models show inconsistent performance on numerical reasoning skills. For different studied types, different models perform well.\n\nThanks for the suggestion to expand the discussion. We will expand the discussion to additional insights as we will have an additional page in the final paper version. Among others, we plan to include the following points (which we also highlighted to another reviewer) for the different hypotheses (Q1-Q4) already included in the paper. \n\nQ1: The question of whether there is one model excelling in all reasoning types will be extended with further discussion on the strengths and weaknesses of each model and how this might be related to the model’s architecture, training procedure, and evaluation setting (for example, the differences we see between finetuned, few-, and zero-shot models)\n\nQ2: This question discusses results per reasoning type. While we already split the discussion into the different reasoning types (representation, number sense, etc.), additional space will allow us not only to discuss the categories of reasoning types but each reasoning type among them individually. For example, for the category “representation,” we find differences between numeration/heterogeneous and the other reasoning types in this category. This can be related to the point that numeration/heterogeneous probes mostly don't have changes in meaning. Whereas changing a number to its negative or flipping shows strong performance shifts in very high and very low accuracies compared to the hypothesis accuracy values. \n\nQ4: While we compare tabular and numerical models and discuss them, we will further include insights for the LLMs which were probed in few/zero-shot settings and more intra (not only inter) group differences and what the reasoning for the differences are. \n\n**Regarding knowledge stored in models’ parameters (i.e., memorization)**\n\nWe thank the reviewer for raising this point! There is an interesting line of work looking into some of these aspects Gupta et al. (2022) [4]. This paper studies the phenomenon of memorization using tabular data. The paper alters the evidence tables for NLI datasets using different operations (e.g., row flipping, perturbation) to understand if (1.) models' results correlate to spurious patterns in the tables and (2.) they change their predictions if information in tables is altered. \n\nHowever, this work doesn't study this phenomenon specifically for numerical tasks.  In L54, we mention that our work focuses on studying numerical reasoning on the hypotheses level and that we plan to extend this in future work with additional evidence-level probing techniques to study phenomenon we observe potential table-related reasons for models’ high-performance changes (such as memorization, hallucination) in more detail.\n\n**Regarding tool-based augmented models in the context of our work**\n\nFor certain tasks, Augmented Language Models (ALMs) equipped with capabilities such as calculators or code generation can be immensely valuable.\n\nHowever, not every task is best suited for such tools. For instance, there are tasks that require implicit numerical reasoning, which might not necessarily involve direct calculations based on numbers. A prime example of this is classifying sentences that incorporate numbers in varied settings, like time indications, dates, or currencies. Such tasks demand a nuanced understanding beyond mere numerical computation.\n\nFurther, using LMs with equipped tools also requires a basic numerical comprehension on the part of the model. Without this understanding, invoking a tool correctly can become a challenge.\n\nWe thank the reviewer for pointing this out to us. We will discuss ALMs in the context of our work (together with a few other points) in a separate section we plan to include before the “conclusion” named “future work, open challenges.\" \n\n**References**\n\n[1] Petroni, Fabio, et al. \"Language Models as Knowledge Bases?.\" Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.\n\n[2] Carlini, Nicholas, et al. \"Extracting training data from large language models.\" 30th USENIX Security Symposium (USENIX Security 21). 2021.\n\n[3] Ishihara, Shotaro. \"Training Data Extraction From Pre-trained Language Models: A Survey.\" arXiv preprint arXiv:2305.16157 (2023).\n\n[4] Gupta, Vivek, et al. \"Is my model using the right evidence? systematic probes for examining evidence-based tabular reasoning.\" Transactions of the Association for Computational Linguistics 10 (2022): 659-679.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents an interesting perspective on how language models represent numbers and how well they are able to reason about them. \n* The paper presents results with a variety of models, this provides an interesting angle on how these models perform. \n* The authors motivate the task and describe their proposed datasets / evaluation methods thoroughly.\n* Benchmarks and standards for evaluating future models on this task.", "reasons_to_reject": "* I find the table, Table 4, is a bit hard to read. Why not present the actual accuracies? I also think that it's a bit hard for the reader to understand some of the large performance swings in the models\n* The answer to Q1 seems to be in the negative. Yet, I feel that the hypotheses explored could be expanded to include a wider variety of analysis. I think devoting more of the body of the paper to analysis of Q1-Q5 would benefit the paper, perhaps removing content from earlier sections.\n* I think that it would be nice to consider the role of facts stored in the parameters of LLMs a bit more. E.g., for some of the tables, I think that the unaltered tables could contain things that the model has memorized. I think that understanding how this changes model behavior could be interesting.", "questions_to_authors": "* How do you view tool-use based methods in the context of your proposed evaluation framework?", "paper_topic_and_main_contributions": "This paper presents a method for analyzing the performance of language models in terms of numerical reasoning. The authors study a wide variety of models such as DeBERTa, NT5, FlanT5 / GPT3.5. The authors define a taxonomy of reasoning types with corresponding datasets build upon previously existing ones. \n\nUpdate after response: Soundness increased to 3.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Can the full version of Table 4 be provided? Rather than accuracy difference?"}, "extra_text": {}}, {"review_id": "TPgh6nWmdn", "reviewer": "Reviewer_FvRi", "summary": "", "questions": "1. what about the statistics aspect of the number beyond range, e.g., mean, sum, correlation etc.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their input and are excited to read they found our work to make a valuable contribution to an important topic.\n \n**Regarding additional benchmarks on numerical reasoning**\n\nWe thank the reviewer for highlighting additional numerical reasoning datasets to extend our work. Following previous work on table-based probing/recasting frameworks (Gupta et al., 2022; Jena et al., 2022) [1,2], we concentrated on five datasets (see Table 2) that capture various types of numerical reasoning skills. In the future, we plan to extend this work with additional datasets, models, reasoning types, and probing strategies (e.g., table-based probes). We will include some of this in the revised final paper version (see our replies to other reviewers) and others in future work. \n\n**Regarding probing additional models**\n\nOur goal on the model level was to cover a broad range of models. Hence, we included at least two models from three different categories (see Sec 5.1 for more details):\n\n1. Numerical LMs: LUNA and PASTA (both are tabular reasoning models focusing on numerical data), NT5\n2. LMs for tabular reasoning: TAPAS and DeBERTa with a table linearization approach\n3. LLMs: FlanT5, GPT3.5 (to include both an open source model, i.e., FlanT5, and a closed model with high performance on various tasks, i.e., GPT3.5)\n\nHowever, we value that the author mentioned including other Tabular models. For the final paper, we plan to extend our three model categories with one additional model per category. Concretely, TAPEX and ReasTAP [4], which we have started probing over the last few days.\n\n**References**\n\n[1] Gupta, Vivek, et al. \"Is my model using the right evidence? systematic probes for examining evidence-based tabular reasoning.\" Transactions of the Association for Computational Linguistics 10 (2022): 659-679.\n\n[2] Jena, Aashna, et al. \"Leveraging Data Recasting to Enhance Tabular Reasoning.\" Findings of the Association for Computational Linguistics: EMNLP 2022.\n\n[3] Eisenschlos, Julian, Syrine Krichene, and Thomas Mueller. \"Understanding tables with intermediate pre-training.\" Findings of the Association for Computational Linguistics: EMNLP 2020. 2020.\n\n[4] Zhao, Yilun, et al. \"ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The topic is important. I have not seen similar empirical studies on this topic before. It's promising to have such fine-grained study.\n2. The paper conducts a thorough evaluation of state-of-the-art models on all reasoning types, using a diverse and extensive set of numerical probes and measuring performance shifts. \n3. The experiment results and analyses are inspiring for future works. According to the study, there is not one best-performing model across all reasoning types, so it will motivate future work on unifying the advantages of different methods.", "reasons_to_reject": "1. Benchmarks of numerical reasoning datasets can be extended, e.g., GSM8K, FinQA, HiTab, DROP, etc.\n2. More table-specific models can be explored like TaBERT, TURL, TUTA, etc", "questions_to_authors": "1. what about the statistics aspect of the number beyond range, e.g., mean, sum, correlation etc.\n", "paper_topic_and_main_contributions": "This paper assesses models on numerical reasoning abilities in a systematic way. It proposes a hierarchical taxonomy for numerical reasoning skills. And it probes existing models systematically. It is inspiring for future works on this topic.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "no", "Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {}}, {"review_id": "gtrNhvdUjX", "reviewer": "Reviewer_CcwK", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort the review dedicated to reviewing our paper and thank you for the questions. \n\n**Regarding comparison to previous works: taxonomy and insights gained**\n\nWhile the mentioned works indeed provide important insights on numerical reasoning, our work fills a gap by introducing and examining a taxonomy that captures reasoning types across four different levels including eleven different reasoning types (Figure 1). Moreover, our work allows automated creation of probes using tabular data, a data source which has a high prevalence of numerical data. \n\nWe thank the author for highlighting the work by Thawani et al. (2021) [1], Xu et al. (2022) [2] and the DROP paper, while we include the references in our paper, we will extend this with a more detailed comparison in the related work section as the reviewer suggests. \n\nWhile **Thawani et al.** [1] address the broader issues of number representation in NLP systems, our work dives into more specific aspects of numerical reasoning. Hereby, we use tabular data to create probes capturing these desired reasoning types. Tables inherently have a structured form of representing numerical data, where numbers don't stand alone but relate to other numbers and categories in a distinct manner.\nMoreover, Thawani et al. [1] offer a taxonomy based primarily on granularity (exact vs. approximate) and units (abstract vs. grounded) of number representation. While we acknowledge the significance of their taxonomy, our framework offers an enhanced taxonomy that captures representation of numbers, number sense (i.e. the meaning of numbers in the given context), manipulation of numbers, and complex reasoning with numerical data. The taxonomies, while having some overlaps, serve different purposes.\nFinally, while Thawani et al. [1] touch upon the concept of probing, our probing techniques are tailored to evaluate the model's reasoning capabilities over numerical data within tables. Their work makes an important contribution on surveying and categorizing previous work on number representation. Our probing evaluates models on their ability to understand numbers and reason with them in the context of structured tabular data.\n\n**Xu et al. [2]** is also an important work on numerical reasoning with LMs. Hence, we will include the paper in our related work and compare it in more detail to our work (e.g. on the taxonomy and findings).\nNext, we provide some details on this comparison. This and additional points will be added to the paper. \n*Robustness vs. reasoning*: Xu et al. [2] focus on the robustness of NLP models in handling numerical data which is a very important constitution to the NLP and numeracy. Our probing study on the other side pivots towards the reasoning capabilities of models when dealing with numerical and tabular data. This means we delve deeper into understanding how well models can make sense of numerical relations in a structured format rather than just testing their resilience.\n*Comparing reasoning types*: [2] gives interesting insights on models reasoning on two levels: a) number detection and extraction; b) semantic parsing of numbers\na) is comparable to our reasoning types \"numeration\". Similar to us, they evaluate number mapping between numerals and number tokens. They also include float numbers while studying number detection. We take a similar approach for reasoning type \"heterogeneous numbers\" but additionally include date, fraction, percentage, scientific notation as heterogeneous number types in addition to floats. These number representations are commonly found in various real-world documents.\nb) is similar to our reasoning type \"arithmetic\". However while this work gives interesting insights on number parsing and using operands for calculations based on parsed numbers, our probes give a broader number of reasoning types and related challenges. For example, understanding the value of numbers by means of comparison between them, sorting numbers to understand their relation to each other, and so on. \nAdditionally, our work also includes further reasoning types, which we motivate and discuss in detail in our paper. \n\nComparing out work to the **DROP** benchmark [4], would like to highlight the following aspects that we will include in the revised paper as well: \nThe DROP paper [4] focuses on a diversified set of reasoning types that revolve largely around discrete reasoning over paragraphs. These reasoning types require a combination of natural language understanding and simple arithmetic. \nOn the other hand, our work categorizes numerical reasoning more granularly, specifically emphasizing language models' capability to reason with numbers in various contexts. The taxonomy in our paper spans over ten reasoning types, some of which encompass the reasoning types in DROP but go deeper in terms of complexity and specificity. For instance, while DROP emphasizes basic arithmetic like addition and subtraction, probes more intricate numerical reasoning types that might not be solely arithmetic-based but hinge on understanding numbers themselves (representation and number sense) and in different numerical contexts.\nMoreover, a distinguishing feature of our work is the usage of table-based Natural Language Inference (NLI) to understand and evaluate the numerical reasoning of models. This is crucial as tables represent structured data are prevalent in the real-world and using our probing approach allow creating probes on a large scale. \nWhile both works share a thematic focus on numerical reasoning, the methodology employed in our paper offers a distinct exploration of models' numerical capabilities, especially in the context of structured data like tables. \n\n**Regarding comparison to Jena et al. (2022) [3]**\n\nThank you for highlighting this work by Jena et al. (2022) [3] While we refer to this work for probe creation and in the related work section of the paper, we acknowledge the feedback and will clarify with a more detailed comparison in the related work section. \nOur research focuses on probing the numerical reasoning capacities of language models, introducing a taxonomy to classify various reasoning types and employing table-based Natural Language Inference (NLI) to evaluate different model categories. In contrast, Jena et al. (2022) [3] concentrate on data transformation, applying data recasting to generate NLI datasets from tabulated structured data for QA. The aim of their work is to enrich the NLI dataset landscape.\nMoreover, while we use tables for numerical reasoning, they use tables to transform questions and answers into NLI statements, emphasizing the process and end-result (i.e. the NLI data), rather than the reasoning behind the answers.\nOur evaluation assesses diverse models on numerical reasoning challenges included in our taxonomy. Jena et al., (2022) [3] however, assess the worth of their recast data based on its efficacy in both augmenting and evaluating tabular inference tasks, showing the utility of their generated datasets in real-world applications.\n\n**Regarding insights of our work**\n\nBelow we highlight the most important insights gained through our probing study (as described in Section 5 of the paper). We appreciate the reviewers feedback and highlight the main findings more clearly in the paper. \n\n*Model Performance Across Numerical Reasoning Types:*\nNo singular model emerged as dominant across all types of numerical reasoning. However, the models FlanT5 and GPT3.5 showcased commendable numerical reasoning skills. GPT3.5, in particular, exhibited significant resilience, except when faced with complex reasoning probes. This might stem from its vast training on the extensive 300 TB Common Crawl data, allowing it a broader numerical data exposure than others. Notably, FlanT5's fine-tuning, emphasizing instructions and chain-of-thought reasoning, enabled it to outshine NT5, even though both have the T5 model as their foundation.\n\n*Representation and Number Sense:*\nWhen it came to representation, models TAPAS and FlanT5 exhibited superior performance, particularly in non-flipping numeration probes. However, a performance dip was observed when models encountered negative number probes, suggesting potential challenges in recognizing the \"-\" sign representation of negative numbers. Yet, few-shot models like FlanT5 and GPT3.5 seemed more adept at comprehending the linguistic expression of negatives. Regarding number sense, while fine-tuned models found comparison probes challenging, FlanT5 and GPT3.5 managed to keep their performance, sometimes even outperforming their base hypotheses.\n\n*Manipulation, Complex Reasoning, and Label Flipping:*\nIn the categories of manipulation and complex reasoning, fine-tuned models generally saw a 23.5% average decline in accuracy, with LUNA being an exception. Surprisingly, few-/zero-shot models improved their performance by 9.5%. Yet, when confronted with complex reasoning challenges, fine-tuned models seem to perform better in general. Label flipping, however, posed a significant challenge. A steeper performance drop in flipping probes as compared to their non-flipping counterparts highlighted models' potential over-reliance on spurious patterns, rather than genuine comprehension.\n\n*Numerical vs. Table-Specific Models:*\nLUNA, which utilized a unique tokenization approach for numbers, showed similar performance to other models across various reasoning types, except for a slight edge in comparison flipping probes. PASTA, despite its numeric-focused design and objectives, only outperformed DeBERTa in handling negative number and scale probes. TAPAS, a table-based model, performed well non-flipping scenarios but showed declines when labels were flipped, emphasizing its challenges with spurious correlations. \n\nIn general, our probing evaluation shows that performance variances across different models for numerical reasoning, underscoring the need for detailed evaluation of models with fine-grained categories of numerical reasoning to understand their strengths and weaknesses in detail. \n\nFor the final paper we further plan to extend our three model categories with one additional model per category. Concretely, TAPEX and ReasTAP [5] which we have started probing over the last days.\n\n**References**\n\n[1] Thawani, Avijit, et al. \"Representing Numbers in NLP: a Survey and a Vision.\" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.\n\n[2] Jialiang Xu, Mengyu Zhou, Xinyi He, Shi Han, and Dongmei Zhang. 2022. Towards robust numerical question answering: Diagnosing numerical capabilities of NLP systems. EMNLP 2022.\n\n[3]  Jena, Aashna, et al. \"Leveraging Data Recasting to Enhance Tabular Reasoning.\" Findings of the Association for Computational Linguistics: EMNLP 2022.\n\n[4] Dua, Dheeru, et al. \"DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs.\" Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.\n\n[5] Zhao, Yilun, et al. \"ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper investigates numerical reasoning which has been and still is one of the toughest skills for language models as they often struggle to reason correctly over the numerical information in text. \n\n2. This paper presents an approach to create probing data for different numerical reasoning skills which may help in doing fine-grained investigations of LLM’s basic understanding of numbers.", "reasons_to_reject": "1. Firstly, the work lacks sufficient contributions for it to be a long paper. Since, most of the strategies to create probing data in section 4 are about replacing the numbers, dates, etc. I think it can be presented in a much more concise manner. Also, previous works such as Thawani et al., 2021b; Xu et al., 2022, DROP dataset have proposed taxonomies and datasets that cover majority of the skills presented in this work such as Numeration, Arithmetic, Comparison, Word Problems, Measurement Estimation, DROP includes Comparison, Subtraction, Selection, Addition, Counting, Sorting, Coreference resolution, Word Problems, etc. Thus, section 2 can also be concisely presented. \n\n2. The need for a new taxonomy is not well motivated in the paper as previous works such as Thawani et al., 2021b; Xu et al., 2022 already provide a detailed taxonomy covering numerical reasoning skills. \n\n3. In the majority of Section 5.3, just the numbers from Table 4 have been mentioned in words. It lacks conclusive analysis. Also, this work doesn’t lead to new results or findings as such. \n\nFurthermore, the paper “Leveraging data recasting to enhance tabular reasoning” uses more techniques to create Tabular NLI instances. It uses augmented instances to improve reasoning; in this work, a simpler augmentation technique is used for creating probing data. This paper and how it is different from the current work needs to be discussed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a new taxonomy for numerical reasoning skills at four levels, namely, representation, number sense, manipulation, and complex reasoning encompassing over 10 reasoning types. Then, they propose an approach to create a probing dataset for each reasoning type using Tabular Natural Language Inference datasets and analyze performance of several models such as FlanT5, GPT 3.5, NT5, and DeBERTa.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "L067 -  four level:  →  four levels: \n\nL260 - Overall, we our framework  → Overall, our framework \n\nL291 - This section discussed probes → This section discusses probes\n\nL296 - Onw converting  → One converting \n\nL427 → which are divers in terms  → which are diverse in terms \n\n..."}}]

Author rebuttal:

Meta-review: The reviewers found that the topic (of categorizing and analyzing numerical reasoning abilities of models) to be important and the empirical study extensive and systematic. There were some questions raised regarding the need for yet another taxonomy of numerical problems. The authors, in their response, clarified in detail important differences from prior work in the area. This would be very helpful to include in the revised version of the paper (as the authors indicated they would).

While it would be even better to expand the set of models used further to other table-specific models, I think the current coverage is broad enough to provide meaningful value to the readers.

The findings seem important and valuable enough to share with the community. At the same time, the paper lacks broad excitement, in part due to the existence of some prior efforts on analyzing numerical skills and in part due to the nature of this work being analysis-oriented rather than proposing a novel methodology that pushes the state of the art forward.
Decision: Accept-Findings