Not all quantifiers are equal: Probing Transformer-based language models' understanding of generalised quantifiers
Authors: Tharindu Madusanka
Iqra Zahid
Hao Li
Ian Pratt-Hartmann
Riza Batista-Navarro
Venue: EMNLP 2023 Main (2023)
Keywords: Natural Language Inference
Transformer-based language models
TL;DR: We investigate how generalised quantifiers affect the behaviour of transformer-based language models employing the textual entailment problem defined in a purely logical sense, namely, model-checking with natural language.

Abstract: How do different generalised quantifiers affect the behaviour of transformer-based language models (TLMs)? The recent popularity of TLMs and the central role generalised quantifiers have traditionally played in linguistics and logic bring this question into particular focus. The current research investigating this subject has not utilised a task defined purely in a logical sense, and thus, has not captured the underlying logical significance of generalised quantifiers. Consequently, they have not answered the aforementioned question faithfully or adequately. Therefore, we investigate how different generalised quantifiers affect TLMs by employing a textual entailment problem defined in a purely logical sense, namely, model-checking with natural language. Our approach permits the automatic construction of datasets with respect to which we can assess the ability of TLMs to learn the meanings of generalised quantifiers. Our investigation reveals that TLMs generally can comprehend the logical semantics of the most common generalised quantifiers, but that distinct quantifiers influence TLMs in varying ways.

Not all quantifiers are equal: understanding of generalised quantifiers
Probing transformer-based language models'
Tharindu Madusanka and Iqra Zahid and Hao Li
Ian Pratt-Hartmann and Riza Batirto-Navarro Department of Computer Science The University of Manchester
Abstract
given concept is best achieved by evaluating their proficiency in applying it. In the case of gen- cralised quantifiers, the most suitable evaluation task is textual entailment. This is particularly rele- vant because altering quantifiers can fundamentally change the logical inferences derived from a given text, reinforcing the integral role that quantifiers occupy within the scope of the textual entailment task.
How do different generalised quantifiers af- fect the behaviour of transformer-bast based lan- guage models (TLMs)? The recent popularity of TLMs and the central role generalised quan- tifiers have traditionally played in linguistics and logic bring this question into particular fo- cus. The current research investigating this subject has not utilised a task defined purely in a logical sense, and thus, has not captured the underlying logical significance of generalised quantifiers. Consequently, they have not an- swered the aforementioned question faithfully adequately. Therefore, investigate how different generalised quantifiers affect TLMs by employing textual entailment problem de- fined in purely logical sense, namely, model- checking with natural language. Our approach permits the automatic construction of datasets with respect to which we can assess the abil- ity of TLMs to learn the meanings of gener- alised quantifiers. Our investigation reveals that TLMs generally can comprehend the logi- cal semantics of the most common generalised quantifiers, but that distinct quantifiers influ- ence TLMs in varying ways.
When discussing entailment, it is vital to ac- knowledge two distinct strands of research in the literature. The first strand incorporates background knowledge and common sense into entailment, im- buing with a probabilistic character (Bowman et al., 2015; Williams et al., 2018; Wang et al., 2019). The second strand examines textual en- tailment in purely logical sense, eliminating the influence of background knowledge and common sense (Richardson and Sabharwal, 2021; Schlegel et al., 2022: Madusanka al., 2023). While the first form of entailment proves beneficial for mul- titude of practical applications, is not ideal in an investigation centred on the impact of linguistic properties with logical significance, such as gen- cralised quantifiers and negation. The empirical evaluation of linguistic constructs under this kind of entailment gets compromised due to its intri- cate association with other concepts. Consequently, challenging to differentiate the performance variation due to linguistic properties from those attributable to concepts like common sense and background knowledge. However, prior literature has only investigated generalised quantifiers in the context of entailment that incorporates background knowledge and common sense (Cui et al., 2022; Apidianaki and Garí Soler, 2021) and naturally suffers from the same predicament. The second strand of textual entailment by defining entailment in purely logical sense circumvents the afore- mentioned shortcoming. Consequently, it offers a conducive environment for conducting evaluations
1 Introduction
Generalised quantifiers have been topic of much interest for more than century in logic and linguis- ties (Frege, 1882; Westerstähl, 1987; Gabbay et al., 1989; Mostowski, 1957). By capturing the interplay between quantity and cardinality, they provide a useful lens through which to understand human lan- guage and cognition (Troiani et al., 2009; Szymanik and Zajenkowski, 2010a). Since transformer- bassed language models (TLMs) strive to stimulate human- like language understanding (Vaswani et al., 2017; Devlin al., 2018; Raffel et al., 2019; Ouyang al., 2022: Chowdhery al., 2022), is essential to determine the extent to which they can com- prehend generalised quantifiers. Assessing the depth of understanding that TLMs possess for any

---

Structure in Natural Language: Talia, Hailee, Ava, Aria, Tony, Roger, Peter and Solomon are members of some group. Talia, Ava, and Solomon are bee-keepers. Hailee, Ava and Tony are scientists. Tony, Roger, Solomon, Avo, Aria and Hailee are Musicians. Roger, Solomon, Ava and Ariu are guitarists. There are designers in the group. Peter, Tony and Roger artists, Only Solomon is engineer.
Sentence: least musicians are guitarists validity: Truc
Sentence: All bee- bee-keepers are scientists validity: False
Figure 1: An instance of the model-checking with natural language problem, the sentence "At least 3 musicians are guitarists" is True according to the structure since the set musicians X = T RRoger, Solomon, Ava, Aria} are also guitarists and [X] 2 However, the sentence " All bee-keepers are scientists" are F False as the set of bee-beee bee-keepers (Talia, Solomon) } are not scientists
centred around linguistic constructs.
junctions and negation with generalised quantifiers. This aspect of our study sheds light on the intri- cate dynamics that exist between these linguistic elements and the challenges they pose to TLMs. This comprehensive analysis paves the way for a more nuanced understanding of how TLMs handle intricate linguistic constructs such generalised quantifiers.
The logical problem that is most suited to study the influence of language constructs of logical sig- nificance is that of model-checking: given for- mula & and a structure 2(, determine whether is true in 2 (Ml + Φ). In the context of nat- ural language, we are interested in a variant of the model-checking problem where the structure and the formula are translated into natural lan- guage. An instance of model-checking with nat- ural language problem is depicted in Figure 1. From complexity-therrete point-of-view. model- checking in most formal languages is, compara- tively speaking, straightforward. Indeed the model- checking problem with a fixed number of free vari- ables and a finite structure is in PTIME. This is in contrast to other logical problems, such as satis- fiability, whose problems for various fragments of logic can belong to different computational complexity classes (Pratt-Hartmans 2004; Pratt- Hartmann and Third, 2006). Yet, solving instances of the model-checking problem with natural lan- guage requires comprehensive understanding of the logical semantics of the expressions involved. Thus, it provides an ideal test environment to faith- fully evaluate the extent to which generalised quan- tifiers affect transformer-based language models.
The key contributions of the present research can be summarised as follows: (1) To the best of our knowledge, this study represents the first exploration into the effects of generalised quanti- fiers within a logical entailment context; (2) We analyse the effect on TLMs when quantifiers are paired with diverse logical constructs like nega- tion and Boolean conjunctions; (3) We compare and contrast the behaviour of TLMs with quanti- fiers with that of quantifier verification experiments done with human beings; and (4) We delve into how well TLMs comprehend generalised quantifiers zero-shot context employing prompt engineering approaches such as chain-of-thouglh thoghhtproootptiing (Wei et al., 2022b) and provide comparisons be- tween pre-trained and fine-tuned models.
2 Related Work
Our work follows the literature on probing how dif- ferent linguistic properties affect the behaviour of neural approaches such as transformer-buse based lan- guage models (Madusanka et al., 2023; Clark et al., 2021; Buijtelaar and Pezzelle, 2023; Jawahar et al., 2019; Ettinger, 2020). Specifically, our investiga- tion is closely related to the literature whose lin- guistic properties of interest are generalised quanti- fiers (Cui et al., 2022; Apidianaki and Garí Soler, 2021). Our exploration differentiates from prior research in two key ways. First, we explore gener- alised quantifiers employing a task that is defined purely in a logical sense. Thus, we provide a more faithful investigation into how TLMs comprehend
In this study, we embark on an in-depth inves- tigation into TLMs' understanding of generalised quantifiers utilising the model checking problem, juxtaposing this with cognitive science research on quantifier verification tasks (Szymanik and Za- jenkowski, b010a,b; McMillan et al., 2005). A critical part of our exploration involves the evalua- tion of tre-trained models prior to any fine-tuning. Thus, allowing us to discern whether any differ- ences identified are intrinsic to the models them- selves or introduced through the process of fine- tuning. Additionally, we consider the complexi- ties arising from the integration of Boolean con-

---

GQ
Logical Denotation
All
{(X, Y) X CYC C 21
Some
((XY)) Xny + and X,Y S )
At least
((X,Y)
Xny 2 and X, Y C
At most K
((X,Y) XnY I Kand Y 2
Less than K
((X,Y) XXYY < Kand X, X,Y [1]
More than
((X,Y) XnY
and X,Y C 21)
K
{(X,
| XNY = K and X, Y [
Most
((XY))
XNY
X Y and X,Y C 2
Few
(X,Y)
XNY
X Y and X, C 21)
Table 1: The generalised quantifiers (GQ) we used in our experimental setup, along with their logical denotation defined on some structure 21
generalised quantifiers. Second, our research also integrates comprehensive analysis of how the in- teraction negations and Boolean conjunctions with quantifiers influences TLMs' performance in a simple entailment task. We follow the logical denotations introduced in logical studies to for- malised generalised quantifiers when formulating our task (Westerstähl, 1987; Mostowski, 1957: Gab- bay et al., 1989; Fuhrken, 1970; Peters and West- erstähl, 2006) and draw parallels with cognitive science work on quantifier verification our exper- imental setup (Szymaník and Zajenkowski, 2010b; McMillan et al., 2005; Szymanik et al., 2016).
ments, set of sentence templates is a natural way of representing them. For example the Aristotelian syllogism (Smith et al., 1989) can be defined using the following set of templates.
All As are Bs Some As are As No A is a B
Some are not Bs
In this work of literature, we employ a slightly extended version of the Aristotelian syllogistic to allow negations at the subject, (e.g., Some non- musicians are beekeepers) and generalised quanti- fiers when generating sentences.
Generalised quantifiers define the semantics of sentences that include them in terms of relations between subsets of the structure (Szymanik et al., 2013). Consider for example "All musicians are artists". The determiner phrase "All" in this sen- tence specifies a relation between the set of musi- cians and the set of artists, namely that the former is subset of the latter. More generally, "All" in structure 21 expresses the binary quantifier:
Our evaluation scheme for evaluating TLMs in a zero-shot setting builds upon prior literature on prompt engineering (Brown et al., 2020; Kojima et al., 2022; Wei et al., 2022b). However, ours is the first literature evaluating TLMs on the model- checking problem in zero-shot settings.
3 Methodology
3.1 Language Fragments and Generalised Quantifiers
XX C
This idea can be generalised to accommodate other quantifiers. Consider the sentence "At least K musicians are artists" where K N. The phrase "At least K" likewise expresses relation between the of musicians and the set of artists, namely the cardinality of their intersection at least K, that is, "At least K"in I expresses the binary quan- tifier:
We define language fragment to be set of sen- tence forms equipped with semantics translating those sentences to some formal system such as first- order logic (Pratt-Hartmann, 2004) and perhaps the simplest way to define a language fragment is via a finite set of sentence templates. A sentence template is a sentence in which certain open- plen-class words have been replaced by schematic variables. For example, "All As are Bs" is sentence tem- plate where A and B substitute ordinary nouns (e.g., artist, musician beekeeper, ...). and by substi- tuting A and B with such nouns, we can formulate sentences such as "All musicians are artists". Due to the formal structure that exists in language frag-
(X,Y) K and X, X)
In our scholarly inquiry, we examine logical quantifiers such as "All", numerical quantifiers such as "At least KS and propositional quantifiers such as "Most". The quantifiers employed, and

---

their logical denotation on structure are depicted in Table 1. We utilise these generalised quantifiers when defining language fragments for sentence gen- eration. Let Tq be the sentence template which defines the language fragment corresponding to the quantifier Q. For example, consider the quanti- fier "All", the corresponding template TAu takes the form of "All (non-)AAs are (not) Bs" where A and B are replaced by ordinary nouns. Appendix A depicts the sentence templates used define language fragments for each of the quantifiers.
true in M. To construct (M, 8, s), apart from To. the algorithm takes the vocabularies D, P also inputs. The vocabulary D comprises proper nouns employed to characterise domain elements, while the vocabulary P comprises ordinary nouns that characterise predicates. We draw random sample of elements D and P. from vocabularies D and P to construct the structure. Two random nouns are sampled from P, each is then negated with proba- bility Pneg- and these are finally substituted for the two schematic variables in the template To form the sentence s.
3.2 Data Construction
Given (probably negated) words A, B, gener- alised quantifier Q and a structure 2, the model- checker determines } s, where s is the sen- tence formed by substituting A. B for schematic variables in the templates To. This involves first determining the extensions of Á and B in 2 and then applying the meaning of gener- alised quantifier Q to these sets. Consider the example put forth in the section 1, (Q, A, B) corresponding the sentence "All bee-keepers are scientists" is (All, beekeepers, scientist). the model-checker determines the extensions of beekeepers and scientists in the corresponding structure 21 to be {Talia, Ava, Solomon) and [aailee, Ava, T Tony), respectively. The quanti- fier All dictates that in order for the sentence to be True, the former needs to be subset of the lat- ter. However, the set (Talio, Ava, Solomon) & {H Ava, T Tony), thus, the model-checker as- signs False as the validity label t.
Algorithm Data Construction Model checking with Generalised Quantifiers
Input The Quantifier Q and corresponding sentence template Tq. natural language template M to convert the structure to natural language, the vocabulary of proper nouns 7) and ordinary nouns P. minimum and maximum number of domain elements dmin and d'uaz, minimum and maximum number predicates pm and p Output: model checking dataset D
1: D -
2: repeat
3: D, P.P + sample from vocabularies D and
P such that dmnin
< [ S mma mnnn x
PP < Mer
4:
A, B +- sample predicates from P
5:
A, B negate A, B with Preg
6: s + substitute predicates A and B for
can be extended This setup with relative when Boolean conjunctions are introduced the sentences. Consider, for instance, sen- tence pair 81 and 82. formed using the predi- cates (A1, B1) and (A2, B2), respectively, for some quantifier Q, merged using Boolean con- junction @ E (A,V). To adapt to this scenario, the algorithm can be augmented by effecting a simple modification to step 10, transforming it into é + MODELCHECKER (Q, A1, O MODEI SADDELCCECCKR(SA (Q, 72, B2)).
schematic variables in the template To
7: L + -ample from (True, False}
8: repeat
9:
21 + generate structure randomly using
(D, P)
10:
(Q,A,B) M + translates 2 to natural language using
f + MODELCHECKER
until = 7
the template M
13:
- DU (M, s,
14: until stop condition is met
3.3 Prompts for Zero-shot Evaluation
We develop data construction algorithm (Algo- rithm 1) to construct balanced dataset free from easily exploitable trivial linguistic patterns. The al- gorithm constructs set of triplets (M, s, E), where M the natural language translation of the struc- ture, is sentence of the relevant fragment To and l a label (True/ False) specifying whether s is
Given that transformer-based language models un- dergo pre-training through certain form of lan- guage modelling objective, the most common ap- proach to evaluate these models in the zero-shot setting is by employing prompt engineering (Brown et al., 2020). Consequently, we formulate prompts

---

following a template-based strategy, utilising the constructed tuples (M, s, f). We adopt two distinct types of templates. The first adheres to more traditional form of prompting, which we refer to as standard prompting. The second type of tem- plate is based on the concept of chain-of-thought prompting (Wei et al., 2022b), Chain-of-thoughh prompting is technique in which an example prob- lem instance, accompanied by an explanation of the underlying thought process, used guide the model towards generating more precise responses. We depict the cxact templates in Appendix A.
decoder. DeBerta-v3 further improves the architec- ture by utilising an ELECTRA-style pre-training with Gradient Disentangled Embedding Sharing. We fine-tune the DeBERTa-v3-Lare model with around 304M parameters.
ChatGPT Due to the recent success of ChatGPT in solving many natural language tasks in a zero- shot setting (Bang et al., 2023), we employ it in a similar context. Similar to InstructGPT (Ouyang et al., 2022), ChatGPT is trained to follow human instructions but follows a slightly different data collection approach.
4 Experimental Setup
LLaMA Considering that Flan-T5 and ChatGPT are trained to follow instructions, we decided to use TLM which has not been explicitly trained to follow instructions as one of our baselines. Thus, employ LLaMA-30B model in zero-shot settings, The LLaMA is said to outperform GPT-3 in most baselines and achieve comparable performance with respect to state-of-the-art TLMs (Touvron et al., 2023).
4.1 Transformer-based language models
To explore the transformer- based language mod- els' ability to comprehend different generalised quantifiers, we employ set of TLMs that have proven track record in textual entailment problems, namely, T5, Flan-T5, DeBERTa, LLaMA and GPT.
T5 Following the prior work on textual entail- ment defined purely in a logical sense (Richardson and Sabharwal, 2021; Tafjord et al., 2021; Madu- sanka et al., 2023), we utilise the T5 model in our experimental setup as one of the baseline models. The T5 model (Raffel et al., 2019) employs a uni- fied text-to-text format where all inputs and out- puts are textual strings. We fine-tune the T5-large model with 770M parameters to perform the model- checking task.
4.2 Dataset and Evaluation
To fine-tune and evaluate TLMs, we construct train and test sets with 72K and 36K unique prob- lem instances with 8K and 4K data points for each generalised quantifier'. We arbitrarily select [dmin, _dmeaa = [8, 14] and [pmmin, "y""tinp""ee p= = [5, 10) when constructing problem instances. We con- struct balanced dataset, and thus, use accu- racy as the main metric but chose depict the overall precision, recall and fl-score to provide a more detailed analysis. We deem this setup an- swers the question. "How do different quantifiers affect the behaviour of TLMs?". As the prob- lem instances contain negations, our experiment will also provide insight on the effect of negation when intertwined with quantifiers TLMs' under- standing of language. We construct separate train and test sets with 72K and 36K problem instances with sentences containing Boolean conjunctions to answer the question, "How do Boolean conjunc- tions affect the behaviour of TLMs when coupled with different quantifiers?", By evaluating these fine-tuned models against problem instances with higher K values in the numerical quantifiers than that of the train set, ask the question "Do TLMs learn to comprehend the logical semantics of gen- eralised quantifiers?". To answer the questions,
Flan-TS Fine-tuned Language Net (Chung et al., 2022), also known as Flan, is based on instruc- tion fine-tuning (Wei et al., 2022a) with the ob- jective of making the transformer model gener- alise better to unseen tasks. The Flan-T5 model, considered to be improvement T5, applies instruction fine-tuning on the T5-model family. Thus, we primarily centred our experimental setup around the Flan-T5 model. We fine-tune the Flan-T5-large model with 770M parameters and utilise F5-aae with 220M parameters, Flan- F1an-T5-xl x1 with 3B parameters and Flan-T5-xxl with 11B parameters in the zero- shot setting.
DeBERTa-v3 Due to the recent success of the DeBERTa-V3 model (He et al., 2021) in solving nat- ural language inference tasks, we utilise it as base- line model. The DeBERTa architecture improves upon the BERT and Roberta models using a disen- tangled attention mechanism and enhanced mask
'The dataset and code available at https: t/github.. com/ iThar i indu cerooo/ggeeralll aidd sediissanddiiiie quantifiers model- chee-kcceccngggg

---

"How do pre-trained TLMs comprehend different quantifiers?" and "Do they have any biases when performing a simple entailment lask?", we eval- uate TLMs in zero-shot setting. We found that the problem instances with [dmin, _dmaa] = [8, 14] and [p"iin,"""rr p= [5, 10] are challenging for TLMs in zero-shot settings. Consequently, the use of the same test set did not yield any meaning- ful insights. Thus, we formulate a much simple problem instance with [dmin _dmar]] [ and Ipmin _pmaa]] = 4]. more detailed descrip- tion of the dataset and fine-tuning is provided in Appendix B.
GQ
ac
re
fl
All
91.4
88.6
95.3
91.8
Some
92.2
96.8
88.0
92.2
At least K
94.2
97.7
90.4
93.9
At most K
96.6
96.5
96.6
96.5
Less than K
95.1
93.4
97.2
95.3
More than
94.9
96.9
93.0
94.9
K
90.8
86.7
96.2
91.2
Most
90.4
92.9
87.3
90.0
Few
91.1
93.1
89.2
91.1
Table 2:
The test scores for the Flan-T5-large model across various generalised quantifiers. The abbrevia- tions ac, pr, re and denote accuracy, precision, recall and Fl score values.
5 Results and Discussion
The ability of transformer-basee language mod- els to solve instances of the model-checking prob- lem differentially influenced by various gen- eralised quantifiers. As demonstrated Table 2, TLMs appear to encounter the most difficulty with proportional quantifiers such as "Most" and "Few". Interestingly, this empirical observation aligns with cognitive science research, which also highlights the complexities faced by humans in in- terpreting proportional quantifiers (Szymaník and Zajenkowski, 2010a; McMillan al., 2005; Troiani et al., 2009). addition, the performance related to the quantifier "K" is notably lower compar- ison to other numerical quantifiers. A sentence incorporating the quantifier "K" is probabilistically more likely to be False given a random structure. Therefore, in balanced dataset, the determination of the truth value of a sentence containing the quan- tifier "K" necessitates a more detailed examination compared to other numerical quantifiers consid- ered in this study. However, as illustrated in Figure 2. given an adequate number of training steps, all TLMs attain satisfactory performance levels across all quantifiers. Moreover, the newer TLM models, such as DeBERTa-v3 and Flan-T5, exhibit faster convergence rate compared to T5.
since there are more than 14 artists who are en- gineers in the world. This proposition remains true even when negations are introduced to the sen- tences. Thus, we speculate that TLMs overcorrect during fine-tuning and predict T True or False ac- cordingly.
TLMs show evidence of learning to under- stand the logical semantics of generalised quan- tifiers. As illustrated in Figure 3, when tested with dataset containing higher K values than that of the train set, the accuracy of TLMs only decreases slightly for all numerical quantifiers. Therefore, we posit that TLMs possess the capacity to learn the logical semantics associated with generalised quantifiers. Our conclusions regarding generalisa- tion bear resemblances to prior work conducted on model-checking with natural language (Madu- sanka et al., 2023). Their research also supports the premise that TLMs are capable of comprehending the logical semantics of natural language. Addition- ally, we highlight the contrast between the demon- strated ability of TLMs to generalise in the context of model-checking problems, and their apparent lack of such generalisation when solving satisfia- bility problems (Schlegel et al., 2022; Richardson and Sabharwal, 2021). We hypothesise that this distinction is due to the different complexity levels associated with these two types of problems and the necessity to understand complex inference rules when solving satisfiability problems.
As expressed by their precision and recall values, TLMs often predict Truc for quantifiers such as "All" and "Less than K", but often predict False with respect to quantifiers like "Some" and "More than K" We attribute this to be overcorrection in- troduced during fine-tuning. Consider the sentence with the quantifier "Less than K" "Less than K artists are engineers". This sentence more likely to be False in the context of the real world for K values we consider this study (8 < K < 14)
The Boolean conjunctions have significant effect, while negation has much less effect on fine-tuned TLMs when coupled with generalised quantifiers. As demonstrated in Table 3, it is ap- parent that fine-tume tuned TLMs possess the capacity to

---

300RSC
0.4
0.3
6003
stops
Figure 2: The rate of convergence of (a) Flan-T5-large and (b) DeBERTa-v3-Large and (c) T5-large models break down based on different quantifiers
jr
/|
Moantaller Hore
AVD
OR
AND
AND
AND
02t
0.95
91.0
83.7
89.0
80.9
93.8
89.5
91.3
84.9
Al least K
90.9
86.9
93.3
84.7
75.9
88.8
88.7
89.4
91.0
87.0
6.94
bend
76.5
93.0
87.5
74.2
88.9
69.7
85.7
86.3
87.3
77.1
6.47
Less than
90.6
92.3
90.3
84.0
91.0
87.9
90.7
than
Most
82.2
81.3
82.6
M0.7
83.0
83.0
$2.8
81.8
80.4
Figure 3: The accuracy value when tested against prob- lem instances with sentences containing higher val- than that of the train set. The results broken down based on the numerical quantifier.
Table 3: The test accuracy values for the various generalised
Flan-T5-large model quantifiers, broken down based on the Boolean conjunc- tion (AN D) Oll) in the sentence. abbreviations ac, pr. and fl denote accuracy, precision, recall and F1 score values.
tors such as the number of parameters, dataset size and computational resources (Kaplan et al., 2020). However, upon breaking down the per- formance metrics based on the quantifiers, the re- sulting graph (Figure 4 (b)) is observed to be less uniform compared to the representation of overall performance. We attribute this behaviour to the in- herent probabilistic aspect of the predictions formu- lated by TLMs since language models are trained to find the most probable next word given a of words. This probabilistic nature language mod- els can lead to inaccurate predictions, especially a logical context.
comprehend negations. In contrast, depicted in Figure (c), prior to the fine-tuning process, nega- tions exert considerable influence. Consequently, we propose that fine-tuning plays a significant role in enhancing the ability of TLMs to understand negations. The inclusion of Boolean conjunctions significantly reduces the accuracy for all quanti- fiers. Noticeably, quantifiers for whom TLMs tend to predict True often tend to have higher accuracy in the context of the OR operation compared to AND and vice-versa. In our findings, we discov- ered that quantifiers for which TLMs frequently predict the label as True also display elevated re- call values for OR operations and diminished recall for AND operations. Conversely, quantifiers that TLMs often predict as False exhibit higher preci- sion for AND D operations and lower precision for OR operations.
The Flan-T5 model with fewer parameters out- performed the ChatGPT and LLaMA models in zero-shot setting, depicted in Table 5. This phenomenon is unsurprising since Flan-based mod- els are very effective tasks naturally verbalised as instructions due to their employment of instruc- tion fine-tuning (Wei et al., 2022a). Upon con- trasting the efficiency of ChatGPT and Flan-T5 models in the context of standard and chain-on thought prompting techniques, it is observed that the discrepancy in accuracy metrics across these two distinct prompting methodologies is not sub- stantial. However, the LLaMA model generated both Truc and False when generating the label when standard prompting is used, failing to follow the instruction properly. We attribute this failure
The number of parameters, training pro- cess and type of prompting can influence the TLMs' performance when solving model- checking problem instances in zero-shot settings. The performance for Flan-T5 models exhibits the power law relationship with the number of param- eters, as illustrated in Figure (a). This empirical finding is consistent with prior research analysing language models' performance variation with fac-

---

restoon
15
mam paramettr
10
Figure 4: The (a) overall test loss and (b) test loss break down based on quantifier and (c) accuracy values breakdown based on the availability of negations for the Flan-T5 model with a different number of parameters zero-shot settings, the number of parameters variates from 220M to 11B.
GQ
n
s s'n00
s'n=
ChatGPT
Flan-T5
LLaMA
All
91.6
94.7
89.0
89.6
st
ch
st
ch
ch
Some
95.3
91.8
91.9
89.7
All
57.8
59.3
59.5
65.6
50.5
Some
77.7
80.0
79.4
79.1
58.0
At least K
94.7
94.6
93.9
94.6
At least K
84.1
80.0
87.1
87.1
67.3
At most K
96.2
97.0
96.5
96.6
At most K
42.9
42.2
50.6
46.9
45.3
Less than
96.1
95.5
95.7
95.0
Less than K
48,5
44.4
63.0
64.6
49.2
More than K
95.6
95.2
95.4
94.9
More than
82.6
75.9
86.5
88.2
53.2
K
90.9
91.8
91.5
89.7
K
68.1
70.
76.6
70.5
53.4
Most
93.6
92.4
88.1
88.6
Most
54.6
65.4
65.7
69.1
57.9
Few
91.7
92.5
87.4
89.3
Few
44.4
43.2
57.6
55.9
46.4
Overall
62.3
62.3
69.6
69.9
53.5
Table The test accuracy values for the Flan-T5-large model across various generalised quantifiers breakdown based on the negations in the sentence. The 8, n denote subject and predicate nominative, 1,0 denotes having or not having a negation 8, n. For example n denote no negation at subject and negation predicate nominative.
Table 5: The test accuracy values for the ChatGPT, Flan-T5-xx1 and LLaMA-30B model in zero shot set- tings, st denotes standard-prompting approach while ch denotes the thaingtf-thought prompting approach.
scenario. Second, the prior cognitive research on quantifiers has demonstrated that quantifiers with a downward monotone, such as "Few" or "Less than K" present processing challenges for humans compared those with upward monotone, such as "Most" and "More than K" (Geurts and van der Slik, 2005; Zeijlstra, 2020; Agmon et al., 2019). Since TLMs are trained on human- -ennrrtte data, highly likely that these models have incorpo- rated this cognitive trait into their understanding of language, which, in turn, affects their responses. Morcover, a deeper analysis of the answers gener- ated through chain-of- offthoughtt prompting revealed that even when the predicted label correct, the overall answer often incoherent. This coherence deficit in TLMs, coupled with their difficulties handling certain quantifiers, suggests that these models are yet to achieve proficiency learning even the simplest inferential rules.
the LLaMA model to its training process, which, unlike the other two models, is not trained to fol- low instructions. When subjected the chain-of- thought prompting approach, the LLaMA model displayed more consistency, generating either True or False label. Thus, we infer that the in- clusion of examples assists the LLaMA model in generating more concise outputs.
Accuracy values for TLMs in zero-shot set- tings vary drastically with different quantifiers. As depicted Table 5, TLMs struggle with numer- ical quantifiers whose cardinality of intersection has an upper bound, such as "At most K" and "Less than K". This diminished performance can be at- tributed primarily to the TLMs' tendency to predict the label False more frequently in sentences incor- porating these specific quantifiers. We hypothesise this phenomenon due two factors. First, the background knowledge already embedded in TLMs from pretraining. As indicated previously, the sen- tences containing the above quantifiers coupled with a low K value are often False real-world
Conclusion
We investigated how generalised quantifiers affect the behaviour of transformer based language mod-

---

els by employing the problem of model-checking. We found that different generalised quantifiers af- fect TLMs in varying ways when solving model- checking problems in both fine-tuned and zero-shot settings. Based on empirical findings on generalisa- tion, we posited TLMs can learn to understand the logical semantics of generalised quantifiers. More- over, our experimental setup in the zero-shot setting demonstrated that a multitude of factors, such as the training process, size of the models and type of prompts, can affect the ability of TLMs solve a simple entailment task. Thus, a compelling av- enue for future research is to probe how varying factors affect transformer-based language models when solving a more complex entailment task, like determining satisfiability.
Dominican Republic. Association for Computational Linguistics.
Yejin Bang. Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt reasoning. hallucination, and interactivity. arXiv preprint 2302::2300.0022.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large amno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632 642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
7 Limitations
Due to the empirical nature of this study, it suffers from an inductive dilemma on three fronts. One, in the front of transformers, the second related to generalised quantifiers and the third in relation to prompts we explored in zero-shot settings. We explored several transformer-basee language mod- els that are in line with prior literature and probe how different generalised quantifiers affect their be- haviour. Nonetheless, due to the empirical nature of this investigation, is plausible that some TLM architectures could deviate from the behavioural norms discussed in this paper when interacting with generalised quantifiers. A similar limitation applies to the range of generalised quantifiers examined, the ones employed in our study do not represent the entire spectrum of generalised quantifiers. In zero-shot settings, this limitation further extends to the prompt templates we employed. We consider two types of prompt templates, but there is multi- tude of alternative ways prompts can be formulated by using the [M, s, E) triplets.
Lars Buijtelaar and Sandro Pezzelle. 2023, A psycholin- guistic analysis of BERT's representations of com- pounds. In Proceedings of the 17th Conference of the European Chapter of the Association for Compu- national Linguistics, pages 2230-2241, Dubrovnik, Croatia. Association for Computational Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sehastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint 220.:2204..23111.
Hyung Won Chung. Le Hou, Shayne Longpre. Bar- ret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-fieetueed language models. arXiv preprint arXiv:2210.1 11416.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2021. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, 20.
References
Ruixiang Cui, Daniel Hershcovich, and Anders Søgaard. 2022. Generalized quantifiers as a source of error in multilingual NLU benchmarks. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4875-4893, Seattle, United States. Association for Computational Linguistics.
Galit Agmon, Yonatan Loewenstein, and Yosef Grodzin- sky. 2019. Measuring the cognitive cost of down- ward monotonicity by controlling for negative polar- ity. Glossa: a journal of general linguistics, 4(1).
Marianna Apidianaki and Aina Garí Soler. 2021, ALL dolphins are intelligent and SOME are friendly: Prob- ing BERT for nouns' semantic properties and their prototypicality. In Proceedings of the Fourth Black- boxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 79-94, Punta Cana,
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint

---

Allyson Ettinger. 2020. What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models. Transactions of the Association for Computational Linguistics, 8:34-48.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
Gottlob Frege. 1882. Über die wissenschaftliche berech- tigung einer begriffsschrift.
Stanley Peters and Dag Westerstähl. 2006. Quantifiers in language and logic. OUP Oxford.
Fuhrken. 1970. Per lindström. first order predicate logic with generalized quantifiers. theoria, vol. 32 (1966), pp. 186-195. The Journal of Symbolic Logic, 34(4):650-650.
lan Hart-Hartmann. 2004. Fragments of language. Journal of Logic, Language and Information, 13(2):207-223.
D. Gabbay, F Guenthner, and Dag Westerståhl. 1989. Quantifiers in formal and natural languages. Springer.
Ian
and Allan Third. 2006. More frag- ments of language. Notre Dame Journal of Formal Logic, 47(2):151-177.
Bart Geurts and Frans van der Slik. 2005. Mono- tonicity and Processing Load. Journal of Semantics, 22(1):97-177.
Colin Raffel, Noam Shazeer. Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2019, Exploring the limits of transfer learning with unified text-to-text trans- former. arXiv preprint arXiv:19100 10683.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with gradient- ing. arxiv preprint arXiv:2111.0954:
embedding shar-
Kyle Richardson and Ashish Sabharwal. 2021. Pushing the limits of rule reasoning in transformers through natural language satisfiability. In AAAI Conference on Artificial Intelligence.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 3651-3657, Florence, Italy, Association for Computational Linguistics.
Viktor Schlegel. Kamen Pavlov, and Ian 2022. Can transformers reason in fragments of natu- ral language? In Proceedings of the 2022 Conference on Empirical Methods Natural Language Process- ing, pages 11184-1111 11199, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.0883
Robin Smith et al. 1989. Aristotle's 's Prior Analytics. Hackett Publishing.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Asso- ciation for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICIR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
Jakub Szymanik, Shane Sleinert-Threlkel Marein Za- jenkowski, and Thomas F Icard III. 2013. Automata and complexity in multiple-quantifer sentence verifi- cation. Logic and Interactive RAtionality Yearbook 2012, page 133.
Tharindu Madusanka, Riza Batista-navarro, and Ian Pratt-hartmann 2023. Identifying the limits of trans- formers when performing model-checking with natu- ral language. In Proceedings of the 17th Conference of the European Chapter of the A Association for Com- putational Linguistics, pages 3539-3550 Dubrovnik, Croatia. Association for Computational Linguistics.
Jakub Szymanik and Marcin Zajenkowski. 2010a. Com- prehension of simple quantifiers: Empirical evalua- tion of a computational model. Cognitive Science, 34(3):521-532
Jakub Szymanik and Marcin Zajenkowski. 2010b. Quantifiers and working memory. In Logic, Lan- guage and Meaning: 17th Amsterdan Colloquium, Amsterdam, The Netherlands, December 16-18, 2009, Revised Selected Papers, pages 456-464. Springer.
Corey T McMillan, Robin Clark, Peachie Moore, Chris- tian Devita, and Murray Grossman. 2005. Neural basis for generalized quantifier comprehension. Neu- ropsychologia, 43(12):1729-19) 1737.
Jakub Szymanik et al. 2016. Quantifiers and cognition: Logical and computational perspectives, volume 96. Springer.
Andrzej Mostowski. 1957. On a generalization of quan- tifiers. Fundamenta mathematicue, 44(2).

---

Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Find- ings of the Association for Computational Linguis- lics: ACL-IJCNLP 2021, pages 3621 1621,, Online. Association for Computational Linguistics.
Hedde Zeijlstra. 2020. 426Negative Quantifiers. The Oxford Handbook of Negation. Oxford University Press.
A Appendix: Templates
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
A.I Sentence Templates
When constructing sentences, mentioned in the methodology section, employ sentence tem- plates. Let Q be a generalised quantifier, and Tq be the sentence template for the corresponding quanti- fier Q. Then TQ take the general form,
Vanessa Troiani, Jonathan E Peelle, Robin Clark, and Murray Grossman. 2009. Is it logical count quantifiers? dissociable neural networks underlying numerical and logical quantifiers. Neuropsychologia, 47(1):104-111. 111.
(non-) As (not) Bs
The inclusion of "non/not" is determined by the availability the negations and A, B are ordinary nouns. Consider the quantifier "At most K" for some natural number K. Then the correspond- ing sentence template takes the form "At most K (non-) As are (not) Bs", Table 6 depicts sentence templates corresponding quantifiers considered in this study.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. Advances in neural information processnnnn systems, 30.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, stick-
and Samuel Bowman. 2019. Superglue: ier benchmark for general-purpose language under- standing systems. Advances in neural information processing systems, 32.
A.2 Prompt templates for Zero-shot settings
We employ the tuples (M, to delineate prompts for the language modelling objective, providing framework for evaluating the effee- tiveness of TLMs in zero-shot settings. As mentioned in the methodology section, we ex- plored two types prompts. One, we informally called standard prompts and the other is based on chain-of- oh-huuhh-t prompting. The standard prompt- ing is conceptualised by the following template, Q: Given the following scenario, M. Is the sentence True or False according the scenario?
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022a. Finetuned language models are zero-shot learners. In International Con- ference Learning Representations.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022b. Chain thought prompting elicits reasoning in large language models. arXiv preprini arXiv:2201.1190 1900.
wttouggh-promptingg
Dag Westerstähl. 1987. Branching generalized quanti- fiers and natural language. Generalized quantifiers: Linguistic and logical approaches, pages 269-298.
A: The chain-of-thouuhtt prompting employs an ex- ample problem instance with an explanation of the thought process, thereby facilitating a more pre- cise response from TLMs. If we let (Me, Se, Le) represent this example problem instance and F elu- cidate the thought process, the chain of thought prompting can then be defined using the template, Q: Given the following scenario, Mc. the sentence se True or False according to the
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A proad-coverage challenge corpus for sen- tence understanding through inference. Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
scenario?
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2019, Huggingface's 's transformers: State-of- the-art natural language processing. arXiv preprint 1rXiv=1910.03771.
A: Pe- E
Q: Given the following scenario, M. Is the sentence True or False according to the scenario?
A:

---

GQ
Sentece Template
All
All (non-) As are (not) Bs
Some
Some (non-) ) As are (not) Bs
At least
At least K (non-) As are (not) Bs
At most K
At most K (non-) As are (not) Bs
Less than K
Less than K (non-) As are (not) Bs
More than K
More than K (non-) As are (not) Bs
K
(non-) As are (not) Bs
Most
Most (non-) As are (not) Bs
Few
Few (non-) As are (not) Bs
Table 6: The generalised quantifiers (GQ) used in experimental setup, along with their sentence templates
GQ
maximum
minimum
mean
depicted in Table 7. To evaluate TLMs' behaviour with boolean conjunctions, we also constructed separate train and test sets with 72K and 36K data points.
All
161
47
90.1 1
Some
165
48
89.7
least K
160
47
92.4
The dataset contains an equal number of problem instances for each conjunction, and quan- to
most K
162
48
92.2
Less than K
163
49
92.4
tifier pair. Morcover, since the intention compare the effects of generalised quantifiers on transformer- based language models, so we decided to use the simplest form of language templates, i.e. sy syllogistic.
More than K
168
49
92.1
K
164
46
89.9
Most
162
45
89.8
Few
158
47
89.6
We also emphasise the rationality behind the iter- ative approach we used in constructing the data. An alternative way of constructing problem instances derive the label Ê using the model-checker instead of iteratively creating structures to match a pre-defined label and a sentence. However, this alternative approach can induce easily exploitable patterns. Consider the quantifier "K", "All" and "Some". For random structure, quantifiers "K" and "All" are more likely F False, while the quanti- fier "Some" is probabilistically Truc.
Table 7: minimum. maximum and mean number of words (tokens) in problem instances (M + s) when seperated by SPACE
B Appendix: Dataset and Training Details
B.1 Dataset details
We utilised nine quantifiers when constructing sen- tences. We construct train and test sets with 72K and 36K unique data points with 8K and 4K data points for each quantifier for fine-tuning and evalu- ating TLMs. As the vocabulary, We employed the Richardson and Sabharwal (2021) vocabulary of nouns, which contains list of professions (e.g.. "artist", "doctor"), that we extended by adding more professions. The dataset contains an equal number of Truc and False problem instances for each generalised quantifier. When construct- ing the dataset for fine-tuning TLMs, we select idmiin dmas] [8, 14] and mp _y [5, 10). For sentences with numerical quantifiers, we se- lect K values randomly from the range [1, ID]. where [D] denotes the number of domain elements selected when formulating the problem instance. The minimum, maximum and mean number of to- kens for problem instances of each quantifier is
B.2 Fine-tuning Details
Formally, we define the task as binary classifica- tion problem where the objective of the transformer- based language model (True or False) given the natural language in- terpretation of the structure M and the sentence 8 as the inputs. We select and fine-tune three TLMs, namely T5, Flan-T5, and DeBERTa-v3, all of which have previously demonstrated their efficiency and reliability in resolving textual en- tailment tasks. According to prior literature, the performance of TLMS mostly depends on the pre- trained data, and size of the models rather than the architectural choice (Raffel et al., 2019; Ka- plan et al., 2020). Morcover, the accuracy values yielded for all TLMs are similar, Thus, we expect
prediet the label &

---

similar behaviour for other TLM architectures as well. Since the TLMs achieve satisfactory accuracy and since the central research interest is to analyse the behaviour TLMs rather than identifying the best-performing TLM, we do not perform any hy- perparameter tuning. Morcover, exploring several different TLMs and performing hyperparameter tuning leaves a higher carbon footprint (Strubell et al., 2019). Due the nature of the research ques- tion, we consider such an exploration unnecessary.
Loss function and optimizer We fine-tune each TLM to predict the label & given the (M, 8) by reducing the binary cross entropy loss over the target using the ADAM (Kingma and Ba, 2015) optimizer.
Batch size Utilising gradient checkpointing for memory-efficient fine-tuning, we set the batch size to 36.
Number of epochs We fine-tune each TLM for 4 epochs, resulting in 8000 steps.
set the maximum maximum token length
token length to 512 since the maximum problem length is much lower than that, thus, we do not truncate the inputs.
learning rate We set the learninig rate of 1 x 10-5
We utilise Huggingface (Wolf et al., 2019) im- plementation when experimenting with the TLM models we consider this study.


Reviews:
[{"review_id": "Vctd7UsJf5", "reviewer": "Reviewer_sJL9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\n\nRegarding the comment on reproducibility: *“This work fails to meet the Reproducibility Criteria of the EMNLP call: The absence of dataset and code availability hinders the ability of the research community to verify and reproduce the findings, undermining the credibility of the results.”*\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; We are more than happy to share the code and it was our intention to add the link to the code and data in the camera-ready version. \n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Regarding data generation, it is worth noting that the specifications of the data sets, and indeed the algorithms to realise them---which are relatively straightforward---were given in full detail.\n\n*“The lack of hyper-parameter tuning and arbitrary parameter selection introduces biases and unfairness in comparing models and settings, rendering the experimental setup unreliable.”*\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; We did not perform hyperparameter tuning of any of the transformer-based language models (TLMs) for the following reasons which were mentioned in Appendix B.2:\n1. All TLMs when fine-tuned achieved adequate performance and did not exhibit much variation among themselves (variation in average accuracy between transformer models is less than 1.5%) in terms of their performance. \n2. The primary point of this research is not to identify which specific TLM achieves the best performance but rather to identify how TLMs behave as a class of models. Thus, we believe that performing hyperparameter tuning is not necessary. \n3. Moreover, exploring several different TLMs and performing hyperparameter tuning leaves a higher carbon footprint. \n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Given the above, we decided to use the TLMs following their default parameter setup. \n\nRegarding the comment on data construction: *“The need for dataset construction details, such as the ambiguous generation of noun vocabularies and missing label distribution information, raises concerns about data quality and potential biases in the evaluation”*\n\n1. Noun vocabulary -  We use the Richardson and Sabharwal (2021) vocabulary of nouns, which contains a list of professions (e.g., \"artist\", \"doctor\"), that we extended by adding more professions. \n2. Label distribution - As mentioned in line 861 in Appendix B.1, the dataset contains an equal number of True and False problem instances for each quantifier. \n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Other details such as the number of domain elements, the number of predicates, the K value range we explored, and the number of data points generated for each quantifier are mentioned in Appendix B.1. \n\n*\"The limited number of templates used for sentence generation hampers linguistic variability and could impact the results' validity and generalizability.\"*\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; The intention was to compare the effects of generalised quantifiers on transformer-based language models, so we decided to use the simplest form of language templates, i.e. syllogistic. This in line with prior work of research when analysing the effect of different language properties on transformers using synthetic data (Richardson et al. (2020), Richardson and Sabharwal (2021), Madusanka et al. (2023)). Moreover, the constructed sentences have a close resemblance to data used when performing quantifier verification on human beings. \n\nReference\n\n1. *Richardson, K. and Sabharwal, A., 2022, June. Pushing the limits of rule reasoning in transformers through natural language satisfiability. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 10, pp. 11209-11219).*\n\n2. *Richardson, K., Hu, H., Moss, L. and Sabharwal, A., 2020, April. Probing natural language inference models through semantic fragments. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 05, pp. 8713-8721).*\n\n3. *Madusanka, T., Batista-navarro, R. and Pratt-Hartmann, I., 2023, May. Identifying the limits of transformers when performing model-checking with natural language. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (pp. 3521-3532).*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents an extensive and well-written methodology section. The detailed explanation of the research approach allows for a clear understanding of the experimental setup. Furthermore, experiments with state-of-the-art models, such as T5, FlanT5, DeBERTa, ChatGPT, and LLaMA, provide comprehensive insights into how different transformer-based language models are affected by different quantifiers. Incorporating multiple models enhances the study's reliability and generalizability, allowing for robust comparisons and insightful conclusions.", "reasons_to_reject": "This work fails to meet the Reproducibility Criteria of the EMNLP call:\n- The absence of dataset and code availability hinders the ability of the research community to verify and reproduce the findings, undermining the credibility of the results.\n- The lack of hyper-parameter tuning and arbitrary parameter selection introduces biases and unfairness in comparing models and settings, rendering the experimental setup unreliable. \n- The need for dataset construction details, such as the ambiguous generation of noun vocabularies and missing label distribution information, raises concerns about data quality and potential biases in the evaluation.\n\nThe limited number of templates used for sentence generation hampers linguistic variability and could impact the results' validity and generalizability.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper delves into the impact of different quantifiers on Transformer-based Language Models (TLMs) through the utilization of prompting strategies. The conducted experiments specifically address how the choice of models and experimental setup influences the performance in assessing TLMs' ability to comprehend generalized quantifiers. By exploring these factors, the study sheds light on the effectiveness of TLMs in dealing with linguistic challenges related to quantifiers.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "n1gEzCMIly", "reviewer": "Reviewer_SHqe", "summary": "", "questions": "No questions for the authors.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper considers and important, but little-researched topic, and demonstrates (i) a sound approach to the problems it considers; (ii) interesting results and (iii) convincing analyses -- especially the results shown in Tables 4 and 5 and their analyses.\n", "reasons_to_reject": "I don't see any reason to reject the paper.\n", "questions_to_authors": "No questions for the authors.", "paper_topic_and_main_contributions": "The work aims to assess the ability of transformer-based language models (TLMs) to learn the meanings of generalised quantifiers using a logically-defined textual entailment problem (i.e., one that attempts to avoid the influence of background knowledge and common sense). The form of textual entailment used is model-checking of natural language statements, and the evaluation occurs prior to fine-tuning, so that any differences found are intrinsic to the models. Since the definitions of some generalized quantifiers involve negation and conjunction, the work contributes towards our knowledge of whether and how TLSs can deal with negation. The work is assessed by comparison with human understanding of quatifiers, and by using chain-of-thought prompting in a zero-shot setting.  The workd considers logical quantifiers, numerical quantifiers and propositional quantifiers.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I did not notice any missing references", "Typos_Grammar_Style_and_Presentation_Improvements": "No suggestions for improving the presentation."}, "extra_text": {}}, {"review_id": "9rYPlrzxYg", "reviewer": "Reviewer_bRRx", "summary": "", "questions": "- Which k (or multiple k) was used for \"at least k\" et al?  And was there an effect of accuracy on this k?\n- Line 514-516: \"We attribute this behaviour to the inherent probabilistic aspect of the predictions formulated by TLMs.\" Can you say more here? \n- For fine-tuning: it wasn't clear to me whether you did \"instruction-tuning\", i.e. continued training as a language mode., but to predict the word True/False, or whether you trained a new classifier head for the labels.  Which one?\n- Do you think factors like complexity or monotonicity may effect LM performance?  I'm particularly struck by \"at most k\" and \"less than k\" being harder than their \"upward\" counterparts; see references below for evidence that these are also harder for people.  Or are these ore likely to be artifacts in the model case?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\n\n*“The analysis could be more thorough, in particular focusing on differences between different quantifiers.”*\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; We have had to prioritise the discussion points in our analysis due to page constraints. \n\n*“Some details about the experiments (values of k, exact nature of fine-tuning and prompting setup) are either missing or hard to find (in Appendices).”*\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; The value of K is expressed in line 861 in Appendix B.1. The K value is randomly sampled to be an integer between 1 and D where D is the number of domain elements in the problem instance. This is true for both fine-tuning and prompting setups. As explained in line 403 in the dataset and evaluation section, the problem instances for the prompting setup have fewer domain elements and predicates. Then, the final prompt is constructed using the template introduced in Appendix A.1. \n\n*“Some of the framing is a bit hard to follow: most of the introduction talks about the task of inference, but then the actual task in the paper is verification (\"model-checking\"). I'm a fan of using the model-checking task, but it took awhile to figure out whether they were going to be using an inference format or not in the experiments, so I think some re-wording at the beginning would be welcome.”*\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; We attempted to make clear that the task is identifying whether a sentence is True or False according to some paragraph in natural language (logical structure in natural language). To perform the said problem the transformer model needs to comprehend logical semantics of natural language and perform very simplistic reasoning operations. Although this can, formally, presumably be characterised as an entailment problem, we accept that it would have been better to use the term “model-checking” more prominently. In the revised version, we will reword at the beginning to make it more clear.\n\n*“Which k (or multiple k) was used for \"at least k\" et al? And was there an effect of accuracy on this k?”*\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; As mentioned above, the K value is selected randomly to be an integer between 1 and D where D is the number of domain elements in the problem instance. We did not explicitly test the effect of K values since the range of K values is not that high. Also, since a higher K value more often means higher domain elements in the problem instance, the problems with higher K values would be longer in length. Therefore, it would be difficult to decouple the effect of K values from the effect of the length of the text. \n\n*“Line 514-516: \"We attribute this behaviour to the inherent probabilistic aspect of the predictions formulated by TLMs.\" Can you say more here?”*\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; We meant that the language models are trained to find the most probable next word given a set of words. This probabilistic nature of language models can lead to inaccurate predictions, especially in a logical context. We would include the clarification in the revised version of the paper. \n\n*“For fine-tuning: it wasn't clear to me whether you did \"instruction-tuning\", i.e. continued training as a language model but to predict the word True/False, or whether you trained a new classifier head for the labels. Which one?”*\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; We trained the language models to predict True or False with a sequence classification head; we did not perform instruction fine-tuning. \n\n*“Do you think factors like complexity or monotonicity may effect LM performance? I'm particularly struck by \"at most k\" and \"less than k\" being harder than their \"upward\" counterparts; see references below for evidence that these are also harder for people. Or are these ore likely to be artifacts in the model case?”*\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; The difference in performance is evident in the zero-shot setting, and not so much in the fine-tuned version. Considering language models are trained on language corpora produced by humans, it is very likely that the pre-trained models inherit difficulties that humans possess when processing different quantifiers. \n\nRegarding comments on reproducibility: we are happy to share the code and data and intend to share it in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Uses logical semantics of natural language to test how well language models learn the meaning.  This is a very nice case study for these questions.\n- Compares multiple prompting methods, and also does some fine-tuning experiments.\n- The use of quantifiers enables careful control over the data-generation process (for model-checking), enabling balance among True/False examples, among other factors.\n- Uses open-source LMs, in addition to API-only models.", "reasons_to_reject": "- The analysis could be more thorough, in particular focusing on differences between different quantifiers.\n- Some details about the experiments (values of k, exact nature of fine-tuning and prompting setup) are either missing or hard to find (in Appendices).\n- Some of the framing is a bit hard to follow: most of the introduction talks about the task of inference, but then the actual task in the paper is verification (\"model-checking\").  I'm a fan of using the model-checking task, but it took awhile to figure out whether they were going to be using an inference format or not in the experiments, so I think some re-wording at the beginning would be welcome.", "questions_to_authors": "- Which k (or multiple k) was used for \"at least k\" et al?  And was there an effect of accuracy on this k?\n- Line 514-516: \"We attribute this behaviour to the inherent probabilistic aspect of the predictions formulated by TLMs.\" Can you say more here? \n- For fine-tuning: it wasn't clear to me whether you did \"instruction-tuning\", i.e. continued training as a language mode., but to predict the word True/False, or whether you trained a new classifier head for the labels.  Which one?\n- Do you think factors like complexity or monotonicity may effect LM performance?  I'm particularly struck by \"at most k\" and \"less than k\" being harder than their \"upward\" counterparts; see references below for evidence that these are also harder for people.  Or are these ore likely to be artifacts in the model case?", "paper_topic_and_main_contributions": "This paper studies the ability of language models to understand/learn the meanings of generalized quantifiers: the meanings of expressions like \"All\", \"most\", \"at least n\", and so on.  These expressions have been central in the study of natural language semantics for decades, since they have a precise but quite abstract semantics.  The paper uses both prompting in the zero-shot setting and fine-tuning to evaluate several different language models' abilities.\n\nIn general, they find some success, especially in the fine-tuning scenarios.  In the zero-shot scenarios, there are some differences between prompts (\"standard\" versus chain-of-thought) and also an effect of scale (among other factors).  \n\nThese results are not uninteresting, and it is very good to probe language models for their understanding of so-called \"logical semantics\", since this is a form of meaning in natural language that is very precise and may be learnable from text.  There are some areas where I think the paper could benefit from a bit of re-framing.  And I also would have appreciated some analyses that focus on the differences between quantifiers more, as opposed to just differences between models.  In other words: are there facts about the quantifiers that also explain the differences?\n\nREBUTTAL EDIT: I thank the authors for their rebuttal and have upgraded my reproducibility score in response, but am leaving my other scores unchanged.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "While I do value the use of primary source literature, there are two excellent surveys that could be cited:\n- For the logical semantics of quantifiers, Peters and Westerstahl 2006 _Quantifiers in Language and Logic_.\n- For the cognitive aspects, Szymanik 2016 _Quantifiers and Cognition_\n\nSome possible inspirations for analyzing differences between quantifiers:\n- For some properties of quantifiers (monotonicity, etc) and other measures of complexity that may be relevant: van de Pol et al 2023 and some references therein: https://www.sciencedirect.com/science/article/pii/S001002772200138X\n- For the difficulty of negative monotone compared to upward monotone quantifiers: Agnon et al 2019 https://doi.org/10.5334/gjgl.770 , Guerts 2003 https://www.sciencedirect.com/science/article/pii/S0010027702001804 , Guerts 2005 https://academic.oup.com/jos/article-abstract/22/1/97/1686377?redirectedFrom=fulltext"}}]

Author rebuttal:

Meta-review: The paper presents a probing study to explore the ability of LLMs to understand various types of generalized quantifiers. Reviewers and I agree about the methodological soundness of the work, in which state-of-the-art models, such as T5, FlanT5, DeBERTa, ChatGPT, and LLaMA, are tested. Moreover, the paper addresses a relatively little-studied topic in nowadays NLP and represents an interesting example of using formal semantics to explore the inferential competences of LLM.
Decision: Accept-Main