Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models
Authors: Yiyuan Li
Rakesh R Menon
Sayan Ghosh
Shashank Srivastava
Venue: EMNLP 2023 Main (2023)
Keywords: Pragmatic Reasoning
Rational Speech Act
Quantifier Understanding
Generalized Quantifiers
TL;DR: We crowdsourced dataset for understanding quantifier semantics and propose a pragmatic-based quantifier understanding framework that outperforms literal interpretation of quantifier semantics.

Abstract: Generalized quantifiers (e.g., $\textit{few}$, $\textit{most}$) are used to indicate the proportions predicates satisfy (for example, $\textit{some}$ apples are red). One way to interpret quantifier semantics is to explicitly bind these satisfactions with percentage scopes (e.g., 30%-40% of apples are red). This approach can be helpful for tasks like logic formalization and surface-form quantitative reasoning (Gordon and Schubert, 2010; Roy et al., 2015). However, it remains unclear if recent foundation models (Bommasani et al., 2021) possess this ability due to the absence of direct training signals. To explore this, we introduce QuRe, a crowd-sourced dataset of human-annotated generalized quantifiers in Wikipedia sentences featuring percentage-equipped predicates. We explore quantifier comprehension using PRESQUE, a framework that combines natural language inference and the Rational Speech Acts framework. Experimental results on the HVD dataset (Herbelot and Vecchi, 2015)  and QuRe demonstrate PRESQUE's superiority over a literal listener baseline, showing a 20% relative improvement in F1 in predicting percentage scopes for quantifiers, even with no additional training.

Pragmatic Reasoning Unlocks Quantifier Semanties for Foundation Models
Yiyuan Li
Rakesh R. Menon
Sayan Ghosh
Shashank Srivastava
UNC Chapel Hill
Cyiyuanli, rrmenon, sayghosh, ssrivastava)@cs. unc.
Abstract
In this work we present PRESQUE (Pragmatic REasoning for Semantics of QUantiffErs), frame- work to model the semantics of quantifiers for text- based foundation models, such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), through the lens of pragmatic reasoning. While foundation models have shown impressive perfor- mance various text-based tasks (Bommasani et al., 2021: Wei al., 2022), their ability infer the semantic meanings of generalized quantifiers remains relatively unexplored.
Generalized quantifiers (e.g., few, most) are used to indicate the proportions predicates are satisfied (for example, some apples are red). One way to interpret quantifier semantics is explicitly bind these satisfactions with per- centage scopes (e.g., 30%-40% of apples red). This approach can be helpful for tasks like logic formalization and surface-fome quan- titative reasoning (Gordon and Schubert, 2010; Roy et al., 2015). However, remains un- clear if recent foundation models possess this ability, as they lack direct training signals. To explore this, we introduce QuRe, crowd- sourced dataset of human-annotated general- ized quantifiers in Wikipedia sentences featur- ing seccetaaee-qqipper predicates. We ex- plore quantifier comprehension in language models using PRESQUE, a framework that com- bines natural language inference and the Ra- tional Speech Acts framework. Experimental results on HVD dataset and QuRe illustrate that PRESQUE, employing pragmatic reasoning. performs 20% better than literal reasoning baseline when predicting quantifier percentage scopes, with additional training required'.
In PRESQUE, we represent quantifier semantics in terms of percentage scopes, which indicate the proportion of cases where the associated predicate holds true, For example, in the sentence 'some ap- ples are red', the quantifier 'some' could be associ- ated with a percentage scope of 30-40%, indicating that 30-40% of all apples are red. Our framework consists of two components: (1) a natural language inference (NLI) component (Bowman al., 2015) that models sentence-level semantics between a sentence containing quantifier word and another sentence containing a percentage value, and (2) a rational speech act (RSA) component (Frank and Goodman. 2012) for pragmatic reasoning. Using these components, PRESQUE takes sentence with quantifier input and outputs the corresponding percentage scope (further discussed in Section 2).
1 Introduction
Generalized quantifiers (Mostowski, 1957) are used to express relations between subsets of concepts or entities. For instance, the quantifier 'some" in the statement 'some apples are red' indicates that at least one apple is red. Quantifiers, being in- herently fuzzy, are prevalent in both real-world communication and natural language processing (NLP) benchmarks (Joshi et al., 2020). Conse- quently, developing a formal framework for un- derstanding quantifier semantics is essential to en- hance the language understanding capabilities of NLP systems, particularly in facilitating natural Auman-Al language- based interactions, such as in huban-robot collaborative tasks (Alami, 2013).
Ambiguity, as highlighted by Piantadosi et al. (2012), beneficial for efficient communication via language. Since the percentage values of quan- tifiers are not universally defined, humans often need to infer the exact percentage value, which is not explicitly conveyed in the utterance (Horowitz et al., 2018). Furthermore, the interpretation of quantifier semantics can be influenced by linguistic and social cues (Bergen et al., 2016). The prag- matic theory proposed by Grice (1975) emphasizes the role of communicative goals in interpreting the semantic meaning natural language expressions, simplifying the required semantic theories (Bergen
Code: httpe//u...oo/aattatttt/////.//.

---

al., 2016). Lastly, NLI models are also shown to struggle with ambiguous premises (Thukral al.. 2021), quantifier inference (Richardson et al., 2020; Joshi et al., 2020), and quantitative reasoning (Naik et al., 2018; Ravichander et al., 2019), making a di- rect literal interpretation of generalized quantifiers less reliable.
chies from previous works (Solt, 2016; Srivastava et al., 2018) that involve strong human priors, and findings from Pezzelle et al. (2018a), who asso- ciates quantifier usage with the counting behavior of human beings.
In summary, contributions are two-fold: we develop PRESQUE based on pragmatic reasoning and NLI, and crowd-source dataset QuRe support quantifier understanding investigation of foundation models. Our results on HVD and QuRe demonstrate that foundation models equipped with pragmatic reasoning (PRESQUE) can perform quan- tifier reasoning similar to humans.
To address these challenges, PRESQUE employs RSA, Bayesian framework that follows a Gricean approach for modeling communication by reason- ing about agent states (Grice, 1975). PRESQUE in- corporates a literal speaker role, based on a foun- dation model fine-tuned on NLI datascts, and a pragmatic listener role, computed using Bayesian rules, to reason between the quantifier space and the space of percentage values.
2 Quantifier Semanties Understanding through RSA
Existing datasets like HVD (Herbelot and Vec- chi, 2015) and GQNLI (Cui et al., 2022) that inves- tigate quantifier semantics either lack gold annota- tions of the percentage scopes for interpreting quan- tifier semantics, or are based on artificial setups us- ing a small number of countable objects (Pezzelle et al., 201 2018b). Such fictional settings are not gener- alizable to broader and more complex real-world settings (e.g. describing concepts about a popula- tion using quantifiers). For a fair evaluation of the quantifier understanding capabilities acquired by foundation models through their pre-training, we should evaluate these models using text of simi- lar style and content as the pre-training corpora. To address the aforementioned issues with cur- rent evaluation corpora for quantifier understand- ing, we crowd-source a dataset, QuRe (Quantifier Reasoning), which contains sentences containing quantifiers paired with annotations for quantifier semantics in terms of percentage scopes. Addition- ally, we characterize the ease of making quantifier predictions for different sentences in QuRe.
We frame the task of quantifier understanding as the prediction of the percentage scope (e.g.. 30%-50% given a quantified sentence Sgg (e.g.- Some apples are red.). Specifically, given an in- terval width B, we divide the percentage spec- trum between O and 1 into evenly spaced inter- vals, denoted as Wg [0, 3%, 10%, .... 95%, 100%)). The goal of a quantifier understanding model is to determine the percentage range in W3 where the associated pred- icate holds true (e.g.. the proportion of red apples among all apples, 30%-50%)
{pi} (e.g.. W3=0.05 =
To interpret quantifiers as percentage scopes, we develop PRESQUE, framework that adopts the ra- tional speech act (RSA) framework, with natural language inference (NLI) the backbone for text understanding. The RSA framework consists of speaker and listener, where the listener infers the world state from the speaker's s utterance by mod- eling the speaker's s state of mind (Goodman and Frank, 2016). In PRESQUE, the world state corre- sponds to percentage values of predicates, while utterances correspond to quantifiers used with those predicates.
Using PRESQUE to evaluate the quantifier reason- ing ability of foundation models on QuRe, we ob- serve 20% span-based Fl boost over the literal lis- tener baseline at all specificity levels (Section 5.2). Our experiments highlight the improved quanti- fier understanding of foundation models when ap- proached from a pragmatic perspective rather than relying direct interpretation using textual un- derstanding frameworks like NLI. Although our framework does not explicitly model mathematical concepts, is noteworthy that the mean strengths quantifiers in foundation models, as revealed by PRESQUE, echo observations of quantifier hierar-
Given a premise ğ (e.g., Some apples are red.) with quantifier g (some) and a hypothesis h (c.g.. 30% apples are red.) with a percentage value p (30%), we use the entailment score between the premise and hypothesis, obtained from an NLI model, to define the literal listener Lo:
Lo(plg) Entailment(f,
(1)
The pragmatic listener L1, in the PRESQUE framework, interprets the semantics of the quanti-

---

fier word as:
WIKIPEDIA SENTENCE
[SPECIFICITY, EXPRES- SION] QuRe SENTENCE
(p(gg) & So(glp)P(p)
(2)
Squirrel Hill North's popu- lation is 75% White, 17% Asian, 4% Hispanic, and 3% Black.
[Partial, 0.04] Squirrel
Hill North's population 75% White, 17% Asian, few Hispanic, and 3% Black.
Here, So represents literal speaker that maps the semantics of percentage values to quantifier words. Practically, we model the speaker by swap- ping the premise and hypothesis in To::
Coconut milk contains 5% 20% fat, while coconut cream contains around 20% to 50% fat..
[Indeterminable, 0.2 0.5] Coconut milk con- tains 5% 20% fat, while coconut cream contains moderate fat.
So(gpp) x Entailment(h, p)
(3)
The prior P(p) in Eq. 2 can be expanded as:
Table I: Examples of QuRe, with target percentage men- tion and the quantifier underlined. The headers of the QuRe also provide information about specificity and percentage expression generated. More examples are included in Appendix A.
P(p) = Σp(pla)P(O
QEU
Here, Ppl()) is computed similarly to Lo. and P(q) represents the word frequency of g, obtained from the WORDFREQ dataset (Speer, 2022).
The annotations in QuRe are obtained through a mix of crowd-sourcing (using Amazon Mechanical Turk) and assistance from gpt-3.5-turbo (OpenAI, 2022). We describe the annotation procedure in more detail below.
3 QuRe: Quantifier Reasoning Dataset
Existing datasets for quantifier understanding like HVD (Herbelot and Vecchi, 2015) comprised of triples of the form (concept, feature, quantifier) (e.g. (shrimp, is sy_white, some)) that denote how of- ten a 'feature' satisfied for a 'concept'. Notably, these datasets do not provide annotated percentage scopes that can be used to decipher the semantics of the quantifiers, i.e., how often (in numerical terms) the "feature' is satisfied for the 'concept' in the real world, and the supporting documents (e.g. knowledge-bases in Wikipedia any pub- licly available corpus) about the percentage scope of those triples are not easily accessible. There- fore, the judgments are based on subjective obser- vation and experience (e.g. the proportion of white shrimps.) and are hence inaccurate. To address this shortcoming in available resources for quantifier understanding, we contribute a dataset, QuRe, for evaluating the quantifier understanding capabilities of language models. QuRe consists of sentences (from Wikipedia) containing percentage mentions annotated with the quantifiers.
Wikipedia sentences collection We utilize the (concept, feature, quantifier) triples from the HVD dataset and convert them into sen- tences programmatically through templates (e.g. (shrimp. is s_wwhite,, some) -> 'some shrimps are white"). We then link these sentences to the most related Wikipedia entities² using off-the-shelf li- brary'. For example, the related Wikipedia entities for the running example would be /Shrimp, Prawn, Indian prawn, etc.). In practice, we find this set- ting links to more diverse entities than simply link- ing the concepts. We then use regular expres- sions to extract around 5,000 candidate sentences containing percentage values from the Wikipedia pages of these entities. For example, 'Among the prawn species entering the field F. indicus consti- tute around 36%.43%. is one such sentence from the Wikipedia page of the entity Indian prawn, with the percentage mention underlined.
Stage 1:
Table presents examples from QuRe. Of note, in addition to the quantifier annotation and percent- age scopes, for each example in QuRe, we also pro- vide specificity as additional metadata. Specificity measures the difficulty of deciphering the percent- age scope of the quantifier from the sentence ex- cluding the quantifier (i.e., if someone can deduce the percentage scope of a quantifier fully/partially from the sentence contents when the quantifier is absent; more details are provided later in Stage 4).
Stage 2: Sentence Filtering The candidate sen- tences are further filtered based on two criteria: (1) the length of the sentences should be between 10 and 40 tokens (space-tokenized)) and (2) the percentage mentioned in the sentence should not indicate a comparative change (e.g., 'increased by 20% " To identify whether the sentence describes
'aah Wikipedia entity title of Wikipedia article. /https://pyp... http://ppppppgogoe////p org/

---

comparative change, we used regular expressions. However, capturing all possible variations of de- scribing such comparative changes through regular expressions is cumbersome. Hence we employ GPT-3.5-turbo to annotate sentences that contain comparative changes. To validate the efficacy of GPr-b,5-turbo, we manually annotate held-out set of 50 sentences based on our aforementioned fil- tering criteria. On this held-out set GPT-3.5-turbh achieves 0.76 F1. More details on the annotation usage of GPT-3.5-turbo this stage are included in Appendix H. The filtered sentences are then paired up with all percentage mentions in the sentence and manually validated by the authors. Around half of the percentage mentions were deemed inap- propriate for the quantifier understanding task and removed. We include examples, metadata, and the instruction used in Appendix and J.
tics of the quantifier used in Sg is associated to the percentage scope in Sp- In the second task, we measure specificity, or the difficulty of specitying the target per- centage scope from Sq without the quantifier g (e.g. removing large amount from the previ- ous Sy). In our study, we discretize the speci- ficity values into three distinct levels of difficulty: fullpaartalinneeermnnaabi
Full means the tar- get percentage scope can be fully determined by other contents in the sentence, like One in ten for 10%; partial means the percentage scope can be narrowed but not determined by the contents (e.g. an incomplete percentage breakdown), and inde- terminable means there is no information in the content of the sentence determine the percent- age scope. This task aims gauge the extent of information that the context contributes to the deter- mination of the quantifier's percentage scope. For example, the specificity of the previous Sg about prawn would be indeterminable since the rest of the sentence after removing a large amount does not provide information to determine the percentage scope of large. But with additional contents like ... constitute large amount (around one-third)." the specificity level would become partially. More examples are included in Appendix A.
Stage 3: Percentage Expression Generation In many Wikipedia sentences, the percentage value is surrounded by texts like around, less than, more than, etc. denote a percentage scope rather than the individual percentage value or a percentage range. We use GPT-3.5-turbo to obtain those per- centage scopes and the instruction is included in Appendix I. The variations that we capture in this stage to obtain the percentage scopes are mentioned in Table 2.
We use majority voting (amongst three anno- tations) to choose the final annotated quantifier among all annotations for each sentence. The in- struction used, examples, and example annotation interface are included in Appendix M. The set of quantifiers to select from is l {all, generally, most, usually, some, likely, few, little, occasionally, none, seldom, tiny, small, moderate, large), which largely comes from Srivastava et al. (2018), and is slightly extended based on preliminary annotator feedback. We leave the choice of nouns that are attached to adjective quantifiers (e.g. amount in small amount), like small and large, in sentences to the annotators.
Or.
PERCENTAGE MENTION: EXPRESSION
None
89%: 0.89
over 93%: > 0.93
>=
least 45%: >= 0.45
less than 1%: 0.01
not exceeding 19%: <= 0.19
between 24% and 40%: 0.21 - 0.4
about 98%: ~ 0.98
Table 2: Operators (Op.) in percentage expression gen- eration and examples.
Statistics We have collected 744 5g sentences, of which 47% and 17% contain no and one per- centage mention respectively and others contain more than one percentage mention. The average sentence length is 26.3 tokens. Each sentence is annotated by annotators. The Fleiss' Kappa for quantifier choices and specificity are 0.37 and 0.80, meaning fair agreement in quantifier choices and substantial agreement in specificity levels. The distribution of quantifiers in QuRe is shown in Fig-
Stage 4: Quantifier and Specificity Annotation We design two human annotation tasks. The first task is rephrasing a sentence, Sp. with a target per- centage mention (c.g. 'around 36%-43% of t...the field indicus constitule around 36% in the previous example) to 3g with minimal semantic changes using a quantifier from 2 (e.g. Among the prawn species entering the field, F. indicus consti- tute a large amount.). This step ensures the seman-

---

26 3%
Quantifier
y = 0.01
g 0.01 w
15.5%
all
0.885 0.087
0.892 0.085
Proportion
12 12.0%
10 10.89
10 10.1%
20%
10.2
generally
0.730 1 0.205
0.708 1 0.212
10%
4.2%
4.0%
2.0%
1.9%
generally 1.6%
1.5%
usually
0.686
0.249
0.674 + 0.242
some
most
moderate
large
Few
small
tny
iitll
usually
all
others
most
0.687 0.193
0.693 0.195
large
0.624 0.217
0.628 0.223
likely
0.473 1 0.287
0.504 1 0.266
moderate
0.369 + 0.154
0.372 + 0.156
Figure Distribution of quantifiers in QuRe. Quan- tifiers with less than 1% frequencies (likely, seldom, occasionally, none) are merged into others. Some, most and moderate are the most frequent quantifiers QuRe.
some
0.225 0.185
0.218 0.182
small
0.183 0.184
0.172 0.172
occasionally
0.119 + 0.037
0.124 + 0.037
seldom
0.112 + 0.117
0.093 + 0.106
ure 1, where some is used in over 25% of sentences, followed by most, moderate, large and few. The most popular quantifiers for different percentage scopes are shown in Figure 2, where some is pre- ferred in over 30% of the cases with target percent- age values lower than 40%, and most is selected in over 40% of the cases with target percentage value greater than 60%. Overall, 17% of the sentences have target percentages fully specified, 32% par- tially specified, and 50% are indeterminable. We include examples across different specificity levels in Appendix A.
little
0.104 0.109
0.117 0.135
few
0.074 0.087
0.081 0.098
tiny
0.024 + 0.048
0.031 + 0.046
none
0.004 + 0.007
0.004 + 0.007
Table 3: Average strengths of quantifiers in all annota- tions of QuRe under different grounding configurations. The average strengths are stable with different window sizes.
stable among configurations, and show interest- ing hierarchies: all (0.88) higher than generally (0.73), and generally is higher than most (0.69). These patterns closely align with previous manual strength assignments like Srivastava al. (2018), and Testoni al. (2019)'s quantifier collection from multimodal simulations. It also echoes Solt (2016)'s finding that the strength of most higher than more than half.
most
80%-100%
large
60%--80%
large
Percentage Ranges
moderate
moderate
40%-60%
most large
4 Experimental setup
some
20%-400%
moderate
For the experiment in HVD, we compute L(pl4) for PRESQUE and Lo among different foundation models and compare them with human interpreta- tions. In QuRe, with the target percentage given in Section 3, we generate the percentage scope that S, satisfies. All percentage choices are selected from Wa. and experiments are run without training.
Jarge
some
0%--0%
small few
0.2
0.3
0.4
Quantifier Preference
Quantifier preferences in difference percent- age scopes, e.g., most is chosen around 35% of the times if percentage mentioned lies 60%%-1000 100%.
Figure 2:
Percentage Scope Generation With specific granularity g and window size for the operators, percentage expression in Section 3 is converted into golden scope {Pmin: Pmax } E W3 (Pmin Pmax). For example, if f 0.05, 9 0.01 and w 2, the golden scope of 0.59 [0.55, 0.65]. The full generation rules are in Appendix
We also include the average strength of quanti- fiers under different grounding configurations in Table 3.4 We that the mean strengths
For the definition of and u. please refer to Appendix E.

---

0.1
Scope
No
Some
Most
All
1,0
1.0
18
0.8
0.2
0.3
0.8
000000
0.6
0.3
0.2
0.6
0.2
0.2
0.1
0.1
0.2
0.4
0.0
0.0
0.0
0.0
0.0
dadadede dede
100%
de da
100%
20
100%
100%
W 8
6
No
Few
Some
Most
All
1.0
1.0
0.8
0.2
0.2
0.8
0.6
0.1
0.6
0.4
0.2
0.0
0%
20%
40%
60%
80%
100%
0.0
0%
20%
40%
60%
60%
100%
0.0
60
80%
100%
0.0
20%
40%
60%
80%
100%
0.0
20%
40%
60%
80%
100%
Percentage Values
Figure 3: Human interpretations (upper) from 25 annotators and PRESQUE scores (I.,) from RoBERTa-large (hottom) quantifier percentage scopes in HVD. The cyan bars indicate percentage values chosen by more than 10% of the annotators (red line) and, therefore could in human interpretations. The threshold is only used for illustration and not in experiments.
approximate percentage scopes. For example, 10%-30% for few
Evaluation Metries For HVD, given listener Lm based on an NLI model M. (plg)) is com- puted by averaging the entailment scores over all Sys for all values in W3 and normalize them to be a distribution. We can then compute the cross entropy between the human interpretation of quan- tifiers Ph from Section 5.1 and (x(p)g) to measure the similarity of quantifier interpretation between humans and M.
predictions (F1 @) under W3. which is used in question answering (Rajpurkar al., 2016). The metrics are averaged over the entire dataset, and Lpplg) is computed through Eq. Eq. 4.
Experiments and Results
We perform experiments determine the percent- age scope of quantifiers two datasets: the HVD dataset, which includes predicates annotated with quantifiers but lacks percentage scope annotations, and the QuRe datasct, which provides annotations for both quantifiers and percentage scopes. As a baseline, we use the literal listener, Lo-
(h(()g)) log (plg) [
CrossEntropy
Σ Σ yEU PEW,
For S, in Quîe, we compute the following metries,
5.1 Human Interpretation of Quantifiers
To quantitatively assess the similarity between quantifier understanding between foundation mod- els and humans, we first collect interpretations (plg) from human participants. For this, ploy 25 Mechanical Turk annotators who are tasked with providing the percentage scope of quantifiers. To guide them, we provide an instruction (see Ap- pendix K for details) and present them with five questions. Each question requires the annotator to indicate the strength of given quantifier word in i by providing either percentage scope single percentage value in W9=0.11 without resorting online searching. The distribution of the annota- tors' choices, shown in Figure 3, reveals that humans interpret different percentage peaks for dif- ferent quantifier words. The percentage scope few, some, most indicated by the selection ratio
HIT@1 [[arg max L(plg) E Sgoldea] NEW,
MRR (Bm En Engadenn Rank,')
CrossEntropy Epersoeen log (p(((g))
where PP)llg)) L(p'la// Ep L(pl4) Pmin + 3, p' E W8
where Bm Pmax
where II(-) is an indicator, Sgolden is the gold scope [Pmin: ).ax], and Rank,' is the rank of p' in W3 by L(plg). HIT@1 measures whether the top inference percentage lies in the gold scope. MRR and cross entropy measure the average rank and confidence of the gold scope. We also compute the span-based based Fl score between the gold scope and the primary scope (Section 5.2) of the top K

---

more than 10% are larger than those of no and all. Meanwhile, the percentage scope of some is lean- ing to few rather than most, where few and most have little scope overlap.
SENTENCE
SCOPE
PREF
Lo: 0%-40%
0.34
No ostriches are strange looking.
11: 0%-10%
0.66
Lo: 0%
0.12
Few tomatoes are green.
5.2 NLI Model's Interpretation of Quantifiers
1. 0%-300
0.88
We evaluate the quantifier reasoning ability of the 'large' (or 'xxlarge') variants of AL- BERT (Lan et al., 2020), XLNet (Yang et al., 2019), BART (Lewis et al., 2020) and RoBERTa (Liu et al., 2019) that are fine-tuned on the NLI tasks (using Adversarial NLI (Nie et al., 2020), SNLI (Bowman al., 2015) MNLI (Williams et al., 2018), and NLI-style FEVER (Nie et al., 2019) datasets).
Lo: 80%-
0.38
Some kites are made of plastic.
L:: 10%-50% 0.62
Lo: 80%- 100%
0.66
Most owls live in forests,
L1: 600%
0.34
Lo: 60%-100%
0.22
All gates are used for enclosing.
L:: 70%-100%%
0.78
Table 5: Examples of percentage preferences between PRESQUE (L,) and Lo in HVD. The primary scope (Scope) is scope with the maximum subarray sum of L(plg) within top K inference values, which stands for the most confident percentage scope of the model. The human preference (Pref.) is the ratio of scopes preferred by the human annotators. Green and blue rep- resent preference to Lo and PRESQUE, respectively.
CROSSENTROPY
BASE (ODEL[(PARAM))
La
PRESQUE
ALBERT (222M)
1.76
1.48
XI.Net (361M)
1.64
1.35
BART (407M)
1.89
1.32
RoBERTa (355M)
1.69
1.29
Table 4: Comparison of different NLI models in HVD with Lg being the baseline using NLJ models for di- rect interpretation and PRESQUE is the pragmatic-based interpretation. PRESQUE is better than Lo and RoBERTa- large has best cross entropy in PRESQUE.
Next, we compare the results of PRESQUE with that of a literal listener baseline, Lo (Equation 1). As the percentage scope of a quantifier measured by the consecutive percentage range among the top K-ranked percentage values, we compare the consecutiveness of Lo and PRESQUE, which is mea- sured by the proportion of sentences with the entire top K choices being able to constitute single con- secutive range. For example e.g. {10% 20%, 30%) } constitutes consecutive range from 10% to 30% while ( (10%, 30%, 50%) ) does not. Consecutive- ness based on the assumption that consecutive ranking of percentage values indicates better quan- tifier understanding as the semantic meaning of quantifier words does not leap between disjoint per- centage values. To enlarge the possible ranges, we start by K = 3 and include the results in Figure 4, where PRESQUE has higher consecutiveness than Lo. showing that PRESQUE has more consistent per- centage inference behavior. Moreover, We select the primary scope by finding the consecutive scope (e.g. [10-33%) and (10%, 30%, 50% in the previous example) of the largest aggregated L(plg) among all consecutive scopes.
The comparison of quantifier understanding us- ing PRESQUE and Lo is presented in Table 4. The results show that PRESQUE achieves lower cross entropies compared to Lp. Among the NLI models, RoBERTa performs the best within the PRESQUE framework, and therefore, it is chosen the pri- mary model for subsequent experiments. The (plgg) scores of PRESQUE from RoBERTa, which are used to represent the model's interpretation of the percentage scopes of different quantifiers, are displayed in the lower half of Figure 3. In general, different quantifier words exhibit distinct percent- age distributions. Similar to Section 5. 1, the scopes of few, some, and most can be approximated as 0% 30%, 10% 50%, and 60% 100%, respectively, with cutoff criteria L(plg)= 2 0.1. These ranges align closely with the scopes determined by human evaluation (upper half of Figure 3). Further, the L(plg) q) scores change in a smooth way as the per- centages increase or decrease within the regions where T-(plz) 2 0.1. This suggests that the model can understand percentage values quite well.
We additionally compare the primary scopes between Lo and PRESQUE, through human pref- erences. For each quantifier word, we randomly sample 10 sentences where the top K inferences be- tween Lo and PRESQUE differ for the same S, with
'In preliminary experiments, we found that foundation models without NI.1 finc-tuning performed worse on the quan- tifier prediction task.

---

BART
RoBERTa
XLNet
ALBERT
PRESQUE
Consecutienne
Lo
0.6
0.4
0.2
0.0
5
10
5
10
5
10
5
Top K
Figure 4: The ratio of top K percentage values from PRESQUE (lime) and Lo (blue) that can form a single consecutive scope in HVD. PRESQUE has higher consecutiveness than Lo among all models.
HIT@ It
MRR1
CROSSENTROPY
F1 @ (1, 5)1
SPECIFICITY
Rnd.
Lg
L
Rnd.
Lo
L
Rnd.
Lg
L
Rnd.
Lo
L
Fully
4.1
27.3
29.7
12.3
22.1
24.3
6.44
5.64
5.74
2.8/8.6
19.5/24.3
21.5/26.5
Partial
8.2
26.4
28.5
11.6
21.2
21.7
7.78
6.99
7.06
4.3/8.3
16.9/25.9
18.3/27.3
Indeterminable
9.7
21.4
21.4
12.5
18.1
22.7
7.76
7.20
6.69
5.3/10.1
14.9/18.2
14.8/25.6
Total
7.9
24.0
25.1
11.8
19.8
22.7
7.47
6.86
6.78
4.4/9.3
16.3/21.7
17.1/26.3
Table Performance of PRESQUE (L1) versus Lo on QuRe using RoBERTa-large. Metries are displayed on 0-100 scale except for cross-entropy. Rnd. is random baseline (averaged over 5 seeds) where L(plg) is sampled from (O, 1) and normalized with softmax. The best results are bolded, The results show that the random baseline is than both Lg and PRESQUE in most metrics. PRESQUE performs better than Lo almost all specificity levels and metrics.
K = 5. We then recruit 40 annotators from Ama- zon Mechanical Turk to select the more reasonable primary scope between Lo and PRESQUE given Sy. We displayed the primary scopes for each S, in ran- dom order to avoid biases. Examples are included in Table 5 where PRESQUE generates smaller pri- mary scopes for universal quantifiers like No and All, and larger primary scopes of other quantifiers which incorporate more vagueness. We leave the more general analysis in Appendix F.
clusion of Carcassi and Szymanik (2021), which suggests that listeners aim minimize the distance from the speaker's s observation when the communi- cation success is measured based on distance.
Qualitative Analysis. Examining examples gen- erated by PRESQUE, we make several interesting observations. For fully determinable contexts, such as "... only (2 out of the total 27) few school children were recognized..." with a gold scope 5%-10% (the true rate was 2 children out of 27 7%), PRESQUE provides more accurate primary scope. In this case, PRESQUE predicted a scope of 0%-5%, while I. predicted scope of 0%. For partially determinable contexts. such as "... calcu- lating from less than few ... 13%..." (indicating a partially determinable percentage scope less than 13%), with gold scope of 5%- 10%, PRESQUE often generates a broader scope than Lo- In this case, PRESQUE predicts 0%-15% which is more expansive than T.g's 's prediction of 15%. For some indeterminable sentences like "... its alcohol content usually is very little." with gold scope of 0%-5%, PRESQUE provides primary scope of 0%.5%, while To predicts significantly distant scope 60%-70% Appendix B provides more comprehensive set of examples.
Table 6 provides a comparison of the top per- centage predictions with the gold scopes from PRESQUE and Lo in QuRe. We observe that, in general, PRESQUE outperforms L.0 in several as- pects. Firstly, the topmost prediction value from PRESQUE appears more frequently within the gold scope, leading to a higher HIT@ score. Addition- ally, the percentage values within the gold scope are ranked higher among the top predictions by PRESQUE, resulting in higher Mean Reciprocal Rank (MRR). Furthermore, there is larger over- lap between the primary scopes of PRESQUE and the gold scopes, indicated by higher F1 score. Moreover, PRESQUE predicts better primary scopes on a distance-based metric designed to measure the distance between scopes, and we include the results in Appendix G. This finding aligns with the con-

---

6 Conclusion
vs Frequentist interpretations of percentage num- bers, among many other factors. Thus, ambiguities and subjectivity are natural when determining a quantifier's scope. Our dataset and analysis largely skirt many of these complex issues.
Generalized quantifiers widely used for ad- dressing satisfaction in natural language. However, the quantifier understanding abilities of founda- tion models are not well-studied or well-supported by the existing benchmarks based on factual con- text. In this work, we study quantifier understand- ing by proposing the PRESQUE framework that for- mulates the problem in a pragmatics reasoning perspective and the format of NLL. And col- lect dataset QuRe that includes human annotated quantifiers and percentage mentions for Wikipedia sentences. From the experimental results on the HVD dataset and our collected QuRe dataset, the PRESQUE achieves better performance compared to the direct interpretation of quantifier semantics by a literal listener model.
Ethics and Broader Impact
We employ crowdsourcing through Amazon Me- chanical Turk for (a) certain annotations of our dataset, QuRe, (b) understanding human interpreta- tion of quantifier semantics, and (c) human evalu- ation of PRESQUE and baselines. In all our crowd- sourcing tasks we do not collect any personally identifiable information about the turkers and all the turkers were paid above minimum wage, which is included in Appendix D. We released our crowd- sourcing tasks to turkers in the USA and con- strained the approval rate of the turkers to be above 98% to ensure good-faith turkers.
Acknowledgments
Besides, the prevalence of quantifiers naturally occurring corpora would inherit the generation be- havior of models. PRESQUE, as one step towards revealing the quantifier understanding ability of foundation models, could be helpful in more accu- rately interpreting the meaning in model--gnerated text. It could also support automatic pipelines for knowledge-intensing reasoning that include quan- tifications, logical reasoning in natural language.
The authors would like to thank the anonymous reviewers for their suggestions and feedback on the work. This work was supported in part by NSF grant DRL2112633 The views contained in this article are those of the authors and not of the funding agency. We also thank Daniel Fried for useful discussion and suggestions to this work.
Limitations
In this work, we investigate the quantifier under- standing abilities of several foundation models and collect a dataset QuRe that we expect will sub- stantially benefit research on quantifier semantics. However, despite the value of our dataset and the promising results from the PRESQUE framework, our analysis and findings have some notable lim- itations. First, we note that our study and dataset still focus small part of the generalized quanti- fiers and likely do not cover the entire spectrum of quantifier semanties. Second, the sentences in our dataset all come from Wikipedia. Consequently, the performance of PRESQUE and the generalizabil- ity of our findings to other domains or languages remains an open question. Finally, assigning pre- cise percentage scopes to quantifiers can be a chal- lenging even impossible task, since quantifier semantics are complex and depend on many factors beyond those analyzed here. In particular, these may subjectively depend on the domain, an annota- tor's background knowledge or culture, comfort with the mathematics of percentages, and Bayesian
References
Rachid Alami. 2013. On human models for collab- orative robots. In 2013 International Conference Collaboration Technologies and Systems (CTS), pages 191-194.
Leon Bergen, R. Levy, and Noah D. Goodman. 2016. Pragmatic reasoning through semantic inference. Se- mantics and Pragmatics. 9.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji. Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Dem- szky, Chris Donahue, Moussa Doumbouya, Esin Dur- mus, Stefano Ermon, John Etchemendy, Kawin Etha- yarajh, Fei-Fei, Chelsea Finn, Trevor Gale, Lau- ren E. Gillespic, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guba, Tatsunori Hashimoto. Peter Henderson, John Hewitt, Daniel Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Khat- tab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna,

---

Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak. Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Ma- lik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadim- itriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan. Robert Re- ich, Hongyu Ren, Frieda Rong. Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R' Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas. Florian Tramèr, Rose E. Wang, William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Ya- sunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang. Tianyi Zhang, Xikum Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021. On the opportunities and risks of foundation models. ArXiv, abs/2108.072 07258....
Jonathan Gordon and Lenhart K. Schubert. 2010. Quan- tificational sharpening of commonsense knowledge. In AAAI Fall Symposium: Commonsense Knowledge.
H. P. Grice. 1975. Logic and Conversation, pages 41 58. Brill, Leiden, The Netherlands.
Aurélie Herbelot and Eva Maria Vecchi. 2015. From concepts models: Some issues in quantifying fea- ture norms. In Linguistic Issues in Language Tech- nology (LiLT).
Alexandra Horowitz, Rose M. Schneider, and Michael C. Frank. 2018. The trouble with quantifiers: Explor- ing children's 's deficits in scalar implicature. Child development, 89 6:e572-e593.
Pratik Joshi, Somak Aditya, Aalok Sathe, and Monojit Choudhury. 2020. TaxiNLI: Taking ride up the NLU hill. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 41-55, Online. Association for Computational Lin- guistics.
Samuel R. Bowman, Gabor Angeli. Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 332.642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Confer- ence on Learning Representations.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence- to-eqqueee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Fausto Carcassi and Jakub Szymanik. 2021. 'most' vs 'more than half"; : An alternatives explanation. In Proceedings of the Society for Computation in Lin- guistics 2021, pages 334-343, Online, Association for Computational Linguistics.
Ruixiang Cui, Daniel Hersheovich, and Anders Søgaard. 2022. Generalized quantifiers as a source of error in multilingual NLU benchmarks. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages Seattle, United States. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv, abs/1907, 1692..
Andrzej Wlodzimierz Mostowski. 1957. On a general- ization of quantifiers. Studies in logic and the foun- dations of mathematics, 93:311-335.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), ). pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
A Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose. and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference Computational Linguisties. pages --40--353, Santa Fe, New Mexico, USA. Association for Com- putational Linguistics.
Michael C. Frank and Noah D. Goodman. 2012. Predict- ing pragmatic reasoning in language games. Science, (36(6084)9998-999 998.
Yixin Nie. Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neu- ral semantic matching networks. In Association for the Advancement of Artificial Intelligence (AAAI).
Noah D. Goodman and Michael C. Frank, 2016. Prag- matic language interpretation as probabilistic infer- ence. Trends in Cognitive Sciences, 20(1 1):818-822
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela, 2020. Adversarial

---

NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4885-4901, Online. Association for Computa- tional Linguistics.
Shashank Srivastava, Igor Labutov, and Tom Mitchell. 2018. Lero-shor learning of classifiers from natural language quantification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 306-316.
OpenAI. 2022. Introducing chatgpt.
Alberto Testoni, Sandro Pezzelle, and Raffaella Bernardi. 2019. Quantifiers in a multimodal world: Hallucinating vision with language and sound. In Proceedings of the Workshop on Cognitive Modeling and Compatational Linguistics, pages 105-1 16, Min- neapolis, Minnesota. Association for Computational Linguistics.
Sandro Pezzelle. Raffaella Bernardi, and Manuela Pi- 8a. Probing the mental representation of quantifiers. Cognition, 181:117-126.
Sandro Pezzelle, lonut-Teodor Sorodoc, and Raffaella Bernardi. 2018b. Comparatives, quantifiers, propor- tions; multi-task model for the learning of quan- tities from vision. In Proceedings of the 2018 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume (Long Papers), pages 419-430, New Orleans, Louisiana. Association for Computational Linguistics.
Shivin Thukral, Kunal Kukreja, and Christian Kavouras. 2021. Probing language models for understanding of temporal expressions. In Proceedings of the Fourth BlackboxVLP Workshop on Analyzing and Interpret- ing Neural Networks for NLP. pages 396-406, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2012. The communicative function of ambiguity in language. Cognition, 122:280-291.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals. Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilities of large language models. Transactions on Machine Learning Research. Survey Certifica- tion.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A croad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, Association for Computational Linguistics.
Abhilasha Ravichander. Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019, EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference. In Proceedings of the 23rd Conference on Computational Natural Lan- guage Learning (CoNLL), pages 349-361, Hong Kong, China. Association for Computational Lin- guistics.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ Salakhutdinov, and Quoc V Le. 2019, Xlnet: Generalized autoregressive pretraining for lan- guage understanding. In Advances in Neural Infor- mation Processing Systems, volume 32, Curran Asso- ciates, Inc
Kyle Richardson, Hai Hu, Lawrence Moss, and Ashish Sabharwal. 2020. Probing natural language inference models through semantic fragments. Proceedings of the AAAI Conference Artificial Intelligence, 34(00):8713-8721.
A Example Metadata of QuRe
Kyle Richardson, Hai Hu, Lawrence S Moss, and Ashish Sabharwal. 2019. Probing Natural Language Infer- ence Models through Semantic Fragments. In Asxo- ciation for the Advancement of Artificial Intelligence (AAAI).
To investigate the topic coverage of QuRe sentences, we use GPT-3.5-turds to generate 3 topics for each sentence, using instruction in Appendix J. The most frequent topics are listed in Figure 5, where nearly 10% sentences are about statistics, followed by animal, percentage and demographics.
Subhro Roy, Tim Vieira, and Dan Roth. 2015. Reason- ing about quantities in natural language. Transac- tions of the Association for Computational Linguis- tics, 3:1-13.
B PRESQUE Examples in QuRe
Stephanie Solt 2016. On measurement and quantifica- tion: The case of most and more than half. Language, 92:100 65.
We include several examples in Table 8 where S, as well as the specificity level, and the S, as well as the golden percentage scope are located on the upper and lower half of each block. We can see that
Robyn Speer. 2022. rspeer/wordfreq: v3.0.

---

[WIK] E ORIGINAL SENTENCES
[SPECIFICITY, EXPRESSION] Quee SENTENCES
TOPICS
[Human] Most humans (61%) live in Asia; the re- mainder live the Americas (14%), Africa (14%), Europe (11%), and Oceania (0.5%). Within the last century, humann have explored challenging environ- ments such as Antarctica, the deep sea, and outer space.
[Fully, 0.1I] Most humans (61%) live in Asia; the remainder live the Americas (14%), Africa (14%), some Europe, and Oceania (0.5%), Within the last century, humans have explored challenging environ- ments such as Antarctica, the deep sea, and outer space.
population continents
exploration
[The Jungle Book (2016 film) The Jungle Book was shown across 4,028 theaters of which 3,100 theaters (75%) were in 3D, including 376 IMAX screens, 463 premium large format sereens, and 145 D-Box locations.
[Fully, 0.75] The Jungle Book was shown across 4,028 theaters of which most (3,100) theaters were 3D, including 376 IMAX screens, 463 premium large format screens, and 145 D-Box locations.
theaters
movie release 3D technol- ogy
[Electric car by country] The EV market share of total and used cars first registered during 2018 was 2.8% based 5,557 out of total of 198,600 first registered 7,55. vehicles were reg- istered in this country over 2019.
[Fully, 0.028] The EV market share of total
electric vehi- cles
and used cars first registered during 2018 was small based 5,557 out of a total of 198,600 first reg- istered cars. 7,542 vehicles country 2019.
market share registration numbers
registered in this
[List of blade materials] Prior to 2002, INFI con- tained 0.5% Carbon, 0.74% Nitrogen, about 1% Cobalt, and about 0.1% Nickel.
[Partially, 0.005] Prior to 2002, INFI contained tiny levels of Carbon, 0.74% Nitrogen, about 1% Cobalt, and about 0.1% Nickel.
chemical com- position INFI elements
[Housing in the United Kingdom] British dwellings had the oldest age profile in the EU with over 60% being built before 1960, and with only just over 10% being built between 1991-2010.
[Partially, > 0.1] British dwellings had the oldest age profile in the EU with over 60% being built before 1960, and with some being built between 1991-2010.
age
housing statis-
tics construction date
[Ice cream] Gelato typically contains 7-8% fat, less than ice cream's 's minimum of 10%.
[Partially, >= 0.1] Gelato typically contains 7-8% fat, less than the moderate amount found in ice cream.
food comparison fat percentage
[Toobacco] A study published in Morbidity and Mor- tality Weekly Report found that 2019 approxi- mately one in four youths (23.0%) in the U.S. had used a tobacco product during the past 30 days.
[Partially, 0.23] A study published in Morbidity and Mortality Weekly Report found that in 2019, some (one in four) youths the U.S. had used a tobacco product during the past 30 days.
youth tobacco use
scientific study
[Polish cuisine] It typically made from rye hread, usually known as black bread, and not classified as an alcoholic beverage Poland, as alcohol content usually ranges from 0% 2%.
[Indeterminable, 0 0.02 is typically made from rye bread, usually known as black bread, and not classified as an alcoholic beverage in Poland, as its alcohol content usually very little.
food
beverage alcohol con-
tent
[List of blade materials] In order for steel to be considered stainless must have Chromium content of least 10.5%.
[Indeterminable, >= 0.105] In order for steel be considered stainless it must have some Chromium content.
steel
metallurgy
composition
[British military bands] The average age of the 304 drummers Waterloo was 25, with about 10% being boys under 16.
[Indeterminable, ~ 0.1] The average age of the 304 drummers at Waterloo was 25, with some being boys under 16.
age
music
statistics
Example data of QuRe, with target percentage mention and quantification underlined. The header marks either the Wikipedia entity where the sentence is extracted or the specificity and the generated percentage scopes. For example, for the Jungle Book sentence, the percentage scope 75% can be fully specified by the proportion of 3100 over 4028 theatres, while for the sentence about Gelato, the content before the percentage mention indicates that the fat content of ice cream is higher than 7-8%, but cannot provide more information to further narrow down the scope, and therefore the specificity partially.
Table 7:

---

with thresholds, 0.1 and 0.5, We can observe that the NLI model is more sensitive to the percent- age values the premise than quantifiers. The en- tailment scores of percentage inference is relatively low, which led by high neutral scores, making challenging to identify the percentage scope for each quantifier. For example. few, some and most don't have any percentage values that exceed even the lower threshold. The lower half of Figure 6, however, demonstrates more interpretable entail- ment distribution, where the percentage scopes of few, some and most can be interpreted as 0%-20% 0%-90% and 60%-100% by the higher threshold. In short, NLI models are more sensitive to interpret- ing accurate numerical premises, which has also been observed that NLI performs better with ac- curate premises (Thukral et al., 2021; Richardson et al., 2019) where premises with quantifiers are less accurate than premises with percentage values.
statistics
animal
percentage
demographics
food
population
race
mortality rate
composition
Topics
ethnicity
percentages
agriculture
scientific study
ingredients
nutrition
chemistry
animal behavior
countries
comparison market share
D Annotator Recruitment
0.00
0.05 0.10
We use Amazon Mechanical Turk as the crowd- sourcing platform for different annotation aspects of QuRe described in previous paragraphs. To finalize our pool of turkers, released a qualifi- cation task test basic understanding of quantifier semantics. The annotators are selected base on the United States, completed more than 1000 HITs with more than 98% approval rate. For annota- tor that participates the final QuRe collection in Section 3, they can make at most one annotation mistake in sentence rephrasing of the qualification task. The qualification task had pass rate of 38% and we recruited 18 turkers for the main annotation tasks of QuRe. Annotators are paid about 7.330 7.30/hr on average. Besides, annotators recruited for hu- man interpretation of quantifiers are paid about $ 9/hr on average. And the annotators recruited for human preference of percentage scopes are paid about $ 9.60/hr on average.
Sentence Proportion
Figure 5: Top 20 sentence topic statistics in QuRe. The most frequent sentence topics, where nearly 10% sen- tences include the topic statistics and 8% sentences include animal. Topics like demographics, food and population also cover more than 5% sentences QuRe.
PRESQUE provides more accurate primary scopes for fully determinable sentences.
Discussion
The semantic understanding of pragmatic listener is proportional to the product of two entailment- based probabilities, where the premise and hypoth- esis are flipped with respect to each other (i.c., the premise used for (glg)) is the hypothesis for cal- culating P(plg) in the prior (Eq. 4) To arrive at an intuitive understanding of why considering the
E Percentage Scope Generation Details
With granularity g, window size w, the grounded percentage scope can be determined Table 10. For HVD, / is set to be 0.1. And in experiments on QuRe, 8, w, 9 are set to be 0.05, 2, 0.01 respec- tively unless specified.
flipped premise-hypothest pairs, we analyze the sensitivity of NLI models (specifically, RoBERTa fine-tuned NLI model) towards percentage values (quantifier inference) and quantifier words (per- centage inference) in premises. The distribution of average entailment scores over all the premises is shown in Figure 6. The upper part of the figure shows the result of percentage inference and the lower part shows the result of quantifier inference,
F Human Preference of HVD Examples
Figure shows PRESQUE is in general preferred over La by the annotators, while the preferences

---

[GS.] SENTENCE, [SPC.] SENTENCEP
PRIMARY SCOPE
MRR
FI@5
CE
[F] In 57 separate fights, loss observed to Neope goschkevitschii, giving V. mandarinia large winning rate.
La: 5%--20%
0.11
0.00
7.67
(95%-100% In 57 separate fights, loss observed Ncope goschkevitschii, giving mandarinia win rate 98.3%.
L1: 85%-100%
0.67
0.67
3.52
[F] In the 2017 Dutch study. only (2 out of the total 27) few school children recognized that the website was hoax.
Lo: 0%
0.08
0.00
7.79
[5%-10%] In the 2017 Dutch study only 2 out the total 27 school children (7%) recognized that the website was a hoax.
L1: 0%-55%
0.11
0.50
6.36
[P) From 4 locations different parts of Europe, a large num- ber had clutch size of 2, 41 had size of 3, clutches of and 4 each constituted about 8%.
Lo: 30%%440%
0.22
0.40
6.29
[40% %455% From locations different parts of Europe, 43% clutch size of 2, 41% had size of 3, clutches of and each constituted about 8%.
L1: 30%-455%
0.33
0.67
[P] The empirical occurrence of regenerated claws in fishery crabs calculating from
Ly: 10%-15%
0.17
0.50
7.79
harvests is low, with studies less than few (Davis et 1978). 13% (Florida Fish and Wildlife Conservation Commission, 2011).
(5 10% ( The empirical occurrence of regenerated claws in fishery harvests is low, with studies stone crabs calculating from less than 10% (Davis et al., 1978), to 13% (Florida Fish and Wildlife Conservation Commission, 20.1).
L1: 0%-15%
0.50
0.67
4,40
[ It is typically made from rye bread, usually known as black bread, and is not classified as an alcoholic beverage in Poland, its alcohol content usually is very little.
Lo: 60%--70%
0.06
0.00
6.97
[0-5 % I It typically made from rye bread, usually known as black bread, and not classified as an alcoholic beverage Poland, as its alcohol content usually ranges from 0% to 2%.
L1: 0%6-5%
0.33
1.00
4.16
(1 Chlamydospore germination requires moderate germination success rate. Spore production highest at midday, relative to temperature increase and relative humid- ity decrease.
hours, with a
Lo: 30%6335
0.13
0.50
18.85
[30%-55% |Chlamydospore germination requires 30 52 hours, with germination success 32 54%. Spore pro- duction is highest midday, relative temperature increase and relative humidity decrease.
L1: 40%-550%
0.22
0.67
16.17
Table 8: Examples of PRESQUE (L,) versus Lp. The sentences are paired with percentages and the corresponding sentence with quantifiers, with the target percentage and quantification phrase underlined. The headings mark either the gold scope (GS) or the specificity levels (SPC.) with [F/P/T] being CE stands for cross-entropy. Bolded figures are better results. Predictions collected from Wa=0.05- L1 achieves better MRR and cross entropy then lop among different sentence inferring categories.
respectively.
CONCEPT
FEATURE
ANNOTATIONS
SENTENCE BASED ON TEMPLATE
rock
has minerals
all, all, most most, most, most some, some, all
All rocks have minerals.
van
has sei__sidinngddde doors has fine_ fine_sand saand_cooeriii it
Most vans have sliding doors.
sandpaper
Some sandpapers have fine sand cov- ering it.
banana
is_ round
no. no all, few, few
No bananas are round.
tricycle
used for for_ffforrrrspal transportatioonn
Few tricycles are used for transporta- tion.
Table 9: Sample (concept, feature), human annotations for the quantifiers, and the corresponding HVD sentences that serve as S, using the majority quantifier annotation.

---

Percentage Inference
No
Few
Same
Most
1.0
0.8
0.6
0.2
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Quantifier Inferenc
1.0
0.8
0.6
0.1
0.4
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Percentage Values
Figure 6: Comparison of the ILI model conducting percentage inference (Entailment( Sy. Sp). upper) and quantifier inference (Entailment( Sy). lower) in HVD. The NLI model is more sensitive in quantifier inference in general. Cyan bars indicate values higher than the upper threshold (0.5). Note that the bar values do not stand for probabilities and not sum up
EXPRESSION
PERCENTAGE SCOPE
PRESQUE
p
No
51.8
48.2
>p
(p.p w
Few
49.4
50.6
>= p
b,, w
Some
62.2
37.8
|p 9.P)
Most
48.3
51.7
<=
|p w0 9.
All
55.3
44.7
P1 Pa
[Pr.Pa]
Total
53.4
46.6
p
Ip 10 9.P
Table 10: Percentage scope grounding rules with gran- ularity g. window size for approximation w. And [Pain Pmax] is the smallest scope in W3 that includes above scope. scope would cut off 0 and I.
Figure 7: Listener preference from humans for HVD examples. The inference from PRESQUE is in general, preferred by humans than Lo. while the preference of each quantifier differs (PRESQUE is more preferred for No, Some and All).
H Instruction for Sentence Filtering
may differ for different quantifiers. The primary scopes of PRESQUE for no, some and all are more preferred than Lo by the annotators. Some and all have p < 0.05 in chi-squared test.
In this task, you will determine whether a given sentence that has one or more quantifier values mentioned can have those quantifier values replaced by natural language quantifier like 'some", 'most' or 'generally'.
G Distance-based Scope Evaluation
Example sentences that meet the criteria like 'It consists of ahout 80% water, soluble minerals (ncarly 3% with half of the potassium) and polyphenols. and sentences that don't meet the requirement 1180.1 million network. Ridership had decreased 13% from 2014 2019 due many factors." where the percentage value represents incremental percentage changes or comparisons (c.g. 'drop by 50%* 220% higher', 'X% better") ) instead of absolute percentages.
To measure the primary scope of L(plq) and the gold scope (Pminn Pmax in distance-based metrics. We compute the minimal scope distance (MSD) over the top K predictions (MSD@K) Specificaly
like rides SEPTA's 'city transit'
Ippl & Spolien
MSD@K
Σ of TTTppk
min Pain p'.p' p Pmns)
Bm
where
Bmn
Peus
Panin + 8, p' E W/
Do you think the following sentence meets require- ment? Answer in Yes

---

Total
Fully
Partially
Indeterminable
0.8
Consecutiveness
0.6
0.4
0.2
0.0
5
6
7
10
5
6
9
10
3
4
7
8
9
10
3
7
8
10
3.5
MSD
3.0
2.5
2.0
2
8
9
10
3
5
6
7
8
9
10
2 3
4
5
6
9
10
2
10
0.4
0.3
PRESQUE
0.2
Lo
2
7
8
9
10
2
3
4
5
6
7
8
9
10
1
2
3
4
5
6
8
9
2 3
4 5
9 10
Top
Figure 8: Consecutiveness (t), MSD (L) and (1) of PRESQUE (lime) and Lo (blue) on QuRe with for better illustration. PRESQUE has higher consecutiveness, lower MSD and higher Fl than Ly across all specificity levels.
I Instruction for Percentage Scope Generation
J Instruction for Sentence Topic Generation
Please use three or four labels to categorize given sentence (starts with "sentence"), including the topics of the contents, split with semicolons. For example,
The instruction for percentage scope generation is shown below.
Sentence: In fact, 2006 survey found that trapping as solution beaver problems had 79% failure rate within two years due to resettlement by new beavers. Lahels: scientific study; animal; rate
Sentence: Among individual countries, the proportion of urban residents living in slum areas in 2009 was highest in the Central African Republic (95. 9%),, Chad (89.3%) Niger (81.7%). and Mozambique (80.5%). The distribution of slums within city varies throughout the world. Labels: population; ranking: countries. Now, please label the following sentence:
lask, you give sentence a quantifier expression the sentence, and you need to convert that quantifier expression into mathematical expression. For example, for the sentence AAbout 30% homes are owned outright by their occupants. [30%]', you are corresponding mathe- matical expression 0.3, where means approximate. Similarly, all the other mathematical operations sup- ported include meaning 'more than' (e.g. "more than 80% would be > 0.8), < meaning 'less than', mean- ing range (e.g. '20% to 50%* would be 0.2-0.5) and null meaning exact (e.g. 'takes up 20% would be 0.2). Answering the expression itself is enough, don't repeat the sentence or use additional English words other than the operations. Also, try to avoid using "<" and ">" if you can formulate the range by using" '-' Now, please do the same conversion for the following sentence:
given 30% the bracket,
K Instruction for Human Evaluation
The instruction for collecting human perception of quantifier words in Section 5.1 is displayed as

---

This form contains several natural language quantifiers. The users are expected to pick one/two numerical percentages from the provided list of percentages such that most accurately bound the range of the given quantifier to the best of his/her knowledge and online scarching not encouraged.
In this task, given sentence (starts with 'Sentence:") containing predicate with a quantifier. and you need to provide percentage scope that the predicate satisfies.
For example, if you given "Sentence: Some apples red." for the quantifier 'some', and you believe 37%--42% apples are red. Then the pereentage scope for "some apples red" would be 37%-42%.
Users can different (real imaginative) statements examples to help estimate the range, such 'No water comes from the sky." and 'Most cucumbers are scavengers.".
The scope you choose should be rounded in the granularity of 5 %. In the previous 'apples red' example, your answer will be "35%-45%% " Not that the percentage value cannot exceed 0% and 100%. You can also select single percentage value for the scope.
The users can select more than options mark the lower and upper bound of the range, they believe only one percentage would apply, they can seleet only option.
Please provide percentage scope for "some" in the following sentence.
An example 'All' stands for' with a statement is 'All sugars are white.' The users are expected to select '100% or a range (based on the user's understanding) from all provided percentages as the range for 'All', the paraphrase becomes 'A% to B% sugars are white.' (A and are selections and can be the same) which becomes the most appropriate paraphrase of 'All sugars are white', Note that the statement itself does not necessarily involve factuality (in fact, sugars can have various colors).
Sentence:
In the example instruction shown above, the quantifier some would be replaced by the target quantifier that appeared in the sentence attached to the instruction. For example, for sentence Adult clams can get most of their nutrients from the algae and the rest from filter feeding. (gold scope 65%- 100%), the output of GPT-turbo-3.5 for quantifier most is 60%-80%
The instruction for collecting listener preference of Lo and L, in Section 5.2 is displayed as
Overall, GPT-3.5-turbo achieves 0.28 Fl score of the quantifier understanding task on Quîe, un- der the same configuration in Appendix E, which is slightly higher than the FI @5 performance of PRESQUE. However, we aware that text-to- text models like GPT-3.5-turbo still suffer from hallucination and the output is unstable due temperature-based sampling. Mcanwhile, the PRESQUE in this work is agnostic to the backbone model choices and can be applied to any mod- els that score the entailment relation between sen- tences.
This form contains several statements (c.g. sugars are white) with natural language quantifiers (e.g. all). The users are expected to pick the more appropriate percentage range from the provided two options such that accurately bound the range of the given quantifier to the best of his/her knowledge and online searching is not encouraged.
An example statement with quantifier is 'All sugars white.", and two example options are '0%-30%% and 100%-100%''
In this example, the users expected to select '90%- 100%', which results in that "90%-100% sugars white." better describes 'All sugars white.'. Note that the statement itself does not necessarily involve factuality (in fact, sugars can have various colors).
M Annotation Task Interface
The instruction for the qualification task in collect- ing QuRe is included Figure 9, and Figure 10 shows the example tasks annotators need to com- plete.
L Quantifier Understanding of GPT-3.5-turbo
Although mainly focus on NLI models to develop PRESQUE, we also test the performance of QuRe on GPT-3.5-turbo using the following instruction.

---

disective sasccate numerical parcer dages quarcihers lalsas sentence contaniing
vakaaharge with sutude Engliit la saaage quarthhe so trut the mesnin of centarans namaneo undfanges a an posa die. The saague of quarthere harw and staps the lac ard provisked instrut ona carllllly before procteciin
Instruction:
cumeikal parconlage a quart'le for word You reed
owing que words
all gencrall, most, ssubity some. likey, lew, Iife occes onally, none addonn tiny amall moderali, lange
Noshonicette If none of the quantffrrs can applied but lset itt one of them
Use Sataraal Patacanoa Lazs addiiionall cortedt to help dezamrine min quart fir word the senaance.
2. Also ploaso conduct recessary ponslrnzinng to mako the quart centence fuom with nir onar onarges to the sontonco mooning noving some wordsin the original sonoonoo diso wane sadect
ecampie,
appeor
coliones how
speci fied cuncuactile canteed cuantifwe sertence treeiing bee
not sund.
Fully: the exnct perowrnng walce can be spectted from the content by some "guess mantiond the conteat. H4l Faaiippe
"Guantiiee raeat broken the armet
of (Quentiiee) can specreed hrmm out
parsar
spaci
"Guantaa"
percemaaee
Indeterminate contact dows provide enough informaten to oeternane scope replacedd percentage value.
Example
"Guantia blick can By ypu enough information provided for the (Cu (Cuantiiid
Optional other fredi the taok, had thee providee cnulul fnectaads, any contus one help improse in the heedback bee. Thanika
Importent intornation on belore proceed
The tote number seriannes finsh a 10.
Your arawar will be end usled by the tot coring crforia Sppocial attantion to Elanne to racaine the sward)
Surreittanea Compliee
leseupt agrional al beadbacie to recence rowers.
Bastence Fulizeaa Illl cuarchied seriances inslwed of mevely ghvn oor, nemrove conmapording serornage
Demazrnableess Whather The quarthoattan uses resadiablee blea quantiee word replace the pansemaae valus. Dlamsy Use receussry edios make line senrense in Ergi
1ida Eorp glass
Examptas
Origial sentence: About 30% of homne oaned outrigtt thei oocuparts, (D%) (Conact
by outight occuperts. X)
Original
species up owing your i grow [000]
Quantiiod
specis allowing the tairy quickyy (fluency X)
Quantifed sertence:
(ncomech Duanttted sentance: None (50%))
/laaaarabbeeee Gerrance T'Bnnes
Spechicttt inderemmaclle
Figure 9: Instruction for the annotation task.

---

Those who viclate the face fhe of up CHF 10,000.r Sestember 2018, ban face covering valls was approved with a 67% vate in favour in the canton of St. Gallen, (87%)
Show Exteral fleference Unk (Oprione))
Quantifad serbence:
Spuniiiitt
Pully Partiall Indeterminable
Thes mortaliy raie amrong untrested bite vichirm rearly 100%. [100%]
(Cotiona)) link fnacommend
Quantified
Sueaificity
C Fully Pertislly Indeterminable
(Aptiona))
Figure 10: Interface of the example annotation task. Each sentence comes with a target percentage in the bracket at the end of the sentence that directs the target percentage mentioned in the sentence (e.g. 100% for nearly 700% in the second sentence). If there are multiple target percentage mentions that share the percentage value, positional indicator would be attached. Besides, optional reference link is provided by checking the box.


Reviews:
[{"review_id": "MtlQh9h9oQ", "reviewer": "Reviewer_dNQH", "summary": "", "questions": "1. Annotation clarification: \n\n    a. There are certain operators in Table 2 without a percentage upper or lower bound, such as “over”, and “not exceeding”. For these operators, how the percentage scopes are determined?\n\n    b. Could you elaborate more on the design of the window size? And how does the window size influence the operators and the percentage scope during annotation? \n\n2. Evenly spaced model output: \n\n    a. For different quantifiers, the corresponding ranges can vary a lot as shown in Table 3. However, the percentage spectrum is divided into evenly spaced intervals. The model’s categorical output may influence the model’s inference on certain quantifiers that can cover a large range. \n\n    b. Can the framework be adapted to a value regression setting, which may better fit the way that humans interpreted it?\n\n3. Framework clarification: the RSA framework is good at solving tasks that need to speak or comprehend sentences heavily dependent on the current dialogue context. Could you further clarify the following in your task:\n\n    a. How the dialogue context influences the quantifier semantics influences in this task? \n\n    b. And, how this influences in (a) reflected in terms of equations (1-4)? From my understanding, the pragmatic listener in equation (2) only takes additional information about the word frequency compared with the literal listener but not grounds in the current sentence context.\n\n    c. How the entailment score is computed?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough feedback. We address the questions raised by the reviewer below:\n\nQuestion 1.a In general, all the percentage scopes are bounded between 0 and 100%. Moreover, we apply several rules to generate the percentage scope using few hyperparameters (the detailed production rules are listed in Table 10 of Appendix E). \nQuestion 1.b The window size is separate from the annotation, the annotators are only required to provide a quantified sentence and the specificity. The window size is introduced by us to help frame the original ambiguity in the percentage expressions (e.g. around 10%) in a way a NLI model like RoBERTa is able to evaluate.\n\nSince the original sentences (with percentage mentions) in QuRe include different percentage expressions (and some of them like ‘around 10%' do not refer to exact percentage values), they need to be formatted in a way the model with unified granularity is able to handle. Therefore we introduce the window size to convert the percentage expressions into percentage scope with more explicit boundaries to meet the setup. \n\nQuestion 2.a The final percentage scope from the model is aggregated from the top categorical outputs (L461-463). Therefore, with the granularity and the number of top outputs as hyperparameters, the model is able to output different percentage scopes without training instances.\n\nQuestion 2.b Our framework can also be extended to accommodate the regression setting, where we train models to output percentage values corresponding to a quantifier. However, since the same quantifier may not correspond to a single percentage value in different contexts, we may have to develop other approaches to make model results interpretable to humans (for example, learning a distribution instead of single values). \nIn this regard, we would like to emphasize the broader impact of the data/annotations collected by us, as it can be adapted into various formats for different techniques of modeling quantifier semantics. We encourage the community to explore such avenues in the future to further expand the capabilities of PRESQUE.\n\nQuestion 3.a Specific topic that the context involves can skew the quantifier choices, for example, the threshold of ‘large’ in mortality or the salinity of water bodies can be lower than other topics (we list several annotated sentences with their original counterpart in wikipedia):\n\n“The NCDUs suffered 31 killed and 60 wounded, a large casualty rate.”\n“The NCDUs suffered 31 killed and 60 wounded, a casualty rate of 52%.”\n\n“The Dead Sea, known for its large salinity levels, is really a salt lake.”\n“The Dead Sea, known for its salinity levels between 30 and 40%, is really a salt lake.”\n\nIn comparison, sentences from other topics may look like,\n“The bitumen content is large (soluble in carbon disulphide), with a penetration value near to zero and a softening point (ring and ball) around 120 °C.”\n“The bitumen content varies from 83% to 92% (soluble in carbon disulphide), with a penetration value near to zero and a softening point (ring and ball) around 120 °C.”\n\nQuestion 3.b $S_{0}(q|p)$ includes the contextual understanding of the quantifier from the NLI framework. And the computation of $S_{0}(q|p)$ (Eq. 3) is different from $L_{0}(p|q)$ (Eq. 1) by taking a different order of inputs. We explain this choice in the fourth paragraph of the introduction (L70) addressing the capability of current NLI frameworks in handling ambiguity and elaborate it in Appendix C. Also, we can notice that the improvement of PRESQUE over the L_0 is different among different specificities, which is affected by the comprehension of context.\n\nQuestion 3.c The entailment score is computed by setting the premise and hypothesis as $\\tilde{S_{q}}$ and $\\tilde{S}_{p}$ respectively (or the other way around) in L168-188, the output scores are the scores corresponds to entailment/netural/contradiction respectively that sum up to be 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper smartly designed a new corpus with quantifiers mapped to a percentage scope which be easily evaluated for quantifier semantics inference. \n2. The authors presented the annotation and data statistics in detail. A suite of comprehensive evaluations is conducted including the comparison of the distributions between human and model interpretation, automatic metrics, consecutiveness, etc.", "reasons_to_reject": "Overall this paper presented a thorough introduction to data collection, analysis, and experimental results. I have several clarification questions regarding the framework design and annotation procedure. Please check below.", "questions_to_authors": "1. Annotation clarification: \n\n    a. There are certain operators in Table 2 without a percentage upper or lower bound, such as “over”, and “not exceeding”. For these operators, how the percentage scopes are determined?\n\n    b. Could you elaborate more on the design of the window size? And how does the window size influence the operators and the percentage scope during annotation? \n\n2. Evenly spaced model output: \n\n    a. For different quantifiers, the corresponding ranges can vary a lot as shown in Table 3. However, the percentage spectrum is divided into evenly spaced intervals. The model’s categorical output may influence the model’s inference on certain quantifiers that can cover a large range. \n\n    b. Can the framework be adapted to a value regression setting, which may better fit the way that humans interpreted it?\n\n3. Framework clarification: the RSA framework is good at solving tasks that need to speak or comprehend sentences heavily dependent on the current dialogue context. Could you further clarify the following in your task:\n\n    a. How the dialogue context influences the quantifier semantics influences in this task? \n\n    b. And, how this influences in (a) reflected in terms of equations (1-4)? From my understanding, the pragmatic listener in equation (2) only takes additional information about the word frequency compared with the literal listener but not grounds in the current sentence context.\n\n    c. How the entailment score is computed?", "paper_topic_and_main_contributions": "This paper introduced a new corpus QuRe for quantifier semantics inferences. Given quantifiers in each sentence, it is annotated with a percentage scope. A framework PRESQUE combing natural language inference with Rational Speech Acts framework is proposed to predict the percentage scopes for quantifiers. Detailed annotation procedures and statistical analysis are presented. Preliminary experimental results show that PRESQUE can perform better than a literal listener on various evaluation", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EivO1ByeJJ", "reviewer": "Reviewer_VNWH", "summary": "", "questions": "Question A: Fleiss kappa for quantifier choice was only 0.37 Fleiss' kappa for 3 annotators.  How often did the annotators choose 3 different quantifiers? Will the release of QuRe include all three annotations for each instance (as in the Herbelot-Vecchio dataset)?\n\nQuestion B: Will the crowd-sourced percentage interpretations for the 5 quantifiers in the Herbelot-Vecchio by 25 judges be published as a dataset?\n\nQuestion C:  I don’t understand the relevance of Pandia et al. (2021)’s finding for distance-based metrics (line 514). Is this the correct reference?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate  the reviewer’s detailed feedback and insightful comments. We address the concerns below:\n\nData leakage of Wikipedia: We use Wikipedia since it is the best source, to the best of our knowledge, with rich percentage mentions that are relatively objective compared to other choices. While we acknowledge the concern about generalization and potential impacts from pretraining exposure, we would like to emphasize that the mere exposure to these sentences doesn't necessarily equate to the direct learning of quantifier semantics (since we recruit annotators to provide the quantified version of those sentences which may not appear in Wikipedia). We hope our work could serve as an exploratory step to collect better benchmarks in the future.\n\nDistribution of generalized quantifiers: We found the imbalance in the use of generalized quantifiers to be related to the natural preference of annotators (e.g. people might be hesitant to provide absolute quantification like ‘none’ or ‘all’), and we would lose a large proportion of the annotations if we plan to balance the quantifiers. We don't find a succinct approach to represent the scope distribution in a figure as the percentage scopes are in diverse values and formats, which is why we apply percentage scope generation based on a few hyperparameters (Section 4 and Appendix E).\n\nRegarding the linguistic soundness, we extended the quantifier set with adjectives based on the limitation of only using adverbs we observed in preliminary experiments and extended the coverage. However, we are aware that the entire set of generalized quantifiers can be much larger and requires future efforts to incorporate. We also decided to leave the choice of nouns that attach to those adjectives to the annotators as it will not significantly change the strength of quantifiers.\n\nThoroughness/Informativeness: We will attach a more fine-grained discussion in our final version.\n\nSoundness:  The p-values of chi-square test for No, Few, Some, Most, All are 0.258, 0.44, < 1e-6, 0.258 and 0.02. Therefore, as the reviewer correctly guessed, differences in results for Some and All are significant.\n\nImproving clarity: We thank the suggestion to improve clarity and pointing out redundant references. We will make appropriate changes in the final version.\n\nAddressing questions posed by the reviewer:\n\nQuestion A: In 19% of the cases, all the annotators choose different quantifiers. We will provide all three annotations when releasing the dataset.\n\nQuestion B: Yes, we will release those annotations.\n\nQuestion C: Thanks for pointing this out. The correct reference would be Carcassi and Szymanik, 2021. [1]\n\n[1] Fausto Carcassi, Jakub Szymanik, ‘Most’ vs ‘More Than Half’: An Alternatives Explanation, SCiL 2021\n\n\n\nWe thank the reviewer for providing the detailed text and reference correction, as well as writing suggestions, we would incorporate them in the final version. \n\n\nWe address other comments posed by the reviewer below:\n\n- Section 3: Why are the HVD triples converted into sentences for collecting the wikipedia sentences? Why not just use the concept nouns? (lines 232 ff.) It is confusing that this information is given in section 3 (on QuRe), “Stage 1: Wikipedia sentences collection”.\n\nThere are no limitations for using sentences or concepts for the sentence collection. In practice, we found that using the sentences returned more diverse wikipedia entities since one concept might be associated with more than one sentence.\nSection 3, Statistics: Table 3 should include absolute frequencies of the quantifiers.\nThe absolute strengths (if that is what the question means) of quantifiers cannot be determined without hyperparameters like granularity and window size since the percentage mentions usually refer to scopes (Table 2 includes some examples).\nThe reason for using HVD concepts as seeds for collecting wikipedia sentences is not explained.\nSince we use an off-the-shelf entity linking library, we need a seed corpora to start the searching, and therefore we choose the HVD concepts as the start. However, the procedure can be easily substituted with other resources.\n\n- 223 Clarification needed: “some is used in 25% of sentences” (L223) vs. figure 1: proportion of “some” is 26,3% of all quantifiers?\n\nWe missed the word ‘over’, thanks for pointing it out.\n\n- 323: \"We have collected 746 ˜ Sq sentences, of which 74% and 13% contain one and two percentage mention(s), respectively.\" => What about the other 13 %?\n\nThe other sentences include more than two percentage mentions.\n\n- 512 Which distance-based metric?\n\nThe distance-based metric we use is minimal scope distance (MSD) which we elaborate in Appendix G. We would rephrase it in a more explicit way in L512 in the final version. \n\n- 514 I don’t understand the relevance of Pandia et al. (2021)’s finding for distance-based metrics. Is this the correct reference?\nThanks for pointing it out. We have included the correct reference in the answer to Question C521 Why is “only a few school children were recognized...\"” fully determinable?\n\nThanks for pointing out our mistake. The original sentence was “In the 2017 Dutch study only 2 out of the total 27 school children (7%) recognized that the website was a hoax.” (which also shows up in Table 8) and we mistakenly placed the corresponding sentence with the quantifier in the text. We would correct this in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper makes a contribution to the NLP interpretation of generalized quantifiers (GQs), which are frequent in natural language communication but have been mostly neglected in NLP. GQs are notoriously vague in their meaning but for some applications  (e.g. in human-robot interaction) it is necessary to interpret the quantification in percentages. The paper offers an interesting pragmatically inspired algorithm to determine percentage interpretation on the basis of existing LLMs that are fine-tuned for NLI.\n\nIt also introduces the new dataset QuRe of sentences that feature GQs. The QuRe sentences are automatically (GPT-3.5-turbo) annotated with percentage range (derived from the original sentences that included percentage information plus potential modifications such as \"about\", \"at least\")  and 3 topic labels. They are also manually annotated with a 3-level “specificity” score which indicates whether the sentence without the quantifier gives partial or full information about the percentages or whether it is indeterminable from the context.\n\nFor the 5 quantifiers in the Herbelot-Vecchio dataset percentage interpretations are crowd-sourced from 25 human judges and visually compared to (RoBERTa-large NLI-based ) PRESQUE results.\n\nA random sample of the results on the Herbelot-Vecchio dataset is evaluated by human judges.\n\nThe results on the QuRe dataset are evaluated by employing five evaluations measure  that emphasize different properties (HIT@1, MRR, Cross-Entropy, F1@n, and consecutiveness}. Although not good, the results are better than (random seeds) baselines and the generall quantifier strengths correspond to previous hierarchies based on human judgements.", "reasons_to_reject": "Origin of QuRe und its distribution of GQs: The QuRe dataset is based on Wikipedia sentences that include the target percentages which means that the LLMs have most likely seen the target interpretation in their pre-training data. That decreases the usefulness of the data set. \n\nIn addition, the distribution of the quantifier expressions in QuRe is very skewed (26,3 % \"some\" aka about 196 instances but less than 1% aka 7 or less instances of \"likely\", \"seldom\", \"occasionally\", \"none\" - the authors only provide relative frequencies). The distribution of percentage scopes is not reported, but doesn't seem to be balanced either.\n\nLinguistic soundness: The set of \"quantifier words\" is very heterogenous, some are not normally seen as quantifiers: The reference to Srivastava et al. (2018) accounts for the inclusion of frequency adverbs such as \"usually\", \"rarely\", but the inclusion of bare adjectives such as \"tiny\"or \"large\" without a noun such as \"part\" or \"amount\" requires further explanation (I don't think that it is a problem that they are listed in the crowd-sourcing task without explanation, though).\n\nThoroughness/Informativeness: The quantitative evaluations on QuRe report only averaged values over all quantifiers.\n\nSoundness: The human evaluation on the performance of L0 vs. L1 on the 50 test sentences that were randomly chosen from the Herbelot-Vecchio dataset (10 sentences per each of the 5 quantifiers) evaluated by 40 annotators is not tested for statistical significant differences.  [In favour of the authors' interpretation: If I interpret the numbers in figure 7 correctly the results for \"some\", \"all\" and the overall values are statistically significant (based on a chi-square test) and do indeed prefer the output of the PRESQUE L1].\n\nClarity: The paper works with two datasets with two different sets of \"quantifiers\". The paper should always indicate which dataset is referred to,  e.g. in all table/figure captions (as it is the case in figure 1 and table 3).", "questions_to_authors": "Question A: Fleiss kappa for quantifier choice was only 0.37 Fleiss' kappa for 3 annotators.  How often did the annotators choose 3 different quantifiers? Will the release of QuRe include all three annotations for each instance (as in the Herbelot-Vecchio dataset)?\n\nQuestion B: Will the crowd-sourced percentage interpretations for the 5 quantifiers in the Herbelot-Vecchio by 25 judges be published as a dataset?\n\nQuestion C:  I don’t understand the relevance of Pandia et al. (2021)’s finding for distance-based metrics (line 514). Is this the correct reference?", "paper_topic_and_main_contributions": "The paper introduces an algorithm for interpreting generalized quantifiers (GQs) semantically in terms of percentage ranges (PRESQUE). The interpretation algorithm is theoretically grounded in Gricean pragmatics of modelling a listener who takes into account assumptions about the speaker’s intentions. This is operationalized in a Bayesian approach by using NLI relations between sentences with quantifier and sentences with percentages as well as relative quantifier frequencies.\n\nSecond, the paper introduces a new test suite (QuRe) for probing the ability of LLMs to interpret English GQs in terms of percentage ranges. In particular, it consists of 746 sentences that contain one of the following 15 “quantifier words”: {all, generally, most, usually, some, likely, few, little, occasionally, none, seldom, tiny, small, moderate,  large}. The sentences are based on Wikipedia sentences that include percentage information, which were manually rewritten as semantically equivalent sentences with a quantifier expression. \n\nThird, the interpretations of five “classic” GQs (all, most, some, few, no) are crowdsourced in terms of percentage ranges.\n\nThe performance of LLMs (and in particular RoBERTa-large) fined-tuned for NLI and of the PRESQUE algorithm are evaluated using (1) the independently existing Herbelot-Vecchio dataset (featuring the five classic GQs), and (2) the newly introduced QuRe dataset (featuring the 15 \"quantifier words\" above). The results show that the LLMs grasp general differences between GQs scopes but have limited predictive power for percentage ranges. The authors conclude that the pragmatic PRESQUE algorithm produces better percentage predictions than a \"literal\" NLI based predictor.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Bowman at al. 2015a,b, refer to the same publication."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* 140 “HVD” => the acronym for the Herbelot-Vecchi dataset has not been explicitly introduced in the main text, only in the abstract.\n* Section 3: Why are the HVD triples converted into sentences for collecting the wikipedia sentences? Why not just use the concept nouns? (lines 232 ff.) It is confusing that this information is given in section 3 (on QuRe), “Stage 1: Wikipedia sentences collection”.\n* Section 3, Statistics: Table 3 should include absolute frequencies of the quantifiers.\n* The reason for using HVD concepts as seeds for collecting wikipedia sentences is not explained.\n* 223 Clarification needed: “some is used in 25% of sentences” (line 223) vs. figure 1:  proportion of “some” is 26,3% of all quantifiers?\n* 288 “Therefore” => maybe rather: This step ensures ….??\n* 323: \"We have collected 746 ˜ Sq sentences, of which 74% and 13% contain one and two percentage mention(s), respectively.\" =>  What about the other 13 %? \n* Section 5.1. Figure 3:  the number of human interpreters (n=25) should be provided in the caption also the number of instances per quantifier analyzed by PRESQUE.\n* Section 5.2 Table 5 => the number of human annotators (n=40) should be provided in the caption (or the actual number for each sentence, if it differed per sentence).\n* 512 Which distance-based metric?\n* 514 I don’t understand the relevance of Pandia et al. (2021)’s finding for distance-based metrics. Is this the correct reference?\n* 521 Why is “only a few school children were recognized...\"” fully determinable?\n* Table 7: Table 7 should be explicitly anchored in Appendix A (cf. caption of table 1). The header QuRe “sentence” is not clear, because some examples in Table 7 include more than one syntactic sentence.\n\nTypos:\n\n* 321 “and slightly extended” => and was is/slightly extended\n* 336 “over 30% of cases” => over 30% of the cases\n* 337 “most is selected” => … is selected in\n* 339 “18% of sentences” => … of the sentences\n* 341 “and 48% indeterminable” => … are indeterminable\n* 346 “and shows interesting hierarchies:” => incomplete clause\n* 380 “compute following metrics” => compute the following metrics\n* 460 “We further calculate L0 result through Eq. 1.” => … the L0 result using Eq. 1”\n* check: fine-tuned, cross-entropy for consistent spelling\n\nBibliography:\n\n* Partly incomplete entries"}}, {"review_id": "l2WtFnw8eX", "reviewer": "Reviewer_PLvc", "summary": "", "questions": "- Is there potential for research on pragmatic quantifiers to be used in quantifier-related inference in 1st-order logic formalism parsed from natural language? I wonder if it is possible to formalize arguments using existential and universal quantification from the range of quantifiers.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing and suggesting the impacts of our work. Applying the pragmatic-inference framework of PRESQUE to other inference and reasoning tasks that involve quantifiers (as part of the input) would be an interesting direction. As the reviewer pointed out, some adaptations and attention to the designs are needed, like formalizing the arguments with existential and universal quantification, which are supported by the current 1st-order logic framework.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The motivation and problem statement of the paper is clear, and the theoretical background and experimental design to support the authors’ claims are clearly explained, making the text itself highly complete.\n- The attempt to limit the range of Generalized Quantifiers using the perspective of pragmatics seems novel, and the QuRe dataset also has a versatility that can be useful in the NLI field where quantifier-related reasoning is required in the future.", "reasons_to_reject": "- I have no reason to reject.", "questions_to_authors": "- Is there potential for research on pragmatic quantifiers to be used in quantifier-related inference in 1st-order logic formalism parsed from natural language? I wonder if it is possible to formalize arguments using existential and universal quantification from the range of quantifiers.", "paper_topic_and_main_contributions": "- This paper presents QuRe, a dataset for converting generalized quantifiers into percentages based on crowdsourcing, and proposes the PRESQUE framework, which combines an RSA framework based on NLI and Gricean theory.\n- PRESQUE shows an improvement of about 20% in predicting the percentage scopes of generalized quantifiers.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper conceptualizes generalized quantifier semantics as percentage ranges, and introduces 1) an annotated dataset of generalized quantifiers in Wikipedia sentences, and 2) a model framework combining NLI and RSA to predict percentage ranges. The paper shows that the proposed pragmatic model outperforms a “literal” RoBERTa-large model fine-tuned on NLI.

The reviewers express that the paper is clear and well-motivated, addressing a worthwhile and understudied topic, and contributing a valuable corpus and an interesting pragmatically-driven model. Some concerns are raised about clarity and soundness, but these have largely been addressed in the discussion period. Overall, this paper will likely be a solid, interesting, and novel contribution to the conference.
Decision: Accept-Main