SlowGuess/NAACL_2022_OCR huggingface dataset:
Ruixiang Cui, Daniel Hershcovich, Anders Søgaard
{rc，dh，soegaard}@di.ku.dk
Logical approaches to representing language have developed and evaluated computational models of quantifier words since the 19th century, but today's NLU models still struggle to capture their semantics. We rely on Generalized Quantifier Theory for language-independent representations of the semantics of quantifier words, to quantify their contribution to the errors of NLU models. We find that quantifiers are pervasive in NLU benchmarks, and their occurrence at test time is associated with performance drops. Multilingual models also exhibit unsatisfying quantifier reasoning abilities, but not necessarily worse for non-English languages. To facilitate directly-targeted probing, we present an adversarial generalized quantifier NLI task (GQNLI) and show that pre-trained language models have a clear lack of robustness in generalized quantifier reasoning.
Quantifier words—such as each or most or more than three—have been extensively studied, both in logic and in linguistics (Westerståhl, 1989; Peters and Westerståhl, 2006), going all the way back to Frege (1879). In this paper, we examine the extent to which they present a challenge to modern NLU systems. Our analysis is motivated by three observations:
Quantifier words are abstract Unlike nouns, verbs and adjectives, quantifier words do not have referents out in the world. Rather, quantifier words specify relationships between sets of entities, events and properties. To provide intuitions about the semantics of quantifier words, and to be able to refer to quantifiers in a language-independent way, we rely on the notion of generalized quantifiers (Mostowski, 1957), as described in §2.
Quantifier words vary across languages Quantifier word inventories differ across languages.
QA_English CONTEXT: A piece of paper was later found on which he had written his last statements in two languages, Latin and German. Only one statement was in Latin and the rest in German. QUESTION: In what language were most statements written? ANSWER: German PREDICTED AN- SWER: Latin and German
PREMISE: Más de tres personas resultaron heridas en un accidente de dos vehículos elunes por la noche. (translation:More than three people were injured in a two-vehicle crash Monday evening.) HYPOTHESIS: Había 4 personas involucradas. (translation:There were 4 people involved. LABEL: Neutral PREDICTED LABEL: Entailment
Table 1: Examples of quantifiers (marked in bold texts) in NLP tasks, with RoBERTa's prediction for QA and XLM-R's prediction for NLI after fine-tuning.
Often what is considered rough translation equivalents also differ in syntax, fine-grained semantics or pragmatics. Stateva et al. (2019) show, e.g., that perceptions of the numerical bounds of existential quantifiers differ across speakers of English, French, Slovenian, and German. Other papers showing discrepancies between quantifier systems include comparisons of Salish to English (Matthewson, 2001), Adyghe to English (Nikolaeva, 2012), or of Dutch, Hebrew and Bengali (Gil, 1982). The cross-linguistic differences in how generalized quantifiers are expressed motivates a cross-lingual error analysis, since quantifiers may contribute more to error when processing some languages rather than others.
Quantifier words are important Quantifier words are extremely important for tasks that require inference, including natural language inference, question answering, fact-checking, etc. Datasets have, for example, been developed for numerical reasoning in English (Dua et al., 2019). Several researchers have identified quantifier words as important sources of errors for natural language processing systems (Joshi et al., 2020); see Table 1 for examples of such errors. Unfortunately, most
Human Language Technologies, pages 4875 - 4893
Generalized Quantifiers	Logical Denotation	Example
some(A)(B) = 1	A∩B≠∅	This process is known to increase security in several ways.
all(A)(B) = 1	A⊆B	Everyone agreed the food was terrible.
more than k the(A)(B) = 1	|A∩B| > k	They do let them go more than twice a week.
less than k the(A)(B) = 1	|A∩B| < k	San Augustin Acolman has less than 1,000 residents.
k (A)(B) = 1	|A∩B| = k	Please donate 100 million to the School of Nursing.
between p and k the(A)(B) = 1	p < |A∩B| < k	The USA added ten states to its nation between 1800 and 1850.
the p/k (A)(B) = 1	|A∩B| = p·(|A|/k)	Captain Blood has 20/20 vision.
the k% (A)(B) = 1	|A∩B| = k·(|A|/100)	The lending fund is always guaranteed 9% interest.
most (A)(B) = 1	|A∩B| > |A\B|	Most ZIP Codes cover roughly ten thousand addresses.
few (A)(B) = 1	|A∩B| < |A\B|	Only a few teenagers were still listening to Rock ’n’ Roll.
each other (A)(B) = 1	∀a ∈ (A∩B)∃b ∈ (A∩B)(a ≠ b)	All of these trails are located within the a one hour drive of each other.
Table 2: The categorization set of quantifiers for task analysis. The first six are Aristotelian/counting quantifiers and the following four are proportional quantifiers. The last one is a Ramsey quantifier (Schmerl and Simpson, 1982). For each quantifier, its logical denotation is listed in the second column. The third conlumn contains English examples with quantifiers taken from XNLI.
efforts have concentrated on subsets of quantifier words and on English.
Contributions We analyze how quantifiers are represented in NLU benchmarks, and how their occurrence at test time contributes to errors by neural language models (LMs). We derive a linguistically motivated 11-way categorization set for generalized quantifiers and look into their distribution in three steps: (a) monolingual NLI; (b) cross-lingual NLI; (c) cross-lingual question answering. We also propose GQNLI<sup>1</sup>, an adversarial generalized quantifier NLI challenge dataset. Our work shows that (i) generalized quantifiers are pervasive and cause overall performance drops in NLU benchmarks; (ii) the contribution of quantifier words to system error varies across languages; and (iii) generalized quantifiers are particularly difficult for LMs in interaction with negation and subsumption.
Generalized quantifiers (GQs) are developed upon first-order predicate logic, denoting relations between sets (Mostowski, 1957). Given a universe  $E$ , a quantifier  $Q$  would be treated as a mapping  $Q_{E}$  from the Cartesian product of powersets  $\mathcal{P}(E)\times \mathcal{P}(E)$  to the set  $\{\text{false, true}\}$  or, as a binary relation on subsets of  $E$  (Dvořák and Holcapek, 2015). GQs are generalizations of the  $\forall, \exists$  quantifiers from first-order predicate logic (Mostowski, 1957; Lindström, 1966; Montague, 1973; Bach et al., 1995; Keenan and Paperno, 2012). A generalized quantifier is, abstractly, a relation between sets. Generalized quantifier theory, while developed by logicians, is used by formal linguists to analyze the
meaning of quantifier words in combination with referential expressions (Barwise and Cooper, 1981; Higginbotham and May, 1981).
Most human languages contain ways of expressing generalized quantifiers, and their semantics exhibit striking similarities across languages (Matthewson, 2004; Fintel and Matthewson, 2008; Steinert-Threlkeld, 2019). At the same time, generalized quantifiers can be instantiated very differently across languages due to pragmatic considerations (Grice, 1989) or cognitive economy and cost-benefit optimisation in the exchange of information (Levinson et al., 2000; Steinert-Threlkeld, 2021; Uegaki, 2022). Quantifier words also exhibit syntactic differences, e.g., with some languages having specialized words to express quantity, while others rely on metaphorical usage of common nouns (Katsos et al., 2012). In English, most is a determiner, but Spanish and French express the same concept through common nouns, la mayoría and la majorité. The relative stability of the core semantics of quantifiers makes a cross-linguistic comparison possible, but the syntactic and pragmatic variation associated with the expression of generalized quantifiers poses a challenge for multilingual NLU. We consult quantifier taxonomy studies (Keenan and Westerstål, 1997; Peters and Westerstål, 2006; Szymanik and Thorne, 2015; Szymanik, 2016) and derive a categorization set for quantifier analysis in NLU benchmarks. In Table 2, we list the 11-way quantifier categorization set and their logical denotation based on set theory.
While other foci of formal linguistics have attracted the attention of NLP researchers—including coreference (Ogrodniczuk et al., 2019, 2020), nega
https://github.com/ruixiangcui/GQNLI
Quantifier	English	Cross-lingual
MNLI_m	MNLI_mm	SNLI	ANLI_R1	ANLI_R2	ANLI_R3	XNLI
some	171	132	191	5	1	17	115
all	255	239	65	15	8	29	166
>k	14	23	8	10	16	14	16
<k	3	3	0	6	7	5	1
k	266	269	988	55	62	48	159
between	2	3	0	3	2	0	1
p/k	1	5	1	1	1	0	2
k%	10	7	0	0	0	1	5
most	35	39	1	0	2	1	9
few	14	15	11	0	0	6	11
each other	4	3	35	0	0	2	5
Total	775	738	1300	95	99	124	499
Frequency	7.9%	7.5%	13.2%	9.5%	9.9%	12.4%	10.0%
Table 3: Quantifier distribution in four NLI tasks, among which three are monolingual English and one is cross-lingual. The table show statistics of the test set, if not available, dev set, of the target task. All but the last rows show the occurrence time of the type of quantifier in the first column. The last row represents the distribution rate of any quantifier in the dataset.
tion (Hossain et al., 2020; Hartmann et al., 2021), and consistency (Li et al., 2019; Ribeiro et al., 2019; Asai and Hajishirzi, 2020; Geva et al., 2022)—there has been little work on generalized quantifiers as a source of error in NLU, let alone in multilingual NLU. It remains an open problem whether LMs represent the semantics of quantifiers words adequately, or if they provide a basis for resolving scopal ambiguities.[2]
3 NLU Benchmarks
We conduct an error analysis focusing on the role of generalized quantifiers in two NLU tasks, Natural Language Inference (NLI) and Question Answering (QA), which generally require understanding of quantifiers. For each type of task, both monolingual and cross-lingual evaluation are conducted. We focus on generalized quantifiers in the hypotheses in NLI examples—and on generalized quantifiers in the question fields in question answering. To this end, we identify quantifiers by the lemma and the universal dependency relation (Nivre et al., 2020) of a quantifier after preprocessing the sentences using Stanza (Qi et al., 2020). Take the sentence "The Yiddish culture has survived for more than a thousand years," we annotate it as
Figure 1: Relative distribution of quantifiers in NLI and QA tasks ranked by semantic complexity. The bars show the relative frequency of such quantifier and the lines indicate the cumulative frequency for a task.
"The/det Yiddish/amod culture/nsubj have/aux survive/root for/case more/advmod than/fixed a/det thousand/nummod year/obl .punct". By matching the regex pattern of the quantifier "more than k", in this case ((more|great)/advmod than/V(fixed|case)|at|case least/Vnmod).+Vnummod.+V(nsubj|obj|obl) ", we approximate the surface form of the type "more than k".Through matching quantifier patterns, we are able to find entries in which quantifiers are instantiated. See Appendix A for the list of regex patterns we write to identify GQs. In Table 3 and Table 6, we present the statistics of the quantifier distributions in NLI and QA tasks, respectively. As can be seen, quantifiers are indeed widespread in NLU tasks, accounting for roughly  $10\%$  in NLI tasks and  $5\%$  in QA tasks. We will further discuss the statistics and experiments in the following section.
4 Quantifiers in English NLI Benchmarks
NLI is commonly framed as a three-way classification task with labels entailment, contradiction and neutral (Bowman et al., 2015a). While SOTA models exhibit low error rates on NLI benchmarks, it is unclear when they succeed or fail in their underlying reasoning. We are interested in whether generalized quantifiers challenge modern NLI models. In our error analysis, we initially focus on three English NLI datasets, MultiNLI (MNLI; Williams et al., 2018), SNLI (Bowman et al., 2015a) and ANLI (Nie et al., 2020) as testbeds.
Table 3 presents statistics of quantifier distribution in these datasets, where we observe that,
2Note that generalized quantifiers are not always explicit in discourse. The sentence inadequate sleep causes obesity should be interpreted as Most of those who do not sleep adequately, gain weight (Zadeh, 1983). Such implicit quantifiers related to pragmatic variation are important for language understanding, but will be ignored in this work.
Quantifier	BERT	RoBERTa
M_m	M_mm	SNLI	A_R1	A_R2	A_R3	weig.	M_m	M_mm	SNLI	A_R1	A_R2	A_R3	weig.
some	82.5	84.1	86.9	100	0	47.1	83.4	83	84.8	86.9	100	100	41.1	83.7
all	85.9	88.3	89.2	46.7	37.5	34.5	83.2	85.9	92.1	90.8	66.7	37.5	34.5	85.3
>k	85.7	100	87.5	70	43.8	42.9	73	85.7	91.3	87.5	80	37.5	28.5	68.2
<k	100	100	33.3	57.1	80	66.7	100	100	83.3	85.7	100	91.7
k	87.2	81.8	92.4	43.6	43.5	33.3	84.8	88.3	88.8	92.9	56.3	61.3	43.8	87.8
between	100	100	66.7	50	80	100	66.7	66.7	50	70
p/k	100	60	100	100	100	77.8	100	80	100	100	0	77.8
k%	90	100	100	94.4	70	85.7	0	72.2
most	74.3	79.5	0	50	0	74.4	77	87.2	100	59	0	80.9
few	78.6	73.3	90.9	33.3	73.9	85.7	80	90.9	33.3	78.3
each other	75	100	85.7	50	84.1	50	100	88.6	50	84.1
all GQs	85	84.8	91.2	50.5	44.4	39	83.3	85.4	88.8	91.7	65.3	56.5	40.3	85.5
full	86.5	86.1	91.3	58.6	48	43.2	84.4	89.5	89.4	92.3	71.7	49.6	49	87.3
Table 4: BERT and RoBERTa performance on NLI tasks. The weig. column represents the percentage of all true predictions in six subtasks over total instances. The penultimate row stands for the overall performance when quantifiers exist in a dataset. The last row reports the overall performance in a dataset. Number marked in bold signifies a lower score than the overall performance.
across, about  $10\%$  of all hypotheses contain quantifier words, indicating the pervasiveness of quantification. We also plot the frequency of quantifiers in NLI in Figure 1 and find the quantifier word distribution follows Zipf's law (Zipf, 1949). Note the top three most common quantifiers account for more than  $90\%$  of all.
Experiments and Results In order to investigate whether NLU systems can solve quantifiers in NLI, we experiment with two pretrained LMs: BERT $^3$  (Devlin et al., 2019) and RoBERTa $^4$  (Liu et al., 2019). We use the codebase by Nie et al. (2020). The training data combines SNLI, MNLI, FEVER-NLI (Nie et al., 2019) and ANLI.
In Table 4, we report the test set performance on SNLI and ANLI, and the dev set performance on MLNI matched and mismatched sections. We can observe that SOTA models suffer from performance drops across almost all quantification phenomena in every task. When it comes to performance over all quantifiers, the improvement from RoBERTa to BERT  $(2.2\%)$  is less prominent than that over full datasets  $(2.9\%)$ , suggesting RoBERTa is particularly challenged.
Taking a closer look at error by category, proportional quantifiers seem harder to solve than Aristotelian/counting quantifiers. Except for  $k\%$ , all proportional quantifiers— $p/k$ , most, and few—are about  $10\%$  lower than the five counting quantifiers (except less than  $k$ ) with BERT; and about  $5\%$  lower with RoBERTa. RoBERTa is not generally
superior to BERT; e.g., for  $k\%$ , BERT outperforms it by  $22\%$ . We show a pairwise analysis of how GQs affect performance when they appear in both the premises and hypotheses in the Appendix B. Generally, our results attest to the difficulty of resolving GQs in NLI benchmarks.
5 Quantifiers in Cross-lingual NLU Benchmarks
Quantifiers are acquired in similar orders across languages (Katsos et al., 2016), although languages express quantifiers in different ways. For example, there are eight different universal quantifiers with different level of distributivity in Malagasy (Matthewson, 2008). This poses challenges to training multilingual LMs and transfer learning. We are interested in whether quantifiers are universally and evenly challenging for all languages.
Quantifiers in Cross-lingual NLI We choose XNLI (Conneau et al., 2018), a manual translation of the development and test set of MNLI into 15 languages, for this multilingual error analysis. We should clarify that for XNLI, the authors annotate entailment labels for the English data only and apply them to the other languages. We do not assume label changes due to translation in this study, but it is worth investigate in the future. We choose five languages belonging to different language families, namely Arabic, Chinese, German, Spanish and Vietnamese as targets. The last column in Table 3 shows the numbers of quantifiers in XNLI. The distribution rate is  $10\%$ . Note that the universal quantifier is the most common quantifier in XNLI.
3wwmcased_L-24_H-1024_A-16
roberta-large
Quantifier	en	zh	es	ar	vi	de	weig.	en	zh	es	XLM
ar	vi	de	weig.
some	85.2	69.6	80	63.5	67.8	74.8	73.4	85.2	70.3	79.1	71.3	73.9	69.6	69.6
all	80.1	65.7	72.8	69.3	63.9	74.1	70.9	82.5	62.7	74.1	67.5	71.7	73.5	72
>k	87.5	50	68.8	43.8	56.2	62.5	61.6	81.2	62.5	56.2	62.5	50	75	75
<k	100	100	100	100	100	100	100	100	100	100	100	100	100	100
k	86.2	69.1	80.5	71.7	76.7	82.4	77.7	83	66.7	78.6	71.7	74.2	81.1	75.8
between	100	100	100	100	100	100	100	100	100	100	100	100	100	100
p/k	100	50	100	100	100	100	91.7	100	0	100	100	50	50	66.7
k%	100	100	80	100	100	100	96.7	80	80	80	100	100	80	86.7
most	55.6	55.6	66.7	66.7	33.3	66.7	57.4	55.6	33.3	66.7	55.6	44.4	77.8	55.6
few	72.7	54.5	72.7	63.6	45.5	72.7	63.6	63.6	36.4	54.5	63.6	54.5	72.7	57.5
each other	60	60	60	60	80	80	66.7	80	20	60	20	40	60	46.7
all GQs	83	67.1	76.7	68.1	68.3	76.9	73.3	82.4	64.2	75.7	69.3	71.4	74.8	73
comp.	82.6	88.9	74.7	65.6	70.7	71.4	72.4	83.1	64.8	76.3	66.9	71.6	71.3	72.3
Table 5: Results of mBERT and XLM performance on XNLI tasks decomposed by quantifier categories.
Quantifier	MLQA	XQuAD
en	zh	es	ar	vi	de	...
some	66	39	41	44	37	33	12
all	31	14	26	21	19	16	7
< k	1	0	0	0	1	0	0
k	322	168	166	195	204	149	32
between	4	2	2	2	3	0	3
p/k	1	1	1	0	0	0	0
k%	1	1	0	1	0	0	0
most	27	19	11	30	17	9	5
Total	453	244	247	293	281	207	59
Frequency	3.9%	4.7%	4.7%	5.4%	5.1%	4.5%	5.0%
Table 6: Quantifier distribution in two multilingual QA tasks, MLQA and XQuAD. We choose six common languages appearing in both tasks to facilitate comparisons. XQuAD is strictly parallel while MLQA is not, hence only the latter has statistics by languages. Categories that no entry exists are omitted.
We fine-tune mBERT $^5$  (Devlin et al., 2019) and  $\mathrm{XLM^6}$  (Lample and Conneau, 2019) on the MNLI training set and evaluate them on XNLI. We report the results in Table 5. We find that performance varies across languages. For Chinese and Vietnamese, we see significant drops in performance for examples with GQs, whereas for Arabic and German, we see improvements. The results per quantifier are more homogeneous, however.
Similar to our results for English, we can see that the lowest accuracies in XNLI are with proportional quantifiers, such as most and few. But the gap in non-English languages is wider for these two categories, especially for Chinese, the difference reaches  $30\%$ . Other hard quantifiers include all,  $>k, < k$ , and each other.
Quantifiers in Cross-lingual QA Cross-lingual question answering (XQA) is another important NLU task that evaluates the cross-lingual transferability of LMs. We evaluate the effect of quantifiers on system errors across two XQA datasets, namely XQuAD (Artetxe et al., 2020) and MLQA (Lewis et al., 2020). As demonstrated in Figure 1, quantifier word distributions in XQA tasks also follow Zipf's law, as in NLI tasks, but  $k$  is more frequent (perhaps because of a traditional emphasis on numerical reasoning), and we see less variance across languages. This is probably because question answering is targeting quantification less directly. To evaluate cross-lingual QA performance on GQs, we fine-tune mBERT and XLM-R $^{7}$  (Conneau et al., 2020) using Hu et al. (2020)'s architecture. We present results for mBERT in Table 7; for XLM-R results, please refer to Appendix D.
Just as with XNLI, LMs suffer from performance drops across all languages for almost all GQ phenomena with significant, cross-lingual variation. The most distinguished is that Exact Match (EM) suffers from a greater deterioration than F1 scores for all languages. For example, the weighted EM difference for mBERT on MLQA is  $2.9\%$  while the weighted F1 is  $1\%$ . As one example in Table 1, we observe that the plausible answers selected by models, while being incorrect, result in a sharper decrease of EMs comparing to F1s. Questions containing GQs also tend to have less verbal answers comparing to those without GQs, and therefore require higher precision.
Regarding cross-lingual comparisons, Chinese and Arabic are the two languages that do not have
multicased_L-12_H-768_A-12
$6_{\mathrm{x1m - mlm - 100 - 1280}}$
7x1m-roberta-large
Quantifier	en	zh	es	XQuAD	vi	de	weighted	en	zh	es	MLQA	vi	de	weighted
EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1
some	75	84.2	50	55.5	58.3	76.1	50	50	16.6	42.4	33.3	43.8	47.2	58.7	59	80	28.2	52.1	34.1	59.2	36.3	54.9	5.4	24	33.3	58.4	32.7	54.8
all	28.5	62.2	14.2	35.2	28.5	82	42.8	52.3	14.2	29.4	28.5	56	26.1	52.9	67.7	79.8	14.2	46.4	38.4	62.8	33.3	57.9	10.5	30.1	31.2	51.6	32.6	54.8
< k	0	0	0	13.3	0	6.7
k	78.1	90.1	68.7	80.4	56.2	72.1	40.6	64.3	12.5	35.7	56.2	77.1	52.1	70	74.9	79.4	47	63.4	41.5	65.9	27.6	50.3	6.3	23.7	38.2	53	39.3	56
between	100	100	33.3	72.2	66.6	93.3	100	100	0	19	0	56.5	50	73.5	50	88.5	50	83.3	0	26.6	0	68.7	0	26.6	20	58.7
p/k	100	100	0	0	0	0	33.3	33.3
k%	100	100	0	26.6	0	23.7	33.3	50.1
most	40	53.3	40	40	0	10	0	26.6	0	0	20	49.3	16.7	29.9	55.5	76	47.3	62.1	45.4	61.7	30	46.8	5.8	15.7	33.3	40.7	36.2	50.3
all GQs	70	83.2	55	66.7	50	70.3	41.6	58.2	11.6	32.5	43.3	65	45.3	62.7	63.5	79.2	41.8	60.3	39.6	63.7	29.3	51.3	6.4	23.6	36.1	53.2	36.1	55.2
comp.	71.8	83.7	48	59.1	56	74.5	40.8	57.9	13.9	32.4	50.7	67.2	46.9	62.5	67.2	80.6	37.5	57.9	47.3	66	30	48.4	11.2	28	40.8	56	39	56.2
Table 7: Results of mBERT performance on XQA tasks decomposed by quantifier categories.
lower performance over GQs compared to the performance over the complete dataset. Despite the overall trends, subtle differences from XNLI performance still exist. For example, XLM-R is worse than mBERT on quantifier reasoning on XQuAD Chinese, especially at proportional quantifiers, but this is not the case on MLQA Chinese.
6 GQNLI
We have seen how quantifiers present challenges to NLI and QA models. Using an approach similar to ANLI (Nie et al., 2020) and DynaBench (Kiela et al., 2021), we use model difficulty (RoBERTa's) as a heuristic to select hard examples for a challenge dataset that can hopefully be used to evaluate any future progress on this. We propose GQNLI, a generalized quantifier NLI challenge dataset, consisting of 30 premises and 300 hypotheses. The average sentence lengths of hypothesis and premises are 15.97 and 7.35, respectively. Both numbers are comparable to those of MNLI, but lower than ANLI's (Williams et al., 2020). It should be noted that GQNLI is designed for evaluating future models; obviously not for benchmarking RoBERTa.
Dataset Creation Firstly, we manually create 100 premise-hypothesis pairs, in which various types of GQs appear. For each premise and hypothesis, the number of GQs varies from one to three. To choose the premises, we randomly sampled 100 premises with GQs from SNLI and ANLI test sets, respectively, and selected 10 premises in total, that we consider are semantically adequate for adding GQs and making simple hypotheses.
To construct the hypotheses, we rely on RoBERTa fine-tuned on MNLI and manually select examples about which the model is unsure or incorrect. To focus on GQs, we keep the challenge examples otherwise simple (Ribeiro et al., 2020), and avoid lexical variations in the hypotheses. Hard examples were found to be characterized by (i) mixing generalized quantifiers with other logical
operators, such as subsumption or negation, and (ii) combining multiple different generalized quantifiers. We discuss these observations in Section 7.