SlowGuess/NAACL_2022_OCR huggingface dataset:
Lisa Matthewson. 2001. Quantification and the nature of crosslinguistic variation. Natural Language Semantics, 9(2):145-189.
Lisa Matthewson. 2004. On the methodology of semantic fieldwork. International journal of American linguistics, 70(4):369-415.
Lisa Matthewson. 2008. Quantification: A cross-linguistic perspective.
Richard Montague. 1973. The proper treatment of quantification in ordinary english. In Approaches to natural language, pages 221-242. Springer.
Andrzej Mostowski. 1957. On a generalization of quantifiers. Fundamenta Mathematicae, 44(1):12-36.
Mathijs Mul and Willem Zuidema. 2019. Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization.
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340-2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Association for the Advancement of Artificial Intelligence (AAAI).
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-4901, Online. Association for Computational Linguistics.
Liudmila Nikolaeva. 2012. Quantifiers in Adyghe, pages 21-82. Springer Netherlands, Dordrecht.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajic, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4034-4043, Marseille, France. European Language Resources Association.
Maciej Ogrodniczuk, Vincent Ng, Yulia Grishina, and Sameer Pradhan, editors. 2020. Proceedings of the Third Workshop on Computational Models of Reference, Anaphora and Coreference. Association for Computational Linguistics, Barcelona, Spain (online).
Maciej Ogrodniczuk, Sameer Pradhan, Yulia Grishina, and Vincent Ng, editors. 2019. Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference. Association for Computational Linguistics, Minneapolis, USA.
Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alexia Warstadt, Karmanya Aggarwal, Emily Allaway, Tal Linzen, and Samuel R. Bowman. 2021. Does putting a linguist in the loop improve NLU data collection? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4886-4901, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Barbara Hall Partee. 1970. Negation, conjunction, and quantifiers: Syntax vs. semantics. Foundations of Language, 6(2):153-165.
Stanley Peters and Dag Westerstahl. 2006. Quantifiers in language and logic. Oxford University Press.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101-108, Online. Association for Computational Linguistics.
Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019. Are red roses red? evaluating consistency of question-answering models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6174-6184, Florence, Italy. Association for Computational Linguistics.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912, Online. Association for Computational Linguistics.
James H. Schmerl and Stephen G. Simpson. 1982. On the role of ramsey quantifiers in first order arithmetic. J. Symb. Log., 47:423-435.
Shashank Srivastava, Igor Labutov, and Tom Mitchell. 2018. Zero-shot learning of classifiers from natural language quantification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 306-316, Melbourne, Australia. Association for Computational Linguistics.
Penka Stateva, Arthur Stepanov, Viviane Deprez, Ludivine Emma Dupuy, and Anne Colette Reboul. 2019. Cross-linguistic variation in the meaning of quantifiers: Implications for pragmatic enrichment. Frontiers in Psychology, 10:957.
Shane Steinert-Threlkeld. 2019. Learnability and semantic universals. Semantics and Pragmatics, 12:4.
Shane Steinert-Threlkeld. 2021. Quantifiers in natural language: Efficient communication and degrees of semantic universals. Entropy, 23(10).
Jakub Szymanik. 2016. Cognitive Processing of Quantifiers, pages 51-83. Springer International Publishing, Cham.
Jakub Szymanik and Camilo Thorne. 2015. Semantic complexity of quantifiers and their distribution in corpora. In Proceedings of the 11th International Conference on Computational Semantics, pages 64-69, London, UK. Association for Computational Linguistics.
Wataru Uegaki. 2022. The Informativeness/Complexity Trade-Off in the Domain of Boolean Connectives. Linguistic Inquiry, pages 1-39.
Yuval Varkel and Amir Globerson. 2020. Pre-training mention representations in coreference models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8534-8540, Online. Association for Computational Linguistics.
Sara Veldhoen and Willem Zuidema. 2017. Can neural networks learn logical reasoning? CLASP Papers in Computational Linguistics, page 34.
Dag Westerstahl. 1989. Quantifiers in formal and natural languages. In Handbook of philosophical logic, pages 1-131. Springer.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Adina Williams, Tristan Thrush, and Douwe Kiela. 2020. Anlizing the adversarial natural language inference dataset.
William A. Woods. 1991. Understanding subsumption and taxonomy: A framework for progress. In *Principles of Semantic Networks*.
Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, and Kentaro Inui. 2020. Do neural models learn systematicity of monotonicity inference in natural language? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6105-6117, Online. Association for Computational Linguistics.
Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019. Can neural networks understand
monotonicity reasoning? In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 31-40, Florence, Italy. Association for Computational Linguistics.
Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential Reasoning Learning for Language Representation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7170-7186, Online. Association for Computational Linguistics.
John Yen. 1991. Generalizing term subsumption languages to fuzzy logic. In *IJCAI*.
Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021. DocNLI: A large-scale dataset for document-level natural language inference. In *Findings of the Association for Computational Linguistics: ACLIJCNLP* 2021, pages 4913-4922, Online. Association for Computational Linguistics.
Lotfi A Zadeh. 1983. A computational approach to fuzzy quantifiers in natural languages. In Computational linguistics, pages 149-184. Elsevier.
Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth. 2020. Do language embeddings capture scales? In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 4889-4896, Online. Association for Computational Linguistics.
George Kingsley Zipf. 1949. Human behavior and the principle of least effort.
Generalized Quantifiers	Regular Expressions
some(A)(B) = 1	(some|severalmuchmany) \/ det .*\/(nsubj|obj|obl) | (some|severalmuchmany) \/ nsubj |(some|severalmuchmany) \/ amod \w+ \nsubj : pass
all(A)(B) = 1	(every | all | each) \/ det .*\/(nsubj|obj|obl) | all \/ det : predet .*\/(nsubj|obj|obl) | everything | everyone |everybody
more than k the(A)(B) = 1	((more|great) \/ advmod than \/(fixed|case) | at \/ case least \/ nmod) .+ \/ nummod .+ \/(nsubj|obj|obl)
less than k the(A)(B) = 1	((few|less) \/ advmod than \/(fixed|case) | at \/ case most \/ amod) .+ \/ nummod .+ \/(nsubj|obj|obl)
k(A)(B) = 1	\w+ \/ nummod .+ \/(nsubj|obj|obl)
between p and k the(A)(B) = 1	between \/ case \w+ \/(nummod|nsubj|obj|obl) and \/ cc \w+ \/ conj | between \/ case .+ \/(nummod|nsubj|obj|obl) % \/obl
the p/k(A)(B) = 1	\d+ \ \d+ \/(nummod|nsubj|obj|obl) | half \/ nummod | third \/(nsubj|obj|obl) | fourth \/(nsubj|obj|obl) | fifth \/(nsubj|obj|obl)
the k% (A)(B = 1)	\d+ \/ nummod % \/(nsubj|obj|obl)
most (A)(B) = 1	most \/ amod \w+ \/(nsubj|obj|obl) | most \/ nsubj : pass of \ / case .+ \/ nmod
few (A)(B) = 1	few \/ amod \w+ \/(nsubj|obj|obl) | few \/ nsubj : pass of \ / case .+ \/ nmod
each other (A)(B) = 1	each \/ det other \/(nsubj|obj|obl)
Table 9: Regular Expressions for generalized quantifiers.
A Regular Expressions for Generalized Quantifiers
Table 9 lists the regex we use to parse generalized quantifiers in sentences augmented with universal dependency tags. The approach does not find all the generalized quantifiers exhaustively but rather approximates the common distributions.
B Pairwise Observation
While the analysis in Section 4 is based on quantifiers in hypotheses, next we consider the interaction of quantifiers in hypotheses and quantifiers in premises. To this end, we calculate the difference between overall performance and performance for premise-hypothesis pairs of GQs. In Figure 2, we visualize the results as heatmaps (see Table 10 for exact numbers of occurrences and accuracies). Surprisingly, whenever quantifiers appear in both the premise and the hypothesis, LMs largely fail to predict the entailment. Percentage quantifiers, supposed to be semantically more complex than counting quantifiers, are not de facto harder in NLI. We studied all 27 cases of percentage quantifiers in the English NLI datasets, and found that in most cases, percentage quantifiers occurrences are identical across premises and hypotheses, i.e., triggering little or no inference. The other two proportional quantifiers, most and few, are hard for
Figure 2: Fine-grained analysis of RoBERTa performance on 6 English NLI subtasks. Each heatmap represents hypotheses with a type of quantifier. The rows stand for premises with the quantifier of that label. The numbers are calculated as the accuracy over the whole dataset minus the fine-grained accuracy given a specific premise and hypothesis (the higher the number, the worse the performance). For each heatmap, the last column represents the accuracy gap weighted by all 6 tasks. "UN" stands for an entry where no explicit quantifier is identified.
LMs to resolve, e.g., in some quantifier pairs, models yield  $0\%$  accuracy. Although each other is supposed to be hardest to resolve due to the complex semantics of reciprocals (Szymanik and Thorne, 2015), it is not reflected in NLI tasks as such. The reason is similar to percentage quantifiers, while annotators intend to alter counting quantifiers when writing hypotheses, reciprocity is seldomly considered a linguistic ability that needs testing for NLU systems. And the annotation for Ramsey quantifier is simply a knockoff, making reciprocal relation identification unwarranted through shallow correlations.
C Fine-grained NLI Analysis
D XQA Result: mBERT and XLM-R
Table 11 compares the results of mBERT and XLM-R on two XQA tasks, XQuAD and MLQA.
E GQNLI Examples
Table 12 list one example per category in GQNLI.
F GQNLI Negation Cases
We present the results of seven models' performance on cases with negation cues in GQNLI in Table 13.
G GQNLI Subsumption Cases
See Table 14 for models 'performance on cases requiring subsumption reasoning in GQNLI. We also break down subsumption results by entailment labels into two categories: neutral and non-neutral.
H GQNLI Experiment Details
We reused the fine-tuned BERT and RobERTa in Section 4. The other fine-tuned LMs are from Hugging Face. We list the models and thier links in Table 15.
Hypothesis	Premise	MNLL m, dev #occurrence	MNLL mm, dev #occurrence	SNLL test #occurrence	ANLL R1 test #occurrence	ANLL R2 test #occurrence	ANLL R3 test #occurrence	Total #occurrence
all	k	k	k	k	k	k	k	k	k	k	k	k	k
some	some	45	93.3	38	86.8	16	90	4	100	0	2	50	6	86.8
all	8	87.5	8	50	3	100	3	0	22	14
k
k
between most	12	75	10	80	40	90	4	100	0	2	50	68	85.3
few	2	50	2	50
p/k	110	79.1	83	84.3	137	85.4	1	100	1	0	8	75	340	82.6
k/ k
eachother "unmatched"
all	some	11	100	12	100	2	100	1	100	26	100
all	73	82.2	74	86.5	3	100	4	25	2	50	6	50	162	81.5
k	1	0	1	0
k
between most	28	85.7	19	100	22	81.8	9	22.2	5	40	13	23.1	96	70.8
few	4	100	2	100	1	0	2	50
p/k	1	0	6	100
k/ k	1	0
eachother "unmatched"	0	0	1	100	0	0	0	1	100	1
k/ k
> k	some	1	100	2	50	1	0	4	50	2
all	1	0	2	0	2	50	5	20
k	100	1	100	2	50	5	80
k
between most	1	0	3	100	2	100	3	66.7	9	55.6	10	40	28	57.1
few	1	100	2	50	2	50
p/k	1	100
k/ k
eachother "unmatched"	12	100	18	100	5	80	6	66.7	6	16.7	3	66.7	50	82
< k	some
all
k
k
between most	12	75	14	78.6	28	85.7	2	50	2	0	4	0	58	72.4
few	12	83.3	14	71.4	22	95.5	1	100	3	0	4	75	56	80.4
p/k
k
between most	12	84.3	121	76.9	593	92.2	42	42.9	49	44.9	37	32.4	982	82.5
few	2	50	2	100	1	0	5	60
p/k
k/ k
eachother "unmatched"	3	100	3	100	3	33.3	2	100	2	50	13	76.9
k	some	8	75	14	78.6	28	85.7	2	50	2	0	4	0	58	72.4
all	12	83.3	14	71.4	22	95.5	1	100	3	0	4	75	56	80.4
k
between most	3	66.7
k
k
between most	140	84.3	121	76.9	593	92.2	42	42.9	49	44.9	37	32.4	982	82.5
few	2	50	2	100	1	0	5	60
p/k	1	100	1	100	1	100
k/ k
eachother "unmatched"	12	100	1	100	1	100	1	100	1	100
k	some
all	12	75	14	78.6	28	85.7	2	50	2	50	4	50
k
between most	12	83.3	14	71.4	22	95.5	1	100	3	0	4	75	56	80
few	12	83.3	14	71.4	22	95.5	1	100	3	0	4	75	56	80
p/k
k/ k
eachother "unmatched"	2	100	1	100	1	100	4	100
most	some	2	50
all	12	75	14	78.6	28	85.7	2	50	2	50	1	0	58	72.4
k
between most	5	60	1	100	2	50	1	0	9	55.6
few	7	85.7	4	75	11	81.8
p/k	1	100
k/ k
eachother "unmatched"	23	73.9	30	83.3	1	0	54	77.8
few	some
all	1	100	1	0	1	0	2	50	2	50
k	1	100
k	0	50
between most	4	75	3	33.3	4	100	33.3	14
few	0	0	0	0	1	0
p/k	0	0
k/ k
eachother "unmatched"	9	77.8	9	88.9	7	85.7	2	50	27	81.5
p/k	some
all	12	75	14	78.6	28	85.7	2	50	2	50	3	100
k	3	100
k	1	0
between most	6	83.3	6	100	1	100	0	1	100
few	2	50	2	100	1	100	3	100
p/k	66.7	4
k/ k
eachother "unmatched"	4	100	1	100	1	100	0	5	100
k/%	some
all	12	75	14	78.6	28	85.7	2	50	2	50	1	100
k	3	100
k	1	100
between most	6	83.3	6	100	1	100	0	1	100
few	2	50	2	100	1	100	3	100
p/k	66.7	4
Table 10: Statistics of pairwise analysis in Monolingual NLI Benchmarks
Quant.	en	zh	es	mBERT	ar	vi	de	weighted	en	zh	es	XLM-R	ar	vi	de	weighted
EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1
XQuAD
some	75	84.2	50	55.5	58.3	76.1	50	50	16.6	42.4	33.3	43.8	47.2	58.7	66.7	76.1	41.6	51.3	50	71.5	66.7	73.6	66.7	76.9	66.7	80.6	59.7	71.7
all	28.5	62.2	14.2	35.2	28.5	82	42.8	52.3	14.2	29.4	28.5	56	26.1	52.9	57.1	91.8	14.2	21.4	57.1	78.6	42.8	54.9	85.7	85.7	79.3	52.3	68.6
> k
< k
k	78.1	90.1	68.7	80.4	56.2	72.1	40.6	64.3	12.5	35.7	56.2	77.1	52.1	70	75	87.4	53.1	58.8	46.8	77.4	65.6	86.3	62.5	85.4	62.5	86.9	60.9	80.4
between	100	100	33.3	72.2	66.6	93.3	100	100	0	19	0	56.5	50	73.5	100	100	66.7	66.7	33.3	60	100	100	100	100	33.3	55.5	72.2	80.4
p/k
k%
most	40	53.3	40	40	0	10	0	26.6	0	0	20	49.3	16.7	29.9	40	48	20	33.3	40	50	0	26.6	0	0	20	49.3	20	34.5
each other
all GQs	70	83.2	55	66.7	50	70.3	41.6	58.2	11.6	32.5	43.3	65	45.3	62.7	70	83.6	43.3	50.2	48.3	73.6	60	76	68.3	83.6	58.3	80.3	58	74.6
comp.	71.8	83.7	48	59.1	56	74.5	40.8	57.9	13.9	32.4	50.7	67.2	46.9	62.5	74.5	86	43	52.8	61	80	53.3	71.7	58.1	78	61.1	77.1	58.5	74.3
MLQA
some	59	80	28.2	52.1	34.1	59.2	36.3	54.9	5.4	24	33.3	58.4	32.7	54.8	69.6	86.1	33.3	60.6	41.4	70	43.1	62.9	43.2	78	45.4	61.1	46	69.8
all	67.7	79.8	14.2	46.4	38.4	62.8	33.3	57.9	10.5	30.1	31.2	51.6	32.6	54.8	77.4	90.6	35.7	70	42.3	66.4	38	60	57.8	79.8	37.5	51	48.1	69.6
> k
< k	0	0	0	13.3	0	6.7	0	40	0	20	0	30
k	74.9	79.4	47	63.4	41.5	65.9	27.6	50.3	6.3	23.7	38.2	53	39.3	56	69.2	82.1	45.2	66.2	48.7	73.3	43	64.9	48.5	71.9	46.3	62.1	50.2	70.1
between	50	88.5	50	83.3	0	26.6	0	68.7	0	26.6	20	58.7	50	88.5	50	50	50	65.3	0	54.6	0	77.4	30	67.2
p/k
k%
most	100	100	0	0	0	0
each other	55.5	7	47.3	62.1	45.4	61.7	30	46.8	5.8	15.7	33.3	40.7	36.2	50.3	59.2	76	47.3	69.5	45.4	59.5	40	63.2	47	75.7	22.2	31.7	43.5	62.6
all GQs	63.5	79.2	41.8	60.3	39.6	63.7	29.3	51.3	6.4	23.6	36.1	53.2	36.1	55.2	69	83	43	65.6	46.9	71.5	41.9	64.1	47.6	73.2	44.4	59.8	48.8	69.5
comp.	67.2	80.6	37.5	57.9	47.3	66	30	48.4	11.2	28	40.8	56	39	56.2	70.4	83.3	38.7	62.5	54.1	72.2	42.5	62.9	50.5	72.3	52.2	67.3	51.4	70.1
Table 11: Results of mBERT and XLM-R performance on XQA tasks decomposed by quantifier categories.
Quantifier	Premise	Hypothesis	Label
some	“There are six dogs. Three brown dogs, a black dog and a white dog run along the green grass.”	“Some dogs sit.”	Neutral
all	“In 2021, there are 490 million people in Africa living in extreme poverty, or 36% of the total population.”	"Not all people in Africa live in extreme poverty.”	Entailment
>k	“Two young men in blue stand over a stove and look at the camera while another young man in red stands behind them.”	"At least two men wear red."	Contradiction
<k	“More than five guys chased two girls in the classroom.”	“No less than four guys chased two girls in the classroom.”	Entailment
k	“There are twelve singers on a stage, less than half from Argentina and one from Cape Verde.”	“Two singers come from Argentina.”	Neutral
between	“Only half out of six cleaners are sweeping up animal faeces from the street during a parade.”	“Between four and five cleaners are sweeping up animal faeces.”	Contradiction
p/k	“More than 50% but less than 65% of Americans worry about global warming.”	“Two thirds of Americans worry about global warming.”	Contradiction
k%	“More than five guys chased two girls in the classroom.”	“100% of the guys chased two girls in the classroom.”	Neutral
most	“Two young men in blue stand over a stove and look at the camera while another young man in red stands behind them.”	“Most men wear blue.”	Entailment
few	“More than 50% but less than 65% of Americans worry about global warming.”	“A few people from America do not worry about global warming.”	Entailment
each other	“There are 100 villagers and 100 townsmen. Most villagers and most townsmen hate each other.”	"All villagers and all towns-men hate each other.”	Neutral
Table 12: GQNLI examples.
Quantifier
# Occurrence with negation cues	some
9	all
6	>k
6	<k
9	k
18	between
3	p/k
6	k%
6	most
6	few
9	each other
3	Overall
81
Model	Training Data	% Performance
BERT	S,M,F,ANLI	0	66.7	100	33.3	50	0	50	0	50	22.2	33.3	39.2
ELECTRA	S,M,F,ANLI	33.3	50.0	100.0	33.3	50.0	0.0	50.0	0.0	66.7	0.0	0.0	43.1
SBERT	S,M,F,ANLI	55.6	50.0	66.7	11.1	27.8	0.0	50.0	0.0	100.0	66.7	0.0	54.9
RoBERTa	MNLI	33.3	16.7	0	33.3	27.8	66.7	33.3	33.3	50	33.3	33.3	31.4
S,M,F,ANLI	66.7	83.3	100.0	33.3	66.7	100.0	50.0	50.0	50.0	33.3	66.7	58.8
ALBERT	S,M,F,ANLI	88.9	50.0	66.7	33.3	55.6	100.0	0.0	50.0	50.0	11.1	0.0	49.0
BART	MNLI	33.3	0.0	50.0	66.7	66.7	100.0	0.0	100.0	0.0	33.3	0.0	35.3
S,M,F,ANLI	66.7	50.0	100.0	33.3	50.0	0.0	50.0	0.0	50.0	66.7	100.0	52.9
DeBERTa-v3	MNLI	33.3	0.0	50.0	33.3	50.0	100.0	66.7	50.0	0.0	33.3	0.0	37.3
M,F,ANLI	55.6	66.7	100.0	33.3	66.7	100.0	50.0	50.0	100.0	55.6	33.3	66.7
M,F,Ling,DocNLI	33.3	100.0	100.0	0.0	33.3	0.0	83.3	0.0	50.0	66.7	100.0	51.0
Table 13: Models' performance on instances with negation cues in GQNLI.
Type
# Occurrence	Subsumption (neutral)
90	Subsumption (non-neutral)
99	Subsumption (total)
189	Non-subsumption
111
Model	Training Data	% Performance
BERT	S,M,F,ANLI	22.2	24.2	23.3	41.4
ELECTRA	S,M,F,ANLI	3.3	52.5	29.1	53.2
SBERT	S,M,F,ANLI	68.9	35.4	51.3	18.9
RoBERTa	MNLI	27.8	18.2	22.8	37.8
S,M,F,ANLI	21.1	33.3	27.5	59.5
ALBERT	S,M,F,ANLI	33.3	38.4	36.0	49.5
BART	MNLI	36.7	46.5	41.8	40.5
S,M,F,ANLI	44.4	23.2	33.3	58.6
DeBERTa-v3	MNLI	45.6	26.3	35.4	33.3
M,F,ANLI	52.2	37.4	44.4	54.1
M,F,Ling,DocNLI	86.7	17.2	50.3	36.0
Table 14: Models' performance on instances requiring subsumption reasoning.
Model	Training Data	Model's link
ELECTRA	S,M,F,ANLI	https://huggingface.co/ynie/electra-large-discriminator-snli_mnli_fever_anli_R1_R2_R3-nli
SBERT	S,M,F,ANLI	https://huggingface.co/usc-isi/sbert-roberta-large-anli-mnli-snli
BART	MNLI	https://huggingface.co/facebook/bart-large-mnli
S,M,F,ANLI	https://huggingface.co/ynie/bart-large-snli_mnli_fever_anli_R1_R2_R3-nli
ALBERT	S,M,F,ANLI	https://huggingface.co/ynie/albert-xxlarge-v2-snli_mnli_fever_anli_R1_R2_R3-nli
DeBERTa-v3	MNLI	https://huggingface.co/MoritzLaurer/DeBERTa-v3-base-mnli
M,F,ANLI	https://huggingface.co/MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli
M,F,Ling,DocNLI	https://huggingface.co/MoritzLaurer/DeBERTa-v3-base-mnli-fever-docnli-ling-2c
Table 15: Links to the models we use to test on GQNLI.