SlowGuess/ACL_2020_OCR huggingface dataset:
Nam Khanh Tran and Weiwei Cheng. 2018. Multiplicative tree-structured long short-term memory networks for semantic representations. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics (*SEM-2018), pages 276–286.
Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC-2018).
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Learning Representations (ICLR-2019).
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT-2018), pages 1112-1122.
Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019a. Can neural networks understand monotonicity reasoning? In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 31-40.
Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019b. HELP: A dataset for identifying shortcomings of neural models in monotonicity reasoning. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM* 2019), pages 250-255.
A.1 Lexical entries and replacement examples
Table 6 shows a context-free grammar and a set of predicate replacements used to generate inference examples. Regarding the context-free grammar, we consider premise-hypothesis pairs containing the quantifier  $Q$  in the subject position, and the predicate replacement is performed in both the first and second arguments of the quantifier. When generating premise-hypothesis pairs involving embedding monotonicity, we consider inferences involving four types of predicate replacements (hyonyms  $N_{hypn}$ , adjectives  $Adj$ , prepositions  $PP$ , and relative clauses  $RelC$ ) in the
6115
Context-free grammar for premise sentences
S	→	NP IV1
NP	→	QN | QN S
S	→	WhNP TV NP | WhNP NP TV | NP TV
Lexicon
Q	→	{no, at most three, less than three, few, some, at least three, more than three, a few}
N	→	{dog, rabbit, lion, cat, bear, tiger, elephant, fox, monkey, wolf}
IV1	→	{ran, walked, came, waltzed, swam, rushed, danced, dawdled, escaped, left}
IV2	→	{laughed, groaned, roared, screamed, cried}
TV	→	{kissed, kicked, hit, cleaned, touched, loved, accepted, hurt, licked, followed}
WhNP	→	{that, which}
Nhypn	→	{animal, creature, mammal, beast}
Adj	→	{small, large, crazy, polite, wild}
PP	→	{in the area, on the ground, at the park, near the shore, around the island}
RelC	→	{which ate dinner, that liked flowers, which hated the sun, that stayed up late}
Adv	→	{slowly, quickly, seriously, suddenly, lazily}
Predicate replacements for hypothesis sentences
N	to	Nhypn | Adj N | N PP | N RelC
IV1	to	IV1 Adv | IV1 PP | IV1 or IV2 | IV1 and IV2
Table 6: A context-free grammar and a set of predicate replacements used to generate inference examples. Predicate replacement is applied to  $N$  or  $IV_{1}$ , replacing it with a corresponding phrase.
first argument of the quantifier. To generate natural sentences consistently, we use the past tense for verbs; for lexical entries and predicate replacements, we select those that do not violate selectional restriction.
To check the gold labels for the generated premise-hypothesis pairs, we translate each sentence to a first-order logic (FOL) formula and test if the entailment relation holds by theorem proving. The FOL formulas are compositionally derived by combining lambda terms assigned to each lexical item in accordance with meaning composition rules specified in the CFG rules in the standard way (Blackburn and Bos, 2005). Since our purpose is to check the polarity of monotonicity marking, vague quantifiers such as few are represented according to their polarity. For example, we map the quantifier few onto the lambda-term  $\lambda P\lambda Q\neg \exists x(\mathbf{f e w}(x)\land P(x)\land Q(x))$
A.2 Results on embedding monotonicity
Table 7 shows all results on embedding monotonicity. This indicates that all models partially generalize to inferences containing embedded clauses one level deeper than the training set, but fail to generalize to inferences containing embedded clauses two or more levels deeper.
6116
Train	Test	BERT	LSTM	TreeLSTM
D1	D1	100.0±0.0	91.1±5.4	100.0±0.0
D2	44.1±6.4	34.1±3.8	48.1±1.2
D3	47.6±3.2	45.1±5.1	48.5±1.8
D4	49.6±1.0	44.4±6.5	50.1±2.1
D5	49.9±1.1	44.1±5.3	50.3±1.1
D1∪D2	D1	100.0±0.0	100.0±0.0	100.0±0.1
D2	100.0±0.0	99.8±0.2	99.5±0.1
D3	75.2±10.0	75.4±10.8	86.4±4.1
D4	55.0±3.7	57.7±8.7	58.6±7.8
D5	49.9±4.4	45.8±4.0	48.4±3.7
D1∪D2∪D3	D1	100.0±0.0	100.0±0.0	100.0±0.0
D2	100.0±0.0	95.1±7.8	99.6±0.0
D3	100.0±0.0	85.2±8.9	97.7±1.1
D4	77.9±10.8	59.7±10.8	68±5.6
D5	53.5±19.6	55.1±8.2	49.6±4.3
D1∪D2∪D3∪D4	D1	100.0±0.0	100.0±0.0	100.0±0.1
D2	100.0±0.0	99.4±1.1	99.7±0.2
D3	100.0±0.0	91.5±4.0	98.9±1.1
D4	100.0±0.0	74.1±4.2	94.0±2.3
D5	89.1±5.4	64.2±4.7	69.5±4.1
D1∪D2∪D3∪D4∪D5	D1	100.0±0.0	100.0±0.0	100.0±0.1
D2	100.0±0.0	95.8±7.3	99.8±0.1
D3	100.0±0.0	90.5±13.1	99.1±0.2
D4	100.0±0.0	90.2±6.0	94.8±0.1
D5	100.0±0.0	93.6±3.1	83.2±12.1
D2	D1	36.4±14.4	25.3±9.3	44.9±4.1
D2	100.0±0.0	100.0±0.0	100.0±0.2
D3	47.6±10.3	43.9±17.5	51.8±1.1
D4	61.7±7.8	57.9±14.7	51.7±0.6
D5	42.6±5.1	47.2±2.9	50.9±0.4
D3	D1	49.6±0.5	48.8±13.2	49.8±4.1
D2	49.8±0.6	47.3±12.1	51.8±1.1
D3	100.0±0.0	100.0±0.0	100.0±0.2
D4	49.7±1.0	42.0±0.6	51.3±0.7
D5	50.0±0.4	38.4±9.6	49.8±0.3
D4	D1	50.3±1.0	46.8±6.5	49.0±0.4
D2	49.6±0.8	45.4±1.8	49.7±0.3
D3	50.2±0.7	45.1±0.6	50.5±0.7
D4	100.0±0.0	100.0±0.0	100.0±0.1
D5	49.7±0.5	45.1±0.9	50.5±1.1
D5	D1	49.9±0.7	43.7±4.4	49.1±1.1
D2	49.1±0.3	43.4±3.9	51.4±0.6
D3	50.6±0.2	44.3±2.7	50.5±0.3
D4	50.9±0.8	44.4±3.4	50.3±0.4
D5	100.0±0.0	100.0±0.0	100.0±0.1
Table 7: All results on embedding monotonicity.
6117