Implementing Natural Language Inference for Comparatives
Department of Computer Science Izumi Haruta (Supervisor : Daisuke Bekki)
1

Introduction

Comparative constructions pose a challenge in Natural Language Inference (NLI), which is the task ofdetermining whether a text entails a hypothesis. In
formal semantics, there is a rich body of work on
comparatives and gradable expressions using the notion of degree. However, a logical inference systemfor comparatives has not been suﬃciently developedfor use in the NLI task. In this paper, we present a
compositional semantics that maps various comparative constructions in English to logical forms (LFs)
via Combinatory Categorial Grammar (CCG) parsers
and combine it with an inference system based on automated theorem proving.1

2

Table 1: Logical forms of gradable constructions
based on A-not-A analysis
Sentence

Logical form

(1) Ken is 2 inches taller than Harry.

∀δ(tall(harry, δ − 2′′ ) → tall(ken, δ))(2) Most apples are red.

∃δ(∃x(apple(x) ∧ red(x) ∧ many(x, δ))∧¬∃x(apple(x) ∧ ¬red(x) ∧ many(x, δ)))(3) Few children ran.

¬∃x∃δ(child(x) ∧ many(x, δ) ∧ (δ > θmany (child))∧∃e(run(e) ∧ (subj(e) = x)))(4) Bob drove as carefully as John.

∃e1 ∃e2 (drive(e1 ) ∧ (subj(e1 ) = bob) ∧ drive(e2 )∧(subj(e2 ) = john) ∧ ∀δ(careful(e2 , δ) → careful(e1 , δ)))this reason, we modify the derivation trees provided
by CCG parsers in post-processing. For instance, wecompound expressions for comparatives and quantiﬁers are combined as one word, such as a few, a lotof, and at most.
few boys
N/NNa>
N P/NN>NPSystem architecture

Figure 1 shows the pipeline of the proposed system.
First, the premises and hypothesis are mapped to LFs
based on A-not-A analysis via CCG parsing and treetransformation. Next, a theorem prover judges yes,no, or unknown with the axioms for comparatives andlexical knowledge.
Degree semantics: A-not-A analysis To analyze
gradable adjectives, we use the two-place predicateof entities and degrees as developed in degree-based
semantics [1]. For instance, the sentence Ann is 5 feettall is analyzed as tall(ann, 5 feet), where tall(x, δ)
is read as “x is (at least) as tall as degree δ”. Comparative expressions are analyzed in terms of ﬁrstorder logic, using the so-called A-not-A analysis [2];for example, Chris is taller than Alex is analyzed as∃δ(tall(chris, δ) ∧ ¬tall(alex, δ)), which asserts that
there is a degree δ of tallness that Chris satisﬁes but
Alex does not. We present semantic LFs for someexample sentences using A-not-A analysis, as shownin Table 1. This analysis can be naturally extended to
not comparative forms of adjectives such as (1), butalso generalized quantiﬁers, adverbial phrases, andthe comparative forms of adverbs such as (2-4).
Compositional semantics in CCG In CCG, themapping from syntax to semantics is deﬁned by assigning syntactic categories to words [3]; the LF of
a sentence is then compositionally derived using λcalculus. However, there is a gap between the syntactic structures assumed in formal semantics and theoutput derivation trees of existing CCG parsers. For
1GitHub repository with code and data: https://github.
com/izumi-h/ccgcomp

99K

a~few boys
N P/NN>
NPInsertion of lexical knowledge To test the compatibility of logical inferences and inferences involving lexical knowledge, we implement an abduction
mechanism to search for useful axioms drawn fromknowledge bases before the process of theorem proving. The strategy is similar to the one used in previous
studies [4] in which the system searches for lexical
relations from WordNet [5] and VerbOcean [6].

3Experiments

For evaluation, we use ﬁve datasets: FraCaS [7];MED [8]; SICK [9]; HANS [10]; CAD. CAD wascreated for this study and includes problems related
to pragmatic inference.
Table 2 gives the results of the evaluation. Maj
is the accuracy of the majority baseline and Ours
the accuracy of our system. For FraCaS, MED, andCAD, +rule shows the accuracy achieved by the addition of hand-coded rules, which correct the errors inPOS tagging and lemmatization. For CAD, we also
experimented with an implementation for pragmaticinference. The accuracy is shown in +imp.
We compared our system with four logic-based systems (MN [11], LP [12, 13], MG [4], and GKR4 [14]),three DL-based systems (RB, BERT [8], BERT+ [8],and BF [15]), and two hybrid systems (HNB [14]and HNX [14]). RB is a system that used a stateof-the-art model, RoBERTa (RB) [16], trained on
MultiNLI [17].
The results show that our system achieved high accuracy on the logical inferences with adjectives, comparatives, quantiﬁers, numerals, and adverbs. For

Logical Forms (A-not-A)

Sentences
P1 :
P2 :
H:

Transformed
CCG Trees

John is tall.
::Syntactic Parsing
CCG parsers

••

P1 :
P2 :
H:

•

Semantic Parsing
ccg2lambda
Modifying Trees
Tsurgeon

CCG Derivation TreesistallJohn
N lex (S\N P )/(Sadj \N P ) Sadj \N P >NPS\N P<S

AxiomsComp∃δ(tall(john, δ) ∧ (δ = θtall ))::Abduction Mechanism
WordNet, VerbOcean•

Insertion oflexical knowledge

TPTP
formatTheorem Proving
Vampire•Yes, No,
Unknown

Figure 1: Overview of the proposed system
Table 2: Accuracy on FraCaS, MED, SICK, HANS, and CAD datasetsFraCaS
Section
#All
Maj
RB
MN
LP
Ours+rule

GQ
73.49
.73
.77
.93
.97
.99

Adj
22.41.45
.68
.73
.82
.95

Com
31.61
.52
.48
–.90
.90

Att
13.62
.69
.77
.92
.92
.92

MED
Label
#All
Maj
BERT
BERT+RB
Ours
+rule

gq
498
.58
.56
.54
.57
.97
.97

gqlex
691
.63
.58
.68
.55
.91
.92

HANS, [10] reported that DL-based systems tend toerroneously output yes for cases in which the hypothesis was a constituent or a sub-string of the premise,such as disjunctive sentences. To see how a system
performs on these cases, we present the accuracy foreach gold answer label (yes and unknown). While theaccuracy when the gold label was yes was close to100% in both our system and the DL-based system(RB), the accuracy of our system outperformed RB
when the label is unknown.

4Conclusion

In this study, we presented an end-to-end logicbased inference system for handling complex inferences with comparatives, quantiﬁers, numerals, andadverbs. The entire system is transparently composedof several modules and can solve complex inferences
for the right reason. This study contributes to thestudy of computational modeling and the evaluationof formal semantic theories, as well as to the creationof challenging NLI problems that DL-based models
need to address.
References[1] M. J. Cresswell. The semantics of degree. In Barbara Partee,editor, Montague Grammar, pp. 261–292. Academic Press, 1976.
[2] E. Klein. The interpretation of adjectival comparatives. Journalof Linguistics, Vol. 18, No. 1, pp. 113–136, 1982.
[3] M. Steedman. The Syntactic Process. MIT press Cambridge, MA,
2000.

SICK
#All 4927
Maj.57
RB.56
LP.81
MG.83
Ours .82

HANSGoldyes#All 15000
Maj.50
BF.87
RB
1.0
GKR4.84
HNB.84
HNX.83
Ours.98

unknown
15000.50.61.56.59.54.25.83

CAD
#All
MajRB
Ours+rule+rule +imp

257.43.58.81.82.92[4] P. Martínez-Gómez, K. Mineshima, Y. Miyao, and D. Bekki. Ondemand injection of lexical knowledge for recognising textual entailment. In In Proc. of EACL, pp. 710–720, 2017.
[5] G. A. Miller. WordNet: A lexical database for english. Commun.
ACM, Vol. 38, No. 11, pp. 39–41, 1995.
[6] T. Chklovski and P. Pantel. VerbOcean: Mining the web for ﬁnegrained semantic verb relations. In Proc. of EMNLP, pp. 33–40,2004.
[7] R. Cooper, R. Crouch, J. van Eijck, C. Fox, J. van Genabith,
J. Jaspers, H. Kamp, M. Pinkal, M. Poesio, S. Pulman, et al.
FraCaS–a framework for computational semantics, 1996.
[8] H. Yanaka, K. Mineshima, D. Bekki, K. Inui, S. Sekine, L. Abzianidze, and J. Bos. Can neural networks understand monotonicity
reasoning? In In Proc. of ACL Workshop, pp. 31–40, 2019.
[9] M. Marelli, S. Menini, M. Baroni, L. Bentivogli, R. Bernardi, andR. Zamparelli. A SICK cure for the evaluation of compositional
distributional semantic models. In In Proc. of LREC, pp. 216–223,2014.
[10] T. McCoy, E. Pavlick, and T. Linzen. Right for the wrong reasons:Diagnosing syntactic heuristics in natural language inference. InIn Proc. of ACL, pp. 3428–3448, 2019.
[11] K. Mineshima, P. Martínez-Gómez, Y. Miyao, and D. Bekki.
Higher-order logical inference with compositional semantics. InIn Proc. of EMNLP, pp. 2055–2061, 2015.
[12] L. Abzianidze. A tableau prover for natural logic and language. InProc. of EMNLP, pp. 2492–2502, 2015.
[13] L. Abzianidze. Natural solution to FraCaS entailment problems.
In In Proc. of *SEM, pp. 64–74, 2016.
[14] A.-L. Kalouli, R. Crouch, and V. de Paiva. Hy-NLI: a hybrid
system for natural language inference. In In Proc. of COLING, pp.
5235–5249, 2020.
[15] Y. Yaghoobzadeh, R. Tachet, T.J. Hazen, and A. Sordoni. Robustnatural language inference models with example forgetting. arXiv
preprint arXiv:1911.03861, 2019.
[16] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy,
M. Lewis, L. Zettlemoyer, and V. Stoyanov. RoBERTa: A robustly
optimized bert pretraining approach, 2019.
[17] A. Williams, N. Nangia, and S. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In In
Proc. of NAACL-HLT, pp. 1112–1122, 2018.