Venue: NAACL 2018

# Acquisition of Phrase Correspondences using Natural Deduction Proofs

 Hitomi Yanaka\({}^{1}\)

hitomiyanaka@g.ecc.u-tokyo.ac.jp &Koji Mineshima\({}^{2}\)

mineshima.koji@ocha.ac.jp &Pascala Martinez-Gomez\({}^{3}\)

pascual.mg@aist.go.jp &Daisuke Bekki\({}^{2}\)

bekki@is.ocha.ac.jp \({}^{1}\)The University of Tokyo

\({}^{2}\)Ochanomizu University

\({}^{3}\)Artificial Intelligence Research Center, AIST

Tokyo, Japan

###### Abstract

How to identify, extract, and use phrasal knowledge is a crucial problem for the task of Recognizing Textual Entailment (RTE). To solve this problem, we propose a method for detecting paraphrases via natural deduction proofs of semantic relations between sentence pairs. Our solution relies on a graph reformulation of partial variable unifications and an algorithm that induces subgraph alignments between meaning representations. Experiments show that our method can automatically detect various paraphrases that are absent from existing paraphrase databases. In addition, the detection of paraphrases using proof information improves the accuracy of RTE tasks.

## 1 Introduction

Recognizing Textual Entailment (RTE) is a challenging natural language processing task that aims to judge whether one text fragment logically follows from another text fragment Dagan et al. (2013). Logic-based approaches have been successful in representing the meanings of complex sentences, ultimately having a positive impact on RTE Bjerva et al. (2014); Beltagy et al. (2014); Mineshima et al. (2015, 2016); Abzianidze (2015, 2016). Although logic-based approaches succeed in capturing the meanings of functional or logical words, it is difficult to capture the meanings of content words or phrases using genuine logical inference alone. This remains a crucial problem in accounting for lexical relations between content words or phrases via logical inference. To solve this problem, previous logic-based approaches use knowledge databases such as WordNet Miller (1995) to identify lexical relations within a sentence pair. While this solution has been successful in handling word-level paraphrases, its extension to phrase-level semantic relations is still an unsolved problem. There are three main difficulties that prevent an effective identification and use of phrasal linguistic knowledge.

The first difficulty is the presence of out-of-context phrase relations in popular databases such as the Paraphrase Database (PPDB) Ganitkevitch et al. (2013). PPDB may suggest paraphrases that do not adhere to the context of our relevant text segments nor to their semantic structure, which might be problematic.

The second difficulty is finding semantic phrase correspondences between the relevant text segments. Typical approaches only rely on surface Beltagy et al. (2013) or syntactic correspondences Arase and Tsujii (2017), often producing inaccurate alignments that significantly impact our inference capabilities. Instead, a mechanism to compute semantic phrase correspondences could potentially produce, if available, more coherent phrase pairs and solve the recurring issue of discontinuity.

The third difficulty is the intrinsic lack of coverage of databases for logical inference despite their large size. Whereas there is a relatively small number of possible word-to-word correspondences and thus their semantic relations can be enumerated, the same is not true for all phrase pairs that might be of interest. One alternative is to use functions of infinite domain (e.g., cosine similarity) between phrase representations Tian et al. (2016), but these techniques are still under development, and we have not seen definitive successful applications when combined with logic systems.

In this study, we tackle these three problems. The contributions of this paper are summarized as follows: First, we propose a new method of detecting phrase correspondences through natu

ral deduction proofs of semantic relations for a given sentence pair. Second, we show that our method automatically extracts various paraphrases that compensate for a shortage in previous paraphrase databases. Experiments show that extracted paraphrases using proof information improve the accuracy of RTE tasks.

## 2 Related Work

In this section, we review previous logical inference systems that are combined with lexical knowledge. The RTE system developed by Abzianidze (2016) uses WordNet as axioms and adds missing knowledge manually from the training dataset; however, this technique requires considerable human effort and is not extended to handle phrasal knowledge.

Martinez-Gomez et al. (2017) proposed an RTE system with an on-the-fly axiom injection mechanism guided by a natural deduction theorem prover. Pairs of unprovable sub-goals and plausible single premises are identified by means of a variable unification routine and then linguistic relations between their logical predicates are checked using lexical knowledge such as WordNet and VerbOcean Chklovski and Pantel (2004). However, this mechanism is limited to capturing word-to-word relations within a sentence pair.

Bjerva et al. (2014) proposes an RTE system where WordNet relations are used as axioms for word-to-word knowledge in theorem proving. For phrasal knowledge, PPDB is used to rephrase an input sentence pair instead of translating paraphrases into axioms. However, this solution ignores logical contexts that might be necessary when applying phrasal knowledge. Moreover, it does not apply to discontinuous phrases.

Beltagy et al. (2016) uses WordNet and PPDB as lexical knowledge in the RTE system. To increase their coverage of phrasal knowledge, the system combines a resolution strategy to align clauses and literals in a sentence pair and a statistical classifier to identify their semantic relation. However, this strategy only considers one possible set of alignments between fragments of a sentence pair, possibly causing inaccuracies when there are repetitions of content words and meta-predicates.

In our research, we propose an automatic phrase abduction mechanism to inject phrasal knowledge during the proof construction process. In addition, we consider multiple alignments by backtracking the decisions on variable and predicate unifications, which is a more flexible strategy. We represent logical formulas using graphs, since this is a general formalism that is easy to visualize and analyze. However, we use _natural deduction_ (see Section 3.2) as a proof system instead of Markov Logic Networks for inference. Some research has investigated graph operations for semantic parsing Reddy et al. (2014, 2016) and abstractive summarization Liu et al. (2015); we contribute to these ideas by proposing a subgraph mapping algorithm that is useful for performing natural language inferences.

Considerable research efforts have been focused on the identification and extraction of paraphrases. One successful technique is associated with bilingual pivoting Bannard and Callison-Burch (2005); Zhao et al. (2008), in which alternative phrase translations are used as paraphrases at a certain probability. However, this technique requires large bilingual parallel corpora; moreover, word alignment errors likely cause noisy paraphrases. Another strategy is to extract phrase pairs from a monolingual paraphrase corpus using alignments between syntactic trees, guided by a linguistically motivated grammar Arase and Tsuji (2017). The main difference between these studies and ours is that they typically attempt alignment between words or syntactic trees, whereas we perform alignments between meaning representations, which enables the acquisition of more general paraphrases by distinguishing functional words from content words. This point is important in distinguishing among different semantic relations (e.g., antonyms and synonyms). In addition, word and syntactic alignments potentially ignore coreferences, making it difficult to find relations between many-to-many sentences. Semantic alignments enable this because coreferences must refer to the same variable as the original entity.

## 3 Logic-based Approach to RTE

### Meaning representation

In logic-based approaches to RTE, a text \(T\) and a hypothesis \(H\) are mapped onto logical formulas \(T^{\prime}\) and \(H^{\prime}\). To judge whether \(T\) entails \(H\), we check whether \(T^{\prime}\Rightarrow H^{\prime}\) is a theorem in a logical system.

For meaning representations, we use Neo-Davidsonian event semantics Parsons (1990). In this approach, a verb is analyzed as a one-place predicate over events. Both the arguments of a 

erb and modifiers are linked to events by semantic roles, and the entire sentence is closed by existential quantification over events. For example, (1) is mapped onto (2).

* A girl is skipping rope on a sidewalk.
* \(\exists x_{1}\exists x_{2}\exists x_{3}\exists y_{1}\left(\textbf{girl}(x_{1}) \wedge\textbf{rope}(x_{2})\land\right.\\ \left.\textbf{sidewalk}(x_{3})\wedge\textbf{skip}(y_{1})\wedge\left( \textbf{subj}(y_{1})=x_{1}\right)\land\right.\\ \left.\left(\textbf{obj}(y_{1})=x_{2}\right)\land\mathbf{on}(y_{1},x_{3})\right)\]

We use \(x_{i}\) as a variable for entities and \(y_{j}\) for events. In this semantics, we represent all content words (e.g., _girl_ and _skip_) as one-place predicates. Regarding functional words, we represent a preposition like _on_ as a two-place predicate, e.g., \(\mathbf{on}(y_{1},x_{3})\). We also use a small set of semantic roles such as **subj** and **obj** as a functional term and use equality (\(=\)) to connect an event and its participant, as in \(\textbf{subj}(y_{1})=x_{1}\).

To be precise, the set of _atomic formulas_\(\mathcal{A}\) in this event semantics is defined by the rule

\[\mathcal{A}:=\mathbf{F}(t)\mid\mathbf{G}(t,u)\mid t=u\]

where \(\mathbf{F}(t)\) is a one-place predicate (for content words), \(\mathbf{G}(t,u)\) is a two-place predicate (for prepositions), \(t\) and \(u\) are a term. A term is defined as a constant, a variable, or a functional term of the form \(f(t)\) where \(f\) is a semantic role and \(t\) is a term.

We call a formula constructed by conjunctions and existential quantifiers a _basic formula_ in event semantics. Thus, a set of basic formulas \(\varphi\) in event semantics is defined as:

\[\varphi::=\mathcal{A}\mid\varphi\land\varphi\mid\exists t\,\varphi\]

The formula in (2) is an instance of a basic formula, which captures the predicate-argument structure of a sentence.

On top of the system of basic formulas, we have a full language of event semantics with negation (\(\neg\)), disjunction (\(\vee\)), implication (\(\rightarrow\)), and a universal quantifier (\(\forall\)). These operators are used to represent additional logical features.

There is a natural correspondence between basic formulas and directed acyclic graphs (DAGs). Figure 1 shows an example1. In the graph representation, constants and variables correspond to vertices; both two-place predicates for prepositions (e.g., \(\mathbf{on}(y_{1},x_{1})\)) and functional terms for semantic roles (e.g., \(\mathbf{subj}(y_{1})=x_{1}\)) are represented as edges. A one-place predicate \(\mathbf{F}(t)\) in a logical formula can be represented as a functional relation \(\textbf{isa}(t,\mathbf{F})\), where \(\textbf{isa}\) is an expression relating a term \(t\) and a predicate \(\mathbf{F}\) represented as a vertex. The \(\textbf{isa}\) edges are unlabeled for simplicity.

Footnote 1: See Jones (2016) for some variants of graphical representations of logical formulas.

### Natural deduction and word abduction

We use the system of natural deduction (Prawitz, 1965; Troelstra and Schwichtenberg, 2000) to capture phrase correspondences from a sentence pair \((T,H)\), following the strategies for word axiom injection developed by Martinez-Gomez et al. (2017) and Yanaka et al. (2017). The sentence pair \((T,H)\) is first mapped to a pair of formulas \((T^{\prime},H^{\prime})\). \(T^{\prime}\) is initially set to the premise \(P\), and \(H^{\prime}\) is set to the goal \(G\) to be proved.

If formulas \(P\) and \(G\) are basic formulas, then the proving strategy is to decompose them into a set of atomic formulas using inference rules for connections and existential quantifiers. The premise \(P\) is decomposed into a pool of premises \(\mathcal{P}=\{\mathbf{p}_{i}(\theta_{i})\mid i\in\{1,\ldots,m\}\}\), where each \(\mathbf{p}_{i}(\theta_{i})\) is an atomic formula and \(\theta_{i}\) is a list of terms appearing in \(\mathbf{p}_{i}(\theta_{i})\). The goal \(G\) is also decomposed into a set of sub-goals \(\mathcal{G}=\{\mathbf{g}_{j}(\theta^{\prime}_{j})\mid j\in\{1,\ldots,n\}\}\), where \(\theta^{\prime}_{j}\) is a list of terms appearing in \(\mathbf{g}_{j}(\theta^{\prime}_{j})\).

The proof is performed by searching for a premise \(\mathbf{p}_{i}(\theta_{i})\) whose predicate matches that of a sub-goal \(\mathbf{g}_{j}(\theta^{\prime}_{j})\). If such a premise is found, then variables in \(\theta^{\prime}_{j}\) are unified to those in \(\theta_{i}\) and the sub-goal \(\mathbf{g}_{j}(\theta^{\prime}_{j})\) can be removed from \(\mathcal{G}\). If all the sub-goals can be removed, we prove \(T^{\prime}\to H^{\prime}\). In the presence of two or more variables with the same predicate, there might be multiple possible variable unifications. Modern theorem provers explore these multiple possibilities in search of a configuration that proves a theorem.

Sub-goals may remain unpored when \(T\) logically does not entail \(H\) i.e., when there are no premise predicates \(\mathbf{p}_{i}\) that are matched with \(\mathbf{g}_{j}\). In this case, the system tries word axiom injection, called _word abduction_. More specifically, if there

Figure 1: A graph for the basic formula (2).

 

is a premise \({\bf p}_{i}(\theta_{i})\) whose predicate has a linguistic relation (according to linguistic knowledge2) with that of a sub-goal \({\bf g}_{j}(\theta^{\prime}_{j})\), then variables in \(\theta^{\prime}_{j}\) are unified with those in \(\theta_{i}\) and the sub-goal \({\bf g}_{j}(\theta^{\prime}_{j})\) can be removed from \(\mathcal{G}\).

Footnote 2: As given in a linguistic ontology or database such as WordNet or VerbOcean.

### Graph illustration

Figure 2 shows an example to illustrate how the system works. To begin with, the input sentence pair \((T,H)\) is mapped onto a pair of formulas, \((T^{\prime},H^{\prime})\). \(T^{\prime}\) is initially placed to the premise \(P\), and \(H^{\prime}\) to the goal \(G\). Note that these are basic formulas, and they are thus decomposed to the following sets of formulas \(\mathcal{P}\) and \(\mathcal{G}\), respectively:

Steps 10 to 3 in Figure 2 demonstrate the variable unification routine and word axiom injection using graphs. Note that in step 1, all variables in formulas in \(\mathcal{P}\) or \(\mathcal{G}\) are initially different.

In step 2, we run a theorem proving mechanism that uses graph terminal vertices as anchors to unify variables between formulas in \(\mathcal{P}\) and those in \(\mathcal{G}\). The premise \(\textbf{meat}(x_{2})\) in \(\mathcal{P}\) matches the predicate **meat** of the sub-goal \(\textbf{meat}(x_{4})\) in \(\mathcal{G}\) and the variable unification \(x_{4}:=x_{2}\) is applied (and similarly for the sub-goal \(\textbf{cut}(y_{2})\) in \(\mathcal{G}\) with the variable unification \(y_{2}:=y_{1}\)).

In step 3, we use the previous variable unification on \(y_{1}\), the **subj** edge in \(\mathcal{P}\) and \(\mathcal{G}\) and the axiom \(\forall x.\textbf{lady}(x)\rightarrow\textbf{woman}(x)\) from external knowledge to infer that \(x_{3}:=x_{1}\).

## 4 Phrase Abduction

There is one critical reason that the word-to-word axiom injection described in Section 3.2 fails to detect phrase-to-phrase correspondences. That is, the natural deduction mechanism decomposes the goal \(G\) into atomic sub-goals that are then proved _one-by-one_ (word-by-word), independently of each other except for the variable unification effect. This mechanism is particularly problematic when we attempt to prove phrases that resist decomposition, two-place predicates (e.g., \(\textbf{into}(x,y)\)), or failures in variable unification (e.g., due to inaccurate semantics). Thus, we propose a method to detect phrase-to-phrase correspondence through natural deduction proofs.

### Phrase pair detection

We detect phrase-to-phrase entailing relations between \(T^{\prime}\) and \(H^{\prime}\) by finding alignments between the subgraphs of their meaning representations when \(T^{\prime}\Rightarrow H^{\prime}\) or \(T^{\prime}\Rightarrow\neg H^{\prime}\) hold. Finding subgraph alignments is a generalization of the subgraph isomorphism problem, which is NP-complete3. In this paper, we approximate a solution to this problem by using a combination of a backtracking variable unification and a deterministic graph search on the neighborhood of non-unified variables.

Footnote 3: Emmert-Streib et al. (2016) gives a good overview.

Using our running example in Figure 2, step 4 displays our proposed subgraph alignment. The variable \(x_{5}\) in the graph of \(\mathcal{G}\) cannot be unified with any variable in the graph of \(\mathcal{P}\). This is a very common case in natural language inferences, as there might be concepts in \(H\) that are not directly supported by concepts in \(T\). In this research, we propose spanning a subgraph starting at non-unified variables (e.g., \(x_{5}\) in \(\mathcal{G}\)) whose boundaries are semantic roles (e.g., **subj, obj**). Its candidate semantics from \(\mathcal{P}\) are then the attributes of its corresponding unified variables from \(\mathcal{G}\) (e.g. _cut up precisely \(\rightarrow\) cut into pieces_).

### Graph alignments

To formalize this solution we introduce some graph notation. Let \(V=\mathcal{V}^{u}~{}\cup~{}\mathcal{V}^{\bar{u}}~{}\cup~{}\mathcal{L}\) be the set of vertices, where \(\mathcal{V}^{u}\) is the set of unified variables (e.g. \(x_{1},x_{2},y_{1}\)), \(\mathcal{V}^{\bar{u}}\) is the set of non-unified variables (e.g. \(x_{5}\)), and \(\mathcal{L}\) is a set of predicates (e.g., _lady_, _woman_). Let \(E\) be the set of labeled, directed edges \(\langle v,l,v^{\prime}\rangle\) where \(v,v^{\prime}\in V\) and \(l\) are labels that may represent a functional relation **isa**, a preposition or a semantic role. We denote a set of two-place predicates for prepositions as PREP and a set of functional terms for semantic roles as ARGS; e.g., \(\text{ARGS}=\left\{\mathbf{subj},\mathbf{obj}\right\}\). A graph that represents \(\mathcal{P}\) is then a tuple \(G_{\mathcal{P}}=\langle V_{\mathcal{P}},E_{\mathcal{P}}\rangle\), and similarly, for \(\mathcal{G}\), \(G_{\mathcal{G}}=\langle V_{\mathcal{G}},E_{\mathcal{G}}\rangle\).

We can now define a function to span a subgraph in the neighborhood of non-unified variables \(v\in\mathcal{V}^{\bar{u}}_{\mathcal{G}}\) in the graph of \(\mathcal{G}\). We call a connected set of edges in which no semantic roles appear, i.e., \(\left\{\langle v,l,v^{\prime}\rangle\mid l\not\in\text{ARGS}\right\}\), a _phrase set_. Let \(E(x)\) be the phrase set in \(E\) such that each vertex is connected to \(x\) with an incoming or outgoing edge, that is, \(E(x)=\left\{(v_{i},l,v_{k})\in E\mid(x=v_{i}\lor x=v_{k})\wedge l\not\in\text{ ARGS}\right\}.\) 

Note that \(E(x)\) induces a subgraph in a given graph \(G\) and the condition \(l\notin{\rm ARGS}\) sets the boundaries of the subgraph by excluding the semantic roles of verb phrases. Given two phrase sets \(E\) and \(E^{\prime}\), we say \(E^{\prime}\) is reachable from \(E\), written \(E\sim E^{\prime}\), if \(E\) and \(E^{\prime}\) share at least one variable vertex. Let \(\sim^{*}\) be the transitive closure of \(\sim\). Given a set of edges \(E_{\mathcal{G}}\) and a variable \(v\), we define the _extended phrase set_, written Reach\((v)\), as follows:

\[\text{Reach}(v)=\{e\in E\mid E_{\mathcal{G}}(v)\sim^{*}E\}\]

that is, the set of edges \(e\) that can be reached from \(v\) without crossing an edge with a semantic role label. This function defines a partition or equivalence class for non-unified variables \(v\in V_{\mathcal{G}}^{\underline{u}}\), and each of these partitions induce a (possibly discontinuous) phrase in \(\mathcal{G}\) that remains unproved.

The corresponding subgraph in \(\mathcal{P}\) to each of these partitions is given by the vertices and edges connected with a path of length one to the unified variables that appear in Reach\((v)\). That is,

\[\text{Corr}(v)=\{e\in E_{\mathcal{P}}(v^{\prime}),v^{\prime}\in V_{\mathcal{ G}}^{[v]}\cap V_{\mathcal{P}}\}\]

where \(V_{\mathcal{G}}^{[v]}\) denotes the vertices in the subgraph of \(\mathcal{G}\) induced by the partition Reach\((v)\).

Figure 2: A graph representation of a theorem proving routine on basic formulas and variable unification. Dotted circles represent non-unified variables at each step, whereas edges without labels are attributes. The graph of the left side is the set of premises \(\mathcal{P}\) and the graph of the right side is the set of sub-goals \(\mathcal{G}\). Colored subgraphs represent a word or a phrase to which our axiom injection mechanism applies.

 

A subgraph alignment between \({\cal P}\) and \({\cal G}\) is given by the pair of \(\left<{\rm Corr}(v),{\rm Reach}(v)\right>\) for all \(v\in{\cal V}_{\cal G}^{\bar{u}}\), where the phrases can be read from the predicates in the vertices and edges labeled with prepositions.

We define a mapping \((\cdot)^{\bullet}\) from a labeled edge \(\langle v,l,v^{\prime}\rangle\) to an atomic formula as follows.

Let \(E\) be a set of labeled edges, and let \(E^{\bullet}\) be \(\big{\{}\langle v,l,v^{\prime}\rangle^{\bullet}\mid\langle v,l,v^{\prime} \rangle\in E\big{\}}\). The phrase axiom generated for each non-unified variable \(v\in{\cal V}_{\cal G}^{\bar{u}}\) is defined as

where \(\theta_{\sf C}\) is a set of free variables appearing in \({\rm Corr}(v)^{\bullet}\) (which includes \(v\)) and \(\theta_{\sf R}\) is a set of free variables appearing in \({\rm Reach}(v)^{\bullet}\) but not in \({\rm Corr}(v)^{\bullet}\).

In Figure 2, the only non-unified variable in the sub-goal in step 4 is \(x_{5}\), that is, \({\cal V}_{\cal G}^{\bar{u}}=\{x_{5}\}\). Then, starting from the variable \(x_{5}\), \({\rm Reach}(x_{5})\) is

Now \(V_{\cal G}^{[x_{5}]}=\{y_{1},x_{5}\}\), and thus \({\rm Corr}(x_{5})\) is

\[\small\begin{array}{c}\left\{\langle y_{1},{\bf i}{\bf s}{\bf a}{\bf c}{\bf u }\right\rangle,\left\langle y_{1},{\bf i}{\bf s}{\bf a}{\bf u}\right\rangle, \left\langle y_{1},{\bf i}{\bf s}{\bf a}{\bf p}\right\rangle,\left\langle y_{1} ,{\bf i}{\bf s}{\bf a}{\bf p}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e }{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e }{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e }{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{\rm e}{ \rm e 

basic formulas, the procedure described in the previous sections applies to the pair \((P_{0},G_{1})\); these basic formulas are decomposed into atomic ones, and then the word-to-word abduction generates the desired axiom \(\forall y_{1}(\textbf{cut}(y_{1})\!\rightarrow\!\textbf{slice}(y_{1})).\) Finally, the graph alignment applies in the same way as described in Figure 2, which generates the phrase axiom:

\[\forall y_{1}(\textbf{cut}(y_{1})\rightarrow\exists x_{5}(\textbf{into}(y_{1} ,x_{5})\wedge\textbf{piece}(x_{5})))\]

Using this axiom, one can complete the proof of the contradiction between \(T^{\prime}\) and \(H^{\prime}\).

## 5 Experiments

### Dataset selection

We use the SemEval-2014 version of the SICK dataset [36] for evaluation. The SICK dataset is a dataset for semantic textual similarity (STS) as well as for RTE. It was originally designed for evaluating compositional distributional semantics, so it contains logically challenging problems involving quantifiers, negation, conjunction, and disjunction, as well as inferences with lexical and phrasal knowledge.

The SNLI dataset [7] contains inference problems requiring phrasal knowledge. However, it is not concerned with logically challenging expressions; the semantic relationships between a premise and a hypothesis are often limited to synonym/hyponym lexical substitution, replacements of short phrases, or exact word matching. This is because hypotheses are often parallel to the premise in structures and vocabularies. The FraCaS dataset [8] also contains logically complex problems. However, it is confined to purely logical inferences and thus does not contain problems requiring inferences with lexical and phrasal knowledge. For these reasons, we choose the SICK dataset to evaluate our method of using logical inference to extract phrasal knowledge.

The SICK dataset contains \(9927\) sentence pairs with a \(5000/4927\) training/test split. These sentence pairs are manually annotated with three types of labels _yes_ (entailment), _no_ (contradiction), or _unknown_ (neutral) (see Table 1 for examples). In RTE tasks, we need to consider a directional relation between words such as hypernym and hyponym to prove entailment and contradiction. Hence, to extract phrasal knowledge for RTE tasks, we use the training data whose gold label is _entailment_ or _contradiction_, excluding those with the _neutral_ label.

### Experimental setup

For the natural deduction proofs, we used ccg2lambda [36]5, a higher-order automatic inference system, which converts CCG derivation trees into semantic representations and conducts natural deduction proofs automatically. We parsed the tokenized sentences of the premises and hypotheses using three wide-coverage CCG parsers: C&C [36], EasyCCG [10], and depccg [11]. CCG derivation trees (parses) were converted into logical semantic representations based on Neo-Davidsonian event semantics (Section 3.1). The validation of semantic templates used for semantic representations was conducted exclusively on the trial split of the SICK dataset. We used Coq [1], an interactive natural deduction theorem prover that we run fully automatically with a number of built-in theorem-proving routines called tactics, which include first-order logic.

Footnote 5: Available at [https://github.com/mynlp/ccg2lambda](https://github.com/mynlp/ccg2lambda).

We compare phrase abduction with different experimental conditions. **No axioms** is our system without axiom injection. **W2W** is the previous strategy of word abduction [36]. **P2P** is our strategy of phrase abduction; **W2W+P2P** combines phrase abduction with word abduction. In addition, we compare our system with three purely logic-based (unsupervised) approaches: **The Meaning Factory**[1], **LangPro**[1], and **UTexas**[1]. We also compare our system with machine learning-based approaches: the current state-of-the-art deep learning model **GRU**[22], a log-linear regression model **SemEval-2014 best**[11], and a hybrid approach combining a logistic regression model and probabilistic logic **PL+eclassif**[1].

### Extracted paraphrases

We extracted 9445 axioms from the SICK training dataset. The proving time average to extract phrasal axioms was only 3.0 seconds for a one-sentence pair6. Table 2 shows some examples of 

paraphrases we extracted from the natural deduction proof in the training set. In particular, the examples of verb phrases show our method has the potential to capture long paraphrases. Each paraphrase in Table 2 is not contained in WordNet and PPDB. There are many instances of non-contiguous phrases in the SICK dataset, in particular, verb-particle phrases. Shown in Table 2, our semantic alignment can detect non-contiguous phrases through the variable unification process, which is one of the main advantages over other shallow/syntactic methods. In addition, Table 2 shows our method is not limited to hypernym or hyponym relations, but it is also capable for detecting antonym phrases.

### RTE evaluation results

Table 3 shows the experimental results. The results show that the combination of word abduction and phrase abduction improved the accuracy. When the **W2W+P2P** result is substituted for the **W2W** result, there is a 1.1% increase in accuracy (from 83.1% to 84.3%). The accuracy of **P2P** is almost equal to that of **W2W**. This is because the recall improves from 63.6% to 72.1% while the precision decreases from 97.1% to 85.6%. The increase in false positive cases caused this result; some details of false positive cases are described in the next subsection. **W2W+P2P** outperformed other purely logic-based systems. The machine learning-based approaches outperform **W2W+P2P**, but unlike these approaches, parameter estimation is not used in our method. This suggests that our method has the potential to increase the accuracy by using a classifier.

### Positive examples and error analysis

Table 4 shows some positive and negative examples on RTE with the SICK dataset. For ID 9491, the sentence pair requires the paraphrase from _a field of brown grass_ to _a grassy area_, not included in previous lexical knowledges. Our phrasal axiom injection can correctly generate this paraphrase from a natural deduction proof, and the system proves the entailment relation.

ID 2367 is also a positive example of phrasal axiom injection. The phrasal axiom between _set fire to cameras_ and _burn cameras with a blow torch_ was generated. This example shows that our semantic alignment succeeds in acquiring a general paraphrase by separating logical expressions such as _some_ from content words and also by accounting for syntactic structures such as the passive-active alternation.

For ID 3628, the axiom shown in the table was extracted from the following sentence pair with

\begin{table}
\begin{tabular}{l l c c} \hline ID & Text & Hypothesis & Entailment \\ \hline \hline
3941 & _A boy is looking at a calendar_ & _There is nobody checking a calendar_ & No \\ \hline
5938 & _Vegetables are being put into a pot by a man_ & _Someone is pouring ingredients into a pot_ & Yes \\ \hline
5930 & _The man is not doing exercises_ & _Two men are fighting_ & Unknown \\ \hline \end{tabular}
\end{table}
Table 1: Examples in the SICK dataset with different entailment labels and similarity scores.

\begin{table}
\begin{tabular}{c c c} \hline  & Prec. & Rec. & Acc. \\ \hline \hline GRU & \(-\) & \(-\) & \(87.1\) \\ \hline PL+eclassif & \(-\) & \(-\) & \(85.1\) \\ \hline SemEval2014 Best Score & \(81.6\) & \(81.9\) & \(84.6\) \\ \hline \hline The Meaning Factory & \(93.6\) & \(60.6\) & \(81.6\) \\ \hline LangPro & \(98.0\) & \(58.1\) & \(81.4\) \\ \hline UTexas & \(-\) & \(-\) & \(80.4\) \\ \hline \hline W2W+P2P & \(84.2\) & \(77.3\) & \(84.3\) \\ \hline W2W & \(97.1\) & \(63.6\) & \(83.1\) \\ \hline P2P & \(85.6\) & \(72.1\) & \(83.0\) \\ \hline No axioms & \(98.9\) & \(46.5\) & \(76.7\) \\ \hline \end{tabular}
\end{table}
Table 2: Examples of phrase alignments constructed by phrasal axiom injection.

 

their _entailment_ label:

\(T_{1}\): _A woman is putting meat in a pan \(H_{1}\)_: Someone is dropping the meat into a pan

But the phrase _drop over_ does not entail the phrase _drop into_, and a proof for the inference is over-generated in ID 3628. We extracted all possible phrasal axioms from the training dataset, so noisy axioms can be extracted as a consequence of multiple factors such as parsing errors or potential disambiguation in the training dataset. One possible solution for decreasing such noisy axioms would be to use additive composition models Tian et al. (2016) and asymmetric learnable scoring functions to calculate the confidence on these asymmetric entailing relations between phrases.

ID 96 is also an example of over-generation of axioms. The first axiom, \(\forall y_{1}(\text{jump}(y_{1})\rightarrow\exists x_{1}(\mathbf{in}(y_{1},x_ {1})\wedge\mathbf{air}(x_{1})))\) was extracted from the proof of \(T_{1}\Rightarrow H_{1}\):

\(T_{1}\): _A child in a red outfit is jumping on a tramploline \(H_{1}\)_: A little boy in red clothes is jumping in the air The second axiom \(\forall y_{1}(\textbf{man}(y_{1})\rightarrow\textbf{biker}(y_{1}))\) was extracted from the proof of \(T_{2}\Rightarrow H_{2}\):

\(T_{2}\): _A man on a yellow sport bike is doing a wheelie and a friend on a black bite is catching up \(H_{2}\)_: A biker on a yellow sport bike is doing a wheelie and a friend on a black bite is catching up_

Although these axioms play a role in the proofs of \(T_{1}\Rightarrow H_{1}\) and \(T_{2}\Rightarrow H_{2}\), the wrong axiom \(\forall y_{1}(\textbf{man}(y_{1})\rightarrow\textbf{biker}(y_{1}))\) causes the over-generation of a proof for the inference in ID 96. The correct one would rather be \(\forall z_{1}\forall y_{1}(\textbf{man}(y_{1})\land\textbf{on}(y_{1},x_{1}) \land\textbf{like}(x_{1})\rightarrow\textbf{biker}(y_{1}))\). In this case, it is necessary to bundle predicates in a noun-phrase by specifying the types of a variable (entity or event) when making phrase alignments.

For ID 408, the word _explorer_ is not contained in the training entailment dataset and hence the relevant axiom \(\forall x_{1}(\textbf{explorer}(x_{1})\to\textbf{people}(x_{1}))\) was not generated. While our logic-based method enables detecting semantic phrase correspondences in a sentence pair in an unsupervised way, our next step is to predict unseen paraphrases of this type.

## 6 Conclusion

In this paper, we proposed a method of detecting phrase correspondences through natural deduction proofs of semantic relations between sentence pairs. The key idea is to attempt a proof with automatic phrasal axiom injection by the careful management of variable sharing during the proof construction process. Our method identifies semantic phrase alignments by monitoring the proof of a theorem and detecting unproved sub-goals and logical premises. The method of detecting semantic phrase alignments would be applicable to other semantic parsing formalisms and meaning representation languages such as abstract meaning representations (AMR) Banarescu et al. (2013). Experiment results showed that our method detected various phrase alignments including non-contiguous phrases and antonym phrases. This result may contribute to previous phrase alignment approaches. The extracted phrasal axioms improved the accuracy of RTE tasks.

In future work, we shall enhance this methodology of phrasal axiom injection to predict unseen paraphrases. The pairs of premises and sub-goals that can be detected through the proof process conduct semantic alignments in a sentence pair. With the use of an additive composition model of distributional vectors, we can evaluate the validity of such semantic alignments. A combination of our phrasal axiom injection and additive composition model of distributional vectors has the potential to detect unseen paraphrases in a sentence pair.

### Acknowledgments

We thank the three anonymous reviewers for their detailed comments. This work was supported by JST CREST Grant Number JPMJCR1301 and AIP Challenge Program, Japan.

\begin{table}
\begin{tabular}{c c l p{170.7pt}|p{56.9pt}} \hline ID & Sentence Pair & Gold & Pred & Axiom \\ \hline
* & **a group of brown dogs are playing in a field of brown grass** & Yes & Yes \\  & & Yes & \(\gamma_{i}(\textbf{bird}(z_{j})\land\textbf{brown}(z_{j})\land\textbf{grass}(z_{ j})\) \\  & _Four dog are playing in a grass area_ & Yes & \(\gamma_{i}(\textbf{bird}(z_{j})\land\textbf{area}(z_{j}))\) \\ \cline{2-3}
[FOOTNOTE:]Footnote : footnotemark: [ENDFOOTNOTE] & **a person in thursing some cameras with a slow rock** & Yes & Yes & \(\gamma_{i}(\textbf{bird}(z_{j})\land\textbf{with}(y_{i})\land\textbf{by}(z_{j}) \land\textbf{bow}\textbf{torch}(x_{j})\land\textbf{camera}(\textbf{obj}(y_{j})))\) \\ \cline{2-3}
[FOOTNOTE:]Footnote : footnotemark: [ENDFOOTNOTE] & **bereport is testing for the concurrent** & Yes & \(\neg\textbf{set}(y_{1}(\textbf{born}(y_{1})\land\textbf{with}(y_{1})\land\textbf{ dot}(y_{1})\land\textbf{camera}(\textbf{obj}(y_{1})))\) \\ \cline{2-3}
[FOOTNOTE:]Footnote : footnotemark: [ENDFOOTNOTE] & **came**(\(\textbf{orb}(y_{1})\land\textbf{note}(y_{1},\textbf{obyl}(y_{1}))\land\textbf{ camera}(\textbf{obs}(y_{1})))\) \\ \cline{2-3}
[FOOTNOTE:]Footnote : footnotemark: [ENDFOOTNOTE] & **durn** is being dropped into a pan** & Unk & Yes & \(\forall y_{1}(\textbf{pan}(\textbf{obyl}(y_{1})\rightarrow\textbf{into}(y_{1}, \textbf{obyl}(y_{1})))\) \\ \cline{2-3}
[FOOTNOTE:]Footnote : footnotemark: [ENDFOOTNOTE] & **e** & \(\textit{An atomic is playing isn’t an empty pool}\) & Unk & No & \(\forall y_{1}(\textbf{burn}(y_{i})\rightarrow\exists x_{i}(\textbf{imb}(y_{i},x_ {1})\land\textbf{air}(x_{1})))\) \\ \cline{2-3}
[FOOTNOTE:]Footnote : footnotemark: [ENDFOOTNOTE] & **f** & \(\forall y_{1}(\textbf{ham}(y_{i})\rightarrow\exists x_{i}(\textbf{imb}(y_{i},x_ {1})\land\textbf{air}(x_{1})))\) \\ \cline{2-3}
[FOOTNOTE:]Footnote : footnotemark: [ENDFOOTNOTE] & **g** & \(\forall y_{1}(\textbf{ham}(y_{i})\rightarrow\exists\textbf{bler}(y_{i}))\) \\ \cline{2-3}
[FOOTNOTE:]Footnote : footnotemark: [ENDFOOTNOTE] & **g** & \(\forall y_{1}(\textbf{ham}(y_{i})\rightarrow\exists\textbf{bler}(y_{i}))\) \\ \cline{2-3}
[FOOTNOTE:]Footnote : footnotemark: [ENDFOOTNOTE] & **s** & \(\neg\textbf{Set}(y_{1}(\textbf{ham}(y_{1})\boldsymbol{\cdot}(\textbf{body}(y_{1})) \land\textbf{camera}(\textbf{obj}(y_{1})))\) \\ \cline{2-3} \end{tabular}
\end{table}
Table 4: Positive and negative examples on RTE from the SICK dataset.

 



## References

* Abzianidze (2015) Lasha Abzianidze. 2015. A tableau prover for natural logic and language. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Lisbon, Portugal, pages 2492-2502.
* Abzianidze (2016) Lasha Abzianidze. 2016. Natural solution to FraCaS entailment problems. In _Proceedings of the 5th Joint Conference on Lexical and Computational Semantics_. Association for Computational Linguistics, Berlin, Germany, pages 64-74.
* Arase and Tsujii (2017) Yuki Arase and Jun'ichi Tsujii. 2017. Monolingual phrase alignment on parse forests. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Copenhagen, Denmark, pages 1-11.
* Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjkabch, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_. Association for Computational Linguistics, Sofia, Bulgaria, pages 178-186.
* Bannard and Callison-Burch (2005) Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In _Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics_. Association for Computational Linguistics, Stroudsburg, PA, USA, pages 597-604.
* Beltagy et al. (2013) Islam Beltagy, Cuong Chau, Gemma Boleda, Dan Garrette, Katrin Erk, and Raymond Mooney. 2013. Montague meets Markov: Deep semantics with probabilistic logical form. In _Proceedings of the Second Joint Conference on Lexical and Computational Semantics (*Sem-2013)_. Atlanta, GA, pages 11-21.
* Belltagy et al. (2014) Islam Beltagy, Stephen Roller, Gemma Boleda, Katrin Erk, and Raymond J. Mooney. 2014. UTeXas: Natural language semantics using distributional semantics and probabilistic logic. In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014)_. Association for Computational Linguistics and Dublin City University, Dublin, Ireland, pages 796-801.
* Beltagy et al. (2016) Islam Beltagy, Stephen Roller, Pengxiang Cheng, Katrin Erk, and Raymond J. Mooney. 2016. Representing meaning with a combination of logical and distributional models. _Computational Linguistics_ 42(4):763-808.
* Bertot and Castran (2010) Yves Bertot and Pierre Castran. 2010. _Interactive Theorem Proving and Program Development: Coq'Art The Calculus of Inductive Constructions_. Springer Publishing Company, Incorporated, New York, USA.
* Bjerva et al. (2014) Johannes Bjerva, Johan Bos, Rob van der Goot, and Malvina Nissim. 2014. The Meaning Factory: Formal semantics for recognizing textual entailment and determining semantic similarity. In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014)_. Association for Computational Linguistics and Dublin City University, Dublin, Ireland, pages 642-646.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Lisbon, Portugal, pages 632-642.
* Chklovski and Pantel (2004) Timothy Chklovski and Patrick Pantel. 2004. Verbocean: Mining the web for fine-grained semantic verb relations. In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Barcelona, Spain, pages 33-40.
* Clark and Curran (2007) Stephen Clark and James R. Curran. 2007. Wide-coverage efficient statistical parsing with CCG and log-linear models. _Computational Linguistics_ 33(4):493-552.
* Cooper et al. (1994) Robin Cooper, Richard Crouch, Jan van Eijck, Chris Fox, Josef van Genabith, Jan Jaspers, Hans Kamp, Manfred Pinkal, Massimo Poesio, Stephen Pulman, et al. 1994. FraCaS-a framework for computational semantics. _Deliverable_ D6.
* Dagan et al. (2013) Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. _Recognizing Textual Entailment: Models and Applications_. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.

* Ganitkevitch et al. (2013) Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie_. Association for Computational Linguistics, Atlanta, Georgia, pages 758-764.
* Jones (2016) Bevan Keeley Jones. 2016. _Learning words and syntactic cues in highly ambiguous contexts_. Ph.D. thesis, The University of Edinburgh.
* Lai and Hockenmaier (2014) Alice Lai and Julia Hockenmaier. 2014. Illinois-lh: A denotational and distributional approach to semantics. In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_. Association for Computational Linguistics and Dublin City University, Dublin, Ireland, pages 329-334

Mike Lewis and Mark Steedman. 2014. A* CCG parsing with a supertag-factored model. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Doha, Qatar, pages 990-1000.
* Liu et al. (2015) Fei Liu, Jeffrey Flanigan, Sam Thomson, Norman M. Sadeh, and Noah A. Smith. 2015. Toward abstractive summarization using semantic representations. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. The Association for Computational Linguistics, pages 1077-1086.
* Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In _Proceedings of the 9th International Conference on Language Resources and Evaluation_. European Language Resources Association, Reykjavik, Iceland, pages 216-223.
* Martinez-Gomez et al. (2016) Pascual Martinez-Gomez, Koji Mineshima, Yusuke Miyao, and Daisuke Bekki. 2016. ccg2lambda: A compositional semantics system. In _Proceedings of the 2016 System Demonstrations of the Association for Computational Linguistics_. Association for Computational Linguistics, Berlin, Germany, pages 85-90.
* Martinez-Gomez et al. (2017) Pascual Martinez-Gomez, Koji Mineshima, Yusuke Miyao, and Daisuke Bekki. 2017. On-demand injection of lexical knowledge for recognising textual entailment. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics_. Association for Computational Linguistics, Valencia, Spain, pages 710-720.
* Miller (1995) George A. Miller. 1995. WordNet: A lexical database for English. _Communications of the ACM_ 38(11):39-41.
* Mineshima et al. (2015) Koji Mineshima, Pascual Martinez-Gomez, Yusuke Miyao, and Daisuke Bekki. 2015. Higher-order logical inference with compositional semantics. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Lisbon, Portugal, pages 2055-2061.
* Mineshima et al. (2016) Koji Mineshima, Ribeka Tanaka, Pascual Martinez-Gomez, Yusuke Miyao, and Daisuke Bekki. 2016. Building compositional semantics and higher-order inference system for a wide-coverage japanese CCG parser. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 2236-2242.
* Parsons (1990) Terence Parsons. 1990. _Events in The Semantics of English: a Study in Subatomic Semantics_. MIT Press, Cambridge, USA.
* A Proof-Theoretical Study_. Almqvist & Wiksell, Stockholm, Sweden.
* Reddy et al. (2014) Siva Reddy, Mirella Lapata, and Mark Steedman. 2014. Large-scale semantic parsing without question-answer pairs. _Transactions of the Association for Computational Linguistics_ 2:377-392.
* Reddy et al. (2016) Siva Reddy, Oscar Lackstrom, Michael Collins, Tom Kwiatkowski, Dipjanjan Das, Mark Steedman, and Mirella Lapata. 2016. Transforming Dependency Structures to Logical Forms for Semantic Parsing. _Transactions of the Association for Computational Linguistics_ 4:127-140.
* Tian et al. (2016) Ran Tian, Naoaki Okazaki, and Kentaro Inui. 2016. Learning semantically and additively compositional distributional representations. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Berlin, Germany, pages 1277-1287.
* Troelstra and Schwichtenberg (2000) Anne Troelstra and Helmut Schwichtenberg. 2000. _Basic Proof Theory_. Cambridge University Press.
* Yanaka et al. (2017) Hitomi Yanaka, Koji Mineshima, Pascual Martinez-Gomez, and Daisuke Bekki. 2017. Determining semantic textual similarity using natural deduction proofs. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Copenhagen, Denmark, pages 692-702.
* Yin and Schutze (2017) Wenpeng Yin and Hinrich Schutze. 2017. Task-specific attentive pooling of phrase alignments contributes to sentence matching. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_. Association for Computational Linguistics, Valencia, Spain, pages 699-709.
* Yoshikawa et al. (2017) Masashi Yoshikawa, Hiroshi Noji, and Yuji Matsumoto. 2017. A* CCG parsing with a supertag and dependency factored model. In _Proceedings of the 55nd Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Vancouver, Canada, pages 277-287.
* Zhao et al. (2008) Shiqi Zhao, Cheng Niu, Ming Zhou, Ting Liu, and Sheng Li. 2008. Combining multiple resources to improve SMT-based paraphrasing model. In _Proceedings of the 46rd Annual Meeting on Association for Computational Linguistics_. Association for Computational Linguistics, Columbus, Ohio, pages 1021-1029.