Robust Natural Language Representation Learning for Natural Language Inference by Projecting Superficial Words out
Authors: Wanyun Cui
Guangyu Zheng
Wei Wang
Venue:  (2020)
Keywords: natural language inference
first order logic

Abstract: In natural language inference, the semantics of some words do not affect the inference. Such information is considered superficial and brings overfitting. How can we represent and discard such superficial information? In this paper, we use first order logic (FOL) - a classic technique from meaning representation language – to explain what information is superficial for a given sentence pair. Such explanation also suggests two inductive biases according to its properties. We proposed a neural network-based approach that utilizes the two inductive biases. We obtain substantial improvements over extensive experiments.

Under review as a conference paper at ICLR 2020
ROBUST NATURAL LANGUAGE REPRESENTATION LEARNING FOR NATURAL LANGUAGE INFERENCE BY PROJECTING SUPERFICIAL WORDS OUT
Anonymous authors Paper under double-blind review
ABSTRACT
natural language inference, the semantics some words do not affect the infer- ence. Such information is considered superficial and brings overfitting. How can we represent and discard such superficial information? this paper, we use first order logic (FOL) classic technique from meaning representation language explain what information is superficial for a given sentence pair. Such explana- tion also suggests inductive biases according its properties. We proposed a neural network-based approach that utilizes the two inductive biases. We obtain substantial improvements over extensive experiments.
INTRODUCTION
In natural language inference (Bowman et al., 2015), the semantics of some words do not affect the inference. figure la, we discard the semantics of some words (e.g. Avatar, fun, adults, children) from 8ı and $2, obtain s and 5/2 respectively. Without figuring out the specific meaning of these words, one can still infer that they are contradictory. this case, the semantics of Avatar,fun, adults, and children superficial for the inference.
Such superficial information brings overfitting to models. Recent studies already noticed that su- perficial information will hurt the generalization of the model (Jia and Liang, 2017), especially in unseen domains (Wang et al., 2019). Without distinguishing the superficial semantics, an NLI model learn to predict contradiction for sentence pairs with "children" or "adults" by example in Fig- ure la. On the other hand, if we discard the superficial information during inference, we can prevent such overfitting.
81 Avatar is fun for children, not adults. Label: contradiction
82: Avalar is fun for adults, not children.
After discarding Avatar, Avn, adults, children 81= A is B for C, not D.
8/z A is B for D, not C.
Label: contradiction
After discarding Avatar, fun, adults, children and their correspondence information: sy: - is for not
$ is for not
Label: unknown
(a)
83t Avatar is fun for all people. Common sense: People include adults and children. Label: contradiction
84: Avalar is fun for adults only.
Figure 1: Examples.

---

Under review as a conference paper at ICLR 2020
Some approaches have been proposed to reduce such overfitting. HEX (Wang et al., 2019) identifies the superficial information by projecting the textural information out. HEX defines the textural information w.r.t. the background of images for image classification, which cannot be generalized to other tasks (e.g. NLP). For NLP, the attention mechanism (Bahdanau et al., 2015) is able to discard some words by assigning them low attention scores. But such mechanism is more about the semantic similarity or relatedness of the words, not the superficial semantics. In example I of figure 1, the two Avatar the two sentences will have a high attention score, since their similarity is (Vaswani et al., 2017). But we have shown that these words are superficial for inference. So previous approaches cannot be applied to modeling the superficial information in natural language inference.
On top of that, a more critical issue the lack of mathematical definition of such superficial informa- fion in previous studies. Why do people think the semantics of adults and children are superficial? In this paper, we tackle this question via the toolkit of first-order logic (FOL). FOL is a classic technique of meaning representation language, which provides sound computational basis for the inference. We explain such superficial information from the perspective of FOL. Furthermore, such explanation suggests two inductive biases, which are used to design NLI model.
FOL(s1): Vx, Fun(r, Avalar) = Adult(x) A -Child(z)
FOL(s)): Vx, People(a) = Fur(æ, Avalar) FOL(sa); Vr, Fun(a, Avatar) = Adull(()) FOL(CS): lz, People(x) -Adult(x)
FOL()2): Vx, Fun(r, Avatar) = Child(()) A -Adult(z))
Label: contradiction
Label: contradiction
(b)
Figure 2: The FOLs of figure 1.
By representing natural language sentences by FOL, the sentence pair and its FOLs are logically equivalent. The conversion of figure 1a shown in figure 2a. The entailment (resp. contradiction) between and $2 is equivalent to FOL(s;) = FOL(s)) (resp. FOL(s;) + -FOL(ss)). Thus we successfully convert the problem of identifying superficial information in NLI to identifying the superficial information in FOL inference.
The superficial information exists in the non-logical symbols in FOL. From the specification of the FOL representation (Russell and Norvig, 1995), the symbols of FOL include the logical symbols and non-logical symbols. In figure la, the contradiction remains if we discard the semantics of Avatar,fun, adults, children, which are non-logical symbols. We can surely change these non-logical symbols to new symbols without changing the results of FOL(s1) - FOL( or FOL(s1) F -s2)(s)).
However, there is a big gap between the FOL representation and the natural language: people use common sense when understanding the natural language. For example, people are able to infer the contradiction between $3 and in figure lb, because they have the common sense that people include adults and children. The FOLs of 83. $1 and the common sense are shown in figure 2b. With the common sense, the contradiction between sy and is equivalent to CS A FOL(ss) } -FOL(ss), where CS denotes the FOL of the common sense.
With the common sense, some non-logical symbols in the two sentences are not superficial, because we need these non-logical symbols for joint inference with the common sense. For example, in figure 2b, the non-logical symbols Adult and People are not superficial. This brings the major challenge of using FOL. to identify the superficial information, because the common sense can hardly be obtained.
Since the common sense is unknown, we restrict the definition of superficial symbols. We regard non-logical symbol as superficial, if it is superficial for all possible common sense. We show the necessary condition of the superficial symbols to avoid the effect of the common sense, which is unknown. We show that the necessary condition is related to the semantical formula-variable (FV) independence (Lang et al., 2003), which is NP-complete. Nevertheless, the properties of the FOL suggest two inductive biases for superficial information identification: word information discard
2

---

Under review as a conference paper at ICLR 2020
and correspondence information representation. We propose a neural network- based approach to incorporate such two inductive biases.
We point out that need to retain the correspondence information of the discarded words. From the perspective of FOL, although the semantics of some non-logical symbols are independent for inference, the correspondence information still affects the inference. More specifically, we need to represent the occurrence of one word in different positions in the sentence pair. This is also intuitive from the perspective of natural language inference. For example, in figure la, although adults and children are superficial, we need to be aware fort is followed by adults in 81, while for is followed by adults in 82. Otherwise, as illustrated sf and sy, we cannot infer their relation.
We summarize our contributions in this paper below:
We proposed the problem of identifying and discarding superficial information for robust natural language inference. We use FOL to precisely define what information is superficial.
We analyze the superficial information from the perspective of FOL. We show that the superficial non-logical symbols are related to the semantical formula-variable (FV) inde- pendence in reasoning. We give two properties of the superficial information, and design neural networks to reflect the two inductive biases accordingly.
We implement a neural tetwork-based algorithm based on the two inductive biases. The ex- perimental results over extensive settings verify the effectiveness of our proposed method.
2 RELATED WORK
Learning Robust Natural Language Representation. Noticing that traditional neural networks for the natural language casily fail in adversarial examples (Jia and Liang, 2017; Rajpurkar et al., 2018), learning robust representations is important for NLP tasks. A critical metric of the robustness whether the model can be applied to a different data distribution Wang et al. (2019). Adversarial training (Goodfellow et al., 2014) is way to increase the robustness for NLP models (Goodfel- low et al., 2014). It has been applied to NLP tasks such as relation extraction (Wu et al., 2017), sentence classification (Liu et al., 2017). The idea is to use adversarial training to learn unified data distribution for different domains. But the domain-speciio must be known. In contrast, we want to learn robust model that can be applied without knowing the target domain. And we learn robust representations by projecting superficial information out. HEX (Wang et al., 2019) a recent approach to project textural information out of images. relies on two models to represent the whole semantics and superficial semantics, respectively. Few studies reveal how to do this for NLP.
information of the target domain
Omit Superficial Information by Attention. The attention mechanism (Bahdanau al., 2015) gives different weights to different words according to their attention scores. Attention and its variations successful in many NLP tasks (Vaswani et al., 2017: Devlin et al., 2018; Cui et al., 2019). Literally, attention also projects some words out by assigning them low attention scores. However, the attention scores cannot be used to project superficial information of the overlapping words out. Attention gives two words high attention scores they are similar or equal, even if they superficial. So we cannot use attention discard superficial information of overlapping words. As illustrated in section 1, much superficial information for cross-sentence inference lies in these overlapping words.
Natural Language Inference uses neural networks to improve its accuracy (Bowman et al., 2016). Recent studies (Shen et al., 201 20;)b;a) apply attention mechanism (Bahdanau et al., 2015) to model the word correlations. Maate-of-the-al approaches (Devlin et al., 2018; Liu et al., 2019) are fine-tuned over the acaee-cale pre-training models.
3 PRELIMINARIES OF FIRST-ORDER LOGIC
According to the specification of FOL in (Russell and Norvig, 1995). the atoms of FOL include logical symbols (connective, quantifier), and non-logical symbols (constant, variable, predicate, and function). We show the context-free grammar specification of the syntax of them in Table 6. We
3

---

Under review as a conference paper at ICLR 2020
omit the syntaxes of more complicated elements of FOL (e.g. formula) since they are irrelevant to this paper. Examples FOLs are shown in figure 2.
PROBLEM ANALYSIS: FROM THE FIRST-ORDER LOGIC PERSPECTIVE
4.1 FROM NATURAL LANGUAGE INFERENCE TO FIRST-ORDER LOGIC INFERENCE
Firstly, we revealed the relation between natural language inference and FOL inference. The general purpose of NIJ is to determine the contradiction, entailment, and neutral relations of two sentences. If we convert the two sentences into two FOLs, the relation of the FOLs directly reflects the inference label of the two sentences, as shown in Table 1.
NLT label
FOL.
FOL with common sense
entailment
FOL(31 FOL(32)
CS A POL(s) FOL(82)
contradiction
FOL 82] F -F))(s))
-FOL(s2)
neural
otherwise
otherwise
Table 1: NLI labels and FOL relations.
People understand natural language with external common sense. We show the mapping between natural language inference and FOL inference with common sense in table
Obviously, the conversion from natural language sentence FOL sentence is not trivial. We highlight that our paper do not require an algorithm to implement such conversion. We only use FOL explain the superficial information in NLI, and to suggest inductive biases for our algorithm.
4.2 SUPERFICIAL INFORMATION ANALYSIS IN FOLS
We analyze the superficial information in the entailment relation. The other two relations (i.e. con- tradiction and neural) can analyzed similarly. Note that the entailment relation depends on the common sense, which is unknown for NLL. So we restrict the definition of the superficial information FOLS w.r.t. all possible common sense.
Definition 1. Given F()(ss). F(2(s,), with non-logical symbol space V. we define a non-logical symbol ns E V is superficial, if replacing ns to with ns' (s.t. ns' & V) in FOL(s1), FOL(s2) satisfies that VCS.
CS FOL(s) := FOL(s2)
(1)
equivalent to
CS A (OL' += (OL
where (O)) (8)) are the FOLs after the replacement.
Since CS have arbitrary sentences, analyzing the superficial symbols with CS is challenging. We first derive a necessary condition in theorem to avoid the effect of CS.
Theorem 1. Given FO)(s:), FOL(s)), a non-logical symbol ns is superficial, only if FOL(s1) -= FOL(s2)
(3)
equivalent to
FOL'(s)) FOL'(s))
(4)
Theorem provides a necessary condition for identifying superficial non-logical symbols that only considers FOL(s1) and FOL(sz). Thus it is feasible to address whether the necessary condition is true by only using FOL(s)) and FOL()2). The condition in theorem is similar to the semantic FV independence problem (Lang al., 2003) in reasoning, which NP-complete (Lang et al., 2003). However, we can still utilize its properties to help identify the superficial information. We show this in theorem 2.
Theorem 2. Given two FOLs (s) and FOL* with their non-logical symbol set A
{a1," .0,)- }- VB = {br,... b,). where each by is a non-logical symbol, if we replace each O; with b, in FOIA ( and FOLA(82) to get FOL (s1) and FOL (sz) respectively, we have
POLA(ss) (s))
4

---

Under review as a conference paper at ICLR 2020
is equivalent
FOL FOL
Note that both & and B contain distinct non-logical symbols.
Theorem 2 points out that, from the perspective of FOL, the semantics about non-logical symbols do not affect the implication of two FOLs. Note that we need to guarantee that the non-logical symbols in are distinct. We need to reserve the correspondence of these symbols to reserve their relation. The theorem is easy prove because uniformly modifying the non-logical symbols in two FOL does not change their implication.
4.3 FROM SUPERFICIAL INFORMATION IN FOLS TO INDUCTIVE BIAS IN NEURAL NETWORKS
The properties of superficial information in FOL.s suggests what information should discarded in natural language inference. In this subsection, we elaborate two types of inductive biases, and how we use neural network to represent these inductive biases. More details of the neural network are shown in section 4.4.
Word Information Discard From theorem I, the necessary condition of word being superfi- cial is that it corresponds to non-logical symbol, and FOL(s1) } FOL(s2) is equivalent to F(L'(s1) + (82) As we use the word embedding to represent the word information, we scalar for each word to indicate how likely the word is superficial. We multiply the word emhedding by a for each word. Note that one word in different positions should have a unique a, since we assume they correspond to the same symbol and thereby whether they are superficial are identical.
Correspondence Information Representation In theorem although can replace each symbol to a new symbol, the symbols should be replaced accordingly. So for the superficial non-logical symbols, their correspondence information affects the inference. This can be easily illustrated from the perspective of NLI in figure la. If we discard the superficial symbols but reserve their correspon- dence information, we will get s and s/2, from which their contradiction can be still inferred. But if we discard both the superficial symbols and their correspondence information to get s and sy their relation is infeasible to infer. order to represent the correspondence information, we use graph neural network which connects the same words in different positions of the word pairs. Thus the correspondence information is able to propagate through these positions.
4.4 NEURAL NETWORK IMPLEMENTATION
Architecture Our proposed neural network consists of three major modules, which is shown in figure 3. The first module the superficial information projection module, which motivated by the word information discard in section 4.3. For each word w,, we compute its superficial factor ag which is scalar indicating how superficial the word a; means the word corresponds to non-logical symbols that we want to keep the information during inference, or the word corresponds logical symbol. ag = means the word totally useless. The embedding of each word is multiplied by the O:;-
The second module is a standard NLI model. We can use arbitrary NLI models (e.g. ESIM Chen et al. (2017), MwAN Tan et al. (2018)) as this module. The output of this model is sequence of embeddings, indicating the states of the words.
The third module represents the correspondence information in section 4.3. We need to keep the correspondence of the superficial symbols via a graph neural network.
Superficial information projection To discard the words with superficial information, we multiply the embedding of each word by its superficial factor a. More specifically, the embedding of a word w; is computed by:
where wi is in the one-hot representation, F. is the embedding matrix.
5

---

Under review as a conference paper at ICLR 2020
2
1- a
1-
a2
correspondence representation
standard NLI
Fun
for
adults
children
Fun
for
only
children
word embedding for sentencel
word embedding for sentence2
Figure Architecture of the proposed neural network.
Note that or is the same for one word in different positions the sentence pair. To achieve this, we simply use a single perceptron layer over the embeddings compute such a.
a; (M [Ew;; t.] + b)
where M is the parameter matrix for a, [:] denotes the concatenation operation, denotes whether W; is overlapped in the sentence pair (f; = 1) or not (4: 0).
represent the cross-sentence correspondence information, we use graph neural network. For the same word which occurs in different positions in the sentence pair, we use an edge between all position pairs to represent the correspondence information. Intu- itively, for words that superficial, we only need retain their correspondence information, and vice versa. As a; denotes whether the information should be retained, we set the weight the edge to O; for word w;- More formally, total length of the sentence pair and is the dimension of the hidden states. By following the graph neural network in Kipf and Welling (2016), update 5T by:
Correspondence representation
denote the states at time S Raxd where is the
S = o -w
where W is the parameter matrix, 5 is the output of the standard NLI module, and A E Raxx the adjacency matrix represent such correspondence:
04 0 as)) if j. Wy 0
if
Arj
(10)
otherwise.
where 0 is used to make the sum of each row in A equals to 1. Figure show how we connect the words "fun", "for", and "children" in different positions in the sentence pair. By using the edges, even if the model discards the semantics of "children". behind "and" in the first sentence, and behind "only" in the second sentence. Therefore retain the correspondence information by the graph neural network.
able to represent that the word
EXPERIMENTS
SETUP
Datasets We use the datasets including MNLI (Williams et al., 2018), SNLI (Bowman al., 2015), QNLI (Wang et al., 2018), DNLI (Welleck et al., 2018), RTE (Dagan et al., 2005), MRPC (Dolan and Brockett, 2005), and SciTail (Khot al., 2018). More details shown in appendix D.

---

Under review as a conference paper at ICLR 2020
Competitors Since our proposed framework different NLI models the second module, use standard NLI models for both comparison and for NLI module. These models include BiLSTM, ESIM (Chen et al., 2017), MwAN (Tan al., 2018), and CAFE (Tay et al., 2018). We compare with HEX (Wang et al., 2019), which projects superficial statistics out. We also compare with the pre- training model Elmo (Peters et al., 2018), Roberta (Liu et al., 2019), which achieves aate-of-the-art results in NLI. More details of the experimental setup are shown in appendix D and appendix E.
5.2 SINGLE DOMAIN EVALUATION
Effectiveness We evaluate the effectiveness of our proposed approaches in the single domain setting. The training and test data are from the same domain. Table 2 shows the performances of different models. Ours A denotes applying algorithm A as the standard NLI module in our proposed neural network. Our proposed method constantly outperforms the original model by large margin.
Model
MRPC
RTE
QNLI
SeiTail
SNLI
MNLI
DNLI
DNLI(gold) Avg.
HEX
73.6/82.8
53.1
49.6
84.3
52.8
60.6/60.9
69.5
70.8
65.8
BiLSTM
69.7/80.5
54.7
74.0
77.0
82.5
68.8/69
86.7
91.7
75.5
BiLSTM+ours
77.6/84.5
58.5
80.3
83.8
85.5
75.2/74.2
87.0
91.4
79.8(+4.3)
ESIM
68.7/80.8
53.4
80.9
82.8
88.1
77.4/76.7
87.9
92.8
79.0
ESIM+ours
76.9/84.1
57.6
80.8
84.1
88.3
78.5/77.5
88.7
93.2
81.0(+2.0)
MwAN
68 68.87
51.9
69.4
71.2
85.2
74.1/73.3
86.0
90.3
75.1
MwAN+ourS 76.7/83.6
59.9
81.9
84.2
82.6
73/73
85.3
88.9
78.9(43.8)
CAFE
69.1/80.6
53.4
82.2
81.2
86.8
76.3/76
88.1
92.8
78.7
CAFE+ours
76.5/83.8
58.4
83.6
85.6
86.4
75.2/74.7
89.0
93.3
80.7(+2.0)
Table 2: Performance over single domain NLI and single domain PI (MRPC). For MRPC, we report the accuracy and fl -corc. For MNLI, we report the accuracy on both matched and mismatched test sets. For the rest datasets, we report the accuracy.
Ablations We evaluate the effectiveness of the two inductive biases in section 4.3, i.e., word infor- mation discard and correspondence information representation. We use an ablation study in Table 3 to evaluate them. Here - word means no word discard (i.e. a only works in the correspondence representation module). -orrespond means no correspondence representation module. From the results, both inductive biases improve the effectiveness. The word information discard is more cru- cial.
Model
MRPC
RTE
QNLI
SciTail
SNLI
MNLI
DNLI
DNLI(gold)
Avg.
BiLSTM+ouu
77.6/84.5
58.5
80.3
83.8
85.5
75,2/74.2
87.0
91.4
79.8
-correspond
77/84.1
60.1
79.9
77.1
86.1
75/74.3
86.7
91.2
79.2(-0.6)
-word
74.3/82.5
50.9
80.2
82.4
84.2
72.5/71.8
85.9
90.3
77.5(-2.3)
ESIM+ours
76.9/84.1
57.6
80.8
84.1
88.3
78.5/77.5
88.7
93.2
81.0
correspond
75.1/83.3
58.6
81.5
83.8
88.1
78.6/77
88.6
93.3
80.8(-0.2)
-word
69.4/80.1
55.6
80.2
83.6
88.2
77.9/76.9
88.7
93.2
79.4(-1.6)
Table 3: Ablation over single domains.
5.3 RESULTS OVER PRE-TRAINING MODELS
-fate-of-the-ar NLI results are from the fine-tuning of pre-training models. We use Elmo (Peters et al., 2018) and Roberta Liu et al. (2019), a recent pre-training model, as the word embeddings module in our architecture Liu et al. (2019). We use the pooling layer in ESIM for final classi- fication. The results are shown in Table 4. While our proposed method outperforms the original ESIM+ELMO by large margin. the accuracies are slightly improved for Roberta. This makes sense because Roberta already reached a very high accuracy.
5.4 EVALUATION FOR UNSEEN DOMAINS
We evaluate the robustness of our approaches in unseen domains. We choose one dataset as the source domain for training, and another dataset as the target unseen domain for testing. The model is only trained by the training data in the source domain.
7

---

Under review as a conference paper at ICLR 2020
Model
MRPC
RTE
QNLI
Scifail
SNLI
MNLI
DNLT
(Nold)
Avg.
Moo+ESIM
70.8/81.4
54.1
81.3
81.8
88.4
79.7/78.5
88.8
93.7
79,9
Elmo+ESIM+ourS 80.0/85.8
60.8
82.5
86.3
88.7
79.8/79.2
89.2
93.5
82.6(+2.7)
Roberta
88.2/91.4
72.1
92.6
93.6
91.0
87.2/86.8
91.2
95.9
89.0
Roherta+ours
88.6/91.6
73.1
92.8
93.5
91.4
87.3/86.7
91.6
95.9
89.3(+0.3)
Table 4: Results over pre-training models.
NLI (3 classes)
NLI(2 classes)
Source
DNLI
DNLI MNLI
MNLI SNLI
MNLI
SNLI
SNLI MNLI
RTE
SciTail
AVG.
Target
SNLI
DNLI
Gold
DNLI
Gold
SciTail
RTE
HEX
33.3
36.9/36.5
52.8
49.6
50.9
34.4
50.9
38.0/38.5
52.9
53.4
44.0
BiLSTM
37.0
38.5/38.2
54.5
46.5
48.9
39.4
40.4
54.3/56.1
47.3
54.1
46.3
BiLSTM+ours
36,4
37.4/37.9
64.1
56,2
58.7
46.9
48.3
60.7/60.2
63.5
56.7
52.3(+6.0)
ESIM
36.7
37.2/37.5
68.1
61.4
64.8
47.5
48.8
62.9/62.6
55.8
53.6
53.2
ESIM+ours
37.7
38.5/39.6
69.2
62.2
65.3
48.8
49.9
63.4/63.5
57.3
58.4
54.5(+1.3)
MwAN
38.0
38.4/38.2
63.6
55.5
58.7
39.7
40.3
58.2/58.9
55.1
49.7
49.5
MwAN+ours
36.9
38.9/39.9
62.0
57.4
60.3
48.1
49.3
59.3/59.3
54.2
58.6
52.0(+2.5)
CAFE
37.9
38.5/39.2
67.5
60.1
63.5
48.2
49.7
62. 1/61/4
41.4
56.1
52.1
CAFE+ourS
38.0
37.6/38.4
67.8
59.6
63.1
48.0
49.4
62.3/62.2
60.1
56.3
53.6(+1.55)
Table 5: Performance in unseen domains.
"Gold" denotes the gold-standard test set of DNLI.
Table shows the performance of different models. From the results, we see that by using our proposed method, the accuracy improves significantly.
5.5 VISUAIIZZTTION OF THE PROJECTION
We visualize the to deeply analyze its performance in Figure 4. Each grid of word represent its a. Our approach successfully projects superficial words out. For example, in figure 4a, the words "women" and "bar" are mostly discarded, while both words do not affect the inference. The same intuitive discarding happens in the words "man" and "shirt" in figure 4b. We also visualize and analyze the attention mechanism in appendix G.
Two
Pavion
dtoke
smatong
mmm
black
ahirc
slering
Delr
outside
1.0
0.8
0,6
Nornen
the
in
bacck
trackes
Das pakeman
cards
with
hhs
0.4
(b)
Figure 4: for two sentence pairs.
CONCLUSION
In this paper, study the problem of projecting superficial information out for NLI. The projection prevents models from overfitting and makes them more robust. Specially, we explain the superficial information from the perspective of FOL, and project them out in neural network- based architec- ture. We conduct extensive experiments to verify the effectiveness of our proposed approach. The results verify that proposed approaches increase the baselines by large margin.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. 2014. adwaan-adversarial neural networks. In NeurIPS.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning align and translate.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In Pro- ceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1466-1477.
Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced Istm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017). ACL.
Wanyun Cui, Guangyu Zheng, Zhigiang Shen, Sihang Jiang, and Wei Wang. 2019. Transfer learning for sequences via learning to collocate. International Conference on Learning Representations.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177-190. Springer.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of
deep bidirectional transformers for language understanding. arXiv preprint 1xXiv:1810.04800
William B Dolan and Chris Brockett. 2005. Automatically constructing corpus of sentential para- phrases. In Proceedings of the Third International Workshop on Paraphrasing (/WP2005).
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension sys- tems. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Pro- cessing, pages 2021-2031.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A textual entailment dataset from science question answering. In AAAI.
Thomas N. Kipf and Max Welling, 2016. Semi-supervised classification with graph convolutional networks. ICLR.
Jérôme Lang, Paolo Liberatore, and Pierre Marquis. 2003. Propositional independence-formui variable independence and forgetting. Journal of Artificial Intelligence Research, 18:391-443.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text clas- sification. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), volume 1, pages 1-10.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized BERT pretraining approach. CoRR, abs/1907. 11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. 2013. Domain generalization via invariant feature representation. In International Conference Machine Learning, pages 10-18.

---

Under review as a conference paper at ICLR 2020
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't 't know: Unanswerable questions for squad. In Proceedings of the 56th Anmual Meeting of the Association for Computa- fional Linguistics (Volume 2: Short Papers), volume 2, pages 784-789.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. 2018. On the convergence of adam and beyond. In International Conference on Learning Representations.
Stuart Russell and Peter Norvig. 1995. Artificial intelligence: a modern approach.
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018a. Disan: Directional self-attention network for an/cnn-free language understanding. Thirty-Second AAAI Conference Artificial Intelligence.
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Sen Wang, and Chengqi Zhang. 2018b. Re- inforced self-attention network: hybrid of hard and soft attention for sequence modeling. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4345- 4352.
Chuanqi Tan, Furu Wei, Wenhui Wang, Weifeng Lv, and Ming Zhou. 2018. Multiway attention networks for modeling sentence pairs. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4411-4417.
Yi Tay, Anh Tuan Luu, and Siu Cheung Hui. 2018. Compare, compress and propagate: Enhancing neural architectures with alignment factorization for natural language inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1565-1575.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NewrlPS, pages 5998- 6008.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018, page 353.
Haohan Wang, Zexue He, and Eric P. Xing. 2019. Learning robust representations by projecting superficial statistics out. In International Conference on Learning Representations.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2018. Dialogue natural language inference. arXiv preprint
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 1112-1122.
Yi Wu, David Bamman, and Stuart Russell. 2017. Adversarial training for relation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1778-1783.
A FORMAL PROBLEM SETUP FOR THE SINGLE DOMAIN AND THE UNSEEN DOMAIN
Single domain In the single domain setting, the training data and the test data have the same dis- tribution Px, y. More formally, given training data Train (In Un Px,y)
IN the goal is to
10

---

Under review as a conference paper at ICLR 2020
predict the labels of the test data Test {m ym Px.y 1 Each 2; consists of two tences 8/, sf. For example, in NLI, each yı E [neutral, contradiction, entailment) indicates the relation between of and sf. Each sentence is sequence of words.
Unseen domains To evaluate the generality of the model after discarding superficial information, we also consider its effectiveness an unseen domain. More formally, suppose the source domain and the target domain have distribution PX y and P respectively. We evaluate the model that trained on the source domain (rain(*) {ah y
(a) P(+] 1. and is tested on the
target domain Test()) = {a'm u
Xxy m=1- Note that in the unseen domain NLI, PX p!f) JM
and (est(*) unknown during training. This setting is more challenging than traditional domain- adaptation (Ajakan et al., 2014; Cui et al., 2019) and domain generalization (Muandet et al., 2013) from the perspective that the test domain is unknown during training.
B THE SYNTAX OF FOL
Syntax of the atoms in FOL
Symbol type
Connective + VlA I=
logical
Quantafier - V/
logical
Constant A VegetarianFood
non-logical
Varaible + xlyl...
non-logical
Predicate + Serves Near
non-logical
Function LocationOf [ocationO[[CsssseOOO
non-logical
Table The syntax of FOL, specified Backus-Naur form (Russell and Norvig, 1995).
C
PROOF OF THEOREM 1
Proof. For a non-logical symbol ns, since VC',
CS A FOL(s)) F FOL(s2)
(11)
is equivalent to
CS F = (32)
(12)
For CS = True, CS A FOL(ss) = FOL(s1), CS A FOL'(s1) FOL'(s)). Thus for non- logical symbol ns, we have
FOL(s)) F (OL(82)
(13)
equivalent to
FOL FOL
(14)
D EXPERIMENTAL SETTINGS AND DATASETS
All the experiments run over computer with Intel Core i7 4.0GHz CPU, 32GB RAM, and GeForce GTX 1080 Ti GPU. For SNLI, we remove the "the other" category to make its labels comparable with MNLL We evaluate the accuracy and fl-score for MRPC, since its labels are im- balanced. We list the statistics the datasets Table 7.
E NEURAL NETWORK DETAILS
We use pooling layer over proposed architecture for sentence pair classification. For ESIM, MwAN and CAFE, we use the pooling layer in their original papers. For BiL.TM, we follow (Wang et al.. 2018). We use
pooling layer to produce the vectors u, of each sentence, and pass [u: v; ju w/: u * vi to an MLP classifier which has a hidden layer with tanh activation. We apply flma the output layer.
11

---

Under review as a conference paper at ICLR 2020
#Train
#Dev
#lest
Avg.L.
Vocab.
MNLI
392702
9815/9832
9796/9847
17.0
95871
SNLT
549367
9842
9824
11.2
40257
MRPC
4076
1725
22.3
18765
QNLI
I04743
5463
5463
21.2
89686
RTE
2490
277
3000
29.5
26751
DNLI
310110
16500
16500/12376
9.4
15568
Scilail
23596
1304
2126
16.4
28169
Table 7: Statistics of datasets. Avg.L. denotes the average length of each review. Vocab. denotes the vocabulary size. We show the size of the matched/mismatched test sets for MNLI.
For Roberta, we split the outputs of Roberta into Uaa th according to the (SEP) separator. Then we use the pooling layer in ESIM. We our proposed method and its ablations, and choose the best model w.r.t. the developing dataset.
Hex (Wang al., 2019) relies on textural model to generate superficial information, and raw model to generate all information. We use a two-layer BiLSTM over the overlapping words to generate the superficial information. And we another BiLSTM over the raw sentence generate all information.
Hlper-paraameerd For BiLSTM, the dimension of the hidden states is 300. For other models, we use the dimension of the hidden states as their original papers. For ESIM, CAFE, and MwAN, their dimensions are set to 300, 300, 75, respectively. We use the AMSGrad (Reddi al.. 2018) optimizer except Roberta, in which we use AdamW (Loshchilov and Hutter, 2019). We use 300d Vl vectors (Pennington et al., 2014) as the initialization for the word embedding except Roberta. For the T-layer graph neural network, to achieve the best performance, we set T = 3 for BiLSTM, ESIM and Roberta, T for MwAN, and T 1 for CAFE.
F ABLATIONS OVER UNSEEN DOMAINS
We show the ablations over unseen domains in table
NLI (3 classes)
NLI(2 classes)
Source
DNLI
DNLI MNLI
MNLI SNLI
MNLI
SNLI
SNLI
RTE
Sci
AVG.
Target
SNLI
DNLI
Gold
DNLI
Gold
MNLI
SciTail
RTE
STM
36.4
37.4/37.9
64.1
56.2
58.7
46.9
48.3
60.7/60.2
63.5
56.7
52.3
-correspond
36.0
38.2/38.1
65.2
57.2
60.2
46.3
47.9
60.7/60.3
60.9
53.4
52.2(-0.1)
-word
37.8
40.1/39.3
61.2
50.1
52.3
40.6
40.7
57.6/57.1
48.5
53.5
48.2(-4.1)
ESIM+ours
37.7
38.5/39.6
69.2
62.2
65.3
48.8
49.9
63.4/63.5
57.3
58.4
54.5
-correspond
37.0
38.2738.5
68.8
61.0
64.1
50.0
51.3
64.9/64.0
56.3
59.0
54.4(-0.1)
-word
36.0
37.3/38.1
67.3
59.8
62.7
46.7
47.7
62.5/62.5
49.7
55.6
52.2(-2.3)
Table 8: Ablations over unseen domains. For MNLI, we report the accuracy on both matched and mismatched test sets. "Gold" denotes the gold-standard test set of DNLI.
G EFFECT OF THE SUPERFICIAL INFORMATION IN ATTENTION
We further investigate what is projected and what is reserved from the attention perspective. Figure show the attention matrix w/o discarding superficial information in ESIM. Clearly, the attention matrix after discarding superficial information is more intuitive. It concentrates on "usual" and "slightly lower", which imply the contradiction relation. In contrast, the matrix of the standard ESIM focus on the repeated words (c.g. "Toronto",, "stock"), which are not critical for the inference.
12

---

Under review as a conference paper at ICLR 2020
stock
exchange
exchange
said
said
tt
It-
wil
will
be
bushess
bushess
0.3
0.2
friday
moming
moming
0.1
und
torant
umme
tower anghtyy and
the
tomett
mach rarge sock
opennd
time
and
an angghty
towar
(a) Attention matrix standard ESIM.
(b) Attention matrix our method.
Figure 5: Attention matrix visualization. The x-axis and the y-axis denote words in the two sen- tences.
13


Reviews:
[{"review_id": "rke9JLOAYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tried to reduce superficial information in natural language inference (NLI) to prevent overfitting. It utilized the first order logic to explain what is superficial information.\nThen, it introduced a superficial factors in the existing neural networks. Furthermore, they introduce a graph neural network (GNN) to model relation between premise and hypothesis.\nIt was evaluated on a bunch of NLI benchmarks including SNLI, SciTail, MNLI etc, showing the effectiveness of the proposed model. \n\nThis paper is well motivated and the ideas are interesting. However, there are a few concerns detailed as follows: \n\n1. Do these methods only work on small tasks? For example, the big improvement only appears in small tasks such as MRPC and RTE. However, the proposed method experiences performance decreases on large tasks such as SNLI and MNLI. E.g., in CAFE settings, the proposed approach got  75.2/74.7 (matched/mismatched) vs 76.3/76 (baselines). The similar observations are found in MwAN settings on MNLI and SNLI. I’d like to see some discussion in the paper on this.\n\n2. What common logic patterns did the model learn when removing all the superficials? For different relations, e.g., entailment vs contradiction, are these patterns different? This may help the reader understand whether the model really filtered these information. \n\n3. It requires more discussions on Table 5. E.g., in NLI (2 classes), the random guess should be 50%. But the model performance was 41.4% on CAFE when transferring from RTE to SciTail, which is even worse than random guess. In contrary, when transferring from SciTail to RTE, the model performance was 56.1%, which seems reasonable. I believe more analysis and discussions are required to understand this model.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1lxoVPRKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses first order logic (FOL) to help reduce so-called “superficial” information/semantics that is less relevant to the judgement of natural language inference relations. The submission misses the major literature of and comparison to previous work that uses FOL for natural language inference (aka. RTE), for example, [Bos and Markert ‘05], [Beltagy et al. ‘13], [Abzianidze ‘17], among others, as well as work based on natural logic, e.g., [MacCartney ‘09], which operates directly on parsed sentences. The submission contains little contribution with regard to the exiting work. Key concepts such as “superficial semantics” is vague and not well defined. I do not recommend it for the conference. \n\nBos and Markert ‘05, Recognising Textual Entailment with Robust Logical Inference. \nBeltagy et al. ‘13, Montague Meets Markov: Deep Semantics with Probabilistic Logical Form.\nAbzianidze ’17, A Natural Proof System for Natural Language.\nMacCartney ’09, natural language inference (PhD thesis)\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byx4K0PJqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to treat natural language inference using first-order logic, and to infuse neural NLI models with logical information to be more robust at inference. However, the paper does not contain a single reference to the computational semantics literature, where logical approaches towards semantics were the dominant trend for many years (see e.g. [1, 2]). Indeed, 'neuralising' first order logic has been an active area of recent research ([3] or indeed much of the recent work coming from Sebastian Riedel's group). This is a glaring oversight.\n\nThe paper starts by introducing background on first-order logic, and then gives a definition of a 'superficial' predicate, namely one whose extension is not necessary to prove an implication for any collection of background facts. However, by extension, this makes s_1 -> s_2 a tautology, which is the 'true' notion that the authors are looking for. Indeed, if |- (s_1 -> s_2), then for any collection of formulae \\Delta then \\Delta |- (s_1 -> s_2) (by monotonicity of entailment) and clearly if for any \\Delta we have \\Delta |- (s_1 -> s_2), we can take \\Delta to be the empty set. Finally, the authors show that tautologies are still tautologies under change of predicates (i.e. if we only require logical rules to prove one statement from another, then the extensions of predicates in those statements do not matter).\n\nThe authors then use this to motivate two extensions to inference models. One is to 'drop out' word information, and the other is to treat different occurrences of the same word as reflecting the same underlying predicate. The first somewhat transparently forces the model to care less about the exact meaning (i.e. extension in the logical world) of words (indeed, word vectors have been shown to capture extensional information [4, 5]), and so may force the inference model to learn more 'logical' inference rules. Further, the word dropout calculation includes whether the word is in both sentences, which is a strong signal that its extension may not be necessary. However, the second only forces the intuition that different mentions of the same word are likely to be coreferent, which is a weak assumption that models may already pick up. Indeed, it is noticeable that this component seems to be less necessary in the authors' ablation study.\n\nIn summary, while I am sympathetic to the aim of grounding neural models in explicit notions of semantics, this paper shows such a lack of awareness of previous literature that I cannot recommend acceptance. \n\n[1] The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity, Bjerva et al. 2014\n[2] Natural Logic for Textual Inference, MacCartney and Manning 2009\n[3] End-to-end Differentiable Proving, Rocktaschel and Riedel 2017\n[4] Building a shared world: mapping distributional to model-theoretic semantic spaces, Vecchi and Herbelot 2015\n[5] Deriving Boolean structures from distributional vectors, Kreuzewski et al 2015", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes using first order logic to rule out superficial information for improved natural language inference. While the topic is of interest, reviewers find that the paper misses much of the previous literature on semantics which is highly relevant. 

I thank the authors for submitting this paper to ICLR. Please take the reviewers' comments, especially recommended references, to improve the paper for future submission.
Decision: Reject