Robust Natural Language Representation Learning for Natural Language Inference by Projecting Superficial Words out

1 INTRODUCTION . In natural language inference ( Bowman et al. , 2015 ) , the semantics of some words do not affect the inference . In figure 1a , if we discard the semantics of some words ( e.g . Avatar , fun , adults , children ) from s1 and s2 , we obtain s′1 and s ′ 2 , respectively . Without figuring out the specific meaning of these words , one can still infer that they are contradictory . In this case , the semantics of Avatar , fun , adults , and children are superficial for the inference . Such superficial information brings overfitting to models . Recent studies already noticed that superficial information will hurt the generalization of the model ( Jia and Liang , 2017 ) , especially in unseen domains ( Wang et al. , 2019 ) . Without distinguishing the superficial semantics , an NLI model can learn to predict contradiction for sentence pairs with “ children ” or “ adults ” by example 1 in Figure 1a . On the other hand , if we discard the superficial information during inference , we can prevent such overfitting . Some approaches have been proposed to reduce such overfitting . HEX ( Wang et al. , 2019 ) identifies the superficial information by projecting the textural information out . HEX defines the textural information w.r.t . the background of images for image classification , which can not be generalized to other tasks ( e.g . NLP ) . For NLP , the attention mechanism ( Bahdanau et al. , 2015 ) is able to discard some words by assigning them low attention scores . But such mechanism is more about the semantic similarity or relatedness of the words , not the superficial semantics . In example 1 of figure 1 , the two Avatar in the two sentences will have a high attention score , since their similarity is 1 ( Vaswani et al. , 2017 ) . But we have shown that these words are superficial for inference . So previous approaches can not be applied to modeling the superficial information in natural language inference . On top of that , a more critical issue is the lack of mathematical definition of such superficial information in previous studies . Why do people think the semantics of adults and children are superficial ? In this paper , we tackle this question via the toolkit of first-order logic ( FOL ) . FOL is a classic technique of meaning representation language , which provides a sound computational basis for the inference . We explain such superficial information from the perspective of FOL . Furthermore , such explanation suggests two inductive biases , which are used to design our NLI model . By representing natural language sentences by FOL , the sentence pair and its FOLs are logically equivalent . The conversion of figure 1a is shown in figure 2a . The entailment ( resp . contradiction ) between s1 and s2 is equivalent to FOL ( s1 ) |= FOL ( s2 ) ( resp . FOL ( s1 ) |= ¬FOL ( s2 ) ) . Thus we successfully convert the problem of identifying superficial information in NLI to identifying the superficial information in FOL inference . The superficial information exists in the non-logical symbols in FOL . From the specification of the FOL representation ( Russell and Norvig , 1995 ) , the symbols of FOL include the logical symbols and non-logical symbols . In figure 1a , the contradiction remains if we discard the semantics of Avatar , fun , adults , children , which are non-logical symbols . We can surely change these non-logical symbols to new symbols without changing the results of FOL ( s1 ) |= FOL ( s2 ) or FOL ( s1 ) |= ¬FOL ( s2 ) . However , there is a big gap between the FOL representation and the natural language : people use common sense when understanding the natural language . For example , people are able to infer the contradiction between s3 and s4 in figure 1b , because they have the common sense that people include adults and children . The FOLs of s3 , s4 and the common sense are shown in figure 2b . With the common sense , the contradiction between s3 and s4 is equivalent to CS ∧ FOL ( s3 ) |= ¬FOL ( s4 ) , where CS denotes the FOL of the common sense . With the common sense , some non-logical symbols in the two sentences are not superficial , because we need these non-logical symbols for joint inference with the common sense . For example , in figure 2b , the non-logical symbols Adult and People are not superficial . This brings the major challenge of using FOL to identify the superficial information , because the common sense can hardly be obtained . Since the common sense is unknown , we restrict the definition of superficial symbols . We regard a non-logical symbol as superficial , if it is superficial for all possible common sense . We show the necessary condition of the superficial symbols to avoid the effect of the common sense , which is unknown . We show that the necessary condition is related to the semantical formula-variable ( FV ) independence ( Lang et al. , 2003 ) , which is NP-complete . Nevertheless , the properties of the FOL suggest two inductive biases for superficial information identification : word information discard and correspondence information representation . We propose a neural network-based approach to incorporate such two inductive biases . We point out that we need to retain the correspondence information of the discarded words . From the perspective of FOL , although the semantics of some non-logical symbols are independent for inference , the correspondence information still affects the inference . More specifically , we need to represent the occurrence of one word in different positions in the sentence pair . This is also intuitive from the perspective of natural language inference . For example , in figure 1a , although adults and children are superficial , we need to be aware that for is followed by adults in s1 , while for is followed by adults in s2 . Otherwise , as illustrated in s′′1 and s ′′ 2 , we can not infer their relation . We summarize our contributions in this paper below : • We proposed the problem of identifying and discarding superficial information for robust natural language inference . We use FOL to precisely define what information is superficial . • We analyze the superficial information from the perspective of FOL . We show that the superficial non-logical symbols are related to the semantical formula-variable ( FV ) independence in reasoning . We give two properties of the superficial information , and design neural networks to reflect the two inductive biases accordingly . • We implement a neural network-based algorithm based on the two inductive biases . The experimental results over extensive settings verify the effectiveness of our proposed method . 2 RELATED WORK . Learning Robust Natural Language Representation . Noticing that traditional neural networks for the natural language easily fail in adversarial examples ( Jia and Liang , 2017 ; Rajpurkar et al. , 2018 ) , learning robust representations is important for NLP tasks . A critical metric of the robustness is whether the model can be applied to a different data distribution Wang et al . ( 2019 ) . Adversarial training ( Goodfellow et al. , 2014 ) is one way to increase the robustness for NLP models ( Goodfellow et al. , 2014 ) . It has been applied to NLP tasks such as relation extraction ( Wu et al. , 2017 ) , sentence classification ( Liu et al. , 2017 ) . The idea is to use adversarial training to learn a unified data distribution for different domains . But the domain-specific information of the target domain must be known . In contrast , we want to learn a robust model that can be applied without knowing the target domain . And we learn robust representations by projecting superficial information out . HEX ( Wang et al. , 2019 ) is a recent approach to project textural information out of images . It relies on two models to represent the whole semantics and superficial semantics , respectively . Few studies reveal how to do this for NLP . Omit Superficial Information by Attention . The attention mechanism ( Bahdanau et al. , 2015 ) gives different weights to different words according to their attention scores . Attention and its variations are successful in many NLP tasks ( Vaswani et al. , 2017 ; Devlin et al. , 2018 ; Cui et al. , 2019 ) . Literally , attention also projects some words out by assigning them low attention scores . However , the attention scores can not be used to project superficial information of the overlapping words out . Attention gives two words high attention scores if they are similar or equal , even if they are superficial . So we can not use attention to discard superficial information of overlapping words . As illustrated in section 1 , much superficial information for cross-sentence inference lies in these overlapping words . Natural Language Inference uses neural networks to improve its accuracy ( Bowman et al. , 2016 ) . Recent studies ( Shen et al. , 2018b ; a ) apply attention mechanism ( Bahdanau et al. , 2015 ) to model the word correlations . State-of-the-art approaches ( Devlin et al. , 2018 ; Liu et al. , 2019 ) are fine-tuned over the large-scale pre-training models . 3 PRELIMINARIES OF FIRST-ORDER LOGIC . According to the specification of FOL in ( Russell and Norvig , 1995 ) , the atoms of FOL include logical symbols ( connective , quantifier ) , and non-logical symbols ( constant , variable , predicate , and function ) . We show the context-free grammar specification of the syntax of them in Table 6 . We omit the syntaxes of more complicated elements of FOL ( e.g . formula ) since they are irrelevant to this paper . Examples of FOLs are shown in figure 2 . 4 PROBLEM ANALYSIS : FROM THE FIRST-ORDER LOGIC PERSPECTIVE . 4.1 FROM NATURAL LANGUAGE INFERENCE TO FIRST-ORDER LOGIC INFERENCE . Firstly , we revealed the relation between natural language inference and FOL inference . The general purpose of NLI is to determine the contradiction , entailment , and neutral relations of two sentences . If we convert the two sentences into two FOLs , the relation of the FOLs directly reflects the inference label of the two sentences , as shown in Table 1 . People understand natural language with external common sense . We show the mapping between natural language inference and FOL inference with common sense in table 1 . Obviously , the conversion from a natural language sentence to a FOL sentence is not trivial . We highlight that our paper do not require an algorithm to implement such conversion . We only use FOL to explain the superficial information in NLI , and to suggest inductive biases for our algorithm . 4.2 SUPERFICIAL INFORMATION ANALYSIS IN FOLS . We analyze the superficial information in the entailment relation . The other two relations ( i.e . contradiction and neural ) can be analyzed similarly . Note that the entailment relation depends on the common sense , which is unknown for NLI . So we restrict the definition of the superficial information in FOLs w.r.t . all possible common sense . Definition 1 . Given FOL ( s1 ) , FOL ( s2 ) , with non-logical symbol space V , we define a non-logical symbol ns ∈ V is superficial , if replacing ns to with ns′ ( s.t . ns′ 6∈ V ) in FOL ( s1 ) , FOL ( s2 ) satisfies that ∀CS , CS ∧ FOL ( s1 ) |= FOL ( s2 ) ( 1 ) is equivalent to CS ∧ FOL′ ( s1 ) |= FOL′ ( s2 ) ( 2 ) , where FOL′ ( s1 ) , FOL′ ( s2 ) are the FOLs after the replacement . Since CS can have arbitrary sentences , analyzing the superficial symbols with CS is challenging . We first derive a necessary condition in theorem 1 to avoid the effect of CS . Theorem 1 . Given FOL ( s1 ) , FOL ( s2 ) , a non-logical symbol ns is superficial , only if FOL ( s1 ) |= FOL ( s2 ) ( 3 ) is equivalent to FOL′ ( s1 ) |= FOL′ ( s2 ) ( 4 ) Theorem 1 provides a necessary condition for identifying superficial non-logical symbols that only considers FOL ( s1 ) and FOL ( s2 ) . Thus it is feasible to address whether the necessary condition is true by only using FOL ( s1 ) and FOL ( s2 ) . The condition in theorem1 is similar to the semantic FV independence problem ( Lang et al. , 2003 ) in reasoning , which is NP-complete ( Lang et al. , 2003 ) . However , we can still utilize its properties to help identify the superficial information . We show this in theorem 2 . Theorem 2 . Given two FOLs FOLA ( s1 ) and FOLA ( s2 ) , with their non-logical symbol set A = { a1 , · · · , an } . ∀B = { b1 , · · · , bn } , where each bi is a non-logical symbol , if we replace each ai with bi in FOLA ( s1 ) and FOLA ( s2 ) to get FOLB ( s1 ) and FOLB ( s2 ) respectively , we have FOLA ( s1 ) |= FOLA ( s2 ) ( 5 ) is equivalent to FOLB ( s1 ) |= FOLB ( s2 ) ( 6 ) . Note that both A and B contain n distinct non-logical symbols . Theorem 2 points out that , from the perspective of FOL , the semantics about non-logical symbols do not affect the implication of two FOLs . Note that we need to guarantee that the n non-logical symbols in B are distinct . We need to reserve the correspondence of these symbols to reserve their relation . The theorem is easy to prove because uniformly modifying the non-logical symbols in two FOL does not change their implication .

Summary: This paper presents an approach to treat natural language inference using first-order logic, and to infuse neural NLI models with logical information to be more robust at inference. However, the paper does not contain a single reference to the computational semantics literature, where logical approaches towards semantics were the dominant trend for many years (see e.g. [1, 2]). Indeed, 'neuralising' first order logic has been an active area of recent research ([3] or indeed much of the recent work coming from Sebastian Riedel's group). This is a glaring oversight.