Logic Pre-Training of Language Models

1 INTRODUCTION . Machine reasoning in natural language understanding ( NLU ) aims to teach machines to understand human languages by building and analyzing the connections between the facts , events , and observations using logical analysis techniques like deduction and induction , which is one of the ultimate goals towards human-parity intelligence . Although pre-trained language models ( PrLMs ) , such as BERT ( Devlin et al. , 2018 ) , GPT ( Radford et al. , 2018 ) , XLNet ( Yang et al. , 2019 ) and RoBERTa ( Liu et al. , 2019 ) , have established state-of-the-art performance on various aspects in NLU , they are still short in complex language understanding tasks that involve reasoning ( Helwe et al. , 2021 ) . The major reason behind this is that they are insufficiently capable of capturing logic relations such as negation ( Kassner & Schütze , 2019 ) , factual knowledge ( Poerner et al. , 2019 ) , events ( Rogers et al. , 2020 ) , and so on . Many previous studies ( Sun et al. , 2021 ; Xiong et al. , 2019 ; Wang et al. , 2020 ) are then motivated to inject knowledge into pre-trained models like BERT and RoBERTa . However , they too much rely on massive external knowledge sources and ignore that language itself is a natural knowledge carrier as the basis of acquiring logic reasoning ability ( Ouyang et al. , 2021 ) . Taking the context in Figure 1 as an example , previous approaches tend to focus on entities such as the definition of `` government '' and the concepts related to it like `` governor '' , but overlook the exact relations inherent in this example , thus failing to model the complex reasoning process . Given the fact that PrLMs are the key supporting components in natural language understanding , in this work , we propose a fundamental solution by empowering the PrLMs with the capacity of capturing logic relations , which is necessary for logical reasoning . However , logical reasoning can only be implemented on the basis of clear , accurate , and generalized knowledge . Therefore , we leverage fact as the conceptual knowledge unit to serve the basis for logic relation extraction . Fact is organized as a triplet , i.e. , in the form of predicate-argument structures , to represent the meaning such as `` who-did-what-to-whom '' and `` who-is-what '' . Compared with existing studies that inject complex knowledge like knowledge graphs , the knowledge structure based on fact is far less complicated and more general in representing events and relations in languages . On top of the fact-based knowledge structure , we present PROPHET , a logic-aware pre-trained language model to learn the logic-aware relations in a universal way from very large texts . In detail , we introduce three novel pre-training objectives based on the newly introduced knowledge structure basis fact : 1 ) logical connectives masking for learning sentence-level logic connection . 2 ) logical structure completion task on top of facts for regularization , aligning extracted fact with the original context . 3 ) logical path prediction to capture the logic relationship between facts . PROPHET is evaluated on a broad range of language understanding tasks : natural language inference , semantic similarity , machine reading comprehension , etc . Experimental results show that the fact is useful as the carrier for knowledge modeling , and the newly introduced pre-training tasks can improve PROPHET and achieves significant performance on downstream tasks.1 2 RELATED WORK . 2.1 PRE-TRAINED LANGUAGE MODELS IN NLP . Large pre-trained language models ( Devlin et al. , 2018 ; Liu et al. , 2019 ; Radford et al. , 2018 ) have brought dramatic empirical improvements on almost every NLP task in the past few years . A classical norm of pre-training is to train neural models on a large corpus with self-supervised pre-training objectives . `` Self-supervised '' means that the supervision provided in the training process is automatically generated from the raw text instead of manually generation . Designing effective criteria for language modeling is one of the major topics in training pre-trained models , which decides how the model captures the knowledge from large-scale unlabeled data . The most popular pre-training objective used today is masked language modeling ( MLM ) , initially used in BERT ( Devlin et al. , 2018 ) , which randomly masks out tokens , and the model is asked to uncover it given surrounding context . Recent studies have investigated diverse variants of denoising strategies ( Raffel et al. , 2020 ; Lewis et al. , 2020 ) , model architecture ( Yang et al. , 2019 ) , and auxiliary objectives ( Lan et al. , 2019 ; Joshi et al. , 2020 ) to improve the model strength during pre-training . Although the existing techniques have shown effectiveness in capturing syntactic and semantic information after large-scale pre-training , they perform sensitivity to role reversal and struggles with pragmatic inference and role-based event knowledge ( Rogers et al. , 2020 ) , which are critical to the ultimate goal of complex reasoning that requires to uncover logical structures . However , it is difficult for pre-trained language models to capture the logical structure inherent in the texts since logical supervision is rarely available during pre-training . Therefore , we are motivated to explicitly guide the model to capture such clues via our newly introduced self-supervised tasks . 2.2 REASONING ABILITY FOR PRE-TRAINED LANGUAGE MODELS . There is a lot of work in the research line of enhancing reasoning abilities in pre-trained language models via injecting knowledge . The existing approaches mainly design novel pre-training objectives and leverage abundant knowledge sources such as WordNet ( Miller , 1995 ) . Notably , ERNIE 3.0 ( Sun et al. , 2021 ) uses a broad range of pre-training objectives from word-aware , structure-aware to knowledge-aware tasks , based on a 4TB corpus consisting of plain texts and a large-scale knowledge graph . WKLM ( Xiong et al. , 2019 ) replaces entity mentions in the document with other entities of the same type , and the objective is to distinguish the replaced entity from the original ones . KEPLER ( Wang et al. , 2021b ) encodes textual entity descriptions using embeddings from a PrLM to take full advantage of the abundant textual information . K-Adapter ( Wang et al. , 2020 ) designs neural adapters to distinguish the type of knowledge sources to capture various knowledge . Our proposed method differs from previous studies in three aspects . Firstly , our model does not require any external knowledge resources like previous methods that use WordNet , WikiData , etc . We only use small-scale textual sources following the standard PrLMs like BERT ( Devlin et al. , 2018 ) , along with an off-the-shelf dependency parser to extract facts . Secondly , previous works only consider triplet-level pre-training objectives . We proposed a multi-granularity pre-training strategy , considering not only triplet-level information but also sentence-level and global knowledge to enhance logic reasoning . Finally , we propose a new training mechanism apart from masked language modeling ( MLM ) , hoping to shed light on more logic pre-training strategies in this research line . 1Our codes have been uploaded as supplemental material , which will be open after the double review period . 3 PRELIMINARIES . In this section , we will introduce the concept of fact and logical graph , which is the basis of PROPHET . We will also describe extracting the fact for logical graph construction , as an example shown in Figure 1 . 3.1 FACT . Following Nakashole & Mitchell ( 2014 ) and Ouyang et al . ( 2021 ) , we extract facts which are triplets represented as T = { A1 , P , A2 } , where A1 and A2 are the arguments and P is the predicate between them . It can well represent a broad range of facts , reflecting the notion of `` who-did-what-to-whom '' and `` who-is-what '' , etc . We extract such facts in a syntactic way , which makes our approach generic and easy to apply . Given a document , we first split the document into multiple sentences . For each sentence , we conduct dependency parsing using StanfordCoreNLP ( Manning et al. , 2014 ) .2 For the analyzed dependencies , basically , we consider verb phrases and some prepositions in the sentences as `` predicates '' , and then we search for their corresponding actors and actees as the `` arguments '' . 3.2 LOGICAL GRAPH . A logical graph is an undirected ( but is not required to be connected ) graph that represents logical dependency relation between components in facts . In logical graphs , nodes represent argument/predicates in the fact , and edges indicate whether two nodes have relations in a fact . Such a structure can well unveil and organize semantic information captured by facts . Besides , a logical graph supports considerations among long-range dependencies via connecting arguments and their relations in different facts across different spans . We further show how to construct such graphs based on facts . Despite given relations in facts , we design another two types of edges based on identical mentions and coreference information . ( 1 ) There can be identical mentions in different sentences , resulting in repeated nodes in facts . We connect nodes corresponding to the same non-pronoun arguments by edges with edge type same . ( 2 ) We conduct coreference resolution on context using an off-to-shelf model to identify arguments in facts that refer to the same one.3 We add edges with type coref between them . The final logical graph is denoted as S = ( V , E ) , where V = Ai ∪ P and i ∈ { 1 , 2 } . 2https : //stanfordnlp.github.io/CoreNLP/ , we also tried to use OpenIE directly ; however , the performance is not satisfactory . 3https : //github.com/huggingface/neuralcoref . 4 PROPHET . 4.1 MODEL ARCHITECTURE . We follow BERT ( Devlin et al. , 2018 ) and use a multi-layer bidirectional Transformer ( Vaswani et al. , 2017 ) as the model architecture of PROPHET . For keeping the focus on the newly introduced techniques , we will not review the ubiquitous Transformer architecture in detail . We develop PROPHET by using exactly the same model architecture as BERT-base , where the model consists of 12 transformer layers , with 768 hidden size , 12 attention heads , and 110M model parameters in total . 4.2 LOGIC-AWARE PRE-TRAINING TASKS . We describe three pre-training tasks used for pre-training PROPHET in this section . Figure 2 is an illustration of PROPHET pre-training . The first task is logical connectives masking ( LCM ) generalized from masked language modeling ( Devlin et al. , 2018 ) for logical connectives to learn sentence-level representation . The second task is logical structure completion ( LSC ) for learning logic relationship inside a fact , where we first randomly mask arguments in facts , and then predict those items . Finally , a logical path prediction ( LPP ) task is proposed for recognizing the logical relations of randomly selected node pairs . Logical Connective Masking Logical connective masking is an extension of the masked language modeling ( MLM ) pre-training objective in Devlin et al . ( 2018 ) , with a particular focus on connective indication tokens . We use the Penn Discourse TreeBank 2.0 ( PDTB ) ( Prasad et al. , 2008 ) to draw the logical relations among sentences . Specifically , PDTB 2.0 contains relations that are manually annotated on the 1 million Wall Street Journal ( WSJ ) corpus and are broadly characterized into `` Explicit '' and `` Implicit '' connectives . We use the `` Explicit '' type ( in total 100 such connectives ) , which apparently presents in sentences such as discourse adverbial `` instead '' or subordinating conjunction `` because '' . Taking all the identified connectives and some randomly sampled other tokens ( for a total 15 % of the tokens of the original context ) , we replace them with a [ MASK ] token 80 % of the time , with a random token 10 % of the time and leave them unchanged 10 % of the time . The MLM objective is to predict the original tokens of these sampled tokens , which has proven effective in previous works ( Devlin et al. , 2018 ; Liu et al. , 2019 ) . In this way , the model learns to recover the logical relations for two given sentences , which helps language understanding . The objective of this task is denoted as Lconn . Logical Structure Completion To align representation between the context and the extracted fact , we introduce a pre-training task of logical structure completion . The motivation here is to encourage the model to learn the structure-aware representation that encodes the `` Who-did-What-to-Whom '' like meanings for better language understanding . To speak in detail , we randomly select a specific proportion λ of the total facts ( λ = 20 % in this work ) , from a given context . For each chosen fact , we either ask the model to complete `` Argument-Predicate- ? '' or `` Argument- ? -Argument '' ( the templates are selected based on equal probability ) . We denote all the blanks that need to be completed as ma and mp , denoting arguments and predicates , respectively . In our implementation , this objective is the same as masked language modeling to keep simplicity , by using the original loss following Devlin et al . ( 2018 ) . Lalign = − ∑ i∈a∪p logD ( xi|ma , mp ) , ( 1 ) where D is the discriminator to predicts a token from a large vocabulary . Logical Path Prediction To learn representation from the constructed logical graph , thus endowing the model with global logical reasoning ability , we propose the pre-training task of predicting whether there exists a path between two selected nodes in the logical graph . In this way , the model learns to look at logical relations across a long distance of arguments and predicates in different facts . We randomly sample 20 % nodes from logical graph to form set V ′ , there are in total C2|v′| node pairs . We have a maximum number maxp of node pairs to predict . To avoid bias in the training process , we try to make sure that maxp2 are positive samples and the rest are negative samples , thus balancing positive-negative ratios . If the number of positive/negative samples is less than maxp2 , we just keep the original pairs . Formally , the pre-training objective of this task is calculated as below following Guo et al . ( 2020 ) : LPath = − ∑ vi , j∈V ′ [ δ log σ [ vi , vj ] + ( 1− δ ) log ( 1− σ [ vi , vj ] ) ] , ( 2 ) where δ is 1 when vi and vj have a connected path and 0 otherwise . [ vi , vj ] denotes the concatenation of representations of vi and vj . The final training objective is the weighted sum of the above mentioned three losses . L = Lconn + Lalign + LPath . ( 3 )

Summary: The paper proposes a new pre-training technique to induce a logical prior in the language model representation. Concretely, they propose pre-training on facts, represented as knowledge base triples (source, sink, relation) (knowledge-base completion) and link prediction, alongside traditional masked language modeling objective. Their proposed method achieves some improvement over downstream tasks, including a subset of GLUE benchmark and a couple of relation prediction datasets.