Does Entity Abstraction Help Generative Transformers Reason?

1 INTRODUCTION . Transformer language models ( TLMs ; Vaswani et al . 2017 ) have enabled rapid progress in natural language processing ( NLP ) . When pre-trained on large corpora ( such as the web ) to predict the next tokens or a set of masked tokens from an input sequence , TLMs can capture linguistic knowledge ( Peters et al. , 2018 ; Goldberg , 2019 ; Tenney et al. , 2019b ) , and yield state-of-the-art performance on many NLP tasks with little to no task supervision ( Devlin et al. , 2019 ; Radford et al. , 2018 ; 2019 ; Brown et al. , 2020 ) . However , it is not clear if these models can capture higher level knowledge such as reasoning skills that can be re-used in arbitrary contexts , and in ways that leverage the compositionality of those skills ( Lake & Baroni , 2018 ; Liška et al. , 2018 ) , something logical reasoners can do relatively well on a smaller scale ( De Raedt et al. , 2007 ; Fierens et al. , 2015 ) . Simple compositional tasks such as SCAN ( Lake & Baroni , 2018 ) , CLUTRR ( Sinha et al. , 2019 ) , and ProofWritter ( Clark et al. , 2020 ; Tafjord et al. , 2020 ) can help diagnose the compositional generalization behavior of language models . Recent work on some of these datasets showed that TLMs still struggle to learn reasoning strategies that can be re-used in out-of training distribution settings ( Lake & Baroni , 2018 ; Gontier et al. , 2020 ) . If we look at how logical reasoners operate , we find that they have an important abstraction component ( going from grounded entities to higher level concepts ) before logical reasoning can start ( De Raedt et al. , 2007 ) . Going from an original text sequence to its higher-order meaning is an important part of the NLP pipeline ( part of it being entity type tagging ) . Similarly in mathematics , the introduction of generic variables allows to progress in a logical reasoning process without keeping track of every ( grounded ) atomic entity . Overall , this idea that we call abstraction , seems to be an important part of logical reasoning . Recent work suggests that incorporating external knowledge about grounded entities could improve language models ’ abilities to reason and generalize ( Ji et al. , 2017 ; Zhang et al. , 2019 ; Moosavi et al. , 2020 ; Rosset et al. , 2020 ) . However the empirical effect of incorporating generic entity types remains unclear , especially with recent studies suggesting that pre-trained models already encode some of that linguistic knowledge in their parameters ( Hewitt & Manning , 2019 ; Liu et al. , 2019a ; Clark et al. , 2019 ; Tenney et al. , 2019a ) . In this work , we study the effect of explicitly providing entity type abstraction in addition to the original input to pre-trained Transformers . We explore and evaluate different ways to incorporate entity type abstraction and observe that some methods are more efficient than others . To constructs the abstract representation incorporated into TLMs , we leverage symbolic NLP representations such as entity type information given by popular NLP libraries . This allows for automatic and reproducible data processing . In general , our approach is the following : given an input sequence , we build a simplified version of it by replacing entity names by their corresponding entity types . This simplified sequence can then be used as extra input ( Sections 3.1 & 3.2 ) or as extra training signal to the model ( Sections 3.3 ) . In particular , we explore three different ways to augment pre-trained Transformers with this abstract knowledge : 1. by combining token embeddings from both the original and the abstract sequence before encoding ( Section 3.1 ) ( Figure 1a & 1b ) . 2. by encoding both the original and the abstract sequence and combining them before decoding the target output ( Section 3.2 ) ( Figure 1c & 1d ) . 3. by adding a second language model head on top of the Transformer decoder to predict the abstract sequence ( Section 3.3 ) ( Figure 1e ) . A series of controlled experiments on two synthetic datasets show that models having access to abstract knowledge about entity types yield better performance at inference time both when interpolating and extrapolating to unseen lengths of reasoning chains . Synthetic data is used in order to control for the degree of compositional generalization required . Furthermore , in order to understand if the benefits observed in the synthetic cases are applicable in more realistic settings , we ran a series of experiments on two question answering datasets requiring various degrees of multi-hop reasoning . Unfortunately , results on these more natural language datasets show that abstraction aware models are not significantly better than baseline models . Overall our work contributions are the following : 1. we introduce and compare empirically different ways to incorporate abstraction into pre-trained TLMs . 2. we show that incorporating abstract knowledge can significantly improve reasoning and compositional generalization in both interpolation and extrapolation when the environment is formally defined in a logical reasoning setting . 3. we show that abstraction aware models may not benefit much when language is more natural and less procedural . We hope that our work will inspire future research in the field to look for simple inductive biases that can complement pre-trained models in their quest to achieve logical reasoning at scale . 2 RELATED WORK . Augmenting neural language models with knowledge about entities has been a popular method to improve their functionality . Ji et al . ( 2017 ) trained an entity neural language model to predict sequences of entities with an LSTM ( Hochreiter & Schmidhuber , 1997 ) . At each sampling step , they predict the next word alongside a categorical variable indicating the current token ’ s entity ID . They obtained lower perplexity and better results on co-reference resolution and entity prediction tasks than a variety of baselines . Similarly , Rosset et al . ( 2020 ) trained a GPT2 model ( Radford et al. , 2019 ) by giving it access to entity knowledge at the input level and as an additional pre-training loss . Their model achieved better factual correctness on benchmarks such as LAMA ( Petroni et al. , 2019 ) , and performed better than a baseline GPT2 model in various question answering tasks . Inspired by this work and motivated by the goal of building better reasoning language models , we instead focus on the prediction of entity types rather than entity identifiers taken from a fixed list of entities . This allows our solution to be robust to new entities . In addition , we explore and compare different ways to incorporate the entity knowledge in an encoder-decoder architecture . Besides entity knowledge , other types of explicit information has also been given to Transformer models . Levine et al . ( 2019 ) trained a BERT-like model to learn word senses . They gave their model access to WordNet supersenses at the input level and as an additional training loss . They achieve better performance than other baselines on the SemEval Word Sense Disambiguation task ( Raganato et al. , 2017 ) . In addition , Moosavi et al . ( 2020 ) propose to improve robustness to data biases by augmenting the training data with predicate-argument structures . They train a BERT-base model ( Devlin et al. , 2019 ) with PropBank-style semantic role labeling ( Shi & Lin , 2019 ) on MultiNLI ( Williams et al. , 2017 ) and SWAG ( Zellers et al. , 2018 ) datasets . Their results show that incorporating predicate-argument structure in the input sequence ( only during training ) makes the model more robust to adversarial examples in MultiNLI . Furthermore , Sachan et al . ( 2020 ) ask if syntax trees can help Transformers to better extract information . They augmented a pre-trained BERT model with a syntax graph neural network in order to encode syntaxt trees and measure the performance of their model against a BERT model on various tasks . Their results showed that the quality of the trees are highly tied to the performance boost observed . More recently , Porada et al . ( 2021 ) extended a RoBERTa model ( Liu et al. , 2019b ) with hypernym abstraction based on WordNet to evaluate the plausibility of events . Their model is able to better predict human plausibility judgement than other RoBERTa baselines . Although different in application , all these prior works leverage the general idea of explicitly giving more abstract knowledge to Transformer models , hence showing how flexible and generic this strategy can be . We take a similar approach with entity types , but in the hope of improving the reasoning skills of our baseline model . A flurry of recent work has also examined ways to augment TLMs with entities from external knowledge bases ( Zhang et al. , 2019 ; Peters et al. , 2019 ; Févry et al. , 2020 ; Verga et al. , 2020 ) . However , most of the time , these solutions rely on external components such as knowledge graphs with pre-trained entity embeddings , and/or an additional memory . While they often use entity linking as a way to perform co-reference resolution , they do not incorporate higher level of abstractions such as entity types like we do here . 3 INTRODUCING ABSTRACTION INDUCTIVE BIASES . In this section we describe five different ways to incorporate abstraction into a pre-trained encoderdecoder model . Given an input sequence X , we use existing NLP tools such as spacy named entity tagger1 to make a simplified copy Xs of the input . This is a more generic copy of X . We run the spacy recognizer on X to extract entity tags such as PERSON , ORG , GPE , etc ... For each entity type , we create n additional vocabulary entries ( with randomly initialized embeddings ) such as [ PERSON 1 , . . . , PERSON n , ORG 1 , . . . , ORG n , . . . ] . Every token in X is then replaced by their ( randomly numbered ) entity tag to make the simplified sequence Xs . If the same entity is present multiple times in X , each occurrence will be replaced by the same entity tag in Xs . If no entity is found for a token in X , the original token ’ s text is kept in Xs ( e.g . “ Bob Smith has a cat that he loves . Bob also loves Alexandra. ” would be transformed into “ PERSON 11 has a cat that he loves . PERSON 11 also loves PERSON 23. ” ) . The hyper-parameter n is set such that each distinct entity within the same sequence gets a different ID . If the same entity appears more than once in a single example , it will get the same ID every time within that sequence . However , we re-use the same set of entity tags across different examples . The value of n depends on each dataset , details can be seen in Appendix A . Although this may look similar to attributing different entity IDs to each entity regardless of their type , we believe that after seeing many examples , all IDs of the same type will have a similar embedding since we randomly select such ID across examples and across epochs . The number of overlapping entity tags across the entire dataset during many epochs will be large , resulting in pushing the embedding of same-type entity tokens closer together . To test the effect of having multiple tags for the same entity type , we also ran experiments in which we set n = 1 thus forcing overlapping entity tags within each example . However we didn ’ t notice any concluding difference , so the rest of this work will focus on the n > 1 setting described above . 1https : //spacy.io/models/en In the following subsections , we describe different strategies to incorporate Xs into an encoderdecoder Transformer model .

Summary: The paper investigates the effect of incorporating entity type abstraction into pre-trained Transformers. To achieve that, the authors have tried five different architectures to build the abstraction aware model. The proposed model is tested on three NLP datasets for reasoning. Empirical results show that entity type abstraction is beneficial in formally defined logical reasoning environments with simple language. While for QA datasets with more natural language, the baseline is already very strong and the improvement of incorporating abstraction is minor.