Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning

1 . Introduction . In recent years , language models have achieved impressive results on a variety of natural language processing ( NLP ) tasks , such as recognizing textual entailment , machine reading comprehension , and machine translation . Most of these language models are based on variants of the transformer architecture [ Vaswani et al. , 2017 ] , for example BERT [ Devlin et al. , 2019 ] , T5 [ Raffel et al. , 2020a ] , and GPT-3 [ Brown et al. , 2020 ] . These models depend entirely on the attention mechanism , and thus eliminate the need for recurrent computations used by LSTMs [ Hochreiter and Schmidhuber , 1997 ] and GRUs [ Cho et al. , 2014 ] . They can easily learn long-range dependencies , and the computation can be parallelized efficiently . Today ’ s models contain millions or even billions of parameters . The models are pre-trained on large unlabeled corpora , and then later fine-tuned to tackle a specific NLP task . For example , the pre-trained BERT can reply to questions such as the following : Context : The iPhone is produced by [ MASK ] . Expected answer : Apple Model answer : Apple However , this performance is deceiving : If we introduce a trap word , the pre-trained BERT model replies completely differently : Context : Samsung . The iPhone is produced by [ MASK ] . Expected answer : Apple Model answer : Samsung Here , the BERT model got distracted by the additional word ( a technique called mispriming [ Kassner and Schütze , 2020 ] ) . Thus , the question arises to what degree such models really “ understand ” the natural language text , and to what degree they merely respond to statistical cues . This question is of utmost importance , because if we start relying on such language models , there is the danger that we obtain good responses only in common test settings , and completely abstruse replies in less common settings . In this survey paper , we shed light on this question by investigating some of the most complex natural language tasks : those that involve reasoning . That is , we look at test data sets that have explicitly been designed to test the limitations of transformer-based models , and we investigate to what degree the models really “ understand ” these tasks . While several survey papers have focused on transformer-based models and on their applications [ Rogers et al. , 2020b , Qiu et al. , 2020 , Xia et al. , 2020 , Yates et al. , 2021 ] , the capabilities of transformer-based models in reasoning tasks have so far not been surveyed . Our paper is organized as follows . In Section 2 , we describe some common pitfalls that all models need to handle in order to reason on natural language text . Section 3 analyzes the performance of transformer-based models on different reasoning tasks . In Section 4 , we describe the theoretical limitations of the transformer architecture , and show , in Sections 4.2-4.3 , that they impact natural language reasoning . We conclude in Section 5 . The appendix contains a detailed list of basic models ( Appendix A ) and challenging datasets ( Appendix B ) , as well as of the model performances ( Appendix C ) . 2 . Common Pitfalls for BERT-like Models . We discuss here some common pitfalls that any approach needs to handle in order to reason on natural language . Our discussion focuses on BERT , but the phenomena may affect other transformer-based models as well . 2.1 Negation . The pre-trained BERT model can not differentiate between positive and negative statements . As an example , take this sentence from the LAnguage Model Analysis ( LAMA ) dataset [ Petroni et al. , 2019 ] , where BERT performs well : Context : Marcel Oopa died in the city of [ MASK ] . Expected answer : Paris Model answer : Paris ( -2.3 ) , Lausanne ( -3.3 ) , Brussels ( -3.3 ) When Kassner and Schütze [ 2020 ] added the word “ not ” , BERT delivered the exact same top-ranked result : Context : Marcel Oopa did not die in the city of [ MASK ] . Expected answer : any city different from Paris Model answer : Paris ( -2.4 ) , Helsinki ( -3.5 ) , Warsaw ( -3.5 ) This phenomenon was also confirmed by Ettinger [ 2020 ] . Kassner and Schütze [ 2020 ] show that BERT can be fine-tuned to pay attention to the negation . Thus , it is essential to add examples with negation to the training set . Niven and Kao [ 2019 ] point out that these examples should be diverse enough to not rely only on the word “ not ” , and Hosseini et al . [ 2021 ] propose an unlikelihood objective function to learn to differentiate between positive and negative statements . 2.2 Mispriming . The ability to distinguish useful from distracting contexts is an essential building block for any reasoning task . We have already seen an example of mispriming in the introduction . Mispriming can , in principle , affect any task , and thus also reasoning in particular . Interestingly , mispriming works only when the distracting word is of the same type as the expected answer ( companies , in our example ) . The pre-trained BERT is not easily misled by primes of other types [ Niven and Kao , 2019 ] . Misra et al . [ 2020 ] also show that the problem of mispriming can be overcome by providing more context . In the following sentence , for example , the mispriming fails : Context : Samsung . The iPhone was produced by [ MASK ] , whose CEO was Steve Jobs Expected answer : Apple Model answer : Apple This shows that , although there is some dependency on misprimes , their power decreases when sentences provide more context . 2.3 Pattern Heuristics . Fine-tuned BERT models have a tendency to learn simple pattern-based heuristics . For example , BERT can be trained to perform well on textual entailment in the MNLI dataset [ Williams et al. , 2018 ] : Premise : The actor and the professor mentioned the lawyer . Hypothesis : The professor mentioned the lawyer . Expected answer : Entailment Model answer : Entailment To better understand the performance of BERT , McCoy et al . [ 2019b ] designed the HANS ( Heuristic Analysis for NLI Systems ) dataset . HANS makes BERT fail as follows : Premise : The doctors advised the presidents and the tourists . Hypothesis : The presidents advised the tourists . Expected answer : Non entailment Model answer : Entailment This shows that the model learned the “ lexical overlap heuristic ” , which assumes that a premise entails all hypotheses constructed from words in the premise . This problem can be addressed by adding more HANS-like examples to the training dataset . 2.4 Word Order . Different studies [ Ettinger , 2020 , Sankar et al. , 2019 , Pham et al. , 2020 , Gupta et al. , 2021 ] have shown that BERT-like models are unperturbed by grammatically incorrect sentences : If presented with a sentence of randomly shuffled words , they will still reply correctly . This insensitivity to order can also mislead textual entailment . For example , the pre-trained BERT fine-tuned on the MNLI dataset fails to provide the correct answer in the following case [ McCoy et al. , 2019b , a ] : Premise : The doctor visited the lawyer Hypothesis : The lawyer visited the doctor Expected answer : Non entailment Model answer : Entailment This issue can be solved by augmenting the training set with modified word order instances with their respective labels or by fine-tuning the model on sensitive word ordering tasks such as CoLA [ Warstadt et al. , 2019 ] . 3 . Types of Reasoning with Transformer-based Models . 3.1 Horn Rule Reasoning . A rather simple way of logical reasoning is to infer a conclusion from a set of premises and rules . Transformer-based models are able to perform such kind of reasoning [ Clark et al. , 2020 , Talmor et al. , 2020 , Betz et al. , 2020 ] without any external knowledge , if both the rules and the facts are mentioned explicitly in the text . They can even generate the proofs [ Saha et al. , 2020 , Tafjord et al. , 2020 , Gontier et al. , 2020 ] . Here is an example from the ParaRules dataset [ Clark et al. , 2020 ] : Context : Fact 1 : Erin is young . Fact 2 : Erin is not kind . Fact 3 : Peter is nice . Rule 1 : If someone is young and not kind then they are big . Question : Is Erin big ? Expected answer : Conclusion : Erin is big . Proof : ( Fact 1 & Fact 2 ) → Rule 1 → Conclusion In this task , the best model , a fine-tuned T5-11B , achieves an accuracy above 95 % in proof generation and question answering . A transformer-based model can thus solve the task nearly perfectly . 3.2 Commonsense Reasoning . Commonsense reasoning is any reasoning task that requires background knowledge that humans commonly have . For example , the instruction “ Can you do a Napoleon for the camera ? ” requires commonsense reasoning to realize that the word “ Napoleon ” expresses a specific pose [ Bender and Koller , 2020 ] . Several studies have shown that BERT learned a certain amount of commonsense knowledge during pre-training [ Petroni et al. , 2019 , Davison et al. , 2019 , Bosselut et al. , 2019 , Zhou et al. , 2020b , Cui et al. , 2020 ] . Consider , for example , the LAMA dataset [ Petroni et al. , 2019 ] , which asks : Context : Ravens can [ MASK ] Expected answer : fly Model answer : fly The model ( the pre-trained BERT-large ) is able to recall such commonsense knowledge . This good performance has prompted the research community to develop datasets that specifically probe the commonsense reasoning of transformer models . Prominent datasets are COSMOS QA [ Huang et al. , 2019 ] , CommonsenseQA [ Talmor et al. , 2019 ] , the Winograd Schema Challenge [ Levesque et al. , 2012 ] , SWAG [ Zellers et al. , 2018 ] , ReCoRD [ Zhang et al. , 2018 ] , CODAH [ Chen et al. , 2019 ] , and PIQA [ Bisk et al. , 2020 ] . Transformer-based models can indeed achieve a high performance ( often > 75 % ) on these datasets , but only with additional methods . These include data augmentation techniques [ Yang et al. , 2020 ] , multi-task learning [ Lourie et al. , 2021 ] , and fusing knowledge graphs into language models [ Xu et al. , 2021 ] . The following is an example from the CommonsenseQA dataset [ Talmor et al. , 2019 ] : Question : Bats have many quirks , with the exception of ? Expected Answer : Laying eggs Model w/o knowledge graph fusing : Eating bugs Model w/ knowledge graph fusing : Laying eggs The above example shows that providing the model with information from a knowledge graph helps the model to correctly answer the question . However , several studies [ Forbes et al. , 2019 , Zhou et al. , 2020b , Lin et al. , 2020 , Boratko et al. , 2020 , Singh et al. , 2021 ] show that when the datasets are specifically changed to target the weaknesses of transformerbased models ( for example , by adversarial instances ) , the models fail . Here is an example from the COM2SENSE dataset [ Singh et al. , 2021 ] , which asks the model to judge whether a given sentence is logically coherent or not : Context : Expecting ten fish in the net , Sammy was thrilled to see forty fish swimming in there . Expected answer : Coherent Model answer : Coherent The authors created a counterpart to this question by modifying a few words : Context : Expecting ten fish in the net , Sammy was thrilled to see five fish swimming in there . Expected answer : Incoherent Model answer : Coherent When the model ( UnifiedQA-3B [ Khashabi et al. , 2020 ] , a multi-task trained model ) is tricked this way , it fails to predict correctly . This shows that the model can fall prey to relatively simple modifications , and does not really reason .

Summary: Authors survey the literature on probing the understanding, reasoning, and commonsense properties of transformer-based models, outlining several limitations of current models. They cover mispriming, negations, the tendency of such models to learn simple heuristics, under-sensitivity to word order, reasoning with patterns resembling Horn clauses, commonsense reasoning, reasoning with temporal knowledge. Furthermore, they also cover several natural language understanding tasks such as natural language inference and mathematical reasoning.