Think Big, Teach Small: Do Language Models Distil Occam’s Razor?

1 Introduction . An effective and reliable interaction between humans and machines depends on a proper understanding of their priors . The alignment between these priors , in the form of knowledge and other predispositions used as bias , is especially critical for communication and learning ( Sun et al. , 2020 ) . For instance , after a human utters the prompt “ on off on off on ” , another human would expect “ off ” to be a more likely continuation than “ on ” . In contrast , after the prompt “ twinkle twinkle little ” we would expect “ star ” to be more likely . The former depends on an abstract pattern intrinsic to the sequence ( a simple algorithmic sequence alternating “ on ” and “ off ” ) , while the latter depends on some language pattern extrinsic to the sequence ( common use in the English language because of a popular song ) . Language models are a kind of AI systems extrapolating sequences of symbols , and are very good at exploiting the extrinsic patterns that are borrowed from humans after training over massive natural language corpora . However , it is only recently , with systems such as BERT ( Devlin et al. , 2018 ) , GPT-2 ( Radford et al. , 2019 ) and GPT-3 ( Brown et al. , 2020 ) , that we see how these models can also discover some intrinsic patterns provided in the prompts . This kind of inference is called ‘ few-shot learning ’ , even if there is no further training or fine-tuning of the model . Many of these language models are based on transformers and other attention-based architectures ( Vaswani et al. , 2017 ) , which do not feature recursion or loops , unlike some approaches based on recurrent neural networks . Some particularly well-chosen and optimised prompts have shown remarkable extrapolations ( Xu et al. , 2020 ; Izacard and Grave , 2020 ; Hendrycks et al. , 2021 ) , illustrating this new kind of abstraction on the go . However , it is unclear how elaborate or algorithmic these abstractions can be . More studies are needed to determine the optimal prompts and example order that make the model capture 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . a range of different patterns ( Zhao et al. , 2021 ; Lu et al. , 2021 ; Liu et al. , 2021 ; Webson and Pavlick , 2021 ; Kirstain et al. , 2021 ) . Even when the prompt template is fixed , it is an open question what the minimal set of examples should be for learning a concept , how this minimality is evaluated and what priors pretrained models incarnate for few-shot learning to work . Precisely , machine teaching is the area of artificial intelligence that looks for the optimal examples or curriculum that a teacher should use to make a learner capture a concept ( Zhu , 2015 ) . This is of enormous importance today in applications such as digital assistants , where we would like to teach them new concepts or routines with a few examples , in the same way humans teach or communicate with other humans ( Cakmak and Thomaz , 2014 ; Goodman and Frank , 2016 ; Degen et al. , 2020 ) . It is also fundamental for explainable AI , as the dual problem appears : how a machine can teach ( explain ) a concept to a human in an efficient way ( Khan et al. , 2011 ; Basu and Christensen , 2013 ; Hernández-Orallo and Ferri , 2021 ) . When dealing with rich representation languages ( e.g. , regular languages or Turing-complete languages ) , closer or similar to human language in expressive power , it has recently been shown that strong simplicity priors can make effective teaching possible with very few examples : the overall size of the example set is usually shorter than the algorithmic expression of the concept itself ( Telle et al. , 2019 ) . For instance , teaching the concept of what an odd number is becomes shorter by using a few examples than by giving the full definition of what an odd number is . Language models and machine teaching can now meet because of this new few-shot learning ability of language models . We can explore the question of how many examples are needed ( or how long they have to be ) to teach something to a language model . These examples and their template are seen as teaching prompts : we want the language model to continue the sequence as if it had captured the pattern that has been expressed in the prompt . From the machine teaching perspective , language models are general systems , as they have not been trained for simplicity or for any particular domain , other than a partial sample of human knowledge expressed as natural language . With their conjunction , we can now ask key questions for AI such as the relevance of simplicity or acquired priors to make learning from a few examples possible ( Chater and Vitányi , 2003 ) . In this paper , we examine how a machine teaching setting inspired by Occam ’ s razor can help us understand how language models ‘ learn ’ and choose prompts more efficiently . We use the term ‘ prior distillation ’ , overloading the notion of knowledge distillation within the field of machine learning , as we analyse whether a simplicity prior is acquired as the result of training large language models such as GPT-2 or GPT-3 . To complete the picture , we also compare them with some other AI systems ( e.g. , program induction systems ) and humans on the same task , which are known to have this prior . The key contributions of this paper are : • We analyse language models in the context of machine teaching , using minimal teaching sets for few-shot inference . • We compare the acquired priors of language models with humans and program induction systems ( Gulwani et al. , 2015 ) , when facing patterns preferred by a simplicity prior . • We find that language models such as GPT-3 have similar or better performance than humans . The performance negatively correlates with the complexity of the concept . • Language models and humans differ in their capability for producing explanations . The length of human explanations correlates with the complexity of the concept . Having a better understanding of language models and its relation to the way other systems , including humans , make inferences is fundamental research that can make the use of these systems safer and more effective , through more robust prompts to solve a variety of tasks . However , this better understanding can also be used maliciously to cunningly modify prompts that alter the results of some applications built on top of language models . A query may fail when using dates , numbers or other strings that have intrinsic patterns . For instance , GPT-3 usually continues “ Mary ’ s father has 5 daughters , Lucy , Anne , Alice , Jane and ” correctly , but fails on “ Mary ’ s father has 5 daughters , Ba , Baba , Bababa , Babababa and ” , systematically continued with ‘ Bababababa ’ . The rest of the paper follows with related work in language models , the comparison of machine and human learning , as well as Occam ’ s razor and machine teaching . Section 3 contains the experimental design , and introduces the systems we consider for comparison . Section 4 presents and analyses the results . Finally , we close the paper with a discussion of the implications of this work . The supplementary material includes additional experiments , plots , tables and specific analyses . 2 Related Work . It is only recently that deep language models are starting to show some minimal capabilities for few-shot learning . Systems such as GPT-3 solve “ tasks which require to perform simple on-the-fly computational reasoning , recognize a novel pattern that is unlikely to have occurred in training , or adapt quickly to an unusual task ” ( Brown et al. , 2020 ) . Examples of these tasks are simple arithmetic operations and the induction of symbolic manipulations such as anagrams or reversed words . Hendrycks et al . ( 2021 ) compare the performance of a text model with humans on 57 tasks including elementary mathematics , US history , law , and more . Sinha et al . ( 2019 ) show that language models obtain poor robustness and generalisation capabilities when compared to graph neural networks that work with symbolic inputs . The experiments also show that humans ( with limited time ) performed well in simple problems but the performance decreases with complex examples . There are also important limitations of language models on simple generalisation tasks ( Bender and Koller , 2020 ; Marcus and Davis , 2020 ) . On the one hand , recurrent neural networks , such as long-short-term-memory ( LSTM ) architectures , have shown the capacity of capturing syntax-sensitive generalisations such as long distance number agreement ( Lakretz et al. , 2019 ) , or language modelling for the game of chess ( Toshniwal et al. , 2021 ) . The abstract reasoning capacity of transformers and other deep learning architectures has been studied by Hupkes et al . ( 2020 ) . In sum , as language models based on transformers are not recursive , it is an open debate whether they are just memorising a huge corpus , or they really have the capacity of learning abstract concepts from data . The comparison between humans and AI systems has usually been performed with systems that are trained for a task , but not for a variety of tasks that are introduced with a few words or examples . For instance , Kühl et al . ( 2020 ) present an experiment that compares the performance of several machine learning algorithms and 44 human participants recognising patterns in 3x3 matrices . The results in this case show that humans , as expected , tend to achieve better performance than the AI systems when few examples are provided . This work suggests that many AI systems may not be competitive with humans when limited examples are given for a range of tasks . Further insight can be obtained when we try to look into the extracted models through explanations , and analyse them in the context of concept complexity . For instance , the relation between complexity of concepts and explanations is discussed by Ai et al . ( 2021 ) . 2.1 Occam ’ s Razor . Occam ’ s razor , the a priori preference for simple hypotheses over more complex ones , has pervaded philosophy of science , inductive inference , and human communication , and it is ubiquitous in statistics and machine learning , explicitly —through criteria such as the MML/MDL principles ( Wallace and Boulton , 1968 ; Rissanen , 1983 ) and the use of regularisation terms— or implicitly —in the optimisation algorithms or parameter-function maps in many methods , including deep learning ( Valle-Perez et al. , 2019 ; Shah et al. , 2020 ) . Occam ’ s razor is backed by strong theoretical and empirical reasons ( Solomonoff , 1964 ; Dingle et al. , 2018 ) , which oppose the idea that all hypotheses should be equally likely a priori , an assumption that would lead to the no free lunch theorems results ( Wolpert , 1996 ) , making learning impossible . While simplicity is always relative to a reference language , the invariance theorem ( see , e.g. , Li and Vitányi , 2008 ) ensures that the use of different ( optimal ) languages gives at most a constant difference that only depends on these two languages . While these constants may be very large , some level of agreement in what simplicity is can happen even for very different representations , as we explore in this paper . A prior for simplicity is hard to understand in large language models , a kind of architecture that is usually overparametrised , compared to the size of the evidence . While the model is not necessarily the most compressed representation of the data , it leads to continuations with low perplexity , which can be used to compress future data , as language models were introduced for communication ( Shannon , 1949 ) . It is this interpretation that sheds light on the role of priors , compression and simplicity in human communication and understanding ( Chater and Vitányi , 2003 ; Dowe et al. , 2011 ) . It is precisely in the context of few-shot inference , once the model has been trained , that we can reformulate the question as whether these models make inferences based on a preference for simpler hypotheses . We use the term ‘ distillation ’ , in a somewhat related way to ‘ knowledge distillation ’ ( Gou et al. , 2021 ) , to represent that the simplicity prior has emerged or have been distilled in the trained model , and used as ‘ learning ’ bias in few-shot inference . For further discussion about Occam ’ s razor , compression and inference , see Section 15 in the supplementary material .

Summary: This paper studies the type of patterns learned by large pre-trained language models (LMs) in a few-show setting. The authors differentiate between external patterns (e.g., common-sense or world-knowledge), and internal patterns, such as ABAB, and focus on the latter. They frame this as a teaching problem with strong priors, and study the size of the example set (number of shots in the few-shot setting) compared to the length of explanation that can be provided to the model in order to teach it the given pattern. The authors focus on the simplicity prior, and study whether it is implicitly encoded in such large models. Experiment that compare LMs, program induction systems and humans are performed. The main findings are that LMs perform as well as humans in few-shot settings, but the two populations differ in their ability to generate explanations.  I would say upfront that I am not an expert in this topic, and have found many points in the paper unclear, perhaps due to this. However, I wasn't fully convinced by the setup used by the authors. I am looking forward to the author response that might clarify some of these points.