Rethinking supervised learning: insights from biological learning and from calling it by its name

The renaissance of artificial neural networks was catalysed by the success of clas-1 sification models , tagged by the community with the broader term supervised2 learning . The extraordinary results gave rise to a hype loaded with ambitious3 promises and overstatements . Soon the community realised that the success owed4 much to the availability of thousands of labelled examples and supervised learning5 went , for many , from glory to shame : Some criticised deep learning as a whole and6 others proclaimed that the way forward had to be “ alternatives ” to supervised learn-7 ing : predictive , unsupervised , semi-supervised and , more recently , self-supervised8 learning . However , these seem all brand names , rather than actual categories9 of a theoretically grounded taxonomy . Moreover , the call to banish supervised10 learning was motivated by the questionable claim that humans learn with little or11 no supervision and are capable of robust out-of-distribution generalisation . Here,12 we review insights about learning and supervision in nature , revisit the notion that13 learning and generalization are not possible without supervision or inductive biases14 and argue that we will make better progress if we just call it by its name.15 1 Introduction16 The re-emergence of deep learning during the last decade due to the noteworthy achievements17 of artificial neural networks ( ANN ) built up a sort of philosophy that nearly anything could be18 automatically learnt from data without human intervention , in contrast to the previous approaches:19 [ hand designing good feature extractors , engineering skill and domain expertise ] 20 can all be avoided if good features can be learned automatically using a general-21 purpose learning procedure . This is the key advantage of deep learning ( LeCun22 et al. , 2015 ) .23 Read in hindsight , this claim was clearly an overstatement . The success of deep learning has24 required iterative hand design of network architectures and techniques that demanded collective , high25 engineering skill and large doses of interdisciplinary domain expertise . Furthermore , deep learning26 owes much to the immense computational power poured into training artificial networks ( Amodei & 27 Hernandez , 2018 ; Schwartz et al. , 2019 ) and to the human effort of manually collecting and labelling28 thousands of images and other data modalities ( Russakovsky et al. , 2015 ; Cao et al. , 2018 ) . However,29 the gist of the claim has permeated machine learning research and is pervasive up to these days.30 The realisation that the success of deep learning was largely due to the availability of huge labelled data31 sets prompted various reactions : some authors strongly questioned the usefulness of the algorithms32 ( Marcus , 2018 ) ; some delved into the question of whether neural networks generalise beyond or33 simply memorise the training examples ( Zhang et al. , 2017 ; Arpit et al. , 2017 ) ; and some proposed34 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . new research horizons that can be overly ambitious and potentially misleading : “ learning a class35 from a single labelled example ” , based on the statement that “ humans learn new concepts with very36 little supervision , [ but ] the standard supervised deep learning paradigm does not offer a satisfactory37 solution for learning new concepts rapidly from little data ” ( Vinyals et al. , 2016 ) . As a consequence,38 multiple research programmes , with various brand names , followed up with the aim of minimising or39 removing the need for “ supervision ” to train neural networks : few-shot , one-shot , zero-shot , predictive,40 unsupervised , semi-supervised and self-supervised learning are only a few popular examples.41 Exploring alternatives to classification and improving the efficiency of learning algorithms should42 indeed be a priority of machine learning research . As a matter of fact , related approaches have43 been subject of study since long before the explosion of deep learning ( Hinton & Sejnowski , 1999 ; 44 Chapelle et al. , 2006 ) . However , the current publication and discussion trends in the field denote45 overambitious promises that are in part based on misconceptions and overstatements about biological46 learning , and amplified by overselling nomenclature . While much of the research output derived from47 these programmes does provide us with useful techniques and insight , it leaves behind a landscape of48 confusing terminology and tangled research directions that are hard to navigate and lead many astray.49 In this paper , we reflect upon fundamental concepts in machine learning such as supervision , in-50 ductive biases and generalisation , which in spite of resting on theoretical grounds , are at the core51 of misconceptions and overstatements about deep learning commonly seen in the literature . First,52 we review aspects from biological learning , and compare them to the traits often ( mis ) attributed to53 human learning and generalisation in the machine learning literature ( Section 2 ) . Second , we revisit54 insights from classical statistical learning and critically review the terminology and current trends in55 deep learning research ( Section 3 ) . Altogether , we aim at tempering certain claims and promises of56 deep learning , helping mitigate the confusion over the terminology and suggesting desirable—in our57 opinion—directions and changes in machine learning research.58 2 Supervision in biological learning59 The link between artificial intelligence—specifically artificial neural networks ( Rosenblatt , 1958 ; 60 Fukushima & Miyake , 1982 ) —and biological learning systems is intrinsic to the field , as one long-61 term goal of artificial intelligence is to mirror the capabilities of human intelligence . However , these62 capabilities are , in our view , often overestimated . One example is the argument that intelligence in63 nature evolves without supervision and is capable of robust out-of-distribution generalisation . In64 particular , it is often claimed that humans and other animals learn to visually categorise objects with65 little or no supervision from a few examples ( Vinyals et al. , 2016 ; Marcus , 2018 ; Morgenstern et al.,66 2019 ) . In what follows , we will discuss three aspects of biological learning to argue against this67 view , so as to gain insights that better inform our progress in machine learning : first , we will discuss68 how generalisation requires exposure to relevant training data ; second , we will review the variety of69 supervised signals that the brain has access to ; third , we will comment on the role of evolution and70 brain development.71 2.1 Generalisation requires exposure to relevant training data72 In the argument that machine learning models should generalise from a few examples , there seems73 to be a promise or aspiration that future better methods will be able to perform robust visual object74 categorisation—for instance—among many object classes after being trained on one or a few examples75 per class . While a primary objective is to develop techniques that efficiently extract the maximum76 possible information from the available examples , we should also remind ourselves that no machine77 learning algorithm can robustly learn anything that can not be inferred from the data it has been trained78 on . Although this may seem to contradict certain current trends and statements in the literature , we79 should also bear in mind that learning in nature is not different.80 First , the amount of data that animals and humans in particular are exposed to is often underestimated.81 A biological brain continuously receives , processes and integrates multimodal inputs from various82 sensors—images ( light ) , sound , smell , etc . Humans do not learn to recognise objects by looking83 at photos from ImageNet , but are rather exposed to a continuous flow of visual stimuli with slow84 changes of the viewing angle and lighting conditions . Furthermore , the stimuli are coherent across85 modalities , we are allowed to interact with the objects and we even receive multiple supervision86 signals , as we discuss later.87 The exposure to so much training data makes the human visual system remarkably robust , but still its88 capabilities are optimised for the tasks it needs to perform and largely determined by the training data89 distribution—and years of evolution , as we will discuss below . For instance , a well-studied property90 of human vision is that our face recognition ability is severely impaired if faces are presented upside91 down ( Yin , 1969 ; Valentine , 1988 ) . Setting aside the specific complexity of face processing in the92 brain , a compelling explanation for this impairment is that we are simply not used to seeing and93 recognising inverted faces . More generally , while human perception of objects is largely invariant94 under certain conditions ( Biederman & Bar , 1999 ) , object recognition is sensitive to changes in view95 angle ( Tarr et al. , 1998 ) , especially when we see objects from unfamiliar viewpoints ( Edelman & 96 Bülthoff , 1992 ; Bülthoff & Newell , 2006 ; Milivojevic , 2012 ) .97 Furthermore , although better than the one-shot or few-shot generalisation of current ANNs , humans98 also have limited ability to recognise truly novel classes ( Morgenstern et al. , 2019 ) . Interestingly,99 experiments with certain novel classes of objects known as Greebles showed that , with sufficient100 training , humans can acquire expertise in recognising new objects from different viewpoints , even101 making use of an area of the brain—the fusiform face area—that typically responds strongly with102 face stimuli ( Gauthier et al. , 1999 ) . This provides evidence that recognition from multiple viewpoints103 is possible but only developed after exposure to similar conditions , that is relevant data . This is104 reminiscent of the effectiveness of data augmentation in deep learning , compared to more naïve105 regularisation methods ( Hernández-García & König , 2018 ) .106 The need for exposure to relevant stimuli challenges the notion that humans are capable of strong107 out-of-distribution generalisation . Rather , it seems that the transfer learning capabilities of humans108 are limited to relatively small changes in the data distribution . A compelling example is our difficulty109 to learn new languages : someone who natively speaks or has learnt Spanish will be able to transfer a110 significant amount of knowledge if they are to learn Italian , due to the overlap in the data distribution,111 but they will have very little to transfer for learning Kanien ’ kéha or Mandarin.112 2.2 Supervised signals for the brain113 Another commonly found argument has it that children—animals in general—learn robust object114 recognition without supervision : “ a child can generalize the concept of ‘ giraffe ’ from a single picture115 in a book ” ( Vinyals et al. , 2016 ) . First of all , we should mention the role of evolution ( expanded in116 Section 2.3 ) , which can be interpreted as a pre-trained model , optimised through millions of years of117 data with natural selection as a supervisory signal ( Zador , 2019 ) . Second , there is abundant evidence118 to argue against the very claim that children—and adults—learn in fully unsupervised fashion.119 Obviously , the kind of supervision that humans make use of is not that of classification algorithms—120 we do not see a class label on top of every object we look at . However , we receive supervision121 from multiple sources . Even though not for every visual stimulus , children do frequently receive122 information about the object classes they see . For instance , parents would point at objects and name123 them , then we learn how to read , and generally play a crucial role as teachers in language development124 ( Kuhl , 2007 ) . Non-human animals such as zebra finches learning to sing have also been found to rely125 on feedback ( supervision ) from the female adult and not just imitation Carouso-Peck & Goldstein126 ( 2019 ) . Furthermore , humans usually follow guided hierarchical learning : children do not directly127 learn to tell apart breeds of dogs , but rather start with umbrella terms and then progressively learn128 down the class hierarchy ( Bornstein & Arterberry , 2010 ; Spriet et al. , 2021 ) . Gopnik ( 2021 ) has129 asserted that “ we learn more from other people than we do from any other source ” and Hasson et al.130 ( 2020 ) mention other examples of supervision from social cues , that is from other humans , such as131 learning to recognise individual faces , produce grammatical sentences , read and write ; as well as132 from embodiment and action , such as learning to balance the body while walking or grasping objects.133 In all these actions , we can identify a supervisory signal that surely influences learning in the brain134 ( Shapiro , 2012 ; Gopnik et al. , 2020 ) .135 While these supervision signals largely differ from what is most commonly considered supervised136 learning in machine learning , we can still draw some parallels with human learning . We learn to137 categorise many concepts and objects as children , but most people carry on learning new categories as138 adults . For example , some people put effort in improving their understanding of the natural world by139 learning to recognise and name trees , plants or birds . Those who have engaged in such an endeavour140 may have noticed that the learning process is easier and faster if we count upon the expert knowledge141 of a friend or of technology such as iNaturalist ( Van Horn et al. , 2018 ) . Another example : those142 who have—or attempted to learn—a new language as an adult may have realised that whereas it is143 possible to learn the meaning of a new word by repeated exposure to it in multiple contexts , it is144 certainly easier if we look up the ground truth definition in a dictionary or , even easier , if there exists145 a direct mapping to a word in our native language . Summing up , not only does supervision facilitate146 learning , but human beings actively seek for it.147 Besides this kind of explicit supervision , the brain certainly makes use of more subtle , implicit148 supervised signals , such as temporal stability ( Becker , 1999 ; Wyss et al. , 2003 ) : The light that enters149 the retina , and the sound waves that reach the cochlea , are not random signals from a sequence of150 rapidly changing arbitrary photos or noise , but highly coherent and regular flows of slowly changing151 stimuli , especially at the higher , semantical level ( Kording et al. , 2004 ) . At the very least , this is how152 we perceive it and if such a smooth perception turns out to be a consequence rather than a cause , then153 it should be a by-product of a long process of evolution that would be worth taking into account.154 2.3 The role of evolution and brain development155 In the previous sections , we have discussed some misconceptions or overstatements about how156 humans learn and generalise that are often found in the machine literature . Namely , that humans157 are able to generalise from a few examples and that this occurs with little or no supervision . Still,158 the commonplace comparison of artificial neural networks with human learning and the brain often159 misses a fundamental component of biology , recently brought to the fore by Zador ( 2019 ) and Hasson160 et al . ( 2020 ) , although considered since the early days of artificial intelligence ( Turing , 1968 ) : the161 role that millions of years of evolution have played in developing the nervous systems of organisms162 in nature , including the human brain.163 The most common way of training artificial neural networks , especially in machine learning research,164 is from tabula rasa , that is from randomly initialised parameters1 . In contrast , a large part of the165 brain connectivity is encoded genetically and certain properties and behaviour are known to be166 innate , that is developed without prior exposure to stimuli ( Farroni et al. , 2005 ; Spriet et al. , 2021 ) .167 Importantly , evolution not only provides innate behaviour , but also determines what can not be learnt,168 or relevant constraints—scientists who have trained animals in the laboratory for psychological169 or neuroscientific studies are well aware that tasks have to be carefully adapted to the ecological170 behaviour and limitations of the animal , determined by evolution.171 Taking into account the role of evolution , we can draw conclusions that relate to the claims discussed172 in the previous sections . If our brains are the product of millions of years of exposure to relevant173 stimuli and adaptation , is it really fair to say that humans are capable of robust out-of-distribution174 generalisation and that we learn from from a few examples ? If evolution has largely determined175 what our brain can and can not learn , providing as with a “ pre-trained model ” , is it really fair to176 say that humans learn in a unsupervised fashion ? This questions are relevant for machine learning177 research : if we take biological learning as motivation for artificial intelligence , should we not temper178 our expectations of what learning algorithms should aspire to ? And , therefore , would it not be worth179 reconsidering some research programmes ? 180 On the flip side , insights from evolutionary theory are likely to be a fruitful source of inspiration181 for machine learning ( Hasson et al. , 2020 ; Zador , 2019 ) . As we have observed , training a neural182 network from scratch may be more similar to a simulation of evolution than to the process by which183 an adult learns a new concept . As a shortcut to simulating evolution , neuroscience is a rich source184 of inspiration of constraints and inductive biases that determine learning in the biological brain and185 can potentially inform machine learning ( Hassabis et al. , 2017 ; Lindsey et al. , 2019 ) . For instance,186 simulating properties of the primary visual cortex in the early layers of an artificial neural network187 has been shown to improve adversarial robustness ( Dapello et al. , 2020 ; Malhotra et al. , 2020 ) .188 Besides evolution , the focus on the capabilities of adults often makes us miss another important189 aspect of biological learning , particularly important in humans : the role of learning in infancy and190 brain development . While learning occurs too in adulthood , childhood is a particularly important and191 1Some interesting and promising areas in machine learning research deviate from this standard approach . For example , transfer learning and domain adaptation study the potential of features learnt on one task to be reused in different , related tasks ( Zhuang et al. , 2019 ) , and continual learning studies the ways in which machine learning models can indefinitely sustain the acquisition of new knowledge without detriment of the previously learnt tasks ( Mundt et al. , 2020 ) . These approaches are inspired by biological learning or share interesting properties with it . active time for learning ( Atkinson , 2002 ; Gelman & Meyer , 2011 ) . In fact , sensitive or critical periods192 for learning in infancy have been described or hypothesised , for example for vision ( Harwerth et al.,193 1986 ) and language development ( Lenneberg , 1967 ) . Machine learning papers that draw motivation194 from the alleged generalisation capabilities of humans often underestimate the amount of input stimuli195 and supervision that infants receive ( Gopnik , 2020 ) . However , childhood can be regarded as period196 dedicated almost exclusively to learn , not only formally from parents and teachers , but also through197 playing , which plays a critical role in cognitive development Burghardt ( 2005 ) ; Pelz & Kidd ( 2020 ) .198 Finally , the fact that humans—and other cognitively advanced animals , such as corvid birds , which199 also exhibit cultural learning—have a comparatively long childhood period , has led Uomini et al.200 ( 2020 ) to recently proposed that extended parenting is pivotal in the evolution of cognition . This201 adds to the discussion on the undervalued role of supervision . In sum , we propose machine learning202 research can benefit from drawing inspiration from both evolutionary biology and the literature on203 developmental psychology , brain development and life history and learning ( Gopnik et al. , 2020 ) .204 3 Supervision in machine learning205 If we open a machine learning textbook ( Murphy , 2012 ; Abu-Mostafa et al. , 2012 ; Goodfellow et al.,206 2016 ) , we will most surely find a taxonomy of learning algorithms with a clear distinction between207 supervised and unsupervised learning . However , while this separation can be useful , the boundaries208 are certainly not clear . As a matter of fact , if we take a look at the deep learning literature of the past209 years , we will also find abundant work on some variants supposedly in between—semi-supervised210 learning , self-supervised learning , etc.—whose definitions are all but clear.211 3.1 Catastrophic forgetting of old concepts212 If we recall a classical result in statistical learning theory and inference , the no free lunch theorem213 ( Wolpert , 1996 ) , no learning algorithm is better than any other at classifying unobserved data points,214 when averaged over all possible data distributions . Therefore , we need to constrain the distributions215 or , in other words , introduce prior knowledge—that is supervision . Recently , Locatello et al . ( 2018 ) 216 obtained a related result for the case of unsupervised learning of disentangled representations : without217 inductive biases for both the models and the data sets , unsupervised disentanglement learning is218 fundamentally impossible . These results are purely theoretical and have limited impact on real219 world applications ( Giraud-Carrier & Provost , 2005 ) , precisely because in practice we use multiple220 inductive biases and implicit supervision , even when we do so-called unsupervised learning.221 In a strict sense , even the classical , purely unsupervised methods , such as independent component222 analysis or nearest neighbours classifiers , make use of inductive biases , such as independence or223 minimum distance , respectively . Without inductive bias , learning is not possible : purely unsupervised224 learning is an illusion . While this is not news , the terminology used in the recent and current machine225 learning literature seems to reject supervision and neglect these nuances , evidencing that the field226 suffers catastrophic forgetting of well-established notions.227 3.2 The brands of alt-supervised learning228 Particularly in deep learning and computer vision , the term supervised learning has adopted , in229 practice , the meaning of classification of examples annotated by humans , that is models trained230 on examples labelled according to , for instance , the object classes . This is yet another instance of231 catastrophic forgetting—or , at best , abuse—of well-established concepts . It should not be necessary232 to recall that , first of all , supervised learning is a broader category than classification , which includes233 also regression and ranking , among other learning modalities . Second , even if we narrow our view234 to classification only , supervised learning is not restricted to learning from examples annotated by235 humans . Goodfellow et al . ( 2016 ) did not overlook this in their definition of supervised learning : “ In236 many cases the outputs y may be difficult to collect automatically and must be provided by a human237 ‘ supervisor , ’ but the term still applies even when the training set targets were collected automatically ” .238 In turn , the term unsupervised learning is now used for any model that does not use manually collected239 labels , regardless of what other kind of supervision it may use . Further , the term semi-supervised240 learning generally refers in practice to models that are trained with a fraction of the labels , but are241 tested on the same classification benchmarks . Finally , the term self-supervised learning has recently242 gained much popularity , referring to models that are trained on tasks other than the standard task243 defined by classification labels.244 Some of the methods proposed under these categories are certainly useful—that is not the subject of245 criticism of this work—but the terminology is overly confusing and unnecessary . A newcomer would246 easily fall into a scientific rabbit hole trying to discern the meaning of each of these names through247 publications—not to mention if they incorporated social media discussions into their endeavour . By248 way of illustration , the authors of this paper have witnessed how a recurrent question by students who249 learn about recent deep learning methods is whether there is any difference between self-supervised250 and unsupervised learning . Are students missing something fundamental ? The following anecdotal251 recall of influential keynote talks at artificial intelligence conferences should shed some light on part252 of the origins of this confusion : In December 2016 , Prof. Yann LeCun titled his NeurIPS keynote253 presentation “ Predictive Learning ” , to refer to “ what many people mean by unsupervised learning ” 254 ( LeCun , 2016 ) . A few years later , in his keynote presentation at ISSCC in February 2019 , he spoke255 about similar ideas , but this time the title was “ Self-Supervised Learning ” ( LeCun , 2019 ) . In social256 media , he wrote : “ I now call it ‘ self-supervised learning ’ , because ‘ unsupervised ’ is both a loaded257 and confusing term ” . Students may be getting things rather right.258 Is there then a fundamental difference—a theoretically grounded one—between the deep learning259 methods labelled as unsupervised learning and more recently self-supervised learning ? We argue260 that these are mostly brand names that reflect trends in the field , adding noise to the scientific261 progress and leading many astray . Therefore , we propose that , given the recent progress , the field262 of machine learning research would benefit from an exercise of self-reflection and from an effort to263 devise a rigorous taxonomy of the variety of methods . From a theoretical point of view , both the264 conventional classification models and the recent wave of self-supervised tasks can all be formalised265 as sub-categories of supervised learning.266

Summary: With this opinion paper the authors aim at clarifying some terminology commonly used in the modern deep learning literature. The goal is to highlight the strong overlap between several approaches that have been proposed as "alternatives" to supervised learning: predictive, unsupervised, semi-supervised and self-supervised learning. According to the authors, such proliferation of terms has been promoted by the (wrong) conviction that supervised learning is too limited in scope and does not account for the great variety of learning mechanisms observed in biological agents. The authors argue that the deep learning community would benefit from a clarification in terminology, which could highlight that, overall, all learning approaches can be defined as “supervised”.