Rethinking supervised learning: insights from biological learning and from calling it by its name
Year: 2021

Abstract:
The renaissance of artificial neural networks was catalysed by the success of clas1 sification models, tagged by the community with the broader term supervised 2 learning. The extraordinary results gave rise to a hype loaded with ambitious 3 promises and overstatements. Soon the community realised that the success owed 4 much to the availability of thousands of labelled examples and supervised learning 5 went, for many, from glory to shame: Some criticised deep learning as a whole and 6 others proclaimed that the way forward had to be “alternatives” to supervised learn7 ing: predictive, unsupervised, semi-supervised and, more recently, self-supervised 8 learning. However, these seem all brand names, rather than actual categories 9 of a theoretically grounded taxonomy. Moreover, the call to banish supervised 10 learning was motivated by the questionable claim that humans learn with little or 11 no supervision and are capable of robust out-of-distribution generalisation. Here, 12 we review insights about learning and supervision in nature, revisit the notion that 13 learning and generalization are not possible without supervision or inductive biases 14 and argue that we will make better progress if we just call it by its name. 15

text: The renaissance of artificial neural networks was catalysed by the success of clas-1 sification models, tagged by the community with the broader term supervised2 learning. The extraordinary results gave rise to a hype loaded with ambitious3 promises and overstatements. Soon the community realised that the success owed4 much to the availability of thousands of labelled examples and supervised learning5 went, for many, from glory to shame: Some criticised deep learning as a whole and6 others proclaimed that the way forward had to be “alternatives” to supervised learn-7 ing: predictive, unsupervised, semi-supervised and, more recently, self-supervised8 learning. However, these seem all brand names, rather than actual categories9 of a theoretically grounded taxonomy. Moreover, the call to banish supervised10 learning was motivated by the questionable claim that humans learn with little or11 no supervision and are capable of robust out-of-distribution generalisation. Here,12 we review insights about learning and supervision in nature, revisit the notion that13 learning and generalization are not possible without supervision or inductive biases14 and argue that we will make better progress if we just call it by its name.15
1 Introduction16
The re-emergence of deep learning during the last decade due to the noteworthy achievements17 of artificial neural networks (ANN) built up a sort of philosophy that nearly anything could be18 automatically learnt from data without human intervention, in contrast to the previous approaches:19
[hand designing good feature extractors, engineering skill and domain expertise]20 can all be avoided if good features can be learned automatically using a general-21 purpose learning procedure. This is the key advantage of deep learning (LeCun22 et al., 2015).23
Read in hindsight, this claim was clearly an overstatement. The success of deep learning has24 required iterative hand design of network architectures and techniques that demanded collective, high25 engineering skill and large doses of interdisciplinary domain expertise. Furthermore, deep learning26 owes much to the immense computational power poured into training artificial networks (Amodei &27 Hernandez, 2018; Schwartz et al., 2019) and to the human effort of manually collecting and labelling28 thousands of images and other data modalities (Russakovsky et al., 2015; Cao et al., 2018). However,29 the gist of the claim has permeated machine learning research and is pervasive up to these days.30
The realisation that the success of deep learning was largely due to the availability of huge labelled data31 sets prompted various reactions: some authors strongly questioned the usefulness of the algorithms32 (Marcus, 2018); some delved into the question of whether neural networks generalise beyond or33 simply memorise the training examples (Zhang et al., 2017; Arpit et al., 2017); and some proposed34
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
new research horizons that can be overly ambitious and potentially misleading: “learning a class35 from a single labelled example”, based on the statement that “humans learn new concepts with very36 little supervision, [but] the standard supervised deep learning paradigm does not offer a satisfactory37 solution for learning new concepts rapidly from little data” (Vinyals et al., 2016). As a consequence,38 multiple research programmes, with various brand names, followed up with the aim of minimising or39 removing the need for “supervision” to train neural networks: few-shot, one-shot, zero-shot, predictive,40 unsupervised, semi-supervised and self-supervised learning are only a few popular examples.41
Exploring alternatives to classification and improving the efficiency of learning algorithms should42 indeed be a priority of machine learning research. As a matter of fact, related approaches have43 been subject of study since long before the explosion of deep learning (Hinton & Sejnowski, 1999;44 Chapelle et al., 2006). However, the current publication and discussion trends in the field denote45 overambitious promises that are in part based on misconceptions and overstatements about biological46 learning, and amplified by overselling nomenclature. While much of the research output derived from47 these programmes does provide us with useful techniques and insight, it leaves behind a landscape of48 confusing terminology and tangled research directions that are hard to navigate and lead many astray.49
In this paper, we reflect upon fundamental concepts in machine learning such as supervision, in-50 ductive biases and generalisation, which in spite of resting on theoretical grounds, are at the core51 of misconceptions and overstatements about deep learning commonly seen in the literature. First,52 we review aspects from biological learning, and compare them to the traits often (mis)attributed to53 human learning and generalisation in the machine learning literature (Section 2). Second, we revisit54 insights from classical statistical learning and critically review the terminology and current trends in55 deep learning research (Section 3). Altogether, we aim at tempering certain claims and promises of56 deep learning, helping mitigate the confusion over the terminology and suggesting desirable—in our57 opinion—directions and changes in machine learning research.58
2 Supervision in biological learning59
The link between artificial intelligence—specifically artificial neural networks (Rosenblatt, 1958;60 Fukushima & Miyake, 1982)—and biological learning systems is intrinsic to the field, as one long-61 term goal of artificial intelligence is to mirror the capabilities of human intelligence. However, these62 capabilities are, in our view, often overestimated. One example is the argument that intelligence in63 nature evolves without supervision and is capable of robust out-of-distribution generalisation. In64 particular, it is often claimed that humans and other animals learn to visually categorise objects with65 little or no supervision from a few examples (Vinyals et al., 2016; Marcus, 2018; Morgenstern et al.,66 2019). In what follows, we will discuss three aspects of biological learning to argue against this67 view, so as to gain insights that better inform our progress in machine learning: first, we will discuss68 how generalisation requires exposure to relevant training data; second, we will review the variety of69 supervised signals that the brain has access to; third, we will comment on the role of evolution and70 brain development.71
2.1 Generalisation requires exposure to relevant training data72
In the argument that machine learning models should generalise from a few examples, there seems73 to be a promise or aspiration that future better methods will be able to perform robust visual object74 categorisation—for instance—among many object classes after being trained on one or a few examples75 per class. While a primary objective is to develop techniques that efficiently extract the maximum76 possible information from the available examples, we should also remind ourselves that no machine77 learning algorithm can robustly learn anything that cannot be inferred from the data it has been trained78 on. Although this may seem to contradict certain current trends and statements in the literature, we79 should also bear in mind that learning in nature is not different.80
First, the amount of data that animals and humans in particular are exposed to is often underestimated.81 A biological brain continuously receives, processes and integrates multimodal inputs from various82 sensors—images (light), sound, smell, etc. Humans do not learn to recognise objects by looking83 at photos from ImageNet, but are rather exposed to a continuous flow of visual stimuli with slow84 changes of the viewing angle and lighting conditions. Furthermore, the stimuli are coherent across85 modalities, we are allowed to interact with the objects and we even receive multiple supervision86 signals, as we discuss later.87
The exposure to so much training data makes the human visual system remarkably robust, but still its88 capabilities are optimised for the tasks it needs to perform and largely determined by the training data89 distribution—and years of evolution, as we will discuss below. For instance, a well-studied property90 of human vision is that our face recognition ability is severely impaired if faces are presented upside91 down (Yin, 1969; Valentine, 1988). Setting aside the specific complexity of face processing in the92 brain, a compelling explanation for this impairment is that we are simply not used to seeing and93 recognising inverted faces. More generally, while human perception of objects is largely invariant94 under certain conditions (Biederman & Bar, 1999), object recognition is sensitive to changes in view95 angle (Tarr et al., 1998), especially when we see objects from unfamiliar viewpoints (Edelman &96 Bülthoff, 1992; Bülthoff & Newell, 2006; Milivojevic, 2012).97
Furthermore, although better than the one-shot or few-shot generalisation of current ANNs, humans98 also have limited ability to recognise truly novel classes (Morgenstern et al., 2019). Interestingly,99 experiments with certain novel classes of objects known as Greebles showed that, with sufficient100 training, humans can acquire expertise in recognising new objects from different viewpoints, even101 making use of an area of the brain—the fusiform face area—that typically responds strongly with102 face stimuli (Gauthier et al., 1999). This provides evidence that recognition from multiple viewpoints103 is possible but only developed after exposure to similar conditions, that is relevant data. This is104 reminiscent of the effectiveness of data augmentation in deep learning, compared to more naïve105 regularisation methods (Hernández-García & König, 2018).106
The need for exposure to relevant stimuli challenges the notion that humans are capable of strong107 out-of-distribution generalisation. Rather, it seems that the transfer learning capabilities of humans108 are limited to relatively small changes in the data distribution. A compelling example is our difficulty109 to learn new languages: someone who natively speaks or has learnt Spanish will be able to transfer a110 significant amount of knowledge if they are to learn Italian, due to the overlap in the data distribution,111 but they will have very little to transfer for learning Kanien’kéha or Mandarin.112
2.2 Supervised signals for the brain113
Another commonly found argument has it that children—animals in general—learn robust object114 recognition without supervision: “a child can generalize the concept of ‘giraffe’ from a single picture115 in a book” (Vinyals et al., 2016). First of all, we should mention the role of evolution (expanded in116 Section 2.3), which can be interpreted as a pre-trained model, optimised through millions of years of117 data with natural selection as a supervisory signal (Zador, 2019). Second, there is abundant evidence118 to argue against the very claim that children—and adults—learn in fully unsupervised fashion.119
Obviously, the kind of supervision that humans make use of is not that of classification algorithms—120 we do not see a class label on top of every object we look at. However, we receive supervision121 from multiple sources. Even though not for every visual stimulus, children do frequently receive122 information about the object classes they see. For instance, parents would point at objects and name123 them, then we learn how to read, and generally play a crucial role as teachers in language development124 (Kuhl, 2007). Non-human animals such as zebra finches learning to sing have also been found to rely125 on feedback (supervision) from the female adult and not just imitation Carouso-Peck & Goldstein126 (2019). Furthermore, humans usually follow guided hierarchical learning: children do not directly127 learn to tell apart breeds of dogs, but rather start with umbrella terms and then progressively learn128 down the class hierarchy (Bornstein & Arterberry, 2010; Spriet et al., 2021). Gopnik (2021) has129 asserted that “we learn more from other people than we do from any other source” and Hasson et al.130 (2020) mention other examples of supervision from social cues, that is from other humans, such as131 learning to recognise individual faces, produce grammatical sentences, read and write; as well as132 from embodiment and action, such as learning to balance the body while walking or grasping objects.133 In all these actions, we can identify a supervisory signal that surely influences learning in the brain134 (Shapiro, 2012; Gopnik et al., 2020).135
While these supervision signals largely differ from what is most commonly considered supervised136 learning in machine learning, we can still draw some parallels with human learning. We learn to137 categorise many concepts and objects as children, but most people carry on learning new categories as138 adults. For example, some people put effort in improving their understanding of the natural world by139 learning to recognise and name trees, plants or birds. Those who have engaged in such an endeavour140 may have noticed that the learning process is easier and faster if we count upon the expert knowledge141 of a friend or of technology such as iNaturalist (Van Horn et al., 2018). Another example: those142
who have—or attempted to learn—a new language as an adult may have realised that whereas it is143 possible to learn the meaning of a new word by repeated exposure to it in multiple contexts, it is144 certainly easier if we look up the ground truth definition in a dictionary or, even easier, if there exists145 a direct mapping to a word in our native language. Summing up, not only does supervision facilitate146 learning, but human beings actively seek for it.147
Besides this kind of explicit supervision, the brain certainly makes use of more subtle, implicit148 supervised signals, such as temporal stability (Becker, 1999; Wyss et al., 2003): The light that enters149 the retina, and the sound waves that reach the cochlea, are not random signals from a sequence of150 rapidly changing arbitrary photos or noise, but highly coherent and regular flows of slowly changing151 stimuli, especially at the higher, semantical level (Kording et al., 2004). At the very least, this is how152 we perceive it and if such a smooth perception turns out to be a consequence rather than a cause, then153 it should be a by-product of a long process of evolution that would be worth taking into account.154
2.3 The role of evolution and brain development155
In the previous sections, we have discussed some misconceptions or overstatements about how156 humans learn and generalise that are often found in the machine literature. Namely, that humans157 are able to generalise from a few examples and that this occurs with little or no supervision. Still,158 the commonplace comparison of artificial neural networks with human learning and the brain often159 misses a fundamental component of biology, recently brought to the fore by Zador (2019) and Hasson160 et al. (2020), although considered since the early days of artificial intelligence (Turing, 1968): the161 role that millions of years of evolution have played in developing the nervous systems of organisms162 in nature, including the human brain.163
The most common way of training artificial neural networks, especially in machine learning research,164 is from tabula rasa, that is from randomly initialised parameters1. In contrast, a large part of the165 brain connectivity is encoded genetically and certain properties and behaviour are known to be166 innate, that is developed without prior exposure to stimuli (Farroni et al., 2005; Spriet et al., 2021).167 Importantly, evolution not only provides innate behaviour, but also determines what cannot be learnt,168 or relevant constraints—scientists who have trained animals in the laboratory for psychological169 or neuroscientific studies are well aware that tasks have to be carefully adapted to the ecological170 behaviour and limitations of the animal, determined by evolution.171
Taking into account the role of evolution, we can draw conclusions that relate to the claims discussed172 in the previous sections. If our brains are the product of millions of years of exposure to relevant173 stimuli and adaptation, is it really fair to say that humans are capable of robust out-of-distribution174 generalisation and that we learn from from a few examples? If evolution has largely determined175 what our brain can and cannot learn, providing as with a “pre-trained model”, is it really fair to176 say that humans learn in a unsupervised fashion? This questions are relevant for machine learning177 research: if we take biological learning as motivation for artificial intelligence, should we not temper178 our expectations of what learning algorithms should aspire to? And, therefore, would it not be worth179 reconsidering some research programmes?180
On the flip side, insights from evolutionary theory are likely to be a fruitful source of inspiration181 for machine learning (Hasson et al., 2020; Zador, 2019). As we have observed, training a neural182 network from scratch may be more similar to a simulation of evolution than to the process by which183 an adult learns a new concept. As a shortcut to simulating evolution, neuroscience is a rich source184 of inspiration of constraints and inductive biases that determine learning in the biological brain and185 can potentially inform machine learning (Hassabis et al., 2017; Lindsey et al., 2019). For instance,186 simulating properties of the primary visual cortex in the early layers of an artificial neural network187 has been shown to improve adversarial robustness (Dapello et al., 2020; Malhotra et al., 2020).188
Besides evolution, the focus on the capabilities of adults often makes us miss another important189 aspect of biological learning, particularly important in humans: the role of learning in infancy and190 brain development. While learning occurs too in adulthood, childhood is a particularly important and191
1Some interesting and promising areas in machine learning research deviate from this standard approach. For example, transfer learning and domain adaptation study the potential of features learnt on one task to be reused in different, related tasks (Zhuang et al., 2019), and continual learning studies the ways in which machine learning models can indefinitely sustain the acquisition of new knowledge without detriment of the previously learnt tasks (Mundt et al., 2020). These approaches are inspired by biological learning or share interesting properties with it.
active time for learning (Atkinson, 2002; Gelman & Meyer, 2011). In fact, sensitive or critical periods192 for learning in infancy have been described or hypothesised, for example for vision (Harwerth et al.,193 1986) and language development (Lenneberg, 1967). Machine learning papers that draw motivation194 from the alleged generalisation capabilities of humans often underestimate the amount of input stimuli195 and supervision that infants receive (Gopnik, 2020). However, childhood can be regarded as period196 dedicated almost exclusively to learn, not only formally from parents and teachers, but also through197 playing, which plays a critical role in cognitive development Burghardt (2005); Pelz & Kidd (2020).198 Finally, the fact that humans—and other cognitively advanced animals, such as corvid birds, which199 also exhibit cultural learning—have a comparatively long childhood period, has led Uomini et al.200 (2020) to recently proposed that extended parenting is pivotal in the evolution of cognition. This201 adds to the discussion on the undervalued role of supervision. In sum, we propose machine learning202 research can benefit from drawing inspiration from both evolutionary biology and the literature on203 developmental psychology, brain development and life history and learning (Gopnik et al., 2020).204
3 Supervision in machine learning205
If we open a machine learning textbook (Murphy, 2012; Abu-Mostafa et al., 2012; Goodfellow et al.,206 2016), we will most surely find a taxonomy of learning algorithms with a clear distinction between207 supervised and unsupervised learning. However, while this separation can be useful, the boundaries208 are certainly not clear. As a matter of fact, if we take a look at the deep learning literature of the past209 years, we will also find abundant work on some variants supposedly in between—semi-supervised210 learning, self-supervised learning, etc.—whose definitions are all but clear.211
3.1 Catastrophic forgetting of old concepts212
If we recall a classical result in statistical learning theory and inference, the no free lunch theorem213 (Wolpert, 1996), no learning algorithm is better than any other at classifying unobserved data points,214 when averaged over all possible data distributions. Therefore, we need to constrain the distributions215 or, in other words, introduce prior knowledge—that is supervision. Recently, Locatello et al. (2018)216 obtained a related result for the case of unsupervised learning of disentangled representations: without217 inductive biases for both the models and the data sets, unsupervised disentanglement learning is218 fundamentally impossible. These results are purely theoretical and have limited impact on real219 world applications (Giraud-Carrier & Provost, 2005), precisely because in practice we use multiple220 inductive biases and implicit supervision, even when we do so-called unsupervised learning.221
In a strict sense, even the classical, purely unsupervised methods, such as independent component222 analysis or nearest neighbours classifiers, make use of inductive biases, such as independence or223 minimum distance, respectively. Without inductive bias, learning is not possible: purely unsupervised224 learning is an illusion. While this is not news, the terminology used in the recent and current machine225 learning literature seems to reject supervision and neglect these nuances, evidencing that the field226 suffers catastrophic forgetting of well-established notions.227
3.2 The brands of alt-supervised learning228
Particularly in deep learning and computer vision, the term supervised learning has adopted, in229 practice, the meaning of classification of examples annotated by humans, that is models trained230 on examples labelled according to, for instance, the object classes. This is yet another instance of231 catastrophic forgetting—or, at best, abuse—of well-established concepts. It should not be necessary232 to recall that, first of all, supervised learning is a broader category than classification, which includes233 also regression and ranking, among other learning modalities. Second, even if we narrow our view234 to classification only, supervised learning is not restricted to learning from examples annotated by235 humans. Goodfellow et al. (2016) did not overlook this in their definition of supervised learning: “In236 many cases the outputs y may be difficult to collect automatically and must be provided by a human237 ‘supervisor,’ but the term still applies even when the training set targets were collected automatically”.238
In turn, the term unsupervised learning is now used for any model that does not use manually collected239 labels, regardless of what other kind of supervision it may use. Further, the term semi-supervised240 learning generally refers in practice to models that are trained with a fraction of the labels, but are241 tested on the same classification benchmarks. Finally, the term self-supervised learning has recently242
gained much popularity, referring to models that are trained on tasks other than the standard task243 defined by classification labels.244
Some of the methods proposed under these categories are certainly useful—that is not the subject of245 criticism of this work—but the terminology is overly confusing and unnecessary. A newcomer would246 easily fall into a scientific rabbit hole trying to discern the meaning of each of these names through247 publications—not to mention if they incorporated social media discussions into their endeavour. By248 way of illustration, the authors of this paper have witnessed how a recurrent question by students who249 learn about recent deep learning methods is whether there is any difference between self-supervised250 and unsupervised learning. Are students missing something fundamental? The following anecdotal251 recall of influential keynote talks at artificial intelligence conferences should shed some light on part252 of the origins of this confusion: In December 2016, Prof. Yann LeCun titled his NeurIPS keynote253 presentation “Predictive Learning”, to refer to “what many people mean by unsupervised learning”254 (LeCun, 2016). A few years later, in his keynote presentation at ISSCC in February 2019, he spoke255 about similar ideas, but this time the title was “Self-Supervised Learning” (LeCun, 2019). In social256 media, he wrote: “I now call it ‘self-supervised learning’, because ‘unsupervised’ is both a loaded257 and confusing term” . Students may be getting things rather right.258
Is there then a fundamental difference—a theoretically grounded one—between the deep learning259 methods labelled as unsupervised learning and more recently self-supervised learning? We argue260 that these are mostly brand names that reflect trends in the field, adding noise to the scientific261 progress and leading many astray. Therefore, we propose that, given the recent progress, the field262 of machine learning research would benefit from an exercise of self-reflection and from an effort to263 devise a rigorous taxonomy of the variety of methods. From a theoretical point of view, both the264 conventional classification models and the recent wave of self-supervised tasks can all be formalised265 as sub-categories of supervised learning.266

heading: 3.3 Supervision comes in different flavours267
text: In Section 2.2, we have seen examples of different forms of supervision used by humans and other268 animals. In machine learning, the field focused for many years on a few loss functions, such as269 classification and simple forms of regression. The relatively recent explosion of deep learning has270 brought about the development of several libraries for automatic differentiation (Baydin et al., 2017),271 which in turn have enabled the proposal of multiple loss functions and learning tasks with various272 types of supervision that can easily be optimised numerically by stochastic gradient descent and273 artificial neural networks. This has certainly opened promising and already fruitful avenues to274 incorporate richer forms of supervision and inductive biases other than classification, some inspired275 by biological learning, into machine learning algorithms.276
A currently popular example is image data augmentation: Although until recently it was seen as a277 naïve technique to simply create additional training data, data augmentation actually encodes rich prior278 knowledge about human visual perception, in the case of computer vision. This is why it outperforms279 explicit regularisation methods, which provide less effective inductive biases Hernández-García &280 König (2018), and was used in “semi-supervised” tasks Laine & Aila (2016). The rich information281 embedded in image transformations has been used to encourage invariant outputs under different282 augmentations through contrastive losses (Ye et al., 2019), and even at intermediate representations,283 inspired by the invariance in the visual cortex (Hernández-García et al., 2019), although these methods284 were not branded as self-supervision. The use of this term for losses based on data augmentation285 was further popularised after the success of similar methods such as SimCLR (Chen et al., 2020).286 Beyond data augmentation invariance, the zoo of self-supervised learning tasks in computer vision is287 rich and diverse: classifying the rotation applied to image patches (Gidaris et al., 2018), predicting288 image colourisation (Larsson et al., 2017), classifying the relative position of two image patches289 (Doersch et al., 2015), or even solving full jigsaw puzzles (Noroozi & Favaro, 2016) (Jing & Tian290 (2020) recently performed an extensive review).291
The current trend is to refer to these methods as self-supervised learning, but similar methods were292 referred to in the past as semi-supervised, unsupervised, and even predictive learning, as we have seen.293 A look at the papers reveals that these terms have been used mostly interchangeably. The terms self-294 and semi- and unsupervised learning imply that less supervision is used, but it would be misleading to295 seriously argue that the tasks are devoid of supervision. Most of these techniques make use of a wide296 range of surrogate tasks with supervisory signals defined by humans. In fact, they could have been297
called hyper-supervised2 learning. Here, we contend that these methods are all variants of supervised298 learning, only that supervision comes in different flavours, both in biological and machine learning,299 and we should call it by its name and ideally develop a rigorous taxonomy.300
4 Discussion301
In this paper, we have discussed some of the overambitious promises of the deep learning hype,302 namely that machine learning models should be able to generalise to unseen distributions, from a303 few examples, without human intervention or supervision. These claims have often been motivated304 by alleged generalisation capabilities of humans. In order to assess these motivations, we have305 first reviewed, in Section 2, some often overlooked characteristics of biological learning relevant306 to machine learning research. In particular, we have argued that humans and other animals receive307 extensive and diverse input stimuli as well as multiple supervisory signals, including the long history308 of evolution and cultural transmission. In the light of these insights from biological learning, we have309 then, in Section 3, critically reviewed the various terms that are currently used to refer to supposed310 alternatives to supervised learning: semi-, self- and unsupervised learning, among others. In sum,311 we pointed out that all these approaches are in fact supervised learning—though not necessarily312 classification—and the machine learning (research) community would benefit from using more313 rigorous, less overselling nomenclature, and from devising a more rigorous taxonomy.314
Supervision is not evil. It is at the core of statistical learning theory: learning is impossible without315 inductive biases or supervision. But supervision comes in different flavours, not only as classification316 labels. Neither is deep learning some sort of exceptional solution to learn without human intervention317 and supervision, nor is it a hopeless model class because it requires large data sets (Marcus, 2018).318 The human visual system is exposed to a lot of stimuli too. One exceptional advantage of deep319 learning is precisely that it is possible to effectively optimise different learning objectives, almost320 end-to-end, from large collections of nearly naturalistic sensory signals, such as digital images (Saxe321 et al., 2020). While other models are known to scale poorly as the amount of data increases, neural322 networks excel at fitting the training data and interpolating on unseen examples (Belkin et al., 2019;323 Hasson et al., 2020). This is a feature, not a bug. But we will make better progress if we exploit324 these advantages of deep learning without neglecting that supervision will always be necessary—the325 critical goal is how to best incorporate it and exploit it.326
In this regard, we argue that deep learning needs more supervision, and not less. A major focus327 of the deep learning community in the last decade has been image object classification. This has328 brought about unprecedented progress and unveiled the limitations of having classification as chief329 task and class labels as main supervisory signal. For example, deep classifiers have been found to330 learn spurious features that are highly discriminative for the classification task but with little true331 generalisation power and clearly not aligned with perceptual features (Jo & Bengio, 2017; Wang et al.,332 2019; Geirhos et al., 2020). In fact, this mismatch has been argued to be at the root of adversarial333 vulnerability (Ilyas et al., 2019) and seems to be the consequence of training highly expressive,334 over-parameterised models in heavily unconstrained tasks. This can be addressed with meaningful335 constraints, that is more and richer supervision, possibly inspired by human perception and biological336 learning. For example, combining a classification loss with a similarity loss inspired by the invariance337 in the visual cortex yields more robust representations without detriment to categorisation (Hernández-338 García et al., 2019), and simulating the properties of the primary visual cortex may improve the339 adversarial robustness of neural networks (Dapello et al., 2020). Expanding in this direction leads to340 biologically-inspired, multi-task and representation learning, and away from just classification.341
5 Conclusions for future research directions342
The chief goal of this paper is rather descriptive than prescriptive. We have aimed to identify and343 describe aspects of the current trends in machine learning research that could be improved, in the344 hope of inspiring future work that effectively address them. Nonetheless, throughout the paper we345 have made suggestions that may help mitigate the confusion with the terminology, clarify research346 directions and ultimately bring about scientific progress in machine learning research. We outline347 these suggestions here to conclude the paper.348
2The authors explicitly discourage the addition of a new term to the already too confusing list.
We have drawn parallels from cognitive neuroscience to contend that learning in nature also requires349 abundant data and supervision in multiple forms. Even evolution can be regarded as an optimisation350 process where natural selection is the supervisory signal. We have argued, as others have before, that351 these insights from biology, neuroscience and developmental psychology, among other fields, offer a352 great opportunity for machine learning research to draw inspiration and calibrate its compass.353
As we have discussed, research in deep learning has departed from pure classification and has been354 exploring new learning tasks and ways of training artificial neural networks. Nonetheless, in some355 fields such as computer vision, the ultimate benchmark to assess the value of a method is still the356 accuracy on classification data sets, such as ImageNet, even though there is evidence of overfitting357 the test set. While object recognition will remain an important benchmark, as deep learning is358 well suited to learn representations, we should develop methods to assess the quality of the learnt359 representations for tasks other than classification. In this regard, we encourage researchers to evaluate360 their models with tests that are still not widespread, such as the suitability for transfer learning,361 adversarial robustness, comparison with brain measurements, behavioural tasks, etc.362
We have also argued that the field would benefit from an effort to devise a rigorous taxonomy of363 learning methods that sheds light on the ocean of methods proposed in the past years. The terms364 self-, semi- and unsupervised learning have been used interchangeably and this is often a source of365 confusion for students and newcomers. While confusing terminology is natural in a rapidly growing,366 the time might have come for distilling the progress of the past years into rigorous nomenclature that367 better survive the test of time.368
Finally, we recall that most of the learning theory has been developed for simple loss functions such369 as binary classification or mean squared error regression, but certain methods successfully used in370 practice today escape the available theory. Given the success of this kind of more complex supervised371 objectives, the study of these methods from a theoretical point of view might be a fruitful direction372 for future work.373
Broader Impact374
Since this article does not present a new method or results from data sets, potential risks of “bias in375 the data” or “failure of the system” do not apply. As a critical review of current trends in the field and376 cite multiple research articles, some researchers could potentially feel addressed and affected by our377 mentions. We declare that we do not intend to negatively affect any individual researcher and we have378 only referred to individuals directly in the case of well-established scientist with a reputation. Our379 goal has been in any case to potentially improve scientific progress through a constructive reflection.380
References381 Abu-Mostafa, Y. S., Magdon-Ismail, M., and Lin, H.-T. Learning from data. AMLBooks, 2012.382
Amodei, D. and Hernandez, D. AI and compute. Accessed: 2020-09-28, 2018. URL https:383 //openai.com/blog/ai-and-compute/.384
Arpit, D. et al. A closer look at memorization in deep networks. arXiv preprint arXiv:1706.05394,385 2017.386
Atkinson, J. The developing visual brain. Oxford Scholarship Online, 2002.387
Baydin, A. G., Pearlmutter, B. A., Radul, A. A., and Siskind, J. M. Automatic differentiation in388 machine learning: a survey. Journal of Machine Learning Research (JMLR), 2017.389
Becker, S. Implicit learning in 3D object recognition: The importance of temporal context. Neural390 Computation, 1999.391
Belkin, M., Hsu, D., Ma, S., and Mandal, S. Reconciling modern machine-learning practice and the392 classical bias–variance trade-off. Proceedings of the National Academy of Sciences (PNAS), 2019.393
Biederman, I. and Bar, M. One-shot viewpoint invariance in matching novel objects. Vision Research,394 1999.395
Bornstein, M. H. and Arterberry, M. E. The development of object categorization in young chil-396 dren: Hierarchical inclusiveness, age, perceptual attribute, and group versus individual analyses.397 Developmental Psychology, 2010.398
Bülthoff, I. and Newell, F. N. The role of familiarity in the recognition of static and dynamic objects.399 Progress in Brain Research, 2006.400
Burghardt, G. M. The genesis of animal play: Testing the limits. Mit Press, 2005.401
Cao, Q., Shen, L., Xie, W., Parkhi, O. M., and Zisserman, A. VGGFace2: A dataset for recognising402 faces across pose and age. In IEEE International Conference on Automatic Face & Gesture403 Recognition. 2018.404
Carouso-Peck, S. and Goldstein, M. H. Female social feedback reveals non-imitative mechanisms of405 vocal learning in zebra finches. Current Biology, 2019.406
Chapelle, O., Schölkopf, B., and Zien, A. (eds.). Semi-supervised learning. MIT Press, 2006.407
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of408 visual representations. In International conference on machine learning. 2020.409
Dapello, J., Marques, T., Schrimpf, M., Geiger, F., Cox, D. D., and DiCarlo, J. J. Simulating a410 primary visual cortex at the front of cnns improves robustness to image perturbations. BioRxiv,411 2020.412
Doersch, C., Gupta, A., and Efros, A. A. Unsupervised visual representation learning by context413 prediction. In Proceedings of the IEEE international conference on computer vision, 2015.414
Edelman, S. and Bülthoff, H. H. Orientation dependence in the recognition of familiar and novel415 views of three-dimensional objects. Vision Research, 1992.416
Farroni, T., Johnson, M. H., Menon, E., Zulian, L., Faraguna, D., and Csibra, G. Newborns’ preference417 for face-relevant stimuli: Effects of contrast polarity. Proceedings of the National Academy of418 Sciences, 2005.419
Fukushima, K. and Miyake, S. Neocognitron: A new algorithm for pattern recognition tolerant of420 deformations and shifts in position. Pattern Recognition, 1982.421
Gauthier, I., Tarr, M. J., Anderson, A. W., Skudlarski, P., and Gore, J. C. Activation of the middle422 fusiform’face area’increases with expertise in recognizing novel objects. Nature Neuroscience,423 1999.424
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A.425 Shortcut learning in deep neural networks. arXiv preprint arXiv:2004.07780, 2020.426
Gelman, S. A. and Meyer, M. Child categorization. Wiley Interdisciplinary Reviews: Cognitive427 Science, 2011.428
Gidaris, S., Singh, P., and Komodakis, N. Unsupervised representation learning by predicting image429 rotations. arXiv preprint arXiv:1803.07728, 2018.430
Giraud-Carrier, C. and Provost, F. Toward a justification of meta-learning: Is the no free lunch431 theorem a show-stopper. In Proceedings of the ICML-2005 Workshop on Meta-learning, 2005.432
Goodfellow, I., Bengio, Y., and Courville, A. Deep learning. MIT Press, 2016.433
Gopnik, A. Childhood as a solution to explore–exploit tensions. Philosophical Transactions of the434 Royal Society B, 2020.435
Gopnik, A. Brain Inspired podcast, by Paul Middle Middlebrooks. BI 094 Alison Gopnik: Child-436 Inspired AI. https://braininspired.co/podcast/94/, 2021.437
Gopnik, A., Frankenhuis, W. E., and Tomasello, M. Introduction to special issue:‘life history and438 learning: how childhood, caregiving and old age shape cognition and culture in humans and other439 animals’, 2020.440
Harwerth, R. S., Smith, E. L., Duncan, G. C., Crawford, M., and Von Noorden, G. K. Multiple441 sensitive periods in the development of the primate visual system. Science, 1986.442
Hassabis, D., Kumaran, D., Summerfield, C., and Botvinick, M. Neuroscience-inspired artificial443 intelligence. Neuron, 2017.444
Hasson, U., Nastase, S. A., and Goldstein, A. Direct fit to nature: An evolutionary perspective on445 biological and artificial neural networks. Neuron, 2020.446
Hernández-García, A. and König, P. Data augmentation instead of explicit regularization. arXiv447 preprint arXiv:1806.03852, 2018.448
Hernández-García, A., König, P., and Kietzmann, T. Learning robust visual representations using449 data augmentation invariance. arXiv preprint arXiv:1906.04547, 2019.450
Hinton, G. and Sejnowski, T. (eds.). Unsupervised learning. MIT Press, 1999.451
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. Adversarial examples are452 not bugs, they are features. arXiv preprint arXiv:1905.02175, 2019.453
Jing, L. and Tian, Y. Self-supervised visual feature learning with deep neural networks: A survey.454 IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020.455
Jo, J. and Bengio, Y. Measuring the tendency of CNNs to learn surface statistical regularities. arXiv456 preprint arXiv:1711.11561, 2017.457
Kording, K. P., Kayser, C., Einhauser, W., and Konig, P. How are complex cell properties adapted to458 the statistics of natural stimuli? Journal of Neurophysiology, 2004.459
Kuhl, P. K. Is speech learning ‘gated’by the social brain? Developmental science, 2007.460
Laine, S. and Aila, T. Temporal ensembling for semi-supervised learning. arXiv preprint461 arXiv:1610.02242, 2016.462
Larsson, G., Maire, M., and Shakhnarovich, G. Colorization as a proxy task for visual understanding.463 In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.464
LeCun, Y. Predictive learning, December 2016. URL https://nips.cc/Conferences/2016/465 ScheduleMultitrack?event=6197. Invited talk at the Thirtieth Conference on Neural Informa-466 tion Processing Systems (NeurIPS) 2016. Slide 24, starting on minute 14:34.467
LeCun, Y. Deep learning hardware: Past, present, and future, February 2019. URL https://www.468 youtube.com/watch?v=YzD7Z2yRL7Y. Invited talk at the International Solid-State Circuits469 Conference (ISSCC) 2019. Slide 57, starting on minute 27:24.470
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. Nature, 2015.471
Lenneberg, E. H. The biological foundations of language. Hospital Practice, 1967.472
Lindsey, J., Ocko, S. A., Ganguli, S., and Deny, S. A unified theory of early visual representations473 from retina to cortex through anatomically constrained deep CNNs. In International Conference474 on Learning Representations (ICLR), arXiv:1901.00945, 2019.475
Locatello, F., Bauer, S., Lucic, M., Rätsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging476 common assumptions in the unsupervised learning of disentangled representations. arXiv preprint477 arXiv:1811.12359, 2018.478
Malhotra, G., Evans, B. D., and Bowers, J. S. Hiding a plane with a pixel: examining shape-bias in479 CNNs and the benefit of building in biological constraints. Vision Research, 2020.480
Marcus, G. Deep learning: A critical appraisal. arXiv preprint arXiv:1801.00631, 2018.481
Milivojevic, B. Object recognition can be viewpoint dependent or invariant–it’s just a matter of time482 and task. Frontiers in Computational Neuroscience, 2012.483
Morgenstern, Y., Schmidt, F., and Fleming, R. W. One-shot categorization of novel object classes in484 humans. Vision Research, 2019.485
Mundt, M., Hong, Y. W., Pliushch, I., and Ramesh, V. A wholistic view of continual learning with486 deep neural networks: Forgotten lessons and the bridge to active and open world learning. arXiv487 preprint arXiv:2009.01797, 2020.488
Murphy, K. P. Machine learning: a probabilistic perspective. MIT Press, 2012.489
Noroozi, M. and Favaro, P. Unsupervised learning of visual representations by solving jigsaw puzzles.490 In European conference on computer vision. 2016.491
Pelz, M. and Kidd, C. The elaboration of exploratory play. Philosophical Transactions of the Royal492 Society B, 2020.493
Rosenblatt, F. The perceptron: a probabilistic model for information storage and organization in the494 brain. Psychological Review, 1958.495
Russakovsky, O. et al. ImageNet large scale visual recognition challenge. International Journal of496 Computer Vision (IJCV), 2015.497
Saxe, A., Nelli, S., and Summerfield, C. If deep learning is the answer, then what is the question?498 arXiv preprint arXiv:2004.07580, 2020.499
Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, O. Green ai. arXiv preprint arXiv:1907.10597,500 2019.501
Shapiro, L. Embodied cognition. Oxford Handbooks Online, 2012.502
Spriet, C., Abassi, E., Hochmann, J.-R., and Papeo, L. Visual object categorization in infancy.503 bioRxiv, 2021.504
Tarr, M. J., Williams, P., Hayward, W. G., and Gauthier, I. Three-dimensional object recognition is505 viewpoint dependent. Nature Neuroscience, 1998.506
Turing, A. M. Cybernetics; (Key papers). University Park Press, 1968.507
Uomini, N., Fairlie, J., Gray, R. D., and Griesser, M. Extended parenting and the evolution of508 cognition. Philosophical Transactions of the Royal Society B, 2020.509
Valentine, T. Upside-down faces: A review of the effect of inversion upon face recognition. British510 Journal of Psychology, 1988.511
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and512 Belongie, S. The inaturalist species classification and detection dataset. In Proceedings of the513 IEEE conference on computer vision and pattern recognition, 2018.514
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al. Matching networks for one shot learning.515 In Advances in Neural Information Processing Systems (NeurIPS), 2016.516
Wang, H., Wu, X., Yin, P., and Xing, E. P. High frequency component helps explain the generalization517 of convolutional neural networks. arXiv preprint arXiv:1905.13545, 2019.518
Wolpert, D. H. The lack of a priori distinctions between learning algorithms. Neural Computation,519 1996.520
Wyss, R., König, P., and Verschure, P. F. Invariant representations of visual patterns in a temporal521 population code. Proceedings of the National Academy of Sciences (PNAS), 2003.522
Ye, M., Zhang, X., Yuen, P. C., and Chang, S.-F. Unsupervised embedding learning via invariant and523 spreading instance feature. In Proceedings of the IEEE/CVF Conference on Computer Vision and524 Pattern Recognition, 2019.525
Yin, R. K. Looking at upside-down faces. Journal of Experimental Psychology, 1969.526
Zador, A. M. A critique of pure learning and what artificial neural networks can learn from animal527 brains. Nature Communications, 2019.528
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires529 rethinking generalization. In International Conference on Learning Representations (ICLR),530 arXiv:1611.03530, 2017.531
Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., and He, Q. A comprehensive survey532 on transfer learning. arXiv preprint arXiv:1911.02685, 2019.533
Checklist534
1. For all authors...535 (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s536 contributions and scope? [Yes]537 (b) Did you describe the limitations of your work? [Yes] See the beginning of Section 5.538 (c) Did you discuss any potential negative societal impacts of your work? [Yes] See539 Section 5.540 (d) Have you read the ethics review guidelines and ensured that your paper conforms to541 them? [Yes]542 2. If you are including theoretical results...543
(a) Did you state the full set of assumptions of all theoretical results? [N/A]544 (b) Did you include complete proofs of all theoretical results? [N/A]545
3. If you ran experiments...546 (a) Did you include the code, data, and instructions needed to reproduce the main experi-547 mental results (either in the supplemental material or as a URL)? [N/A]548 (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they549 were chosen)? [N/A]550 (c) Did you report error bars (e.g., with respect to the random seed after running experi-551 ments multiple times)? [N/A]552 (d) Did you include the total amount of compute and the type of resources used (e.g., type553 of GPUs, internal cluster, or cloud provider)? [N/A]554 4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...555
(a) If your work uses existing assets, did you cite the creators? [N/A]556 (b) Did you mention the license of the assets? [N/A]557 (c) Did you include any new assets either in the supplemental material or as a URL? [N/A]558
559
(d) Did you discuss whether and how consent was obtained from people whose data you’re560 using/curating? [N/A]561
(e) Did you discuss whether the data you are using/curating contains personally identifiable562 information or offensive content? [N/A]563
5. If you used crowdsourcing or conducted research with human subjects...564 (a) Did you include the full text of instructions given to participants and screenshots, if565 applicable? [N/A]566 (b) Did you describe any potential participant risks, with links to Institutional Review567 Board (IRB) approvals, if applicable? [N/A]568 (c) Did you include the estimated hourly wage paid to participants and the total amount569 spent on participant compensation? [N/A]570

Review summaries:
This article is a position statement that aims to point out weaknesses in the  authors' conception of the ML field's agenda. As the authors state, the work is descriptive, not prescriptive: it does not provide specific directions, but rather points to trends that (in the extreme) are problematic. The key points in the article include: (1) natural environments provide rich support for learning beyond traditional supervised classification labels; (2) evolutionary and developmental processes provide strong inductive biases that can  inform model design; (3) the amount of data that natural learners have available is often underestimated (as is one-shot learning abilities); (4) there are many signals in natural environments which can constrain learning beyond traditional class labels.

With this opinion paper the authors aim at clarifying some terminology commonly used in the modern deep learning literature. The goal is to highlight the strong overlap between several approaches that have been proposed as "alternatives" to supervised learning: predictive, unsupervised, semi-supervised and self-supervised learning. According to the authors, such proliferation of terms has been promoted by the (wrong) conviction that supervised learning is too limited in scope and does not account for the great variety of learning mechanisms observed in biological agents. The authors argue that the deep learning community would benefit from a clarification in terminology, which could highlight that, overall, all learning approaches can be defined as “supervised”.