Rethinking supervised learning: insights from biological learning and from calling it by its name
Authors: Alex Hernández-García
Venue: NeurIPS 2021 Submitted (2021)
Keywords: supervised learning
self-supervised learning
biological learning
generalization

Abstract: The renaissance of artificial neural networks was catalysed by the success of classification models, tagged by the community with the broader term supervised learning. The extraordinary results gave rise to a hype loaded with ambitious promises and overstatements. Soon the community realised that the success owed much to the availability of thousands of labelled examples. And supervised learning went, for many, from glory to shame: Some criticised deep learning as a whole and others proclaimed that the way forward had to be alternatives to supervised learning: predictive, unsupervised, semi-supervised and, more recently, self-supervised learning. However, these seem all brand names, rather than actual categories of a theoretically grounded taxonomy. Moreover, the call to banish supervised learning was motivated by the questionable claim that humans learn with little or no supervision and are capable of robust out-of-distribution generalisation. Here, we review insights about learning and supervision in nature, revisit the notion that learning and generalization are not possible without supervision or inductive biases and argue that we will make better progress if we just call it by its name.

Rethinking supervised learning: insights from biological learning and from calling it by its name
Anonymous Author(s)
Affiliation
Address
email
Abstract
The renaissance of artificial neural networks catalysed by the success of clas- sification models, tagged by the community with the broader term supervised learning. The extraordinary results gave rise to hype loaded with ambitious promises and overstatements. Soon the community realised that the success owed much to the availability of thousands of labelled examples and supervised learning went, for many, from glory to shame: Some criticised deep learning as a whole and others proclaimed that the way forward had to be "alternatives" supervised learn- ing: predictive, unsupervised, semi-supervised and, more recently, self-supervised learning. However, these seem all brand names, rather than actual categories of theoretically grounded taxonomy. Moreover, the call to banish supervised learning was motivated by the questionable claim that humans learn with little no supervision and are capable of robust we review insights about learning and supervision in nature, revisit the notion that learning and generalization are not possible without supervision or inductive biases and argue that we will make better progress if we just call it by its name.
2
4
6
8
10
11
12
generalisation. Here,
13
14
15
Introduction
16
17
The re-emergence of deep learning during the last decade due to the noteworthy achievements of artificial neural networks (ANN) built up sort of philosophy that nearly anything could be automatically learnt from data without human intervention, in contrast to the previous approaches:
18
19
20
[hand designing good feature extractors, engineering skill and domain expertise] can all be avoided if good features can be learned automatically using a general- purpose learning procedure. This is the key advantage of deep learning (LeCun) et al. 2015).
21
22
23
Read in hindsight, this claim clearly an overstatement. The success of deep learning has required iterative hand design of network architectures and techniques that demanded collective, high engineering skill and large doses of interdisciplinary domain expertise. Furthermore, deep learning
25
27 owes much to the immense computational power poured into training artificial networks (Amodei & Hernandez 2018; Schwartz al. 2019) and to the human effort of manually collecting and labelling thousands of images and other data modalities (Russakovsky et al. 2015 Cao et al.) 2018). However, the gist of the claim has permeated machine learning research and is pervasive up these days.
29
30
31
The realisation that the success of deep learning was largely due the availability of huge labelled data sets prompted various reactions: some authors strongly questioned the usefulness of the algorithms (aarcss 2018); some delved into the question of whether neural networks generalise beyond or simply memorise the training examples (Zhang et al. 2017 Arpit et al. 2017); and some proposed
33
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

35
In this paper, we reflect upon fundamental concepts in machine learning such as supervision, in- ductive biases and generalisation, which in spite of resting on theoretical grounds, are at the core of misconceptions and overstatements about deep learning commonly seen in the literature. First, we review aspects from biological learning, and compare them to the traits often (mis jattribbrb to human learning and generalisation in the machine learning literature (Section 2}. Second, we revisit insights from classical statistical learning and critically review the terminology and current trends in new research horizons that can be overly ambitious and potentially misleading: "learning class from a single labelled example", based on the statement that "humans learn new concepts with very little supervision, [but] the standard supervised deep learning paradigm does not offer a satisfactory solution for learning new concepts rapidly from little data" (Vinyals et al.) (0T6). As a consequence, multiple research programmes, with various brand names, followed up with the aim of minimising or removing the need for "supervision" " to train neural networks: few-shot, one-shot, zero-shot, predictive, unsupervised, semi-supervised and self-supervised learning are only few popular examples.
36
37
38
39
40
41
42
Exploring alternatives to classification and improving the efficiency of learning algorithms should indeed be a priority of machine learning research. As matter of fact, related approaches have been subject of study since long before the explosion of deep learning (Hinton & Sejnowski] 1999) Chapelle et al.] 2006). However, the current publication and discussion trends in the field denote overambitious promises that are in part based on misconceptions and overstatements about biological learning, and amplified by overselling nomenclature. While much of the research output derived from these programmes does provide us with useful techniques and insight, it leaves behind a landscape of confusing terminology and tangled research directions that are hard to navigate and lead many astray,
43
44
45
46
47
48
19
50
51
52
53
54
55
56 deep learning research (Section 3). Altogether, we aim at tempering certain claims and promises of deep learning, helping mitigate the confusion over the terminology and suggesting desirable- our opinion-direction directions and changes in machine learning research.
57
58
59
2
Supervision in biological learning
60
The link between artificial intelligence-speciieee
artificial neural networks (Rosenblatt, 1958]
61
Fukushima & Miyake 1982) and biological learning systems is intrinsic to the field, as one long- term goal of artificial intelligence is to mirror the capabilities of human intelligence. However, these capabilities are, in our view, often overestimated. One example is the argument that intelligence in generalisation. In
62
63
64
nature evolves without supervision and is capable of robust out-of-distributin particular, it is often claimed that humans and other animals learn to visually categorise objects with little or no supervision from a few examples (Vinyals et al. 2016; Marcus 2018 Morgenstern et al., 2019). In what follows, we will discuss three aspects of biological learning to argue against this view, so as to gain insights that better inform our progress in machine learning: first, we will discuss how generalisation requires exposure to relevant training data; second, we will review the variety of supervised signals that the brain has access to; third, we will comment on the role of evolution and brain development.
66
68
70
71
72
2.1
Generalisation requires exposure to relevant training data
73 In the argument that machine learning models should generalise from a few examples, there seems
74 to be promise or aspiration that future better methods will be able to perform robust visual object
75 categorisation-for instance-amon many object classes after being trained on one or a few examples
76 per class. While a primary objective is to develop techniques that efficiently extract the maximum
77 possible information from the available examples, we should also remind ourselves that no machine
78 learning algorithm can robustly learn anything that cannot be inferred from the data it has been trained
Although this may seem to contradict certain current trends and statements in the literature, should also bear in mind that learning in nature is not different.
80
81 First, the amount of data that animals and humans in particular are exposed to is often underestimated.
82 biological brain continuously receives, processes and integrates multimodal inputs from various
83 sensors immges (light), sound, smell, etc. Humans do not learn to recognise objects by looking
84 at photos from ImageNet, but are rather exposed to a continuous flow of visual stimuli with slow changes of the viewing angle and lighting conditions. Furthermore, the stimuli are coherent across 86 modalities, we are allowed to interact with the objects and we even receive multiple supervision signals, as we discuss later.

---

88
The exposure to so much training data makes the human visual system remarkably robust, but still its capabilities are optimised for the tasks it needs to perform and largely determined by the training data distribution and years of evolution, as we will discuss below. For instance, well-studied property of human vision is that our face recognition ability is severely impaired if faces are presented upside down Yin 1969, Valentine| 1988). Setting aside the specific complexity of face processing in the brain, a compelling explanation for this impairment is that we are simply not used to seeing and recognising inverted faces. More generally, while human perception of objects is largely invariant under certain conditions (Biederman & Bar 1999), object recognition is sensitive to changes in view angle TTrr et al., 1998), especially when see objects from unfamiliar viewpoints (Edelman & Bülthoft 1992; Bülthoff& & Newell 2006 Milivojevic 2012).
89
90
91
92
93
94
95
97
98
Furthermore, although better than the one-shot or few-shor generalisation of current ANNs, humans also have limited ability to recognise truly novel classes (Morgenstern et al.) 2019). Interestingly, experiments with certain novel classes of objects known as Greebles showed that, with sufficient training, humans can acquire expertise in recognising new objects from different viewpoints, even making use of area of the brain-the the fusiform face area- that typically responds strongly with face stimuli (Gauthier et al. 1999). This provides evidence that recognition from multiple viewpoints is possible but only developed after exposure to similar conditions, that is relevant data. This is reminiscent of the effectiveness of data augmentation in deep learning, compared to more naive regularisation methods Glemández--Grríi & König 2018).
99
100
101
102
103
105
106
107
The need for exposure to relevant stimuli challenges the notion that humans are capable of strong out-of-distrin -istriiibittnnnnnnnn generalisation. Rather, it seems that the transfer learning capabilities of humans are limited to relatively small changes in the data distribution. A compelling example is our difficulty
108
109
110
to learn new languages: someone who natively speaks or has learnt Spanish will be able to transfer a significant amount of knowledge if they are to learn Italian, due to the overlap in the data distribution, but they will have very little to transfer for learning Kanien kéha or Mandarin.
111
112
115
2.2
Supervised signals for the brain
114
in general-leand -earn robust object recognition without supervision: "a child can generalize the concept of "giraffe' from a single picture in book" (Vinyals et al. 2016). First of all, we should mention the role of cvolution (expanded in Section 2.3). which can be interpreted as a pre-trained model, optimised through millions of years of
Another commonly found argument has it that
115
116
117
118
data with natural selection as a supervisory signal (Zador 2019). Second, there is abundant evidence to argue against the very claim that children and adults learn in fully unsupervised fashion.
119
120
Obviously, the kind of supervision that humans make use of is not that of classification algorithms-
121
we do not see class label on top of every object we look at. However, we receive supervision from multiple sources. Even though not for every visual stimulus, children do frequently receive information about the object classes they see. For instance, parents would point at objects and name
122
123
124
them, then we learn how to read, and generally play crucial role as teachers in language development Kuhl 2007). Non-human animals such as zebra finches learning to sing have also been found rely on feedback (supervision) from the female adult and not just imitation CCarouso-Peck & Goldstein 2019). Furthermore, humans usually follow guided hierarchical learning: children do not directly leann to tell apart breeds of dogs, but rather start with umbrella terms and then progressively learn down the class hierarchy (Bornstein & Arterberry 2010 Spprie et al., (02T). Gopnik (202T) has asserted that "we learn more from other people than we do from any other source" and Hasson et al. 2020) mention other examples of supervision from social cues, that is from other humans, such as learning to recognise individual faces, produce grammatical sentences, read and write; as well as from embodiment and action, such as learning to balance the body while walking or grasping objects.
125
128
127
128
129
130
131
132
133
In all these actions, we can identify supervisory signal that surely influences learning in the brain Shapiro 2012 Gopnik et al. 2020).
135
136
While these supervision signals largely differ from what is most commonly considered supervised learning in machine learning, we can still draw some parallels with human learning. We learn to categorise many concepts and objects as children, but most people carry on learning new categories as adults. For example, some people put effort in improving their understanding of the natural world by learning to recognise and name trees, plants or birds. Those who have engaged in such an endeavour may have noticed that the learning process is easier and faster if we count upon the expert knowledge
137
138
139
141
of a friend or of technology such as iNaturalist (Van Horn et al. 2018). Another example: those
3

---

143
who have- or attempted to learn- a new language as an adult may have realised that whereas it is possible learn the meaning of new word by repeated exposure it in multiple contexts, it is certainly easier if we look up the ground truth definition in dictionary or, even easier, if there exists a direct mapping to a word in our native language. Summing up, not only does supervision facilitate learning, but human beings actively seek for it.
144
145
146
147
148
Besides this kind of explicit supervision, the brain certainly makes use of more subtle, implicit supervised signals, such as temporal stability (Becker 1999. Wyss et al. 2003): The light that enters the retina, and the sound waves that reach the cochlea, are not random signals from a sequence of rapidly changing arbitrary photos or noise, but highly coherent and regular flows of slowly changing stimuli, especially the higher, semantical level (Kording al.) (004). At the very least, this is how
149
150
151
152
153
we perceive it and if such a smooth perception turns out to be consequence rather than a cause, then should be a by-product of a long process of evolution that would be worth taking into account.
154
155
2.3
The role of evolution and brain development
156
In the previous sections, we have discussed some misconceptions or overstatements about how humans learn and generalise that are often found in the machine literature. Namely, that humans are able to generalise from a few examples and that this occurs with little or no supervision. Still, the commonplace comparison of artificial neural networks with human learning and the brain often misses a fundamental component of biology, recently brought to the fore by Zador (2019) and Hasson et al. (2020), although considered since the early days of artificial intelligence (Turing 1968): the
158
159
160
role that millions of years of evolution have played in developing the nervous systems of organisms in nature, including the human brain.
162
163
164
The most common way of training artificial neural networks, especially in machine learning research, is from tabula rasa, that is from randomly initialised parameter:" In contrast, large part of the brain connectivity is encoded genetically and certain properties and behaviour are known to be innate, that is developed without prior exposure stimuli (Farroni et al. 2005; Spriet et al. 2021). Importantly, evolution not only provides innate behaviour, but also determines what cannot be learnt, or relevant constraints scientists who have trained animals in the laboratory for psychological or neuroscientific studies are well aware that tasks have to be carefully adapted to the ecological behaviour and limitations of the animal, determined by evolution.
166
167
168
170
171
172 Taking into account the role of evolution, we can draw conclusions that relate to the claims discussed in the previous sections. If our brains are the product of millions of years of exposure to relevant stimuli and adaptation, is it really fair to say that humans are capable of robust out-of--istribution generalisation and that we learn from from a few examples? If evolution has largely determined what our brain can and cannot learn, providing as with a "pre-trained model", is it really fair to say that humans learn in unsupervised fashion? This questions are relevant for machine learning research: if we take biological learning as motivation for artificial intelligence, should we not temper expectations of what learning algorithms should aspire to? And, therefore, would it not be worth reconsidering some research programmes?
174
176
178
179
180
On the flip side, insights from evolutionary theory are likely to be fruitful source of inspiration for machine learning (Hasson et al., 2020 [aaor 2019). As we have observed, training neural network from scratch may be more similar to a simulation of evolution than to the process by which an adult learns a new concept. As a shortcut to simulating evolution, neuroscience is a rich source of inspiration of constraints and inductive biases that determine learning in the biological brain and can potentially inform machine learning (Hassabis et al.) 2017 Lindsey et al. 2019). For instance, simulating properties of the primary visual cortex in the early layers of an artificial neural network
182
184
186
187
has been shown to improve adversarial robustness (Dapello et al. 2020 Malhotra et al.) 2020).
188
189
Besides evolution, the focus on the capabilities of adults often makes us miss another important aspect of biological learning, particularly important in humans: the role of learning in infancy and brain development. While learning occurs too in adulthood, childhood is a particularly important and
190
Some interesting and promising areas in machine learning research deviate from this standard approach. For example, transfer learning and domain adaptation study the potential of features learnt on one task to be reused in different, related tasks (Zhuagg et al. (011)), and continual learning studies the ways in which machine learning models can indetinitely sustain the acquisition of new knowledge without detriment of the previously learnt tasks (Mundt al. 2020). These approaches are inspired by biological learning or share interesting properties with it.

---

192
active time for learning (Atkinson 2002 Gelman & Meyer 2011). In fact, sensitive or critical periods for learning infancy have been described or hypothesised, for example for vision (Harwerth et al.) 1986) and language development (Lenneberg 1967). Machine learning papers that draw motivation
193
194
195
from the alleged generalisation capabilities of humans often underestimate the amount of input stimuli and supervision that infants receive (Gopnik (020). However, childhood can be regarded as period dedicated almost exclusively to learn, not only formally from parents and teachers, but also through playing, which plays critical role in cognitive development Burghardi (2005); Pelz
196
197
(Pelz & Kidd (2020). Finally, the fact that humans-ann and other cognitively advanced animals, such as corvid birds, which also exhibit cultural learning havein comparatively long childhood period, has led Uomini et al. (020) to recently proposed that extended parenting is pivotal in the evolution of cognition. This adds to the discussion on the undervalued role of supervision. In sum, we propose machine learning research can benefit from drawing inspiration from both evolutionary biology and the literature on developmental psychology, brain development and life history and learning (Gopnik et al. 2020).
199
201
202
203
204
205
3
Supervision in machine learning
206
If we open a machine learning textbook (Murphy) 2012 Abu- Abu-Mostafa et al., 2012 Goodfellow et al. 2016), we will most surely find taxonomy of learning algorithms with clear distinction between supervised and unsupervised learning. However, while this separation can be useful, the boundaries certainly not clear. As matter of fact, if we take a look at the deep learning literature of the past years, we will also find abundant work on some variants supposedly in between-ssemissupere learning, self-supervised learning, whose definitions are all but clear.
208
209
210
212
3.1
Catastrophic forgetting of old concepts
213
If we recall a classical result in statistical learning theory and inference, the no free lunch theorem
214
(Wolpert 1996). no learning algorithm is better than any other at classifying unobserved data points, when averaged over all possible data distributions. Therefore, we need to constrain the distributions
215
216
or, in other words, introduce prior knowledge that is supervision. Recently, Locatello et al. (2018)
217
obtained a related result for the case of unsupervised learning of disentangled representations: without
inductive biases for both the models and the data sets, unsupervised disentanglement learning is fundamentally impossible. These results are purely theoretical and have limited impact on real world applications (Giraud-Carrier & Provost] 2005). precisely because in practice we use multiple inductive biases and implicit supervision, even when we do so-called unsupervised learning.
218
219
221
222
In a strict sense, even the classical, purely unsupervised methods, such as independent component analysis or nearest neighbours classifiers, make use of inductive biases, such as independence or minimum distance, respectively. Without inductive bias, learning is not possible: purely unsupervised
223
224
225
learning an illusion. While this is not news, the terminology used in the recent and current machine learning literature seems to reject supervision and neglect these nuances, evidencing that the field suffers catastrophic forgetting of eellbestabl notions.
226
227
228
3.2
The brands of alt-supervised learning
229
Particularly in deep learning and computer vision, the term supervised learning has adopted, in practice, the meaning of classification of examples annotated by humans, that is models trained on examples labelled according to, for instance, the object classes. This is yet another instance of catastrophic forgetting-
231
at best, abuse-or of well-established concepts. It should not be necessary to recall that, first of all, supervised learning is a broader category than classification, which includes also regression and ranking, among other learning modalities. Second, even if narrow view to classification only, supervised learning is not restricted to learning from examples annotated by humans. Goodfellow et al. (2016) did not overlook this in their definition of supervised learning: "In many cases the outputs y may be difficult to collect automatically and must be provided by a human 'supervisor," but the term still applies even when the training set targets were collected automatically".
233
235
237
238
239 In turn, the term unsupervised learning is now used for any model that does not use manually collected labels, regardless of what other kind of supervision it may use. Further, the term semi-supervised learning generally refers practice to models that are trained with a fraction of the labels, but are tested on the same classification benchmarks. Finally, the term sulf-vuee learning has recently
241
5

---

243
gained much popularity, referring to models that are trained on tasks other than the standard task defined by classification labels.
244
245
Some of the methods proposed under these categories are certainly useful-that is not the subject of criticism of this work-but the terminology is overly confusing and unnecessary. A newcomer would
246
247
casily fall into scientific rabbit hole trying to discern the meaning of each of these names through publications-not to mention if they incorporated social media discussions into their endeavour. By way of illustration, the authors of this paper have witnessed how a recurrent question by students who learn about recent deep learning methods is whether there is any difference between ceffesupereieed and unsupervised learning. Are students missing something fundamental? The following anecdotal recall of influential keynote talks at artificial intelligence conferences should shed some light on part of the origins of this confusion: In December 2016, Prof. Yann LeCun titled his NeurIPS keynote
248
249
250
251
252
253
254
presentation "Predictive Learning", to refer to "what many people mean by unsupervised learning" [eeCun] 2016). A few years later, in his keynote presentation at ISSCC in February 2019, he spoke about similar ideas, but this time the title was "Self- SeefrvSupe Learning" (LeCun 2019). In social media, he wrote: "I now call it 'self-supervised learning', because "unsupervised" is both a loaded and confusing term". Students may be getting things rather right.
255
256
258
259
Is there then a fundamental difference-a theoretically grounded one-between the deep learning methods labelled as unsupervised learning and more recently self-supervised learning? We argue that these are mostly brand names that reflect trends in the field, adding noise to the scientific progress and leading many astray. Therefore, we propose that, given the recent progress, the field of machine learning research would benefit from an exercise of self-reflection and from an effort to devise rigorous taxonomy of the variety of methods. From a theoretical point of view, both the conventional classification models and the recent wave of self-supervised tasks can all be formalised as sub-categories of supervised learning.
260
261
262
263
264
265
266
267
3.3
Supervision comes in different flavours
268
In Section 2.2, we have seen examples of different forms of supervision used by humans and other animals. In machine learning, the field focused for many years on a few loss functions, such as classification and simple forms of regression. The relatively recent explosion of deep learning has
289
270
brought about the development of several libraries for automatic differentiation (Baydin et al. 2017). which in turn have enabled the proposal of multiple loss functions and learning tasks with various types of supervision that can easily be optimised numerically by stochastic gradient descent and artificial neural networks. This has certainly opened promising and already fruitful avenues to
271
272
273
274
275
incorporate richer forms of supervision and inductive biases other than classification, some inspired by biological learning, into machine learning algorithms.
276
277 A currently popular example is image data augmentation: Although until recently it was seen as a naive technique to simply create additional training data, data augmentation actually encodes rich prior knowledge about human visual perception. in the case of computer vision. This is why it outperforms explicit regularisation methods, which provide less effective inductive biases Giernnn &
278
279
280
281
König 20188, and was used in "semi-supervised" tasks Laine & Aila (2016). The rich information embedded in image transformations has been used to encourage invariant outputs under different
282
283
augmentations through contrastive losses (Ye et al. 2019), and even at intermediate representations, inspired by the invariance in the visual cortex (Hernández- Garcia et al.. [201),, although these methods were not branded self-supervision. The use of this term for losses based on data augmentation was further popularised after the success of similar methods such as SimCLR (Chen et al.) 2020). Beyond data augmentation invariance, the zoo of self-supervised learning tasks in computer vision is rich and diverse: classifying the rotation applied to image patches (Gidaris et al. 2018)) predicting image colourisation (Larsson et al. 2017). classifying the relative position of two image patches Doersch et al,, 2015), or even solving full jigsaw puzzles (Noroozi & Favaro 2016) (Jing & Tian 2020) recently performed an extensive review).
284
286
286
290
292
The current trend is to refer to these methods as self-supervised learning, but similar methods were referred to in the past as semi-supervised unsupervised, and even predictive learning, as we have seen, look at the papers reveals that these terms have been used mostly interchangeably. The terms self- and semi- and unsupervised learning imply that lexs supervision is used, but it would be misleading to seriously argue that the tasks are devoid of supervision. Most of these techniques make use of a wide range of surrogate tasks with supervisory signals defined by humans. In fact, they could have been
293
294
296

---

298
called hyper-superrised learning. Here, we contend that these methods are all variants of supervised learning, only that supervision comes in different flavours, both in biological and machine learning. and we should call it by its name and ideally develop a rigorous taxonomy.
299
300
4
Discussion
301
302
In this paper, we have discussed some of the overambitious promises of the deep learning hype, namely that machine learning models should be able to generalise to unseen distributions, from a few examples, without human intervention or supervision. These claims have often been motivated by alleged generalisation capabilities of humans. In order to assess these motivations, we have first reviewed, in [Section 2] some often overlooked characteristics of biological learning relevant to machine learning research. In particular, we have argued that humans and other animals receive extensive and diverse input stimuli well as multiple supervisory signals, including the long history of evolution and cultural transmission. In the light of these insights from biological learning, we have then, in Section 3 critically reviewed the various terms that currently used to refer to supposed alternatives to supervised learning: semi-, self- and unsupervised learning, among others. In sum, we pointed out that all these approaches are in fact supervised learning- classification- and the machine learning (research) community would benefit from using more rigorous, less overselling nomenclature, and from devising more rigorous taxonomy.
303
304
305
307
309
311
312
not necessarily
313
314
315
Supervision is not evil. It is at the core of statistical learning theory: learning is impossible without inductive biases or supervision. But supervision comes in different flavours, not only as classification labels. Neither is deep learning some sort of exceptional solution to learn without human intervention and supervision, nor is it a hopeless model class because it requires large data sets (Marcus 2018). The human visual system is exposed to a lot of stimuli too. One exceptional advantage of deep learning is precisely that it is possible to effectively optimise different learning objectives, almost end-to-end from large collections of nearly naturalistic sensory signals, such as digital images (Saxe) al. 2020). While other models are known to scale poorly the amount of data increases, neural networks excel at fitting the training data and interpolating on unseen examples (Belkin et al.) 2019)]
317
319
321
322
323
324
Hasson et al., 2020). This is feature, not a bug. But will make better progress if we exploit these advantages of deep learning without neglecting that supervision will always be necessary- the critical goal is how to best incorporate it and exploit it.
325
326
327 In this regard, we argue that deep learning needs more supervision. and not less. A major focus of the deep learning community in the last decade has been image object classification. This has brought about unprecedented progress and unveiled the limitations of having classification as chief
329
330
task and class labels as main supervisory signal. For example, deep classifiers have been found to learn spurious features that are highly discriminative for the classification task but with little true generalisation power and clearly not aligned with perceptual features (Jo & Bengio 2017 Wang et al.,
331
332
333
2019; Geirhos et al.] 2020). In fact, this mismatch has been argued to be at the root of adversarial vulnerability (Iyya et al. 2019) and seems to be the consequence of training highly expressive, over-parameterissd models in heavily unconstrained tasks. This can be addressed with meaningful constraints, that is more and richer supervision, possibly inspired by human perception and biological learning. For example. combining a classification loss with a similarity loss inspired by the invariance in the visual cortex yields more robust representations without detriment to categorisation (Hernández) Garcia et al. 2019), and simulating the properties of the primary visual cortex may improve the adversarial robustness of neural networks (Dapello et al. 2020). Expanding in this direction leads to pelloiccally-inspreede multi-task and representation learning, and away from just classification.
334
335
336
337
338
339
340
341
5
Conclusions for future research directions
342
343
The chief goal of this paper is rather descriptive than prescriptive. We have aimed to identify and describe aspects of the current trends in machine learning research that could be improved, in the hope of inspiring future work that effectively address them. Nonetheless, throughout the paper we have made suggestions that may help mitigate the confusion with the terminology, clarify research directions and ultimately bring about scientific progress in machine learning research. We outline these suggestions here to conclude the paper.
344
345
346
347
348
The authors explicitly discourage the addition of new term the already confusing list.
7

---

349
We have drawn parallels from cognitive neuroscience to contend that learning in nature also requires abundant data and supervision in multiple forms. Even evolution can be regarded as an optimisation process where natural selection is the supervisory signal. We have argued, as others have before, that these insights from biology, neuroscience and developmental psychology, among other fields, offer a great opportunity for machine learning research to draw inspiration and calibrate its compass.
350
352
353
354 As we have discussed, research in deep learning has departed from pure classification and has been exploring new learning tasks and ways of training artificial neural networks. Nonetheless, in some
355
356
fields such as computer vision, the ultimate benchmark to assess the value method is still the accuracy on classification data sets, such as ImageNet, even though there is evidence of overfitting
357
358
the test set. While object recognition will remain an important benchmark, as deep learning is well suited to learn representations, we should develop methods to assess the quality of the learnt
359
360
representations for tasks other than classification. In this regard, we encourage researchers to evaluate their models with tests that are still not widespread, such as the suitability for transfer learning, adversarial robustness, comparison with brain measurements, behavioural tasks, etc.
362
385
We have also argued that the field would benefit from an effort to devise a rigorous taxonomy of learning methods that sheds light on the ocean of methods proposed in the past years. The terms self-, semi- and wnsupervised learning have been used interchangeably and this is often a source of confusion for students and newcomers. While confusing terminology natural in a rapidly growing,
364
385
366
387
the time might have come for distilling the progress of the past years into rigorous nomenclature that better survive the test of time.
388
369
Finally, we recall that most of the learning theory has been developed for simple loss functions such as binary classification mean squared error regression, but certain methods successfully used in practice today escape the available theory. Given the success of this kind of more complex supervised objectives, the study of these methods from theoretical point of view might be fruitful direction for future work.
370
371
372
373
Broader Impact
375
Since this article does not present a new method or results from data sets, potential risks of "bias in the data" or "failure of the system" do not apply. As critical review of current trends in the field and cite multiple research articles, some researchers could potentially feel addressed and affected by our mentions. We declare that we do not intend to negatively affect any individual researcher and we have only referred to individuals directly in the case of well-established scientist with a reputation. Our goal has been in any case to potentially improve scientific progress through a constructive reflection.
376
377
378
379
380
References
381
382 Abu-Mostafa, Y. S., Magdon-Ismail, M., and Lin, H.-T. Learning from data. AMLBooks, 2012.
Amodei, D. and Hernandez. D. Al and compute. Accessed: 2020-09-28, 2018 URL https
384
//openai... com hanm//log/aaa ai - and compute/
Arpit, D. et al. A closer look at memorization in deep networks. arXiv preprint 2017.
386
Atkinson, J. The developing visual brain. Oxford Scholarship Online, 2002.
388
Baydin, A. G., Pearlmutter, B. A., Radul, A. A., and Siskind, J. M. Automatic differentiation in machine learning: survey. Journal of Machine Learning Research (JMLR), 2017.
390
Becker, S. Implicit learning in 3D object recognition: The importance of temporal context. Neural
Computation, 1999.
382
Belkin, M., Hsu, D., Ma, S., and Mandal, Reconciling modern machine-learning practice and the classical varinvar trade-off. Proceedings of the National Academy of Sciences (PNAS), 2019.
394
Biederman, and Bar, M. One-shot viewpoint invariance in matching novel objects. Vision Research, 1999.
8

---

396
Bornstein, M. H. and Arterberry, M. E. The development of object categorization in young chil- dren: Hierarchical inclusiveness, age, perceptual attribute, and group versus individual analyses. Developmental Psychology, 2010.
397
Bülthoff, 1. and Newell, F. N. The role of familiarity in the recognition of static and dynamic objects. Progress in Brain Research, 2006.
400
401
Burghardt, G. M. The genesis of animal play: Testing the limits. Mit Press, 2005.
402
Cao, Q. Shen, L.,, Xie, W., Parkhi, O. M., and Zisserman, A. VGGFace2: A dataset for recognising faces across pose and age. In IEEE International Conference Automatic Face & Gesture Recognition. 2018.
404
405
Carouso-Peck. S. and Goldstein, M. H. Female social feedback reveals non-imitative mechanisms of vocal learning in zebra finches. Current Biology. 2019.
408
407
Chapelle, O., Schölkopf, B., and Zien, A. (eds.). Gemi-supervised learning. MIT Press, 2006.
408
Chen, T., Komblithh, S., Norouzi, M., and Hinton, G. simple framework for contrastive learning of visual representations. In International conference machine learning. 2020.
Dapello, J., Marques, T., Schrimpf, M., Geiger, F., Cox, D. D., and DiCarlo. J. Simulating primary visual cortex at the front of cnns improves robustness to image perturbations. BioRxiv, 2020.
411
Doersch, C., Gupta, A., and Efros, A. A. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, 2015.
414
415
Edelman, S. and Bülthoff, H. H. Orientation dependence in the recognition of familiar and novel views of dimensionan objects. Vision Research, 1992.
416
417
Farroni, T., Johnson, M. H., Menon, E., Zulian, L.,, Faraguna, D., and Csibra, G. Newborns' preference for face-relevant stimuli: Effects of contrast polarity. Proceedings of the National Academy of Sciences, 2005.
419
420
Fukushima, K. and Miyake, S. Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position. Pattern Recognition, 1982.
Gauthier, I., Tarr, M. J., Anderson, A. W., Skudlarski, P., and Gore, J. C. Activation of the middle fusiform' face area increases with expertise in recognizing novel objects. Nature Neuroscience, 1999.
423
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F, A. Shortcut learning in deep neural networks. arXiv preprint arXiv:2004.07780, 2020.
428
427
Gelman, S. A. and Meyer, M. Child categorization. Wiley Interdisciplinary Reviews: Cognitive
428
Science, 2011.
429
Gidaris, S., Singh, P., and Komodakis, Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1rXXv:1803.07788, 2018.
Giraud-Carrier C. and Provost. F. Toward justification of meta-learning Is the free lunch thcorem a show-stopper. In Proceedings of the ICML-2005 Workshop on Meta-learning, 2005.
432
433
Goodfellow, I., Bengio, Y., and Courville, A. Deep learning. MIT Press, 2016.
434
Gopnik, A. Childhood as a solution to explore-exploit tensions. Philosophical Transactions of the Royal Society B, 2020.
435
436
Gopnik, Brain podast, by Paul Middle Middlebrooks. BI 094 Alison Gopnik: Child- Inspired AI. https://braininspiiree cttpp//aeeecc/popccs//////////// 2021.
Gopnik, A., Frankenhuis, W. E., and Tomasello, M. Introduction to special issue: life history and learning: how childhood, caregiving and old age shape cognition and culture in humans and other animals'. 2020.
439

---

Harwerth, R. S., Smith, E. L., Duncan, G. C.. Crawford, M., and Von Noorden, G. K. Multiple sensitive periods in the development of the primate visual system. Science, 1986.
442
443
Hassabis, D., Kumaran, D.. Summerfield, C., and Botvinick, M. Neuroscience-inspired artificial
intelligence. Neuron, 2017.
445
Hasson, U., Nastase, S. A., and Goldstein, A. Direct fit to nature: An evolutionary perspective on biological and artificial neural networks. Neuron, 2020.
447
Garian, A. and König, P. Data augmentation instead of explicit regularization. arXiv preprint www.iv:/00600000000 2018.
448
Hernández- A., König, P., and Kietzmann, 'T. Learning robust visual representations using
450
data augmentation invariance. arXiv preprint (rXiv:1906.045:0 2019.
Hinton, G. and Sejnowski, T. (eds.). Unsupervised learning. MIT Press, 1999.
Hyas, A., Santurkar, S., Tsipras, D., Engstrom, L.., Tran, B., and Madry, A. Adversarial examples not bugs, they are features. arXiv preprint arXiv:1905.02175. 2019.
453
Jing, L. and Tian, Y. supe-vupd visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020.
455
456
Jo, J. and Bengio, Y. Measuring the tendency of CNNs to learn surface statistical regularities. arXiv preprint ar 1rXiv:1111.11561, 2017.
457
458
Kording, K. P., Kayser, C.,, Einhauser, W., and Konig, P. How are complex cell properties adapted to the statisties of natural stimuli? Journal of Neurophysiology 2004.
459
460
Kuhl, K. Is speech learning 'gated'by the social brain? Developmental science, 2007.
461
Laine, S. and Aila, T. Temporal ensembling for semi-supervised learning. arXiv preprint 1rXiv:1610.0224. 2016.
462
463
Larsson, G., Maire, M., and Shakhnarovich, G. Colorization as a proxy task for visual understanding.
In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2017.
465 1 Cun, Predictive learning, December 2016. URL htttt:tt///pppttt http:///ipps.ccc/offeeeeeeeeee www.c//eeeeenecccee/ccecoo 2016/ ScheduleMin wwt.eeeeeuttttctcteooe wttplttttttacc/eettill... Invited talk at the Thirtieth Conference on Neural Informa- tion Processing Systems (NeurIPS) 2016. Slide 24, starting on minute 14:34.
467
468
Cun, Y. Deep learning hardware: Past, present, and future, February 2019. httphtts::/ youtube.
469
Invited talk at the International Solid-State Circuits Conference (ISSCC) 2019. Slide 57, starting on minute 27:24.
470
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. Nature, 2015.
472
Lenneberg, E. H. The biological foundations of language. Hospital Practice, 1967.
473
Lindsey, J., Ocko, S. A., Ganguli, S., and Deny, S. A unified theory of early visual representations from retina cortex through anatomically constrained deep CNNs. In International Conference on Learning Representations (ICLR), 1901.09901.00045 2019.
474
475
Locatello, F., Bauer, S., Lucic, M., Rätsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. arXiv preprint 1rXiv:1811.12359, 2018.
477
2018.
479
Malhotra, G., Evans, D., and Bowers, Hiding plane with a pixel: examining shape-bias in CNNs and the benefit of building in biological constraints. Vision Research, 2020.
Mareus, G. Deep learning: A critical appraisal. arXiv preprint arXiv: 2018.
482
Milivojevic, B. Object recognition can be viewpoint dependent or invariant-it's just a matter of time and task. Frontiers in Computational Neuroscience, 2012.
10

---

484
Morgenstern, Y., Schmidt, F., and Fleming, R. W. One-shot categorization of novel object classes in
485
humans, Vision Research, 2019.
486
Mundt, M., Hong, Y. W., Pliushch, I., and Ramesh, V. A wholistic view of continual learning with
487
deep neural networks: Forgotten lessons and the bridge active and open world learning. arXiv preprint urXiv2209.00799. 2020.
488
Murphy, K. P. Machine learning: probabillistic perspective. MIT Press, 2012.
490
Noroozi, M. and Favaro, P. Unsupervised learning of visual representations by solving jigsaw puzzles.
In European conference on computer vision. 2016.
Pelz, M. and Kidd, C. The elaboration of exploratory play. Philosophical Transactions of the Royal Society B, 2020.
493
Rosenblatt, F. The perceptron: probabilistic model for information storage and organization in the brain. Psychological Review, 1958.
495
496
Russakovsky, O. et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 2015.
497
498
Saxe, A., Nelli, S., and Summerfield, C. If deep learning is the answer, then what is the question? arXiv preprint ar arXiv:2004.07580, 2020.
500
Schwartz, R., Dodge. J., Smith, N. A., and Etzioni, O. Green ai. arXiv preprint arXiv:1907.100.
2019.
502
Shapiro, L. Embodied cognition. Oxford Handbooks Online, 2012.
503
Spriet, C., Abassi, E., Hochmann, J.-R., and Papeo, L. Visual object categorization in infancy.
bioRxiv, 2021.
505
Tarr, M. J., Williams, P., Hayward, W. G., and Gauthier, Three-dimensional object recognition is viewpoint dependent. Nature Neuroscience, 1998.
506
Turing, A. M. Cybernetics; (Key papers). University Park Press, 1968.
506
Uomini, N., Fairlie, J., Gray, R. D., and Griesser, M. Extended parenting and the evolution of cognition. Philosophical Transactions of the Royal Society B, 2020.
510
Valentine, T. Upside-down faces: A review of the effect of inversion upon face recognition. British
511
Journal of Psychology, 1988.
Van Horn, G., Mac Aodha, O., Song. Y., Cui, Y., Sun, C.,, Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, 2018.
513
515
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Wang, H., Wu, X., Yin, P., and Xing, E. High frequency component helps explain the generalization of convolutional neural networks. arXiv preprint arXiv: 13545, 2019.
518
Wolpert, D. H. The lack of a priori distinctions between learning algorithms. Neural Computation, 1996.
520
521
Wyss, R., König, P., and Verschure, P. F. Invariant representations of visual patterns in a temporal
522
population code. Proceedings of the National Academy of Sciences (PNAS), 2003.
523
Ye, M., Zhang. X., Yuen, P. C., and Chang, S.-F. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2019.
525
Yin, R. K. Looking upside-down faces. Journal of Experimental Psychology, 1969.
11

---

527
Zador, A. M. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature Communications, 2019.
528
529
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires
rethinking generalization. In International Conference Learning Representations (ICLR), arXiv:1611.03530 2017.
531
532
Zhuang, F., Qi, Z., Duan, K., Xi, Zhu, Y., Zhu, H., Xiong, H., and He, Q. A comprehensive survey transfer learning. arXiv preprint arXiv: 1.02685, 2019.
Checklist
535
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
537
538
(b) Did you describe the limitations of your work? [Yes] See the beginning of Seetion 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
540
Section 5]
541
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
544
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
546
3. If you ran experiments..
547
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [N/A]
519
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [N/A]
550
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [N/A]
$52
559
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
556
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
559
580
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
561
562
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
564
5. If you used crowdsourcing conducted research with human subjects...
565
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
567
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
570
12


Reviews:
[{"review_id": "vRuoTCjj6g4", "reviewer": "Reviewer_pJTi", "summary": "This article is a position statement that aims to point out weaknesses in the \nauthors' conception of the ML field's agenda. As the authors state, the work is\ndescriptive, not prescriptive: it does not provide specific directions, but rather\npoints to trends that (in the extreme) are problematic.\nThe key points in the article include: (1) natural environments provide rich\nsupport for learning beyond traditional supervised classification labels; (2)\nevolutionary and developmental processes provide strong inductive biases that can \ninform model design; (3) the amount of data that natural learners have available\nis often underestimated (as is one-shot learning abilities); (4) there are many\nsignals in natural environments which can constrain learning beyond traditional\nclass labels.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work would be a provocative position paper for a workshop,\nbut I fail to see what research contribution it makes. The authors' caricature\nof the field's naivete does not match my impression of the field (having been\nan active researcher for over 30 years). They point to relevant psychological\nand neuroscientific literature (which the authors do know well) that the \nML field would do well to be more aware of. Much of the literature cited has\nbeen the subject of more cognitive science focused research at NeurIPS.\nLike the authors, I wish the field would focus more on cognitive inductive bias \nrather than building larger and larger systems with larger and larger data sets. \nBut the way to convince the field is not to lecture it, but to demonstrate that\nthe authors' insights into biology and cognition can be productively incorporated\ninto computational ML approaches. For example, in small data regimes, show that\ncognitive biases (whether in architecture, objective functions, or data sources\navailable for learning) allow for what would otherwise be unlearnable.\nThe article also points to some boastful and unfortunate quotes from prominent figures,\nbut to shoot down the field for a few quotes is unnecessary. What the authors need\nto make their argument convincing is specific examples of research done wrong and\nresearch done right, rather than speaking in generalities. I do not believe\nthat the article in its current form would perturb any researcher from their\ncurrent trajectory.\n\nThe article reads as a stream-of-consciousness series of thoughts and contains many\ndated and incorrect, or at least arguable, points, particularly with the authors' conception\nof the 'supervised' and 'unsupervised' labels.  Let me illustrate with just one \nparticularly frustrating section, Section 3.1.\n\n* The section is headed \"catastrophic forgetting\" yet the content of the section says\nnothing about this classic issue in neural nets.  Did the authors change their mind\nabout what content to include in the section?\n\n* The authors' take from the no-free-lunch theorem is incorrect: the fact that no \none model/architecture is superior to all others on expectation across data \ndistributions does not motivate supervised learning per se. It motivates choosing \ninductive bias appropriate for a task. (The theorem assumes a certain objective function,\nwhich could be either supervised or what is usually considered an unsupervised\ncriterion, such as cluster compactness.) \n\n* Any well-formulated unsupervised learning method requires an objective function  and optimization. The authors appear to consider this a form of 'implicit supervision'.   Unsupervised learning does not mean that the algorithm does something random. \nIt means that the objective does not require guidance concerning specific responses from a teacher.   And unsupervised learning does not mean that the model is without inductive bias. As the authors state, \"this is not news\", but they then go on to claim (without support) that the field is unaware of these facts.\n\n* Note that from a probabilistic perspective, unsupervised and supervised learning\nshare a common foundation. Rather than treating supervised and unsupervised\nlearning as two qualitatively different paradigms, every decent probabilistic ML textbook makes \nthe point that follows.  Unsupervised learning is about modeling a distribution \nover observed features.  When those features are images and class labels,\nlearning a joint distribution over the observations will permit estimation of\nconditional probabilities. Thus, from a probabilistic perspective, unsupervised\nand supervised learning aren't qualitatively different methods; supervised\nlearning is a special case in which a model must estimate the _conditional_\ndistribution P(output|input), whereas\nunsupervised learning does not split the data elements into 'input' and\n'output' and thus requires modeling the _joint_ distribution, from which any\nfeature can be predicted from any other. One reason why the probabilistic\nperspective is so popular in ML is because it provides this unified perspective.\n\n* The authors state that supervised learning goes beyond classification to include\nregression problems. Every textbook makes this point. Every ML course makes this\npoint.  The wikipedia 'supervised learning' page makes this point. I make this point in the first day when I teach ML. Again,\nthis is not news to anyone in the field. The authors may be expressing their frustration at\nthe predominance of research on classification, but there is no shortage of work on regression\nproblems at NeurIPS and other ML conferences. \n\n* The authors seem to have a somewhat dated conception of the field. Sure, in 2012\nimage classification was the focus of the groundbreaking research. \nOne reason why image classification continues to be so popular is that the image \ndata sets are well understood, and these image data sets provide sensible test beds for \nbetter understanding deep learning and exploring novel optimization methods, architectures, etc. However, the field is much \nricher than the authors allow. For example, the introduction of the transformer \narchitecture for language processing--as horribly non-cognitive as it is--has become a \nhuge industry in and of itself. Video understanding and interpretation (not just\nclassification) can be found at every recent ML conference. Work on video-text interactions\nis now moving to the foreground. The field is advancing and the problems researchers \nchoose to focus on changes over time.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "C5hUdgSiruh", "reviewer": "Reviewer_6UDa", "summary": "With this opinion paper the authors aim at clarifying some terminology commonly used in the modern deep learning literature. The goal is to highlight the strong overlap between several approaches that have been proposed as \"alternatives\" to supervised learning: predictive, unsupervised, semi-supervised and self-supervised learning. According to the authors, such proliferation of terms has been promoted by the (wrong) conviction that supervised learning is too limited in scope and does not account for the great variety of learning mechanisms observed in biological agents. The authors argue that the deep learning community would benefit from a clarification in terminology, which could highlight that, overall, all learning approaches can be defined as “supervised”.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The perspective advocated by the authors is provocative and original. It addresses an important topic with broad vision and well-motivated arguments.\n\nQuality: I enjoyed reading this paper. Though (to the best of my knowledge) opinion papers are not common in NeurIPS, I believe the subject of the article is relevant and timely, since it is often a matter of heated discussions with colleagues at AI conferences. I think it is very useful to clarify common terminology and try to organize the taxonomy of modern machine learning approaches. Maybe this would not be tremendously useful for experts and people working in the field from a long time, but it could be an important guide for newcomers and students. At the same time, however, I feel that the paper is sometimes a bit superficial and could be improved.\n\nThe most critical point, in my opinion, is that the discussion should be accompanied with more technical definitions: the different approaches mentioned by the authors as “alternatives” to supervised learning do have peculiar features that justify calling them with specific names, and such features should be clearly explained if the aim is to clarify the terminology. For example, semi-supervised learning entails that some labels are directly provided by human experts, while self-supervised learning usually refers to the setting where the agent discovers by itself the relevant categories, for example by exploiting related tasks. Unsupervised and predictive learning, instead, usually do not involve any label, and the “supervision” process is mostly related to the design of proper distance metrics or “introduction of prior knowledge” (Line 216).\n\nI think that most of the confusion stems from the ambiguous definition of “supervision”, which is normally associated with the use of explicit labels (discrete classes or real numbers) to characterize each input pattern. In this paper, instead, the authors seem to associate the term “supervision” with the more general notion of “feedback”. According to their view, “supervision” just implies that the agent can obtain some useful learning signals, for example through active interaction with the environment. If interpreted in this sense, I agree with the authors that all learning approaches should be considered supervised. However, I think that the most common way to think about “supervision” is to assume that the emerging semantics in the model should align with the conceptual semantics externally imposed by humans (i.e., classification, regression and ranking). If taken in this sense, I do not agree that all learning approaches should be considered supervised.\n\nIt would also be great if the authors could more explicitly put each learning approach in relation with biological learning principles, rather than just generically saying that biological learning and evolution imply lot of supervision signals. For a relevant discussion about the role of supervision and top-down feedback in cognitive models, the authors could refer to Testolin and Zorzi (Frontiers in Computational Neuroscience, 2016), which use “generative learning” as yet another alternative term that could be disambiguated. In relation to predictive learning, the authors could refer to Friston (Nature Reviews Neuroscience, 2010) and Clark (Behavioral and Brain Sciences, 2013), which provide a wealth of biologically-rooted perspectives on this powerful learning modality.\n\nFew typos:\n-\tLine 177: these\n-\tLine 201: propose\n-\tLine 366: growing…\n\nClarity: The paper is written in a very clear way and properly organized.\n\nSignificance: Deep learning researchers can benefit from a thoughtful clarification of relevant terminology and for a condensed overview of how state-of-the-art models incorporate biological learning principles.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not explicitly discuss the limitations of their research approach, but being this an opinion article I think this issue is less relevant.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see ethical issues related to the content of this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "1C9kgdJt2JO", "reviewer": "Reviewer_YQEE", "summary": "The manuscript offers perspectives on machine learning and connects some general claims in ML with some findings in cognitive science. The authors make four main suggestions for moving forward in the field (see below for a summary of contributions). ", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors examine important claims in machine learning. However, in its current state, this manuscript reads more like a high-level blogpost than a paper that would appear as a full publication in a top ML conference. The manuscript can be substantially improved if the authors provide more evidence for their claims about ML research directions. It would be especially beneficial to back up their argument with some statistics that can quantify the trends they describe instead of solely relying on quotes from a handful of papers ( for example, some statistics about number of papers on a certain topic with a certain claim). Below are some more specific issues that I have with the manuscript:\n\n1. The exposition is filled with unnecessarily inflammatory language. Some examples just from the first 50 lines: L4 \"gave rise to a hype loaded with ambitious promises and overstatements\", L6 'supervised learning went, for many, from glory to shame\", L49 \"based on misconceptions and overstatements about biological\n47 learning, and amplified by overselling nomenclature\". The manuscript would benefit from toning this down and allowing factual evidence to make its point. Here is the one that gave me the most pause: L180 \"would it not be worth reconsidering some research programmes?\". This seems to be completely the opposite of \nthe authors' self-described goal of being descriptive rather than prescriptive. \n\n2. The authors ​dismiss important challenges to modern ML by unfairly equating them to much more extreme versions of these challenges in human intelligence. For example, the authors dismiss the importance of addressing out-of-distribution generalization in ML by saying that humans are also limited in their \nability to transfer to new domains, as exemplified by the difficulty to learn a new unrelated language. Out-of-distribution generalization comes in many challenging forms to ML systems (e.g. changes in the label distribution (P(Y)), changes in the input feature distribution (P(X)), and changes in P(Y|X))). The example of humans learning a different language is an example of changing all of these distributions, which is an extreme form of out-of-distribution generalization. ML systems are shown to be sensitive to much milder changes in the problem setting. \n\n3. The contributions of this manuscript are not clear. It's not clear to me what this manuscript offers as a conclusion that is not already pursued in ML.\n- Two of the four conclusions that are drawn -- that evaluation beyond singe task performance is important and that a theoretical understanding of complex supervised objectives is important -- have been steadily growing directions in the field for over 5 years now.\n- One of the other 2 conclusions is that ML should temper its expectation about how sample efficient learning should be because humans are exposed to more supervision than is suggested in the ML literature. The authors oversimplify the aims in many of these ML works.\nML researchers don't expect that an ML system will develop abilities to generalize to a new class out of thin air. Of course a system needs to be exposed to \"relevant samples\" in order to learn. Not all of these samples however have to come from the specific class. \nThe hope is that a sample efficient ML system will be able to generalize from previously learned related examples in order to support fast learning of new examples.\n- The last remaining conclusion is that it would be great to distill progress in ML methods into a standardized nomenclature. I agree, as do many other ML researchers. A fitting contribution for a venue such as NeurIPS would be to actually suggest such a nomenclature. I don't consider a call for such nomenclature to be enough of a contribution.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a position paper, which is less common at NeurIPS.  It received very thorough reviews, and the authors responded in earnest.  However, the general consensus from the reviews was that the arguments posed by the paper did not have the depth or clarity required for a position paper to have a real impact as part of the NeurIPS program.

I would also encourage the authors to consider taking a less combative tone in future rebuttals.  Though the review process may seem adversarial, a lot of good can come out of it when it is approached as an opportunity for improvement. 
Decision: Reject