Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary: This article is a position statement that aims to point out weaknesses in the 
authors' conception of the ML field's agenda. As the authors state, the work is
descriptive, not prescriptive: it does not provide specific directions, but rather
points to trends that (in the extreme) are problematic.
The key points in the article include: (1) natural environments provide rich
support for learning beyond traditional supervised classification labels; (2)
evolutionary and developmental processes provide strong inductive biases that can 
inform model design; (3) the amount of data that natural learners have available
is often underestimated (as is one-shot learning abilities); (4) there are many
signals in natural environments which can constrain learning beyond traditional
class labels.
main_review: This work would be a provocative position paper for a workshop,
but I fail to see what research contribution it makes. The authors' caricature
of the field's naivete does not match my impression of the field (having been
an active researcher for over 30 years). They point to relevant psychological
and neuroscientific literature (which the authors do know well) that the 
ML field would do well to be more aware of. Much of the literature cited has
been the subject of more cognitive science focused research at NeurIPS.
Like the authors, I wish the field would focus more on cognitive inductive bias 
rather than building larger and larger systems with larger and larger data sets. 
But the way to convince the field is not to lecture it, but to demonstrate that
the authors' insights into biology and cognition can be productively incorporated
into computational ML approaches. For example, in small data regimes, show that
cognitive biases (whether in architecture, objective functions, or data sources
available for learning) allow for what would otherwise be unlearnable.
The article also points to some boastful and unfortunate quotes from prominent figures,
but to shoot down the field for a few quotes is unnecessary. What the authors need
to make their argument convincing is specific examples of research done wrong and
research done right, rather than speaking in generalities. I do not believe
that the article in its current form would perturb any researcher from their
current trajectory.

The article reads as a stream-of-consciousness series of thoughts and contains many
dated and incorrect, or at least arguable, points, particularly with the authors' conception
of the 'supervised' and 'unsupervised' labels.  Let me illustrate with just one 
particularly frustrating section, Section 3.1.

* The section is headed "catastrophic forgetting" yet the content of the section says
nothing about this classic issue in neural nets.  Did the authors change their mind
about what content to include in the section?

* The authors' take from the no-free-lunch theorem is incorrect: the fact that no 
one model/architecture is superior to all others on expectation across data 
distributions does not motivate supervised learning per se. It motivates choosing 
inductive bias appropriate for a task. (The theorem assumes a certain objective function,
which could be either supervised or what is usually considered an unsupervised
criterion, such as cluster compactness.) 

* Any well-formulated unsupervised learning method requires an objective function  and optimization. The authors appear to consider this a form of 'implicit supervision'.   Unsupervised learning does not mean that the algorithm does something random. 
It means that the objective does not require guidance concerning specific responses from a teacher.   And unsupervised learning does not mean that the model is without inductive bias. As the authors state, "this is not news", but they then go on to claim (without support) that the field is unaware of these facts.

* Note that from a probabilistic perspective, unsupervised and supervised learning
share a common foundation. Rather than treating supervised and unsupervised
learning as two qualitatively different paradigms, every decent probabilistic ML textbook makes 
the point that follows.  Unsupervised learning is about modeling a distribution 
over observed features.  When those features are images and class labels,
learning a joint distribution over the observations will permit estimation of
conditional probabilities. Thus, from a probabilistic perspective, unsupervised
and supervised learning aren't qualitatively different methods; supervised
learning is a special case in which a model must estimate the _conditional_
distribution P(output|input), whereas
unsupervised learning does not split the data elements into 'input' and
'output' and thus requires modeling the _joint_ distribution, from which any
feature can be predicted from any other. One reason why the probabilistic
perspective is so popular in ML is because it provides this unified perspective.

* The authors state that supervised learning goes beyond classification to include
regression problems. Every textbook makes this point. Every ML course makes this
point.  The wikipedia 'supervised learning' page makes this point. I make this point in the first day when I teach ML. Again,
this is not news to anyone in the field. The authors may be expressing their frustration at
the predominance of research on classification, but there is no shortage of work on regression
problems at NeurIPS and other ML conferences. 

* The authors seem to have a somewhat dated conception of the field. Sure, in 2012
image classification was the focus of the groundbreaking research. 
One reason why image classification continues to be so popular is that the image 
data sets are well understood, and these image data sets provide sensible test beds for 
better understanding deep learning and exploring novel optimization methods, architectures, etc. However, the field is much 
richer than the authors allow. For example, the introduction of the transformer 
architecture for language processing--as horribly non-cognitive as it is--has become a 
huge industry in and of itself. Video understanding and interpretation (not just
classification) can be found at every recent ML conference. Work on video-text interactions
is now moving to the foreground. The field is advancing and the problems researchers 
choose to focus on changes over time.

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 2: Strong rejection
review_content: summary: The manuscript offers perspectives on machine learning and connects some general claims in ML with some findings in cognitive science. The authors make four main suggestions for moving forward in the field (see below for a summary of contributions). 
main_review: The authors examine important claims in machine learning. However, in its current state, this manuscript reads more like a high-level blogpost than a paper that would appear as a full publication in a top ML conference. The manuscript can be substantially improved if the authors provide more evidence for their claims about ML research directions. It would be especially beneficial to back up their argument with some statistics that can quantify the trends they describe instead of solely relying on quotes from a handful of papers ( for example, some statistics about number of papers on a certain topic with a certain claim). Below are some more specific issues that I have with the manuscript:

1. The exposition is filled with unnecessarily inflammatory language. Some examples just from the first 50 lines: L4 "gave rise to a hype loaded with ambitious promises and overstatements", L6 'supervised learning went, for many, from glory to shame", L49 "based on misconceptions and overstatements about biological
47 learning, and amplified by overselling nomenclature". The manuscript would benefit from toning this down and allowing factual evidence to make its point. Here is the one that gave me the most pause: L180 "would it not be worth reconsidering some research programmes?". This seems to be completely the opposite of 
the authors' self-described goal of being descriptive rather than prescriptive. 

2. The authors ​dismiss important challenges to modern ML by unfairly equating them to much more extreme versions of these challenges in human intelligence. For example, the authors dismiss the importance of addressing out-of-distribution generalization in ML by saying that humans are also limited in their 
ability to transfer to new domains, as exemplified by the difficulty to learn a new unrelated language. Out-of-distribution generalization comes in many challenging forms to ML systems (e.g. changes in the label distribution (P(Y)), changes in the input feature distribution (P(X)), and changes in P(Y|X))). The example of humans learning a different language is an example of changing all of these distributions, which is an extreme form of out-of-distribution generalization. ML systems are shown to be sensitive to much milder changes in the problem setting. 

3. The contributions of this manuscript are not clear. It's not clear to me what this manuscript offers as a conclusion that is not already pursued in ML.
- Two of the four conclusions that are drawn -- that evaluation beyond singe task performance is important and that a theoretical understanding of complex supervised objectives is important -- have been steadily growing directions in the field for over 5 years now.
- One of the other 2 conclusions is that ML should temper its expectation about how sample efficient learning should be because humans are exposed to more supervision than is suggested in the ML literature. The authors oversimplify the aims in many of these ML works.
ML researchers don't expect that an ML system will develop abilities to generalize to a new class out of thin air. Of course a system needs to be exposed to "relevant samples" in order to learn. Not all of these samples however have to come from the specific class. 
The hope is that a sample efficient ML system will be able to generalize from previously learned related examples in order to support fast learning of new examples.
- The last remaining conclusion is that it would be great to distill progress in ML methods into a standardized nomenclature. I agree, as do many other ML researchers. A fitting contribution for a venue such as NeurIPS would be to actually suggest such a nomenclature. I don't consider a call for such nomenclature to be enough of a contribution.



limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: With this opinion paper the authors aim at clarifying some terminology commonly used in the modern deep learning literature. The goal is to highlight the strong overlap between several approaches that have been proposed as "alternatives" to supervised learning: predictive, unsupervised, semi-supervised and self-supervised learning. According to the authors, such proliferation of terms has been promoted by the (wrong) conviction that supervised learning is too limited in scope and does not account for the great variety of learning mechanisms observed in biological agents. The authors argue that the deep learning community would benefit from a clarification in terminology, which could highlight that, overall, all learning approaches can be defined as “supervised”.
main_review: Originality: The perspective advocated by the authors is provocative and original. It addresses an important topic with broad vision and well-motivated arguments.

Quality: I enjoyed reading this paper. Though (to the best of my knowledge) opinion papers are not common in NeurIPS, I believe the subject of the article is relevant and timely, since it is often a matter of heated discussions with colleagues at AI conferences. I think it is very useful to clarify common terminology and try to organize the taxonomy of modern machine learning approaches. Maybe this would not be tremendously useful for experts and people working in the field from a long time, but it could be an important guide for newcomers and students. At the same time, however, I feel that the paper is sometimes a bit superficial and could be improved.

The most critical point, in my opinion, is that the discussion should be accompanied with more technical definitions: the different approaches mentioned by the authors as “alternatives” to supervised learning do have peculiar features that justify calling them with specific names, and such features should be clearly explained if the aim is to clarify the terminology. For example, semi-supervised learning entails that some labels are directly provided by human experts, while self-supervised learning usually refers to the setting where the agent discovers by itself the relevant categories, for example by exploiting related tasks. Unsupervised and predictive learning, instead, usually do not involve any label, and the “supervision” process is mostly related to the design of proper distance metrics or “introduction of prior knowledge” (Line 216).

I think that most of the confusion stems from the ambiguous definition of “supervision”, which is normally associated with the use of explicit labels (discrete classes or real numbers) to characterize each input pattern. In this paper, instead, the authors seem to associate the term “supervision” with the more general notion of “feedback”. According to their view, “supervision” just implies that the agent can obtain some useful learning signals, for example through active interaction with the environment. If interpreted in this sense, I agree with the authors that all learning approaches should be considered supervised. However, I think that the most common way to think about “supervision” is to assume that the emerging semantics in the model should align with the conceptual semantics externally imposed by humans (i.e., classification, regression and ranking). If taken in this sense, I do not agree that all learning approaches should be considered supervised.

It would also be great if the authors could more explicitly put each learning approach in relation with biological learning principles, rather than just generically saying that biological learning and evolution imply lot of supervision signals. For a relevant discussion about the role of supervision and top-down feedback in cognitive models, the authors could refer to Testolin and Zorzi (Frontiers in Computational Neuroscience, 2016), which use “generative learning” as yet another alternative term that could be disambiguated. In relation to predictive learning, the authors could refer to Friston (Nature Reviews Neuroscience, 2010) and Clark (Behavioral and Brain Sciences, 2013), which provide a wealth of biologically-rooted perspectives on this powerful learning modality.

Few typos:
-	Line 177: these
-	Line 201: propose
-	Line 366: growing…

Clarity: The paper is written in a very clear way and properly organized.

Significance: Deep learning researchers can benefit from a thoughtful clarification of relevant terminology and for a condensed overview of how state-of-the-art models incorporate biological learning principles.

limitations_and_societal_impact: The authors did not explicitly discuss the limitations of their research approach, but being this an opinion article I think this issue is less relevant.
review_title: null

Meta-review: This is a position paper, which is less common at NeurIPS.  It received very thorough reviews, and the authors responded in earnest.  However, the general consensus from the reviews was that the arguments posed by the paper did not have the depth or clarity required for a position paper to have a real impact as part of the NeurIPS program.

I would also encourage the authors to consider taking a less combative tone in future rebuttals.  Though the review process may seem adversarial, a lot of good can come out of it when it is approached as an opportunity for improvement. 
Decision: Reject