Deep learning
Authors: Yoshua Bengio
Geoffrey Hinton
Yann LeCun
Journal: Nature (2015)
Discipline: Multidisciplinary
Keywords: Deep learning; computational models; multiple processing layers; data representations; abstraction
Affiliations: 1] Facebook AI Research, 770 Broadway, New York, New York 10003 USA. [2] New York University, 715 Broadway, New York, New York 10003, USA., Department of Computer Science and Operations Research Université de Montréal, Pavillon André-Aisenstadt, PO Box 6128 Centre-Ville STN Montréal, Quebec H3C 3J7, Canada., 1] Google, 1600 Amphitheatre Parkway, Mountain View, California 94043, USA. [2] Department of Computer Science, University of Toronto, 6 King's College Road, Toronto, Ontario M5S 3G4, Canada.

Abstract:  Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. These methods have dramatically improved the state-of-the-art in speech recognition, visual object recognition, object detection and many other domains such as drug discovery and genomics. Deep learning discovers intricate structure in large data sets by using the backpropagation algorithm to indicate how a machine should change its internal parameters that are used to compute the representation in each layer from the representation in the previous layer. Deep convolutional nets have brought about breakthroughs in processing images, video, speech and audio, whereas recurrent nets have shone light on sequential data such as text and speech. 

Retracted: No

PubPeer comments:
 I sent a letter to Editor-in-Chief of Nature titled: A Report to Nature about Technical Flaws of Post-Selections Using Test Sets (PSUTS) Charge C1: Post-Selection. Their experimental protocols should have included a stage, called Post-Selections Using Test Sets (PSUTS). In the simplest form, all the data were divided into two disjoint sets, T and Tu2032. In a training stage, n systems where trained to fit the training set T, each starting from a different set of weights determined by a different random seed. Next, in the Post-Selection stage, all the n systems were tested on the test set Tu2032 and finally only the performance of the luckiest system that has the best performance on the test set Tu2032 was reported in the corresponding paper, but not the performances of the remaining less lucky n u2212 1 systems. This is a version of u201ctesting on the training setu201d as textbook Duda, Hart Stork, Pattern Classification (Wiley, New York, 2001) warned against, because the Post-Selection stage utilized the test set Tu2032. In competitions, the Post-Selections take a form of machine PSUTS or human PSUTS. Charge C2: Lack of transparency. There is a lack of transparency in reporting the Post-Selection stage. This paper misleadingly claimed: u201c... until ImageNet competition in 2012. When deep convolutional systems were applied to a data set of about a million images from the web that contained 1,000 different classes, they achieved spectacular results, almost halving the error rates of the best competing approaches.u201d But the paper did not explain the Post-Selection stage, which should have used PSUTS on team-labeled test sets. For more detail, see http: www.cse.msu.edu ~weng research 2021-06-28-Report-to-Nature-specific-PSUTS.pdf , 
 Alex Krizhevsky used a validation set to choose the hyper parameters. So the u201cluckiestu201d net was the one that did best on the validation set and it would indeed be wrong to report this performance. But Alex reported performance on the test set which gives an unbiased estimate. In the Imagenet competition the test set was only known to the people who ran the competition in order to guarantee that the cheating that Weng attributes to us could not possibly occur. , 
 #2 Surprising a man of your stature is taking the time to respond to this, Dr. Hinton. Nonetheless, perhaps your student s statement:  In the remainder of this paragraph, we use validation and test error rates interchangeably because in our experience they do not differ by more than 0.1% (see Table 2).  in the apparently-seminal NeurIPS publication, is a possible source for the confusion? (page 7, https: papers.nips.cc paper 2012 file c399862d3b9d6b76c8436e924a68c45b-Paper.pdf) , 
 #4 Dr. Weng, is it possible for you to provide any sources for the claims in point 3? i did a rudimentary google using the terms Krizhevsky, cracked, imagenet and could not find any information about this disclosure. that is not to say it didn t happen, but certainly it does not seem to be publicised. , 
 #2 This statement from Dr. Geoffrey Hinton is misleading and is without disclosed SuperVision source programs data to support. 1. Dr. Hinton wrote  In the ImageNet competition the test set was only known to the people who ran the competition . This is not accurate because, unfortunately, the test sets of ILSCRC 2012 (without answers) were released at least by July 12, 2012, but the contest answers (not including teams programs) were not due till Sept. 30, 2012. https: image-net.org challenges LSVRC 2012 index.php All ImageNet 2012 contest teams had as long as over 2.5 months to look at the test sets while there were no explicit ImagetNet Contest rules that ban teams from labeling the test sets themselves during algorithm developments. This time period seems too long as each team had a great pressure to label the test sets, at least to find out how their algorithms did on the test sets before submission. 2. In fact, from as early as January 26, 2012, ImageNet organizers (improperly?) provided an Evaluation Server, stating  you can evaluate you[sic] own results . That seems to mean the test answers were at least partially disclosed in an interactively way. https: image-net.org challenges LSVRC 2012 index.php [Moderator: part of this post did not comply with PubPeer s  no hearsay rule and has been deleted. Please see our FAQ.] , 
 Alex Krizhevsky in the Communications ACM paper apparently was using not only Post-Selections Using Validation Sets (PSUVS) but also Post-Selections Using Test Sets (PSUTS). For example, in Table 2, the test error is lower than the validation error, indicating that the Post-Selections probably used the test errors instead of the validation errors. The fact that Alex Krizhevsky was able to report test errors in Table 2 is another evidence that Alex Krizhevsky had u201ccrackedu201d the test sets of ILSVRC-2012. Alex Krizhevsky did not seem to understand that his report of test errors for 5 CNNs and 7 CNNs in Table 2 defeated his claimed reason of u201ccannot report test errorsu201d for 1 CNN and 1 CNN* because the former (5 CNNs and 7 CNNs) requires the same class labels for the test sets of ILSVRC-2012 as the latter (1 CNN and 1 CNN*). ,