Neural Networks 

Post-Selections in AI and How to Avoid Them --Manuscript Draft--

Manuscript Number: 

Article Type: Article

Section/Category: Mathematical and Computational Analysis

Keywords: Experimental Protocol; Error-Backprop; Deep Learning; Performance Evaluation;

Maximum Likelihood; Turing Machines

Corresponding Author: Juyang Weng, Ph.D.

Michigan State University

East Lansing, Michigan UNITED STATES

First Author: Juyang Weng, Ph.D.

Order of Authors: Juyang Weng, Ph.D.

Abstract: Neural network based Artificial Intelligence (AI) has reported increasing scales in

experiments. However, this paper raises a rarely reported stage in such experiments

called Post-Selection alter the reader to several possible protocol flaws that may result

in misleading results. All AI methods fall into two broad schools, connectionist and

symbolic. The Post-Selection fall into two kinds, Post-Selection Using Validation Sets

(PSUVS) and Post-Selection Using Test Sets (PSUTS). Each kind has two types of

post-selectors, machines and humans. The connectionist school received criticisms for

its “black box” and now the Post-Selection; but the seemingly “clean” symbolic school

seems more brittle because of its human PSUTS. This paper first presents a

controversial view: all static “big data” are non-scalable. We then analyze why errorbackprop from randomly initialized weights suffers from severe local minima, why

PSUVS lacks cross-validation, why PSUTS violates well-established protocols, and

why every paper involved should transparently report the Post-Selection stage. To

avoid future pitfalls in AI competitions, this paper proposes a new AI metrics, called

developmental errors for all networks trained, under Three Learning Conditions: (1) an

incremental learning architecture (due to a “big data” flaw), (2) a training experience

and (3) a limited amount of computational resources. Developmental Networks avoid

Post-Selections because they automatically discover context-rules on the fly by

generating emergent Turing machines (not black boxes) that are optimal in the sense

of maximum-likelihood across lifetime, conditioned on the Three Learning Conditions.

Suggested Reviewers: 

Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation 
Title Page (With all author details listed)

Post-Selections in AI and How to Avoid Them

Juyang Weng

Department of Computer Science and Engineering

Cognitive Science Program

Neuroscience Program

Michigan State University, East Lansing, MI, 48824 USA

and

GENISAMA LLC

4460 Alderwood Drive, Okemos, Michigan 48864 USA
Manuscript Click here to view linked References

Post-Selections in AI and How to Avoid Them 

Juyang Weng1,2,3,4

1Department of Computer Science and Engineering 

2Cognitive Science Program 

3Neuroscience Program 

Michigan State University, East Lansing, MI, 48824 USA 

4GENISAMA LLC, 4460 Alderwood Drive, Okemos, Michigan 48864 USA 

Abstract 

Neural network based Artificial Intelligence (AI) has reported increasing scales in experiments. However, this paper raises a rarely reported stage in such experiments called 

Post-Selection alter the reader to several possible protocol flaws that may result in misleading results. All AI methods fall into two broad schools, connectionist and symbolic. The Post-Selection fall into two kinds, Post-Selection Using Validation Sets 

(PSUVS) and Post-Selection Using Test Sets (PSUTS). Each kind has two types of 

post-selectors, machines and humans. The connectionist school received criticisms for 

its “black box” and now the Post-Selection; but the seemingly “clean” symbolic school 

seems more brittle because of its human PSUTS. This paper first presents a controversial view: all static “big data” are non-scalable. We then analyze why error-backprop 

from randomly initialized weights suffers from severe local minima, why PSUVS lacks 

cross-validation, why PSUTS violates well-established protocols, and why every paper 

involved should transparently report the Post-Selection stage. To avoid future pitfalls in 

AI competitions, this paper proposes a new AI metrics, called developmental errors for 

all networks trained, under Three Learning Conditions: (1) an incremental learning architecture (due to a “big data” flaw), (2) a training experience and (3) a limited amount 

of computational resources. Developmental Networks avoid Post-Selections because 

they automatically discover context-rules on the fly by generating emergent Turing machines (not black boxes) that are optimal in the sense of maximum-likelihood across 

lifetime, conditioned on the Three Learning Conditions. 

Preprint submitted to Neural Networks September 9, 2021 
Keywords: Experimental Protocols, Error-Backprop, Deep

Learning, Performance Evaluation, Maximum Likelihood,

Turing Machines

1. Introduction 

AI research dates back at least to early 1910 when Leonardo Torres y Quevedo built 

a chess end game player called EI Ajedrecista [1]. In 1950, Alan Turing published his 

now celebrated paper [2] titled Computing Machinery and Intelligence. Turing [2] was 

5 impressive to have discussed a wide variety of considerations for machine intelligence, 

as many as nine categories. Unfortunately, he suggested to consider what is now called 

the Turing Test that has inspired and misled many AI researchers. 

Much progress has been made in AI since then and many methods have been developed to deal with AI problems. As the scope of this paper, we will focus on general10 ization. All AI methods fall into two schools [3], symbolic and connectionist, although 

many published methods are a mixture of both. 

1.1. Symbolic school 

Symbols are used in many AI methods (e.g., states in HMMs, nodes in Graphical Models and attributes in SLAM). Although symbols are intuitive to a human pro15 grammer since he defines the associated meanings, symbols are static and have some 

fundamental limitations that have not received sufficient attention. 

The symbolic school [4] assumes a micro-world in 4D space-time in which a set of 

objects or concepts, e.g., L = {l1, l2, ..., ln}, is assumed to be uniquely defined among 

many human programmers and their computers, represented by a series of symbols in 

20 time {l1(t), l2(t), ...ln(t)|t0 ≤ t < t1}. The correspondences among all these symbols 

{li} of the same object across different times are known as “the frame problem” [4] in 

AI which means that the programmer must manually link every symbol along time with 

its corresponding physical object. In computer vision, the symbolic school assumes a 

single symbol oi, for all its 3D positions in its 3D trajectory {x(t)|t0 ≤ t ≤ t1}

25 and uses certain techniques, such as feature tracking through video (e.g., for driverless 

2 
cars). Therefore, the symbolic school is based on human-handcrafted set of symbols 

and their assumed meanings. Marvin Minsky wrote that symbols are “neat” [5], but in 

fact, symbols are “neat” mainly in a single human programmer’s understanding but not 

between different programmers and not in relating computer programs to a real world. 

30 We will see the Developmental Network (DN) model of a brain is free from any 

symbols in its full version. Abstract symbols correspond to action/state vectors in 

the motor area of DN. Therefore, the frame problem is automatically solved through 

emergent action/state vectors in a physically grounded DN, without using any symbols 

in the DN’s internal representations. 

35 A major problem for symbolic AI is the generalization issue of symbols as defined 

here. 

Definition 1 (Brittleness of static symbols). Suppose a symbolic AI machine M(L)

designed for a handcrafted set L of symbols is applied to a real world that requires a 

new set of symbols L0

, with L∩L0 6= ∅, M(L) fails without a human programmer who 

40 handcrafts an appropriate mapping function f : L0 7→ L that maps every element of 

L0 to an element in L so that M(f(L0

)) works correctly as before. 

Many expert systems (e.g., CYC, WordNet and EDR [6]) and “big data” projects [7] 

require a human programmer to be in the loop of handcrafting such a mapping f during 

deployment. For example, an machine M developed in Florida is deployed in Michigan 

45 but Michigan has snow but Florida does not. Because it is extremely challenging for a 

human programmer to understand many implicit limitations of M(L), the mapping f

that the human handcrafts typically makes M(F(L0

)) fail, resulting in the well-known 

high brittleness of symbolic systems. 

Due to emergent representations as numeric vectors, a DN robot discussed below 

50 learns snow settings and the snow concept when it sees snow scenes for the first time, 

because there are no symbols that correspond to snow in DN’s representations. 

In general, the developmental methods to be discussed below automatically address 

such new concept problems without a need for a human programmer to be in the loop 

of handcrafting a symbolic mapping f during a deployment. In this paper, the author 

55 will further argue that the symbolic school suffers from human PSUTS. 

3 
1.2. Connectionist School 

The connectionist school claimed to be less brittle [8, 9]. However, a network 

is egocentric—meaning that the agent starts from its own (neural) network, instead 

of a symbolic world. It must learn from the external world without a handcrafted, 

60 world-centered object model. Although connectionist methods often assume some 

task-specific symbols, e.g., a static set L of object labels, they also assume a restricted 

world implicitly. Therefore, a connectionist model typically needs to sense and learn 

from a restricted world using a network. The use of L by any neural networks (e.g., 

ImageNet [10] and many other competitions) as a set of object labels is a fundamental 

65 limitation that also causes the resulting system to be brittle for the same reason as the 

symbolic school. 

Typically, a neural network is meant to establish only a mapping f from the space 

of input X to the space of class labels L, 

f : X 7→ L (1) 

[11, 12]. X many contains a few time frames. Many video analysis problems, speech 

recognition problems, and computer game-play problems are also converted into this 

static input space so that the input space also includes L, so as to learn 

f : X × L 7→ L. (2) 

Without a pressure of performance characterization during learning other than the 

performance of the final network, a self-organization map (e.g., SOM) has been used 

often as an unsupervised but slow learning method [13, 14, 15] . 

70 In contrast, with a pressure of performance characterization during learning, Cresceptron [16] used a “skull-closed” incremental-learning Hebbian-like scheme with receptivefield based competitions. 

Other than the Hebbian mechanisms which are strictly “unsupervised” used by he 

Cresceptron and the DN explained below, two other types of learning schemes have 

75 been published: 

A Human handpicking features: after knowing the test set, humans handpick features, 

4 
reported explicitly [17, 18, 19] or implicitly as “weakly supervised” [19]. This 

author called them “skull-open” [20]. 

B Error-backprop: Locally train multiple networks each from a different set of random 

80 weights. After the training, post-select the luckiest network. Report the luckiest 

network only [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 

38] but many publications do not report the post-selection stage at al, with few 

exceptions [27]. 

Below, this author will argue that (A) suffers from human Post-Selections and (B) 

85 suffers from machine Post-Selections. While Cresceptron, the first deep learning for 

a 3D world, generates a single large network, it showed an impressive generalization 

power due to the use of the nearest neighbor scheme at every layer of an automatically 

generated deep network. This author will argue that Post-Selections in (A) and (B) 

suffer from weak generalizations (due to three types of lucks to be discussed below) and 

90 did not count the cost of training multiple networks many of which were not reported. 

By the way, genetic algorithms offer another approach to such network learning. 

These algorithms study changes in genomes across different life generations. However, 

many genetic algorithms do not deal with lifetime development [39, 40]. We argue that 

handcrafting functions of a genome as a Developmental Program (DP) seems to be a 

95 clean and tractable problem, which avoids the extremely high, cost of evolution on DP. 

Many genetic algorithms further suffer from the PSUTS problems, since they often use 

test sets as training sets (i.e., vanished tests) as explained below. 

Marvin Minsky [5] among many scholars complained that neural networks are 

“scruffy” or “black boxes”. This problem is not addressed holistically until the frame100 work of Emergent Turing Machine was introduced [41] into Developmental Networks 

(DNs) by the Developmental School discussed below. A lack of Emergent Turing Machine mechanisms or being “scruffy” in sample fitting appears to be the main cause of 

PSUTS in traditional neural networks trained by human feature-handpicking or errorbackprop methods. 

5 
105 1.3. Developmental School 

The main thrust of the Developmental School, formally presented 2001 by Weng 

and six co-authors [39] is the task-nonspecificity for lifetime development, known as 

Developmental Programs (DPs) that simulate the functions of genome without simulating the genome encoding. Although a DP generates a neural network, a DP is very 

110 different from a conventional neural network in the evaluation of performance across 

each life—all errors from the inception time 0 of each life is recorded and reported up 

to each frame time t > 0, as explained further below. 

The first developmental program seems to be the Cresceptron by Weng et al. [42, 

43, 16] which appears to be, as far as the author is aware, the first deep-learning Con115 volutional Neural Network (CNN) for a 3D world. As explained in [44, 45] other 

well-known CNNs for 3D recognition, although they do not use a generative DP, followed many key ideas of Cresceptron. Cresceptron seems to be the first incremental 

neural network whose evaluation of performance is across its entire “life” and only one 

network was generated (developed) from the given training data set. 

120 Cresceptron did not deal with time. A developmental approach that deals with both 

space and time in a unfired fashion using a neural network started from Developmental Networks (DNs) [46] whose experimental embodiments range from Where-What 

Networks 1 (WWN-1) [47] to WWN-9 [48]. The DNs went beyond vision problems to 

attack general AI problems including vision, audition, and natural language acquisition 

125 as emergent Turing machines [41]. 

DNs overcame the limitations of the framewise mapping in Eq. (2) by dealing with 

lifetime mapping: 

f : X(t − 1) × Z(t − 1) 7→ Z(t), t = 1, 2, ... (3) 

where X(t) and Z(t) are the sensory input space and motor space, respectively, and ×

denotes the Cartesian product of sets. A fundamental difference between Eq. (2) and 

Eq. (3) is that in the latter the Z space contains exclusively emergent vectors, instead 

of any symbols, so that the actions/states are incrementally taught and learned across a 

130 lifetime. 

6 
As we will see in Section 4.1, all the errors occurred during any time of each life is 

recorded and taken into account in the performance evaluation. 

It is important to extend Eq. (3) to include the hidden area Y that generates international (hidden) representations. To model how Y -to-Y connections enable something 

similar to higher and dynamic order in Markov models (but not symbolic), the above 

lifetime mapping is extended to: 

f : X(t − 1) × Y (t − 1) × Z(t − 1) 7→ Y (t) × Z(t), t = 1, 2, ... (4) 

Note that Z(t − 1) here is extremely important since it corresponds to the state 

of a Turing machine. Namely, all the errors occurred during any time of each life is 

135 recorded and taken into account in the performance evaluation. 

Different from the static symbols in the symbolic school and the space of class 

labels L of static symbols in Eq. (2) of the connectionist school, the space Z(t) of 

numeric vectors of the developmental school is free from symbols. Therefore, these 

states/actions are directly teachable or self-generative, inspired by brains [49, 50, 51, 

140 52, 53, 51, 54]. This new symbol-free formulation is necessary to model not only 

brain’s spatial processing [55] and temporal processing [56], but also Autonomous 

Programming for General Purposes (APFPG) [57]. Based on the APFGP capability, we open the door towards the next step—conscious learning [58]—learning while 

being partially and increasingly conscious. By conscious learning, we do not mean 

145 “open-skulledly” handcrafting general-purpose consciousness, which is probably too 

complicated to handcraft. But instead we enable fully autonomous machine learning while machines being partially conscious—autonomously learn more sophisticated 

consciousness skills using their partial, earlier, and simpler conscious skills across the 

lifetime. 

150 This is the journal archieval version of the earlier conference papers [59, 60] with 

significantly refined additional material and analysis. 

In the following, we will discuss Post-Selection in Section 2. Section 3 addresses 

why error-backprop algorithms suffer from severe local minima problems. Section 4 

explains how a Developmental Network solves the local minima problems, since only 

155 one network is needed for each life and the evaluation of performance across the entire 

7 
life. Section 5 discusses experiments. Section 6 provides concluding remarks. 

2. Post-Selections 

AI has made impressive progress, gained much visibility, and attracted the attention 

of many government officials. However, there are protocol flaws that have resulted in 

160 misleading results. 

First, let us consider three learning conditions that any fair comparisons of AI methods should take into account. 

2.1. The Three Learning Conditions 

Many AI methods were evaluated without considering how much computational 

165 recourses are necessary for the development of a reported system. Thus, comparisons 

about the performance of the system have been tilted toward competitions about how 

much resources a group has at its disposal, regardless how many networks have been 

trained and discarded, and how much time the training takes. 

Here we explicitly define the Three Learning Conditions for development of an AI 

170 system: 

Definition 2 (The Three Learning Conditions). The Three Learning Conditions for 

developing an AI system are: (1) a set of restrictions of learning framework, including 

whether task-specific or task-nonspecific, batch learning or incremental learning, and 

the body’s sensors and effectors; (2) a training experience and (3) a limited amount of 

175 computational resources including the number of hidden neurons. 

The competing standard of the ImageNet competitions [61] did not include any 

of these three conditions. The AIML Contests [62] considered all the three in performance evaluation. In the following Subsection, we discuss why task-nonspecificity and 

incremental mode should be considered in any comparisons. 

180 2.2. Task-specific vs. Task-nonspecific 

A task-specific learning approach learns less because much is handcrafted by a 

human according to the given task. Furthermore, a task-specific method is brittle. 

8 
In Condition (1) of the Three Learning Conditions, the task-nonspecific learning 

paradigm is significantly different from the task-specific traditional AI paradigm as 

185 explained in Weng et al. 2001 [39]. In a task-specific paradigm, the system developer 

is given a task e.g., constructing a driverless car. Then, it is a human programmer who 

chooses a world model, such as a model of lane edges. Next, he picks an algorithm 

based on this world model, e.g., the well-known Hough transform algorithm [63, 64] 

for line detection which makes every pixel that is detected as edge cast votes for lines 

190 of all possible orientations o and distances d from the origin that go through the pixel. 

Then the top-two “peaks” of line parameters (o, d) that have received the highest votes 

are adopted to declare a line detected from the image. Here “edges” and “lane lines" 

are two symbolic concepts picked up by the programmer. Such systems will fail when 

lanes are unclear or totally disappear due to weather or road conditions, leading to a 

195 brittle system. Human brains appear to be more resilient. 

In contrast, a task-nonspecific approach [39] not only avoids any symbolic model, 

but also does not requires that a task is given. The desirable actions at any time are 

taught, tried, and recalled automatically by the learner based on system’s learned context q [65] that includes automatically figured-out goal and state, as well as the cur0 200 rent input. The mapping function f(z, x) = z , representing the symbolic mapping 

0 fs(q, σ) = q , corresponds to a finite automaton. Weng has proven that the control 

of any Turing machine is a finite automaton [41]. Thus, this framework is of generalpurposes in the sense of universal Turing machines. Any universal Turing machine is 

of general purposes, because it can read any program written for any purposes and run 

205 it for the purposes. Any neural network that learns a universal Turing machine become 

of general purposes in the sense of any programs, not just in the sense of any mappings 

like that in Eq. (1). Thanks to the absence of any world model, such as lanes, this 

task-nonspecific approach has a potential to be more robust than a world-model based 

approach. The task-nonspecific approach typically uses a neural network to learn be0 210 cause the need to learn vector mapping function f(z, x) = z . We will discuss internal 

response vector y in Section 4 but task-nonspecificity holds true without y. 

9 
2.3. Batch vs. Incremental Learning Modes 

Neural network learning for the mapping f has two learning modes, batch learning 

and incremental learning. 

215 With batch learning, a human first collects a set D of data (e.g., images) and then 

labels each datum with a desirable output (e.g., command of navigation or class label). A neural network is trained to approximate a mapping f in Eq. (1) or Eq. (2). 

Many batch-learning projects use an error-backprop method [23, 66, 24] which uses a 

gradient-based method to find a local minimum in error. 

220 As we will discuss in Section 3, the gradient in the error-backprop method does not 

contain key information of many other data if the learning mode is incremental. Thus, 

error-backprop on a large data set does poorly using a purely incremental learning 

mode. Many used a block-incremental learning mode which suffers from the big data 

flaw in Theorem 1 below. 

225 In contrast, all developmental methods cited here use incremental learning mode 

for long lifetimes, using a closed-form solution to the global lifetime optimization. 

The competition among neurons guarantees that the winner is the most appropriate 

neuron whose memory corresponds to the current working memory [67]. 

However, the batch and incremental learning modes are not capability-equivalent 

230 [67]. The former requires all sensory inputs are available at a batch, independent of 

the corresponding actions. Therefore, the former is easier and also incorrect according 

to sensorimotor recurrence. By sensorimotor recurrence, we mean that sensory inputs 

and motor outputs are mutually dependent on each other in such a recurrent way that 

off-line collection of inputs are technically flawed. We have the following theorem: 

235 Theorem 1 (Big Data Flaw). All static “big data” sets used by machine learning violate the sensorimotor recurrence property of the real world. 

Proof: A learning agent at time t − 1, as shown in Eqs. (3) and (4) does not have 

the next sensory input from X(t) available before the corresponding actions in Z(t−1)

are generated and output, since the sensory input in X(t) varies according to the agent 

240 actions in Z(t − 1). As an example, turning head left or right will result in a different 

10 
image sensed. Therefore, all static “big data” sets violate the sensorimotor recurrence. 



One may say that classifications of static images are fine. We do not agree, because even when a human (or machine) is looking at a static image, he uses attention 

245 (e.g., context-based saccades)which is a sequence of actions. Each saccade results in a 

different fovea image. 

Therefore, incremental learning is necessary for the sensorimotor recurrence. All 

batch-training methods use a static set of training data and, therefore, are inappropriate 

for any of them to claim near-human performance since the two learning problems are 

250 different. This leads to the following theorem. 

Consider a hierarchy of levels of object types, such as nails, fingers, palms, hands, 

arms, limbs, torsos, human bodies, etc. Because vision requires a high level l to understand natural scenes with abstraction of parts with invariances (e.g., all fingers of different scales, looks, and at different locations), each child needs an open-ended world 

255 to learn to learn rules (e.g., finger-parts and hand-whole) instead of simple-minded 

pattern recognition of sensory images. 

Theorem 2 (Nonscalability of Big Data without abstraction). All static “big data” 

sets used by machine learning are nonscalable if they are treated as pattern recognition 

without rule abstractions. 

260 Proof: Suppose that a static data set D has shown the presence of k feature types 

defined at level 1 (e.g., edge pixels are a type). Suppose a combination of k > 1 feature 

types to level l + 1 type (e.g., straight line type is from multiple edge pixels) is defined 

from k types of feature types at level l, l = 1, 2, .... The number of samples for a llevel feature type requires at least kl observations to discover all necessary within-type 

265 equivalence (e.g., logic OR is at l = 2 with k = 2 logic features at l = 1, thus without 

rule abstraction (e.g., parts and whole), it requires kl = 22 = 4 observations, corresponding to 4 rows in the truth table of logic OR). Since f(l) = kl is an exponential 

function in l, kl quickly exceeds any fixed number of observations in the static data set 

D. 

11 
270 Rule abstractions deal with invariances. For example, a “what” concept is “where”-

invariant and a “where” concept is “what”-invariant, as explained in [55, 68]. 

Section 4 discusses an optimal framework through which such abstractions can take 

place from learning simple rules during early life that enable learning of more complex 

rules during later life—called scaffolding [69]. 

275 Theorem 2 leads to two observations on data fitting on a static data set: 

Observation 1: Any data fitting on a static data set without learning invariant 

concepts are nonscalable, including the n-fold cross-validation discussed below. Unfortunately, data fitting on a static data set is a norm in all ImageNet Contests [66]. 

Namely, the remaining subsections in this section analyze approaches that are nonscal280 able. For example, computer vision is not a “one-shot” pattern classification problem as 

argued by Li Fei-Fei et al. [19] (which was questioned in PubMed without responses), 

but rather a spatiotemporal problem to learn various invariant concepts present in cluttered natural scenes through autonomous attention saccades, as explained further in 

Observation 2. 

285 Observation 2: Learning invariant concepts seem nonscalable for any data fitting 

on a static data set either, because there are too many images to be labeled by hand 

(e.g., all pixel locations) [55, 68]. Like a human baby, any scalable machine learning 

methods must be conscious through which the machine learner must consciously guess 

concepts (i.e., not just active learning [70]) (e.g., an object type) and verify their in290 variance rules (e.g., the where-invariance of a what concept). The state-based transfer 

in Theorem 8 of [56] explains how each concept state reduces the number of samples 

to be learned from an exponential kl down to only kl (see Fig. 6 of [56] for intuition 

where k = 10 and l = 3). Thus, Section 4 not only addresses the non-scalability 

problems in this section, but is also necessary for conscious learning whose theory was 

295 recently published in Weng 2020 [58] with some single-sensory-modality experimental 

results, but animal-level conscious robots that are multi-sensory and multi-motor have 

not yet been demonstrated. The availability of real-time learning brain-chip is a current 

bottleneck. 

12 
2.4. Fitting, validation and test errors 

Given an available data set D, D is divided by a partition P = (T, V, T0

) into three 

mutually disjoint sets, a training set T, a validation set V , and a test set T0 so that 

D = T ∪ V ∪ T0

. (5) 

300 Two sets are disjoint if they do not share any elements. The validation set is possessed 

by the trainer, the test set should not be possessed by the trainer since the test should 

be conducted by an independent agency. Otherwise, V and T0 become equivalent. 

As we will see in Section 3, given any architecture parameter vector ai, it is unlikely 

that a single network initialized by a set of random weight vectors can result in an 

acceptable error rate on the training set, called fitting error, that the error-backprop 

training intends to minimize locally. That is how the multiple sets of random weight 

vectors come in. For k architecture vectors ai, i = 1, 2, ...k and n sets of random initial 

weight vectors wj , the error back-prop training results in kn networks 

{N(ai, wj )|i = 1, 2, ..., k, j = 1, 2, ..., n}.

Error-backprop locally and numerically minimizes the fitting error fi,j on the training 

set T. 

305 [27] seems to have mentioned n = 20. [71] did not give n but seems to have 

mentioned 60 million parameters which probably means each wi and each aj combined to be of 60 million dimensional. Using the above example of k = 310 = 59049, 

kn ≈ 1M networks must be trained, a huge number that requires a lot of computational 

resources to do number crunching and a lot of manpower to manually tune the range of 

310 hyper-parameters. 

Definition 3 (Distribution of fitting, validation and test errors). The distributions of 

all kn trained networks’ fitting errors {fij}, validation errors {eij}, and test errors 

0 {e }, i = 1, 2, ...k, j = 1, 2, ...n are random distributions depending on a specific ij

data set D and its partition P = (T, V, T0). The difference between a validation error 

315 and a test error is that the former is computed from the same group using a grouppossessed validation set V but the latter is computed by an independent agency using 

a group-unknown test set T0

. 

13 
We define a simple system that is easy to understand for our discussion to follow. 

Definition 4 (Nearest neighbor classifiers with a confidence threshold). Define a net320 work stores the entire training set T. Suppose the input x matches the nearest sample 

s in T. If the distance between x and s is not larger than a confidence threshold d (a 

hyper-parameter), then the network outputs the associated label of the nearest sample 

s. Otherwise, the system outputs “unknown”. 

Namely, this system uses a lot of resources for over-fitting. It gives up if the dis325 tance is larger than d, but has a perfect fitting error (zero) for any positive d. 

A neural network architecture has a set of hyper parameters represented by a vector 

a, where each component corresponds a scalar parameter, such as convolution kernel 

sizes and stride values at each level of a deep hierarchy, the neuronal learning rate, 

and the neuronal learning momentum value, etc. Let k be a finite number of grid 

330 points along which such architecture parameter vectors need to be tried, A = {ai|i =

1, 2, ..., k}. Suppose there are 10 scalar parameters in each vector ai. For each scalar 

parameter x of the 10 hyper parameters, we need to validate the sensitivity of the 

system error to x. With uncertainty of x, we estimate its initial value as the mean x ̄, 

positively perturbed estimate x ̄ + σx (σ is the estimated standard deviation of x), and 

335 negatively perturbed estimate x ̄ − σx. If each scalar hyper parameter has three values 

to tray in this way, there are a total of k = 310 = 59049 architecture parameter vectors 

to try, a very large number. For example, the initial threshold d

 ̄in the nearest neighbor 

classifier can be estimated by the average of nearest distance between a sample in V

and the nearest neighbor in T and the σd be estimated by the standard deviation of 

340 these nearest distances. 

Let us define the Post-Selection. Suppose that the trainer is first aware of the validation sets (or the test sets). 

Definition 5 (Post selection). A human programmer trains multiple systems using the 

training set T. After these systems have been trained, he post-selects a system by 

345 searching, manually or assisted by computers, among trained systems based on the 

validation set V (or the test set T0

). This is called Post-Selection—selection of one 

network from multiple trained and verified (or tested) networks. 

14 
Obviously, a post-selection wastes all trained systems except the selected one. As 

we will see next, a system from the post-selection tends to have a weak generalization 

350 power. 

First, consider Post-Selection Using Validation Sets (PSUVS): 

2.5. PSUVS 

A Machine PSUVS is defined as follows: If the test set T0 is not available, suppose the validation error of N(ai, wj ) is ei,j on the validation set V , find the luckiest 

network N(ai∗ , wj∗ ) so that it reaches the error of the luckiest-architecture and the 

luckiest initial weights from Post-Selection on Validation Sets: 

ei∗,j∗ = min min ei,j (6)

1≤i≤k 1≤j≤n

and report only the performance ei∗,j∗ but not the performances of other remaining 

kn − 1 trained neural networks. 

355 Similarly, a human PSUVS is a procedure wherein a human selects a system from 

multiple trained systems for {ei,j} using human visual inspection of internal representations of the system and their validation errors. 

2.6. Cross-Validation 

The above PSUVS is an absence of cross-validation [72]. Originally, the cross360 validation is meant to mitigate an unfair luck in a partition of the dataset D into a 

training set T and a test set T0 (empty validation set). For example, an unfair luck is 

such that every point in the test set T0 is well surrounded by points in the training set 

T. But such a luck is hardly true in reality. 

To reduce the bias of such a luck, an n-fold cross-validation protocol, n ≥ 2, 

divides the data set D into n subsets of same size and conducts n experiments. The term 

“cross” refers to switching the roles of training and testing data. In the i-th experiment, 

the i-th subset is left out as the test set and the remaining n − 1 folds of data form 

the training set. Thus, the cross-validation protocol conducts n experiments, for i =

1, ..., n, to obtain n errors, e1, e2, ..., en. The cross-validated error is defined as the 

15 
average of errors from the n tests, to filter out the partition lucks: 

Xn 1

e ̄ = ei (7) 

n

i=1

as well as the distribution of errors {ej}. 

365 The n different numbers here shows a distribution show a distribution {ej} to indicate how sensitive the error is to lucks, such as the number of partition pairs between 

a training set and a validation/test set, the number of tried random seeds for initializing network weights, the number of tried hyper-parameter vectors, or a combination 

thereof. The larger the n, the better the estimated standard deviation of {ej}. 

370 2.7. Types of lucks in a Neural Network 

In a neural network, there are at least three kinds of lucks: 

Type-1 order lucks: The luck in a partition Pi into a training set Ti and a test 

set Ti

0 from a data set D resulting in test error ei, i = 1, 2, ...n. Different partitions 

correspond to different luck outcomes. This kind of outcome variation results in a 

375 variation of performance from different outcomes. Conventionally, this type of lucks 

is filtered out by cross-validation (e.g., n-fold cross-validation) as well as reporting 

the deviation of {ei} during the cross-validation. However, such cross-validation and 

deviation have hardly published for neural networks and reported. The smaller the 

average e ̄ of {ei}, the more accurate the trained network is; the smaller the standard 

380 deviation of {ei}, the more trustable the average error e ̄ is. 

Type-2 weights lucks: As we will discuss below, weights specify the role assignment for all the neurons in the neural network. A random seed value determines the 

initialization of a pseudo-random number generator, which gives initial weights wi for 

a neural network N(wi), resulting in a test error ei, i = 1, 2, ...n, after training of 

385 these n networks and testing on T0. It is unknown that such a luck will be carried 

over to a new test set T00 that is outside the data set D but was drawn from the same 

distribution of S. Because a neural network might not capture the internal rules of the 

training set T, this paper argues that a statistical validation of the reported error should 

be performed by reporting the distribution of {ei|i = 1, 2, ...n}, where ei is from a dif390 ferent initial weight vector wi. For example, Krizhevsky et al. [71] reported 60 million 

16 
parameters, mostly in wi but only the luckiest ei was reported. The smaller the average 

e ̄ of {ei}, the more accurate the trained network is; the smaller the standard deviation 

σ of {ei}, the less sensitive the trained neural network is to the initial weights and thus 

the accuracy is more trustable for real applications. For i.i.d. (identically independently 

395 distributed) errors, we can expect that doubling the number n will reduce the expected 

√ variance of e ̄ by a factor 1/ 2., since the expected variance of n random numbers is 

about σ2/n. 

Type-3 architecture lucks: The initial hyper-parameter vector aj of the neural 

network gives an error ej , j = 1, 2, ...k. Because such a luck of aj might not capture 

400 the internal rules of the training set Tj , this paper argues that a statistical validation 

of the reported error estimate should be performed and the distribution of {ej} be 

reported. In our above example, the number of distinct hyper-parameter vectors to 

310 be tried is k = = 59049. The smaller the average e ̄ of {ej}, the more accurate 

the trained network is; the smaller the sample variance of {ej}, the more trustable e ̄

405 is, namely, the average error e ̄ is less sensitive to the initial hyper-parameters of the 

network. For example, the threshold d of the nearest neighbor classifier in Definition 4 

might result in a large deviation. A good way is to reduce the manual selection nature 

of such hyper-parameters. For example, all hyper-parameters are adaptively adjusted 

from the initial hyper-parameters that are further automatically computed from system 

410 resources, e.g., the resolution of a camera, the total number of available neurons, and 

the firing age of each neuron [67]. 

For notation clarity in the discussion that follows, index j is used in Type 3 to 

distinguish index i in type 2, but the above three types of lucks are all different. 

Let us discuss the case of a developmental network, such as Cresceptron [16] and 

415 DN [41]. Type-1 cross-validation is not needed because of reporting of a lifetime error. 

In other words, errors of all new tests in each life are taken into account throughout 

the lifetime. Type-2 validation is not needed because all different random weights wi

leads to the function-equivalent neural network under certain conditions. For example, 

in top-k competition, with k = 1 different wi give the exactly the same neural network 

420 and with k > 1 different wi give almost the same neural network. The distribution 

of lifetime errors {ei} is expected to have a negligible deviation across different initial 

17 
weight vectors wi, given the same Three Learning Conditions. Type-3 validation might 

be useful but is expected to be negligible since the most obvious parameters such as 

learning rate and momentum of learning rate is automatically and optimally determined 

425 by each neuron, not handcrafted, as in LCA [67]. The synaptic maintenance automatically adjusts all receptive fields [73, 74] so that the neural network performance is not 

sensitive to the initial hyper-parameters. 

In contract, a batch-trained neural network typically uses a Post-Selection to pick 

the luckiest network without cross-validation for either of the above three types of 

430 lucks, e.g., in ImageNet Contest [61]. Namely, errors occurred during batch training 

of the network before the network is finalized and how long the training takes are not 

reported. Below, Fig. 5 will show a huge difference between the luckiest CNN with 

error-backprop and the optimal DN. Many researchers have claimed error-backprop 

works without providing much-needed three types of validations. 

435 Next, let us discuss Types 2 and 3 validations which are new for neural networks 

but hardly done. 

2.8. Post-Selection with Types 2 and 3 Average-Validations 

Type-1 cross-validation should be nested inside the Types 2 and 3 validations, but 

this triple-nested protocol could be too computationally expensive. Below, we delay 

440 Type-1 cross-validation till after Type 2 and Type 3 validations. 

Assume that we use n random weight vectors wi and k grid-search hyper parameters aj . Each combination of wi and aj gives an error ei,j from the corresponding 

validation set. To reduce the effect of such a luck for each vector wi, an average of ei,j

over n values of i should be used instead of the minimum in Eq. (6). This leads to the 

random-weights validated error for the luckiest architecture from PSUVS: 

Xn 1 ∗ a = arg min ei,j . (8)

1≤j≤k n

i=1

We dropped the term “cross” because this validation examines other random seeds 

without switching the roles between training and testing. 

Similarly, we define the hyper-parameter validated luckiest initial weights from 

18 
PSUVS: 

X

k 1 w∗ = arg min ei,j . (9)

1≤i≤n k

j=1

We dropped the term “cross” for the same reason. 

∗ From a statistical point of view, the initial hyper parameter vector a and the ran445 dom initial weights w∗ validated above through averages should be more robust in real 

applications than those without average-validation in Eq. (6). 

∗ ∗ For both the luckiest a and w , the standard deviation under min should be reported to show how sensitive the reported performance is to the validation process. If 

the variation is large, the corresponding network is not very trustable in practice. 

450 We also need to be aware of another protocol flaw: Random seeds and hyper parameters are all coupled. Under such a coupling, Type 2 validation seems unnecessary 

with n = 1 but the search of the luckiest weights is embedded into the search for the 

luckiest hyper-parameter vector where each hyper parameter vector uses a different 

seed. Similarly, Type-3 validation seems unnecessary with k = 1 but the search of the 

455 luckiest hyper-parameter vector is embedded into the search for the luckiest weights, 

where each random seed uses a different hyper parameter vector. 

Since a PSUVS procedure picks the best system based on the errors on the validation set, the resulting system might not do well on the test sets because doing well on 

a validation set does not guarantee doing well on a test set. Typically, due to a very 

460 large number of samples, availability of validation sets and unavailability of test sets 

in a properly managed contest, principles of Post-Selection should cause the validation 

error rate to be smaller than the test error rate. (However, in Table 2 of [71], the test error rate is smaller than the validation error for 7CNNs, causing a reasonable suspicion 

that PSUTS could be used instead of PSUVS.) 

465 The following subsection discusses the luckiest network with the luckiest hyper-

∗ ∗ parameter vector a and the luckiest initial weights w . 

2.9. The Luckiest Network from a Validation Set 

Many people may ask: Are there any technical flaws in at least PSUVS, since it 

does not use the test sets? We analyze the luckiest network in this section and reach 

19 
470 a conclusion that any post-selection is technically flawed and results in misleading results, including both PSUVS and PSUTS. However, in general, Type-1 cross-validation 

is to filter out lucks in data partition that a typical user does not have during a deployment of the method. Namely, it is a severe technical and protocol flaw in reporting only 

the luckiest network, regardless the post-selection uses validation sets or test sets. 

475 This conclusion has a great impact on evolutional methods that often report only 

the luckiest network, instead of those of all networks in a population. Namely, the performances of all individual networks in an evolutionary generation should be reported. 

For simplicity, we assume that the space S, from which random samples in D are 

drawn, is static. Our conclusions here can be readily extended to a time varying D but 

480 the technical flaws are even worse. 

From the sample space S, randomly draw a data set D. D is partitioned into three 

mutually disjoint sets, training set T, validation set V and test set T0

, so that Eq. (5) 

holds true. For realistic applications, we should assume that T, V and T0 are mutually 

independently drawn from S so that T, V and T0 are mutually independent. Identically 

485 independently distributed (i.i.d.) is a sufficient condition, but we do not need such a 

restrictive condition because temporal-dependency often occurs in lifetime development. Namely, we only need that any three vectors from T, V and T0

, respectively, are 

mutually independent. 

Using the training set T, one trains kn networks, where k and n are the number of 

hyper-parameter vectors a’s and random weight vectors w’s, using a training algorithm 

(e.g., error-backprop ), 

N(ai, wj ) ← fai,wj (T). (10) 

This is like a teacher trains kn students in a class. The teacher knows that the fitting 

490 error on T does not predict the validation error well, due to the possibility of over 

fitting. One extreme example is the above nearest-neighbor classifier with confidence 

d = 0. 

The teacher then tests each N(ai, wj ) on the validation set V to get ei,j . This is 

like the teacher observes the performance of kn networks in a mock exam. 

495 The teacher then post-selects and reports only the luckiest network N(ai∗ , wj∗ )

20 
whose validation error ei∗,j∗ is minimum in Eq. (6). This is like the teacher colludes 

with the Educational Test Service (ETS) so that the ETS only reports the luckiest network but not all remaining kn − 1 networks to cover up. 

2.10. Luckiest Network with Type-1 Cross-Validation 

500 Suppose that a user has bought the luckiest network N(ai∗ , wj∗ ) and test on his 

new test data T0 randomly drawn from S, independent of T and V . The luckiest network N(ai∗ , wj∗ ) that reached the minimum error rate in V does not mean that it 

reaches the minimum error rate in T0

. Because T0 is independent of T and V , and 

ai∗ , wj∗ are luckiest on a particular pair (T, V ) only, we need to compute the expected 

505 error rate of N(ai∗ , wj∗ ) on T0. 

Theorem 3 (Type-1 cross-validation of the luckiest). The luckiest network on validation set gives an error rate that is approximately the average error in Type-1 crossvalidation, supposing that, in an n-fold cross validation, n folds of data are drawn i.i.d. 

(independently and identically distributed) among folds from data set D, but individual 

510 samples inside each fold do not need to be i.i.d. 

Proof: Let F denote the event that both the training set and validation set are from 

a fixed data set D from S. Consider in a real application, n tests were conducted on 

the luckiest network N(ai∗ , wj∗ ) using Ti

0

, i = 1, 2, ..., n, where each partition Pi =

(Ti, Vi, Ti

0

) in each of the i-th training and test pair is drawn from the real application 

515 space S. We compute the average error rate from the luckiest network N(ai∗ , wj∗ ): 

Xn 1

e(ai∗ , wj∗ ) = ei(ai∗ , wj∗ ; Ti, Vi, Ti

0

) n

i=1

Xn 1 ≈ ei(ai∗ , wj∗ ; Ti, Vi, Ti

0

|F) (11) 

n

i=1

where the term ei(ai∗ , wj∗ ; Ti, Vi, Ti

0

) means the error of the luckiest network using 

training set Ti, validation set Vi, and test set Ti

0

, and ei(ai∗ , wj∗ ; Ti, Vi, T0|F) means i

the same but Ti, Vi, T0 are all from the same D as one does in n-fold cross-validation. i



21 
520 Note, the n-fold i.i.d. is weaker than i.i.d. for all samples. In practice, i.i.d. is 

rarely true even for pattern recognition problems, such as image classification due to 

sequential attention discuss above. Also note that the left side of ≈ sign in Eq. (11) is 

expected larger because the data Ti, Vi, T0 on the right side are all from a fixed D but i

the left side does not have such a restriction. 

525 The above theorem tells us that the error rate of the luckiest network from a single 

validation set in PSUVS is misleading without any partition validation. This is because the error rate is a random function, depending on not only many random initial 

weights, many hyper parameters, and local lucks of error-backprop, but also a particular partition (T, V, T0

). This seems especially true if the data D were made public and 

530 overworked during 2010-2014 [61, p. 213]. 

In practice, when we report an error rate e(ai∗ , wj∗ ) which is always a random 

number x, depending on how much hand tuning is done, how much computational 

resources are used for a large-scale search for the random seeds and hyper-parameters, 

as well as the validation or a lack thereof. We should also report the distribution of this 

535 random number x, such as the maximum, 75%, 50%, 25%, and the minimum value 

of x, over multiple training-and-test pairs in cross-validation, random seeds and hyperparameters. Otherwise, the error rate, if only as a single number x, is misleading, since 

users of this learning method or buyers of the luckiest network do not have the same 

partition luck. 

540 Up to now, this author has not found any published papers that report not only the 

luckiest network from error-backprop but also Type-1, Type-2 and Type-3 validations. 

Many papers do not report the post-selection stage at all [24, 25, 26, 28, 29, 30, 31, 

32, 33, 34, 35, 36, 37, 38], except [27], let alone whether the reported error is from the 

validation error V or the test set T0

. 

545 Next, we discuss Post-Selections Using Test Set (PSUTS). There are two kinds of 

PSUTS, machine PSUTS and human PSUTS. 

2.11. Machine PSUTS 

If the test set T0 is available which seems to be true for almost all neural network publications other than competitions, we define Post-Selection Using Test Sets 

22 
550 (PSUTS): 

A Machine PSUTS is defined as follows: If the test set T0 is available, suppose the 

0 test error of N(ai, wj ) is ei,j on the test set T0

, find the luckiest network N(ai∗ , wj∗ )

so that it reaches the minimum, called the error of the luckiest architecture and the 

luckiest initial weights from Post-Selection on Test Set: 

0 0 ei∗,j∗ = min min ei,j . (12)

1≤i≤k 1≤j≤n

0 Report only the performance e but not the performances of other remaining kn−1 i∗,j∗

trained neural networks. 

Imagine that we want to remove lucks in the above expression, by using averages 

like we did in Eq. (7) to give the error of the luckiest architecture with validated weights 

from PSUTS: 

Xn 1 0 a∗,j∗ = arg min ei,j . (13)

1≤j≤k n

i=1

But the above error is still flawed since each term under minimization has peeked into 

test sets. Instead, it is better to use Eq. (8) which does not use the test sets. Of course, 

555 the test error rate of that in Eq. (8) tends to be larger than that from Eq. (13). 

A similar discussion can be made for the error of the luckiest initial weights with 

validated architecture from PSUTS. Do not peek into test sets. 

There are some variations of Machine PSUTS: The validation set V or T0 are not 

disjoint with T. If T = V , we call it validation-vanished PSUTS. If T = T0

, we called 

560 it test-vanished PSUTS. 

In general, the more free parameters a network has, the more likely the network can 

report an artificially small error as in Eq. (12). That is why we need the computational 

resource in the Three Learning Conditions. 

Although PSUVS has flaws of post-selection and a lack of three types of validation, 

565 the key difference between PSUVS and PSUTS does not guarantee that PSUVS reports 

a low error rate as PSUTS. In fact, it is expected that the luckiest network from PSUVS 

does better on a validation set V than on a test set T0 because the Post-Selection did 

not “see” the test set T0 but “saw” the validation set V . Likewise, it is expected that 

the luckiest network from PSUTS does better on the test set T0 than on a validation set 

23 
570 V because the Post-Selection did not “see” the validation set V but “saw” the test set 

T0

. In the following paragraph, we discuss that this expectation is reversed in Table 2 

of [71, page 88]. 

In ImageNet Contest 2012, the test sets were released to competition teams over 

2.3 months ahead of the output-result submission date. Although the class labels were 

575 not attached to the test sets other than being available indirectly through an online test 

server provided by the contest organizers, it was not difficult to “crack” a test set by 

manually hand-labeling the test set. The first author of [71] seems not sensitive to 

the fundamental difference between a validation set and a test set by writing: “in the 

remainder of this paragraph, we use validation and test error rates interchangeably”. 

580 By “we cannot report test error rates for all the models that we tried” [71, page 88], 

there is no evidence to rule out what he meant was the possibly “cracked” test set is not 

necessarily exactly the same as the original test set. But in Table 2 of [71, page 88], 

the 7NNs did worse on the validation set (possessed) than the test set (if not “cracked” 

and searched for minimization like in Eq. (12)). This reversed our expectation in the 

585 previous paragraph. Is it an evidence of using PSUTS instead of PSUVS? 

Another interesting phenomenon that is consistent with the likely use of PSUTS 

instead of PSUVS is that the SuperVision Team of ImageNet Contest 2021 did not 

submit any output results for “the fine-grained classification task, where algorithms 

would classify dog photographs into one of 120 dog breeds” [61, footnote, p214]. It 

590 appears that cracking “120 dog breeds” is harder than cracking “a list of object categories present in the image” where the class labels are all available in the provided 

training sets. [71] lacks due transparency about the post-selection stage except that Geoffrey Hinton admitted the “luckiest” network in his brief PubPeer response to questions raised on PubPeer towards [24]. 

595 For more examples, see Fig. 1 from [75, Fig. 7], error-backprop consistently results lower validation accuracies than the test accuracies (about 0.5% lower compared 

to about 0.1% lower in [71]). Are they other evidences of using PSUTS instead of 

PSUVS, similar to [71]? The availability of test sets to the programmers in a project 

seems to be indeed addictive towards PSUTS, away from PSUVS. The standard de600 viation around 1% is clashes with our Theorem 4. Our experience with our own ex24 
Figure 1: The average e ̄ and the standard deviation of {ei} for different values of a regularization hyper 

parameter α. Adapted from [75] 

periments with error-backprop training for CNN indicated that the maximum and the 

minimum values of the distribution of fitting accuracies are drastically different for 

different random seeds, with fitting accuracies spreading uniformly between 20% and 

90%. Section 3 will discuss why. If Theorem 4 in Section 3 is correct, the deviation 

605 bars seem too small and the 20 runs in Fig. 7 of [75] could be the best 20 among many 

more random-seeds the programmer has tried. We hope that authors provide the source 

program. 

2.12. Implications of PSUTS 

Although the set {ei,j |i = 1, 2, ...n; j = 1, 2, ...k} is large, it is necessary to 

610 present some key statistical characteristics of its distribution. For example, rank all 

errors in decreasing order, for each type of errors, fitting, validation and test. Then 

give the maximum, 75% (in ranked population), 50% (median), 25%, the minimum 

value, and the standard deviation of these kn values for the fitting errors, validation errors. and test errors, respectively, not just the standard deviation in Fig. 1 . Such more 

615 complete information of the distribution is critical for the research community to see 

whether error-backprop can indeed avoid local minima in deep learning as some authors claimed. Furthermore, such information is also important for the authors to show 

that the luckiest hyper-parameter vector is not just an over fitting to the validation/test 

set. Unfortunately, none of [23, 66, 24, 25, 27, 76, 29, 31] reported such distribution 

25 
620 characteristics other than the minimum value e0

i∗,j∗ . 

Furthermore, such a use of test sets to post-select networks resembles hiring a larger 

number kn of random test takers and report only the luckiest N(ai∗ , wj∗ ) after the 

grading. This practice could hardly be acceptable to any test agencies and any agencies 

0 that will use the test scores for admission purpose since this submitted error ei∗,j∗

625 misleads due to its lack of validation. 

The error-backprop training tends to locally fit each network on the training set 

T; while the Post-Selection picks the luckiest network with parameter vector ai∗ and 

initial weights wj∗ that has the best luck on T0

. If an unobserved data set T00, disjoint 

, T0 ∩ T00 00 with T0 = ∅, is observed from the same distribution S, the error rate e of i∗,j∗

N 0 (ai∗ , wj∗ ) is predicted to be significantly higher than ei∗,j∗ , 

00 0 ei∗,j∗  ei∗,j∗ (14) 

because Eq. (12) depends on the test set T0 in the post selection from many networks. 

Of course, handling a new test is also challenging for a human student but a human 

learning involves learning invariant rules. Any PSUTS is a technically flawed protocol. 

PSUTS is tempting especially when test sets are available to the authors of paper. 

630 During all error-backprop related paper reviews I have not yet found a case in which 

the authors did not admit that they used USUTS when I asked. The second author of 

[27] claimed to have used PSUVS through a personal email to me but the first author 

who probably performed the experiments did not claim the same. No authors of [27] 

responded to PubPeer questions towards [27]. 

635 Weng 2020 [77, 78] argued that the claims by some public speakers that such misleading errors have approached or even succeeded human performance [61] are controversial, since there are no explicit competition rules that ban test sets to be used for 

Post-Selections. 

2.13. Human PSUTS 

640 Instead of writing a search program in machine PSUTS, human PSUTS defined 

below typically involves less computational resources and programming demands. 

26 
Definition 6 (Human PSUTS). After planning experiments or knowing what will be 

in the training set T and test set T0

, a human post-selects features in networks instead 

of using a machine to learn such features. 

645 Unfortunately, almost all methods in the symbolic school use human PSUTS because it is always the same human who plans for and design a micro-world and collect 

the test set T0

. The key to an acceptable test score lies in how much detail the human 

designer can plan for what is in the test sets and how much freedom s programmer has 

in hand picking features. 

650 Poggio et al. [79] and Fukushima et al. [17] explicitly admitted their use of human 

PSUTS. Li Fei-Fei at al. [19] only vaguely admitted their use of human PSUTS by a 

vague term “weakly supervised” using an extension of formulation by Pietro Perona 

that is originally unsupervised. Questions raised towards [19] on PubPeer were not 

answered by the authors. 

655 3. Why Error-Backprop Needs PSUTS 

This section discusses a global view, which is new as far as the author is aware, 

about why error-backprop suffers from local minima even in the easier batch-learning 

mode. 

Since error-backprop does not perform well for incremental learning mode as we 

660 can see why also from the following discussion, we will concentrate on batch learning 

mode. Namely, we let the network “see” the entire training set T for each network 

update. 

Let us first consider a well-known neuronal model that is applicable to many CNNs. 

Suppose a post-synaptic neuron with activation zj is connected to its pre-synaptic neurons yi, i = 1, 2, ..., n, through synaptic weights wij , by the expression: 

Xn

φ( wijyi) = zj (15) 

i=1

1 where φ(y) = is the logistic function. The gradient of zj with respect to weight 1+e−y

vector wj = (w1,j , w2,j , ..., wn,j ) is 

η(y1, y2, ..., yn) , ηy

27 
where η is the partial derivative of φ(y). Thus, according to gradient direction, the 

change of the weight vector wj is along the direction of pre-synaptic input vector y. If 

the error is negative, zj should increase. Then the weight vector should be incremented 

by 

wj ← wj + w2y (16) 

where w2 is the learning rate. We use the w2 to relate better the optimal Hebbian 

learning, called LCA, used by DN in Section 4. At this point, the following theorem is 

665 in order. 

Theorem 4 (Lacks of error-backprop). Error-backprop lacks (1) energy conservation, (2) an age-dependent learning rate, and (3) competition based role-determination. 

Proof: Proof of (1): If pre-synaptic input vectors {y} are similar, multiple applications of Eq. (16) add many terms of {w2y} into the weight vector wj causing it 

670 to explode, which means a lack of energy conversation. Proof of (2): w2 is typically 

tuned by an ad hoc way, such as a handpicked small value turned by a term called 

momentum, instead of being automatically determined in Maximum Likelihood optimality (ML-optimality) by neuronal firing age to be discussed in Section 4. Proof of 

(3): Suppose neuron zj is in a hidden area of the network hierarchy. This neuron zj

675 updates its pre-synaptic weight using Eq. (16) regardless zj is role-responsible or not 

for the current network error. Likewise, looking upstream, there is also a lack of roledetermination in the gradient-based update for pre-synaptic neurons y1, y2, ..., yn, all 

of which must update their own weights using their own gradients. Namely, there is no 

competition-based role-determination in error-backprop . 

680 The meaning (3) of Theorem 4 are illustrated by Fig. 2. CNNs do not have a 

competition mechanism in any layers. Complete connections initialized with random 

weights are provided for all consecutive areas (also called layers), from input area all 

the way to the output area. If the zj neuron is in the output motor area and each 

output neuron is assigned a single class label, the role of zj (“dog” in the figure) is 

685 determined by human supervised label “dog”. However, let us assume instead that 

zj is in a hidden area, not responsible for the “dog” class. zj still updates its input 

weights using the gradient. Likewise, the pre-synaptic area Y , is characterized by its 

28 
Neurons

without

competion

Past

inputs

Bird Cat Dog

Current

input

Figure 2: Lack of role-determination in hidden neurons due to a lack of competition. The same ideas are true 

for a deeper hierarchy. Color sample images courtesy of [61]. 

label “neurons without competition”. The hidden neurons in this area do not have a 

competition mechanism which would, like in LCA [67], allow a small proportion of 

690 neurons to win the competition and fire so that they automatically take the roles that 

they happened to compete well. This analysis leads us to the following theorem. 

Theorem 5 (Random roles in error-backprop ). A set of random initial weights in 

a network assigns random roles to all hidden neurons, from which a local minimal 

point based on error-backprop learning inherits this particular random-role assign695 ment. Which neurons in each hidden area take a role does not matter, but how hidden 

neurons share a set of roles in each hidden area does matter in the final fitting error, 

validation error, and test error after error-backprop. 

29 
Proof: Without loss of generality, suppose a maximum in the output neuron means 

a positive classification and weights take positive and negative values. Then, a positive 

700 weight to an output neuron zj from a hidden neuron yi means an excitatory role of yi to 

zi and a negative weight means an inhibitory role. A zero weight means an irrelevant 

role. The gradient vector computed in Eq. (16) means such excitatory-inhibitory input 

patterns from pre-synaptic neurons are added through iterative error-backprop procedures. Because of the complete connections and an identical neuronal type, where a 

705 hidden neuron is located in the Fig. 2 does not matter, but each input image must have 

a sufficient number of hidden neurons in every hidden area to excite for its signals to 

reach the corresponding output neuron. The initial role assignment patterns in initial 

weights do matter for the final the fitting error rate, the validation error rate, and the 

test error rate, because gradient updates are local and inherited such initial roles. 

710 Theorem 6 (Percentage luck of error-backprop ). Suppose a CNN has l > 1 areas, 

A0, A1, ..., Al, connected by a cascade or a variation thereof. A0 takes input frames 

{x ∈ X} and Al is the output area for classification. Suppose an area has a total 

of m hidden neurons that share a common receptive field R in A0. Consider a given 

input frame x. Let the percentage of the m hidden neurons that do not fire among all 

715 neurons in the same area with the same receptive field be denoted as p(x). Then, the 

error-backprop depends on the average p ̄ = Ex∈X{p(x)} to be a reasonably small 

value, called the percentage luck. 

Proof: To guide the proof, we should mention that DNs use top-k competition so 

that each receptive field in each area has only k neurons that fires, where k is small, 

720 e.g., top-1, for each receptive field R. Suppose a receptive field R represents a neuronal 

column that has n neurons. A neuronal area at level l is denoted as Al. Every receptive 

field image x ∈ X = A0 is concrete by which we means that its neurons are only pixels 

{x} of a concrete example of a class C with p ̄(x) = P(x fires) ≈ 50% (e.g., 50% back 

and 50% white). Each neuron z in area Al is abstract by which we means that it fires 

725 means an abstract class C that x belongs to, with p ̄(z) being small corresponds to top1 among n neurons. Then, it is necessary for the CNN to convert the most concrete 

representation of pixels in A0 to more abstract representations in Al, l > 0 with a low 

30 
p ̄(z). For example, in Fig. 2, we have l = 2 and there is no completion in the hidden 

area A1. Then error-backprop depends on that each neuron in A2 has only a relatively 

730 smaller percentage among n = 6 neurons in A1 that are positive, i.e., as the features of 

its particular class. The requirement of being a small percentage is due to the need for 

other non-firing neurons to deal with many other patterns in the same receptive field. 



As we can expect, such a low percentage condition is rarely satisfied by a random 

735 weight vector. The more random weight vectors one uses, the better chance to hit the 

luck. 

From Theorems 4 through 6 and their proofs, we can see that the luck of role 

assignment is a critical flaw of error-backprop , and so are the system parameters and 

the simple-minded regularization of the learning rate. Because of these key reasons, 

740 PSUTS plays a critical role to select the luckiest network from many unlucky ones after 

error-backprop . The more networks have trained by error-backprop , the more likely 

the luckiest one has a good role-assignment to start with. 

There has been no lack of papers that claim to justify error-backprop does not 

over fit, e.g., variance based stochastic gradient decent [80], saddle-free deep network 

745 [81], drop out [82], implicit regularization during gradient flow [83]. They all address 

only local issues of neural networks trained by error-backprop and did not mention 

Post-Selections. The theory here addresses, the global role-assignment problem of 

random weights that no local mechanisms can deal with. This seems to be why PSUTS 

is necessary by error-backprop , but PSUTS is controversially fraudulent in terms of 

750 protocol—test sets are meant to test a reported system, not supposed to be used to 

decide which network to report from many. 

4. How a DN Avoids Post-Selections 

Apparently, a brain does not use Post-Selection at al, whether UPSVS or PUUTS, 

because every human child must develop in a human environment to make his living. 

755 He should not be covered up and not reported, regardless how well or bad he performs. 

Cresceptron in the 1990s [42, 43, 16, 44, 45] and later DN [46, 84, 57, 58] were inspired 

31 
760

by the interactive mode that brains learn though lifetime. In other words, Cresceptron 

and DN do not need Post-Selections. Furthermore, every DN must be ML-optimal 

given the same Three Learning Conditions. 

4.1. New AI Metrics: Developmental Errors 

In contrast to Post-Selections likely used by [21, 14, 19, 79, 23, 66, 24, 25, 27, 

76, 29, 31] including probably AlphaGo [26], AlphaGo Zero [28], AlphaZero [85], AlphaFold [30] and MuZero [86] and many others, we define and reported developmental 

errors that includes all errors occurred through lifetime of each learning network: 

Definition 7 (Developmental error). A Developmental Network is denoted as N =

(X, Y, Wy,Z, Wz, A) with sensory area X, skull closed hidden area Y and its weight 

space Wy, and motor area Z and its weight space Wz, and the space of architecture 

parameters A, where X, Y , and Z also denote the spaces of responses of X, Y and 

Z areas, respectively. The space of architecture parameters A includes all remaining 

parameters and memory of the network, other than neuronal weighs, such as ages of 

neurons (for learning rates), neuronal patterning parameters (location and receptive 

fields adapted by synaptic maintenance), neuronal types (for initial connection absences among areas), and neuronal growth rates (for speed of mitosis). It runs through 

lifetime by sampling at discrete time indices as N(t), t = 0, 1, 2, ... . Start at inception 

t = 0 with supervised sensory input x0 ∈ X(0), initial state z0 ∈ Z(0), randomly 

initialized weigh vector y0 ∈ Y (0), initial architecture a0 ∈ A(0). At each time t, 

t = 1, 2, ..., the network N(t) recursively and incrementally updates: 

(xt, yt, zt, at) = f(xt−1, yt−1, zt−1, at−1) (17) 

where f is the Developmental Program (DP) of N. If zt ∈ Z(t) is supervised by the 

teacher, the network complies and the error et is recorded, but if the supervised motor 

vector has error, the error should be treated as teacher’s. Otherwise, the learner is not 

motor-supervised and N(t) generates a motor vector zt and is observed by the teacher 

∗ and its vector difference from the desired z is recorded as error et. The lifetime t

average error for each motor concept or component, from time 0 up to time t is defined 

32 
as 

Xt 1

e ̄(t) , ei, (18) t

i=0

which is computed incrementally in terms of average developmental error e ̄(t): 

t − 1 1

e ̄(t) = e ̄(t − 1) + et. (19) t t

765 Namely, all errors across a lifetime, at every time instance, are caught by the developmental error. In order to reach a small error, a low final error rate that a batch 

learning method tries to reach is not sufficient. Instead, the network must learn as fast 

as possible and avoid errors as much as possible at every time instance t. This is indeed 

important since earlier performance will shape later learning. 

770 An optimal network that gives the lowest possible developmental error, among all 

possible networks under the same Three Learning Conditions, must be optimal at every 

time instance t throughout its life. DN is one such network. Post-Selections are useless among neural networks that give the smallest developmental error under the same 

Three Learning Conditions, because the maximum-likelihood optimality should give 

775 equivalent networks of the same developmental error. 

However, in practice, the learning experience in the Three Learning Conditions is 

unlikely the same among different networks, because each physical robot that runs a 

network at least occupies distinct physical locations in the real world. For example, 

if two physical robot in the same family fight for a toy, the winner gains a winner 

780 experience and the loser may acquire a loser mentality. In other words, even if the 

parents of two boys are not biased toward any boys, the competition among the boys 

results in different learning experiences. 

The developmental error is important. If a competition is based on developmental 

errors (such as during AIML Contests [62]), the winner is unlikely be one that uses a 

785 brute force method but has an excessive amount of computational resources and manpower. ImageNet competitions [61] are flowed also in this sense. 

Although not formally defined as developmental errors, Cresceptron [16] and Developmental Networks [41, 87, 88] reported developmental errors. 

Namely, the developmental error, unless stated otherwise for a particular time pe790 riod, is the average lifetime error from inception. To report more detailed information 

33 
about the process of developmental errors {et|t ≥ 0}, statistics other than the mean 

(average) can be utilized, such as the minimum, 25%, 50% (median), 75%, the maximum, and the standard deviation. 

For more a specific time period, such as the period from age t1 to age t2, the average 

795 error is denoted as e ̄[t1 : t2]. Therefore, e ̄(t) is a short notation for e ̄[0 : t]. 

Because Cresceptron and DN have a dynamic number of neurons up to a system 

memory limit, each new context 

ct , (xt, yt, zt) (20) 

may be significantly different from the nearest matched learned weight vectors of all 

hidden neurons. If that happens and there are still new hidden neuron that have not 

fired, a free-state neuron that happens to be the best match is spawned that perfectly 

memorizes this new context regardless its randomly initialized weights. When all the 

800 free neurons have fired at least once, the DN will update the top-k matched neurons 

optimally in the sense of maximum likelihood (ML), as proven for DN-1 by [41] and 

for DN-2 by [87], as we will discuss below. 

Note that a developmental system has two input areas from the environment, sensory X and motor Z. That is, motor Z is supervisable by the world (including teachers) 

805 but not often. Since there is hardly any sensory input x ∈ X that exactly duplicates 

at two different time indices, almost all sensory inputs from X are sensory-disjoint. 

During motor-supervised learning, if the teacher supervises its motor area Z and the 

learner complies. Since a teacher can make an error, the motor-error that the teacher 

made is also recorded as the developmental error of the motor of the learner but due to 

810 the teacher. 

4.2. Neuronal Competitions 

As discussed above, error-backprop learning is without neuronal competitions. The 

main purpose of competition is to automatically assign roles to hidden neurons. Below, 

we consider two kinds of Convolution Neural Networks (CNNs), sensory networks and 

815 sensorimotor networks. A sensory network is feedforward, from sensor to motor, in 

computation flow and therefore is simpler and easier to understand. A sensorimotor 

34 
Neurons

with top-1

competion

Neurons

with top-1

competion

Past

inputs

Current

input

Bird Cat Dog

I win!

0.4 0.3

0.3

0.3

0.2 0.4

0.2 0.6

0.3

Neurons

without

competion

Past

inputs

More!

More! More!

More!

More!

Bird Cat Dog

Current

input

Completely

not their job!

(a) (b)

Figure 3: How competition automatically assigns roles among hidden neurons without a central controller: 

The case for automatically construct a mapping f : X 7→ L. (a) The number of samples in X is larger 

than the number of hidden neurons such that each hidden neuron must win-and-fire for multiple inputs. (b) 

Error-backprop from the “dog” motor neuron asks some hidden neurons to help but the current input feature 

is not their job. Thus, error-backprop messes up with the role assignment guessed by the random initial 

weights. The same ideas are true for a deeper hierarchy. Color sample images courtesy of [61]. 

network takes both sensor and motor as inputs and is highly recurrent and therefore 

more powerful. 

4.2.1. Sensory networks 

820 Let us first consider the case of feed-forward networks as illustrated in Fig. 3. 

Fig. 3(a) shows a situation where the number of samples in X is larger than the number 

of hidden neurons, which is typical. Otherwise, if there are sufficient hidden neurons, 

each hidden neuron can simply memorize a single sample x ∈ X. 

This means that the total number of hidden neurons must be shared through incre825 mental learning, where each sample image-label pair (x, l) ∈ X ×L arrives incrementally through time, t = 0, 1, 2, .... This is the case with Cresceptron (and some other 

networks) which conducts incremental learning by dealing with image-label pairs one 

at a time and update the network incrementally. 

Every layer in Cresceptron consists of a image-feature kernel, which is very differ830 ent from those in DN where each hidden neuron represents a sensorimotor feature to be 

discussed later. By image-feature, we mean that each hidden neuron is centered at an 

35 
image pixel. Competitions take place within the column for a receptive field centered 

at each pixel at the resolution of the layer. The resolution reduces from lower layer to 

higher layer through was called resolution reduction (also called drop-out). 

835 The competition in incremental learning is represented by incrementally assigning 

a new neuronal plane (convolution plane) where the new kernel memorizes the new 

input pattern if the best matched neuron in a column does not match sufficiently well. 

Suppose images x ∈ X arrives sequentially, the top-1 competition in the hidden layer 

in Fig. 3(a) enables each hidden neuron to respond to multiple features, indicated by the 

840 typically multiple upward arrows, one from each image, pointing to a hidden neuron. 

This amounts to incremental clustering based on top-k competition. The weight vector 

of each hidden Y neuron corresponds to a cluster in the X space. In Fig. 3(a), k = 1

for top-k competition in Y . 

Likewise, suppose top-1 competition in the next higher layer, Y , namely each time 

845 only one Y neuron fires at 1 and all other Y neurons do not fire, resulting the connection 

patterns from the second layer Y to the next higher layer Z. In the output layer Z, top-1 

competition takes place but a human teacher can supervise the pattern. 

The Candid Covariance-free Incremental (CCI) Lobe Component Analysis (LCA) 

in Weng 2009 [67] proved that such automatic assignment of roles through competi850 tion results in a dually optimal neuronal layer, optimal spatially and optimal temporally. 

Optimal spatially means the CCI LCA incrementally computes the first principal component features of the receptive field. Optimal temporally means that the principal 

component vector has the least expected distance to its target—the optimal estimator 

in the sense of minimum variance to the true LCA vector. 

Intuitively, regardless what random weights each hidden neuron starts with, as soon 

as it wins to fire for the first time, its firing age a = 1. Its random weight vector 

is multiplied by the zero retention rate w1 = 1 − 1/a = 0 and this learning rate 

w2 = 1/a = 1 so that the new weight vector becomes the first input rx with r = 1 for 

the firing winner. 

1 1 v ← (1 − )v + rx. (21) 

a a

855 It has been proven that the above expression incrementally computes the first principal 

36 
1 component as v. The learning rate w2 = is the optimal and age-dependent learning a

rate. CCI LCA is a framework for dually optimal Hebbian learning. The property 

1 “candid” corresponds to the property that sum of the learning rate w2 = and the a

retention rate w1 = 1 − 1 is always 1 to keep the “energy” of response r weighted a

860 input x unchanged (e.g., not to explode or vanish). This dually optimality resolves the 

three problems in Theorem 4. 

Fig. 3(b) shows how the three neurons in the Z area updates their weights so that 

the weight from the second area to the third area become the probability of firing, 

conditioned on the firing of the post-synaptic neuron in area Z (Dog, Cat, Bird, etc.). 

865 The CCI LAC guarantees that the sum of weights for each Z neuron sum to 1. This 

automatic role assignment optimally solves the random role problem of error-backprop 

in Theorem 5. 

However, optimal network for incrementally constructing a mapping f : X 7→ L

is too restricted, since f : X 7→ L is only what brains can do, but not all brains can do. 

870 For the latter, we must address sensorimotor networks. 

4.2.2. Sensorimotor networks 

The main reason that Marvin Minsky [5] complained that neural network is scruffy 

was because conventional neural networks lacked not only the optimality described 

above for sensory networks, but also lacked the Emergent Universal Turing Machines 

875 (EUTM) that is ML-optimal we now discuss below. 

First, each neuron in the brain not only corresponds to a sensory feature as illustrated in Fig. 3, but also a sensorimotor feature. By sensorimotor feature, we mean that 

the firing of each hidden neuron in Fig. 3 is determined not just by the current image 

σ represented by a sensory vector x ∈ X, but also the state q represented by a motor 

880 vector z ∈ Z. It is well known that a biological brain contains not only bottom-up 

inputs from x ∈ X but also top-down inputs from z ∈ Z. In summary, each hidden 

neuron represents a sensorimotor feature in a complex brain-like network. 

37 
895

4.3. FA as sensorimotor mapping 

This sensorimotor feature is easier to understand if we use the conventional symbols 

for (symbolic) automata. Let us borrow the idea of Finite Automaton (FA). In an FA, 

transitions are represented by function δ : Q × Σ 7→ Q, where Σ is the set of input 

symbols and Q the set of states. Each transition is represented by 

f 0 (q, σ) −→ q

AFA as a control of any Turing machine. Weng 2015 [41] extended the definition the 

885 FA so that it outputs its state so the resulting FA becomes an Agent FA (AFA). Further, 

Weng 2015 [41] extended the action q to the machinery of Turing machine (see Fig. 4) 

so that action q includes output symbol to the Turing tape and the head motion of the 

read-write head of a Turing machine. With this extension, Weng 2015 [41] proved that 

the control of any Turing machine is an AFA, a surprising result. 

890 Here q ∈ Q is the top-down motor input to a sensorimotor feature neuron; σ is the 

bottom-up sensory input to the same neuron. If δ has n transitions, n hidden neurons 

in the Y area are sufficient to memorize all the transitions that is observed sequentially, 

one transition at a time. 

We should not use symbols like σ and q, but instead sensory vectors x ∈ X and motor vectors z ∈ Z that are emergent as discussed above. At discrete time t = 0, 1, 2, ..., 

we use the hidden neurons in the Y area to incrementally learn the transitions: 

⎤⎡ ⎡⎤⎡ ⎤

⎢

⎢

⎢

⎣

Z(0)

Y (0)

⎥

⎥

⎥

⎦

→

⎢

⎢

⎢

⎣

Z(1)

Y (1)

⎥

⎥

⎥

⎦

→

⎢

⎢

⎢

⎣

Z(2)

Y (2)

⎥

⎥

⎥

⎦

→ ... (22) 

X(0) X(1) X(2)

where → means neurons on the right use the input neurons on the right and compete 

to fire as explained below without iterations. Namely, by unfolding time, the spatially 

recurrent DN becomes non-recurrent in a time-unfolded and time-sampled DN. With 

LCA update, [41] proved that such a DN is ML-optimal and has a constant complexity 

for each update O(1) with a large constant, suited for real-time computation with a 

large memory and many neurons. 

38 
q

Δ Δ x x 11111 Δ Δ Δ

3

Figure 4: A Turing machine has a tape, a read-write head, and a transition function with a current state. 

900 4.4. DN as a ML-Optimal of Emergent Universal Super-Turing Machine 

The traditional Turing Machine (TM) is a human handcrafted machine, as illustrated in Fig. 4. 

A Universal TM (UTM) s still a TM, but its tape contains two parts, a user supplied 

program and the data that the program is applied to. The transition function of the 

905 UTM is designed to simulate any program encoded in the form of transition of a TM 

and to apply the program on the data on the tape and finally to place the output of the 

program on the data onto the tape. 

A UTM is a model of the current general-purpose computers because the user can 

write any program on any set of appropriate data for the UTM to carry out. Because 

910 a DN is an ML-optimal emergent FA, Weng 2015 [41] extended a symbolic Turing 

machine to a super Turing machine by (1) extending the tape to the real world, (2) the 

input symbols to vectors from sensors, (3) the output symbols to vector output from effectors, and (3) the head motion to any action from the agent. Thus, DN ML-optimally 

learns any TM, including UTM, directly from the physical world. The programs on 

915 the tape are learned by the Super UTM incrementally from the real world across its 

lifetime! 

4.5. DN as a ML-Optimal Learning Engine for APFGP and Conscious Learning 

Because DN is an ML-optimal learning engine for any TM, including UTM, DN 

ML-optimally learns any UTM from the physical world, conditioned on those in Defi920 nition 2. This means that a DN ML-optimally learns to Autonomous Programming for 

General Purposes (APFGP) [57, 89]. Based on the capability of APFGP, Weng 2020 

argued that APFGP is a characterization of conscious machines [58] that boots its skill 

of consciousness through conscious learning—being (partially) conscious while learning across lifetime. Hopefully, APFGP is a clearer and more precise characterization 

39 
925 for conscious machines and animals, assuming that we allow a conscious machine to 

develop its degree of consciousness from infancy. 

In the following, we list the DN algorithm so that we can understand APFGP is not 

a vague idea and how APFGP by DN avoids Post-Selections. 

4.6. DN-2 Algorithm 

930 Let us go through the DN-2 algorithm here so that we can see that DN is fully detail 

in computer implementation. 

DN-2 is the latest general-purpose learning engine in the DN family. In DN-1, 

the allocation of neurons in each subarea of the hidden Y area is handcrafted by the 

designer. In DN-2, several biology-inspired mechanisms are added to automatically 

935 allocate neuronal resources and generate a dynamic and fluid hierarchy of internal representations during learning, relieving the human designer from handcrafting a concept 

hierarchy, beyond the rigid hierarchy in deep learning [42, 43, 16, 44, 45, 19, 79, 23, 

66, 24, 25, 27, 76, 29, 31]. Namely, a DN-2 starts with simple internal representations 

which gradually grow to be rich and deep supported by early representations as a “brain 

940 stem”, but it is still ML-optimal conditioned on those in Definition 2. 

Areas from low to high: X: sensory; Y hidden (internal); Z: motor. From low to 

high: bottom-up. From high to low: top-down. From one area to the same area: lateral. 

X does not link with Z directly. 

945 Input areas: X and Z; Output areas: Z; Hidden area: Y , fully closed from t = 0. 

1. At time t = 0, inception. Initialize the X, Y and Z areas. x ∈ X takes the 

first image. Set every Y neuron with random weights, zero firing age, and zero 

response y(0). Set the total number of Y neurons to be nY . A boundary cY indicates the number of active neurons (cY ≤ nY ). Set the Z area and its memory 

950 part MZ similarly, but all concept zones take none vectors if the learner has no 

prenatally learned inborn “reflexes”. 

2. For time t = 1, 2, ..., repeat the following steps forever (executing steps 2a, 2b 

in parallel, before step 2c): 

40 
(a) All Y neurons compute in parallel: 

(y0

, MY

0 ) = fY (cY , MY ) (23) 

where context cY = (x, y, z), MA denotes the memory of area A including 

955 weights and neuronal firing ages, and fY is the Y area function using LCA 

[65, 67]. If the best active Y neurons do not match the input vector well, 

area Y transfers new neurons to active and increment the boundary cY . 

0 (b) Supervise z if the teacher likes. Otherwise, Z neurons compute the response vector z and update memory M0 in parallel: Z

(z0, MZ

0 ) = fZ(cY , MZ) (24) 

where fZ is the Z area function using LCA [65, 67] and cZ = (y, z). 

(c) Replace asynchronously: (y, MY , z, MZ) ← (y0

, MY

0 , z0

, M0 ). Super- Z

960 vise input x. 

The area function fY in Eq.(23) and area function fZ in Eq.(24) include two parts: 

0 (1) The computation of response vectors y0 and z , respectively; (2) The maintenance 

of memory MY

0 and M0 for Y area and Z area, respectively. Z

The ML-optimality of DN-1 and DN-2 is rooted in the optimality of LCA and 

965 extends to the entire network and entire lifetime. 

4.7. Methods for Recursive Optimization 

Given the Three Learning Conditions, at each time t, t = 1, 2, ..., a DN incrementally computes the ML-estimator of its parameters at each time t that minimizes the 

developmental error without doing any iterations. 

Let us first review the maximum likelihood estimator for a batch data. Let x be the 

observed data and fθ(x, z) is the probability density function that depends on a vector 

θ of parameters, there θ(t) = (wy, wz, a) where some parameters of the architecture 

parameter vector a are hand-initialized such as the receptive fields. The maximum 

estimator for θ corresponds to the θ that maximizes the probability density. Regardless 

41 
z is imposed, z is part of the parameters to be computed in a closed-form as a selfgenerated version: 

∗ (θ∗, y , z∗) = argmax fθ(x, z). (25) 

(θ,y,z)

Since the above lifetime estimator is incremental, at each time t, the previous state 

∗z is self-generated or supervised, and the observation is xt−1. The incremental ML- t−1

estimator for θ∗ is computed in a closed-form by the incremental version of Eq. (25) t

where f uses context ct−1 = (xt−1, yt−1, zt−1): 

∗ ∗ ∗ ∗ (θt

∗, yt , zt ) = argmax fθt (xt−1, yt−1, zt−1). (26) 

(θt,yt,zt)

970 The DN computes the above expression for each time t in a closed form without conducting any iterations [41, 87]. 

How about initial weights? Inside θ, the weights of the DN are initialized randomly 

at t = 0. There are k + 1 initial neurons in the Y area, and V = {v ̇ i|i = 1, 2, ..., k + 1}

is the current synaptic vectors in Y . Whenever the network takes an input p, compute 

975 the pre-responses in Y . If the top-1 winner in Y has a pre-response lower than almost 

perfect match m(t) discussed below, activate a free neuron to fire. Eq. (21) showed 

that the initial weights of this free neuron is multiplied by a zero and therefore do not 

affect its updated weights. 

Weng [41] proved that DN-1 computes the ML-estimator of all observations from 

980 the sensory space X and motor space Z using a large constant time complexity for 

each time t. Although DN learns incrementally, such a DN is error-free for learning 

any complex Turing machines, including any universal Turing machines. Weng [87] 

did the same for DN-2. 

4.8. How DN Avoids Post-Selections but is Further ML-Optimal 

Since weights are initialized randomly, how does a DN result in an equivalent network regardless the random seed? There are k + 1 initial neurons in the Y area, and 

V = {v ̇ i|i = 1, 2, ..., k + 1} is the current synaptic vectors in Y . Whenever the network takes an input p, every Y neuron computes the pre-response. If the top-1 winner 

in Y has a pre-response lower than almost perfect match m(t), activate a free neuron 

42 
to fire. The almost perfect match m(t) is defined as follows: 

−t/t1 m(t) = (1 − δ)(1 − e ) (27) 

985 where δ is the bound of machine round-off errors, and t1 the childhood length. 

Using a mathematical induction procedure, Weng [41] proved that DN-1 computes 

the ML-estimator of all observations from (x, y, z) using a large constant time complexity for each time t. Weng et al. 2018 [87] proved the ML-optimality for DN-2. 

Since the number of transition of any Turing machine is finite, when the DN learns a 

990 Turing machine, a finite number of hidden Y neurons is sufficient for the DN to incrementally memorize exactly all the transitions observed from the Turing machine. In 

other words, although DN learns incrementally, such a DN is error-free for learning 

any complex Turing machines, including any universal Turing machines. 

If the DN runs in the real world, any finite size DN is not error-free soon after 

995 inception since the number of observations from the real world is virtually unbounded, 

although each life is time bounded. Namely, the amount of data from the real world 

is so large that any practically large DN will eventually run out of free neurons that 

have not fired yet. From that point on, the DN is no longer guaranteed to be errorfree, although could be sometimes error-free, but is still ML-optimal inside the skull 

1000 conditioned on those in Definition 2. In other words, in the sense of ML, the DN is free 

of local minima inside the skull. That is why only one DN is sufficient for each life and 

the DN avoids Post-Selections. However, because the three conditions in Definition 2, 

a DN is not be optimal unconditionally either. For example, a better designed teaching 

schedule or a more appropriate physical environment may enable a DN to learn and 

1005 discover rules faster and better. 

4.9. Comparison with HMM 

It is important to compare the traditional Hidden Markov Model (HMM) [90] with 

the ML-optimal DN. (1) The former does not have any internal representations other 

than the symbol based probabilities; the latter self-generates internal representations to 

1010 generalize based on internal-representation based probabilities (e.g., weights). (2) The 

former uses batch learning but the latter uses incremental learning. (3) States in the 

43 
former are symbolic, static, only partially observable for HMM, and not teachable but 

those in the latter are emergent, observable and directly teachable if the teacher like. (4) 

The former requires a batch clustering method (e.g., k-mean clustering) to initialize a 

1015 static set of symbolic states, but the states/actions in the latter are incrementally taught 

or autonomously generated and tried. (5) Clusters of states in the former are not supported by a statistical optimality and the probability is only for state estimates but those 

in the latter are ML-optimal throughout the lifetime of learning, not in states/actions 

that the learner must produce and do not have a freedom for, but for the internal repre1020 sentations that the learner does have a high degree of freedom for. (6) Due to the need 

to compute internal representations, the amount of computations in the latter is often 

higher than the former but the computational complexity is linear in time with a large 

constant (the number of weights of all available neurons). 

5. Experiments 

1025 5.1. Vision, Audition and Natural Languages 

The recent experimental results of DN work here include (1) vision that includes 

simultaneous recognition and detection and vision-guided navigation on MSU campus walkways [91], (2) audition to learn phonemes with a simulated cochlea and the 

corresponding behaviors [92], (3) acquisition of English and French in an interactive 

1030 bilingual environment [93], and (4) exploration in a simulated maze environment with 

autonomous learning for vision, path cost, planning, and selection of the least-cost plan, 

where all such emergent actions are either covert (thoughts) or overt (acts) [94]. The 

same network was used to learn these four very different tasks and task environments 

while each task embeds the ML-optimality of the network, under the Three Learning 

1035 Conditions. 

5.2. Error-Backprop vs. ML-Optimal DN 

To show the effects of the absence of ML-optimality in CNN vs. the ML-optimality 

of DN, Fig. 5 shows the errors of the luckiest Convolutional Neural Network (CNN) 

trained by a batch error-backprop method and the errors of a DN trained incrementally. 

44 
Figure 5: Comparison of error between the luckiest CNN trained by batch error-backprop and a DN across 

different epochs through the training data. “Recog” and “Full” means teach where-what rules. Otherwise, 

data-fitting only. Adapted from [95]. 

1040 As we understand, batch learning should not be compared with an incremental learning 

method, because it is not a comparison on an equal footing. However, Fig. 5 shows that 

DN does a harder (incremental) work drastically better than CNN does an easier (batch) 

work. The task is real-world vision-guided navigation on the campus of Michigan State 

University. Because the DN is optimal in maximum-likelihood, it reaches the minimum 

1045 error as soon as it has gone through the data set T once (one epoch). Later epochs 

correspond to reviews of the same data set T. According to the maximum-likelihood 

principle, the optimal estimate of the neuronal weights should not change but the ages 

of the neurons continue to advance. In contract, the luckiest error-backprop trained 

CNN chosen from several random seeds need many epochs to reduce its errors and 

1050 only very slowly. At the end of 500th epoch, the error of the luckiest CNN trained by 

error-backprop is still considerably higher than the full DN. Furthermore, as show in 

Fig. 5, teaching invariant concepts, i.e., abstraction in Theorem 2, are use for reducing 

the optimal errors. For more detail, the reader is referred to [95]. 

5.3. AIML Contests 

1055 In the AIML Contest 2016, all teams are required to use a single learning engine to 

learn three sensory modalities, vision, audition, and bilingual natural languages acqui45 
sition, while the engine learns in “lifetime”. Although all the teams are free to choose 

any existing learning engines such as DN, tensor-flow or other engines, all the teams 

chose DN engine (open source). The supplied simulated sequential data (yes, subject 

1060 to the “big data” flaw) are as follows. When we let the x be the image at each time instance and z be the pattern of landmark location-and-type and action of navigation, the 

DN became a vision-guided navigation machine. When we let x be the frame of firing 

pattern of hair cells in cochlea at each time instance and z be the dense states/contexts 

and the sparse type of sounds, the DN became a auditory-recognizer machine. When 

1065 we let x be a time frame of vector of word (either English or French) and z be the 

language kind (neutral, English, and French) and meaning of each sentence context, 

the DN became a bilingual language learner and recognizer. Thus, the AIML Contest 

appeared to be the first contest that independently demonstrated task-nonspecificity 

and modality-nonspecificity by independent laboratories with the contest teams. All 

1070 the teams are evaluated under the same Three Learning Conditions, e.g., the number 

of neurons in the engine must not exceed the same given bound. The Contest used 

the developmental error like the one defined here averaged over all the three contest 

tasks and across the three lifetimes. The developmental error ranked all the submitted 

contest entries and required all networks not to exceed the specified maximum number 

1075 of neurons for each task so that the competition does not unfairly favor those teams 

that have more computational resources at their disposal but not necessarily that their 

methods are more superior. All the teams have a high degree of freedom to modify the 

learning engine and to modify the supplied motor actions on the given data set, such as 

generating attentive actions on the given training set to train invariant concepts (e.g., 

1080 where and what concepts) which modifies the default training experience supplied by 

the AIML Contest organizers but was still based on the same supplied data set. The 

prudent design of the AIML contests was meant to avoid the corresponding problems 

in ImageNet Contests [61] and many other contests. 

5.4. GENISAMA Applications 

1085 GENISAMA LLC, a startup that the author created, has produced a series of realtime machine learning products, as human-wearable robots. They are the first products 

46 
ever existed as APFGP robots. Hopefully, as a APFGP platform, this new kind of 

human-wearable robots will be useful for practitioners to produce various kinds of 

intelligent auto-programed software. The author predicts that such a new kind of AI 

1090 systems will considerably alleviate the high brittleness of traditional AI software and 

traditional robot software in open and natural world. 

Hopefully, future DN-driven robots will learn consciously and autonomously discover in the real world for Turing machine based general purposes, with relative infrequent interactions from humans similar to what parents do to their children and 

1095 human teachers teach their students in classrooms. The experiments and competitions 

described here are for this grand goals but have not reached this experimental goal yet. 

6. Conclusions 

We used intuitive terms but formal ways to discuss Post-Selections. Public and media have gained an impression that deep learning has approached or even “sometimes 

1100 exceeded” human level performance on certain tasks. For example, the image classification errors from a static image set were compared with those of humans [61, A2, 

p242]) and the work is laudable. However, this paper raises Post-Selections, which 

seem to question such claims since a real human does not have the luxury of PostSelections. The author hopes that the exposure of Post-Selections is beneficial to AI 

1105 credibility and the future healthy development of AI, especially with the concepts of 

developmental errors and the framework of ML-optimal lifetime learning for invariant concepts under the Three Learning Conditions. Some researchers have raised that 

it seems that those who wan a competition were those who have more computational 

resources and manpower at their disposal. The new developmental error metrics un1110 der the Three Learning Conditions hopefully encourages future AI competitions to 

compare methods under the same Three Learning Conditions. Considering DN as a 

much-simplified model for a biological machine, it seems not baseless to guess that 

each biological brain is probably ML-optimal (of course in a much richer sense) across 

lifetime, e.g., due to the pressure to compete at every age. The Three Learning Condi1115 tions explicitly include other factors that greatly affect machine learning performances 

47 
such as learning framework (e.g., task-nonspecificity, incremental learning, the robot 

bodies), learning experiences and computational resources. The analysis that any “big 

data” sets are nonscalable does not mean that we should not create, use and share data 

sets. Instead, we need to pay attention to the fundamental limitations of any static data 

1120 sets, regardless how large their apparent sizes are. 

References 

[1] N. Montfort, Twisty Little Passages: An Approach to Interactive Fiction, MIT 

Press, Cambridge, MA, 2005. 

[2] A. M. Turing, Computing machinery and intelligence, Mind 59 (1950) 433–460. 

1125 [3] J. Weng, Symbolic models and emergent models: A review, IEEE Trans. Autonomous Mental Development 4 (1) (2012) 29–53. 

[4] S. Russell, P. Norvig, Artificial Intelligence: A Modern Approach, 3rd Edition, 

Prentice-Hall, Upper Saddle River, New Jersey, 2010. 

[5] M. Minsky, Logical versus analogical or symbolic versus connectionist or neat 

1130 versus scruffy, AI Magazine 12 (2) (1991) 34–51. 

[6] D. B. Lenat, G. Miller, T. T. Yokoi, CYC, WordNet, and EDR: Critiques and 

responses, Communications of the ACM 38 (11) (1995) 45–48. 

[7] L. Gomes, Machine-learning maestro Michael Jordan on the delusions of big data 

and other huge engineering efforts, IEEE Spectrum. 

1135 [8] D. E. Rumelhart, J. L. McClelland, the PDP Research Group, Parallel Distributed 

Processing, Vol. 1, MIT Press, Cambridge, Massachusetts, 1986. 

[9] J. L. McClelland, D. E. Rumelhart, The PDP Research Group (Eds.), Parallel 

Distributed Processing, Vol. 2, MIT Press, Cambridge, Massachusetts, 1986. 

[10] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalan1140 tidis, L. J. Li, D. A. Shamma, M. S. Ernstein, L. Fei-Fei, Visual genome, International Journal of Computer Vision 123 (1) (2017) 32–73. 

48 
[11] K. I. Funahashi, On the approximate realization of continuous mappings by neural 

networks, Neural Network 2 (2) (1989) 183–192. 

[12] T. Poggio, F. Girosi, Networks for approximation and learning, Proceedings of 

1145 The IEEE 78 (9) (1990) 1481–1497. 

[13] T. Kohonen, Self-Organizing Maps, 3rd Edition, Springer-Verlag, Berlin, 2001. 

[14] K. Fukushima, Neocognitron: A self-organizing neural network model for a 

mechanism of pattern recognition unaffected by shift in position, Biological Cybernetics 36 (1980) 193–202. 

1150 [15] M. Oja, S. Kaski, T. Kohunen, Bibliography self-organizing maps (som) papers: 

1998 - 2001 addendum, Neural Computing Surveys 3 (2003) 1–156. 

[16] J. Weng, N. Ahuja, T. S. Huang, Learning recognition and segmentation using the 

Cresceptron, International Journal of Computer Vision 25 (2) (1997) 109–143. 

[17] K. Fukushima, S. Miyake, T. Ito, Neocognitron: A neural network model for a 

1155 mechanism of visual pattern recognition, IEEE Trans. Systems, Man and Cybernetics 13 (5) (1983) 826–834. 

[18] T. Serre, T. Poggio, M. Riesenhuber, L. Wolf, S. Bileschi, High-performance 

vision system exploiting key features of visual cortex, US Patent US7606777B2. 

[19] L. Fei-Fei, R. Fergus, P. Perona, One-shot learning of object categories, IEEE 

1160 Trans. Pattern Analysis and Machine Intelligence 28 (4) (2006) 594–611. 

[20] J. Weng, Dialog initiation: Modeling amd: Closed skull or not?, IEEE CIS Autonomous Mental Development Newsletter 9 (2) (2012) 10–11. 

[21] P. J. Werbos, The Roots of Backpropagation: From Ordered Derivatives to Neural 

Networks and Political Forecasting, Wiley, Chichester, 1994. 

1165 [22] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to 

document recognition, Proceedings of IEEE 86 (11) (1998) 2278–2324. 

49 
[23] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in Neural Information Processing Systems 25, 2012, pp. 1106–1114. 

1170 [24] Y. LeCun, L. Bengio, G. Hinton, Deep learning, Nature 521 (2015) 436–444. 

[25] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, 

A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, 

A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, D. Hassabis, 

Human-level control through deep reinforcement learning, Nature 518 (2015) 

1175 529–533. 

[26] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, 

K. Kavukcuoglu, T. Graepel, D. Hassabis, Mastering the game of go with deep 

1180 neural networks and tree search, Nature 529 (2016) 484–489. 

[27] A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. GrabskaBarwinska, S. G. Colmenarejo, E. Grefenstette, T. Ramalho, J. Agapiou, A. P. 

Badia, K. M. Hermann, Y. Zwols, G. Ostrovski, A. C., H. King, C. Summerfield, 

P. Blunsom, K. Kavukcuoglu, D. Hassabis, Hybrid computing using a neural net1185 work with dynamic external memory, Nature 538 (2016) 471–476. 

[28] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, 

T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, 

G. van den Driessche, T. Graepel, D. Hassabis, Mastering the game of go without 

human knowledge, Nature (2017) 354–359. 

1190 [29] S. M. McKinney, M. Sieniek, V. Godbole, J. Godwin, N. Antropova, 

H. Ashrafian, T. Back, M. Chesus, G. S. Corrado, A. Darzi, M. Etemadi, 

F. Garcia-Vicente, F. J. Gilbert, M. Halling-Brown, D. Hassabis, S. Jansen, 

A. Karthikesalingam, C. J. Kelly, D. King, J. R. Ledsam, D. Melnick, H. Mostofi, 

L. Peng, J. J. Reicher, B. Romera-Paredes, R. Sidebottom, M. Suleyman, D. Tse, 

50 
1195 K. C. Y. andJeffrey De Fauw, S. Shetty, International evaluation of an AI system 

for breast cancer screening, Nature 577 (2020) 89–94. 

[30] A. W. Senior, R. Evans, J. Jumper, J. Kirkpatrick, L. Sifre, T. Green, C. Qin, 

A. Zidek, A. W. R. Nelson, A. Bridgland, H. Penedones, S. Petersen, K. Simonyan, S. Crossan, P. Kohli, D. T. Jones, D. Silver, K. Kavukcuoglu, D. Hass1200 abis, Improved protein structure prediction using potentials from deep learning, 

Nature 577 (2020) 706–710. 

[31] M. G. Bellemare, S. Candido, J. G. P. S. Castro, M. C. Machado, S. Moitra, 

S. S. Ponda, Z. Wang, Autonomous navigation of stratospheric balloons using 

reinforcement learning, Nature 588 (7836) (2020) 77–82. 

1205 [32] A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, J. Clune, First return, then 

explore, Nature 590 (7847) (2021) 580–586. 

[33] V. Saggio, B. E. Asenbeck, A. Hamann, T. Stromberg, P. Schiansky, V. Dunjko, N. Friis, N. C. Harris, M. Hochberg, D. Englund, S. Wolk, H. J. Briegel, 

P. Walther, Experimental quantum speed-up in reinforcement learning agents, Na1210 ture 591 (7849) (2021) 229–233. 

[34] F. R. Willett, D. T. Avansino, L. R. Hochberg, J. M. Henderson, K. V. 

Shenoy, High-performance brain-to-text communication via handwriting, Nature 

593 (7858) (2021) 249–254. 

[35] N. Slonim, Y. Bilu, C. Alzate, R. Bar-Haim, B. Bogin, F. Bonin, L. Choshen, 

1215 E. Cohen-Karlik, L. Dankin, L. Edelstein, L. Ein-Dor, R. Friedman-Melamed, 

A. Gavron, A. Gera, M. Gleize, S. Gretz, D. Gutfreund, A. Halfon, D. Hershcovich, R. Hoory, Y. Hou, S. Hummel, M. Jacovi, C. Jochim, Y. Kantor, Y. Katz, 

D. Konopnicki, Z. Kons, L. Kotlerman, D. Krieger, D. Lahav, T. Lavee, R. Levy, 

N. Liberman, Y. Mass, A. Menczel, S. Mirkin, G. Moshkowich, S. Ofek-Koifman, 

1220 M. Orbach, E. Rabinovich, R. Rinott, S. Shechtman, D. Sheinwald, E. Shnarch, 

I. Shnayderman, A. Soffer, A. Spector, B. Sznajder, A. Toledo, O. Toledo-Ronen, 

E. Venezian1, R. Aharonov, An autonomous debating system, Nature 591 (7850) 

(2021) 379–384. 

51 
[36] A. Mirhoseini, A. Goldie, M. Yazgan, J. W. Jiang, E. Songhori, S. Wang, Y.-J. 

1225 Lee, E. Johnson, O. Pathak, A. Nazi, J. Pak, A. Tong, K. Srinivasa, W. Hang, 

E. Tuncer, Q. V. Le1, J. Laudon, R. Ho, R. Carpenter, J. Dean, A graph placement 

methodology for fast chip design, Nature 594 (7862) (2021) 207–212. 

[37] M. Y. Lu, T. Y. Chen, D. F. K. Williamson, M. Zhao, M. Shady, J. Lipkova, 

F. Mahmood, AI-based pathology predicts origins for cancers of unknown pri1230 mary, Nature 594 (7861) (2021) 106–110. 

[38] S. Warnat-Herresthal, H. Schultze, K. L. Shastry, S. Manamohan, S. Mukherjee, 

V. Garg, R. Sarveswara, K. Handler, P. Pickkers, N. A. Aziz, S. Ktena, F. Tran, 

M. Bitzer, S. Ossowski, N. Casadei, C. Herr, D. Petersheim, U. Behrends, F. Kern, 

T. Fehlmann, P. Schommers, C. Lehmann, M. Augustin, J. Rybniker, J. Altmuller, 

1235 N. Mishra, J. P. Bernardes, B. Kramer, L. Bonaguro, J. Schulte-Schrepping, E. D. 

Domenico, C. Siever, M. Kraut, M. Desai, B. Monnet, M. Saridaki, C. M. Siegel, 

A. Drews, M. Nuesch-Germano, H. Theis, J. Heyckendorf, S. Schreiber, S. KimHellmuth, C. A. S. (COVAS), J. Nattermann, D. Skowasch, I. Kurth, A. Keller, 

R. Bals, P. Nurnberg, O. Rieb, P. Rosenstiel, M. G. Netea, F. Theis, S. Mukher1240 jee, M. Backes, A. C. Aschenbrenner, T. Ulas, D. COVID-19 Omics Initiative 

(DeCOI), M. M. B. Breteler, E. J. Giamarellos-Bourboulis, M. Kox, M. Becker, 

S. Cheran, M. S. Woodacre, E. L. Goh, J. L. Schultze, Swarm learning for decentralized and confidential clinical machine learning, Nature 594 (7862) (2021) 

265–270. 

1245 [39] J. Weng, J. McClelland, A. Pentland, O. Sporns, I. Stockman, M. Sur, E. Thelen, Autonomous mental development by robots and animals, Science 291 (5504) 

(2001) 599–600. 

[40] J. L. Mcclelland, K. Plunkett, J. Weng, Guest editorial: Convergent approaches 

to the understanding of autonomous mental development, IEEE Trans. on Evolu1250 tionary Computation 11 (2) (2007) 133–136. 

[41] J. Weng, Brain as an emergent finite automaton: A theory and three theorems, 

International Journal of Intelligent Science 5 (2) (2015) 112–131. 

52 
[42] J. Weng, N. Ahuja, T. S. Huang, Cresceptron: a self-organizing neural network 

which grows adaptively, in: Proc. Int’l Joint Conference on Neural Networks, 

1255 Vol. 1, Baltimore, Maryland, 1992, pp. 576–581. 

[43] J. Weng, N. Ahuja, T. S. Huang, Learning recognition and segmentation of 3-D 

objects from 2-D images, in: Proc. IEEE 4th Int’l Conf. Computer Vision, 1993, 

pp. 121–128. 

[44] J. Weng, Life is science (35): Did Turing Awards go to plagiarism?, Face1260 book blog, www.facebook.com/juyang.weng/posts/10158305658699783 (March 

4 2020). 

[45] J. Weng, Did Turing Awards go to plagiarism?, YouTube Video, 1:05 hours, https: 

//youtu.be/EAhkH79TKFU (May 27 2020). 

[46] J. Weng, Why have we passed “neural networks do not abstract well”?, Natural 

1265 Intelligence: the INNS Magazine 1 (1) (2011) 13–22. 

[47] Z. Ji, J. Weng, D. Prokhorov, Where-what network 1: “Where” and “What” assist 

each other through top-down connections, in: Proc. IEEE Int’l Conference on 

Development and Learning, Monterey, CA, 2008, pp. 61–66. 

[48] Q. Guo, X. Wu, J. Weng, Cross-domain and within-domain synaptic maintenance 

1270 for autonomous development of visual areas, in: Proc. the Fifth Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics, 

Providence, RI, 2015, pp. +1–6. 

[49] D. J. Felleman, D. C. Van Essen, Distributed hierarchical processing in the primate cerebral cortex, Cerebral Cortex 1 (1991) 1–47. 

1275 [50] C. M. Super, Environmental effects on motor development: A case of Africa 

infant precocity, Developmental Medicine and Child Neurology 18 (1976) 561– 

567. 

[51] K. A. Thoroughman, J. A. Taylor, Rapid reshaping of human motor generalization, Jounal of Neuroscience 25 (39) (2005) 8948–8953. 

53 
1280 [52] G. Rizzotti, L. Riggio, I. Dascola, C. Umilta, Reorienting attention across the horizontal and vertical meridians: evidence in favor of a premotor theory of attention, 

Neuropsychologia 25 (1987) 31–40. 

[53] T. Moore, K. M. Armstrong, M. Fallah, Visuomotor origins of covert spatial attention, Neuron 40 (2003) 671–683. 

1285 [54] J. M. Iverson, Developing language in a developing body: the relationship between motor development and language development, Journal of child language 

37 (2) (2010) 229–261. 

[55] J. Weng, M. Luciw, Brain-like emergent spatial processing, IEEE Trans. Autonomous Mental Development 4 (2) (2012) 161–185. 

1290 [56] J. Weng, M. Luciw, Q. Zhang, Brain-like temporal processing: Emergent open 

states, IEEE Trans. Autonomous Mental Development 5 (2) (2013) 89 – 116. 

[57] J. Weng, Autonomous programming for general purposes: Theory, International 

Journal of Huamnoid Robotics 17 (4) (2020) 1–36. 

[58] J. Weng, Conscious intelligence requires developmental autonomous program1295 ming for general purposes, in: Proc. IEEE International Conference on Development and Learning and Epigenetic Robotics, Valparaiso, Chile, 2020, pp. 1–7. 

[59] J. Weng, On post selections using test sets (PSUTS) in AI, in: Proc. International 

Joint Conference on Neural Networks, Shengzhen, China, 2021, pp. 1–8. 

[60] J. Weng, A developmental method that computes optimal networks without post1300 selections, in: Proc. IEEE International Conference on Development and Learning, Beijing, China, 2021, pp. 1–6. 

[61] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, 

A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, L. Fei-Fei, ImageNet large 

scale visual recognition challenge, International Journal of Computer Vision 115 

1305 (2015) 211–252. 

54 
[62] J. Weng, Z. Zheng, X. Wu, J. Castro-Garcia, S. Zhu, Q. Guo, X. Wu, Emergent 

Turing machines and operating systems for brain-like auto-programming for general purposes, in: Proc. AAAI 2018 Fall Symposium: Gathering for Artificial 

Intelligence and Natural Systems, Arlington, Virginia, 2018, pp. 1–7. 

1310 [63] D. H. Ballard, C. M. Brown, Computer Vision, Prentice-Hall, New Jersey, 1982. 

[64] L. Shapiro, G. Stockman, Computer Vision, Addison-Wesley, New York, 2001. 

[65] J. Weng, Natural and Artificial Intelligence: Introduction to Computational BrainMind, 2nd Edition, BMI Press, Okemos, Michigan, 2019. 

[66] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, L. Fei-Fei, Large1315 scale video classification with convolutional neural networks, in: Proc. Computer 

Vision and Pattern Recognition, Columbus, Ohio, 2014, pp. +1–8. 

[67] J. Weng, M. Luciw, Dually optimal neuronal layers: Lobe component analysis, 

IEEE Trans. Autonomous Mental Development 1 (1) (2009) 68–85. 

[68] J. Weng, M. D. Luciw, Brain-inspired concept networks: Learning concepts from 

1320 cluttered scenes, IEEE Intelligent Systems Magazine 29 (6) (2014) 14–22. 

[69] D. J. Wood, J. S. Bruner, G. Ross, The role of tutoring in problem-solving, Journal 

of Child Psychology and Psychiatry (1976) 89–100. 

[70] S. Burr, Active learning literature survey, Data Mining and Knowledge Discovery 

2 (2) (1998) 121–167. 

1325 [71] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, Communications of the ACM 60 (6) (2017) 84–90. 

[72] A. K. Jain, R. C. Dubes, Algorithms for Clustering Data, Prentice-Hall, New 

Jersey, 1988. 

[73] Y. Wang, X. Wu, J. Weng, Synapse maintenance in the where-what network, in: 

1330 Proc. Int’l Joint Conference on Neural Networks, San Jose, CA, 2011, pp. 2823– 

2829. 

55 
[74] Q. Guo, X. Wu, J. Weng, WWN-9: Cross-domain synaptic maintenance and its 

application to object groups recognition, in: Proc. Int’l Joint Conference on Neural Networks, Beijing, China, 2014, pp. +1–8. 

1335 [75] Q. Gao, G. A. Ascoli, L. Zhao, BEAN: Interpretable and efficient learning with 

biologically-enhanced artificial neuronal assembly regularization, Front. Neurorobot (2021) 1–13. 

[76] M. Moravcik, M. Schmid, N. Burch, V. Lisy, D. Morrill, N. Bard, T. Davis, 

K. Waugh, M. Johanson, M. Bowling, Deepstack: Expert-level artificial intel1340 ligence in heads-up no-limit poker, Science 356 (2017) 508–513. 

[77] J. Weng, Life is science (36): Did Turing Awards go to fraud?, Facebook blog, 

www.facebook.com/juyang.weng/posts/10158319020739783 (March 8 2020). 

[78] J. Weng, Did Turing Awards go to fraud?, YouTube Video, 1:04 hours, https: 

//youtu.be/Rz6CFlKrx2k (June 4 2020). 

1345 [79] T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, T. Poggio, Robust object recognition with cortex-like mechanisms, IEEE Trans. Pattern Analysis and Machine 

Intelligence 29 (3) (2007) 411–426. 

[80] P. Sermanet, K. Kavukcuoglu, S. Chintala, Y. LeCun, No more pesky learning 

rates, in: Proc. International Conference on Machine learning, Atlanta, GA, 2013, 

1350 pp. 343–351. 

[81] Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli, Y. Bengio, Identifying and attacking the saddle point problem in high-dimensional non-convex 

optimization, in: Advances in Neural Information Processing Systems, Curran 

Associates, Inc., Montreal, Canada, 2014, pp. 2933–2941. 

1355 [82] N. Srivastava, G. E. Hinton, K. Krizhevsky, I. Sutskever, R. Salakhutdinov, 

Dropout: A simple way to prevent neural networks from overtting, Journal of 

Machine Learning Research 15 (1) (2014) 1929–1958. 

56 
[83] T. Poggio, Theoretical issues in deep networks, Proceedings of thr National 

Academy of Sciences 117 (48) (2020) 30039–30045. 

1360 [84] J. Weng, Z. Zheng, X. Wu, J. Castro-Garcia, Auto-programming for general purposes: Theory and experiments, in: Proc. International Joint Conference on Neural Networks, Glasgow, UK, 2020, pp. 1–8. 

[85] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, 

L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, D. Hassabis, A gen1365 eral reinforcement learning algorithm that masters chess, shogi, and go through 

self-play, Science (2018) 1140–1144. 

[86] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, 

A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, D. Silver, Mastering 

atari, go, chess and shogi by planning with a learned model, Science 588 (7839) 

1370 (2020) 604–609. 

[87] J. Weng, Z. Zheng, X. Wu, Developmental Network Two, its optimality, and 

emergent Turing machines, U.S. Provisional Patent Application Serial Number: 

62/624,898, published (Feb. 1 2018). 

[88] J. A. Knoll, V. N. Hoang, J. Honer, S. Church, T. H. Tran, J. Weng, Optimal 

1375 developmental learning for multisensory and multi-teaching modalities, in: Proc. 

IEEE International Conference on Development and Learning, Beijing, China, 

2021, pp. 1–6. 

[89] J. Weng, A unified hierarchy for AI and natural intelligence through autoprogramming for general purposes, Journal of Cognitive Science 21 (2020) 53– 

1380 102. 

[90] L. R. Rabiner, A tutorial on hidden Markov models and selected applications in 

speech recognition, Proceedings of IEEE 77 (2) (1989) 257–286. 

[91] Z. Zheng, X. Wu, J. Weng, Emergent neural turing macine and its visual navigation, Neural Networks 110 (2019) 116–130. 

57 
1385 [92] X. Wu, J. Weng, Muscle vectors as temporally “Dense Labels”, in: Proc. International Joint Conference on Neural Networks, Glasgow, UK, 2020, pp. 1–8. 

[93] J. Castro-Garcia, J. Weng, Emergent multilingual language acquisition using developmental networks, in: Proc. International Joint Conf. Neural Networks, IEEE 

Press, Budapest, Hungary, 2019, pp. +1–8. 

1390 [94] X. Wu, J. Weng, On machine thinking, in: Proc. International Joint Conf. Neural 

Networks, IEEE Press, Shenzhen, China, 2021, pp. 1–8. 

[95] Z. Zheng, J. Weng, Mobile device based outdoor navigation with on-line learning 

neural network: a comparison with convolutional neural network, in: Proc. 7th 

Workshop on Computer Vision in Vehicle Technology (CVVT 2016) at CVPR 

1395 2016, Las Vega, 2016, pp. 11–18. 

58 
 

 

Declaration of Interest Statement

Declaration of interests 

The authors declare that they have no known competing financial interests or personal 

relationships that could have appeared to influence the work reported in this paper. 

☐The authors declare the following financial interests/personal relationships which may be considered 

as potential competing interests: 

None