# An Implicit Memory-Based Method for Supervised Pattern Recognition

Authors: Yu Ma; Shafei Wang; Junan Yang; Yanfei Bao; Jian Yang
Journal: Discrete Dynamics in Nature and Society
 (2021)

---

## Abstract

How the human brain does recognition is still an open question. No physical or biological experiment can fully reveal this process. Psychological evidence is more about describing phenomena and laws than explaining the physiological processes behind them. The need for interpretability is well recognized. This paper proposes a new method for supervised pattern recognition based on the working pattern of implicit memory. The artificial neural network (ANN) is trained to simulate implicit memory. When an input vector is not in the training set, the ANN can treat the input as a “do not care” term. The ANN may output any value when the input is a “do not care” term since the training process needs to use as few neurons as possible. The trained ANN can be expressed as a function to design a pattern recognition algorithm. Using the Mixed National Institute of Standards and Technology database, the experiments show the efficiency of the pattern recognition method.


---

## Body

## 1. Introduction

Pattern recognition methods can be divided into two categories: two-stage and end-to-end. Most traditional pattern recognition methods are two-stage: feature extraction and pattern classification [1]. Feature extraction reduces the number of resources that are required to describe a large amount of raw data. The first step is to identify the measurable quantities that make these training sets X1,…,Xl distinct from each other. The measurements used for the classification, such as mean value and the standard deviation, are known as features. In general, some features constitute a feature vector. Some information gets lost since feature extraction is not a lossless compression approach. The lost information cannot be used for pattern recognition. Therefore, how to generate features is a fundamental issue. In the feature vector selection, two crucial issues are the best number of features and the classifier design [2]. For example, feature selection plays an important role in text classification [3]. The complexity of the practice data makes it arduous to use two-stage methods.Nowadays, deep neural networks can be trained end-to-end [4]. Raw data can reserve all information of the pattern. Inspired by the biological neural networks that constitute animal brains, the artificial neural network (ANN) is applied to do image classification [5], speech separation [6], forest fire prediction [7], etc. However, a major drawback of neural networks is the black-box character. Explaining why the neural networks make a particular decision is formidable. The knowledge representation of neural network is unreadable to humans [8]. The exact reason why trained deep neural networks can implement recognition remains an open question [9]. The training algorithm does not specify the way to recognize. A causal model is formidable to build or acquire. End-to-end learning relies on data for the cognitive task [10, 11], where the data cannot tell the reasons [12].In a supervised pattern recognition task, a set of training data (training set) is used to train a learning procedure. A training set is a set of instances that are properly labeled by hand with the corrected labels. The learning procedure attempts to recognize the instances as accurately as possible. The goal of the learning procedure is to minimize the error rate on a test set. The question arising in the recognition task is why a new data instance can be classified as a particular category. The problem of supervised pattern recognition can be stated as follows. Assume that the training setXi only contains instances with label i where i∈ℒ≜1,…,l, and Xi∩Xi=∅ for any i≠j. Given l training sets X1,…,Xl, the question is how to label a new instance x˜.A memory system is involved in the process of recognition. Jacoby and Kelly posit that memory can serve as both storage and a tool [13]. Memory is treated as storage in a recall. In this case, the focus is on the past, and memory is used as computer storage. Meanwhile, memory (from experience) can be used as a tool to perceive and interpret present events.The implicit memory is acquired and used unconsciously and can serve as a tool/function [14–16]. In one experiment, two groups of people are asked several times to solve a Tower of Hanoi puzzle. One group is amnesic patients with heavily impaired long-term memory, and the other is composed of healthy subjects. The first group shows the same improvements over time as the second one, even if some participants claim that they do not even remember seeing the puzzle before. These findings strongly suggest that procedural memory is completely independent of declarative memory [17]. Given a game state, implicit memory is trained to output an operation. Memorizing the solving steps is not necessary. When the state appears again, the input can evoke the trained operator [18].Usually, humans implement recognition processes unconsciously, and the focus is on current given images. So the memory works as a tool while implementing the recognition processes. The recognition process depends on the similarity comparison between the current input and the labeled instances. The more similar, the more likely they are in the same class and have the same label. However, the way to compare similarities without memorizing any labeled instances is not evident.This paper proposes an implicit memory-based method for supervised pattern recognition. The method does not memorize or recall any labeled instances and is not in the two-stage or end-to-end categories. The proposed method has interpretability since similarity criteria are used in the process of recognition. A new instance is recognized as a particular class because the instance appears similar enough to the training data of the class. Compared with thek-nearest neighbors algorithm [19], the proposed method does not need to recall and iterate through the training sets. The process is consistent with the human ability of pattern recognition. People may forget most of the training instances, but they can recognize a new instance. The Mixed National Institute of Standards and Technology (MNIST) database (General site for the MNIST database: http://yann.lecun.com/exdb/mnist) is used to verify the proposed method.The rest of this paper is organized as follows. First, a model is built to describe implicit memory. Second, with the implicit memory model, a recognition algorithm is proposed. Then an application and the analysis of experimental results are given.Notations: X expresses the cardinality of the set X; 0,1n is an n-dimensional space constructed by 0 and 1; d is a metric on 0,1n; define a distance function between any point z of 0,1n and any nonempty set X of 0,1n by(1)dz,X=infdz,x|x∈X.Given twon-dimensional a and b, the element-wise product of a and b, written a°b, is the vector c with elements given by ci=aibi; xl1 expresses the number of 1 in the binary vector x; the probability of X=x is written as PrX=x; ℒ≜1,…,l; N≜1,…,n; P≜1,…,p.The expression of an inverter (NOT circuit) can be expressed as follows: If the input variable is calledZ and the output variable is called Y, then Y=Z¯. If Z=0, then Y=1 and if Z=1, then Y=0. The expression of a 2-input AND gate can be expressed in equation form as follows: Y=Z1Z2. The output Y of an AND gate is 1 only when both inputs are 1 s. The expression of a 2-input OR gate can be expressed as follows: Y=Z1+Z2. The output Y of an OR gate is 1 when any one or more of the inputs are 1 s.


## 2. Model of Implicit Memory

In this section, a model is built to describe implicit memory. From one given input, implicit memory can give an output. Both the input and output of implicit memory are actual signals. The signals have explicit physical meaning in the real world, such as sound, light, electricity, etc. The types of input-output signals can be the same or different. Meanwhile, the input-output signals can be measured and represented as a binary vector.The action of implicit memory can be represented as a functionψ:Z⟶Y. The domain Z is a set of binary vectors. The element of Z represents the input signal of implicit memory. The codomain Y is also a set of binary vectors. The element of Y represents the output signal of implicit memory. The set of all ordered pairs z,y represents the training result of implicit memory where z is an element of Z and y is an element of Y. An example of the function is given as follows: the domain is chosen to be the set 000,001,100,110, and the codomain is the set 01,10. Figure 1 shows these mapping relationships.Figure 1
Diagram represents a function with domain000,001,101,110, codomain 01,10, and set of ordered pairs 000,01,001,01,101,10,110,10.A computer can store all ordered pairsz,y in a database. Given an input, the computer can search the database for output. With the database, the computer can simulate the external behaviors of implicit memory. However, this method needs to recall the ordered pairs stored in the database. This internal process is different from the process of implicit memory. The implicit memory does not need to execute memorizing or searching. The implicit memory is similar in operation to a high-speed logic circuit. From one given input, the implicit memory would not have a second’s hesitation of giving output.A logic circuit and a database have different implementations. In the preparation phase, the database stores the input-output pairs on a hard disk. However, the logic circuit connects the logic gates to implement the input-output maps. At run time, the database searches the given input on the hard disk to find an output. In the logic circuit, nevertheless, the input goes through the logic gates to get the output. The complexity of the practice data also makes it arduous to manually design logic circuits. However, ANN can simulate the actions of implicit memory automatically.The capacities of the implicit memory can be represented as a setX=z1,y1,…,zn,yn where zi and yi are binary vectors. Without loss of generality, assume that all of z1,…,zn have the same length and all of y1,…,yn have the same length. And then, the operation of an implicit memory can be expressed with a table. The table lists all allowed input vectors with the corresponding output, as illustrated in Table 1 for an example. The table shows the output for each allowed input. Only the first bit’s implementation of the output signal is considered since the other implementations can operate in the same way. The table can serve as a truth table.Table 1
Example of listing the operation of an implicit memory.
InputOutputProduct termZ1Z2Z3Y1000000101011Z1Z¯2Z31101Z1Z2Z¯3Truth tables are widely used to describe the operation of logic circuits. With a truth table, the sum-of-products (SOP) expression can be written. The Boolean SOP expression can be obtained from the truth table by ORing the product terms, for whichY1=1 is(2)Y1=Z1Z¯2Z3+Z1Z2Z¯3.The first term in the expression is formed by ANDing the three variablesZ1, Z¯2, and Z3. The second term is formed by ANDing the three variables Z1, Z2, and Z¯3. The logic gates can be used to implement the expression. The logic gates are constructed as follows: two inverters to form the Z¯2 and Z¯3 variables; two 3-input AND gates to form the terms Z1Z¯2Z3 and Z1Z2Z¯3; and one 2-input OR gate to form the final output function, Z1Z¯2Z3+Z1Z2Z¯3. Figure 2 shows the logic diagram. Therefore, a logic circuit can simulate the actions of the implicit memory.Figure 2
Logic diagram representsY1=Z1Z¯2Z3+Z1Z2Z¯3.


### 2.1. ANN-Based Boolean Operation

NAND gate is a universal gate because it can be used to produce the NOT, the AND, the OR, and the NOR functions [20]. With NAND gates and appropriate connections, all logic circuits can be built.Suppose there is a sigmoid neuron with two inputs,x1 and x2. The sigmoid neuron has weights for each input, w1, w2, and an overall bias, b. The output of the sigmoid neuron is σw1x1+w2x2+b, where σ is called the sigmoid function and is defined by(3)σy=11+ey.Whenw1=w2=−20 and b=30, the sigmoid neuron is shown in Figure 3. Then the input 00 produces output 1 since σ−20∗0−20∗0+30=σ30≈1. Similar calculations show that the inputs 01 and 10 produce output 1. But input 11 produces output 0 since σ−20∗1−20∗1+30=σ−10≈0. Therefore, the sigmoid neuron can implement a 2-input NAND gate.Figure 3
Sigmoid neuron withw1=w2=−20 and b=30.Let the value ofw2 be 0. The output is not affected by the input x2. The sigmoid neuron equals a 1-input neuron. When w1=−20 and b=10, the sigmoid neuron is shown in Figure 4. Then input 0 produces output 1 since σ−20∗0+10=σ10≈1. But input 1 produces output 0 since σ−20∗1+10=σ−10≈0. The sigmoid neuron can implement a 1-input NOT gate.Figure 4
Sigmoid neuron with one input, with weight −20, and an overall bias of 10.Whenw1=20 and b=−10, the sigmoid neuron is shown in Figure 5. Then input 0 produces output 0 since σ20∗0−10=σ−10≈0. But input 1 produces output 1 since σ20∗1−10=σ10≈1. The sigmoid neuron can implement a connecting line.Figure 5
Sigmoid neuron with one input, with weight 20, and an overall bias of −10.


### 2.2. Simulation of Implicit Memory

According to De Morgan’s laws, Boolean expression can be resolved into 2-input NAND and 1-input NOT. For example,(4)Y1=Z1Z¯2Z3+Z1Z2Z¯3=Z1Z2¯¯¯Z3¯¯+Z1Z2¯¯Z3¯¯¯=Z1Z¯¯2Z3¯¯Z1Z¯2Z3¯¯¯¯.A neural network of five layers can implement expression (4), as is shown in Figure 6. This method of constructing the network has generality. ANN can be configured to execute an arbitrary map. Therefore, ANN can simulate the actions of the implicit memory.Figure 6
ANN can implement the Boolean expressionZ1Z¯2Z3+Z1Z2Z¯3. The neurons are configured to implement the NOT, the NAND, and the connecting line. In general, the neurons have more inputs. While the weight of inputs is 0, the outputs are not affected, which are not drawn.Sometimes a situation arises in which some input variable combinations are not allowed. Because these unallowed states never occur in an application, they can be treated as “do not care” terms. For these “do not care” terms, either a 1 or a 0 may be assigned to the output. The “do not care” terms can be used to simplify an expression. Table 2 shows that, for each “do not care” term, a χ is placed in the output. As indicated in Figure 7, when grouping the 1s on the Karnaugh map, the χs can be treated as 1 s to make a larger grouping or as 0 s if they cannot have the advantage. The larger the group, the simpler the resulting term [20]. With “do not care”, the neural network of five layers can be simplified to a connection line between Y1 and Z1.Table 2
Example of a truth table.
InputOutputZ1Z2Z3Y10000001010111101010χDo not care011χDo not care100χDo not care111χDo not careFigure 7
Example of the use of “do not care” conditions to simplify an expression. Without “do not care”Y1=Z1Z¯2Z3+Z1Z2Z¯3. With “do not care” Y1=Z1. So you can see the advantage of using “do not care” terms to get the simplest expression.Using the ordered pair setX as a training set, the ANN can be trained to simulate the implicit memory. The optimization process of training the ANN has some similar properties to a Boolean expression simplification process. Their purpose is to implement the required functionality with minimal resources, such as logic gates and activated neurons. The brain might have less than 1% neurons active at any given time [21–23]. If an input vector is not included in the training set, then the ANN can treat the input as a “do not care” term. The ANN can output either a 1 or a 0 when the input is a “do not care” term.Suppose the supervised learning algorithm trains an ANN. While the given input is in the training set, the output of the trained ANN can be expected. Otherwise, the output of the trained ANN cannot be expected. Only by actual measuring can the output be known. The measurement process is like sampling from a statistical population. The trained ANN model can be expressed as a functionf. The function f has the following properties:(i)
The output of the function is a specified value when the input is in the training set. That is, ifz,y is in the training set, then fz=y.(ii)
Otherwise, the output can be assigned by generating a sample from a statistical population.The following section proposes a pattern recognition algorithm with the above functionf.


## 2.1. ANN-Based Boolean Operation

NAND gate is a universal gate because it can be used to produce the NOT, the AND, the OR, and the NOR functions [20]. With NAND gates and appropriate connections, all logic circuits can be built.Suppose there is a sigmoid neuron with two inputs,x1 and x2. The sigmoid neuron has weights for each input, w1, w2, and an overall bias, b. The output of the sigmoid neuron is σw1x1+w2x2+b, where σ is called the sigmoid function and is defined by(3)σy=11+ey.Whenw1=w2=−20 and b=30, the sigmoid neuron is shown in Figure 3. Then the input 00 produces output 1 since σ−20∗0−20∗0+30=σ30≈1. Similar calculations show that the inputs 01 and 10 produce output 1. But input 11 produces output 0 since σ−20∗1−20∗1+30=σ−10≈0. Therefore, the sigmoid neuron can implement a 2-input NAND gate.Figure 3
Sigmoid neuron withw1=w2=−20 and b=30.Let the value ofw2 be 0. The output is not affected by the input x2. The sigmoid neuron equals a 1-input neuron. When w1=−20 and b=10, the sigmoid neuron is shown in Figure 4. Then input 0 produces output 1 since σ−20∗0+10=σ10≈1. But input 1 produces output 0 since σ−20∗1+10=σ−10≈0. The sigmoid neuron can implement a 1-input NOT gate.Figure 4
Sigmoid neuron with one input, with weight −20, and an overall bias of 10.Whenw1=20 and b=−10, the sigmoid neuron is shown in Figure 5. Then input 0 produces output 0 since σ20∗0−10=σ−10≈0. But input 1 produces output 1 since σ20∗1−10=σ10≈1. The sigmoid neuron can implement a connecting line.Figure 5
Sigmoid neuron with one input, with weight 20, and an overall bias of −10.


## 2.2. Simulation of Implicit Memory

According to De Morgan’s laws, Boolean expression can be resolved into 2-input NAND and 1-input NOT. For example,(4)Y1=Z1Z¯2Z3+Z1Z2Z¯3=Z1Z2¯¯¯Z3¯¯+Z1Z2¯¯Z3¯¯¯=Z1Z¯¯2Z3¯¯Z1Z¯2Z3¯¯¯¯.A neural network of five layers can implement expression (4), as is shown in Figure 6. This method of constructing the network has generality. ANN can be configured to execute an arbitrary map. Therefore, ANN can simulate the actions of the implicit memory.Figure 6
ANN can implement the Boolean expressionZ1Z¯2Z3+Z1Z2Z¯3. The neurons are configured to implement the NOT, the NAND, and the connecting line. In general, the neurons have more inputs. While the weight of inputs is 0, the outputs are not affected, which are not drawn.Sometimes a situation arises in which some input variable combinations are not allowed. Because these unallowed states never occur in an application, they can be treated as “do not care” terms. For these “do not care” terms, either a 1 or a 0 may be assigned to the output. The “do not care” terms can be used to simplify an expression. Table 2 shows that, for each “do not care” term, a χ is placed in the output. As indicated in Figure 7, when grouping the 1s on the Karnaugh map, the χs can be treated as 1 s to make a larger grouping or as 0 s if they cannot have the advantage. The larger the group, the simpler the resulting term [20]. With “do not care”, the neural network of five layers can be simplified to a connection line between Y1 and Z1.Table 2
Example of a truth table.
InputOutputZ1Z2Z3Y10000001010111101010χDo not care011χDo not care100χDo not care111χDo not careFigure 7
Example of the use of “do not care” conditions to simplify an expression. Without “do not care”Y1=Z1Z¯2Z3+Z1Z2Z¯3. With “do not care” Y1=Z1. So you can see the advantage of using “do not care” terms to get the simplest expression.Using the ordered pair setX as a training set, the ANN can be trained to simulate the implicit memory. The optimization process of training the ANN has some similar properties to a Boolean expression simplification process. Their purpose is to implement the required functionality with minimal resources, such as logic gates and activated neurons. The brain might have less than 1% neurons active at any given time [21–23]. If an input vector is not included in the training set, then the ANN can treat the input as a “do not care” term. The ANN can output either a 1 or a 0 when the input is a “do not care” term.Suppose the supervised learning algorithm trains an ANN. While the given input is in the training set, the output of the trained ANN can be expected. Otherwise, the output of the trained ANN cannot be expected. Only by actual measuring can the output be known. The measurement process is like sampling from a statistical population. The trained ANN model can be expressed as a functionf. The function f has the following properties:(i)
The output of the function is a specified value when the input is in the training set. That is, ifz,y is in the training set, then fz=y.(ii)
Otherwise, the output can be assigned by generating a sample from a statistical population.The following section proposes a pattern recognition algorithm with the above functionf.


## 3. Recognition Based on the Implicit Memory Model

The principle of recognition is based on the intuitive assumption that examples in the same class are closer/similar [24, 25]. Therefore, a recognition algorithm is proposed to estimate the similarity via the implicit memory model.Let’s denote a signal byx=x1,x2,…,xn, where xi∈0,1, i∈N≜1,…,n. When a function f can precisely predict any masked part of the signal x and x˜ is not the same as x, distinguishing between the signal x and any other signal x˜ is feasible. The following theorem describes how to recognize the signal x.Theorem 1.
Consider the signalx=x1,x2,…,xn is a constant. For an arbitrary mask m=m1,…,mn∈0,1n, a function g satisfies gm=x°1−m where 1−m=1−m1,…,1−mn. Construct a function f:0,1n⟶0,1n of the form fgm=x°m. If an input z∈0,1n is not included in the codomain of the function g, then the output of the function f is assigned by generating a signal, v=v1,v2,…,vn, from a random number generator. The generator can produce binary digits, 0, 1, through equal probability sampling. Prvi=0=Prvi=1=1/2, where i∈N. Let x˜=x˜1,x˜2,…,x˜n be a new given signal, and following theorem describes how to recognize the signal x.(5)ζ=12n∑m∈0,1nZfx˜°1−m,x˜°m,where(6)Za,b=1,ifa=b,0,ifa≠b.
Ifx˜≠x, then(7)Prζ=1=0,(8)Prζ=0>1−12n2n⟶1e≈0.3679.
Ifx˜=x, then(9)ζ=12n∑m∈0,1nZfx˜°1−m,x˜°m=12n∑m∈0,1nZfx°1−m,x°m=12n∑m∈0,1nZfgm,x°m=12n∑m∈0,1nZx°m,x°m=1.The proof of Theorem1 is given in Appendix.According to equations (7)–(9), recognizing the signal x with the function f is feasible. When a point can help the function f to retrieve the signal x, the point is called an evoked point. Let(10)D≜x°1−m:m∈0,1n.There areDevoked points, where D=2xl1=2x1+x2+⋯+xn. If Prxi=1=1/2, then the expected value of D is 2n/2.The functionf can retrieve the signal x from the evoked point, h=x˜°1−m, in the current input signal x˜ since the signal x can be expressed as h+fh. By comparing fh with x˜°m, identifying whether x˜ is the same as x or not is feasible. If there exists a mask m∈0,1n that can make fx˜°1−m=x˜°m, then Prx˜=x=1−1/2n. If x˜=x, then each mask m∈0,1n can make fx˜°1−m=x˜°m.In a similar way of recognizing the signalx (Theorem 1), identifying whether the new given signal x˜ is in a set X≜x1,x2,…,xp or not is also feasible.Lemma 1.
Supposexi=xi,1,xi,2,…,xi,n∈0,1n. If the signals, x1,x2,…,xp, satisfy xi≠xj for any i≠j, then there exists p−1 positive integer, α1,α2,…,αp−1, s.t., xi,α1,xi,α2,…,xi,αp−1≠xj,α1,xj,α2,…,xj,αp−1 for any i≠j.The proof of Lemma1 is given in the Appendix.Theorem 2.
Suppose the signals,x1,x2,…,xp, satisfy xi≠xj for any i≠j where xi is a constant. Without loss of generality, assume that xi,1,xi,2,…,xi,p−1≠xj,1,xj,2,…,xj,p−1 for any i≠j. Let(11)Bi=Di∖∪j∈P,j≠iDi∩Dj,where(12)Di≜xi°1−m:m∈0,1n,i∈P≜1,…,p.
Ifxi,1,xi,2,…,xi,p−1≠0,0,…,0, then the cardinality of Bi satisfies that(13)Bi≥2xi,p+xi,p+1+⋯+xi,n.
Ifxi,1,xi,2,…,xi,p−1=0,0,…,0, then(14)Bi=2xi,p+xi,p+1+⋯+xi,n−1.The proof of Theorem2 is given in Appendix.Assume that any signal in setX does not equal 0,0,…,0, i.e., xil1=xi,1+xi,2+⋯+xi,n≥1 for each i∈P. Then, Bi≥1 where i∈P. If Prxi=1=1/2, then the expected value of Bi is 2n−p+1/2.When a new signalx˜ appears, it is possible to identify whether x˜ is in the set X or not with a function f:0,1n⟶0,1n. Construct the function f of the form(15)fxi°1−m=xi°m,for any xi°1−m∈Bi, i∈P. If an input z∈0,1n is not included B1∪⋯∪Bp, then the output of the function f is assigned by generating a signal from a random number generator. If there exists a mask m∈0,1n that can make fx˜°1−m=x˜°m, then Prx˜∈X=1−1/2n. If x˜∈X (for example x˜=xj), then there exists at least one mask m∈0,1n that can make fx˜°1−m=x˜°m since Bj≥1.To identify whether a new given signalx˜ has the same label as X, the distance/similarity between x˜ and X can be used, such as(16)dx˜,X=infdx˜,x|x∈X.Without memorizing any element inX, it is also possible to estimate the similarity by using the function f.Let(17)D˜=x˜°1−m:m∈0,1n,B=B1∪⋯∪Bp.IfD˜∩Bi≠∅, then there exist D˜∩Bi evoked points. Let qi=D˜∩Bi. Each evoked point, hi,j, is corresponding to a mask, mi,j.(18)hi,j=x˜°1−mi,j=xi°1−mi,j,where j∈1,2,…,pi. The evoked points and the masks can be used to retrieve xi. With the function f, xi can be gotten by(19)x˜°1−mi,j+fx˜°1−mi,j=xi°1−mi,j+fxi°1−mi,j=xi°1−mi,j+xi°mi,j=xi.The similarity betweenx˜ and X can be estimated by(20)infdx˜,x˜°1−m+fx˜°1−m|m∈0,1n.The process of similarity estimation is influenced byx˜°1−m∉B. If x˜ has the same meaning as element in X, then x˜ has to overcome the influence. The drawback of this estimation is that the algorithm might not traverse all elements in X. If D˜∩Bi=∅, then there are no evoked points that can help us to retrieve xi. However, the advantage is that intentionally recollecting all elements in X is not necessary.By trainingl functions f1,…,fl to predict the masked part of elements in their respective instance sets, i.e., X1,…,Xl, recognizing a new given signal x˜ as one category is feasible. Let Xi≜xi,1,…,xi,pi, Di,j≜xi,j°1−m:m∈0,1n, and(21)Bi,j=Di,j\∪k∈1,…,pi,k≠jDi,j∩Di,k,where j∈1,…,pi, i∈1,…,l. Function fi satisfies(22)fixi,j°1−m=xi,j°m,for any xi,j°1−m∈Bi,j, j∈1,…,pi. If an input z∈0,1n is not included Bi,1∪⋯∪Bi,pi, then the output of the function fi is assigned by generating a signal from a random number generator.When a signalx˜ is to be labeled, the similarity rule is a natural choice. To reduce the influence on the analysis of similarity, replace the infimum with average. The similarity between x˜ and Xi can be estimated by(23)six˜≜12n∑m∈0,1ndx˜,x˜°1−m+fix˜°1−m.The smaller the similarity is, the more likely they are in the same category and have the same label. The above recognition model is presented in Algorithm1, which is called the Implicit Recognition Model.Algorithm 1: Implicit Recognition Model.
input:Xi=xi,1,xi,2,…,xi,pi is a training instance set and yi is the known label of Xi; fi, labeled yi, is an approximator and can be trained; i∈ℒ; x˜ is a testing instance.∗ Cognitive Process ∗(1)
for each i∈ℒdo(2)
repeat(3)
Observing a signal inXi;(4)
Training the prediction functionfi to predict the masked parts of the signal;(5)
until For each signal xi,k∈Xi and any j≠i, sixi,k<sjxi,k.(6)
end for(7)
return Prediction functions f1,…,fl labeled with y1,…,yl respectively.∗ Recognition Process ∗(1)
for each i∈ℒdo(2)
Estimating the similarity between testing signalx˜ and instance set Xi;(3)
Letςi=six˜;(4)
end for(5)
Assigning the imagex˜ to the class of its highest similarity;(6)
l^=argminiςi ;(7)
returnyl^.In summary, the Implicit Recognition Model uses similarity comparison to do the recognition. But intentionally recalling any labeled signals is not necessary. The focus is not on the past, but on the current input. The evoked points are objective features, which can help us to retrieve labeled signals from the current input. Bothevoked points and retrieved signals have explicit physical meaning in the real world, which are objective pieces of evidence to support the judgment.


## 4. Experiment

In the application, the Mixed National Institute of Standards and Technology (MNIST) database is used to verify the Implicit Recognition Model. The MNIST database is one of the famous image classification benchmarks [26]. There are 60,000 instances in the training set and 10,000 instances in the testing set. The database is created by remixing the samples of NIST’s original data sets. The black and white images from NIST are normalized to fit into a 28×28 pixel bounding box and antialiased, which introduces gray-scale levels [27]. The first 500 elements of the training set are used for training. All of the testing instances are used for testing.Xi is labeled i and only contains training images with label i where i=0,1,…,9. Among the first 500 training images, there are p0=62 training images labeled 0, p1=51 training images labeled 1, and so on. X0+⋯+X9=500. All the 500 images are different. Xi∩Xj=∅ for any i≠j.In the process of recognition, an image in the testing set is recognized as the class of the highest similarity. To execute Algorithm1, a distance function d is redefined first. Cosine distance is a usual measure of similarity in machine learning [28, 29]. Given two vectors a and b, the cosine similarity cosθ is represented by(24)cosθ=abab=∑i=1naibi∑i=1nai2∑i=1nbi2,where ai and bi are components of the vectors a and b, respectively. The angle θ∈0,π is used as the distance between two images in this application. Therefore,(25)da,b=argcosabab.With the artificial neural network, the implicit memory is simulated to execute the Implicit Recognition Model (Algorithm1). The main process of Algorithm 1 is to train 10 approximators of the functions f0,f1,…,f9. Then, fi can be used to retrieve images in Xi from the evoked points of the current input image x˜ and estimate the similarity between x˜ and Xi.To imitate the human recognition process, 10 ANNs are trained in two steps. The first step is to train a base neural network to complete all training images by filling in missing regions of a rectangular shape. This step does not use labels. In order to finally generate the specific functionfi, the second step is to add routing layers to the trained base neural network. With supervision, the routing layers are trained on the instance set Xi. Routing neural networks are used to approximate the specific functions.


### 4.1. Architecture of ANN

A fully convolutional neural network is modified to be the base neural network. Refer to [30] for more details of the original fully convolutional neural network. The modified network parameters are given in Tables 3–6. Behind each convolution layer, except the last one, there is a Rectified Linear Unit (ReLU) layer. The output layer consists of a convolutional layer with a sigmoid function instead of a ReLU layer to normalize the output to the [0, 1] range. “Outputs” refers to the number of output channels for the output of the layer (Tables 3–6). Fully connected (FC) layers refer to the standard neural network layers. The output layer of discriminator consists of a fully connected layer with a sigmoid transfer layer. The discriminator outputs the probability that an input image came from real images rather than the completion network.Table 3
Architecture of base neural network.
TypeKernelDilationμStrideOutputsConv.3×311×114Conv.2×211×128Conv.2×211×128Conv.2×211×156Conv.2×211×156Conv.2×211×156Dilated conv.2×221×156Dilated conv.2×221×156Dilated conv.2×221×156Dilated conv.2×221×156Conv.2×211×156Conv.2×211×156Deconv.2×211×128Conv.2×211×128Deconv.2×211×114Conv.2×211×17Output2×211×11Table 4
Architecture of local discriminator.
TypeKernelStrideOutputsConv.2×21×114Conv.2×21×128Conv.2×21×156Conv.2×21×1112FC——112Table 5
Architecture of global discriminator.
TypeKernelStrideOutputsConv.2×21×114Conv.2×21×128Conv.2×21×156Conv.2×21×1112Conv.2×21×1112FC——112Table 6
Concatenation layer of discriminator.
TypeKernelStrideOutputsConcat——224FC——1Routing layers are added to the trained base neural network to generate an approximator of the specific functionfi. Figure 8 shows the architecture of a routing neural network. Three routing layers are inserted in front of each network layer. The routing layer performs element-wise multiplication between the input x=x1,…,xn and the routing weights w=w1,…,wn, which can be represented as x°w=x1w1,…,xnwn. The initial value of routing weight w is set to 1=1,…,1. The role of routing layers is to keep fixi,k°1−m°m+xi,k°1−m similar to xi,k, but the distance between fjxi,k°1−m°m+xi,k°1−m and xi,k is becoming far with the training, for any j≠i, xi,k∈Xi.Figure 8
Network architectures of the routing neural network and 3 routing layers.


### 4.2. Training Method of ANN

While the base neural network is training, global and local context discriminators are trained to distinguish real images from completed ones. The base neural network is trained to complete images by filling in masked regions of a rectangle. The training operation on the mask setM=m:m∈0,128×28 is too large to calculate. A subset (denoted M′) of M is selected with a rectangle masked part of 7 to 14 for the width and height.(26)M′=m|m=028;…;028;︸a00b0,1b1,0b2;⋮0b0,1b1,0b2;a1028;…;028︸a2,a0+a1+a2=28,b0+b1+b2=28,a1,b1∈7,…,14,where(27)0k=0,…,0︸k,1k=1,…,1︸k.All the 500 training images (denotedX) are used to train the base neural network.When the training of the base neural network is finished, routing layers are added to generate an approximator of specific prediction functionfi. While the routing layers are training, the parameters of the base neural network remain unchanged. Ten digit labels/categories correspond to 10 approximators of prediction functions. Each prediction function fi has a unique set of routing parameters.The training method for routing layers is the same as the base neural network. The routing neural network is also trained to complete images by filling in masked regions of a rectangle. However, only the instance setXi is used to train routing parameters of approximator of fi. The three routing layers are trained one after another. While one routing layer is training, the other two routing layers remain unchanged. While sixi,k is becoming less than sjxi,k for each signal xi,k∈Xi and any j≠i, the cognitive process finishes. The trained routing neural network is an approximator of the specific prediction tool/function fi.


### 4.3. Experimental Results

The neural network models are created by TensorFlow 1.8.0. The learning rate of the Adam optimizer is10−3. A batch size of one image is used for training. First, the base neural network is trained for 100 iterations. Then both the discriminator and base neural network are jointly trained. In each iteration, all 500 training images are shuffled and traversed. For each image, a masked rectangle region is randomly selected to train the neural networks. By 1000 iterations, the base neural network can predict the masked part (Figure 9).Figure 9
Prediction result by a base neural network on a training image with a label of (a) 0, (b) 1, (c) 2, (d) 3, (e) 4, (f) 5, (g) 6, (h) 7, (i) 8, and (j) 9 using randomly generated masks. The mask covers up a rectangular part of the image, and the predicted result is in the box.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)Then, keep the base neural network unchanged, and add routing layers. The first routing layers are trained for 260 iterations and then remain unchanged. The second routing layers are also trained for 260 iterations and then remain unchanged. The approximation of a specific prediction function finishes when the third routing layers are also trained for 260 iterations.A set of routing parameters correspond to a specific prediction function. Each approximator of functionsf0,…,f9 is corresponding to a routing neural network with a particular set of routing parameters. The approximator of function fi can only work on the training set Xi. Figure 10 gives an example. The mean distance di,fi¯ is less than di,fj¯ for each j≠i where di,fj¯≜1/pi∑k=1pisjxi,k (Figure 11).Figure 10
Prediction result by a routing neural network of (a)f0, (b)f1, (c)f2, (d)f3, (e)f4, (f)f5, (g)f6, (h)f7, (i)f8, and (j)f9 on a training image in X6 using a randomly generated mask.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)Figure 11
Mean distancedi,fj¯ for each fj on the training set.With the approximator of the prediction functionfi, the algorithm can estimate the similarity between a testing image x˜ and the instance set Xi. A subset of M′ (denoted Mr) is used to estimate the similarity. Equations (23) and (25) are modified by(28)six˜≜1Mr∑m∈Mrdx˜,x˜°1−m+fix˜°1−m°m.30 masks are randomly generated, andMr=30. The masks remain unchanged when the algorithm calculates ς0,…,ς9. Finally, the image x˜ can be recognized as the digit l^ satisfying that ςl^≤ςi for any i∈0,1,…,9.The masked part is the key for doing the similarity comparison. If the mask’s size is too small,x˜ and x˜°1−m+fx˜°1−m°m are mostly the same. There is no enough data to demonstrate the similarity. The main cause of high similarity should be from the accuracy of prediction (i.e., fx˜°1−m°m is similar to x˜°m), not the small mask window (i.e., ml1 is small). An appropriate size can ensure the estimation results would be more comparable and credible. For a similarity comparison, the mask’s window size cannot be too small. Meanwhile, the size cannot be too large to train ANN. The bigger the size is, the harder the masked parts are to be predicted.Using the Implicit Recognition Model (Algorithm1), the correct recognition probability approaches 100% on the training set (contains 500 training images). “Digit 4” and “digit 9” are easily to be confused. 80.44% recognition results of the Implicit Recognition Model are the same as the labels, and the confusion matrix is shown in Figure 12. Experimental results show the efficiency of the proposed Implicit Recognition Model.Figure 12
Performance of Implicit Recognition Model. Each column of the matrix represents the instances in an original class while each row represents the instances in a recognized class. The figure is accurate to two decimal places.The proposed model recognizes imagex˜ as digit l^, because x˜ is most similar to the instances in Xl^. As the similarity becomes lower, both the recognition precision of the Implicit Recognition Model and the one nearest neighbor algorithm (Explicit Recognition Model) decrease while rotating the testing images (Figure 13). Even using the testing image x˜ itself to do the rotation similarity comparison, the distance dx˜,rx˜,α becomes further still (Figure 14), where rx˜,α represents that image x˜ rotates α degrees. While the rotation angle is around 90 and 270 degrees, the distance approaches 1.22 radians, which is the mean distance between a noise image and the testing image. The pixel of testing image takes integer values between 0 and 255. The pixel of noise image obeys discrete uniform distribution on the integers 0,1,2,…,255. In Figures 13 and 14, the curves show W-shape and M-shape, because the images labeled 1 are similar to their rotated images when the rotation angle is around 180 degrees. Rotation of image can cause similarity changes. Therefore, a rotated image can also be used to define a new category if the similarity change is big enough. For example, the main difference between “W” and “M” is the direction of the opening (Figures 13 and 14).Figure 13
Recognition precision when the testing images rotate 0, 15, 30, 45, 90, 180, 270, 315, 330, 345, and 359 degrees, respectively.Figure 14
Rotation similarity comparison. Solid line represents the mean distance between a rotated image and its original image in the testing set. Dashed line represents the mean distance between a noise image and the testing image.Furthermore, the proposed algorithm is robust while adding random noisen to the testing images. Suppose that n=n1,…,n784 where ni is an independent random variable. Noise point ni obeys discrete uniform distribution on the integers β,β+1,β+2,…,γ, where β and γ restrict the value range 0≤β≤γ≤255. β,γ represents the noise level. The bigger the values, the higher the noise level. The testing images x˜=x˜1,…,x˜784 are disturbed by noise n (Figure 15) to test the robustness of algorithms. The image in noise is denoted as x˜noise=Tx˜1+n1,…,Tx˜784+n784 where(29)Tx0+x1=x0+x1,ifx0+x1≤255,255,ifx0+x1>255.Figure 15
Testing image is disturbed at a noise level of (a) {0, 25}, (b) {0, 76}, (c) {0, 127}, (d) {0, 178}, (e) {0, 229}, (f) {25, 255}, (g) {76, 255}, (h) {127, 255}, (i) {178, 255}, and (j) {229, 255}.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)Using PyTorch 1.5.1, the traditional neural network models are modified to do the recognition, too. Two linear layers after the output layer are added. In front of each linear layer, there is a Rectified Linear Unit (ReLU) layer. The testing images are resized and repeated along the channel. Then, the stochastic gradient descent optimizer and a batch size of 32 images are used for training.By training 1000 iterations, the traditional neural network models can work when the noise level is lower than0,127. However, the performance is not stable with increase of the noise level. The recognition precision of many traditional neural networks drops sharply while the change in the noise level is not big, as shown in Figure 16. However, the performance curve of Alexnet and VGG is smooth, as shown in Figure 17. Overfitting cannot provide the reasons behind the phenomenon since the neural networks are black-boxes. The human can recognize the digits even when the noise level is 229,255. Therefore, the more straight the performance curve, the closer it is to the human capability. When the noise level is 102,255, the recognition accuracy of the traditional neural networks is less than 20%, but the accuracy of the Implicit Recognition Model is about 50%. The Implicit Recognition Model improves the robustness significantly.Figure 16
Recognition precision of Implicit Recognition Model, Explicit Recognition Model, ResNet, SqueezeNet, DenseNet, ResNeXt, Wide ResNet, and MNASNet at noise levels {0, 0}, {0, 25}, {0, 51}, {0, 76}, {0, 102}, {0, 127}, {0, 153}, {0, 178}, {0, 204}, {0, 229}, and {0, 255}.Figure 17
Recognition precision of Implicit Recognition Model, Explicit Recognition Model, Alexnet, and VGG at noise levels {0, 0}, {0, 25}, {0, 51}, {0, 76}, {0, 102}, {0, 127}, {0, 153}, {0, 178}, {0, 204}, {0, 229}, {0, 255}, {25, 255}, {51, 255}, {76, 255},{102, 255}, {127, 255}, {153, 255}, {178, 255}, {204, 255}, and {229, 255}.


## 4.1. Architecture of ANN

A fully convolutional neural network is modified to be the base neural network. Refer to [30] for more details of the original fully convolutional neural network. The modified network parameters are given in Tables 3–6. Behind each convolution layer, except the last one, there is a Rectified Linear Unit (ReLU) layer. The output layer consists of a convolutional layer with a sigmoid function instead of a ReLU layer to normalize the output to the [0, 1] range. “Outputs” refers to the number of output channels for the output of the layer (Tables 3–6). Fully connected (FC) layers refer to the standard neural network layers. The output layer of discriminator consists of a fully connected layer with a sigmoid transfer layer. The discriminator outputs the probability that an input image came from real images rather than the completion network.Table 3
Architecture of base neural network.
TypeKernelDilationμStrideOutputsConv.3×311×114Conv.2×211×128Conv.2×211×128Conv.2×211×156Conv.2×211×156Conv.2×211×156Dilated conv.2×221×156Dilated conv.2×221×156Dilated conv.2×221×156Dilated conv.2×221×156Conv.2×211×156Conv.2×211×156Deconv.2×211×128Conv.2×211×128Deconv.2×211×114Conv.2×211×17Output2×211×11Table 4
Architecture of local discriminator.
TypeKernelStrideOutputsConv.2×21×114Conv.2×21×128Conv.2×21×156Conv.2×21×1112FC——112Table 5
Architecture of global discriminator.
TypeKernelStrideOutputsConv.2×21×114Conv.2×21×128Conv.2×21×156Conv.2×21×1112Conv.2×21×1112FC——112Table 6
Concatenation layer of discriminator.
TypeKernelStrideOutputsConcat——224FC——1Routing layers are added to the trained base neural network to generate an approximator of the specific functionfi. Figure 8 shows the architecture of a routing neural network. Three routing layers are inserted in front of each network layer. The routing layer performs element-wise multiplication between the input x=x1,…,xn and the routing weights w=w1,…,wn, which can be represented as x°w=x1w1,…,xnwn. The initial value of routing weight w is set to 1=1,…,1. The role of routing layers is to keep fixi,k°1−m°m+xi,k°1−m similar to xi,k, but the distance between fjxi,k°1−m°m+xi,k°1−m and xi,k is becoming far with the training, for any j≠i, xi,k∈Xi.Figure 8
Network architectures of the routing neural network and 3 routing layers.


## 4.2. Training Method of ANN

While the base neural network is training, global and local context discriminators are trained to distinguish real images from completed ones. The base neural network is trained to complete images by filling in masked regions of a rectangle. The training operation on the mask setM=m:m∈0,128×28 is too large to calculate. A subset (denoted M′) of M is selected with a rectangle masked part of 7 to 14 for the width and height.(26)M′=m|m=028;…;028;︸a00b0,1b1,0b2;⋮0b0,1b1,0b2;a1028;…;028︸a2,a0+a1+a2=28,b0+b1+b2=28,a1,b1∈7,…,14,where(27)0k=0,…,0︸k,1k=1,…,1︸k.All the 500 training images (denotedX) are used to train the base neural network.When the training of the base neural network is finished, routing layers are added to generate an approximator of specific prediction functionfi. While the routing layers are training, the parameters of the base neural network remain unchanged. Ten digit labels/categories correspond to 10 approximators of prediction functions. Each prediction function fi has a unique set of routing parameters.The training method for routing layers is the same as the base neural network. The routing neural network is also trained to complete images by filling in masked regions of a rectangle. However, only the instance setXi is used to train routing parameters of approximator of fi. The three routing layers are trained one after another. While one routing layer is training, the other two routing layers remain unchanged. While sixi,k is becoming less than sjxi,k for each signal xi,k∈Xi and any j≠i, the cognitive process finishes. The trained routing neural network is an approximator of the specific prediction tool/function fi.


## 4.3. Experimental Results

The neural network models are created by TensorFlow 1.8.0. The learning rate of the Adam optimizer is10−3. A batch size of one image is used for training. First, the base neural network is trained for 100 iterations. Then both the discriminator and base neural network are jointly trained. In each iteration, all 500 training images are shuffled and traversed. For each image, a masked rectangle region is randomly selected to train the neural networks. By 1000 iterations, the base neural network can predict the masked part (Figure 9).Figure 9
Prediction result by a base neural network on a training image with a label of (a) 0, (b) 1, (c) 2, (d) 3, (e) 4, (f) 5, (g) 6, (h) 7, (i) 8, and (j) 9 using randomly generated masks. The mask covers up a rectangular part of the image, and the predicted result is in the box.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)Then, keep the base neural network unchanged, and add routing layers. The first routing layers are trained for 260 iterations and then remain unchanged. The second routing layers are also trained for 260 iterations and then remain unchanged. The approximation of a specific prediction function finishes when the third routing layers are also trained for 260 iterations.A set of routing parameters correspond to a specific prediction function. Each approximator of functionsf0,…,f9 is corresponding to a routing neural network with a particular set of routing parameters. The approximator of function fi can only work on the training set Xi. Figure 10 gives an example. The mean distance di,fi¯ is less than di,fj¯ for each j≠i where di,fj¯≜1/pi∑k=1pisjxi,k (Figure 11).Figure 10
Prediction result by a routing neural network of (a)f0, (b)f1, (c)f2, (d)f3, (e)f4, (f)f5, (g)f6, (h)f7, (i)f8, and (j)f9 on a training image in X6 using a randomly generated mask.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)Figure 11
Mean distancedi,fj¯ for each fj on the training set.With the approximator of the prediction functionfi, the algorithm can estimate the similarity between a testing image x˜ and the instance set Xi. A subset of M′ (denoted Mr) is used to estimate the similarity. Equations (23) and (25) are modified by(28)six˜≜1Mr∑m∈Mrdx˜,x˜°1−m+fix˜°1−m°m.30 masks are randomly generated, andMr=30. The masks remain unchanged when the algorithm calculates ς0,…,ς9. Finally, the image x˜ can be recognized as the digit l^ satisfying that ςl^≤ςi for any i∈0,1,…,9.The masked part is the key for doing the similarity comparison. If the mask’s size is too small,x˜ and x˜°1−m+fx˜°1−m°m are mostly the same. There is no enough data to demonstrate the similarity. The main cause of high similarity should be from the accuracy of prediction (i.e., fx˜°1−m°m is similar to x˜°m), not the small mask window (i.e., ml1 is small). An appropriate size can ensure the estimation results would be more comparable and credible. For a similarity comparison, the mask’s window size cannot be too small. Meanwhile, the size cannot be too large to train ANN. The bigger the size is, the harder the masked parts are to be predicted.Using the Implicit Recognition Model (Algorithm1), the correct recognition probability approaches 100% on the training set (contains 500 training images). “Digit 4” and “digit 9” are easily to be confused. 80.44% recognition results of the Implicit Recognition Model are the same as the labels, and the confusion matrix is shown in Figure 12. Experimental results show the efficiency of the proposed Implicit Recognition Model.Figure 12
Performance of Implicit Recognition Model. Each column of the matrix represents the instances in an original class while each row represents the instances in a recognized class. The figure is accurate to two decimal places.The proposed model recognizes imagex˜ as digit l^, because x˜ is most similar to the instances in Xl^. As the similarity becomes lower, both the recognition precision of the Implicit Recognition Model and the one nearest neighbor algorithm (Explicit Recognition Model) decrease while rotating the testing images (Figure 13). Even using the testing image x˜ itself to do the rotation similarity comparison, the distance dx˜,rx˜,α becomes further still (Figure 14), where rx˜,α represents that image x˜ rotates α degrees. While the rotation angle is around 90 and 270 degrees, the distance approaches 1.22 radians, which is the mean distance between a noise image and the testing image. The pixel of testing image takes integer values between 0 and 255. The pixel of noise image obeys discrete uniform distribution on the integers 0,1,2,…,255. In Figures 13 and 14, the curves show W-shape and M-shape, because the images labeled 1 are similar to their rotated images when the rotation angle is around 180 degrees. Rotation of image can cause similarity changes. Therefore, a rotated image can also be used to define a new category if the similarity change is big enough. For example, the main difference between “W” and “M” is the direction of the opening (Figures 13 and 14).Figure 13
Recognition precision when the testing images rotate 0, 15, 30, 45, 90, 180, 270, 315, 330, 345, and 359 degrees, respectively.Figure 14
Rotation similarity comparison. Solid line represents the mean distance between a rotated image and its original image in the testing set. Dashed line represents the mean distance between a noise image and the testing image.Furthermore, the proposed algorithm is robust while adding random noisen to the testing images. Suppose that n=n1,…,n784 where ni is an independent random variable. Noise point ni obeys discrete uniform distribution on the integers β,β+1,β+2,…,γ, where β and γ restrict the value range 0≤β≤γ≤255. β,γ represents the noise level. The bigger the values, the higher the noise level. The testing images x˜=x˜1,…,x˜784 are disturbed by noise n (Figure 15) to test the robustness of algorithms. The image in noise is denoted as x˜noise=Tx˜1+n1,…,Tx˜784+n784 where(29)Tx0+x1=x0+x1,ifx0+x1≤255,255,ifx0+x1>255.Figure 15
Testing image is disturbed at a noise level of (a) {0, 25}, (b) {0, 76}, (c) {0, 127}, (d) {0, 178}, (e) {0, 229}, (f) {25, 255}, (g) {76, 255}, (h) {127, 255}, (i) {178, 255}, and (j) {229, 255}.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)Using PyTorch 1.5.1, the traditional neural network models are modified to do the recognition, too. Two linear layers after the output layer are added. In front of each linear layer, there is a Rectified Linear Unit (ReLU) layer. The testing images are resized and repeated along the channel. Then, the stochastic gradient descent optimizer and a batch size of 32 images are used for training.By training 1000 iterations, the traditional neural network models can work when the noise level is lower than0,127. However, the performance is not stable with increase of the noise level. The recognition precision of many traditional neural networks drops sharply while the change in the noise level is not big, as shown in Figure 16. However, the performance curve of Alexnet and VGG is smooth, as shown in Figure 17. Overfitting cannot provide the reasons behind the phenomenon since the neural networks are black-boxes. The human can recognize the digits even when the noise level is 229,255. Therefore, the more straight the performance curve, the closer it is to the human capability. When the noise level is 102,255, the recognition accuracy of the traditional neural networks is less than 20%, but the accuracy of the Implicit Recognition Model is about 50%. The Implicit Recognition Model improves the robustness significantly.Figure 16
Recognition precision of Implicit Recognition Model, Explicit Recognition Model, ResNet, SqueezeNet, DenseNet, ResNeXt, Wide ResNet, and MNASNet at noise levels {0, 0}, {0, 25}, {0, 51}, {0, 76}, {0, 102}, {0, 127}, {0, 153}, {0, 178}, {0, 204}, {0, 229}, and {0, 255}.Figure 17
Recognition precision of Implicit Recognition Model, Explicit Recognition Model, Alexnet, and VGG at noise levels {0, 0}, {0, 25}, {0, 51}, {0, 76}, {0, 102}, {0, 127}, {0, 153}, {0, 178}, {0, 204}, {0, 229}, {0, 255}, {25, 255}, {51, 255}, {76, 255},{102, 255}, {127, 255}, {153, 255}, {178, 255}, {204, 255}, and {229, 255}.


## 5. Conclusion

Scientists early know the existence of implicit memory. This paper establishes a mathematical model of implicit memory and explains ANN can simulate the model from the view of digital logic circuit design. A trained ANN can be expressed as a function. When a given input is not in the training set, the output of the ANN is hard to control. With the function, this paper proposes a new pattern recognition method, the Implicit Recognition Model. The Implicit Recognition Model works under the similarity rule and has interpretability. Compared to the one nearest neighbor algorithm (Explicit Recognition Model), the Implicit Recognition Model makes similarity comparisons without recalling any instances. The experiment results show the efficiency of the Implicit Recognition Model.



---

*Source: 4472174-2021-07-17.xml*