Patent Publication Number: US-8990132-B2

Title: Artificial neural networks based on a low-order model of biological neural networks

Description:
1 CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is related to the provisional patent application Ser. No. 61/475,819, filed Apr. 15, 2010 and the provisional patent application Ser. No. 61/434,109, filed Jan. 19, 2010 submitted by the present inventor. 
     2 BACKGROUND AND ADVANTAGES OF THE INVENTION 
     The invention herein disclosed comprises artificial neural networks (ANNs) and systems and methods based thereon for most of the applications of ANNs such as clustering, detecting or recognizing spatial, hierarchical and/or temporal patterns of objects or causes; understanding images or videos; recognizing speeches, handwriting and texts; and generating representations of probability distributions of labels of patterns of objects and causes, where data may contain erasure, smear, noise, occlusion, distortion, alteration, rotation, translation and/or scaling. The ANNs (artificial neural networks) are based on a low-order model of biological neural networks and have applications in a large number of fields such as computer vision, signal processing, financial engineering, telecommunication, data clustering, and data mining. Example applications are handwritten character/word classification, face recognition, fingerprint identification, DNA sequence identification, speech recognition, machine fault detection, baggage/container examination, video monitoring/understanding, image understanding, scene analysis, text/speech understanding, automatic target recognition, medical diagnosis, prosthesis control, robotic arm control, and vehicle navigation. 
     A good introduction to the prior art in ANNs (artificial neural networks) and their applications can be found in Simon Haykin,  Neural Networks and Learning Machines , Third Edition, Pearson Education, New Jersey, 2009; Christopher M. Bishop,  Pattern Recognition and Machine Learning , Springer Science, New York, 2006. 
     An ANN, which is a functional model of biological neural networks, was recently reported in James Ting-Ho Lo, Functional Model of Biological Neural Networks,  Cognitive Neurodynamics , Vol. 4, Issue 4, pp. 295-313, November 2010, where the ANN is called the temporal hierarchical probabilistic associative memory (THPAM), and in James Ting-Ho Lo, A Cortex-Like Learning Machine for Temporal and Hierarchical Pattern Recognition, U.S. patent application Ser. No. 12/471,341, filed May 22, 2009; Publication No. US-2009-0290800-A1, Publication Date Nov. 26, 2009, where the ANN is called the probabilistic associative memory (PAM). The ANN is hereinafter referred to as the THPAM. The goal to achieve in the construction of the THPAM was to develop an ANN that performs Hebbian-type unsupervised and supervised learning without differentiation, optimization or iteration; retrieves easily; and recognizes corrupted, distorted and occluded temporal and spatial information. In the process to achieve the goal, mathematical necessity took precedence over biological plausibility. This mathematical approach focused first on minimum mathematical structures and operations that are required for an effective learning machine with the mentioned properties. 
     The THPAM turned out to be a functional model of biological neural networks with many unique features that well-known models such as the recurrent multilayer perceptron, associative memories, spiking neural networks, and cortical circuit models do not have. However, the THPAM has been found to have shortcomings. Among them the most serious one is the inability of its unsupervised correlation rule to prevent clusters from overgrowing under certain circumstances. These shortcomings motivated further research to improve the THPAM. At the same time, the unique features of the THPAM indicated that it might contain clues for understanding the structures and operations of biological neural networks. To achieve this understanding and eliminate the mentioned shortcomings, the components of the THPAM were examined from the biological point of view with the purpose of constructing a biologically plausible model of biological neural networks. More specifically, the components of the THPAM were identified with those of biological neural networks and reconstructed, if necessary, into biologically plausible models of the same. 
     This effort resulted in a low-order model (LOM) of biological neural networks and an improved functional model called the Clustering Interpreting Probabilistic Associative Memory (CIPAM). They were respectively reported in the articles, James Ting-Ho Lo, A Low-Order Model of Biological Neural Networks,  Neural Computation , Vol. 23, No. 10, pp. 2626-2682, October 2011; and James Ting-Ho Lo, A Cortex-Like Learning Machine for Temporal Hierarchical Pattern Clustering, Detection, and Recognition,  Neurocomputing , Vol. 78, pp. 89-103, 2012, which are both incorporated into the present invention disclosure by reference. Note that “dendritic and axonal encoders”, “dendritic and axonal trees” and “dendritic and axonal expansions” in them are collectively called “neuronal encoders”, “neuronal trees” and “neuronal codes” respectively in the present invention disclosure and that “C-neuron”, “D-neuron” and “expansion covariance matrix” are called “nonspiking neuron”, “spiking neuron” and “code covariance matrices” respectively in the present invention disclosure. 
     It was subsequently discovered that the LOM and the CIPAM are equivalent in the sense that their corresponding components can mathematically be transformed into each other. In fact, generalizing the mathematical transformation that transforms the LOM and the CIPAM into each other, we can transform the LOM and the CIPAM into infinitely many equivalent models. 
     The LOM, the CIPAM and their equivalent models are each a network of models of the biological neuronal node or encoder (which is a biological dendritic or axonal node or encoder), synapse, spiking/nonspiking neuron, means for learning, feedback connection, maximal generalization scheme, feedback connection, etc. For simplicity, these component models are sometimes referred to without the word “model”. For example, the model neuronal node, model neuronal encoder, model neuronal tree, model synapse, model spiking/nonspiking neuron, etc. will be referred to as the neuronal node/encoder/tree, synapse, spiking/nonspiking neuron, etc. respectively. The LOM, the CIPAM and all their equivalent models can be used as artificial neural networks (ANNs). To emphasize that their components are artificial components in these artificial neural networks, they are referred to as the artificial neuronal node/encoder/tree, artificial synapse, artificial spiking/nonspiking neuron, etc. respectively. 
     If there is possibility of confusion, the real components in the brain are referred to with the adjective “biological”, for example, the biological neuronal node, biological neuronal encoder, biological neuronal tree, biological spiking/nonspiking neuron, and biological synapse, etc. The components of equivalent models (or equivalent ANNs) that can be obtained from transforming a component model of the LOM are given the same name of the said component of the LOM. In other words, all the model components (of the equivalent ANNs) that are equivalent to one another are given the same component name. 
     All models that are equivalent to the LOM including the LOM and the CIPAM use the “unsupervised covariance rule” instead of the “unsupervised correlation rule” used in the THPAM and can prevent the clusters of patterns or causes formed in synapses from overgrowing. Moreover, all models that are equivalent to the LOM including the LOM and the CIPAM use the “supervised covariance rule” instead of the “supervised correlation rule” used in the THPAM. These are two of the main improvements in the LOM, the CIPAM and other equivalent models over the THAPM. 
     From the application viewpoint, as an ANN, the LOM (or a mathematical equivalent thereof) has the following advantages:
         1. No label of the learning data from outside the ANN is needed for the UPUs (unsupervised processing units) in the LOM to learn.   2. The unsupervised learning by a processing unit clusters data without involving selecting a fixed number of prototypes, cycling through the data, using prototypes as cluster labels, or minimizing a non-convex criterion.   3. Both the unsupervised and supervised covariance rules are of the Hebbian type, involving no differentiation, backpropagation, optimization, iteration, or cycling through the data. They learn virtually with “photographic memories”, and are suited for online adaptive learning. Large numbers of large temporal and spatial data such as photographs, radiographs, videos, speech/language, text/knowledge, etc. are learned easily. The “decision boundaries” are not determined by exemplary patterns from each and every pattern and “confuser” class, but by those from pattern classes. In many applications such as target and face recognition, there are a great many pattern and “confuser” classes and usually no or not enough exemplary patterns for some “confuser classes”.   4. Only a small number of algorithmic steps are needed for retrieving or estimating labels. Detection and recognition of multiple/hierarchical temporal/spatial causes are easily performed. Massive parallelization at the bit level by VLSI implementation is suitable.   5. empirical probability distributions and membership functions of labels are easily obtained by supervised processing units (SPUs) and unsupervised processing units (UPUs).   6. The ANN generalizes not by only a single holistic similarity criterion for the entire input exogenous feature vector, which noise; erasure; distortion and occlusion can easily defeat, but by a large number of similarity criteria for feature subvectors input to a large number of UPUs (processing units) in different layers. These criteria contribute individually and collectively to generalization for single and multiple causes. Example 1: smiling; putting on a hat; growing or shaving beard; or wearing a wig can upset a single similarity criterion used for recognizing a face in a mug-shot photograph. However, a face can be recognized by each of a large number of feature subvectors of the face. If one of them is recognized to belong to a certain face, the face is recognized. Example 2: a typical kitchen contains a refrigerator, a counter top, sinks, faucets, stoves, fruit and vegetable on a table, etc. The kitchen is still a kitchen if a couple of items, say the stoves and the table with fruit and vegetable, are removed.   7. Masking matrices in a PU (processing unit) eliminate effects of corrupted, distorted and occluded components of the feature subvector input to the PU, and thereby enable maximal generalization capability of the PU, and in turn that of the ANN.   8. The ANN is no more a blackbox with “fully connected” layers much criticized by opponents of such neural networks as multilayer perceptrons (MLPs) or recurrent MLPs. In a PU of the ANN, synaptic weights are covariances between neuronal codes and labels of the vector input to the PU. Each PU has a receptive field in the exogenous feature vector input to the ANN and recognizes the pattern(s) or cause(s) appearing within the receptive field. Such properties can be used to help select the architecture (i.e., layers, PUs, connections, feedback structures, etc.) of the ANN for the application.   9. The ANN (or a mathematical equivalent thereof) may have some capability of recognizing rotated, translated and scaled patterns. Moreover, easy learning and retrieving by an ANN allow it to learn translated, rotated and scaled versions of an input image with ease.   10. The hierarchical architecture of the clusterer stores models of the hierarchical temporal and spatial worlds (e.g., letters, words and sentences).   11. Ambiguity and uncertainty are represented and resolved with empirical probabilities and membership degrees in the sense of fuzzy logic.   12. Noises and interferences in inputs self-destruct like random walks with residues eliminated gradually by forgetting factors in the synapses, leaving essential informations that have been learned by repetitions and emphases.   13. The architecture of the ANN can be adjusted without discarding learned knowledge in the ANN. This allows enlargement of the feature subvectors, increase of the number of layers, and even increase of feedback connections.       

     For simplicity and clarity of the present invention disclosure, we will mainly describe the LOM in the present invention disclosure and will also show how the LOM is transformed into the CIPAM and other ANNs that are mathematically equivalent to it by the use of affine functions and their inverses. 
     3 SUMMARY 
     In this Section, we first describe briefly the LOM (Low-Order Model of biological neural networks) and its equivalents including the CIPAM (Clustering Interpreting Probabilistic Associative Memory). All these equivalents and the LOM can be transformed into one another by affine functions and their inverses. The components of the LOM and their corresponding components of those equivalents are models of the corresponding components of biological neural networks. Each model component of the LOM and its corresponding model components of those equivalents can be transformed into one another by affine functions and their inverses and are therefore given the same name. Such components include the model neuronal node, model neuronal encoder, model neuronal tree, model synapse, model spiking neuron, model nonspiking neuron, etc. 
     The LOM and its equivalents form a new paradigm of artificial neural networks and can be used for the wide range of data processing applications that artificial neural networks are intended for. Said applications include clustering data, detecting objects or patterns, recognizing or classifying patterns or objects. In applications, the LOM and its equivalents are artificial neural networks, and their model components, which are mentioned above, are the artificial neuronal (axonal or dendritic) node, artificial neuronal (axonal or dendritic) encoder, artificial neuronal (axonal or dendritic) tree, artificial synapse, artificial spiking neuron, artificial nonspiking neuron, etc. In the present invention disclosure, the words, “model” and “artificial” in front of a component are used interchangeably, depending on whether the emphasis is placed on modeling of a biological component or on application of the model. 
     For simplicity, the LOM is described in the present invention disclosure with the understanding that the description is valid for the equivalents of the LOM after transformation by proper affine functions and their inverses. Notice also that in this invention disclosure, although a dendrite or axon is a part of a neuron, and a dendro-dendritic synapse is a part of a dendrite (thus a part of a neuron), they are treated, for simplicity, as if they were separate entities, and the word “neuron” refers essentially to the soma of a neuron in this specification. Similarly, artificial dendrites, artificial axons, and artificial synapses are treated, for simplicity, as if they were separate from artificial nonspiking/spiking neurons, and the term “artificial nonspiking/spiking neuron” refers essentially to the model of a soma of a neuron in the present invention disclosure. 
     In this Summary, references to subsection and subsubsection numbers in the Section entitled “DESCRIPTION OF PREFERRED EMBODIMENTS” are made. It is stressed that the LOM is described in the present invention disclosure with the understanding that the description is valid for the equivalents of the LOM. 
     Object 1 of the present invention is to provide an artificial neural network, a learning machine, and a method for identifying or approximating a known or unknown function or dynamical system. 
     Object 2 of the present invention is to provide an artificial neural network, a learning machine, and a method for nonparametric nonlinear classification for spatial or temporal data. 
     Object 3 of the present invention is to provide an artificial neural network, a learning machine, and a method for nonparametric nonlinear classification for spatially or temporally hierarchical data. 
     Object 4 of the present invention is to provide an artificial neural network, a learning machine, and a method for recognizing spatial or temporal patterns. 
     Object 5 of the present invention is to provide an artificial neural network, a learning machine, and a method for recognizing spatially or temporally hierarchical patterns. 
     Object 6 of the present invention is to provide an artificial neural network, a learning machine, and a method for understanding images or videos. 
     Object 7 of the present invention is to provide an artificial neural network, a learning machine, and a method for Objects 1-6 where said artificial neural network, learning machine, and said method are trained on data with erasure, smear, noise, occlusion, distortion, and/or alteration. 
     Object 8 of the present invention is to provide an artificial neural network, a learning machine, and a method for generating representations of probability distributions of labels of the vector or its subvectors input to the artificial neural network, learning machine, or the method. 
     Object 9 of the present invention is to provide an artificial neural network, a learning machine, and a method for data fusion, data mining, decision making, predicting a financial time series, or searching the internet. 
     Object 10 of the present invention is to provide an artificial neural network, a learning machine, and a method for unsupervised learning of data or clustering of data. 
     Object 11 of the present invention is to provide an artificial neural network, a learning machine, and a method for supervised learning of data. 
     Object 12 of the present invention is to provide an artificial neural network, a learning machine, and a method to perform both supervised and unsupervised learning of data. 
     Object 13 of the present invention is to provide an artificial neural network and a learning machine whose architecture can be adjusted without discarding learned knowledge. 
     Object 14 of the present invention is to provide a method wherein the dimensionalities of unsupervised or supervised covariance matrices can be adjusted without discarding learned knowledge. 
     Object 15 of the present invention is to provide an artificial neural network and a learning machine with a hierarchical architecture for recognizing hierarchical patterns at different levels. 
     Object 16 of the present invention is to provide an artificial neural network and a learning machine with feedback connections for processing sequences of vectors such as those obtained from examining single images at consecutive time points, multiple images of an object taken from different angles, consecutive frames in a video or movie, and handwritten or printed letters in a word, words in a sentence, and sentences in a paragraph. 
     Object 17 of the present invention is to provide a method that employs feedback structures for processing sequences of vectors such as those obtained from examining single images at consecutive time points, multiple images of an object taken from different angles, consecutive frames in a video or movie, and handwritten or printed letters in a word, words in a sentence, and sentences in a paragraph. 
     Object 18 of the present invention is to provide an artificial neural network, a learning machine, and a method for recognizing rotated, translated and/or scaled patterns. 
     The foregoing objects, as well as other objects of the present invention that will become apparent from the discussion below, are achieved by the present invention. 
     As an artificial neural network (ANN), the LOM (or any of its equivalents) described in the Section on Description of Preferred Embodiments is a discrete-time multilayer network of processing units (PUs) with or without feedback connections. A PU includes some or all of the following components;
         1. at least one artificial neuronal encoder for encoding a vector into a neuronal code;   2. a means for evaluating a code deviation vector, which is the deviation of a neuronal code obtained by said artificial neuronal encoder from a neuronal code average;   3. a plurality of artificial synapses each for storing a component of a code deviation accumulation vector;   4. a first means for evaluating a first product of a component of a code deviation accumulation vector, a masking factor, and a component of a code deviation vector;   5. an artificial nonspiking neuron for evaluating a first sum of first products obtained by said first means;   6. a plurality of artificial synapses each for storing an entry of a code covariance matrix;   7. a second means for evaluating a second product of an entry of a code covariance matrix, a masking factor, and a component of a code deviation vector; and   8. at least one artificial spiking neuron for evaluating a second sum of second products obtained by said second means, for using said second sum and a first sum obtained by said nonspiking neuron processor to evaluate a representation of a first empirical probability distribution of a component of a label of a vector that is input to said processing unit, and for generating a pseudorandom number in accordance with said first empirical probability distribution,
 
wherein the number R of artificial spiking neurons is the number of binary digits in the label of the vector input to the PU, the R pseudorandom numbers generated by said artificial spiking neurons are binary digits (i.e., 0 and 1) and form a point estimate of the label of the vector input to said PU. Over time, the point estimates generated by a PU form R spike trains output from the PU. The effect of the masking factors, which are nonnegative integers, can also be achieved either by having numerous artificial neuronal encoders with input vectors of different dimensionalities (i.e., input vectors with different numbers of components) or by including multiple copies of said first products and multiple copies of said second products in the first and second sum respectively.
       

     A vector input to a PU first goes through artificial neuronal encoders (Subsection 5.1), which are networks of artificial neuronal nodes and form the upper part of the artificial neuronal trees. Artificial neuronal nodes in the LOM are each a hyperbolic polynomial with two variables, which acts approximately like an XOR (exclusive-OR) logic gate with an accuracy depending on how close the two inputs to the node are to binary digits (Subsubsection 5.1.1). By forming function compositions of such hyperbolic polynomials, which are commutative and associative binary operations, an artificial neuronal node may have more than two input variables. 
     An artificial neuronal encoder, which is a network of neuronal nodes or function compositions of many hyperbolic polynomials, can be looked upon as a function that encodes its input vector into a neuronal code with an orthogonal property (Subsubsection 5.1.3). After neuronal codes are produced by artificial neuronal encoders, code deviation vectors that are each a deviation of a neuronal code from a neuronal code average over a time window are then computed. 
     The weight (or strength) of an artificial synapse is either an entry of a code covariance matrix or a component of a code deviation accumulation vector. A component of a code deviation accumulation vector, a masking factor, and a component of a code deviation vector are multiplied to yield a first product. An entry of a code covariance matrix, a masking factor, and a component of a code deviation vector are multiplied to yield a second product. The masking factor is a diagonal entry of a masking matrix corresponding to the artificial synapse. 
     An artificial nonspiking neuron in the PU evaluates a first sum of products each of a first product (Subsubsection 5.5.4). An artificial spiking neuron evaluates a second sum of second products, uses said second sum and a first sum obtained by the artificial nonspiking neuron to evaluate a representation of an empirical probability distribution of a component of a label of a vector that is input to said processing unit, and generates a pseudorandom number in accordance with said empirical probability distribution (Subsubsection 5.5.5). The R pseudorandom numbers generated by the R artificial spiking neurons in the PU are binary digits (i.e., 0 and 1) and form a point estimate of the label of the vector input to said PU, such labels and their point estimates being R-dimensional binary vectors. Over time, the point estimates generated by a PU form R spike trains output from the PU. 
     There are three learning rules: the unsupervised covariance rule, supervised covariance rule and unsupervised accumulation rule. The unsupervised covariance rule is used to adjust weights of synapses with post-synaptic artificial spiking neurons whose outputs are teaching signals (i.e., desired or target outputs from the post-synaptic artificial spiking neurons) (Subsubsection 5.2.1). The supervised covariance rule is used to adjust weights of synapses using teaching signals (i.e., desired or target outputs from the post-synaptic artificial spiking neurons) or a given label from outside the PU (Subsubsection 5.2.2). The unsupervised accumulation rule is used to adjust weights of synapses with a post-synaptic artificial non-spiking neuron by accumulating deviations of the neuronal codes from their averages over time (Subsubsection 5.2.3). 
     A forgetting factor λ and a normalizing constant A are used to keep the entries in code covariance matrices and code deviation accumulation vectors bounded. A PU usually learns by the unsupervised accumulation rule and one of the other two learning rules. In some applications, a processing unit can perform both supervised and unsupervised covariance learning, depending on whether a teaching signal from outside the LOM is available or not. 
     By the unsupervised covariance rule, the output vector from the R artificial spiking neurons (i.e., model spiking neurons) in the PU is assigned as the label (or teaching signals) to be learned jointly with the vector input to the PU. If the input vector or a variation thereof has not been learned before, the output vector from the R artificial spiking neurons is a purely random label. If the input vector or a variation of it has been learned before, the output vector, which is a point estimate of the label or labels of the input vector based on the empirical probability distribution, is learned jointly with the input vector. Supervised covariance learning is performed when teaching signals (i.e., labels of input vectors to the PU) from outside the PU are provided (Subsection 5.2.2). In the third type of learning, namely unsupervised accumulation learning, no label is needed. 
     Maximal generalization capability of the PU is achieved mainly with masking matrices each automatically finding the largest subvector of the code deviation vector that matches a subvector of a code deviation vector that has been learned and stored in the code covariance matrix (or the code deviation accumulation vector) and setting the rest of the components of the former code deviation vector equal to zero. A masking matrix can be viewed as idealization and organization of neuronal encoders with overlapped and nested input vectors (Section 5.4). 
     A PU is sometimes required to learn to distinguish learned input vectors with small differences, but to recognize unlearned input vectors with larger differences with learned input vectors. In this case, masking matrices for retrieving and masking matrices for unsupervised learning should be different. To distinguish these two types of masking matrix, the former and the latter are respectively called the masking matrices and the learning masking matrices, and similarly, their entries are respectively called masking factors and learning masking factors. 
     To use learning masking factors or matrices for unsupervised learning, a processing unit should further include:
         1. a third means for evaluating a third product of a component of a code deviation accumulation vector, a learning masking factor, and a component of a code deviation vector;   2. a summing means for evaluating a third sum of third products obtained by said third means;   3. a fourth means for evaluating a fourth product of an entry of a code covariance matrix, a learning masking factor, and a component of a code deviation vector;   4. a summing-evaluating means for evaluating a fourth sum of fourth products obtained by said fourth means and for using said fourth sum and a third sum obtained by said summing means to evaluate a representation of a second empirical probability distribution of a component of a label of a vector that is input to said processing unit; and   5. an unsupervised learning means for using at least one pseudorandom number generated in accordance with said second empirical probability distribution and at least one component of a code deviation vector to adjust at least one entry of a code covariance matrix by an unsupervised covariance rule in response to a vector that is input to said processing unit.       

     A PU that has completed learning or continues to learn by the unsupervised covariance rule is called an unsupervised processing unit (UPU). A PU that has completed learning or continues to learn by the supervised covariance rule is called a supervised processing unit (SPU). The LOM may have UPU in the lower layers and have SPUs in the higher layers. Alternatively, the LOM may have a network of UPUs only and have SPUs that branch out from the UPUs in said network. As unsupervised learning requires no label from outside the PU, it can be performed anytime on data without a given or handcrafted label. What the network of UPUs actually does is clustering of spatial and temporal patterns in the data. When supervised learning occurs, a cluster is assigned with the same label given or handcrafted. Therefore, the network of UPUs is called a clusterer, and the set of SPUs branching out from the clusterer is called an interpreter. 
     In the following, embodiments of the present invention disclosed herein are described. The mathematical symbols are used for explanation only, and are not intended to restrict the embodiments. 
     Embodiment 1 of the present invention disclosed herein is an artificial neural network that comprises at least one processing unit, a first processing unit including (a) at least one artificial neuronal encoder for encoding a vector v t (ψ) that is input to said encoder into a neuronal code {hacek over (v)} t (ψ) that is output from the encoder; (b) a means for evaluating a code deviation vector {hacek over (v)} τj (ψ)−&lt;{hacek over (v)} τj (ψ)&gt;, which is the deviation of a neuronal code {hacek over (v)} τj (ψ) obtained by said artificial neuronal encoder from a neuronal code average &lt;{hacek over (v)} τj (ψ)&gt;; (c) a plurality of artificial synapses each for storing an entry C j (ψ) of a code deviation accumulation vector 
                 C   ⁡     (   ψ   )       =       Λ   2     ⁢       ∑     s   =   1     t     ⁢         λ     t   -   s       ⁡     (           v   ⋓     s     ⁡     (   ψ   )       -     〈         v   ⋓     s     ⁡     (   ψ   )       〉       )       ′           ,         
(d) a first means for evaluating a first product c τj (ψ)=C j (ψ)M jj ({hacek over (v)} τj (ψ)−&lt;{hacek over (v)} τj (ψ)&gt;) of said entry C j (ψ), a masking factor M jj (ψ) and a component {hacek over (v)} τj (ψ)−&lt;{hacek over (v)} τj (ψ)&gt; of a code deviation vector {hacek over (v)} τ (ψ)−&lt;{hacek over (v)} τ (ψ)&gt; that is the deviation of a neuronal code {hacek over (v)} τ (ψ) from a neuronal code average &lt;{hacek over (v)} τ (ψ)&gt;; (e) an artificial nonspiking neuron processor for evaluating a first sum c τ =Σ ψ=1   Ψ Σ j=1   dim v     τ     (ψ) c τj (ψ) of first products obtained by said first means; (f) a plurality of artificial synapses each for storing an entry D kj (ψ) of a code covariance matrix D(ψ)=ΛΣ s=1   t λ t−s (r s −½)({hacek over (v)} s (ψ)−&lt;{hacek over (v)} s (ψ)&gt;)′; (g) a second means for evaluating a second product d τk =D kj (ψ)M jj ({hacek over (v)} τj (ψ)−&lt;{hacek over (v)} τj (ψ)&gt;) of said entry D kj (ψ), a masking factor M jj , and a component ({hacek over (v)} τj (ψ)−&lt;{hacek over (v)} τj (ψ)&gt; of a code deviation vector {hacek over (v)} τ (ψ)−&lt;{hacek over (v)} τ (ψ)&gt; that is a deviation of a neuronal code {hacek over (v)} τ (ψ) from a neuronal code average &lt;{hacek over (v)} τj (ψ)&gt;; and (h) at least one artificial spiking neuron for evaluating a second sum d τk =Σ ψ=1   Ψ Σ j=1   dim v     τ     (ψ) d τkj (ψ), k=1, . . . , R, of second products obtained by said second means and for using said second sum d τk  and a first sum c τ  obtained by said nonspiking neuron to evaluate a representation of a first empirical probability distribution p τi  of a component of a label of a vector v τ  that is input to said first processing unit. (See  FIGS. 3-4 ,  10 ,  14 - 21 .)
 
     Embodiment 1 can be generalized such that a plurality of code covariance matrices are submatrices of a general code covariance matrix, a plurality of code deviation accumulation vectors are subvectors of a general code deviation accumulation vector, a plurality of masking factors are entries of a masking matrix, a plurality of masking matrices are submatrices of a general masking matrix, and a plurality of neuronal codes are subvectors of a general neuronal code. 
     Embodiment 2 is Embodiment 1, wherein a plurality of code covariance matrices D(ψ), ψ=1, . . . , Ψ, in the processing unit are submatrices of a general code covariance matrix D=[D(1) D(2) . . . D(Ψ)], a plurality of code deviation accumulation vectors C(ψ), ψ=1, . . . , Ψ, in the processing unit in the first embodiment are subvectors of a general code deviation accumulation vector C=[C(1) C(2) . . . C(Ψ)], a plurality of masking factors M jj (ψ) in the processing unit in the first embodiment are entries of a masking matrix M(ψ), a plurality of masking matrices M(ψ), ψ=1, . . . , Ψ, in the processing unit in the first embodiment are submatrices of a general masking matrix M=diag[M(1) M(2) . . . M(Ψ)], a plurality of neuronal codes {hacek over (v)} t (ψ), ψ=1, . . . , Ψ, in the processing unit in the first embodiment are subvectors of a general neuronal code {hacek over (v)} t =[{hacek over (v)} t ′(1) {hacek over (v)} t ′(2) . . . {hacek over (v)} t ′(Ψ)]′ and a plurality of neuronal code averages &lt;{hacek over (v)} t (ψ)&gt;, ψ=1, . . . , Ψ, are subvectors of a general neuronal code average &lt;{hacek over (v)} t &gt;=[&lt;{hacek over (v)} t ′(1)&gt; &lt;{hacek over (v)} t ′(2)&gt;. . . &lt;{hacek over (v)} t ′(Ψ)&gt;]′. (See  FIG. 18 .) 
     Embodiment 3 is Embodiment 1, further comprising feedback connections with time delay means for processing temporal data such as videos or enhancing the performance in processing spatial data. (See  FIGS. 25-27 .) 
     Embodiment 4 is Embodiment 1, wherein said first processing unit further includes an unsupervised accumulation learning means for adjusting (or learning online or offline) at least one component of a code deviation accumulation vector without supervision by an unsupervised accumulation rule, 
                   C   j     ⁡     (   ψ   )       ←         λC   j     ⁡     (   ψ   )       +       Λ   2     ⁢       (           v   ⋓     tj     ⁢     (   ψ   )       -     〈         v   ⋓     tj     ⁡     (   ψ   )       〉       )     ′           ,         
in response to a vector v t  that is input to said first processing unit. (See  FIGS. 7   b ,  22 .)
 
     Embodiment 5 is Embodiment 1, wherein said first processing unit further includes a pseudorandom number generation means for generating at least one pseudorandom number u ti =v{p τi } in accordance with a first empirical probability distribution obtained by said at least one artificial spiking neuron processor. (See  FIG. 20 .) 
     Embodiment 6 is Embodiment 5, wherein said first processing unit further includes an unsupervised learning means for using a pseudorandom number u τi  generated by said pseudo-random number generation means and at least one entry {hacek over (v)} τj (ψ)−&lt;{hacek over (v)} τj (ψ)&gt; of a code deviation vector to adjust at least one entry D ij  of a code covariance matrix by an unsupervised covariance rule, D ij (ψ)←λD ij (ψ)+Λ(u τi −&lt;u τi &gt;)({hacek over (v)} τj (ψ)−&lt;{hacek over (v)} τj (ψ)&gt;)′, in response to a vector v τ  that is input to said first processing unit. (See  FIGS. 5   b ,  22 .) 
     Embodiment 7 is Embodiment 1, wherein said first PU (processing unit) uses learning masking factors M jj   #  (or matrices M # (ψ)) for unsupervised learning. How the learning masking factors are used are described earlier on in this Summary. (See  FIG. 24 .) 
     Embodiment 8 is Embodiment 1, wherein said first processing unit further includes a supervised learning means for using at least one entry {hacek over (v)} τj (ψ)−&lt;{hacek over (v)} τj (ψ)&gt; of a code deviation vector and at least one component w τi  of a given label w τ  of a vector v τ  that is provided from outside said first processing unit to adjust at least one entry D ij (ψ) of a code covariance matrix D(ψ) by a supervised covariance rule, D ij (ψ)←λD ij (ψ)+Λ(w τi −&lt;w τi &gt;)({hacek over (v)} τj (ψ)−&lt;{hacek over (v)} τj (ψ)&gt;)′. A given label w τ  is a label provided from outside the first processing unit. It is usually handcrafted or manually determined. (See  FIGS. 6   b ,  23 .) 
     Embodiment 9 is Embodiment 1 whose architecture comprises a network of unsupervised processing units (UPUs) and set of supervised processing units (SPUs), wherein the code deviation accumulation vectors and the code covariance matrices in the UPUs have been learned by unsupervised learning, and those in the SPUs have been learned by supervised learning. The network of UPUs puts learned data in clusters and is therefore called a clusterer. Each UPU creates its own label of a vector input to it and thus has its own vocabulary. The SPUs are used to translate the labels in the UPUs&#39; vocabularies into the vocabulary of the artificial neural network&#39;s user. Hence, the set of SPUs is called an interpreter. (See  FIG. 27 ) 
     Embodiment 10 of the present invention is a learning machine comprising at least one processing unit, a first processing unit including (a) an encoding means for encoding a vector v τ (ψ) that is input to said encoder into a neuronal code {hacek over (v)} τ (ψ) that is output from the encoder; (b) a means for evaluating a code deviation vector {hacek over (v)} τj (ψ)−&lt;{hacek over (v)} τj (ψ)&gt; that is the deviation of a code {hacek over (v)} τj (ψ) obtained by said encoding means from a code average &lt;{hacek over (v)} τj (ψ)&gt;; (c) memory means for storing at least one first weighted sum αD kj (ψ)+βC j (ψ) of an entry D kj (ψ) of a code covariance matrix D(ψ) and a component C j (ψ) of a code deviation accumulation vector C(ψ), where α and β are real-valued weights; (d) a memory means for storing at least one masking factor M jj ; (e) a multiplying means for evaluating a product of a second weighted sum of an entry of a code covariance matrix and a component of a code deviation accumulation vector, a masking factor, and a code deviation vector; (f) a summing means for evaluating a sum of products obtained by said multiplying means; (g) an evaluation means for using at least a sum obtained by said summing means to evaluate a representation of an empirical probability distribution p τ  of a label r τ  of a vector v τ  input to said first processing unit. 
     The real-valued weights α and β in the second main embodiment can be any real numbers. Embodiment 11 is Embodiment 10, wherein the first weighted sum αD kj (ψ)+βC j (ψ) has α=0 and β=1 and is a component C j (ψ) of the code deviation accumulation vector C(ψ). Embodiment 12 is Embodiment 10, wherein the first weighted sum αD kj (ψ)+βC j (ψ) has α=1 and β=0 and is an entry D kj (ψ) of a code covariance matrix D(ψ). Embodiment 13 is Embodiment 10, wherein the first weighted sum αD kj (ψ)+βC j (ψ) has α=1 and β=1 and is a sum D kj (ψ)+C j (ψ). 
     Embodiment 14 is Embodiment 10, wherein said first processing unit further includes a memory means for storing at least one learning masking factor M jj   # (ψ) (or matrices M # (ψ)); and a second evaluation means for using at least a weighted sum of an entry of a code covariance matrix and a component of a code deviation accumulation vector; a learning masking factor; and a component of a code deviation vector to evaluate a representation of a second empirical probability distribution of a component of a label of a vector input to said first processing unit. The weighted sum αD kj (ψ)+βC j (ψ) learned with the second empirical probability distribution can enable Embodiment 14 to distinguish learned patterns with smaller differences and to recognize unlearned patterns by a better ability of generalization on learned patterns than with the first empirical probability distribution. 
     Embodiment 15 is Embodiment 10, wherein said first processing unit further includes a pseudorandom number generation means for generating at least one pseudorandom number v i {p τ } in accordance with a first empirical probability distribution p τi  obtained by said evaluation means. The pseudorandom number generated is an estimate of a component of the label of a vector input v τ  to said first processing unit. 
     Embodiment 16 is Embodiment 15, wherein said first processing unit further includes an unsupervised learning means for using at least a pseudorandom number generated by said pseudorandom number generation means and an entry of a code deviation vector to adjust at least one first weighted sum αD kj (ψ)+βC j (ψ) of an entry of a code covariance matrix D kj (ψ) and a component C j (ψ) of a code deviation accumulation vector. 
     Embodiment 17 is Embodiment 16, wherein α=0 and β=1 and the weighted sum αD kj (ψ)+βC j (ψ) is a component C j (ψ) of the code deviation accumulation vector C(ψ). Embodiment 18 is Embodiment 10, wherein α=1 and β=0 and the weighted sum αD kj (ψ)+βC j (ψ) is an entry D kj (ψ) of a code covariance matrix D(ψ). Note that if α=1 and β=1, then the weighted sum αD kj (ψ)+βC j (ψ) is a sum D kj (ψ)+C j (ψ) of an entry of a code covariance matrix and a component of a code deviation accumulation vector. 
     Embodiment 19 is Embodiment 15, further comprising a feedback means for feedbacking at least one pseudorandom number generated by said pseudorandom number generation means to a processing unit after a time delay. Feedback means can enable Embodiment 10 to process temporal data and enhance the performance of the learning machine to process spatial data. 
     Embodiment 20 is Embodiment 10, wherein the PU further includes a supervised learning means for adjusting at least one entry of a code covariance matrix by a supervised covariance rule. 
     Embodiment 21 is Embodiment 10, wherein a plurality of processing units are unsupervised processing units, and at least one processing unit is a supervised processing unit. The plurality of processing units is called the clusterer, and the at least one processing unit is called the interpreter. 
     Embodiment 22 is a method comprising a step of encoding a subvector of a first vector into a code; a step of evaluating a component {hacek over (v)} τj (ψ)−&lt;{hacek over (v)} τj (ψ)&gt; of a code deviation vector {hacek over (v)} τ (ψ)−&lt;{hacek over (v)} τ (ψ)&gt; that is the deviation of a code from a code average; a step of evaluating a product of a weighted sum αD kj (ψ)+βC j (ψ) of an entry of a code covariance matrix D kj (ψ) and a component C j (ψ) of a code deviation accumulation vector, where a and/are real-valued weights, a masking factor, and a component of a code deviation vector; a step of evaluating a sum of products obtained by said step of evaluating a product; and a step of using at least a sum of products obtained by said step of evaluating a sum to evaluate a representation of an empirical probability distribution p τ  of a component r τi  of a label r τ  of said first vector v τ . 
     Embodiment 23 is Embodiment 22, wherein a weighted sum αD kj (ψ)+βC j (ψ) has α=0 and β=1 and is C j (ψ). Embodiment 24 is Embodiment 22, wherein a weighted sum αD kj (ψ)+βC j (ψ) has α=1 and β=0 and is D kj (ψ). Embodiment 25 is Embodiment 22, wherein a weighted sum αD kj (ψ)+βC j (ψ) has α=1 and β=1 and is D kj (ψ)+C j (ψ). 
     Embodiment 26 is Embodiment 22, further comprising a step of using at least a component of a code deviation vector and a component w τi  of a given label of said first vector to adjust at least one entry D ij (ψ) of a code covariance matrix by a supervised covariance rule D ij (ψ)←λD ij (ψ)+Λ(w τi −&lt;w τi &gt;)({hacek over (v)} τj (ψ)−&lt;{hacek over (v)} τj (ψ)&gt;)′, said component of a given label being provided for said method to use. w τi  is not produced by Embodiment 26. It is usually handcrafted or manually determined. 
     Embodiment 27 is Embodiment 22, further comprising a step of generating a pseudo-random number v{p τi } in accordance with an empirical probability distribution p τi  of a component of a label of said first vector. The pseudorandom number is an estimate of a component of a label of said first vector. 
     Embodiment 28 is Embodiment 27, further comprising a step of comprising including a pseudorandom number generated in accordance with an empirical probability distribution as a component in said first vector after a time delay. The pseudorandom number is an estimate of a component of a label of said first vector. Said step is a step of providing a feedback of the pseudorandom number after a time delay. Feedbacks are helpful in enhancing performances in processing data. 
     Embodiment 29 is Embodiment 27, further comprising a step of adjusting a weighted sum αD kj (ψ)+βC j (ψ) of a component of a code deviation accumulation vector and an entry of a code covariance matrix. 
     Embodiment 30 is Embodiment 29, wherein said step of adjusting a weighted sum αD kj (ψ)+βC j (ψ) is a step of adjusting a component C j (ψ) of a code deviation accumulation vector. 
     Embodiment 31 is Embodiment 29, wherein said step of adjusting a weighted sum αD kj (ψ)+βC j (ψ) is adjusting an entry D kj (ψ) of a code covariance matrix D(ψ) by an unsupervised covariance rule. 
     Embodiment 32 is Embodiment 29, further comprising a step of using at least a code deviation vector and a given label of said first vector to adjust a code covariance matrix by a supervised covariance rule, said given label being provided for said method to use. Embodiment 32 is a method comprising both unsupervised and supervised learning for both clustering and interpreting data. 
     Embodiment 33 is Embodiment 22, further comprising a step of evaluating a second product of a weighted sum of a component of a code deviation accumulation vector and an entry of a code covariance matrix, a learning masking factor, and a component of a code deviation vector; a step of evaluating a sum of second products obtained by said step of evaluating a second product; and a step of using at least a sum of second products obtained by said step of evaluating a sum of second products to evaluate a representation of a second empirical probability distribution of a component of a label of said first vector. The use of learning masking factors for learning and masking factors for retrieving allow distinguishing learned vectors with smaller differences and recognizing unlearned vectors with larger differences. 
     Embodiment 34 is Embodiment 22, further comprising a step of using at least a sum of second products obtained by said step of evaluating a sum of second products and a component of a given label of said first vector to adjust a code covariance matrix by a supervised covariance rule, said given label being provided for said method to use. Embodiment 34 is a method that includes both supervised learning and unsupervised learning for both clustering and interpreting data, where unsupervised learning is performed with learning masking factors. 
     4 DESCRIPTION OF DRAWINGS 
     In the present invention disclosure, the prime ′ denotes matrix transposition, a vector is regarded as a subvector of the vector itself, a column or row vector is also a matrix, and [a ij ] denotes a matrix whose (i, j)th entry is a ij . If a ij , i=1, . . . , m, j=1, . . . , n, then they are contained in the m×n matrix [a ij ]. Evaluating a ij , i=1, . . . , m, j=1, . . . , n, is equivalent to evaluating the m×n matrix [a ij ]. 
     Preferred embodiments of the present invention will now be further described in the following paragraphs of the specification and may be better understood when read in conjunction with the attached drawings, in which: 
       FIG. 1  illustrates an artificial neuronal node 7, represented by a solid dot, in the LOM. The artificial neuronal node has two input variables, v and u, and one output φ(v, u)=−2vu+v+u. If v and u are binary digits, φ(v, u) acts like the logic gate, XOR (exclusive-or). 
       FIG. 2  is a three-dimensional graph of the output φ(v, u) of an artificial neuronal node over the unit square [0, 1] 2 . Note that the domain of the function φ(v, u) contains the unit square as a subset. The saddle shape of φ(v, u) shows that the artificial neuronal node is robust: If strengths of spikes change in travelling through dendrites/exons or if the spikes are corrupted by biological noises in the dendrites/exons, the outputs of the artificial neuronal node suffer from a graceful degradation. The hyperbolic polynomial −2vu+v+u is an idealized approximation of the XOR logic gate and henceforth called the XOR polynomial. 
       FIG. 3  illustrates an artificial neuronal encoder with four inputs v ti , i=1, . . . , 4; and 16 outputs listed on the right side of the artificial neuronal encoder. It is an upper part of an artificial neuronal tree, performing neuronal encoding. The solid dots represent artificial neuronal nodes as shown in  FIG. 1 . Because of the commutativity and associativity of the XOR polynomial, there are many possible branching structures with the same inputs and outputs. 
       FIG. 4  illustrates an artificial neuronal encoder 6 that generates the 2 m -dimensional neuronal code {hacek over (v)} of an m-dimensional vector v input to the artificial neuronal encoder in a recursive manner. 
       FIG. 5   a  illustrates a deviation evaluator 12 that evaluates the code deviation {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; of an artificial neuronal code {hacek over (v)} tj  from it average &lt;{hacek over (v)} tj &gt;, where the average 
               〈       v   ⋓     tj     〉     ⁢           ⁢   is   ⁢           ⁢     1     q   v       ⁢       ∑     τ   =     t   -     q   v     +   1       t     ⁢       v   ⋓     tj             
of {hacek over (v)} tj  over a time window [t−q v +1, t] with the window width q v ; an unsupervised learning scheme that adjusts an entry D ij  of the code covariance matrix D using the unsupervised covariance rule  14   a ; a unit-time delay  8 ; an artificial synapse denoted by {circle around (x)}  18   d  that stores the entry D ij  of the code covariance matrix D after a unit-time delay  8  and evaluates the product D ij ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;); and artificial spiking neuron i  10  that uses such products from artificial synapses and a graded signal c t  from an artificial nonspiking neuron to generate an output u ti . Note that the unsupervised learning rule  14   a  inputs the artificial code deviation {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; and the output u ti  from the artificial spiking neuron i  10  as the ith component r ti  of the label r t  of v t .
 
       FIG. 6   a  illustrates a deviation evaluator 12 that evaluates the code deviation {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; of an artificial neuronal code {hacek over (v)} tj  from its average &lt;{hacek over (v)} tj &gt;, where the average 
               〈       v   ⋓     tj     〉     ⁢           ⁢   is   ⁢           ⁢     1     q   v       ⁢       ∑     τ   =     t   -     q   v     +   1       t     ⁢       v   ⋓     tj             
of {hacek over (v)} tj  over a time window [t−q v +1, t] with the window width q v ; a supervised learning scheme that adjusts an entry D ij  of the code covariance matrix D using the supervised covariance rule  14   b ; a unit-time delay  8 ; an artificial synapse {circle around (x)}  18   d  that stores the entry D ij  of the code covariance matrix D after a unit-time delay and evaluates the product D ij ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;); and artificial spiking neuron i  10  that uses such products from artificial synapses and a graded signal c t  from an artificial nonspiking neuron to generate an output u ti . Note that the supervised learning rule  14   b  inputs the code deviation {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; and the component w ti  of a label w t  of the measurements in the receptive field of model spiking neuron i, that w t  is provided from outside the LOM (e.g., a handcrafted label w t ) as the ith component r ti  of the label r t  for the unsupervised covariance rule.
 
       FIG. 7   a  illustrates a deviation evaluator 12 that evaluates the code deviation {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; of an artificial neuronal code {hacek over (v)} tj  from its average &lt;{hacek over (v)} tj &gt;, where the average &lt;{hacek over (v)} tj &gt; is 
               1     q   v       ⁢       ∑     τ   =     t   -     q   v     +   1       t     ⁢         v   ⋓     tj     ⁢           ⁢   of   ⁢           ⁢       v   ⋓     tj               
over a time window [t−q v +1, t] with the window length q v ; an unsupervised accumulation learning scheme  20  that adjusts an entry C j  of the code deviation accumulation vector C using the unsupervised accumulation rule  20 ; a unit-time delay  8 ; an artificial synapse {circle around (x)}  18   c  that stores the entry C j  of the code deviation accumulation vector C after a unit-time delay and evaluates the product C j ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;); and artificial nonspiking neuron i  16  that sums such products from artificial synapses as an output c t . Note that the unsupervised accumulation rule  20  inputs only the component {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;) of the code deviation vector {hacek over (v)} t −&lt;{hacek over (v)} t &gt;.
 
       FIG. 8  shows an artificial synapse denoted by {circle around (x)}  18   d  that stores an entry D ij  of the code covariance matrix D and performs the multiplication of its input {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; and its weight D ij .  FIG. 8  also shows an artificial synapse denoted by {circle around (x)}  18   c  that stores an entry C j  of the code deviation accumulation vector C and performs the multiplication of its input {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; and its weight C j . 
       FIG. 9  shows the data for supervised learning in EXAMPLE 2a, unsupervised learning in EXAMPLE 2b, and unsupervised learning in EXAMPLE 2c at the vertices of a cube. The digits  17  in the squares at the vertices are labels for supervised learning. The question marks  15  indicate that the labels are unknown and unsupervised learning is necessary. 
       FIG. 10  shows an artificial synapse denoted by {circle around (x)}  18   d  that stores an entry D ij  of the code covariance matrix D and performs the multiplication of its input {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; and its weight D ij . Denoting the jth diagonal entry of the masking matrix M by M jj , the product D ij ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;) obtained by the artificial synapse is multiplied by the masking factor M jj    22 , resulting in d τij =M jj D ij ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;), which is the (i, j)th entry of the matrix [d τij ].  FIG. 10  also shows an artificial synapse denoted by {circle around (x)}  18   c  that stores an entry C j  of the code deviation accumulation matrix C and performs the multiplication of its input {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; and its weight C j . The product C j ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;) obtained by the artificial synapse denoted by {circle around (x)}  18   c  is multiplied by the masking factor M jj    22 , resulting in c τj =M jj C j ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;), which is the jth entry of the row vector [c τj ]. 
       FIG. 11  illustrates an artificial neuronal encoder, wherein artificial neuronal nodes  7  are represented by large dots; an artificial nonspiking neuron  16 , an artificial spiking neuron  10 , the artificial synapses denoted by {circle around (x)}  18   c ,  18   d , and the masking matrix M denoted by the dashed line  15 , that are used in EXAMPLES 2a, 2b and 2c. The dashed line  15  represents a masking factor following an artificial synapse on each feedforward connection. In unsupervised accumulation learning, the weights in artificial synapses  18   c  are adjusted by the unsupervised accumulation rule  20  as shown in  FIG. 7   b . For unsupervised covariance learning, the teaching signal r τ  is the output vector u τ =v{p τ } of the spiking neuron, and the unsupervised covariance rule ( 14   a  in  FIG. 5   b ) is used to adjust the weights in artificial synapses denoted by  18   d . For supervised covariance learning, the teaching signal r τ  is a label w τ  of input vector v τ , that is provided from outside the LOM, and the supervised covariance rule ( 14   b  in  FIG. 6   b ) is used to adjust the weights in artificial synapses denoted by {circle around (x)}  18   d.    
       FIG. 12  illustrates the masking matrix M  21  for an artificial neuronal encoder that encodes an m-dimensional vector v t  into its 2 m -dimensional neuronal code {hacek over (v)} t . M is a 2 m ×2 m  diagonal matrix diag(M 11  . . . M 2     m     2     m   ), whose diagonal entries are called masking factors. M{hacek over (v)} t  eliminates (or reduces) the effect of the corrupted, distorted or occluded components of v t  to enable generalization on other components. The greater J is, the effect of more components of v t  can be eliminated (or reduced). The factor 2 −nj  is used to de-emphasize the effect of subvectors of v t  in accordance with the number j of components of v t  included in (i 1   − , i 2   − , . . . , i j   − ). The constant J is preselected with consideration of dim v t  and the application. 
       FIG. 5   b  is the same as  FIG. 5   a  except the addition of the masking factor M jj    22  that is multiplied to the output D ij ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;) from the artificial synapse  18   d  to yield the product d tij =M jj D ij ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;). Artificial spiking neuron i  10  then uses such products and a graded signal c t  from an artificial nonspiking neuron to generate an output u ti . Note that the unsupervised learning rule  14   a  inputs the artificial code deviation {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; and the output u ti  from the artificial spiking neuron i  10  as the ith component r ti  of the label r t  of v t . 
       FIG. 6   b  is the same as  FIG. 6   a  except the addition of the masking factor M jj    22  that is multiplied to the output D ij ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;) from the artificial synapse  18   d  to yield the product d tij =M jj D ij ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;). Artificial spiking neuron i  10  then uses such products and a graded signal c t  from an artificial nonspiking neuron to generate an output u ti . Note that the supervised learning rule  14   b  inputs the code deviation {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; and the component w ti  of a label w t  of the measurements in the receptive field of model spiking neuron i, that w t  is provided from outside the LOM (e.g., a handcrafted label w t ) as the ith component r ti  of the label r t  for the unsupervised covariance rule. 
       FIG. 7   b  is the same as  FIG. 7   a  except the addition of the masking factor M jj    22  that is multiplied to the output C j ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;) from the artificial synapse  18   c  to yield the product c tj =M jj C j ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;). Artificial nonspiking neuron  16  then sums such products as an output c t . Note that the unsupervised accumulation rule inputs only the component {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; of the code deviation vector {hacek over (v)} t −&lt;{hacek over (v)} t &gt;. 
       FIG. 13  shows that the product M{hacek over (v)} τ  in EXAMPLE 3 consists of four terms, {hacek over (v)} τ , (diagÎ(1 − )){hacek over (v)} τ , (diagÎ(2 − )){hacek over (v)} τ , and (diagÎ(3 − )){hacek over (v)} τ , where (diagÎ(j − )){hacek over (v)} τ  has the components in {hacek over (v)} τ  that involve v τj  eliminated. These four terms can be viewed as the outputs of four artificial neuronal encoders shown in (a)  26 , (b)  28 , (c)  30  and (d)  32  in response to four overlapped and nested input vectors, v τ =[v τ1  v τ2  v τ3 ]′, [v τ2  v τ3 ]′, [v τ1  v τ3 ]′, and [v τ1  v τ2 ]′, respectively. The product M{hacek over (v)} τ  is the sum of the outputs in (a), (b), (c) and (d). Although there are two parts for (diagÎ(3 − )){hacek over (v)} τ  in (d), they can be replaced with a single neuronal encoder, namely the upper neuronal (e.g., dendritic) part in (d) with φ(v τ2 , v τ2 )=0 added to replace φ(v τ2 , v τ2 )=0 in the lower neuronal (e.g., dendritic) part. 
       FIG. 14  illustrates three neuronal encoders  36  in a PU (processing unit). Let v t (ψ), ψ=1, 2, 3, be subvectors of the vector v t    34  input to the PU, where the components of v t (ψ) are randomly selected from those of v t  such that the union of all the components of v t (ψ), ψ=1, 2, 3, is the set of components of v t . The locations of the components of v t (1), v t (2) and v t (3) are indicated by x, y and z in the circle  34 , respectively. The subvectors v t (ψ), ψ=1, 2, 3, are encoded by model neuronal encoders  36  into neuronal codes v t (ψ), ψ=1, 2, 3, which form the general neuronal code {hacek over (v)} t =[v t ′(1) v t ′(2) v t ′(3)]′ of the vector v t  input to the PU. Note that this v t  here is different from the neuronal code of the vector v t  input to the PU. 
       FIG. 15  illustrates two artificial neuronal encoders that input subvectors, v τ (1) and v τ (2), of the vector v τ . v τ (1) and v τ (2) are overlapped. With the masking factors indicated by the dashed line denoted by M, the two artificial neuronal encoders are used to encode one input vector v τ  into the general neuronal code {hacek over (v)} t  of v t  to reduce the number of synapses (from 16 to 12) required and to increase the generalization capability. 
       FIG. 16  illustrates the group  40  of artificial neuronal encoders (NEs)  36  in a PU (processing unit). The vector input to the PU is v t . Let v t (ψ), ψ=1, . . . , Ψ, be subvectors of the vector v t    34  input to the PU, where the components of v t (ψ) are randomly selected from v t  such that the union of all the components of v t (ψ), ψ=1, . . . , Ψ, is the set of components of v t . The subvectors v t (ψ), ψ=1, . . . , Ψ, are encoded by artificial neuronal encoders  36  into neuronal codes v t (ψ), ψ=1, . . . , Ψ, which form the general neuronal code {hacek over (v)} t =[{hacek over (v)} t ′(1) . . . {hacek over (v)} t ′(Ψ)]′ of the vector input to the PU. 
       FIG. 17  illustrates the general masking matrix M in a PU, wherein M(ψ) is the masking matrix for neuronal encoder ψ for ψ=1, . . . , Ψ. Note that η can be preselected for a given magnitude of dim v t (ψ) and the application. M{hacek over (v)} t , where {hacek over (v)} t  is the general neuronal code, reduces or eliminates the effect of the corrupted, distorted, occluded, noised, or erased components of the vector v t  input to the PU. Note that the vector [τ j  M jj D ij ({hacek over (v)} τj −&lt;{hacek over (v)} τj &gt;)], where Σ j  is taken over j=1, . . . , Σ ψ=1   Ψ 2 dim v(ψ) , is equal to the vector DM({hacek over (v)} τ −&lt;{hacek over (v)} τ &gt;), because M is a diagonal matrix. 
       FIG. 18  provides the formulas 42 of the general neuronal code {hacek over (v)} t , the average general neuronal code &lt;{hacek over (v)} t &gt;, the general code deviation accumulation vectors C, and the general code covariance matrix D, whose D(ψ), ψ=1, . . . , Ψ, are learned by the unsupervised covariance rule in an unsupervised PU (UPU) or by the supervised covariance rule in a supervised PU (SPU). 
       FIG. 19  illustrates an artificial nonspiking neuron  16  in a PU, which evaluates the sum c τ  of its inputs, c τj , j=1, . . . , Σ ψ=1   Ψ 2 dim v     τ     (ψ) , which are computed using the general neuronal code deviation {hacek over (v)} τ −&lt;{hacek over (v)} τ &gt;, the general masking matrix M and the general code deviation accumulation vector C. 
       FIG. 20  illustrates artificial spiking neuron k  10  in a PU. It inputs d τkj , j=1, . . . , Σ ψ=1   Ψ 2 dim v     τ     (ψ) , which are computed using c τ  and the general neuronal code deviation {hacek over (v)} τ −&lt;{hacek over (v)} τ &gt;, the general masking matrix M and the general code covariance matrix D; and evaluates the sum Σ j d τkj  of d τkj , j=1, . . . , Σ ψ=1   Ψ 2 dim v     τ     (ψ) . If c τ =0, the artificial spiking neuron k then sets p τk =½. If c τ ≠0, the artificial spiking neuron k then sets p τk =((Σ j d τkj )/c τ +1)/2. The artificial spiking neuron generates a pseudo-random number v{p τk } as follows: v{p τk }=1 with probability p τk  and v{p τk }=0 with probability 1−p τk . 
       FIG. 21  illustrates a PU (processing unit)  56  without a learning mechanism. A general code deviation accumulation vector C and a general code covariance matrix D are stored in synapses  48 . If the D was learned before in accordance with the unsupervised covariance rule, the PU  56  is called an unsupervised processing unit (UPU). If D was learned before in accordance with the supervised covariance rule, the PU  56  is called a supervised processing unit (SPU). At time or numbering τ, the PU receives v τ . The neuronal encoders  40  encode it into the general neuronal code {hacek over (v)} τ . The synapses  48  compute the entries of the matrix [D kj ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;)] and the vector [C j ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;)], which entries are multiplied to the masking factors M jj    46  to yield the matrix [d τkj ]=[M jj D kj ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;)] and the vector [c τj ]=[M jj C j ({hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;)]. The masking factor M jj  is the jth diagonal entry of the general masking matrix M  46 . 
     There are R artificial spiking neurons  50  and one artificial nonspiking neuron  16  in the PU. The artificial nonspiking neuron sums up c τj  to form c τ . For k=1, . . . , R, artificial spiking neuron k sums up d τkj  to form d τk  and divides d τk  by c τ  to get y τk . The vector p τ  with components p τk =(y τk +1)/2 represents an empirical probability distribution of the label of v τ  input to the PU For k=1, . . . , R, artificial spiking neuron k generates a pseudorandom number u τk =v{p τk } in accordance with the empirical probability p τk . The pseudo-random vector v{p τ } is a point estimate of the label of v τ  according to the empirical probability distribution p τ  and is the output vector of the PU  56  at time or numbering τ. Over time, the R components of u τ =v{p τ } form R spike trains. 
       FIG. 22  illustrates an UPU with an unsupervised learning mechanism  60  to learn the code covariance matrix D and an unsupervised learning mechanism  58  to learn the code deviation accumulation vector C. The former  60  learns in accordance with the unsupervised covariance rule,
 
 D←λD +Λ( u   τ   −&lt;u   τ &gt;)( {hacek over (v)}   τ   −&lt;{hacek over (v)}   τ &gt;)′
 
where u τ =v{p τ } that is the output of the PU and is actually used by the PU as a label r τ  of v τ , and the latter  58  learns in accordance with the unsupervised accumulation rule
 
             C   ←       λ   ⁢           ⁢   C     +       Λ   2     ⁢       (         v   ⋓     τ     -     〈       v   ⋓     τ     〉       )     ′               
The adjusted C and D are delayed for one unit of time or one numbering  8  before being stored in the artificial synapses  48 .
 
       FIG. 23  illustrates an SPU with a supervised learning mechanism  62  to learn the code covariance matrix D and an unsupervised learning mechanism  58  to learn the code deviation accumulation vector C. The former  62  learns in accordance with the supervised covariance rule,
 
 D←λD +Λ( r   τ −½)( {hacek over (v)}   τ   −&lt;{hacek over (v)}   τ &gt;)′
 
where r τ  denotes the label of the vector v τ  input to the SPU 64, and the latter  58  learns in accordance with the unsupervised accumulation rule
 
             C   ←       λ   ⁢           ⁢   C     +       Λ   2     ⁢       (         v   ⋓     τ     -     〈       v   ⋓     τ     〉       )     ′               
The adjusted C and D are delayed for one unit of time or one numbering  8  before being stored in the artificial synapses  48 .
 
       FIG. 24  illustrates an UPU with an unsupervised learning mechanism  60  to learn the code covariance matrix D that uses learning masking factors M jj   #  and an unsupervised learning mechanism  58  to learn the code deviation accumulation vector C. M jj   #  are the diagonal entries of the learning masking matrices, M # (ψ), ψ=1, . . . , Ψ, which have a smaller J in (32) than the masking matrices M(ψ) in the same UPU do. 
     The UPU (with learning masking matrices M # (ψ), ψ=1, . . . , Ψ), generates an estimate v{p τ   # } of the label of the vector v τ  input to the UPU using its masking matrices M(ψ), ψ=1, . . . , Ψ, as before. However, when it comes to learning, the learning masking matrices M # (ψ), ψ=1, . . . , Ψ, are used instead to generate an estimate v{p τ   # } of the label of v t , and the general code covariance matrix D is adjusted by
 
 D (ψ)←λ D (ψ)+Λ( v{p   τ   #   }−&lt;v{p   τ   # }&gt;)( {hacek over (v)}   τ   −&lt;{hacek over (v)}   τ &gt;)′
 
for ψ=1, . . . , Ψ. The generation of v{p τ   # } is described in the Subsection on “Learning Masking Factors for Unsupervised Learning of Smaller Differences” and summarized in  62  in  FIG. 24 . The code deviation accumulation vector C is learned in accordance with the unsupervised accumulation rule
 
             C   ←       λ   ⁢           ⁢   C     +       Λ   2     ⁢       (         v   ⋓     τ     -     〈       v   ⋓     τ     〉       )     ′               
The adjusted C and D are delayed for one unit of time or one numbering  8  before being stored in the artificial synapses  48 .
 
       FIG. 25  illustrates layer l and layer l+2 of an example LOM with a feedback connection from layer l to layer l and another from layer l+2 to layer l. The former contains 1 unit-time delay device  8 , and the latter  5  unit-time delay devices  8 . The box under layer l of PUs does not represent a model of a biological entity. It shows only how both the feedforwarded and feedbacked spike trains are assembled into the vector input to layer l. Note that only the feedbacks from layer l and layer l+2 are shown here. There can be feedbacks from other layers. Note also that for notational simplicity, we feedback all outputs from layer l and layer l+2, but this is not necessary. 
       FIG. 26  illustrates an example LOM with three layers of PUs. There are three types of feedback connection: same-layer feedbacks, one-layer feedbacks, and two-layer feedbacks. The delay durations on the feedback connections are not specified in the delay boxes  76 . The PUs  74  can be UPUs or SPUs. The second delay box on a feedback connection represents an additional delay. 
       FIG. 27  illustrates the entire clusterer with three layers of UPUs  78  in an example LOM. 
       FIG. 28  illustrates an example LOM consisting of the clusterer from  FIG. 27  and an interpreter with three SPUs  80 . The feedback connections and delays on them in the clusterer in  FIG. 27  are not shown for clarify in  FIG. 28 . 
     5 DESCRIPTION OF PREFERRED EMBODIMENTS 
     In the terminology of artificial neural networks, machine learning, and pattern recognition, a feature vector is a transformation of a measurement vector, whose components are measurements or sensor outputs. As a special case, the transformation is the identity transformation, and the feature vector is the measurement vector. Example measurement vectors are digital photographs, frames of a video, segments of speech and handwritten characters/words/numbers. In the present invention disclosure, a feature vector is usually referred to as a vector, unless an emphasis that a vector is a transformation of a measurement vector is needed. 
     The present invention disclosure discloses a low-order model (LOM) of biological neural networks, the Clustering Interpreting Probabilistic Associative Memory (CIPAM), and other ANNs (artificial neural networks) that are equivalent to the LOM and the CIPAM in the sense that they can be obtained by transforming the LOM or the CIPAM with an affine function and its inverse. The LOM and its equivalent ANNs including the CIPAM are systems that receive and process feature vectors or sequences of feature vectors. Such feature vectors input to the LOM, the CIPAM or another ANN equivalent to them are called exogenous feature vectors. The LOM and ANNs (including the CIPAM) equivalent to the LOM can be viewed as a new ANN paradigm and a new type of learning machine. 
     The LOM or an ANN equivalent to it is a network of processing units (PUs) with or without feedback connections. If an LOM or an ANN equivalent to it (e.g., CIPAM) is a multilayer network, a vector input to a layer of the network is a feature vector, because it is a transformation of exogenous feature vectors input to the LOM or the ANN and is in turn a transformation of the measurement vectors. A feature vector input to layer l comprises a vector output from layer l−1 and vectors output and feedbacked from the same or other layers. For example, if there is a feedback connection to layer l, then an exogenous feature vector is not an entire feature vector input to layer l, but only a subvector of said entire feature vector. 
     The LOM, the CIPAM and their equivalent models are each a network of models of the biological neuronal node/encoder, synapse, spiking/nonspiking neuron, means for learning, feedback connection, maximal generalization scheme, feedback connection, etc. For simplicity, these component models are often referred to without the word “model”. For example, the model neuronal node, model neuronal encoder, model neuronal tree, model spiking/nonspiking neuron, model synapse, etc. will be referred to as the neuronal node, neuronal encoder, neuronal tree, spiking/nonspiking neuron, synapse, etc. respectively. The LOM, the CIPAM and all their equivalent models can be used as artificial neural networks (ANNs). To emphasize that their components are artificial components in these artificial neural networks, they are referred to as the artificial neuronal node, artificial neuronal encoder, artificial neuronal tree, artificial spiking/nonspiking neuron, artificial synapse, etc. respectively. 
     We will first describe the LOM and then show how the LOM is transformed into the CIPAM and the general ANN equivalent to the LOM by the use of the affine functions. 
     In the present invention disclosure, the prime ′ denotes matrix transposition, a vector is regarded as a subvector of the vector itself, a column or row vector is also a matrix, and [a ij ] denotes a matrix whose (i, j)th entry is a ij . If a ij , i=1, . . . , m, j=1, . . . , n, then they are contained in the m×n matrix [a ij ]. Evaluating a ij , i=1, . . . , m, j=1, . . . , n, is equivalent to evaluating the m×n matrix [a ij ]. 
     5.1 Model Neuronal Encoders 
     It was discovered in 1980&#39;s and 1990&#39;s that biological dendrites are capable of performing information processing tasks. However, neuronal trees are missing in well-known artificial neural networks, overlooking a large percentage of the neural circuit (Simon Haykin,  Neural Networks and Learning Machines , Third Edition, Pearson Education, New Jersey, 2009 and Christopher M. Bishop,  Pattern Recognition and Machine Learning , Springer Science, New York, 2006). 
     In the low-order model (LOM) disclosed in the present invention disclosure, a model neuronal (dendritic/axonal) encoder, is a network of model neuronal nodes, each of which is a low-order polynomial with two variables that acts like an XOR (exclusive-OR) logic gate when the two input variables are (bipolar) binary digits. 
     In this Subsection, model neuronal nodes and model neuronal encoders are described. An model neuronal encoder is a function, that encodes the vector input to the encoder into the vector output, called the neuronal code, from the encoder. The output vectors have an orthogonality property proven in the Appendix of James Ting-Ho Lo, A Low-Order Model of Biological Neural Networks,  Neural Computation , Vol. 23, No. 10, pp. 2626-2682, 2011. 
     5.1.1 Binary Operation of a Model Neuronal Node 
     A model (or artificial) neuronal node is shown in  FIG. 1 . The input variables, v and u, to a neuronal node are usually spikes in spike trains modeled as Bernoulli processes. The output of the model neuronal node is
 
φ( v,u )=−2 vu+v+u   (1)
 
which is a hyperbolic polynomial depicted over the unit square in  FIG. 2 . The unit square is only a part of the domain of the function φ. If v and u are binary digits, φ(v, u) is the XOR function. If not, φ(v, u) approximates the XOR function nicely. As shown in  FIG. 2 , the closer v and u are to binary values, the more φ acts like XOR. The farther they are to binary values, the less φ acts like XOR. For example, φ(0.9, 0.9)=0.18, φ(0.9, 0.1)=0.82, φ(0.9, 0.75)=0.3, and φ(0.75, 0.1)=0.7. φ acts more like XOR at (0.9, 0.9) and (0.9, 0.1) than at (0.9, 0.75) and (0.75, 0.1). Note that there are other polynomials (e.g., the elliptic polynomial 2v 2 +2u 2 −2vu−v−u) that act like the XOR function at binary inputs, but φ involves the least number of arithmetic operations and approximates XOR in the most reasonable manner as shown in  FIG. 2 . φ(v, u) is henceforth called the XOR polynomial.
 
5.1.2 Composition of Operations of Model Neuronal Nodes
 
     The algebraic binary operation φ(v, u)=−2vu+v+u is commutative and also associative: 
                     ϕ   ⁡     (     w   ,     ϕ   ⁡     (     v   ,   u     )         )       =       ⁢     ϕ   ⁡     (       ϕ   ⁡     (     w   ,   v     )       ,   u     )                   =       ⁢           (     -   2     )     2     ⁢   wvu     -     2   ⁢     (     wv   +   wu   +   vu     )       +   w   +   v   +   u                 
Hence, we can define a symmetric function φ k  by applying the binary operation repeatedly as follows:
 
φ k ( v   1   ,v   2   , . . . ,v   k )=φ( . . . φ(φ( v   1   ,v   2 ), v   3 ), . . . , v   k )
 
where φ 1 (v i )=v i  and φ 2 (v i , v j )=φ(v i , v j ).
 
     It follows that
 
φ k ( v   1   ,v   2   , . . . ,v   k )=φ(φ i ( v   1   ,v   2   , . . . v   i ),φ k−i ( v   i+1   ,v   i+2   , . . . ,v   k ))
 
Therefore, there are many different ways to obtain φ k (v 1 , v 2 , . . . , v k ). For example, φ 4 (v 1 , v 2 , v 3 , v 4 ) can be obtained by φ(φ(v 1 , v 2 ), φ(v 3 , v 4 )) or by φ(φ 3  (v 3 , v 2 , v 4 ), v 1 ).
 
     If the model neuronal encoder has m inputs forming an input set S={v 1 , v 2 , . . . , v m }, then the input set has 2 m  subsets. On each of these subsets, say {v k     1   , v k     2   , v k     i   }, an output of the neuronal encoder is defined to be φ i (v k     1   , v k     2   , . . . , v k     i   ). For example, if the input set is {v 1 , v 2 , v 3 }, then the subsets are Φ, {v 1 }, {v 2 }, {v 2 , v 1 }, {v 3 }, {v 3 , v 1 }, {v 3 , v 2 }, {v 3 , v 2 , v 1 }, where Φ is the empty set. The outputs of the neuronal encoder are φ 0  (Ψ), φ 1 (v 1 ), φ 1 (v 2 ), φ 2 (v 2 , v 1 ), φ 1 (v 3 ), φ 2 (v 3 , v 1 ), φ 2 (v 3 , v 2 ), φ 3 (v 3 , v 2 , v 1 ), where φ 0 (Φ) is defined to be 0. 
     Similarly, if the input set is {v 1 , v 2 , v 3 , v 4 }, the model neuronal encoder has 16 outputs φ i (v k     1   , . . . , v k     i   ), where {v k     1   , . . . , v k     i   } are subsets of {v 1 , v 2 , v 3 , v 4 }.  FIG. 3  shows this model neuronal encoder with four inputs and 16 outputs, where the four inputs v ti , i=1, 2, 3, 4, at time t are each close to a binary digit. φ i (v k     1   , . . . , v k     i   ) can be evaluated by binary φ 2  or other operations φ k  in more than one way if i&gt;2. Therefore, the structure of a model neuronal encoder for more than 2 inputs is not unique. 
     Note that φ(v i , v i )=0 and φ(0, v i )=v i . It follows that for v k     1   , v k     2   , . . . , v k     j    and v k     1   , 
                       ϕ     j   +   1       ⁡     (       v     k   1       ,     v     k   1       ,     v     k   2       ,   …   ⁢           ,     v     k   j         )       =       ⁢     ϕ   ⁡     (         ϕ   2     ⁡     (       v     k   1       ,     v     k   1         )       ,       ϕ     j   -   1       ⁡     (       v     k   2       ,   …   ⁢           ,     v     k   j         )         )                   =       ⁢     ϕ   ⁡     (     0   ,       ϕ     j   -   1       ⁡     (       v     k   2       ,   …   ⁢           ,     v     k   j         )         )                   =       ⁢       ϕ     j   -   1       ⁡     (       v     k   2       ,   …   ⁢           ,     v     k   j         )                   
Hence, a function φ j  with repeated variables can be identified with a function φ j−2i  with different variables for some i&gt;0. Using {v 1 , v 2 , . . . , v m } as the input set and φ(v, u) to compose functions, we can obtain only 2 m  different functions for input variables with binary values.
 
     Note that model axonal encoders, model dendritic encoders, and model dendritic encoders with a dendrite or dendritic node replaced with an axon or axonal node respectively are collectively called neuronal encoders. An orthogonality property of a model neuronal encoder&#39;s output vector is discussed in the next Subsubsection and proven in the Appendix. 
     5.1.3 An Orthogonality Property of Neuronal Codes 
     To describe an orthogonality property of the outputs of a model neuronal encoder with input variables {v 1 , v 2 , . . . , v m }, we organize its 2 m  outputs into a vector, called the neuronal code, as follows: Let u denote a scalar and v=[v 1  v 2  . . . v k ]′ a k-dimensional vector. Define a k-dimensional vector φ(u, v) of polynomials by
 
φ( u,v )=[φ( u,v   1 ) φ( u,v   2 ) . . . φ( u,v   k )]
 
The 2 m  different functions that can be defined by compositions of the binary operation φ on the input set {v 1 , v 2 , . . . , v m } are generated and organized into a 2 m -dimensional column vector {hacek over (v)} by recursively generating row vectors {hacek over (v)}(1, . . . , k), for k=1, 2, . . . , m, as follows:
 
                       v   ⋓     ⁡     (   1   )       =     [         0         v   1           ]             (   2   )                         v   ⋓     ⁡     (     1   ,   2     )       =       ⁢     [             v   ⋓     ⁡     (   1   )           ϕ         (       v   2     ,       v   ⋓     ⁡     (   1   )         )           ]                 =       ⁢     [         0         v   1           v   2               -   2     ⁢           ⁢     v   2     ⁢     v   1       +     v   2     +     v   1             ]                   (   3   )                   v   ⋓     ⁡     (     1   ,   …   ⁢           ,     k   +   1       )       =     [               v   ⋓     ⁡     (     1   ,   …   ⁢           ,   k     )           ϕ         ⁢     (       v     k   +   1       ,       v   ⋓     ⁡     (     1   ,   …   ⁢           ,   k     )         )       ]             (   4   )                 v   ⋓     =         v   ⋓     ′     ⁡     (     1   ,   …   ⁢           ,   m     )               (   5   )               
Denoting the k-th component of {hacek over (v)} by {hacek over (v)} k , the vector {hacek over (v)}=[{hacek over (v)} 1  {hacek over (v)} 2  . . . {hacek over (v)} 2     m   ]′ is called the neuronal code of v. Setting the first component of {hacek over (v)}(1) equal to zero above (instead of one) yields two properties: First, because φ(v, v)=0 and φ(v, 0)=v, two equal binary signals meeting at a neuronal node produce the first component of {hacek over (v)}, and this first component will not change other components through neuronal nodes through neuronal nodes down the stream. Second, this makes {hacek over (0)}=0. Here, 0 are the zero vectors. The above recursive way to generate the neuronal codes is also shown in  FIG. 4 .
 
     It is proven in the APPENDIX of James Ting-Ho Lo, “A Low-Order Model of Biological Neural Networks,”  Neural Computation , vol. 23, no. 10, pp. 2626-2682, 2011, that given two m-dimensional binary vectors, v and u, their neuronal codes, {hacek over (v)} and {hacek over (u)}, satisfy 
                         (       v   ⋓     -       1   2     ⁢   I       )     ′     ⁢     (       u   ⋓     -       1   2     ⁢   I       )       =       2     -   2       ⁢       ∏     k   =   1     m     ⁢           ⁢     (     1   +       2   2     ⁢     (       v   k     -     1   2       )     ⁢     (       u   k     -     1   2       )         )                 (   6   )               
where I=[1 1 . . . 1]′, which we note is not the identity matrix I. If v and u are binary digits, then
 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           ( 
                           
                             
                               v 
                               ⋓ 
                             
                             - 
                             
                               
                                 1 
                                 2 
                               
                               ⁢ 
                               I 
                             
                           
                           ) 
                         
                         ′ 
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             u 
                             ⋓ 
                           
                           - 
                           
                             
                               1 
                               2 
                             
                             ⁢ 
                             I 
                           
                         
                         ) 
                       
                     
                     = 
                       
                     ⁢ 
                     0 
                   
                   , 
                   
                     
                       if 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       v 
                     
                     ≠ 
                     u 
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
             
               
                 
                   
                       
                   
                   ⁢ 
                   
                     
                       = 
                         
                       ⁢ 
                       
                         2 
                         
                           m 
                           - 
                           2 
                         
                       
                     
                     , 
                     
                       
                         if 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         v 
                       
                       = 
                       u 
                     
                   
                 
               
               
                 
                   ( 
                   8 
                   ) 
                 
               
             
           
         
       
     
     Note that if the components of u and v assume values from the set {0, ½, 1}, we have the following additional properties: 
     
       
         
           
             
               
                 
                   
                     
                       1. 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       If 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             v 
                             k 
                           
                           - 
                           
                             1 
                             2 
                           
                         
                         ) 
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             u 
                             k 
                           
                           - 
                           
                             1 
                             2 
                           
                         
                         ) 
                       
                     
                     = 
                     
                       0 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       for 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       some 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       k 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       in 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         { 
                         
                           1 
                           , 
                           … 
                           ⁢ 
                           
                               
                           
                           , 
                           m 
                         
                         } 
                       
                     
                   
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   
                     
                       
                         
                           then 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             
 
                           
                           ( 
                           
                             
                               v 
                               ⋓ 
                             
                             - 
                             
                               
                                 1 
                                 2 
                               
                               ⁢ 
                               I 
                             
                           
                           ) 
                         
                         ′ 
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             u 
                             ⋓ 
                           
                           - 
                           
                             
                               1 
                               2 
                             
                             ⁢ 
                             I 
                           
                         
                         ) 
                       
                     
                     = 
                     
                       
                         2 
                         
                           - 
                           2 
                         
                       
                       ⁢ 
                       
                         
                           ∏ 
                           
                             
                               j 
                               = 
                               1 
                             
                             , 
                             
                               j 
                               ≠ 
                               k 
                             
                           
                           m 
                         
                         ⁢ 
                         
                           ( 
                           
                             1 
                             + 
                             
                               
                                 2 
                                 2 
                               
                               ⁢ 
                               
                                 ( 
                                 
                                   
                                     v 
                                     j 
                                   
                                   - 
                                   
                                     1 
                                     2 
                                   
                                 
                                 ) 
                               
                               ⁢ 
                               
                                 ( 
                                 
                                   
                                     u 
                                     j 
                                   
                                   - 
                                   
                                     1 
                                     2 
                                   
                                 
                                 ) 
                               
                             
                           
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       
                         2. 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         If 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           
                             ( 
                             
                               
                                 v 
                                 ⋓ 
                               
                               - 
                               
                                 
                                   1 
                                   2 
                                 
                                 ⁢ 
                                 I 
                               
                             
                             ) 
                           
                           ′ 
                         
                         ⁢ 
                         
                           ( 
                           
                             
                               u 
                               ⋓ 
                             
                             - 
                             
                               
                                 1 
                                 2 
                               
                               ⁢ 
                               I 
                             
                           
                           ) 
                         
                       
                       ≠ 
                       
                         0 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         then 
                       
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       
 
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       
                         
                           
                             ( 
                             
                               
                                 v 
                                 ⋓ 
                               
                               - 
                               
                                 
                                   1 
                                   2 
                                 
                                 ⁢ 
                                 I 
                               
                             
                             ) 
                           
                           ′ 
                         
                         ⁢ 
                         
                           ( 
                           
                             
                               u 
                               ⋓ 
                             
                             - 
                             
                               
                                 1 
                                 2 
                               
                               ⁢ 
                               I 
                             
                           
                           ) 
                         
                       
                       ⁢ 
                       
                           
                       
                       = 
                       
                         2 
                         
                           
                             4 
                             ⁢ 
                             
                               
                                 ( 
                                 
                                   v 
                                   - 
                                   
                                     
                                       1 
                                       2 
                                     
                                     ⁢ 
                                     I 
                                   
                                 
                                 ) 
                               
                               ′ 
                             
                             ⁢ 
                             
                               ( 
                               
                                 u 
                                 - 
                                 
                                   
                                     1 
                                     2 
                                   
                                   ⁢ 
                                   I 
                                 
                               
                               ) 
                             
                           
                           - 
                           2 
                         
                       
                     
                   
                   ⁢ 
                   
                       
                   
                 
               
               
                 
                   ( 
                   10 
                   ) 
                 
               
             
           
         
       
     
    
    
     EXAMPLE 1a 
     If v=[1 0 1 0]′ and u=[1 0 1 1]′, then {hacek over (v)}(1)=[0 1], {hacek over (v)}(1,2)=[0 1 φ(0,0) φ(0,1)]=[0 1 0 1], {hacek over (v)}(1,2,3)=[0 1 1 1 0 1 0], and
 
 {hacek over (v)}=[ 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 0]′
 
 {hacek over (u)}=[ 0 1 0 1 1 0 1 0 1 0 1 0 0 1 0 1]′
 
It follows that {hacek over (v)}−½I and {hacek over (u)}−½I are respectively
 
                     [           -     1   2             1   2           -     1   2             1   2           1   2           -     1   2             1   2           -     1   2                 -     1   2             1   2           -     1   2             1   2           1   2           -     1   2             1   2           -     1   2             ]     ′                             [           -     1   2             1   2           -     1   2             1   2           1   2           -     1   2             1   2           -     1   2                 1   2           -     1   2             1   2           -     1   2             -     1   2             1   2           -     1   2             1   2           ]     ′                           
Hence, ({hacek over (v)}−½I)′({hacek over (v)}−½I)=2 4−2  and ({hacek over (v)}−½I)′({hacek over (u)}−½I)=0, as predicted by (8) and (7).
 
5.2 Hebbian-Type Learning Rules
 
     Three learning rules, called unsupervised covariance rule, supervised covariance rule and unsupervised accumulation rule, are described in this Subsection. The first two are essentially Terrence J. Sejnowski&#39;s covariance rule. However, the unsupervised covariance rule and supervised covariance rule herein proposed do not build up the covariance between the outputs of the presynaptic and postsynaptic neurons as Sejnowski&#39;s covariance rule does. The unsupervised covariance rule builds up, in synapses, the covariance between the outputs of the presynaptic neuronal encoder and the postsynaptic neurons. The supervised covariance rule builds up the covariance between the outputs of the presynaptic neuronal encoder and the labels provided from outside the LOM that act as teaching signals for the supervised covariance learning. Like Sejnowski&#39;s covariance rule, the unsupervised and supervised covariance rules here, especially the former, can be looked upon as variants of what is commonly known as the Hebb rule. The unsupervised accumulation rule simply accumulates code deviation vectors, which are the deviations of the neuronal codes from their averages over a time window with a certain width. 
     5.2.1 Unsupervised Covariance Rule 
     There are two types of model neuron, namely model spiking neuron (also called D-neuron) and model nonspiking neuron (also called C-neuron), in the LOM. A model spiking neuron generates binary digits, and a model nonspiking neuron outputs graded signals that are transmitted to its neighboring model spiking neurons. Computations performed in these two types of neuron are described in Subsubsections 5.5.4 and 5.5.5. 
     Each of the 2 m  outputs, {hacek over (v)} t1 , {hacek over (v)} t2 , . . . , {hacek over (v)} t2     m   , from a neuronal encoder at time (or numbering) t passes through a synapse to reach each of a number of, say R, postsynaptic model spiking neurons and a postsynaptic model nonspiking neuron. {hacek over (v)} t =[{hacek over (v)} t1  . . . {hacek over (v)} t2     m   ]′ is called a neuronal code, and &lt;{hacek over (v)} t &gt;=[&lt;{hacek over (v)} t1 &gt; . . . &lt;{hacek over (v)} t2     m   &gt;]′ denotes an average of {hacek over (v)} t  over a time window [t−q v +1, t] with width q v , which is preset in consideration of the maximum size of the clusters formed using the unsupervised covariance rule (to be described) for the application. {hacek over (v)} t −&lt;{hacek over (v)} t &gt; is called a code deviation vector. It is the deviation of a neuronal code {hacek over (v)} t  from a neuronal code average &lt;{hacek over (v)} t &gt;.  FIG. 5   a  and  FIG. 5   b  show an output of the neuronal encoder going through a model synapse represented by {circle around (x)} to reach a model spiking neuron, model spiking neuron i, whose output at time t is u ti . The model spiking neuron usually receive signals from the synapses connected to other model neuronal encoders. 
       FIG. 5   a  and  FIG. 5   b  also show how the unsupervised covariance rule updates the weight (or strength) D ij  of the model synapse:
   D   ij   ←λD   ij +Λ( u   ti   −&lt;u   ti &gt;)( {hacek over (v)}   tj   −&lt;{hacek over (v)}   tj &gt;)  (11)
 
where Λ is a proportional constant, λ is a forgetting factor that is a positive number less than one, and &lt;{hacek over (v)} tj &gt; and &lt;u ti &gt; denote, respectively, the averages of &lt;{hacek over (v)} tj &gt; and &lt;u ti &gt; over time windows with preset widths. These widths may be different.
 
     The outputs u ti , i=1, . . . , R, of the R model spiking neurons can be assembled into a vector, u t =[u t1  u t2  . . . u tR ]′, and the strengths DR into a R×2 m  matrix D whose i×j-th entry is D ij . This matrix D is called a code covariance matrix. Using these notations, the unsupervised covariance rule can be expressed as follows:
 
 D←λD +Λ( u   t   −&lt;u   t &gt;)( {hacek over (v)}   t   −&lt;{hacek over (v)}   t &gt;)′  (12)
 
If the vector pairs, (v s , u s ), s=1, . . . , t, have been learned by the 2 m  R synapses, their code covariance matrix D is
 
     
       
         
           
             
               
                 
                   D 
                   = 
                   
                     Λ 
                     ⁢ 
                     
                       
                         ∑ 
                         
                           s 
                           = 
                           1 
                         
                         t 
                       
                       ⁢ 
                       
                         
                           
                             λ 
                             
                               t 
                               - 
                               s 
                             
                           
                           ⁡ 
                           
                             ( 
                             
                               
                                 u 
                                 s 
                               
                               - 
                               
                                 〈 
                                 
                                   u 
                                   s 
                                 
                                 〉 
                               
                             
                             ) 
                           
                         
                         ⁢ 
                         
                           
                             ( 
                             
                               
                                 
                                   v 
                                   ⋓ 
                                 
                                 s 
                               
                               - 
                               
                                 〈 
                                 
                                   
                                     v 
                                     ⋓ 
                                   
                                   s 
                                 
                                 〉 
                               
                             
                             ) 
                           
                           ′ 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   13 
                   ) 
                 
               
             
           
         
       
     
     This learning rule (11) makes the LOM more fault-tolerant and efficient than THPAM:
         1. If the model spiking neuron outputting u ti  or the neuronal node outputting {hacek over (v)} tj  is out of order causing u ti  or {hacek over (v)} tj =0 or 1 or any constant for too long, then u ti −&lt;u ti &gt;=0 or v ti −&lt;v ti &gt;=0, whence D ij ←λD ij  and D ij  shrinks to zero, eliminating the effect of the faulty model spiking neuron or neuronal node.
           2. If {hacek over (v)} tj  assumes the value, 1 (or 0), significantly more often than 0 (or 1), then &lt;{hacek over (v)} tj &gt; is closer to 1 (or 0), {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; is smaller for {hacek over (v)} tj =1 (or 0) than for {hacek over (v)} tj =0 (or 1), and D learns {hacek over (v)} tj  with less intensity. The same happens if u ti  assumes 1 (or 0) significantly more often than 0 (or 1). This automatically balances out the number of additions (to store 1&#39;s) to and subtractions (to store 0&#39;s) from D to avoid memory saturation at a synapse.   
           3. If {hacek over (v)} tj  is replaced with an inhibitory {hacek over (v)} tj  taking on −1 or 0, then {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt;=−(−{hacek over (v)} tj −&lt;−{hacek over (v)} tj &gt;), where −{hacek over (v)} tj  is excitatory, and the orthogonality property among {hacek over (v)} t  remains valid. This will be discussed more in Subsection 5.3.       

     These advantages of the unsupervised covariance rule are valid also for the supervised covariance rule and the unsupervised accumulation rule to be described below. In the LOM used as an artificial neural network, the formulas for the unsupervised covariance rule, (11), (12) and (13) can be replaced with 
                     D   ij     ←       λ   ⁢           ⁢     D   ij       +       Λ   ⁡     (       u   ti     -     1   2       )       ⁢     (         v   ⋓     tj     -     〈       v   ⋓     tj     〉       )                 (   14   )               D   ←       λ   ⁢           ⁢   D     +       Λ   ⁡     (       u   t     -       1   2     ⁢   I       )       ⁢       (         v   ⋓     t     -     〈       v   ⋓     t     〉       )     ′                 (   15   )               D   =     Λ   ⁢       ∑     s   =   1     t     ⁢         λ     t   -   s       ⁡     (       u   s     -       1   2     ⁢   I       )       ⁢       (         v   ⋓     s     -     〈       v   ⋓     s     〉       )     ′                   (   16   )               
for less computation and more accurate estimation of the label u t  once it is generated and learned by the PU, where I=1 1 . . . 1 with R components.
 
     How a vocabulary is built by the unsupervised covariance rule is discussed at the end of Subsubsection 5.5.7. 
     5.2.2 Supervised Covariance Rule 
     The set of sensor elements (e.g., the pixels in the CCD of a camera) whose measurements affect the inputs to a model spiking neuron is called the receptive field of the model spiking neuron. The vectors v t  that are input to all the neuronal encoders that affect such inputs to a model spiking neuron have the same receptive field. 
     If a label w t  of the cause (e.g., object or pattern) that appears in the receptive field of a spiking or nonspiking neuron is provided from outside the LOM, a supervised learning can be performed. For example, if the image or sub-image appearing in the receptive field of the neuron can be identified by a human, s/he can handcraft a label w t  of the image or subimage. 
       FIG. 6   a  and  FIG. 6   b  show how the component {hacek over (v)} tj  and the component w ti  of the label w t  of model spiking neuron i are used to update the weight D ij  in a model synapse {circle around (x)} of postsynaptic model spiking neuron i for j=1, . . . , 2 m . 
     The supervised covariance rule that updates the strength D ij  using {hacek over (v)} tj  and w ti  is the following:
 
 D   ij   ←λD   ij +Λ( w   ti −½)( {hacek over (v)}   tj   −&lt;{hacek over (v)}   tj &gt;)  (17)
 
for j=1, . . . , 2 m  and i=1, . . . , R, where Λ and λ are a proportion constant and a forgetting factor, and &lt;{hacek over (v)} tj &gt;denotes the average of {hacek over (v)} tj  over a time window.
 
     The synaptic weights (or strengths) D ij  form an R×2 m  matrix D whose (i, j)th entry is D ij . This matrix D is again called a code covariance matrix. Using these notations, the supervised covariance rule can be expressed as follows:
 
 D←λD +Λ( w   t −½ I )( {hacek over (v)}   t   −&lt;{hacek over (v)}   t &gt;)′  (18)
 
If the pairs, (v s , w s ), s=1, . . . , t, have been learned by the R(2 m ) synapses, their code covariance matrix D is
 
     
       
         
           
             
               
                 
                   D 
                   = 
                   
                     Λ 
                     ⁢ 
                     
                       
                         ∑ 
                         
                           s 
                           = 
                           1 
                         
                         t 
                       
                       ⁢ 
                       
                         
                           
                             λ 
                             
                               t 
                               - 
                               s 
                             
                           
                           ⁡ 
                           
                             ( 
                             
                               
                                 w 
                                 s 
                               
                               - 
                               
                                 
                                   1 
                                   2 
                                 
                                 ⁢ 
                                 I 
                               
                             
                             ) 
                           
                         
                         ⁢ 
                         
                           
                             ( 
                             
                               
                                 
                                   v 
                                   ⋓ 
                                 
                                 s 
                               
                               - 
                               
                                 〈 
                                 
                                   
                                     v 
                                     ⋓ 
                                   
                                   s 
                                 
                                 〉 
                               
                             
                             ) 
                           
                           ′ 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   19 
                   ) 
                 
               
             
           
         
       
     
     Note that the update formulas, (17) and (18), and the equation (19) are the same as the update formulas, (14) and (15), and the equation (16), respectively; except that u t  is respectively replaced with the label w t  of v t  provided from outside the LOM. In some applications, the term ½I in (17) and (18), and (19) can be replaced with &lt;w ti &gt;, &lt;w t &gt; and &lt;w s &gt;, respectively. In this case, the length of time interval on which the averages, &lt;w ti &gt;, &lt;w t &gt; and &lt;w s &gt;, are taken may be different from that for &lt;{hacek over (v)} tj &gt;, &lt;{hacek over (v)} t &gt; and &lt;{hacek over (v)} s &gt;. 
     EXAMPLE 2a 
     Consider a unit cube shown in  FIG. 9 . The vectors, v t , t=1, 2, . . . , 8, to be input to neuronal encoders in EXAMPLES 2a, 2b and 2c, are shown at the vertices. The signals from a teaching model spiking neuron corresponding to v t , t=1, 2, 3, 7, 8, are available for supervised learning. They are binary digits w t , t=1, 2, 3, 7, 8, respectively, enclosed in the square boxes. The supervised training data consists of the pairs, (v t , w t ), t=1, 2, 3, 7, 8. The question marks in the square boxes indicate no teaching signal is available for supervised learning. 
     The pairs, ({hacek over (v)} t ′, w t ), t=1, 2, 3, 7, 8, are listed as rows in the following table: 
     
       
         
           
               
               
               
               
               
               
               
               
               
               
             
               
                   
               
               
                   
                   
                   
                   
                 φ 
                   
                 φ 
                 φ 
                 φ 3   
                   
               
               
                 {hacek over (v)}′ t   
                 0 
                 v t1   
                 v t2   
                 (v t2 , v t1 ) 
                 v t3   
                 (v t3 , v t1 ) 
                 (v t3 , v t2 ) 
                 (v t3 , v t2 , v t1 ) 
                 w t   
               
               
                   
               
             
            
               
                 {hacek over (v)}′ 1   
                 0 
                 0 
                 0 
                 0 
                 0 
                 0 
                 0 
                 0 
                 0 
               
               
                 {hacek over (v)}′ 2   
                 0 
                 1 
                 0 
                 1 
                 0 
                 1 
                 0 
                 1 
                 1 
               
               
                 {hacek over (v)}′ 3   
                 0 
                 0 
                 1 
                 1 
                 0 
                 0 
                 1 
                 1 
                 1 
               
               
                 {hacek over (v)}′ 7   
                 0 
                 0 
                 1 
                 1 
                 1 
                 1 
                 0 
                 0 
                 1 
               
               
                 {hacek over (v)}′ 8   
                 0 
                 1 
                 1 
                 0 
                 1 
                 0 
                 0 
                 1 
                 1 
               
               
                   
               
            
           
         
       
     
     Assume λ=Λ=1, &lt;{hacek over (v)} t &gt;=I/2, and &lt;w t &gt;=I/2 in (18). The code covariance matrix D is the following: 
                   D   =     [           -     3   4             1   4           3   4           3   4           1   4           1   4           -     1   4             3   4           ]             (   20   )               
5.2.3 Unsupervised Accumulation Rule
 
     The 2 m  synapses on the connections from the output terminals of a neuronal encoder to a model nonspiking neuron are updated by the unsupervised accumulation rule: 
                   C   ←       λ   ⁢           ⁢   C     +       Λ   2     ⁢       (         v   ⋓     t     -     〈       v   ⋓     t     〉       )     ′                 (   21   )                 FIGS. 7   a  and  7   b  show a model nonspiking neuron and one of its synapses {circle around (x)} and how the component {hacek over (v)} tj −&lt;{hacek over (v)} tj &gt; of the code deviation {hacek over (v)} t −&lt;{hacek over (v)} t &gt; is used to adjust C j . Note that the graded output c t  from the model nonspiking neuron is not feedbacked for updating the synaptic strength C j . If the deviations {hacek over (v)} s −&lt;v s &gt;, s=1, . . . , t, have been accumulated in the 2 m  synapses, the strengths (or weights) in them are the row vector,
 
                   C   =       Λ   2     ⁢       ∑     s   =   1     t     ⁢         λ     t   -   s       ⁡     (         v   ⋓     s     -     〈       v   ⋓     s     〉       )       ′                 (   22   )               
This row vector C is called a code deviation accumulation vector.
 
     EXAMPLE 2b 
     This is a continuation of EXAMPLE 2a. For the training data, x t , t=1, 2, 3, 7, 8, from EXAMPLE 2a, which is shown in  FIG. 9 , the code deviation accumulation vector C is the following: 
                   C   =     [           -     5   4             -     1   4             1   4           1   4           -     1   4             -     1   4             -     3   4             1   4           ]             (   23   )               
Note that the teaching signals, w t , t=1, 2, 3, 7, 8, are not needed in obtaining C by the unsupervised accumulation rule (21).
 
     The inputs to the neuronal encoder are v τ1 , v τ2 , v τ3 . The masking matrices M in the figure will be used in EXAMPLE 2c in Section 5.4. The outputs of the neuronal encoder are 0, v τ1 , v τ2 , φ(v τ2 , v τ1 ), v 3 , φ(v τ3 , v τ1 ), φ(v τ3 , v τ2 ), φ 3 (v τ3 , v τ2 , v τ1 ). For the synapses preceding the model spiking neuron to perform supervised covariance learning, the selection lever represented by the thick line segment with a circle at its end is placed in the top position to receive a teaching signal r τ  provided from outside. For these synapses to perform unsupervised covariance learning, the lever is placed in the bottom position to receive a spike or nonspike v{p τ }output from the model spiking neuron. For these synapses to perform no learning, the lever is placed in the middle position to receive the signal ½. The selection lever of a biological synapse is usually permanently set at one position for one type of learning. It is not clear if a biological synapse can perform supervised, unsupervised, and no learning alternately. 
     In  FIG. 11 , the model nonspiking neuron and model spiking neuron share the same neuronal encoder and its outputs. This is not necessary in modeling biological neural networks. As long as the model nonspiking neuron and model spiking neuron jointly provide enough information to produce a good estimate of the empirical probability distribution, they may have different neuronal encoders. 
     5.3 Empirical Probabilities Learned 
     The purpose of this Subsection is to show that an artificial neuronal encoder and the Hebbian-type learning rules disclosed in the last two Subsections working together are able to learn empirical probability distributions of the labels of the vectors input to the artificial neuronal encoder. 
     Once a vector v τ  is received by a neuronal encoder, v τ  is encoded by the neuronal encoder into {hacek over (v)} τ , which is made available to synapses for learning as well as retrieving of a point estimate of the label (u t  or w t ) of the input v τ , whether the label has been learned by the unsupervised covariance rule or the supervised covariance rule. This label of v τ , which may be u t  or w t  as defined in the preceding Subsection, is denoted by r τ . Recall that learned information is stored in the code covariance matrices, D and C. Upon the arrival of {hacek over (v)} τ , the following products, d τ   o  and c τ   o , are computed by the synapses preceding the R model spiking neurons and 1 model nonspiking neuron as shown in c o  8:
 
 d   τ   o   =D ( {hacek over (v)}   τ   −&lt;{hacek over (v)}   τ &gt;)  (24)
 
 c   τ   o   =C ( {hacek over (v)}   τ   −&lt;{hacek over (v)}   τ &gt;)  (25)
 
where d τ   o  is an R-dimensional vector and c τ   o  is a scalar. Recall that {hacek over (v)} t =[{hacek over (v)} t1  . . . {hacek over (v)} t2     m   ]′ is called a neuronal code, and &lt;{hacek over (v)} t &gt;=[&lt;{hacek over (v)} t1 &gt; . . . &lt;{hacek over (v)} t2     m   &gt;]′ denotes an average of {hacek over (v)} t  over a time window [t−q v +1, t] with width q v , which is preset in consideration of the size the maximum size of the clusters formed using the unsupervised covariance rule to be described for the application. {hacek over (v)} t −&lt;{hacek over (v)} t &gt; is called a code deviation vector. It is the deviation of a neuronal code v t  from a neuronal code average &lt;{hacek over (v)} t &gt;.
 
     To gain some intuitive understanding of the meanings of d τ   o  and c τ   o , let us assume that λ=Λ=1, and that the averages, &lt;w t &gt; and &lt;{hacek over (v)} τ &gt;, are all equal to I/2. Here, I=[1 1 . . . 1]′, which we note is not the identity matrix I. Let r s  be an R-dimensional binary vector. Noting that the time average of a biological spike train is usually close to ½ and that the forgetting factor λ is believed to be very close to 1. The only problem with the assumption is that {hacek over (v)} s1 =0 and hence &lt;{hacek over (v)} s1 &gt;=0≠½. If the dimensionality of the vector {hacek over (v)} s  is large, the effect of missing this one term of ½ is negligible. Nevertheless, it would be interesting to see whether and how this one term exists in biological neural networks. 
     Under the above assumptions, 
     
       
         
           
             
               
                 
                   
                     d 
                     τ 
                     o 
                   
                   = 
                     
                   ⁢ 
                   
                     Λ 
                     ⁢ 
                     
                       
                         ∑ 
                         
                           s 
                           = 
                           1 
                         
                         t 
                       
                       ⁢ 
                       
                         
                           ( 
                           
                             
                               r 
                               s 
                             
                             - 
                             
                               〈 
                               
                                 r 
                                 s 
                               
                               〉 
                             
                           
                           ) 
                         
                         ⁢ 
                         
                           
                             ( 
                             
                               
                                 
                                   v 
                                   ⋓ 
                                 
                                 s 
                               
                               - 
                               
                                 〈 
                                 
                                   
                                     v 
                                     ⋓ 
                                   
                                   s 
                                 
                                 〉 
                               
                             
                             ) 
                           
                           ′ 
                         
                         ⁢ 
                         
                           ( 
                           
                             
                               
                                 v 
                                 ⋓ 
                               
                               τ 
                             
                             - 
                             
                               〈 
                               
                                 
                                   v 
                                   ⋓ 
                                 
                                 τ 
                               
                               〉 
                             
                           
                           ) 
                         
                       
                     
                   
                 
               
             
             
               
                 
                   = 
                     
                   ⁢ 
                   
                     Λ 
                     ⁢ 
                     
                       
                         ∑ 
                         
                           s 
                           = 
                           1 
                         
                         t 
                       
                       ⁢ 
                       
                         
                           ( 
                           
                             
                               r 
                               s 
                             
                             - 
                             
                               
                                 1 
                                 2 
                               
                               ⁢ 
                               I 
                             
                           
                           ) 
                         
                         ⁢ 
                         
                           
                             ( 
                             
                               
                                 
                                   v 
                                   ⋓ 
                                 
                                 s 
                               
                               - 
                               
                                 
                                   1 
                                   2 
                                 
                                 ⁢ 
                                 I 
                               
                             
                             ) 
                           
                           ′ 
                         
                         ⁢ 
                         
                           ( 
                           
                             
                               
                                 v 
                                 ⋓ 
                               
                               τ 
                             
                             - 
                             
                               
                                 1 
                                 2 
                               
                               ⁢ 
                               I 
                             
                           
                           ) 
                         
                       
                     
                   
                 
               
             
           
         
       
       
         
           and 
         
       
       
         
           
             
               
                 
                   
                     c 
                     τ 
                     o 
                   
                   = 
                     
                   ⁢ 
                   
                     
                       Λ 
                       2 
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           s 
                           = 
                           1 
                         
                         t 
                       
                       ⁢ 
                       
                         
                           
                             ( 
                             
                               
                                 
                                   v 
                                   ⋓ 
                                 
                                 s 
                               
                               - 
                               
                                 〈 
                                 
                                   
                                     v 
                                     ⋓ 
                                   
                                   s 
                                 
                                 〉 
                               
                             
                             ) 
                           
                           ′ 
                         
                         ⁢ 
                         
                           ( 
                           
                             
                               
                                 v 
                                 ⋓ 
                               
                               τ 
                             
                             - 
                             
                               〈 
                               
                                 
                                   v 
                                   ⋓ 
                                 
                                 τ 
                               
                               〉 
                             
                           
                           ) 
                         
                       
                     
                   
                 
               
             
             
               
                 
                   = 
                     
                   ⁢ 
                   
                     
                       Λ 
                       2 
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           s 
                           = 
                           1 
                         
                         t 
                       
                       ⁢ 
                       
                         
                           
                             ( 
                             
                               
                                 
                                   v 
                                   ⋓ 
                                 
                                 s 
                               
                               - 
                               
                                 
                                   1 
                                   2 
                                 
                                 ⁢ 
                                 I 
                               
                             
                             ) 
                           
                           ′ 
                         
                         ⁢ 
                         
                           ( 
                           
                             
                               
                                 v 
                                 ⋓ 
                               
                               τ 
                             
                             - 
                             
                               
                                 1 
                                 2 
                               
                               ⁢ 
                               I 
                             
                           
                           ) 
                         
                       
                     
                   
                 
               
             
           
         
       
     
     Under the further assumption that v s  and v τ  are binary vectors, by (7) and (8), which are proven in the APPENDIX, if v s ≠v τ , then
 
( {hacek over (v)}   s −½ I )′( {hacek over (v)}   τ −½ I )=0  (26)
 
and  if v   s   =v   τ , then
 
( {hacek over (v)}   s −½ I )′( {hacek over (v)}   τ −½ I )=2 m−2   (27)
 
Therefore, under the above assumptions and the assumption that v s  and v τ  are binary vectors, for j=1, . . . , R,
 
                     d     τ   ⁢           ⁢   j     o     =       ⁢       Λ   (       ∑       s   =   1           v   s     =     v   τ       ,       r   sj     =   1         t     ⁢     +     ∑       s   =   1           v   s     =     v   τ       ,       r   sj     =   0         t         )     ⁢     (       r   sj     -     1   2       )     ⁢     2     m   -   2                     =       ⁢       Λ   ⁢       ∑       s   =   1           v   s     =     v   τ       ,       r   sj     =   1         t     ⁢       (     1   2     )     ⁢     2     m   -   2             +     Λ   ⁢       ∑       s   =   1           v   s     =     v   τ       ,       r   sj     =   0         t     ⁢       (     -     1   2       )     ⁢     2     m   -   2                           =       ⁢         2     m   -   3       ⁢   Λ   ⁢          {           v   s     ❘     v   s       =     v   τ       ,       r   sj     =   1     ,     s   ∈     {     1   ,   …   ⁢           ,   t     }         }            -                     ⁢       2     m   -   3       ⁢   Λ   ⁢          {           v   s     ❘     v   s       =     v   τ       ,       r   sj     =   0     ,     s   ∈     {     1   ,   …   ⁢           ,   t     }         }                            and                     c   τ   o     =       ⁢       Λ   2     ⁢       ∑       s   =   1         v   s     =     v   τ         t     ⁢     2     m   -   2                       =       ⁢       2     m   -   3       ⁢   Λ   ⁢          {           v   s     ❘     v   s       =     v   τ       ,     s   ∈     {     1   ,   …   ⁢           ,   t     }         }                        =       ⁢         2     m   -   3       ⁢   Λ   ⁢          {           v   s     ❘     v   s       =     v   τ       ,       r   sj     =   1     ,     s   ∈     {     1   ,   …   ⁢           ,   t     }         }            +                     ⁢       2     m   -   3       ⁢   Λ   ⁢          {           v   s     ❘     v   s       =     v   τ       ,       r   sj     =   0     ,     s   ∈     {     1   ,   …   ⁢           ,   t     }         }                        
where |S| denotes the number of elements in the set S.
 
     Denoting (c τ   o +d τj   o )/2 by a τj   o , 
                 a     τ   ⁢           ⁢   j     o       c   τ   o       =            {           v   s     ❘     v   s       =     v   τ       ,       r   sj     =   1     ,     s   ∈     {     1   ,   …   ⁢           ,   t     }         }                 {           v   s     ❘     v   s       =     v   τ       ,     s   ∈     {     1   ,   …   ⁢           ,   t     }         }                  
This is the relative frequency that r sj =1 has been learned for v s =v τ  for s=1, . . . , t. Let a τ   o =[a τ1   o  a τ2   o  . . . a τR   o ]′. Then a τ   o /c τ   o  is a relative frequence distribution of r τ  given v τ .
 
     v τ  may be shared by more than one cause (or pattern) and may contain corruption, distortion, occlusion or noise caused directly or indirectly by the sensor measurements such as image pixels and sound recordings. In this case, the label r τ  of v τ  should be a random variable, which can be described or represented only by an empirical probability distribution (or a relative frequency distribution). On the other hand, v τ  may contain parts from more than one cause. In this case, the label r τ  of v τ  should be a fuzzy variable, which can be described or represented only by its membership function as defined in the fuzzy set theory. Fortunately, both the probabilities and membership degrees range between zero and one. The former can be learned mainly as relative frequencies over time and the latter mainly as relative proportions in each v τ  as represented by relative frequencies. a τj   o /c τ   o  evaluated above is a relative frequency representing a probability or a membership degree. The three learning rules in the preceding Subsection facilitate learning both the empirical probabilities and membership degrees and sometimes a combination thereof. For simplicity, the membership degree and membership function will also be referred to as empirical probability and empirical probability distribution in the present invention disclosure. The fact that the empirical probability distribution (or membership function) of the label r τ  can be retrieved from the synapses is striking, but mathematically and naturally necessary. 
     If any number of components in v τ  change their signs and the corresponding components in {hacek over (v)} s  and I change their signs, then the orthogonality property, (7) and (8), still holds. If {hacek over (v)} tq  is inhibitory, the qth component of {hacek over (v)} s −&lt;{hacek over (v)} s &gt; and {hacek over (v)} τ −&lt;{hacek over (v)} τ &gt; change their signs. If additionally ({hacek over (v)} s )=({hacek over (v)} τ )=−I/2, then a τ   o /c τ   o  above is still the relative frequency that r sj =1 has been learned for v s =v τ . In general, a τj   o /c τ   o  above is this relative frequency regardless of how many of the neuronal encoder’ outputs {hacek over (v)} t  are inhibitory. 
     The above empirical-probability and membership-degree interpretation of a τ   o /c τ   o  are valid, only if all the vectors v t , t=1, 2, . . . , input to the neuronal encoder are binary vectors or nearly binary vectors. Biological spiking neurons emit spike trains, where a spike and a nonspike can be looked upon as nearly binary digits. Therefore, the output vectors of a biological neuronal encoder, whose components are spikes and nonspikes, are nearly orthogonal. 
     EXAMPLE 1b 
     This example is a continuation of EXAMPLE 1a. With the v and u from EXAMPLE 1a, let a supervised training data set consists of 8 copies of u with label 1 and 2 copies of u with label 0; and 3 copies of v with label 1 and 27 copies of v with label 0. By (19) and (22), this supervised training data set is learned with λ=Λ=1 (in (19) and (22)) to form the code covariance matrix D and the code deviation accumulation vector C:
 
 D= ½(8−2)( {hacek over (u)}− ½ I )′+½(3−27)( {hacek over (v)}− ½ I )′
 
 C= ½(8+2)( {hacek over (u)}− ½ I )′+½(3+27)( {hacek over (v)}− ½ I )′
 
By (7) and (8), D({hacek over (u)}−½I)=3(4), C({hacek over (u)}−½I)=5(4), D({hacek over (v)}−½I)=−12 (4), C({hacek over (v)}−½I)=15(4). It follows that (D({hacek over (u)}−½I)+C({hacek over (u)}−½I))/(2C({hacek over (u)}−½I))=8/10 is the relative frequency that u has been learned with label 1; and 1−8/10=2/10 is the relative frequency that a has been learned with label 0. Similarly, (D({hacek over (v)}−½I)+C({hacek over (v)}−½I))/(2C({hacek over (v)}−½I))=3/30 is the relative frequency that v has been learned with label 1; and 1−3/30= 27/30 is the relative frequency that v has been learned with label 0.
 
5.4 Masking Factors for Maximal Generalization
 
     Let a vector v τ  that deviates, due to corruption, distortion or occlusion, from each of the vectors v s  that have been learned by the synapses on an artificial neuronal encoder be presented to the neuronal encoder. The neuronal encoder and its synapses are said to have a maximal generalization capability in their retrieval of information, if they are able to automatically find the largest subvector of v τ  that matches at least one subvector among the vectors v s  stored in the synapses and enable post-synaptic neurons to generate an empirical probability distribution of the label of the largest subvector. This maximal capability is achieved by the use of a masking matrix described in this Subsection. A biological interpretation of such a matrix is given at the end of this Subsection. 
     The following are recursive formulas for expanding bipolar binary vectors a=[a 1  a 2  . . . a m ]′ into an orthogonal expansion:
 
 â (1)=[1  a   1 ]′
 
 â (1 , . . . ,j+ 1)=[ â′ (1 , . . . ,j )  a   j+1   â′ (1 , . . . ,j )]′ for  j= 1 , . . . , m− 1
 
 â=â (1 , . . . ,m )  (28)
 
â is called the orthogonal expansion of a. For example, if a=[a 1  a 2  a 3 ], then
 
 â=[ 1  a   1    a   2    a   2   a   1    a   3    a   3   a   1    a   3   a   2    a   3   a   2   a   1 ]
 
     Let us denote the vector a=[a 1  a 2  . . . a m ]′ a with its i 1 -th, i 2 -th, . . . , and i j -th components set equal to 0 by a(i 1   − , i 2   − , . . . , i j   − ), where 1≦i 1 &lt;i 2 &lt; . . . &lt;i≦m, and the neuronal and orthogonal expansions of a(i 1   − , i 2   − , . . . , i j   − ) by {hacek over (a)}(i 1   − , i 2   − , . . . , i j   − ) and â(i 2   − , i 2   − , . . . , i j   − ), respectively. Denoting the m-dimensional vector [1 1 . . . 1]′ by I, the vector I with its i 1 -th, i 2 -th, . . . , and i j -th components set equal to 0 is I(i 1   − , i 2   − , . . . , i j   − ) and the orthogonal expansion of I(i 1   − , i 2   − , . . . , i j   − ) is Î(i 1   − , i 2   − , . . . , i j   − ). For example, if I=[1 1 1]′, then
 
 Î (1 − )=[1 0 1 0 1 0 1 0 1]
 
 Î (2 − )=[1 1 0 0 1 1 0 0]
 
 Î (3 − )=[1 1 1 1 0 0 0 0]  (29)
 
Notice that for the vector a=[a 1  a 2  a 3 ]′,
 
 {hacek over (a)} (1 − )=[0 0  a   2 0  a   3  0 φ( a   3   ,a   2 ) 0]=(diag{circumflex over ( I )}(1 − )) {hacek over (a)} 
 
 {hacek over (a)} (2 − )=[0  a   1  0 0  a   3  φ( a   3   ,a   1 ) 0 0]=(diag{circumflex over ( I )}(2 − )) {hacek over (a)} 
 
 {hacek over (a)} (3 − )=[0  a   1    a   2  φ( a   2   ,a   1 ) 0 0 0 0]=(diag{circumflex over ( I )}(3 − )) {hacek over (a)} 
 
In general, for the vector a=[a 1  a 2  . . . a m ]′,
 
 {hacek over (a)} ( i   1   −   ,i   2   −   , . . . ,i   j   − )=(diag{circumflex over ( I )}( i   1   −   ,i   2   −   , . . . ,i   j   − )) {hacek over (a)} 
 
Notice that diagÎ(i 1   − , i 2   − , . . . , i j   − ) eliminates components in {hacek over (a)} that involve a i     1   , a i     2   , . . . , a i     j   . Therefore, diagÎ(i 1   − , i 2   − , . . . , i j   − ) is called a masking matrix.
 
     An important property of the masking matrix diagÎ(i 1   − , i 2   − , . . . , i j   − ) is the following: Assume that v s  and v τ  are binary vectors. If 
     
       
         
           
             
               
                 
                   
                     
                       
                         ( 
                         
                           diag 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             
                               I 
                               ^ 
                             
                             ⁡ 
                             
                               ( 
                               
                                 
                                   i 
                                   1 
                                   - 
                                 
                                 , 
                                 
                                   i 
                                   2 
                                   - 
                                 
                                 , 
                                 … 
                                 ⁢ 
                                 
                                     
                                 
                                 , 
                                 
                                   i 
                                   j 
                                   - 
                                 
                               
                               ) 
                             
                           
                         
                         ) 
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             
                               v 
                               ⋓ 
                             
                             s 
                           
                           - 
                           
                             
                               1 
                               2 
                             
                             ⁢ 
                             I 
                           
                         
                         ) 
                       
                     
                     = 
                     
                       
                         ( 
                         
                           diag 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             
                               I 
                               ^ 
                             
                             ⁡ 
                             
                               ( 
                               
                                 
                                   i 
                                   1 
                                   - 
                                 
                                 , 
                                 
                                   i 
                                   2 
                                   - 
                                 
                                 , 
                                 … 
                                 ⁢ 
                                 
                                     
                                 
                                 , 
                                 
                                   i 
                                   j 
                                   - 
                                 
                               
                               ) 
                             
                           
                         
                         ) 
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             
                               v 
                               ⋓ 
                             
                             τ 
                           
                           - 
                           
                             
                               1 
                               2 
                             
                             ⁢ 
                             I 
                           
                         
                         ) 
                       
                     
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   then 
                 
               
               
                 
                     
                 
               
             
             
               
                 
                   
                     
                       
                         ( 
                         
                           
                             
                               v 
                               ⋓ 
                             
                             s 
                           
                           - 
                           
                             〈 
                             
                               
                                 v 
                                 ⋓ 
                               
                               s 
                             
                             〉 
                           
                         
                         ) 
                       
                       ′ 
                     
                     ⁢ 
                     
                       ( 
                       
                         diag 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           
                             I 
                             ^ 
                           
                           ⁡ 
                           
                             ( 
                             
                               
                                 i 
                                 1 
                                 - 
                               
                               , 
                               
                                 i 
                                 2 
                                 - 
                               
                               , 
                               … 
                               ⁢ 
                               
                                   
                               
                               , 
                               
                                 i 
                                 j 
                                 - 
                               
                             
                             ) 
                           
                         
                       
                       ) 
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           
                             v 
                             ⋓ 
                           
                           τ 
                         
                         - 
                         
                           〈 
                           
                             
                               v 
                               ⋓ 
                             
                             τ 
                           
                           〉 
                         
                       
                       ) 
                     
                   
                   - 
                   
                     
                       2 
                       
                         m 
                         - 
                         2 
                         - 
                         j 
                       
                     
                     . 
                     
                       
 
                     
                     ⁢ 
                     If 
                   
                 
               
               
                 
                   ( 
                   30 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       
                         ( 
                         
                           diag 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             
                               I 
                               ^ 
                             
                             ⁡ 
                             
                               ( 
                               
                                 
                                   i 
                                   1 
                                   - 
                                 
                                 , 
                                 
                                   i 
                                   2 
                                   - 
                                 
                                 , 
                                 … 
                                 ⁢ 
                                 
                                     
                                 
                                 , 
                                 
                                   i 
                                   j 
                                   - 
                                 
                               
                               ) 
                             
                           
                         
                         ) 
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             
                               v 
                               ⋓ 
                             
                             s 
                           
                           - 
                           
                             
                               1 
                               2 
                             
                             ⁢ 
                             I 
                           
                         
                         ) 
                       
                     
                     ≠ 
                     
                       
                         ( 
                         
                           diag 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             
                               I 
                               ^ 
                             
                             ⁡ 
                             
                               ( 
                               
                                 
                                   i 
                                   1 
                                   - 
                                 
                                 , 
                                 
                                   i 
                                   2 
                                   - 
                                 
                                 , 
                                 … 
                                 ⁢ 
                                 
                                     
                                 
                                 , 
                                 
                                   i 
                                   j 
                                   - 
                                 
                               
                               ) 
                             
                           
                         
                         ) 
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             
                               v 
                               ⋓ 
                             
                             τ 
                           
                           - 
                           
                             
                               1 
                               2 
                             
                             ⁢ 
                             I 
                           
                         
                         ) 
                       
                     
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   then 
                 
               
               
                 
                     
                 
               
             
             
               
                 
                   
                     
                       
                         ( 
                         
                           
                             
                               v 
                               ⋓ 
                             
                             s 
                           
                           - 
                           
                             〈 
                             
                               
                                 v 
                                 ⋓ 
                               
                               s 
                             
                             〉 
                           
                         
                         ) 
                       
                       ′ 
                     
                     ⁢ 
                     
                       ( 
                       
                         diag 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           
                             I 
                             ^ 
                           
                           ⁡ 
                           
                             ( 
                             
                               
                                 i 
                                 1 
                                 - 
                               
                               , 
                               
                                 i 
                                 2 
                                 - 
                               
                               , 
                               … 
                               ⁢ 
                               
                                   
                               
                               , 
                               
                                 i 
                                 j 
                                 - 
                               
                             
                             ) 
                           
                         
                       
                       ) 
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           
                             v 
                             ⋓ 
                           
                           τ 
                         
                         - 
                         
                           〈 
                           
                             
                               v 
                               ⋓ 
                             
                             τ 
                           
                           〉 
                         
                       
                       ) 
                     
                   
                   = 
                   0 
                 
               
               
                 
                   ( 
                   31 
                   ) 
                 
               
             
           
         
       
     
     Using this property, we combine all such masking matrices that set less than or equal to a selected positive integer J of components of v s  equal to zero into the following masking matrix 
                   M   =     I   +       ∑     j   =   1     J     ⁢       ∑       i   j     =   j     m     ⁢           ⁢     …   ⁢           ⁢       ∑       i   2     =   2         i   3     -   1       ⁢       ∑       i   1     =   1         i   2     -   1       ⁢       2       -   η     ⁢           ⁢   j       ⁢     2   j     ⁢   diag   ⁢           ⁢       I   ^     ⁡     (       i   1   -     ,     i   2   -     ,   …   ⁢           ,     i   j   -       )                             (   32   )               
where 2 j  is used to compensate for the factor 2 −j  in 2 m−2−j  in the important property (30), and 2 −nj  is a weight selected to differentiate between different levels j of maskings. Note that the masking matrix M is a diagonal matrix, whose diagonal entries M jj  are called masking factors.
 
     The formulas (24) and (25) together with M constitute the following decovariance rules:
 
 d   τ   =DM ( {hacek over (v)}   τ   −&lt;{hacek over (v)}   τ &gt;)  (33)
 
 c   τ   =CM ( {hacek over (v)}   τ   −&lt;{hacek over (v)}   τ &gt;)  (34)
 
which are equivalent to
 
 d   τkj   =D   kj   M   jj ( {hacek over (v)}   τj   −&lt;{hacek over (v)}   τj &gt;)= M   jj   D   kj ( {hacek over (v)}   τj   −&lt;{hacek over (v)}   τj &gt;)
 
 c   τj   =C   j   M   jj ( {hacek over (v)}   τj   −&lt;{hacek over (v)}   τj &gt;)= M   jj   C   j ( {hacek over (v)}   τj   −&lt;{hacek over (v)}   τj &gt;)
 
for k=1, . . . , R and j=1, . . . ,2 m . c o    10  illustrates synapses and a masking factor retrieve information about the vector v τ  input to the neuronal encoder according to the above decovariance rules.
 
     Note that for k=1, . . . , R, we have the following: 
     
       
         
           
             
               
                 If 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 
                   C 
                   ⁡ 
                   
                     ( 
                     
                       
                         
                           v 
                           ⋓ 
                         
                         τ 
                       
                       - 
                       
                         〈 
                         
                           
                             v 
                             ⋓ 
                           
                           τ 
                         
                         〉 
                       
                     
                     ) 
                   
                 
               
               ≠ 
               0 
             
             , 
             then 
           
         
       
       
         
           
             
               
                 D 
                 k 
               
               ⁡ 
               
                 ( 
                 
                   
                     
                       v 
                       ⋓ 
                     
                     τ 
                   
                   - 
                   
                     〈 
                     
                       
                         v 
                         ⋓ 
                       
                       τ 
                     
                     〉 
                   
                 
                 ) 
               
             
             ≈ 
             
               
                 D 
                 k 
               
               ⁢ 
               
                 M 
                 ⁡ 
                 
                   ( 
                   
                     
                       
                         v 
                         ⋓ 
                       
                       τ 
                     
                     - 
                     
                       〈 
                       
                         
                           v 
                           ⋓ 
                         
                         τ 
                       
                       〉 
                     
                   
                   ) 
                 
               
             
           
         
       
       
         
           
             
               C 
               ⁡ 
               
                 ( 
                 
                   
                     
                       v 
                       ⋓ 
                     
                     τ 
                   
                   - 
                   
                     〈 
                     
                       
                         v 
                         ⋓ 
                       
                       τ 
                     
                     〉 
                   
                 
                 ) 
               
             
             ≈ 
             
               CM 
               ⁡ 
               
                 ( 
                 
                   
                     
                       v 
                       ⋓ 
                     
                     τ 
                   
                   - 
                   
                     〈 
                     
                       
                         v 
                         ⋓ 
                       
                       τ 
                     
                     〉 
                   
                 
                 ) 
               
             
           
         
       
       
         
           
             
               
                 If 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 
                   C 
                   ⁡ 
                   
                     ( 
                     
                       
                         
                           v 
                           ⋓ 
                         
                         τ 
                       
                       - 
                       
                         〈 
                         
                           
                             v 
                             ⋓ 
                           
                           τ 
                         
                         〉 
                       
                     
                     ) 
                   
                 
               
               = 
               0 
             
             , 
             
               
                 but 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 C 
                 ⁢ 
                 
                   
                     ∑ 
                     
                       
                         i 
                         1 
                       
                       = 
                       1 
                     
                     m 
                   
                   ⁢ 
                   
                     
                       ( 
                       
                         diag 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           
                             I 
                             ^ 
                           
                           ⁡ 
                           
                             ( 
                             
                               i 
                               1 
                               - 
                             
                             ) 
                           
                         
                       
                       ) 
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           
                             v 
                             ⋓ 
                           
                           τ 
                         
                         - 
                         
                           〈 
                           
                             
                               v 
                               ⋓ 
                             
                             τ 
                           
                           〉 
                         
                       
                       ) 
                     
                   
                 
               
               ≠ 
               0 
             
             , 
             then 
           
         
       
       
         
           
             
               
                 D 
                 k 
               
               ⁢ 
               
                 
                   ∑ 
                   
                     
                       i 
                       1 
                     
                     = 
                     1 
                   
                   m 
                 
                 ⁢ 
                 
                   
                     ( 
                     
                       diag 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           I 
                           ^ 
                         
                         ⁡ 
                         
                           ( 
                           
                             i 
                             1 
                             - 
                           
                           ) 
                         
                       
                     
                     ) 
                   
                   ⁢ 
                   
                     ( 
                     
                       
                         
                           v 
                           ⋓ 
                         
                         τ 
                       
                       - 
                       
                         〈 
                         
                           
                             v 
                             ⋓ 
                           
                           τ 
                         
                         〉 
                       
                     
                     ) 
                   
                 
               
             
             ≈ 
             
               
                 D 
                 k 
               
               ⁢ 
               
                 M 
                 ⁡ 
                 
                   ( 
                   
                     
                       
                         v 
                         ⋓ 
                       
                       τ 
                     
                     - 
                     
                       〈 
                       
                         
                           v 
                           ⋓ 
                         
                         τ 
                       
                       〉 
                     
                   
                   ) 
                 
               
             
           
         
       
       
         
           
             
               C 
               ⁢ 
               
                 
                   ∑ 
                   
                     
                       i 
                       1 
                     
                     = 
                     1 
                   
                   m 
                 
                 ⁢ 
                 
                   
                     ( 
                     
                       diag 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           I 
                           ^ 
                         
                         ⁡ 
                         
                           ( 
                           
                             i 
                             1 
                             - 
                           
                           ) 
                         
                       
                     
                     ) 
                   
                   ⁢ 
                   
                     ( 
                     
                       
                         
                           v 
                           ⋓ 
                         
                         τ 
                       
                       - 
                       
                         〈 
                         
                           
                             v 
                             ⋓ 
                           
                           τ 
                         
                         〉 
                       
                     
                     ) 
                   
                 
               
             
             ≈ 
             
               CM 
               ⁡ 
               
                 ( 
                 
                   
                     
                       v 
                       ⋓ 
                     
                     τ 
                   
                   - 
                   
                     〈 
                     
                       
                         v 
                         ⋓ 
                       
                       τ 
                     
                     〉 
                   
                 
                 ) 
               
             
           
         
       
       
         
           
             
               
                 If 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 C 
                 ⁢ 
                 
                   ( 
                   
                     
                       
                         v 
                         ⋓ 
                       
                       τ 
                     
                     - 
                     
                       〈 
                       
                         
                           v 
                           ⋓ 
                         
                         τ 
                       
                       〉 
                     
                   
                   ) 
                 
               
               = 
               0 
             
             , 
             
               
                 C 
                 ⁢ 
                 
                   
                     ∑ 
                     
                       
                         i 
                         1 
                       
                       = 
                       1 
                     
                     m 
                   
                   ⁢ 
                   
                     
                       ( 
                       
                         diag 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           
                             I 
                             ^ 
                           
                           ⁡ 
                           
                             ( 
                             
                               i 
                               1 
                               - 
                             
                             ) 
                           
                         
                       
                       ) 
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           
                             v 
                             ⋓ 
                           
                           τ 
                         
                         - 
                         
                           〈 
                           
                             
                               v 
                               ⋓ 
                             
                             τ 
                           
                           〉 
                         
                       
                       ) 
                     
                   
                 
               
               = 
               0 
             
             , 
             
               
 
             
             ⁢ 
             
               
                 but 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 C 
                 ⁢ 
                 
                   
                     ∑ 
                     
                       
                         i 
                         2 
                       
                       = 
                       2 
                     
                     m 
                   
                   ⁢ 
                   
                     
                       ∑ 
                       
                         
                           i 
                           1 
                         
                         = 
                         1 
                       
                       
                         
                           i 
                           2 
                         
                         - 
                         1 
                       
                     
                     ⁢ 
                     
                       
                         ( 
                         
                           diag 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             
                               I 
                               ^ 
                             
                             ⁡ 
                             
                               ( 
                               
                                 
                                   i 
                                   1 
                                   - 
                                 
                                 , 
                                 
                                   i 
                                   2 
                                   - 
                                 
                               
                               ) 
                             
                           
                         
                         ) 
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             
                               v 
                               ⋓ 
                             
                             τ 
                           
                           - 
                           
                             〈 
                             
                               
                                 v 
                                 ⋓ 
                               
                               τ 
                             
                             〉 
                           
                         
                         ) 
                       
                     
                   
                 
               
               ≠ 
               0 
             
             , 
             then 
           
         
       
       
         
           
             
               
                 D 
                 k 
               
               ⁢ 
               
                 
                   ∑ 
                   
                     
                       i 
                       2 
                     
                     = 
                     2 
                   
                   m 
                 
                 ⁢ 
                 
                   
                     ∑ 
                     
                       
                         i 
                         1 
                       
                       = 
                       1 
                     
                     
                       
                         i 
                         2 
                       
                       - 
                       1 
                     
                   
                   ⁢ 
                   
                     
                       ( 
                       
                         diag 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           
                             I 
                             ^ 
                           
                           ⁡ 
                           
                             ( 
                             
                               
                                 i 
                                 1 
                                 - 
                               
                               , 
                               
                                 i 
                                 2 
                                 - 
                               
                             
                             ) 
                           
                         
                       
                       ) 
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           
                             v 
                             ⋓ 
                           
                           τ 
                         
                         - 
                         
                           〈 
                           
                             
                               v 
                               ⋓ 
                             
                             τ 
                           
                           〉 
                         
                       
                       ) 
                     
                   
                 
               
             
             ≈ 
             
               
                 D 
                 k 
               
               ⁢ 
               
                 M 
                 ⁡ 
                 
                   ( 
                   
                     
                       
                         v 
                         ⋓ 
                       
                       τ 
                     
                     - 
                     
                       〈 
                       
                         
                           v 
                           ⋓ 
                         
                         τ 
                       
                       〉 
                     
                   
                   ) 
                 
               
               ⁢ 
               
                 
 
               
               ⁢ 
               C 
               ⁢ 
               
                 
                   ∑ 
                   
                     
                       i 
                       2 
                     
                     = 
                     2 
                   
                   m 
                 
                 ⁢ 
                 
                   
                     ∑ 
                     
                       
                         i 
                         1 
                       
                       = 
                       1 
                     
                     
                       
                         i 
                         2 
                       
                       - 
                       1 
                     
                   
                   ⁢ 
                   
                     
                       ( 
                       
                         diag 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           
                             I 
                             ^ 
                           
                           ⁡ 
                           
                             ( 
                             
                               
                                 i 
                                 1 
                                 - 
                               
                               , 
                               
                                 i 
                                 2 
                                 - 
                               
                             
                             ) 
                           
                         
                       
                       ) 
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           
                             v 
                             ⋓ 
                           
                           τ 
                         
                         - 
                         
                           〈 
                           
                             
                               v 
                               ⋓ 
                             
                             τ 
                           
                           〉 
                         
                       
                       ) 
                     
                   
                 
               
             
             ≈ 
             
               CM 
               ⁡ 
               
                 ( 
                 
                   
                     
                       v 
                       ⋓ 
                     
                     τ 
                   
                   - 
                   
                     〈 
                     
                       
                         v 
                         ⋓ 
                       
                       τ 
                     
                     〉 
                   
                 
                 ) 
               
             
           
         
       
     
     Continuing in this manner, it is seen that D k M({hacek over (v)} τ −&lt;{hacek over (v)} τ &gt;) and CM({hacek over (v)} τ −&lt;{hacek over (v)} τ &gt;) always use the greatest number of uncorrupted, undistorted or unoccluded components of v τ  in estimating d τk , c τ , and a τk . The vector a τj /c τ  is now an estimate of the conditional empirical probability distribution of r τ  given v τ , using the greatest number of uncorrupted, undistorted or unoccluded components of v τ . 
     If some terms in (32) are missing, the generalization effect of M degrades only gracefully. The example weight 2 −5j  in (32) is used to illustrate M generalizing maximally in EXAMPLE 2c. The weight is believed to be 2 −j  in biological neural networks for a reason to be discussed. J in (32) is believed to vary from brain region to brain region. The range of J can be found by biological experiments. 
     EXAMPLE 2c 
     Let the code covariance matrices, D and C, be those obtained in EXAMPLE 2a and EXAMPLE 2b. Using (29), we construct the masking matrix by the formula (32) for J=1, 
     
       
         
           
             
               
                 
                   
                     
                       
                         M 
                         = 
                           
                         ⁢ 
                         
                           I 
                           + 
                           
                             
                               2 
                               
                                 - 
                                 5 
                               
                             
                             ⁢ 
                             2 
                             ⁢ 
                             
                                 
                             
                             ⁢ 
                             
                               diag 
                               ⁡ 
                               
                                 ( 
                                 
                                   
                                     
                                       I 
                                       ^ 
                                     
                                     ⁡ 
                                     
                                       ( 
                                       
                                         1 
                                         - 
                                       
                                       ) 
                                     
                                   
                                   + 
                                   
                                     
                                       I 
                                       ^ 
                                     
                                     ⁡ 
                                     
                                       ( 
                                       
                                         2 
                                         - 
                                       
                                       ) 
                                     
                                   
                                   + 
                                   
                                     
                                       I 
                                       ^ 
                                     
                                     ⁡ 
                                     
                                       ( 
                                       
                                         3 
                                         - 
                                       
                                       ) 
                                     
                                   
                                 
                                 ) 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                           
                         ⁢ 
                         
                           I 
                           + 
                           
                             
                               2 
                               
                                 - 
                                 4 
                               
                             
                             ⁢ 
                             
                               diag 
                               ⁡ 
                               
                                 [ 
                                 
                                   
                                     
                                       3 
                                     
                                     
                                       2 
                                     
                                     
                                       2 
                                     
                                     
                                       1 
                                     
                                     
                                       2 
                                     
                                     
                                       1 
                                     
                                     
                                       1 
                                     
                                     
                                       0 
                                     
                                   
                                 
                                 ] 
                               
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   35 
                   ) 
                 
               
             
           
         
       
     
     Recall that with the masking matrix M, we use d τj =D τj M({hacek over (v)} τ −&lt;{hacek over (v)} τ &gt;) and c τ =C τ M({hacek over (v)} τ −&lt;{hacek over (v)} τ &gt;) in general, where D j  denotes the jth row of D. If c τ ≠0, the empirical probability p τj =(d τj /c τ +1)/2. The masking matrix M for this example is shown in  FIG. 11 . 
     Assume that {hacek over (v)} 1  is presented to the synapses containing the code covariance matrices through M. By matrix multiplication,
 
 DM ( {hacek over (v)}   1   −&lt;{hacek over (v)}   1 &gt;)=−1+2 −4 (−½)=−1.0312
 
 CM ( {hacek over (v)}   1   −&lt;{hacek over (v)}   1 &gt;)=1+2 −4 (5/2)=1.1563
 
The empirical probability the label of v 4  is 1 is (DM({hacek over (v)} 1 −&lt;{hacek over (v)} 1 &gt;)/(CM({hacek over (v)} 1 −&lt;{hacek over (v)} 1 &gt;))+1)/2=0.054, and the empirical probability that the label of v 4  is 0 is 0.946. Note that v 1  with a label of 0 has been learned. The empirical probability that the label of v 4  is 0 should be 1. The use of M causes a very small amount of error to the empirical probability, which can be controlled by changing the weight, 2 −5 .
 
     The neuronal codes of the three vertices, v 4 , v 5  and v 6 , of the cube in  FIG. 9 , which are not included in the supervised learning data, are listed as follows: 
     
       
         
           
               
               
               
               
               
               
               
               
               
             
               
                   
               
               
                   
                   
                   
                   
                 φ 
                   
                 φ 
                 φ 
                 φ 
               
               
                 {hacek over (v)}′ t   
                 0 
                 v t1   
                 v t2   
                 (v t2 , v t1 ) 
                 v t3   
                 (v t3 , v t1 ) 
                 (v t3 , v t2 ) 
                 (v t3 , v t2 , v t1 ) 
               
               
                   
               
             
            
               
                 {hacek over (v)}′ 4   
                 0 
                 1 
                 1 
                 0 
                 0 
                 1 
                 1 
                 0 
               
               
                 {hacek over (v)}′ 5   
                 0 
                 0 
                 0 
                 0 
                 1 
                 1 
                 1 
                 1 
               
               
                 {hacek over (v)}′ 6   
                 0 
                 1 
                 0 
                 1 
                 1 
                 0 
                 1 
                 0 
               
               
                   
               
            
           
         
       
     
     Simple matrix-vector multiplication yields D({hacek over (v)} t −&lt;{hacek over (v)} t &gt;)=0 and C({hacek over (v)} t −&lt;{hacek over (v)} t &gt;)=0 for t=4, 5, 6. Hence no information is provided on v t  by D({hacek over (v)} t −&lt;{hacek over (v)} t &gt;) and C({hacek over (v)} t −&lt;{hacek over (v)} t &gt;) for t=4, 5, 6. This shows that if v t  has not been learned, then generalization is necessary to get information on it from the code covariance matrices. 
     Assume that v 4  is presented to the synapses containing the code covariance matrices. By matrix multiplication,
 
 DM ( {hacek over (v)}   4   −&lt;{hacek over (v)}   4 &gt;)=0+2 −10 (9+2+6−3−2+1−1)=2 −10 (12)
 
 CM ( {hacek over (v)}   4   −&lt;{hacek over (v)}   4 &gt;)=0+2 −10 (15−2+2−1+2−1−3)=2 −10 (12)
 
The empirical probability the label of v 4  is 1 is (DM({hacek over (v)} 4 −&lt;{hacek over (v)} 4 &gt;)/(CM({hacek over (v)} 4 −&lt;{hacek over (v)} 4 &gt;))+1)/2=1. From  FIG. 9 , we see that all the three vertices neighboring v 4  have been learned and have a label of 1. It is a good generalization that a label of 1 is assigned to v 4 .
 
     Assume that {hacek over (v)} 6  is presented to the code covariance matrices. By matrix multiplication,
 
 DM ( {hacek over (v)}   6   −&lt;{hacek over (v)}   6 &gt;)=0+2 −7 (9+2−6+3+2−1−1)=2 −7 (8)  (36)
 
 CM ( {hacek over (v)}   6 −&lt;{hacek over (v)} 6 &gt;)=0+2 −7 (15−2−2+1−2+1−3)=2 −7 (8)  (37)
 
The empirical probability the label of v 6  is 1 is (DM({hacek over (v)} 6 −&lt;{hacek over (v)} 6 &gt;)/(CM({hacek over (v)} 6 −&lt;{hacek over (v)} 6 &gt;))+1)/2=1. From  FIG. 9 , we see that only two vertices neighboring v 4  have been learned, and they both have a label of 1. It is a good generalization that a label of 1 is assigned to v 6 .
 
     Assume that {hacek over (v)} 5  is presented to the synapses containing the code covariance matrices. By matrix multiplication,
 
 DM ( {hacek over (v)}   5   −&lt;{hacek over (v)}   5 &gt;)=0+2 −7 (9−2−6−3+2+1−1)=2 −7 (0)
 
 CM ( {hacek over (v)}   5   −&lt;{hacek over (v)}   5 &gt;)=0+2 −7 (15+2−2−1−2−1−3)=2 −7 (8)
 
The empirical probability the label of v 5  is 1 is (DM({hacek over (v)} 5 −&lt;{hacek over (v)} 5 &gt;)/(CM({hacek over (v)} 5 −&lt;{hacek over (v)} 5 &gt;))+1)/2=½. From  FIG. 9 , we see that only two vertices neighboring v 4  have been learned, and one of them has a label of 1, and the other has a label of 0. No generalization is possible. A label of 1 is assigned to v 6  with an empirical probability of ½ and that a label of 0 is assigned to v 6  with equal empirical probability.
 
     The above example shows that the weight 2 −5j  in (32) is more than adequate to differentiate between different levels j of maskings for a neuronal encoder with only 3 inputs. The greater the number m of inputs to a neuronal encoder, the less two adjacent levels, j and j+1, of maskings need to be differentiated. For example, if the number m of components in the input vector is 12, any 11 of the 12 components should be almost as good as the 12 in determining the label of the input vector. A reduction by 50% of emphasis on the subvector is usually adequate. 
     Therefore, if m is 12 or larger, the weight 2 −nj  in (32) can be set equal to 2 −j  so that the reduction of emphasis by 50% from level j+1 and j is adequate. In this case, the masking matrix M is a mathematical idealization and organization of a large number of neuronal trees with nested and overlapped inputs. The following example illustrates this biological interpretation of the masking matrix M with the weight being 2 −j  in (32). 
     EXAMPLE 3 
     Using (29), we construct the masking matrix by the formula (32) for J=1, 
                         M   =       ⁢     I   +       2     -   1       ⁢   2   ⁢           ⁢     diag   ⁡     (         I   ^     ⁡     (     1   -     )       +       I   ^     ⁡     (     2   -     )       +       I   ^     ⁡     (     3   -     )         )                       =       ⁢     I   +     diag   ⁢           ⁢       I   ^     (     1   -     )       +     diag   ⁢           ⁢       I   ^     (     2   -     )       +     diag   ⁢           ⁢       I   ^     (     3   -     )                       (   38   )               
In the product
 
 M{hacek over (v)}   τ   ={hacek over (v)}   τ +(diag{circumflex over ( I )}(1 − )) {hacek over (v)}   τ +(diag{circumflex over ( I )}(2 − )) {hacek over (v)}   τ +(diag{circumflex over ( I )}(3)) {hacek over (v)}   τ 
 
(diagÎ(k − ){hacek over (v)} τ  eliminates all terms in {hacek over (v)} τ  that contain v τk  and can be viewed as the output vector of a model neuronal encoder for k=1, 2, 3. These model neuronal encoders are shown in  FIG. 13 . M{hacek over (v)} τ  can be viewed as the sum of the output vectors from four model neuronal encoders with nested and overlapped input vectors, v τ =[v τ1  v τ2  v τ3 ]′, [v τ2  v τ3 ]′, [v τ1  v τ3 ]′, and [v τ1  v τ2 ]′.
 
     The difference between  FIGS. 5   a - 7   a  and  FIGS. 5   b - 7   b  is that the former do not contain the masking matrix, but the latter do. The above biological interpretation of the masking matrix explains said difference. In other words, while biologically there is no masking factor being multiplied to a synapse output as shown in  FIG. 5   a - 7   a , a large number of biological neuronal encoders with nested and overlapped inputs are mathematically idealized and organized to have the effect of the masking factor as shown in  FIGS. 5   b - 7   b.    
     5.5 Processing Units (PUs) 
     The LOM (low-order model) disclosed in the present invention disclosure is a network of processing units. A processing unit comprises model neuronal encoders, model synapses, masking matrices, a model nonspiking neuron and a number of model spiking neurons. Let us denote the vector input to the PU at time or numbering t, the number of model neuronal encoders in the PU and the number of model spiking neurons in the PU by v t , R and Ψ, respectively. 
     5.5.1 Multiple Model Neuronal Encoders and General Neuronal Codes 
     Recall that if v t =[v t1  v t2  . . . v tN ], then the neuronal code {hacek over (v)} t  of v t  is 2 N  dimensional. If there were only one neuronal encoder in a PU, there would be two difficulties. First, 2 N  grows exponentially as N increases. Second, a large number of terms in the masking matrix M in (32) are required for masking even a small proportion J/N of the components of v t  if N is large. These difficulties motivated the use of Ψ model neuronal encoders. 
     Let v t (ψ), ψ=1, . . . , Ψ, be subvectors of v t , where the components of v t (ψ) are randomly selected from those of v t  such that the union of all the components of v t (ψ), ψ=1, . . . , Ψ, is the set of components of v t . The subvectors v t (ψ), ψ=1, . . . , Ψ, are encoded by the Ψ model neuronal encoders into neuronal codes {hacek over (v)} t (ψ), ψ=1, . . . , Ψ. Their averages over a time window [t−q v +1, t] of width q v  are denoted by &lt;{hacek over (v)} t (ψ)&gt;, ψ=1, . . . , Ψ. We assemble these vectors into
 
 {hacek over (v)}   t   =[{hacek over (v)}   t ′(1)  {hacek over (v)}   t ′(2) . . .  {hacek over (v)}   t ′(Ψ)]′  (39)
 
&lt; {hacek over (v)}   t   &gt;=[&lt;{hacek over (v)}   t ′(1)&gt; &lt; {hacek over (v)}   t ′(2)&gt; . . . &lt; {hacek over (v)}   t ′(Ψ)&gt;]′  (40)
 
Notice that {hacek over (v)} t  here is not the neuronal code of v t  defined in (5), but consists of the neuronal codes of the Ψ subvectors. {hacek over (v)} t  and &lt;{hacek over (v)} t &gt; above are called a general neuronal code of v t  and an average general neuronal code respectively. This dual use of the symbol {hacek over (v)} t  is not expected to cause confusion. Note that the vectors v t (ψ), ψ=1, . . . , Ψ, may have common components and different dimensionalities, dim v t (ψ), ψ=1, . . . , Ψ, but every component of v t  is included in at least one v t (ψ). Equally importantly, the components of v t (ψ) are selected at random from those of the vector v t  input to the PU. The difference {hacek over (v)} t −&lt;{hacek over (v)} t &gt; between the general neuronal code of v t  and the average general neuronal code is called the general code deviation vector.
 
5.5.2 General Code Covariance Matrix, General Code Deviation Accumulation Vector and General Masking Matrix
 
     For the neuronal codes, {hacek over (v)} t (ψ), ψ=1, . . . , Ψ, let the corresponding code covariance matrices be denoted by D(ψ), ψ=1, . . . , Ψ; and let the corresponding code deviation accumulation vectors be denoted by C(ψ), ψ=1, . . . , Ψ; and the corresponding masking matrices be denoted by M(ψ), ψ=1, . . . , Ψ. We assemble these matrices into
 
 D=[D (1)  D (2) . . .  D (Ψ)]  (41)
 
 C=[C (1)  C (2) . . .  C (Ψ)]  (42)
 
 M =diag[ M (1)  M (2) . . .  M (Ψ)]  (43)
 
Here D and C are respectively called general code covariance matrix and general code deviation accumulation vector, and M is called the general masking matrix for the Ψ neuronal encoders. Note that D, C and M above are not those defined for 2 dim v     t   -dimensional neuronal codes of the vector v t  input to the PU. The dual use of the symbols here is not expected to cause confusion either. The general masking matrix M is a diagonal matrix, whose diagonal entries are called masking factors.
 
     D, C and M above are R×Σ ψ=1   Ψ 2 dim v     t     (ψ) , 1×Σ ψ=1   Ψ 2 dim v     t     (ψ) , and Σ ψ=1   Ψ 2 dim v     t     (ψ) ×Σ ψ=1   Ψ 2 dim v     t     (ψ)  matrices, respectively. Note that by choosing dim v t (ψ) smaller than dim v t , 2 dim v     t     (ψ)  is much smaller than 2 dim v     t   , and the dimensionalities of the general code covariance matrix D, general code deviation accumulation vector C, and the general masking matrix M are much smaller than those obtained from using 2 dim v     t   -dimensional neuronal codes of the vector v t  input to the PU. Therefore, the two aforementioned difficulties with a single neuronal encoder with a 2 dim v     t   -dimensional output vector are alleviated by the use of multiple neuronal encoders in a PU. A third advantage of multiple neuronal encoders in a PU is the enhancement of generalization capability: If the neuronal code from one neuronal encoder fails to retrieve any useful information from a general code covariance matrix and a code deviation accumulation matrix, other neuronal codes from other neuronal encoders may still retrieve enough information for detecting and recognizing the vector v t  input to the PU. 
     An example model spiking neuron with its masking matrix M, synapses, and two neuronal encoders for unsupervised covariance learning in a PU is shown in  FIG. 15 . The two neuronal encoders input subvectors, v τ (1) and v τ (2), of the vector v τ  input to the PU. 
     We stress that in describing a PU, the symbols, v t , {hacek over (v)} t , D, C, and M, denote the vector input to the PU, the general neuronal code, the general code covariance matrix, the general code deviation accumulation vector, and the general masking matrix respectively, unless indicated otherwise. 
     5.5.3 Processing Neuronal Codes 
     By the information retrieval formula, (33) and (34), applied to each of the Ψ neuronal encoder, its synapses, the masking matrices M(ψ), and the model spiking and nonspikeing neurons; upon the arrival of v τ  the following products, d τ (ψ) and c τ (ψ), ψ=1, . . . , Ψ, are obtained for ψ=1, . . . , Ψ:
 
 d   τ (ψ)= D (ψ) M (ψ)( {hacek over (v)}   τ (ψ)−&lt; {hacek over (v)}   τ (ψ)&gt;)  (44)
 
 c   τ (ψ)= C (ψ) M (ψ)( {hacek over (v)}   τ (ψ)−&lt; {hacek over (v)}   τ (ψ)&gt;)  (45)
 
which are a vector with R components and a scalar, respectively. Redefine the symbols d τ , c τ  and a τ  by
 
                     d   τ     =       DM   ⁡     (         v   ⋓     τ     -     〈       v   ⋓     τ     〉       )       =       ∑     ψ   =   1     Ψ     ⁢       d   τ     ⁡     (   ψ   )                   (   46   )                 c   τ     =       CM   ⁡     (         v   ⋓     τ     -     〈       v   ⋓     τ     〉       )       =       ∑     ψ   =   1     Ψ     ⁢       c   τ     ⁡     (   ψ   )                   (   47   )                 a   τ     -       (       d   τ     +       c   τ     ⁢   I       )     /   2             (   48   )               I   =       [         1       1       …       1         ]     ′             (   49   )               
Let us reset the subscripts of the τ ψ=1   Ψ 2 dim {hacek over (v)}     τ     (ψ)  components {hacek over (v)} τ  in 39 from 1 to Σ ψ=1   Ψ 2 dim {hacek over (v)}     τ     (ψ) . We do the same to those of the diagonal entries of the general masking matrix M in 43, those of the general cod deviation accumulation (row) vector C in 42, and those of each row of the general code covariance matrix D in 41. Then the two equations, (46) and (47), are respectively equivalent to
 
 d   τkj   =D   kj   M   jj ( {hacek over (v)}   τj   −&lt;{hacek over (v)}   τj &gt;)  (50)
 
 c   τj   =C   j   M   jj ( {hacek over (v)}   τj   −&lt;{hacek over (v)}   τj &gt;)  (51)
 
for k=1, . . . , R and j=1, . . . , Σ ψ=1   Ψ 2 dim {hacek over (v)}     τ     (ψ) , where M jj , d τkj , C j , and {hacek over (v)} τj  are the jth diagonal entry of the general masking matrix M, the (k, j)th entry of the general code covariance matrix D, the jth entry of the general code deviation accumulation vector C, and the jth entry of the general neuronal code {hacek over (v)} τ  at time or numbering τ, respectively. Using the matrix notation [d τkj ], which denotes a matrix with the (k, j)th entry being d τkj  and the vector notation [c τj ], which denotes a row vector with the jth entry being c τj , the above equations (50) and (51) can be expressed as follows:
 
[ d   τkj   ]=[M   jj   D   kj ( {hacek over (v)}   τj   −&lt;{hacek over (v)}   τj &gt;)]  (52)
 
[ c   τj   ]=[M   jj   C   j ( {hacek over (v)}   τj   −&lt;{hacek over (v)}   τj &gt;)]  (53)
 
where D kj ({hacek over (v)} τj −&lt;{hacek over (v)} τj &gt;) and C j ({hacek over (v)} τj −&lt;{hacek over (v)} τj &gt;) can be looked upon as the outputs of the synapses with weights D kj  and C j  respectively after performing the multiplications involved in response to the input {hacek over (v)} τj −&lt;{hacek over (v)} τj &gt;; and M jj  are called masking factors. The masking factor M jj  is multiplied to the output D kj ({hacek over (v)} τj −&lt;{hacek over (v)} τj &gt;) of synapse j preceding spiking neuron k and to the output C j ({hacek over (v)} τj −&lt;{hacek over (v)} τj &gt;) of synapse j preceding nonspiking neuron.
 
     Denoting (c τ (ψ)+d τj (ψ))/2 by a τj (ψ), For each ψ=1, . . . , Ψ, the ratio a τj (ψ)/c τ (ψ)=(d τj (ψ)/c τ (ψ)+1)/2 is an estimate of the empirical conditional probability that the jth component r τj (ψ) of the label r τ (ψ) of v t (ψ) is equal to 1 given v t (ψ). If v t (ψ), ψ=1, . . . , Ψ, share the same label r τ , that is the label of v t  (i.e., r τ (ψ)=r τ  for ψ=1, . . . , Ψ; then the best estimate of the empirical conditional probability that the jth component r τj  of label r τ  of v t  is equal to 1 given v t (ψ), ψ=1, . . . , Ψ, is 
                 (       ∑     ψ   =   1     Ψ     ⁢       a     τ   ⁢           ⁢   j       ⁡     (   ψ   )         )     /     (       ∑     ψ   =   1     Ψ     ⁢       c   τ     ⁡     (   ψ   )         )       =       (         (       ∑     ψ   =   1     Ψ     ⁢       d     τ   ⁢           ⁢   j       ⁡     (   ψ   )         )     /     (       ∑     ψ   =   1     Ψ     ⁢       c   τ     ⁡     (   ψ   )         )       +   1     )     /   2           
for j=1, . . . , R, or equivalently,
 
 a   τ   /c   τ =( d   τ   /c   τ   +I )/2
 
which is a representation of an empirical probability distribution denoted by p τ . How pseudo-random binary numbers are generated in accordance with p τ  is described in Subsubsection 5.5.5.
 
     Note that {hacek over (v)} τ , &lt;{hacek over (v)} τ &gt;, D, C and M, in describing a PU (and hence in this subsubsection) are the general neuronal code, the general code covariance matrix, the general code deviation accumulation vector, and the general masking matrix defined in (39), (40), (41), (42), and (43), respectively. Note also that a τ , c τ  and d τ  in describing a PU (and hence in this subsubsection) are computed using the general versions of {hacek over (v)} τ , &lt;{hacek over (v)} τ &gt;, D, C and M, unless indicated otherwise. In describing a PU, the symbol v t  remains to denote the vector input to the PU. 
     5.5.4 Model Nonspiking Neurons 
     A model nonspiking neuron in a PU receives signals from dim {hacek over (v)} τ =Σ ψ=1   Ψ 2 dim v     t     (ψ)  synapses connected to the Ψ neuronal encoders in the PU. The strengths of these synapses form the code deviation accumulation vector C. Receiving C j (ψ)M jj (ψ)({hacek over (v)} τ (ψ)−&lt;{hacek over (v)} τ (ψ)&gt;) j , j=1, 2, . . . , 2 dim v     t     (ψ) , ψ=1, 2, . . . , Ψ, the model nonspiking neuron performs addition of them. The output of the model nonspiking neuron is 
                           c   τ     =       ⁢       ∑     ψ   =   1     Ψ     ⁢       C   ⁡     (   ψ   )       ⁢     M   ⁡     (   ψ   )       ⁢     (           v   ⋓     τ     ⁡     (   ψ   )       -     〈         v   ⋓     τ     ⁡     (   ψ   )       〉       )                     =       ⁢     CM   ⁡     (         v   ⋓     τ     -     〈       v   ⋓     τ     〉       )                     (   54   )               
which is an estimate of the total number of times v τ  or its variants have been encoded and stored in C with effects of the forgetting factor, normalizing constant, and input corruption, distortion and occlusion included. c τ  is a graded signal transmitted to the neighboring R model spiking neurons in the same PU (processing unit).
 
     An artificial nonspiking neuron is shown in  FIG. 19 . It evaluates the sum of its inputs, c τj =M jj C j ({hacek over (v)} τj −&lt;{hacek over (v)} τj &gt;), j=1, . . . , Σ ψ=1   Ψ 2 dim v     τ     (ψ) . 
     5.5.5 Model Spiking Neurons 
     Each of the R model spiking neurons in the PU receives signals from Σ ψ=1   Ψ 2 dim v     t     (ψ)  synapses connected to the neuronal encoders in the PU. The entries of the k-th row D k  of D are the strengths of these synapses for model spiking neuron j for j=1, . . . , R. In response to v τ , model spiking neuron k performs addition of D kj (ψ)M jj (ψ)({hacek over (v)} τ (ψ)−&lt; τ (ψ)&gt;) j , j=1, 2, . . . , 2 dim v     t     (ψ) , ψ=1, 2, . . . , Ψ, from the Σ ψ=1   Ψ 2 dim v     t     (ψ)  synapses. The output of model spiking neuron k is 
                           d     τ   ⁢           ⁢   k       =       ⁢       ∑     ψ   =   1     Ψ     ⁢         D   k     ⁡     (   ψ   )       ⁢     M   ⁡     (   ψ   )       ⁢     (           v   ⋓     τ     ⁡     (   ψ   )       -     〈         v   ⋓     τ     ⁡     (   ψ   )       〉       )                     =       ⁢       D   k     ⁢     M   ⁡     (         v   ⋓     τ     -     〈       v   ⋓     τ     〉       )                       (   55   )               
which is an estimate of the total number of times v τ  and its variants have been encoded and stored in D with the kth component r τk  of r τ  being +1 minus the total number of times v τ  and its variants have been encoded and stored in D with the kth component r τj  being −1. Included in this estimate are the effects of the forgetting factor, normalizing constant, and input corruption, distortion and occlusion.
 
     Therefore, (c τ +d τk )/2 is an estimate of the total number of times v τ  and its variants have been encoded and stored in C with the kth component r τk  of r τ  being 1. Consequently, (d τk /c τ +1)/2 is an estimate p τk  of the empirical probability that r τ , is equal to 1 given v t . Model spiking neuron k then uses a pseudo-random number generator to generate a spike (i.e., 1) with probability p τk  or no spike (i.e., 0) with probability 1−p τk . This 1 or 0 is the output u τk =v{p τk } of model spiking neuron k at time or numbering τ. u τk  is thus a point estimate of the k-th component r τk  of the label r τ  of v τ . 
     Note that the vector
 
 p   τ   =[p   τ1    p   τ2    . . . p   τR ]′
 
is a representation of an empirical probability distribution of the label r τ . Note also that the outputs of the R model spiking neurons in response to v τ  form a binary vector u τ =v{p τk }, which is a point estimate of the label r τ  of v τ .
 
     Artificial spiking neuron k in the PU is shown in  FIG. 23 . It inputs d τkj =D kj (ψ)M(ψ)·({hacek over (v)} τ (ψ)−&lt;v τ (ψ)&gt;), j=1, . . . , Σ ψ=1   Ψ 2 dim v     τ     (ψ) , and c τ ; and evaluates the sum Σ j d τkj  of d τkj , j=1, . . . , Σ ψ=1   Ψ 2 dim v     τ     (ψ) . If c τ =0, the artificial spiking neuron k then sets p τk =½. If c τ ≠0, the artificial spiking neuron k then sets p τk =((Σ j d τkj )/c τ +1)/2. The artificial spiking neuron generates a pseudo-random number v{p τk } as follows: v{p τk }=1 with probability p τk  and v{p τk }=0 with probability 1−p τk . An artificial spiking neuron is shown in  FIG. 20 . 
     Note that the PU has Σ ψ=1   Ψ 2 dim v     t     (ψ)  synapses for each of the R spiking neurons and has the same number of synapses for the nonspiking neuron. However, the R spiking neurons and the nonspiking neuron may have different numbers of synapses. 
     A flow chart of a PU without a learning mechanism is shown in  FIG. 21 . At time or numbering τ, the PU receives a (feature) vector v τ . The neuronal encoders in the PU encode v τ  into the general neuronal code {hacek over (v)} τ  by (39). The synapses and the masking matrices compute and output c τj =C j M jj ({hacek over (v)} τj −&lt;{hacek over (v)} τ &gt;) and d τkj =D kj M jj ({hacek over (v)} τj −&lt;{hacek over (v)} τj &gt;) for all k and j, or equivalently the vector [c τj ] and matrix [d τk ], where C, D and M are those in (42, (41) and (43), respectively. The model nonspiking neuron sums up c τj  over all j to form 
               c   τ     =       ∑   j     ⁢     c     τ   ⁢           ⁢   j               
in (47). Model spiking neuron k sums up d τkj  over all j to form
 
               d     τ   ⁢           ⁢   k       =       ∑   j     ⁢       d     τ   ⁢           ⁢   kj       .             
If c τ =0, then the spiking neuron sets p τ =½. Otherwise, for k=1, . . . , R, the spiking neuron computes the empirical probability distribution p τk =(y τk +1)/2, where y τk =d τk /c τ , and generates a pseudo-random number v{p τk } in accordance with the empirical probability p τk : P(v{p τk }=1)=p τk  and P(v{p τk }=0)=1−p τk .
 
     If the general code covariance matrix D in the PU in  FIG. 21  has been learned in unsupervised learning by the unsupervised covariance rule and held fixed, the PU is called an unsupervised PU (UPU). If D in the PU in  FIG. 21  has been learned in supervised learning by the supervised covariance rule and held fixed, the PU is called a supervised PU (SPU). 
     5.5.6 Unsupervised and Supervised Learning 
     The general code deviation accumulation vector C, whose components are stored in the artificial synapses with a feedforward connection to the artificial nonspiking neuron, is adjusted by the following unsupervised accumulation rule: 
                   C   ←       λ   ⁢           ⁢   C     +       Λ   2     ⁢       (         v   ⋓     τ     -     〈       v   ⋓     τ     〉       )     ′                 (   56   )               
where the general neuronal code {hacek over (v)} τ  and its average over time &lt;{hacek over (v)} τ &gt; are defined in (39), and (40). Note that (56) is equivalent to:
 
               C   ⁡     (   ψ   )       ←       λC   ⁡     (   ψ   )       +       Λ   2     ⁢       (           v   ⋓     τ     ⁡     (   ψ   )       -     〈         v   ⋓     τ     ⁡     (   ψ   )       〉       )     ′               
for #=1, . . . , Ψ.
 
     In unsupervised learning by the PU, the general code covariance matrix D, whose entries are stored in the artificial synapses with a feedforward connection to the R artificial spiking neurons, is adjusted by the following unsupervised covariance rule:
 
 D←λD +Λ( v{p   τ   }−&lt;v{p   τ }&gt;)( {hacek over (v)}   τ   −&lt;{hacek over (v)}   τ &gt;)′  (57)
 
where the general neuronal code {hacek over (v)} τ  and its average over time &lt;{hacek over (v)} τ &gt; are defined in (39), and (40), and v{p τ } is the output vector u τ  of the PU. Note that (57) is equivalent to:
 
 D (ψ)←λ D (ψ)+Λ( u   τ   −&lt;u   τ &gt;)( {hacek over (v)}   τ (ψ)−&lt; {hacek over (v)}   τ (ψ)&gt;)′
 
for ψ=1, . . . , Ψ, where u τ =v{p τ }.
 
     In supervised learning by the PU, the general code covariance matrix D, whose entries are stored in the artificial synapses with a feedforward connection to the R artificial spiking neurons, is adjusted by the following supervised covariance rule:
 
 D←λD +Λ( r   τ −½)( {hacek over (v)}   τ   −&lt;{hacek over (v)}   τ &gt;)′  (58)
 
where the general neuronal code {hacek over (v)} τ  and its average over time &lt;{hacek over (v)} τ &gt; are defined in (39), and (40), and r τ  is the label of v τ  (or the measurements in the receptive field of the PU) that is provided from outside of the PU. The provided label r τ  is an R-dimensional binary vector. Note that (58) is equivalent to:
 
 D (ψ)←λ D (ψ)+Λ( r   τ −½)( {hacek over (v)}   τ (ψ)−&lt; {hacek over (v)}   τ (ψ)&gt;)′
 
for ψ=1, . . . , Ψ.
 
     The adjusted C and D are delayed for one unit of time or one numbering before being stored in the model synapses. 
       FIG. 22  illustrates a PU with an unsupervised learning mechanism to learn the general code covariance matrix D by the unsupervised covariance rule and an unsupervised learning mechanism to learn the general code deviation accumulation vector C by the unsupervised accumulation rule. The adjusted C and D are delayed for one unit of time or one numbering before being stored in the model synapses. The PU in  FIG. 22  is called an unsupervised PU (UPU). 
       FIG. 23  illustrates a PU with a supervised learning mechanism to learn the general code covariance matrix D by the supervised covariance rule and an unsupervised learning mechanism to learn the general code deviation accumulation vector C by the unsupervised accumulation rule. The adjusted C and D are delayed for one unit of time or one numbering before being stored in the model synapse. The PU in  FIG. 23  is called a supervised PU (SPU). 
     A network of at least one PU is called a low-order model (LOM) of biological neural networks. The vector v t  input to a PU in a layer contains not only feedforwarded components from outputs of model spiking neurons in the lower layers but also feedbacked components from outputs of model spiking neurons in the same or higher layers. Feedbacked components are delayed for at least one unit of time to ensure stability. Feedforwarded components may come from more than one layer preceding the layer in which the PU belongs to. 
     5.5.7 Creating a Vocabulary by Unsupervised Covariance Learning 
     Pseudo-random binary digit generation performed by the R model spiking neurons in a PU (processing unit) is indispensable in making the unsupervised covariance rule work for the PU. Let us now see how a “vocabulary” is created by unsupervised covariance rule for the PU: If a feature subvector v τ  or a slightly different version of it has not been learned by the PU, and CM({hacek over (v)} τ −&lt;{hacek over (v)} τ &gt;)=0; then d τ /c τ  is set equal to 0 and p τ =(½)I, where I=1 1 . . . 1. The R model spiking neurons use this empirical probability vector to generate a purely random label r τ . Once this r τ  and the output vector {hacek over (v)} τ  have been learned and stored in C and D, if v τ  is input to the PU for a second time, then u τ =r τ  with probability 1, and one more copy of the pair (v τ , r τ ) is included in C and D. 
     If an input vector v τ  or a slightly different version of it has been learned by a PU with different labels for different numbers of times, then y τ =d τk /c τ ≠0 and p τ ≠(½)I. Since v τ  may contain different parts from different causes and are assigned different labels in different rounds of unsupervised learning, p τ  may not be a binary vector. For example, assume that two labels, r τ   1  and r τ   2  of the same input vector v τ  have been learned with relative frequencies, 0.7 and 0.3, respectively. Then in response to v τ , each component of u τ  that is output from the PU is equal to r τ   1  with probability 0.7 and is equal to r τ   2  with probability with probability 0.3. Since these two labels may have common components, the point estimate of the label resembles r τ   1  with a probability of greater than 70% and resembles r τ   2  with a probability of greater than 30%. 
     5.6 Controlling Cluster Sizes 
     The set of vectors that, as input vectors to a PU, will retrieve (or generate) the same label from the PU is called a cluster. The length of the time interval over which the time average &lt;u t &gt; is taken of the PU&#39;s output vector u t =v{p t }can be selected to control the cluster sizes. 
     EXAMPLE 4 
     The Gray codes of two adjacent integers differ by one component. For example, the Gray codes of the integers, 0 to 15, are, respectively, 0000, 0001, 0011, 0010, 0110, 0111, 0101, 0100, 1100, 1101, 1111, 1110, 1010, 1011, 1001, 1000. Let v t , t=1, 2, . . . , 16, be these 16 codes in the same order. For example, v 1 =[0 0 0 0]′, v 2 =[0 0 0 1]′ and so on. Repeating the 16 codes, an infinite sequence of vectors x t , t=1, 2, . . . , is available for learning. Let us use a PU with 10 model spiking neurons and 1 model nonspiking neuron to learn v t , t=1, 2, . . . , 16, in the given order without a supervisor. Assume that Ψ=J=1 in (39) and (32) and that λ=Λ=1 in (12) and (21). 
     Because J=1 in M, an unsupervised learning rule generalizes, if necessary, on three binary digits of each v t . Since two consecutive codes differ by only one bit, the unsupervised correlation rule in THPAM (James Ting-Ho Lo, Functional Model of Biological Neural Networks,  Cognitive Neurodynamics , Vol. 4, Issue 4, pp. 295-313, November 2010) learns all the 16 codes (i.e., v t ) with one single label and thus puts them in one single cluster. Note that components 0 in v t  are replaced with −1 before the unsupervised correlation rule in THPAM can be applied. 
     Let us now use the unsupervised covariance rule to learn the same sequence of codes v t . To initiate the learning, we set &lt;v{p t }&gt;=I/2 for t=1, 0, −1, −2, . . . , −∞ and set C=0 and D=0 at t=0. For simplification, we assume &lt;{hacek over (v)} t &gt;=I/2 for all t. Note that the time averages &lt;v{p t }&gt; and &lt;{hacek over (v)} t &gt; may be taken over different time lengths. 
     EXAMPLE 4a 
     Assume that the time average &lt;v{p t }&gt; is taken over 100,000 time units, namely &lt;v{p t }&gt;=Σ Σ=−100,000+t+1   t  v{p τ }/100,000. Under this assumption, &lt;v{p t }&gt; is virtually ½ for t=1, 2, . . . , 16. For notational simplicity, we take &lt;v{p t }&gt; to be ½ for t=1, 2, . . . , 16 in the following. 
     Retrieval at t=1: c 1 =0, d 1 =0, p 1 =½ and v{p 1i } is purely random (i.e., v{p 1i }=1 with probability ½ for i=1, . . . , 10). 
     Learning at t=1: C=½({hacek over (v)} 1 −I/2)′ and D=(v{p 1 }−I/2)({hacek over (v)} 1 −I/2)′. Note that v{p 1 } is a binary vector. 
     Retrieval at t=2: c 2 =1, d 2 =2(v{p 1 }−I/2), p 2 =v{p 1 }, and v{p 2 }=v{p 1 }, because v{p 1 } is a binary vector. v 2  is assigned the same label v{p 1 } as v 1 , and v 1  and v 2  are put in the same cluster. 
     Learning at t=2: C=½Σ t=1   2 ({hacek over (v)} t −I/2)′ and D=Σ t=1   2 (v{p t }−I/2)({hacek over (v)} t −I/2)′. 
     Retrieval at t=3: c 3 =1, d 3 =2(v{p 2 }−I/2), p 3 =v{p 2 }, v{p 3 }=v{p 2 }. v 3  is assigned the same label v{p 1 } as v 2 , and v 1 , v 2  and v 3  are put in the same cluster. 
     Learning at t=3: C=½Σ t=1   3 ({hacek over (v)} t −I/2)′ and D=Σ t=1   3 (v{p t }−I/2)({hacek over (v)} t −I/2)′ 
     Retrieval at t=4: c 4 =1, d 4 =2(v{p 3 }−I/2), p 4 =v{p 3 }, and v{p 4 }=v{p 3 }. v 4  is assigned the same label v{p 3 } as v 3 , and v 1 , . . . , v 4  are put in the same cluster. 
     Continuing in this manner, all the 16 codes v t  are assigned the same label v{p 1 } and thus put in the same cluster. 
     EXAMPLE 4b 
     Assume that the time average &lt;v{p t }&gt; is taken over 1 time unit, —mely &lt;v{p t }&gt;=v{p t }. 
     Retrieval at t=1: c 1 =0, d 1 =0, p 1 =I/2 and v{p 1 } is purely random (i.e., v{p 1i }=1 with probability ½ for i=1, . . . , 10). 
     Learning at t=1: C=½({hacek over (v)} 1 −I/2)′ and D=(v{p 1 }−&lt;v{p 1 }&gt;)({hacek over (v)} 1 −I/2)′=0. 
     Retrieval at t=2: c 2 =1, d 2 =0, p 2 =I/2 and v{p 2 } is purely random. 
     Learning at t=2: C=½Σ t=1   2 ({hacek over (v)} t −I/2)′ and D=Σ t=1   2 (v{p t }−{p t })({hacek over (v)} t −I/2)′=0. 
     Retrieval at t=3: c 3 =1, d 3 =0, p 3 =I/2 and the label v{p 3 } is purely random. 
     Learning at t=3: C=½Σ t=1   3 ({hacek over (v)} t −I/2)′ and D=Σ t=1,3 (v{p t  }−v{p t−1 })({hacek over (v)} t −I/2)′=0. 
     Retrieval at t=4: c 4 =1, d 4 =0, p 4 =I/2 and the label v{p 4 } is purely random. 
     Continuing in this manner, the 16 codes v t  are each assigned a purely random label. Since there are 10 model spiking neurons and thus each label has 10 entries, the chance is that the 16 codes are assigned with different labels. Note that D remains to be 0 and thus no knowledge is learned. 
     EXAMPLE 4c 
     Assume that the time average &lt;v{p t }&gt; is taken over n time units, namely &lt;v{p t }&gt;=Σ i=0   n−1 v{p t−i }/n. 
     Retrieval at t=1: c 1 =0, d 1 =0, p 1 =I/2 and v{p 1 } is purely random (i.e., v{p 1i }=1 with probability ½ for i=1, . . . , 10). 
     Learning at t=1: C 1 =½({hacek over (v)} 1 −I/2)′ and D 1 =(v{p 1 }−(v{p 1 }+(n−1)I/2)/n)({hacek over (v)} 1 −I/2)′. 
     Retrieval at t=2: c 2 =1, d 2 =2(v{p 1 }−(v{p 1 }+(n−1)I/2)/n), p 2 =v{p 1 }−(v{p 1 }+(n−1)I/2/n)+I/2. 
     Learning at t=2: C 2 =½Σ t=1   2 ({hacek over (v)} t −½)′ and D 2 =D 1 +(v{p 2 }−(Σ τ=2   1 v{p τ }+(n−2)I/2)/n)({hacek over (v)} 2 −½I)′. 
     Continuing in this manner for t≦n, 
     Retrieval at t: c=1, d t =2(v{p t−1 }−(Σ τ=t−1   1 v{p τ }+(n−t+1)I/2)/n), p t =v{p t−1 }−(Σ τ=t−1   1 v{p τ }+(n−t+1)I/2)/n+½I 
     Learning at t: C t =½Σ τ=1   t ({hacek over (v)} τ −½)′ and D t =D t−1 +(v{p t }−(Σ τ=t   1 v{p τ }+(n−t)I/2)/n)({hacek over (v)} t −½)′; 
     and for t&gt;n, 
     Retrieval at t: c t =1, d t =2(v{p t−1 }−Σ τ=t−1   t−n v{p τ }/n), p t =v{p t−1 }−Σ τ=t−1   t−n v{p τ }/n+½I. 
     Learning at t: C t =½Σ τ=1   t ({hacek over (v)} τ −½)′ and D t =D t−1 +(v{p t }−Σ τ=t   t−n+1 v{p τ }/n)({hacek over (v)} t −½)′. 
     Note that lim n→∞ p t =v{p t−1 }. This is consistent with Example 2a. Two observations can be made from these formulas:
         1. Given a fixed t: the greater n is, the closer p t  is to v{p t−1 }, the closer v{p t } is from v{p t−1 }, the more likely v t  is assigned the same label v{p t−1 } as v t−1 , or the more likely v t  and v t−1  are put in the same cluster.   2. Given a fixed n: for t≦n, the deviation of (Σ τ=−1   1 v{p τ }+(n−t+1)I/2)/n from ½I increases as t increases, and hence the deviation of p t  from v{p t−1 }increases. Consequently, the chance that v t  is assigned v{p t−1 } as than v t−1  decreases as t increases, and the chance that v t  and v t−1  are put in the same cluster decreases.       

     EXAMPLE 5 
     Let us now use the unsupervised covariance rule to learn the sequence of identical codes v t (t)=[1 1 1]′, t=1, . . . , 16. 
     Assume that the time average &lt;v{p t }&gt; is taken over n time units, namely &lt;v{p t }&gt;=Σ i=0   n−1 v{p t−i }/n. 
     Retrieval at t=1: c 1 =0, d 1 =0, p 1 =I/2 and v{p 1 } is purely random (i.e., v{p 1i }=1 with probability ½ for i=1, . . . , 10). 
     Learning at t=1: C 1 =½({hacek over (v)} 1 −I/2)′ and D 1 =(v{p 1 }−(v{p 1 }+(n−1)I/2)/n)({hacek over (v)} 1 −I/2)′. 
     Retrieval at t=2: c 2 =2, d 2 =4(v{p 1 }−(v{p 1 }+(n−1)I/2)/n), p 2 =v{p 1 }−(v{p 1 }+(n−1)I/2)/n+I/2. 
     Learning at t=2: C 2 =½Σ t=1   2 ({hacek over (v)} t −½)′ and D 2 =D 1 +(v{p 2 }−(Σ τ=2   1 {p τ }+(n−2)I/2)/n)·({hacek over (v)} 2 −½I)′. 
     Retrieval at t=3: c 3 =4, d 3 =d 2 +4(v{p 2 }−(Σ τ=2   1 v{p τ }+(n−2)I/2)/n), p 3 =(p 2 +v{p 2 }−(Σ τ=2   1 v{p τ }+(n−2)I/2)/n+½I)/2 
     Learning at t=3: C 3 =½Σ t=1   3 ({hacek over (v)} t −½)′ and D 3 =D 2 +(v{p 3 }−(Σ τ=2   1 v{p}−(n−2)I/2)/n)·({hacek over (v)} 3 −½)′ 
     Retrieval at t=4: 
     
       
         
           
             
               
                 c 
                 4 
               
               = 
               6 
             
             , 
             
               
                 d 
                 4 
               
               = 
               
                 
                   d 
                   3 
                 
                 + 
                 
                   4 
                   ⁢ 
                   
                     ( 
                     
                       
                         v 
                         ⁢ 
                         
                           { 
                           
                             p 
                             3 
                           
                           } 
                         
                       
                       - 
                       
                         
                           ( 
                           
                             
                               
                                 ∑ 
                                 
                                   τ 
                                   = 
                                   2 
                                 
                                 1 
                               
                               ⁢ 
                               
                                 v 
                                 ⁢ 
                                 
                                   { 
                                   
                                     p 
                                     τ 
                                   
                                   } 
                                 
                               
                             
                             - 
                             
                               
                                 ( 
                                 
                                   n 
                                   - 
                                   2 
                                 
                                 ) 
                               
                               ⁢ 
                               
                                 I 
                                 / 
                                 2 
                               
                             
                           
                           ) 
                         
                         / 
                         n 
                       
                     
                     ) 
                   
                 
               
             
             , 
             
               
                 p 
                 4 
               
               = 
               
                 
                   ( 
                   
                     
                       
                         ∑ 
                         
                           τ 
                           = 
                           2 
                         
                         3 
                       
                       ⁢ 
                       
                         p 
                         τ 
                       
                     
                     + 
                     
                       v 
                       ⁢ 
                       
                         { 
                         
                           p 
                           3 
                         
                         } 
                       
                     
                     - 
                     
                       
                         1 
                         n 
                       
                       ⁢ 
                       
                         
                           ∑ 
                           
                             τ 
                             = 
                             2 
                           
                           1 
                         
                         ⁢ 
                         
                           v 
                           ⁢ 
                           
                             { 
                             
                               p 
                               τ 
                             
                             } 
                           
                         
                       
                     
                     - 
                     
                       
                         
                           n 
                           - 
                           3 
                         
                         
                           2 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           n 
                         
                       
                       ⁢ 
                       I 
                     
                     + 
                     
                       
                         1 
                         2 
                       
                       ⁢ 
                       I 
                     
                   
                   ) 
                 
                 / 
                 3 
               
             
           
         
       
     
     Continuing in this manner for t≦n, 
     Retrieval at t: c t =2(t−1), d t =4(v{p t−1 }−(Σ τ=t−1   1 v{p τ }+(n−+1)I/2)/n), p t =[Σ τ=2   t−1 p τ +v{p t−1 }−(Σ τ=t−1   1 v{p τ }+(n−t+1)I/2)/n+]/(t−1). 
     Learning at t: C t =½Σ τ=1   t ({hacek over (v)} τ −½)′ and D t =D t−1 +(v{p t }−(Σ τ=t   1 v{p τ }+(n−t)I/2)/n)·({hacek over (v)} t −½)′; 
     and for t&gt;n, 
     Retrieval at t: c t =2(t−1), d=4(v{p t−1 }−Σ τ=t−1   t−n v{p τ }/n), p t =[Σ τ=2   t−1 p τ v{p t−1 }−Σ τ=t−1   t−n v{p t }/n+½I]/(t−1). 
     Learning at t: C t =½Σ τ=1   t ({hacek over (v)} τ −½)′ and D t =D t−1 +(v{p t }−Σ τ=t   t−n+1 v{p τ }/n)({hacek over (v)} t −½). 
     The two observations are valid here. However, p t =(Σ τ=2   t−1 p τ +v{p t−1 }−(Σ τ=t−1   1 v{p τ }+(n−t+1)I/2)/n+½I)/(t−1) shows that the change from p t−1  to p t  for learning identical codes is much smaller than that for learning the sequence of Gray codes in EXAMPLE 4c. This means that for some n, the identical codes can be put in the same cluster, and yet the 16 Gray codes are put in more than one cluster. In fact, it is acceptable by the LOM that the identical codes are assigned different labels as long as the labels differ by a couple of bits, because these labels are inputs to other PUs, which have generalization capability. 
     The Gray codes in the given learning order are an extreme example that does not exist in the real-world. The analysis in the above examples show that sizes of the clusters of the Gray codes in the given learning order can be controlled by selecting n appropriately. In the real world, repeated learning is a frequently encountered situation. Therefore, we should select n large enough to guarantee that the chance for putting the same input vector in repeated learning into two different clusters is negligible. Only under the condition this requirement is fulfilled, we select n as small as possible to prevent a cluster to overgrow. Of course, the maximum size of a cluster depends on the application of the LOM. (In fact, having more clusters does little harm to the processing of the LOM. It is like having multiple words for similar things.) The formulas for determining p t  in the above examples can be generalized easily for any dimensionality of the input vector and any number of presentations for similar input vectors (i.e., those with a small number of different bits) or identical input vectors. Generalizations of the formulas in Example 5 are especially useful for determining the smallest n required for not assigning different labels to identical input vectors. 
     5.7 Learning Masking Factors for Unsupervised Learning of Vectors with Smaller Differences 
     In supervised learning by an SPU (supervised processing unit), the label is not generated by the SPU, but is provided from outside the SPU (or the LOM). If different labels are provided for input vectors with small differences, the SPU learns the different labels and can distinguish said input vectors with small differences thereafter. In unsupervised learning by an UPU, if the UPU is required to be able to distinguish learned input vectors with small differences and to remain to be able to recognize unlearned input vectors that have larger differences with (or are more different from) learned input vectors, other masking matrices that are more strict (or less tolerant) than the masking matrices are used. These other masking matrices have a smaller J in (32) than the masking matrices do and are called the learning masking matrices. The learning masking matrices, denoted by M # (ψ), ψ=1, . . . , Ψ, form a general masking matrix called the general learning masking matrix
 
 M   # =diag[ M   # (1)  M   # (2) . . .  M   # (Ψ)]  (59)
 
It is a diagonal matrix. The diagonal entries are numbered consecutively from 1 to dim {hacek over (v)} τ =Σ ψ=1   Ψ 2 dim v     τ     (ψ) , where {hacek over (v)} τ  is the general neuronal code, whose entries are also numbered consecutively from 1 to dim v τ . The diagonal entries M jj   # =1, . . . , dim {hacek over (v)} τ , of M #  are called learning masking factors.
 
     An UPU (unsupervised processing unit) with learning masking matrices M # (ψ), ψ=1, . . . , Ψ, still generates the binary vector v{p τ } that is output from the UPU in response to the vector v τ  input to the UPU using its masking matrices M(ψ), ψ=1, . . . , Ψ, as before. However, when the UPU learns, the learning masking matrices M # (ψ), ψ=1, . . . , Ψ, are used instead to generate an estimate v{p τ   # } of the label of v t , and the general code covariance matrix D is adjusted by
 
 D (ψ)←λ D (ψ)+Λ( v{p   τ   #   }−&lt;v{p   τ   # }&gt;)( {hacek over (v)}   τ   −&lt;{hacek over (v)}   τ &gt;)′  (60)
 
More specifically, v{p τ   # } is generated as follows:
 
                     [     c     τ   ⁢           ⁢   j     #     ]     =     [       M   jj   #     ⁢       C   j     ⁡     (         v   ⋓     τ     -     〈       v   ⋓     τ     〉       )         ]             (   61   )                 [     d     τ   ⁢           ⁢   kj     #     ]     =     [       M   jj   #     ⁢       D   kj     ⁡     (         v   ⋓     τ     -     〈       v   ⋓     τ     〉       )         ]             (   62   )                 c   τ   #     =       ∑   j`     ⁢     c     τ   ⁢           ⁢   j     #               (   63   )                 p     τ   ⁢           ⁢   k     #     =       (         ∑   j`     ⁢     d     τ   ⁢           ⁢   kj         +     c     τ   ⁢             #       )     /     (     2   ⁢           ⁢     c   τ   #       )               (   64   )               
v{p τk   # } is a pseudorandom number generated in accordance with the empirical probability distribution p τk   # , and then v{p τ   # }=[v{p τ1   # } . . . v{p τR   # }]′, where R is the number of model spiking neurons in the UPU. A typical UPU with learning masking factors M jj   # , j=1, . . . , dim {hacek over (v)} τ  is illustrated in  FIG. 24 .
 
5.8 Spike Trains for Each Exogenous Feature Vector
 
     Recall that a binary vector u t  output from a PU (processing unit), is obtained by a pseudo-random number generator using the empirical probability distribution p t  of the label r t  of a vector v t  input to the PU. Components of such binary vectors u t  with uncertainty form vectors input to other or the same PUs through feedforward or feedback connections. Upon receiving a vector with uncertainty, a PU uses masking matrices to suppress or “filter out” some components so that the remaining components are consistent with those stored in the code covariance matrices. (Masking matrices are described in Section 5.4.) 
     However, there is a chance for the pseudo-random number generator to generate a binary vector u t  that is such an outlier for the empirical probability distribution p t  that causes undesirable effects on learning and retrieving of PUs receiving components of u t  in spite of masking matrices. To minimize such undesirable effects and to represent the empirical probabilities involved in the PUs, the LOM usually completes a certain number of rounds of retrieving and learning for each exogenous vector v t   ex  input to the LOM so that many pseudorandom versions of u t  are generated and learned by each PU for the same v t   ex . 
     To have ζ rounds of retrieving and learning for each exogenous vector v t   ex , the exogenous vector must be held constant for ζ units of time. In other words, the exogenous vector v t   ex  is presented to the LOM with a different time scale. More specifically, v t   ex  changes at t=iζ+1, i=0, 1, 2, . . . . Consequently, a PU generates a sequence of binary vectors denoted by u t , t=iζ+j, j=1, 2, . . . , ζ, for each exogenous feature vector v t   ex , which remains constant for t=iζ+j, j=1, 2, . . . , ζ. More specifically, once a new exogenous vector v iζ+1   ex  is presented, v iζ+1   ex =v iζ+1   ex  for j=2, . . . , ζ. The sequence u t , t=iζ+j, j=1, 2, . . . , ζ, output from the PU consists of R spike trains, each having ζ spikes during the period of time. 
     Alternatively, if each exogenous vector v t   ex  is held constant for 1 unit of time, then the PU generates a sequence of ζ binary vectors within the 1 unit of time; namely the PU generates one binary vector every 1/ζ unit of time. 
     5.9 Multilayer Networks of Processing Units with Feedbacks 
     The low-order model (LOM) of biological neural networks described in the present invention disclosure is a network of processing units (PUs) with or without time-delayed feedbacks. An LOM that is a multilayer network of PUs is described in this Subsection. 
     An external vector input to the LOM is called an exogenous feature vector, and a vector input to a layer of PUs is called a feature vector. A feature vector input to a layer usually contains not only feedforwarded outputs from the PUs in preceding layers but also feedbacked outputs from the PUs in the same or higher layers with time delays. A feature vector may contain components from an exogenous feature vector. For simplicity, we assume that the exogenous feature vector is only input to layer 1 and is thus a subvector of a feature vector input to layer 1. All these feature vectors and output vectors over time usually form spike trains. 
     A subvector of a feature vector that is input to a PU is called a feature subvector. Trace the feedforward connections backward from neurons of a PU to a subvector of a feature vector input to a layer or the exogenous feature vector. This subvector is called the receptive field of the PU in the feature vector input to the layer or the exogenous feature vector. All the measurements that affect the receptive field in the exogenous feature vector are also called the receptive field of the PU in the measurements. For example, let the measurements taken by the pixels in a digital camera be input to a model of a retina, and let the output vector from the model of the retina be used as the exogenous feature vector of the LOM. Then the measurements or the pixels with those measurements that affect the receptive field of a PU in the exogenous feature vector are called the receptive field of the PU in the measurements. 
     The collection of neurons in layer l−i, i=1, 2, . . . , that have a direct feedforward connection (without going through another neuron) to a neuron in a PU in layer l and the unit-time delay devices that hold a feedback that is directly input (without going through another unit-time delay device) to the same PU are called the immediate receptive field of the PU. 
     The feature vector input to layer l at time or numbering τ is denoted by v τ   l−1 , and the output from layer l at τ is denoted by v{p τ   l }. The feature vector v τ   l−1  consists of components of the feedforwarded vector v{p τ   l−1 } and feedbacked vector v{p τ−z(k)   l+k }feedbacked from the same layer l and higher layers l+k and after z(k) time units of delay for k=0, 1, . . . , where z(k) is a function of k. 
       FIG. 25  shows layer l and layer l+2 of PUs of the LOM. In  FIG. 25 , same-layer feedback connections with one unit-time delay device from layer l to itself and 2-layer feedback connections with 5 unit-time delay devices from layer l+2 to layer l are shown. The box under layer l of PUs does not model a biological entity, but illustrates that the feature vector input to layer l comprises feedforward vector v{p τ   l−1 }, the same layer feedback v{p τ−1   l }, and the 2-layer feedback v{p τ−5   l+2 }. Note that the symbols, v{p τ   l−1 }, v{p τ−1   l } and v{p τ−5   l+2 }, include all the components output from layer l−1, layer l+2, and layer l for simplicity in showing the feedback structure. For an application, we do not have to include all of them, but may select some of these components as inputs to layer l. 
     Once an exogenous feature vector is received by the LOM, the PUs perform functions of retrieving and/or learning from layer to layer starting with layer l, the lowest layer. After the PUs in the highest layer, layer L, complete performing their functions, the LOM is said to have completed one round of retrievings and/or learnings. Each exogenous feature vector is held constant for a certain number ζ of time units, during which the LOM completes (of retrievings and/or learnings. 
     We note that retrieving and learning by a PU are performed locally, meaning that only the feature subvector input to the PU and its label are involved in the processing by the PU. Causes in patterns, temporal or spatial, usually form a hierarchy. Examples: (a). Phonemes, words, phrases, sentences, and paragraphs in speech. (b). Musical notes, intervals, melodic phrases, and songs in music. (c). Bananas, apples, peaches, salt shaker, pepper shaker, fruit basket, condiment tray, table, refrigerator, water sink, and kitchen in a house. Note that although Example (c) is a spatial hierarchy, when one looks around in the kitchen, the images scanned and received by the person&#39;s retina form a temporal hierarchy. 
     The higher a layer in the LOM is, the higher in the hierarchy the causes the PUs in the layer handle, and the more time it takes for the causes to form and be detected and recognized by the PUs. Therefore, the number of unit-time delay devices on a feedback connection is a monotone increasing function z(k) of k, which are defined above. This requirement is consistent with the workings in a biological neural network in the cortex. Note that it takes time (i) for biological PUs to process feature subvector, (ii) for spikes to travel along feedforward neural fibers from a layer to the next layer, and (iii) for spikes to travel along feedback neural fibers from a layer to the same or a lower-numbered layer. Note also the subscripts of the input vector v τ   l−1  and output vector v{p τ   l } of all layers l are the same, indicating the same exogenous feature vector v τ   ex  is processed or propagated in all layers. The common subscript τ does not represent the time that the signals in the biological network reach or processed by its layers. However, a feedback v τ−z(k)   l+k  from layer l+k to layer l for inclusion in v τ   l−1  must have a delay z(k) that reflects the sum of the times taken for (i), (ii) and (iii) from the input terminals of layer l back to the same input terminals. 
     For notational simplicity, the superscript l−1 in v t   l−1  and dependencies on l−1 or l in other symbols are sometimes suppressed in the following when no confusion is expected. 
     Let the sequence of feature vectors, v t   l , t=1, 2, . . . , be M-dimensional and thus v t   l =[v t1   l  . . . v tM   l ]′, whose components are ternary numbers. Let n l =[n 1   l  . . . n k   l ]′ be a subvector of [1 . . . M]′ such that n 1   l &lt; . . . &lt;n k   l . The subvector 
                 v   t   l     ⁡     (     n   l     )       :=       [           v     tn   1   l     l         …         v     tn   k   l     l           ]     ′           
of v t   l  is called a feature subvector of v t   l . n l  is called a feature subvector index (FSI), and v t   l (n) is said to be a feature subvector on the FSI n l  or have the FSI n l . Each UPU is associated with a fixed FSI n l  and denoted by UPU(n l ). Using these notations, the sequence of subvectors of v t   l−1 , t=1, 2, . . . , that is input to UPU(n l ) is v t   l−1 (n l ), t=1, 2, . . . . The FSI n l  of a UPU usually has subvectors, n l (ψ), ψ=1, . . . , Ψ, on which subvectors v t   l−1 (n l (ψ)) of v t   l−1 (n l ) are separately processed by the neuronal encoders in UPU(n l ) at first. The subvectors, n l (ψ), ψ=1, . . . , Ψ, are not necessarily disjoint, but are all inclusive in the sense that every component of n l  is included in at least one of the subvectors n l (ψ). Moreover, the components of n l (ψ) are usually randomly selected from those of n l .
 
     The UPUs in layer l have FSIs (feature subvector indices) denoted by 1 l , 2 l , . . . , N l . Upon receiving a feature vector v τ   l−1  by layer l, the feature subvectors, v τ   l−1 (1 l ), v τ   l−1 (2 l ), . . . , v τ   l−1 (N l ), are formed and processed by UPU(1 l ), UPU(2 l ), . . . , and UPU(N l ) to compute the empirical probabilities, p τ   l (1 l ), p τ   l (2 l ), . . . , p τ   l (N l ), first and then generate the pseudo-random ternary vectors v{p τ   l (1 l )}, v{p τ   l (2 l )}, . . . , v{p τ   l (N l )}. Recall that p τ   l (n l ) is the empirical probability of the label r τ (n l ) of v τ   l (n l ), and v{p τ   l (n l )} is the output of UPU(n l ) based on p τ   l (n l ). These empirical probabilities and UPU outputs are grouped into the empirical probability distribution p τ   l  and the output vector v{p τ   l } of layer l as follows:
 
 p   τ   l   =[p   τ   l ′(1 l )  p   τ   l ′(2 l ) . . .  p   τ   l ′( N   l )]′
 
 v{p   τ   l   }=[v′{p   τ   l (1 l )}  v′{p   τ   l (2 l )} . . .  v′{p   τ   l ( N   l )}]′
 
     The components of a feature vector v τ   l−1  input to layer l at time (or numbering) τ comprise components of binary vectors generated by UPUs in layer l−1 and those generated at previous times by UPUs in the same layer l or UPUs in higher layers with layer numberings l+k for some positive integers k. The time delays may be of different durations. 
     For illustration, an example is given in the following: 
     EXAMPLE 4 
     Let us set the number z(k) of unit-time delay devices equal to 4(k+1) for k=0, . . . , 7; and set the number ζ time units that each exogenous feature vector is held constant equal to 16. 
     For k=1 and z(1)=8, the first 8 feedbacks used by layer l in processing an exogenous feature vector v t   ex  are output from layer l+1 in response to v t−1   ex , which provides temporally and spatially associated information from the preceding exogenous feature vector v t−1   ex . 
     For k=5 and z(5)=24, the first 8 feedbacks used by layer l in processing an exogenous feature vector v t   ex  are output from layer l+5 in response to v t−2   ex ; and the next 8 feedbacks are output from layer l+5 in response to v t−1   ex , which provides temporally and spatially associated information from the preceding exogenous feature vectors, v t−2   ex  and v t−1   ex . 
     For k=8 and z(8)=36, the first 4 feedbacks used by layer l in processing an exogenous feature vector v t   ex  are output from layer l+8 in response to v t−3   ex ; and the next 12 feedbacks are output from layer l+8 in response to v t−2   ex , which provides temporally and spatially associated information from the preceding exogenous feature vectors, v t−3   ex  and v t−2   ex . 
     Note that the greater k, the larger the number of unit-delay devices, and the further back the feedbacked information is in processing the current exogenous feature vector v t   ex . Note also that the further back the feedbacked information, the less spatially but more temporally associative information is used in processing v t   ex . Moreover, given the same numbers of unit-delay devices on each feedback connection, if an exogenous feature vector is presented to the LOM for a larger number of time units, then more recent information and less further back information is used in processing v t   ex . This means more spatially associated information but less temporally associated information is brought back by the feedback connections and utilized by the LOM. 
     An example LOM with three layers of PUs and feedbacks is shown in its entirety in  FIG. 26 . There are three types of feedback connection: same-layer feedbacks, one-layer feedbacks and two-layer feedbacks. The numbers of unit-time delay devices on the feedback connections are not specified for simplicity. The second delay box on a feedback connection represents an additional delay. 
     5.10 The Clusterer and Interpreter 
     An architecture of the LOM that makes for an effective artificial neural network comprises a cluster and an interpreter, which are described in the present Subsection. The clusterer in the LOM is a network of unsupervised processing units (UPUs) with or without time-delayed feedbacks. If the cluster has no feedback connection, it is an unsupervised ANN (artificial neural network) that clusters spatial data. If the clusterer has feedback connections, it is an unsupervised ANN that clusters spatial and/or temporal data. 
     The vectors output from an UPU in the clusterer in the LOM are point estimates v{p τ } of the labels of clusters of vectors v τ  input to the UPU. These labels form a vocabulary for the UPU. To interpret such a label or its point estimate v{p τ }generated by UPU into a word, a few words, a sentence or a few sentences using the language of the human user, say English, an SPU is used. 
     Once an exogenous feature vector is received by a clusterer in the LOM, the UPUs perform functions of retrieving and/or learning from layer to layer starting with layer l (i.e., the lowest layer). After the UPUs in the highest layer, layer L, complete performing their functions, the clusterer in the LOM is said to have completed one round of retrievings and/or learnings (or memory adjustments). For each exogenous feature vector, the clusterer in the LOM will continue to complete a certain number of rounds of retrievings and/or learnings. 
       FIG. 27  shows layer l and layer l+2 of UPUs of a clusterer with feedback connections in the LOM, wherein same-layer feedback connections have one unit-time delay device and 2-layer feedback connections have 5 unit-time delay devices from layer l+2 to layer l. The box under layer l of PUs illustrates that the feature vector input to layer l comprises feedforward vector v{p τ   l−1 }, the same layer feedback v{p τ−1   l }, and the 2-layer feedback v{p τ−5   l+2 }. 
       FIG. 28  shows an LOM with a clusterer and an interpreter. The clusterer in the LOM in  FIG. 27  is again shown in  FIG. 28 . The connections and delay devices in the clusterer are not shown for clarity in  FIG. 28 . The three UPUs in the lowest layer are not connected to a SPU, but each of the three PUs in the second and third layers is. UPU(1 2 ) and UPU(2 2 ) in the second layer have feedforward connections to SPU(1 3 ) and SPU(2 3 ) respectively, and UPU(1 3 ) in the third layer has feedforward connections to SPU(1 4 ). 
     The labels, r τ (1 3 ), r τ (2 3 ) and r τ (1 4 ), which are used for supervised learning by the synapses in SPU(1 3 ), SPU(2 3 ) and SPU(1 4 ) respectively are provided by the human trainer of the artificial neural network. For spatial pattern recognition, r τ (n l ) is obtained by tracing the feedforward connections from SPU(n l ) all the way down to input terminals that receive the exogenous feature vector and if necessary further down to the sensor elements such as the pixels in the CCD of a camera. These input terminals or sensory elements are called the receptive field of SPU(n l ) and UPU(n l−1 ) in the measurements. If the human trainer sees a distinct cause such as an apple or John Doe&#39;s face in the receptive field, he/she assigns the bipolar binary code of a word or a phrase such as “apple” or “John Does&#39;s face” to the bipolar binary label r τ (n l ). Since supervised covariance learning is very easy and the output vector of SPU(n l ), which has the same number of bits as r τ (n l ), will not be used as inputs to another PU, the total number of bits in r τ (n l ) can be made as large as needed to hold the longest code for the receptive field. Shorter codes can be made longer by including zeros at the beginning or ending of the shorter codes. 
     The receptive field of an SPU branching out from a higher layer is larger than that of an SPU branching out from a lower layer. The cause in a larger receptive field usually requires a longer bipolar binary code to represent. For example, there are only 26 letters, more than 10,000 commonly used words, and millions of commonly used sentences; which need codes of 5 bits, 14 bits, and 30 bits, respectively. To avoid using a look-up table to translate codes into English word or sentences, we can simply type the letters, the words or the sentences to assign their ASCII codes to r τ (n l ). 
     To avoid an SPU using different codes for the same pattern or cause, we may use SPU(n l ) to retrieve its v{p τ   l (n l )}. If it is not recognizable by the human trainer, if it does not agree with the human trainer, or if the empirical probability distribution p τ   l (n 1 ) does not contain enough information; we assign a new r τ (n l ) to SPU(n l ). A measure of information contained in p τ   l (n l ) is 
               ξ   ⁡     (       p   τ     ⁡     (     n   l     )       )       =     1   -       4   R     ⁢       ∑     k   =   1     R     ⁢         p     τ   ⁢           ⁢   k     l     ⁡     (     n   l     )       ⁢       (     1   -       p     τ   ⁢           ⁢   k     l     ⁡     (     n   l     )         )     .                   
If p τk   l (n l )=0 or 1 for k=1, 2, . . . , R, then ξ(p τ   l (n l ))=1 and ξ(p τ (n l )) is maximized, meaning there is no uncertainty in p τ (n l ). If p τk (n l )=½ for k=1, 2, . . . , R, then ξ(p τ (n l ))=0 and ξ(p τ (n l )) is minimized, meaning there is no information in p τ (n l ).
 
     Recall that the clusterer of the LOM learns independently of the learning (or even existence) of SPUs in the LOM&#39;s interpreter. Whenever new information is acquired for creating a new handcrafted label for the output of an UPU with or without an SPU, a new SPU can be added and learn the new label. Therefore, there can be multiple SPUs for an UPU, providing multiple interpretations of the output from the UPUs. For example, if the head of a man shows up in the receptive field of an UPU, there may be 2 SPUs that output labels, “head of a man with long hair” and “head of Albert Einstein” in response to the vector output from the UPU. If it is later found that the photograph containing the head of Einstein was taken in 1945, a third SPU can be added that outputs the handcrafted label “An image of Albert Einstein in 1945” in response to the same vector output from the UPU. 
     5.11 Preprocessing Exogenous Feature Vectors for the LOM 
     If feature vectors α t   ex , t=1, 2, . . . , to be processed by an artificial neural network have components being real numbers, a and b, or near a and b, where a and b are not unipolar binary digits (i.e., 0 and 1), a preprocessor is used to convert the feature vectors into unipolar binary digits or values near them before processing by the LOM. 
     Given a variable α whose value is either a real number a or a real number b, it can be transformed into a variable v whose value is either 0 or 1, respectively, by the following function: 
             v   =       f   ⁡     (   α   )       =       α   -   a       b   -   a               
If α is near a or near b, then v is near ƒ(a)=0 or near ƒ(b)=1, respectively. Given a vector α=[α 1  α 2  . . . α m ]′ whose components are real numbers, a and b, the vector α can be transformed into a binary vector v by the function
 
 v =ƒ(α)=[ƒ(α 1 ) ƒ(α 2 ) . . . ƒ(α m )]′
 
If the component of α is near a or near b, then the corresponding components of v are near ƒ(a) or near ƒ(b), respectively. Notice that the symbol ƒ denotes both a real-valued function of a real-valued variable α and a vector-valued function of a vector variable α with real components.
 
     If the exogenous feature vectors α t   ex , t=1, 2, . . . , to be processed by an artificial neural network have components being real numbers, a and b, or near a and near b, the vectors α t   ex , t=1, 2, . . . , can be converted into binary or nearly binary vectors v t   ex , t=1, 2, . . . , by v t   ex =ƒ(α t   ex ), t=1, 2, . . . The low-order model (LOM) described hereinabove can then be applied. 
     5.12 the CIPAM—an ANN Mathematical Equivalent to the LOM 
     The LOM is a model of biological neural networks, wherein most biological neurons communicate with spike trains. Spike trains are usually modeled as sequences of 1&#39;s and 0&#39;s. The feature vectors, neuronal codes, neuron inputs, and neuron outputs in the LOM are all (unipolar) binary vectors, whose components are 1&#39;s and 0&#39;s. On the other hand, the corresponding quantities in the functional model of biological neural networks, the THPAM, are bipolar binary vectors, whose components are −1&#39;s and 1&#39;s. The development of the LOM motivated a re-examination of the THPAM to improve the THPAM and eliminate its shortcomings. 
     This re-examination resulted in still another functional model of biological neural networks, namely the Clustering Interpreting Probabilistic Associative Memory (CIPAM), reported in James Ting-Ho Lo, A Cortex-Like Learning Machine for Temporal Hierarchical Pattern Clustering, Detection, and Recognition,  Neurocomputing , Vol. 78, pp. 89-103, 2012; also available online with DOI: 10.1016/j.neucom.2011.04.046, which is incorporated into the present invention disclosure by reference. As in the THPAM, the feature vectors, neuronal codes, neuron inputs, and neuron outputs in the CIPAM are all bipolar binary vectors, whose components are −1&#39;s and 1&#39;s. Among differences between the CIPAM and the THPAM, a main difference is that the former uses the unsupervised and supervised covariance rule to learn code covariance matrices and the latter uses the unsupervised and supervised correlation rule to learn expansion correlation matrices. This main difference necessitates some other differences including that in their retrieving methods; namely the former uses decovariance rule and the latter decorrelation rule. For applications, the LOM, THPAM and CIPAM are all artificial neural networks (ANNs). 
     The high degree of similarity between the LOM and the CIPAM motivated a study of their relation. This study led to the discovery that LOM and the CIPAM together with their corresponding components can be mathematically transformed into each other by the affine transformation and its inverse:
 
 v =ƒ( x )=½(1 −x )  (65)
 
 x=ƒ   −1 ( v )=1−2 v   (66)
 
where ƒ −1  denotes the inverse function of ƒ. Notice that ƒ(1)=0 and ƒ(−1)=1, and that ƒ −1 (0)=1 and ƒ −1 (1)=−1. Note ƒ(0)=½. Using the CIPAM to process vectors with components from {−1, 0, 1} is equivalent to using the LOM to process vectors with components from {1, ½, 0}. However, for simplicity, processing vectors whose components are from these ternary sets are not described in detail in the present invention disclosure. Those skilled in the art are not expected to have difficulty with extending the invention for processing such vectors.
 
     A bipolar binary digit x, whose value is 1 or −1 and a unipolar binary digit v, whose value is 0 or 1, can be transformed into each other, respectively by (65) and (66). Each component of the LOM and its corresponding component of the CIPAM can be transformed into each other by (65) and (66). From this viewpoint, the CIPAM and the LOM are mathematically equivalent artificial neural networks. The CIPAM has advantages similar to those of the LOM over the THPAM. As an artificial neural network, the CIPAM is computationally less expensive than the LOM. 
     In the operation of the CIPAM, the exogenous feature vectors and the feature vectors input to each layer of its neurons or PUs (processing units) are bipolar binary vectors whose components assume values from the bipolar binary set {−1, 1}. As an equivalent of the LOM, the CIPAM does not have the shortcomings of the PAM (i.e., THPAM) mentioned in Subsection 2 (e.g., the inability of the PAM&#39;s unsupervised correlation learning mechanism to prevent clusters from overgrowing under certain circumstances). On the other hand, because the CIPAM processes bipolar binary digits, {−1, 1}, it is computationally less expansive than the LOM, which processes (unipolar) binary digits, {0,1}. The CIPAM is briefly described through transforming the LOM into the CIPAM below. A detailed description is given in the paper by James Ting-Ho Lo, A Cortex-Like Learning Machine for Temporal Hierarchical Pattern Clustering, Detection, and Recognition,  Neurocomputing , DOI: 10.1016/j.neucom.2011.04.046, which is incorporated herein by reference. 
     By (65) and (66), given a bipolar binary variable x, whose value is 1 or −1, we can transform it into a (unipolar) binary variable v by v=ƒ(x)=½(1−x) and the other way around by x=ƒ −1 (v)=1−2v. Let v=ƒ(x) and u=ƒ(y). Recall that the neuronal node performs φ(v, u)=−2vu+v+u. By simple substitution,
 
φ(ƒ( x ),ƒ( y ))=½(1 −xy )
 
ƒ −1 (φ(ƒ( x ),ƒ( y )))= xy  
 
Therefore, this function xy is a representation of the neuronal node operation, where the inputs of the neuronal node are represented by x and y and its output is represented by xy. Note that if x and y are bipolar binary variables, xy is the NXOR (not-exclusive-or) function.
 
     Using this representation of the neuronal node operation, the equations (2)-(5) that generate the neuronal code (or encoding) {hacek over (v)} of an m-dimensional vector v are respectively replaced by the following equations for generating the representation {hacek over (x)} of {hacek over (v)}: 
                       x   ⋓     ⁡     (   1   )       =       [         1         x   1           ]     ′             (   67   )                         x   ⋓     ⁡     (     1   ,   2     )       =       ⁢     [             x   ⋓     ⁡     (   1   )               x   2     ⁢       x   ⋓     ⁡     (   1   )               ]                 =       ⁢     [         1         x   1           x   2             x   2     ⁢     x   1             ]                   (   68   )                   x   ⋓     ⁡     (     1   ,   …   ⁢           ,     k   +   1       )       =       ⁢       [               x   ⋓     ′     ⁡     (     1   ,   …   ⁢           ,   j     )               x     k   +   1       ⁢         x   ⋓     ′     ⁡     (     1   ,   …   ⁢           ,   k     )               ]     ′             (   69   )                 x   ⋓     =       x   ⋓     ⁡     (     1   ,   …   ⁢           ,   m     )               (   70   )               
This representation of the neuronal code is actually the orthogonal expansion of x discussed in the paper by James Ting-Ho Lo, Functional Model of Biological Neural Networks,  Cognitive Neurodynamics , Vol. 4, No. 4, pp. 295-313, November 2010 and the patent application by James Ting-Ho Lo, A Cortex-Like Learning Machine for Temporal and Hierarchical Pattern Recognition, U.S. patent application Ser. No. 12/471,341, filed May 22, 2009; Publication No. US-2009-0290800-A1, Publication Date: Nov. 26, 2009. In the same paper and same patent application, it is proven that given two m-dimensional vectors, x and y, their orthogonal expansions, {hacek over (x)} and {hacek over (y)}, satisfy
 
     
       
         
           
             
               
                 
                   
                     
                       
                         x 
                         ⋓ 
                       
                       ′ 
                     
                     ⁢ 
                     
                       y 
                       ⋓ 
                     
                   
                   = 
                   
                     
                       ∏ 
                       
                         j 
                         = 
                         1 
                       
                       m 
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       ( 
                       
                         1 
                         + 
                         
                           
                             x 
                             j 
                           
                           ⁢ 
                           
                             y 
                             j 
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   71 
                   ) 
                 
               
             
           
         
       
     
     Recall that for real-valued variables, v, u, x, and y, 
               v   -     1   2       =       -     1   2       ⁢   x                     ϕ   ⁡     (     v   ,   u     )       -     1   2       =         -   2     ⁢     (     v   -     1   2       )     ⁢     (     u   -     1   2       )       =     -     xy   2               
The relation between the above formula 71 and the formula (6) can be seen by observing that for m-dimensional vectors, v, u, x, y, and their neuronal codes, {hacek over (v)}, {hacek over (u)}, and orthogonal expansions, {hacek over (x)}, {hacek over (y)}, respectively; we have {hacek over (v)}−½I=−½{hacek over (x)} and {hacek over (u)}=½I=−½{hacek over (y)}, and
 
                             ∏     k   =   1     m     ⁢           ⁢     (     1   +       x   k     ⁢     y   k         )       =       ⁢     4   ⁢       (       -     1   2       ⁢     x   ⋓       )     ′     ⁢     (       -     1   2       ⁢     y   ⋓       )                   =       ⁢     4   ⁢       (       v   ⋓     -       1   2     ⁢   I       )     ′     ⁢     (       u   ⋓     -       1   2     ⁢   I       )                   =       ⁢       ∏     k   =   1     m     ⁢           ⁢     (     1   +       2   2     ⁢     (       v   k     -     1   2       )     ⁢     (       u   k     -     1   2       )         )                     (   72   )               
where I=[1 1 . . . 1]′, which we note is not the identity matrix I. From the formula 71, it follows that if x and y are bipolar binary variables,
 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           
                             
                               
                                 x 
                                 ⋓ 
                               
                               ′ 
                             
                             ⁢ 
                             
                               y 
                               ⋓ 
                             
                           
                           = 
                             
                           ⁢ 
                           0 
                         
                         , 
                         
                           
                             if 
                             ⁢ 
                             
                                 
                             
                             ⁢ 
                             x 
                           
                           ≠ 
                           y 
                         
                       
                     
                   
                   
                     
                       
                         
                           = 
                             
                           ⁢ 
                           
                             2 
                             m 
                           
                         
                         , 
                         
                           
                             if 
                             ⁢ 
                             
                                 
                             
                             ⁢ 
                             x 
                           
                           = 
                           y 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   73 
                   ) 
                 
               
             
           
         
       
     
     Based on the transformations, (65) and (66), and the above corresponding properties between the variables x and y and the variables v and u, the LOM (low-order model of biological neural networks) described in preceding Subsections can be transformed into the CIPAM, which is an equivalent of LOM. In this equivalent of LOM, components of feature vectors, “spikes and nonspikes”, and components of labels are bipolar binary digits or in their vicinities. The variables whose values are such digits and numbers will be denoted by x tj , y ti  and z ti  instead of v tj , u ti  and w ti  in the following lists of formulas for the equivalent, CIPAM. Here x tj =ƒ −1 (v tj ), y ti =ƒ −1 (u ti ), and z ti =ƒ −1 (w ti ). 
     For unsupervised covariance learning by the CIPAM, the formulas, (11), (12) and (13), are respectively transformed into 
                     D   ij     ←       λ   ⁢           ⁢     D   ij       +       Λ   ⁡     (       y   ti     -     〈     y   ti     〉       )       ⁢     (         x   ⋓     tj     -     〈       x   ⋓     tj     〉       )                 (   74   )               D   ←       λ   ⁢           ⁢   D     +       Λ   ⁡     (       y   t     -     〈     y   t     〉       )       ⁢       (         x   ⋓     t     -     〈       x   ⋓     t     〉       )     ′                 (   75   )               D   =     Λ   ⁢       ∑     s   =   1     t     ⁢         λ     t   -   s       ⁡     (       y   s     -     〈     y   s     〉       )       ⁢       (         x   ⋓     s     -     〈       x   ⋓     s     〉       )     ′                   (   76   )               
where x t  is a vector input to a neuronal encoder; each of the 2 m  outputs, {hacek over (x)} t1 , {hacek over (x)} t2 , . . . , {hacek over (x)} t2     m   , from the neuronal encoder at time (or numbering) t, pass through a synapse to reach each of a number, say R, postsynaptic model spiking neurons and a postsynaptic model nonspiking neuron; and the output of model spiking neuron i is denoted by y ti . y t  is the vector of R outputs y ti  of the R model spiking neurons. Furthermore, the formulas, (14), (15) and (16), are respectively transformed into
 
     
       
         
           
             
               
                 
                   
                     D 
                     ij 
                   
                   ← 
                   
                     
                       λ 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         D 
                         ij 
                       
                     
                     + 
                     
                       Λ 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           y 
                           ti 
                         
                         ⁡ 
                         
                           ( 
                           
                             
                               
                                 x 
                                 ⋓ 
                               
                               tj 
                             
                             - 
                             
                               〈 
                               
                                 
                                   x 
                                   ⋓ 
                                 
                                 tj 
                               
                               〉 
                             
                           
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   77 
                   ) 
                 
               
             
             
               
                 
                   D 
                   ← 
                   
                     
                       λ 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       D 
                     
                     + 
                     
                       Λ 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           
                             y 
                             t 
                           
                           ⁡ 
                           
                             ( 
                             
                               
                                 
                                   x 
                                   ⋓ 
                                 
                                 t 
                               
                               - 
                               
                                 〈 
                                 
                                   
                                     x 
                                     ⋓ 
                                   
                                   t 
                                 
                                 〉 
                               
                             
                             ) 
                           
                         
                         ′ 
                       
                     
                   
                 
               
               
                 
                   ( 
                   78 
                   ) 
                 
               
             
             
               
                 
                   D 
                   = 
                   
                     Λ 
                     ⁢ 
                     
                       
                         ∑ 
                         
                           s 
                           = 
                           1 
                         
                         t 
                       
                       ⁢ 
                       
                         
                           λ 
                           
                             t 
                             - 
                             s 
                           
                         
                         ⁢ 
                         
                           
                             
                               y 
                               s 
                             
                             ⁡ 
                             
                               ( 
                               
                                 
                                   
                                     x 
                                     ⋓ 
                                   
                                   s 
                                 
                                 - 
                                 
                                   〈 
                                   
                                     
                                       x 
                                       ⋓ 
                                     
                                     s 
                                   
                                   〉 
                                 
                               
                               ) 
                             
                           
                           ′ 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   79 
                   ) 
                 
               
             
           
         
       
     
     For supervised covariance learning by the CIPAM, (17), (18), and (19) are respectively transformed into 
                     D   ij     ←       λ   ⁢           ⁢     D   ij       +     Λ   ⁢           ⁢       z   ti     ⁡     (         x   ⋓     tj     -     〈       x   ⋓     tj     〉       )                   (   80   )               D   ←       λ   ⁢           ⁢   D     +     Λ   ⁢           ⁢         z   t     ⁡     (         x   ⋓     t     -     〈       x   ⋓     t     〉       )       ′                 (   81   )               D   =     Λ   ⁢       ∑     s   =   1     t     ⁢       λ     t   -   s       ⁢         z   s     ⁡     (         x   ⋓     s     -     〈       x   ⋓     s     〉       )       ′                   (   82   )               
where z t  is a label of the pattern (or cause) inside the receptive field of the R model spiking neurons, and is provided from outside the CIPAM. Here z s , s=1, 2, . . . t, are bipolar binary vectors or nearly bipolar binary vectors.
 
     For unsupervised accumulation learning by the CIPAM, (21) and (22) are respectively transformed into 
     
       
         
           
             
               
                 
                   C 
                   ← 
                   
                     
                       λ 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       C 
                     
                     + 
                     
                       
                         Λ 
                         2 
                       
                       ⁢ 
                       
                         
                           ( 
                           
                             
                               
                                 x 
                                 ⋓ 
                               
                               t 
                             
                             - 
                             
                               〈 
                               
                                 
                                   x 
                                   ⋓ 
                                 
                                 t 
                               
                               〉 
                             
                           
                           ) 
                         
                         ′ 
                       
                     
                   
                 
               
               
                 
                   ( 
                   83 
                   ) 
                 
               
             
             
               
                 
                   C 
                   = 
                   
                     
                       Λ 
                       2 
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           s 
                           = 
                           1 
                         
                         t 
                       
                       ⁢ 
                       
                         
                           
                             λ 
                             
                               t 
                               - 
                               s 
                             
                           
                           ⁡ 
                           
                             ( 
                             
                               
                                 
                                   x 
                                   ⋓ 
                                 
                                 s 
                               
                               - 
                               
                                 〈 
                                 
                                   
                                     x 
                                     ⋓ 
                                   
                                   s 
                                 
                                 〉 
                               
                             
                             ) 
                           
                         
                         ′ 
                       
                     
                   
                 
               
               
                 
                   ( 
                   84 
                   ) 
                 
               
             
           
         
       
     
     For retrieving information from synapses, (24) and (25) are respectively transformed into
 
 d   τ   =D ( {hacek over (x)}   τ   −&lt;{hacek over (x)}   τ &gt;)  (85)
 
 c   τ   =C ( {hacek over (x)}   τ   −&lt;{hacek over (x)}   τ &gt;)  (86)
 
     For maximal generalization, the masking matrix (32) remains the same, but (33) and (34), are replaced with
 
 d   τ   =DM ( {hacek over (x)}   τ   −&lt;{hacek over (x)}   τ &gt;)  (87)
 
 c   τ   =CM ( {hacek over (x)}   τ   −&lt;{hacek over (x)}   τ &gt;)  (88)
 
which are also called the decovariance rule.
 
     To describe the feature subvector input to a processing unit with Ψ multiple neuronal encoders in the CIPAM, the formulas, (39) and (40), are respectively transformed into
 
 {hacek over (x)}   t   =[{hacek over (x)}   t ′(1)  {hacek over (x)}   t ′(2) . . .  {hacek over (x)}   t ′(Ψ)]′  (89)
 
&lt; {hacek over (x)}   t   &gt;=[&lt;{hacek over (x)}   t ′(1)&gt; &lt; {hacek over (x)}   t ′(2)&gt; . . . &lt; {hacek over (x)}   t ′(Ψ)&gt;]′  (90)
 
     The formula, (43), for the general masking matrix M for a PU in the CIPAM remains the same. The general expansion covariance matrix D learned by the unsupervised covariance rule by an UPU in the CIPAM is defined by (41), where the code covariance matrix D(ψ) is defined by (76) or (79), depending on whether y s −&lt;y s &gt; or y s  is used. The general expansion covariance matrix D learned by the supervised covariance rule by an UPU in the CIPAM is defined by (41), where the code covariance matrix D(ψ) is defined by (82). The general expansion covariance matrix C for the CIPAM is defined by (42), where the code covariance matrix C(ψ) is defined by (84). 
     For retrieving information from synapses by the CIPAM, the formulas, (44) and (45), are transformed into
 
 d   τ (ψ)= D (ψ) M (ψ)( {hacek over (x)}   τ (ψ)−&lt; {hacek over (x)}   τ (ψ)&gt;)  (91)
 
 c   τ (ψ)= C (ψ) M (ψ)( {hacek over (x)}   τ (ψ)−&lt; {hacek over (x)}   τ (ψ)&gt;)  (92)
 
for ψ=1, . . . , Ψ.
 
     The computations performed by a nonspiking neuron and by spiking neuron k in a PU in the CIPAM are respectively the addition of components of c τ (ψ), ψ=1, . . . , Ψ, and the addition of kth rows of d τ (ψ), ψ=1, . . . , Ψ to get c, and d τk . Expressions of the resulting sums, c τ  and d τk , can be expressed by transforming (54) and (55) respectively into
 
 c   τ   =CM ( {hacek over (x)}   τ   −&lt;{hacek over (x)}   τ &gt;)  (93)
 
 d   τk   =D   k   M ( {hacek over (x)}   τ   −&lt;{hacek over (x)}   τ &gt;)  (94)
 
(c τ +d τk )/2 is an estimate of 2 m  times the total number of times v τ  and its variants have been encoded and stored in C with the kth component r τk  of r τ  being 1. c τ  is an estimate of 2 m  times the total number of times v τ  and its variants have been encoded and stored in C. Consequently, (d τk /c τ +1)/2 is an empirical probability p τk  that r τk  is equal to 1 given x t  input to the PU. Model spiking neuron k then uses a pseudo-random number generator to generate a number (or “spike”) 1 with probability p τk  and a number −1 (representing “no spike”) with probability 1−p τk . This pseudo-random number denoted by x{p τk } is the output y τk  of model spiking neuron k at time or numbering τ. y τk =x{p τk } is thus a point estimate of the k-th component r τk  of the label r τ  of x τ .
 
     Note that the vector
 
 p   τ   =[p   τ1    p   τ2    . . . p   τR ]′
 
is a representation of an empirical probability distribution of the label r τ . Note also that the outputs of the R model spiking neurons in response to x τ  form a bipolar binary vector y τ =x{p τ }, which is a point estimate of the label r τ  of the vector x τ  input to the PU, whether r τ  is y τ  in unsupervised learning or z τ  in supervised learning.
 
     The CIPAM has the following computational advantage over LOM: While the (model) neuronal node in LOM evaluates −2vu+v+u, the (model) neuronal node in the CIPAM evaluates xy. The former involves 2 multiplications and two additions, and the latter a single multiplication. In a neuronal encoder with an m-dimensional input vector, there are 2 m  neuronal nodes. Therefore, the CIPAM is computationally much less expensive than LOM. 
     The learning masking matrices, denoted by M # (ψ), ψ=1, . . . , Ψ, form a general masking matrix called the general learning masking matrix
 
 M   # =diag[ M   # (1)  M   # (2) . . .  M   # (Ψ)]  (95)
 
which is a diagonal matrix. The diagonal entries are numbered consecutively from 1 to dim {hacek over (x)} τ =Σ ψ=1   Ψ 2 dim x     τ     (ψ) , where {hacek over (x)} τ  is the general neuronal code, whose entries are also numbered consecutively from 1 to dim {hacek over (x)} τ . The diagonal entries M jj   # , j=1, . . . , dim {hacek over (x)} τ , of M #  are called learning masking factors.
 
     An UPU (unsupervised processing unit) with learning masking matrices M # (ψ), ψ=1, . . . , Ψ, generates an estimate x{p τ } of the label of the vector x τ  input to the UPU using its masking matrices M(ψ), ψ=1, . . . , Ψ, as before. However, when it comes to learning, the learning masking matrices M # (ψ), ψ=1, . . . , Ψ, are used instead to generate an estimate x{p τ   # } of the label of x t , and the general code covariance matrix D is adjusted by
 
 D (ψ)←λ D (ψ)+Λ( x{p   τ   #   }−&lt;x{p   τ   # }&gt;)( {hacek over (x)}   τ   −&lt;{hacek over (x)}   τ &gt;)′  (96)
 
More specifically, x{p τ   # } is generated as follows:
 
                     [     c     τ   ⁢           ⁢   j     #     ]     =     [       M   jj   #     ⁢       C   j     ⁡     (         x   ⋓     τ     -     〈       x   ⋓     τ     〉       )         ]             (   97   )                 [     d     τ   ⁢           ⁢   kj     #     ]     =     [       M   jj   #     ⁢       D   kj     ⁡     (         x   ⋓     τ     -     〈       x   ⋓     τ     〉       )         ]             (   98   )                 c   τ   #     =       ∑   j`     ⁢     c     τ   ⁢           ⁢   j     #               (   99   )                 p     τ   ⁢           ⁢   k     #     =       (         ∑   j`     ⁢     d     τ   ⁢           ⁢   kj         +     c   τ   #       )     /     (     2   ⁢           ⁢     c   τ   #       )               (   100   )               
x{p τ   # } is a pseudorandom number generated in accordance with the empirical probability distribution p τk   # , and then x{p τ   # }=[x{p τ1   # } . . . x{p τR   # }]′, where R is the number of model spiking neurons in the UPU.
 
5.13 Preprocessing Exogenous Feature Vectors for the CIPAM
 
     Given a variable α whose value is either a real number a or a real number b, it can be transformed into a variable x whose value is either −1 or 1, respectively, by the following function: 
             x   =       f   ⁡     (   α   )       =         2   ⁢           ⁢   α     -   a   -   b       b   -   a               
If α is near a or near b, then x is near ƒ(a)=−1 or near ƒ(b)=1, respectively. Given a vector α=[α 1  α 2  . . . α m ]′ whose components are real numbers, a and b, the vector α can be transformed into a bipolar binary vector x by the function
 
 x =ƒ(α)=[ƒ(α 1 ) ƒ(α 2 ) . . . ƒ(α m )]′
 
If a component of a is near a or near b, then the corresponding component of x is near ƒ(a) or near ƒ(b), respectively. Notice that the symbol ƒ denotes both a real-valued function of a real-valued variable α and a vector-valued function of a vector variable α with real components.
 
     If the exogenous feature vectors α t   ex =1, 2, . . . , to be processed by an artificial neural network have components being real numbers, a and b, or near a and near b, the vectors α t   ex , t=1, 2, . . . , can be converted into binary or nearly binary vectors x t   ex , t=1, 2, . . . , by x t   ex =ƒ(α t   ex ), t=1, 2, . . . . The CIPAM described hereinabove can then be applied. 
     5.14 A General ANN Mathematically Equivalent to LOM 
     The mathematical equivalence of the LOM and CIPAM motivated search for ANNs (artificial neural networks) that are mathematically equivalent to the LOM and CIPAM. Generalizations of the affine transformations, (65) and (66), are the following: 
                   v   =       f   ⁡     (   α   )       =       α   -   a       b   -   a                 (   101   )               α   =         f     -   1       ⁡     (   v   )       =         (     b   -   a     )     ⁢   v     +   a               (   102   )               
where ƒ −1  denotes the inverse function of ƒ. Notice that ƒ(a)=0 and ƒ(b)=1, and that ƒ −1 (0)=a and ƒ −1 (1)=b. Note that the affine transformations, (101) and (102) with a=1 and b=−1, are (65) and (66). Using (101) and (102), the LOM can be transformed into a general ANN for a and b the way the LOM is transformed into the CIPAM for a=1 and b=−1.
 
     Let v=ƒ(α) and u=ƒ(β). Recall that φ(v, u)=−2vu+v+u=(1−v)u+(1−u)v. By simple substitution, 
                     ϕ   ⁡     (     v   ,   u     )       =       ⁢     ϕ   ⁡     (       f   ⁡     (   α   )       ,     f   ⁡     (   β   )         )                   =       ⁢         (     1   -       α   -   a       b   -   a         )     ⁢     (       β   -   a       b   -   a       )       +       (     1   -       β   -   a       b   -   a         )     ⁢     (       α   -   a       b   -   a       )                     =       ⁢     -           (     α   -   b     )     ⁢     (     β   -   a     )       +       (     α   -   a     )     ⁢     (     β   -   b     )             (     b   -   a     )     2                         and                       f     -   1       ⁡     (     ϕ   ⁡     (       f   ⁡     (   α   )       ,     f   ⁡     (   β   )         )       )       =       ⁢         (     b   -   a     )     ⁢     ϕ   ⁡     (       f   ⁡     (   α   )       ,     f   ⁡     (   β   )         )         +   a                 =       ⁢           -   2     ⁢           ⁢   α   ⁢           ⁢   β     +       (     a   +   b     )     ⁢     (     α   +   β   -   a     )           b   -   a                   
This function ƒ −1 (φ(ƒ(α),ƒ(β))) is denoted by φ(α,β). Notice that φ(a, a)=a, φ(a, b)=b, φ(b, a)=b, φ(b, b)=a. Hence, if α and β only assume values from {a, b}, φ(α, β) is a “general XOR or NXOR function”.
 
     The function φ(α, β) is a general representation of the neuronal node operation φ(v, u), where the inputs of the neuronal node are represented by α and β and its output is represented by φ(α, β). 
     The algebraic binary operation φ(α, β) is a commutative and associative binary operation on α and β:
 
φ(α,β)=φ(β,α)
 
φ(γ,φ(α,β))=φ(φ(γ,α),β)
 
Hence, we can define a symmetric function φ k  by applying the binary operation repeatedly as follows:
 
φ k (α 1 ,α 2 , . . . ,α k )=φ( . . . φ(φ(α 1 ,α 2 ),α 3 ), . . . ,α k )
 
where φ 1 (α i )=α 1  and φ 2 (α i , α j )=φ(α i , α j ).
 
     Let v i =ƒ(α i ) for i=1, . . . k. Recall φ(α 1 , α 2 )=ƒ −1 (φ(v 1 , v 2 ))=ƒ −1 (φ(ƒ(α 1 ),ƒ(α 2 ))). Using this definition of φ(α 1 , α 2 ), we obtain 
                       φ   3     ⁡     (       α   1     ,     α   2     ,     α   3       )       =       ⁢     φ   ⁡     (       φ   ⁡     (       α   1     ,     α   2       )       ,     α   3       )                   =       ⁢     φ   ⁡     (         f     -   1       ⁡     (     ϕ   ⁡     (       v   1     ,     v   2       )       )       ,       f     -   1       ⁡     (     v   3     )         )                   =       ⁢       f     -   1       ⁡     (     ϕ   ⁡     (       ϕ   ⁡     (       v   1     ,     v   2       )       ,     v   3       )       )                   =       ⁢       f     -   1       ⁡     (       ϕ   3     ⁡     (       v   1     ,     v   2     ,     v   3       )       )                   
Assume that φ k (α 1 , . . . , α k )=ƒ −1 (φ k (v 1 , . . . , v k )). Using the definition of φ(α 1 , α 2 ) again yields
 
                       φ     k   +   1       ⁡     (       α   1     ,   …   ⁢           ,     α     k   +   1         )       =       ⁢     φ   ⁡     (         φ   k     ⁡     (       α   1     ,     …   ⁢           ⁢     α   k         )       ,     α     k   +   1         )                   =       ⁢     φ   ⁡     (         f     -   1       ⁡     (       ϕ   k     ⁡     (       v   1     ,     …   ⁢           ⁢     v   k         )       )       ,     α     k   +   1         )                   =       ⁢       f     -   1       ⁡     (     ϕ   ⁡     (         ϕ   k     ⁡     (       v   1     ,     …   ⁢           ⁢     v   k         )       ,     v     k   +   1         )       )                   =       ⁢       f     -   1       ⁡     (       ϕ     k   +   1       ⁡     (       v   1     ,   …   ⁢           ,     v     k   +   1         )       )                   
By mathematical induction, for any positive integer k,
 
φ k (α 1 , . . . ,α k )=ƒ −1 (φ k ( v   1   , . . . ,v   k ))  (103)
 
     To describe an orthogonality property of the outputs of a model neuronal encoder with input variables {α 1 , α 2 , . . . , α m }, we organize its 2 m  outputs into a vector as follows: Let β denote a scalar and α=[α 1  α 2  . . . α k ]′ a k-dimensional vector. Define a k-dimensional vector φ(β,α) by
 
φ(β,α)=[φ(β,α 1 ) φ(β,α 2 ) . . . φ(β,α k )]′
 
The 2 m  different functions that can be defined by compositions of the binary operation φ(α,β) on the input set {α 1 , α 2 , . . . , α m } are generated and organized into a 2 m -dimensional column vector {hacek over (α)} by recursively generating row vectors {hacek over (α)}(1, . . . , k), for k=1, 2, . . . , m, as follows:
 
{hacek over (α)}(1)=[ a α]   (104)
 
{hacek over (α)}(1,2)=[{hacek over (α)}(1) φ(α 2 ,{hacek over (α)}(1))]  (105)
 
{hacek over (α)}(1 , . . . ,k+ 1)=[{hacek over (α)}(1 , . . . ,k ) φ(α k+1 ,{hacek over (α)}(1 , . . . ,k ))]  (106)
 
{hacek over (α)}={hacek over (α)}′(1 , . . . ,m )  (107)
 
By (103),
 
                     α   ⋓     =       ⁢       [             α   ⋓     1             α   ⋓     2         …           α   ⋓       2   m             ]     ′                 =       ⁢       [               f     -   1       ⁡     (       v   ⋓     1     )               f     -   1       ⁡     (       v   ⋓     2     )           …         f     -   1             ⁢     (       v   ⋓       2   m       )       ]     ′                 
where {hacek over (α)} k  and {hacek over (v)} k  denote the k-th components of {hacek over (α)} and {hacek over (v)} respectively. The vector on the right side is denoted by ƒ −1 ({hacek over (v)}) for simplicity. The vector {hacek over (α)} is called the neuronal code of α.
 
     By (101) and (102), 
                         v   ⋓     k     -     1   2       =       ⁢       f   ⁡     (       α   ⋓     k     )       -     1   2                   =       ⁢             α   ⋓     k     -   a       b   -   a       -     1   2                   =       ⁢           α   ⋓     k     -   μ       b   -   a                   
where
 
             μ   =         a   +   b     2     .           
Hence
 
               v   ⋓     -       1   2     ⁢   I     -       1     b   -   a       ⁢     (       α   ⋓     -     μ   ⁢           ⁢   I       )                 and                               (       α   ⋓     -     μ   ⁢           ⁢   I       )     ′     ⁢     (       β   ⋓     -     μ   ⁢           ⁢   I       )       =       ⁢         (     b   -   a     )     2     ⁢       (       v   ⋓     -       1   2     ⁢   I       )     ′     ⁢     (       u   ⋓     -       1   2     ⁢   I       )                   =       ⁢           (     b   -   a     )     2     4     ⁢       ∏     k   =   1     m     ⁢           ⁢     (     1   +       2   2     ⁢     (       v   k     -     1   2       )     ⁢     (       u   k     -     1   2       )         )                     =       ⁢           (     b   -   a     )     2     4     ⁢       ∏     k   =   1     m     ⁢           ⁢     (     1   +       2   2     ⁢     (         α   k     -   μ       b   -   a       )     ⁢     (         β   k     -   μ       b   -   a       )         )                       (   108   )               
It follows that if the components assume values from {a, b}, the neuronal codes have the following orthogonality property:
 
     
       
         
           
             
               
                 
                   
                     
                       
                         ( 
                         
                           
                             α 
                             ⋓ 
                           
                           - 
                           
                             μ 
                             ⁢ 
                             
                                 
                             
                             ⁢ 
                             I 
                           
                         
                         ) 
                       
                       ′ 
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           β 
                           ⋓ 
                         
                         - 
                         
                           μ 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           I 
                         
                       
                       ) 
                     
                   
                   = 
                     
                   ⁢ 
                   
                     
                       
                         2 
                         
                           m 
                           - 
                           2 
                         
                       
                       ⁢ 
                       
                         
                           ( 
                           
                             b 
                             - 
                             a 
                           
                           ) 
                         
                         2 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       if 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       α 
                     
                     = 
                     β 
                   
                 
               
             
             
               
                 
                   = 
                     
                   ⁢ 
                   
                     
                       0 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       if 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       α 
                     
                     ≠ 
                     β 
                   
                 
               
             
           
         
       
     
     If the components of α and β assume values from the set {a, μ, b}, we have the following additional properties from the formula (108):
         1. If (α k −μ)(β k −μ)=0 for some k in {1, . . . , m}, then       

     
       
         
           
             
               
                 
                   
                     
                       
                         ( 
                         
                           
                             α 
                             ⋓ 
                           
                           - 
                           
                             μ 
                             ⁢ 
                             
                                 
                             
                             ⁢ 
                             I 
                           
                         
                         ) 
                       
                       ′ 
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           β 
                           ⋓ 
                         
                         - 
                         
                           μ 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           I 
                         
                       
                       ) 
                     
                   
                   = 
                     
                   ⁢ 
                   
                     
                       
                         
                           ( 
                           
                             b 
                             - 
                             a 
                           
                           ) 
                         
                         2 
                       
                       4 
                     
                     ⁢ 
                     
                       
                         ∏ 
                         
                           
                             k 
                             = 
                             1 
                           
                           , 
                           , 
                           
                             j 
                             ≠ 
                             k 
                           
                         
                         m 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         ( 
                         
                           1 
                           + 
                           
                             
                               2 
                               2 
                             
                             ⁢ 
                             
                               ( 
                               
                                 
                                   
                                     α 
                                     k 
                                   
                                   - 
                                   μ 
                                 
                                 
                                   b 
                                   - 
                                   a 
                                 
                               
                               ) 
                             
                             ⁢ 
                             
                               ( 
                               
                                 
                                   
                                     β 
                                     k 
                                   
                                   - 
                                   μ 
                                 
                                 
                                   b 
                                   - 
                                   a 
                                 
                               
                               ) 
                             
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   109 
                   ) 
                 
               
             
           
         
       
         
         
           
             2. If ({hacek over (α)}−μI)′(β−μI)≠0, then ({hacek over (α)}−μI)′({hacek over (β)}−μI)=2 4(α−μI)′(β−μI)−2 , where the vector I on the left side of the equality sign is 2 m -dimensional, but I on the right side of the equality sign is m-dimensional. 
           
         
       
    
     Based on the transformations, (101) and (102), and the above corresponding properties between the variables α and β and the variables v and u, the LOM (low-order model of biological neural networks) can be transformed into a general representation of the LOM, which is a general ANN mathematically equivalent to the LOM. In this general ANN, components of feature vectors, “spikes and nonspikes”, and components of labels are numbers, a and b, or numbers in their vicinities. The variables whose values are such numbers will be denoted by α tj , β ti  and γ ti  instead of v tj , u ti  and w ti  in the following lists of formulas for the general representation. Here α tj =ƒ −1 (v tj ), β ti =ƒ −1 (u ti ), and γ ti =ƒ −1 (w ti ). 
     For unsupervised covariance learning by the general model, the formulas, (11), (12) and (13), are respectively transformed to 
                     D   ij     ←       λ   ⁢           ⁢     D   ij       +       Λ   ⁡     (       β   ti     -     〈     β   ti     〉       )       ⁢     (         α   ⋓     tj     -     〈       α   ⋓     tj     〉       )                 (   110   )               D   ←       λ   ⁢           ⁢   D     +       Λ   ⁡     (       β   t     -     〈     β   t     〉       )       ⁢       (         α   ⋓     t     -     〈       α   ⋓     t     〉       )     ′                 (   111   )               D   =     Λ   ⁢       ∑     s   =   1     t     ⁢         λ     t   -   s       ⁡     (       β   s     -     〈     β   s     〉       )       ⁢       (         α   ⋓     s     -     〈       α   ⋓     s     〉       )     ′                   (   112   )               
where α t  is a vector input to the neuronal encoder; the 2 m  outputs, {hacek over (α)} t1 , {hacek over (α)} t2 , . . . , {hacek over (α)} t2     m   , from the neuronal encoder at time (or numbering) t, pass through a synapse to reach each of a number, say R, postsynaptic model spiking neurons and a postsynaptic model nonspiking neuron, and the output of model spiking neuron i is denoted by β ti ·β t  is the vector of R outputs β ti  of the R model spiking neurons. Furthermore, the formulas, (14), (15) and (16), are respectively transformed into
 
     
       
         
           
             
               
                 
                   
                     D 
                     ij 
                   
                   ← 
                   
                     
                       λ 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         D 
                         ij 
                       
                     
                     + 
                     
                       
                         Λ 
                         ⁡ 
                         
                           ( 
                           
                             
                               β 
                               ti 
                             
                             - 
                             
                               
                                 ( 
                                 
                                   a 
                                   + 
                                   b 
                                 
                                 ) 
                               
                               / 
                               2 
                             
                           
                           ) 
                         
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             
                               α 
                               ⋓ 
                             
                             tj 
                           
                           - 
                           
                             〈 
                             
                               
                                 α 
                                 ⋓ 
                               
                               tj 
                             
                             〉 
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   113 
                   ) 
                 
               
             
             
               
                 
                   D 
                   ← 
                   
                     
                       λ 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       D 
                     
                     + 
                     
                       
                         Λ 
                         ⁡ 
                         
                           ( 
                           
                             
                               β 
                               t 
                             
                             - 
                             
                               
                                 ( 
                                 
                                   a 
                                   + 
                                   b 
                                 
                                 ) 
                               
                               / 
                               2 
                             
                           
                           ) 
                         
                       
                       ⁢ 
                       
                         
                           ( 
                           
                             
                               
                                 α 
                                 ⋓ 
                               
                               t 
                             
                             - 
                             
                               〈 
                               
                                 
                                   α 
                                   ⋓ 
                                 
                                 t 
                               
                               〉 
                             
                           
                           ) 
                         
                         ′ 
                       
                     
                   
                 
               
               
                 
                   ( 
                   114 
                   ) 
                 
               
             
             
               
                 
                   D 
                   = 
                   
                     Λ 
                     ⁢ 
                     
                       
                         ∑ 
                         
                           s 
                           = 
                           1 
                         
                         t 
                       
                       ⁢ 
                       
                         
                           
                             λ 
                             
                               t 
                               - 
                               s 
                             
                           
                           ⁡ 
                           
                             ( 
                             
                               
                                 β 
                                 s 
                               
                               - 
                               
                                 
                                   ( 
                                   
                                     a 
                                     + 
                                     b 
                                   
                                   ) 
                                 
                                 / 
                                 2 
                               
                             
                             ) 
                           
                         
                         ⁢ 
                         
                           
                             ( 
                             
                               
                                 
                                   α 
                                   ⋓ 
                                 
                                 s 
                               
                               - 
                               
                                 〈 
                                 
                                   
                                     α 
                                     ⋓ 
                                   
                                   s 
                                 
                                 〉 
                               
                             
                             ) 
                           
                           ′ 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   115 
                   ) 
                 
               
             
           
         
       
     
     For supervised covariance learning by the general model, (17), (18), and (19) are respectively transformed into 
                     D   ij     ←       λ   ⁢           ⁢     D   ij       +       Λ   ⁡     (       γ   ti     -       (     a   +   b     )     /   2       )       ⁢     (         α   ⋓     tj     -     〈       α   ⋓     tj     〉       )                 (   116   )               D   ←       λ   ⁢           ⁢   D     +       Λ   ⁡     (       γ   t     -       (     a   +   b     )     /   2       )       ⁢       (         α   ⋓     t     -     〈       α   ⋓     t     〉       )     ′                 (   117   )               D   =     Λ   ⁢       ∑     s   =   1     t     ⁢         λ     t   -   s       ⁡     (       γ   s     -       (     a   +   b     )     /   2       )       ⁢       (         α   ⋓     s     -     〈       α   ⋓     s     〉       )     ′                   (   118   )               
where γ t , t=1, 2, . . . , is a sequence of labels of the causes (e.g., patterns) inside the receptive field of the R model spiking neurons in the PU, and is provided from outside the CIPAM. The components of γ ti  assume values from the set {a, b}.
 
     For unsupervised accumulation learning by the general model, (21) and (22) are respectively transformed into 
     
       
         
           
             
               
                 
                   C 
                   ← 
                   
                     
                       λ 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       C 
                     
                     + 
                     
                       
                         Λ 
                         2 
                       
                       ⁢ 
                       
                         
                           ( 
                           
                             
                               
                                 α 
                                 ⋓ 
                               
                               t 
                             
                             - 
                             
                               〈 
                               
                                 
                                   α 
                                   ⋓ 
                                 
                                 t 
                               
                               〉 
                             
                           
                           ) 
                         
                         ′ 
                       
                     
                   
                 
               
               
                 
                   ( 
                   119 
                   ) 
                 
               
             
             
               
                 
                   C 
                   = 
                   
                     
                       Λ 
                       2 
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           s 
                           = 
                           1 
                         
                         t 
                       
                       ⁢ 
                         
                       ⁢ 
                       
                         
                           
                             λ 
                             
                               t 
                               - 
                               s 
                             
                           
                           ⁡ 
                           
                             ( 
                             
                               
                                 
                                   α 
                                   ⋓ 
                                 
                                 s 
                               
                               - 
                               
                                 〈 
                                 
                                   
                                     α 
                                     ⋓ 
                                   
                                   s 
                                 
                                 〉 
                               
                             
                             ) 
                           
                         
                         ′ 
                       
                     
                   
                 
               
               
                 
                   ( 
                   120 
                   ) 
                 
               
             
           
         
       
     
     For retrieving information from synapses, the decovariance rule, (24) and (25), are respectively transformed into
 
 d   τ   =D ({hacek over (α)} τ −&lt;{hacek over (α)} τ &gt;)  (121)
 
 c   τ   =C ({hacek over (α)} τ −&lt;{hacek over (α)} τ &gt;)  (122)
 
which are also called the decovariance rule.
 
     For maximal generalization, the masking matrix (32) remains the same, but (33) and (34) are replaced with
 
 d   τ   =DM ({hacek over (α)} τ −&lt;{hacek over (α)} τ &gt;)  (123)
 
 c   τ   =CM ({hacek over (α)} τ −&lt;{hacek over (α)} τ &gt;)  (124)
 
which are also called the decovariance rule.
 
     To describe the feature subvector input to a processing unit with Ψ multiple neuronal encoders in the general model, the formulas, (39) and (40), are respectively transformed to
 
{hacek over (α)} t =[{hacek over (α)} t ′(1) {hacek over (α)} t ′(2) . . . {hacek over (α)} t ′(Ψ)]′  (125)
 
&lt;{hacek over (α)} t &gt;=[&lt;{hacek over (α)} t ′(1)&gt; &lt;{hacek over (α)} t ′(2)&gt; . . . &lt;{hacek over (α)} t ′(Ψ)&gt;]′  (126)
 
     The formula, (43), for the general masking matrix M for a PU in the CIPAM remains the same. The general expansion covariance matrix D learned by the unsupervised covariance rule by an UPU in the CIPAM is defined by (41), where the code covariance matrix D(ψ) is defined by (112) or (115), depending on whether y s −&lt;y s &gt; or y s  is used. The general expansion covariance matrix D learned by the supervised covariance rule by an UPU in the CIPAM is defined by (41), where the code covariance matrix D(ψ) is defined by (118). The general expansion covariance matrix C for the CIPAM is defined by (42), where the code covariance matrix C(ψ) is defined by (120). 
     For retrieving information from synapses by the general model, the formulas, (44) and (45), are transformed into
 
 d   τ (ψ)= D (ψ) M (ψ)({hacek over (α)} τ (ψ)−&lt;{hacek over (α)} τ (ψ)&gt;)  (127)
 
 c   τ (ψ)= C (ψ) M (ψ) M ({hacek over (α)} τ (ψ)−&lt;{hacek over (α)} τ (ψ)&gt;)  (128)
 
for ψ=1, . . . , Ψ.
 
     The computation formulas performed by a nonspiking neuron and a spiking neuron in the general model are obtained by transforming (54) and (55) respectively into
 
 c   τ   =CM ({hacek over (α)} τ −&lt;{hacek over (α)} τ &gt;)  (129)
 
 d   τk   =D   k   M ({hacek over (α)} τ −&lt;{hacek over (α)} τ &gt;)  (130)
 
(c τ +d τk )/2 is an estimate of 2 m  times the total number of times v τ  and its general models have been encoded and stored in C with the kth component r τk  of r τ  being a. c τ  is an estimate of 2 m  times the total number of times v τ  and its general models have been encoded and stored in C. Consequently, (d τk /c τ +1)/2 is the empirical probability p τk  that r τk  is equal to a given α t . model spiking neuron k then uses a pseudo-random number generator to generate a number (or “spike”) a with probability p τk  and a number b (or no “spike”) with probability 1−p τk . This pseudo-random number denoted by α{p τk } is the output β τk  of model spiking neuron k at time or numbering τ. β τk =α{p τk } is thus a point estimate of the k-th component r τk  of the label r τ  of α τ  input to the PU.
 
     Note that the vector
 
 p   τ   =[p   τ1    p   τ2    . . . p   τR ]′
 
is a representation of an empirical probability distribution of the label r τ . Note also that the outputs of the R model spiking neurons in response to α τ  form a bipolar binary vector β τ =α{p}, which is a point estimate of the label r τ  of α τ .
 
     6 CONCLUSION, RAMIFICATION, AND SCOPE OF INVENTION 
     Many embodiments of the present invention are disclosed, which can achieve the objects listed in the “Summary” of the present invention disclosure. While our descriptions hereinabove contain many specificities, these should not be construed as limitations on the scope of the invention, but rather as an exemplification of preferred embodiments. In addition to these embodiments, those skilled in the art will recognize that other embodiments are possible within the teachings of the present invention. Accordingly, the scope of the present invention should be limited only by the appended claims and their appropriately construed legal equivalents.