Source: http://www.google.com/patents/US5214715?dq=system+for+measuring+web+traffic&ei=Lg8FT__TIIr-sQKzxaGRCg
Timestamp: 2014-03-07 11:13:38
Document Index: 42020945

Matched Legal Cases: ['ART 1', 'ART 2', 'ART 3', 'ART 1', 'ART 1', 'ART 1', 'ART 1', 'ART 1', 'ART 2', 'ART 3', 'ART 1', 'ART 1', 'ART 1', 'ART 1', 'ART 2', 'ART 1', 'ART 1', 'ART 1', 'ART 1', 'ART 1', 'ART 1', 'ART 2', 'ART 1', 'ART 1', 'ART 1', 'ART 1', 'ART 1', 'ART 1', 'ART 1', 'ART 1', 'ART 1', 'Art 3', 'Art 3']

Patent US5214715 - Predictive self-organizing neural network - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inAdvanced Patent SearchPatentsAn A pattern recognition subsystem responds to an A feature representation input to select A-category-representation and predict a B-category-representation and its associated B feature representation input. During learning trials, a predicted B-category-representation is compared to that obtained through...http://www.google.com/patents/US5214715?utm_source=gb-gplus-sharePatent US5214715 - Predictive self-organizing neural networkAdvanced Patent SearchPublication numberUS5214715 APublication typeGrantApplication numberUS 07/648,653Publication dateMay 25, 1993Filing dateJan 31, 1991Priority dateJan 31, 1991Fee statusPaidAlso published asDE69216077D1, DE69216077T2, EP0569549A1, EP0569549B1, WO1992014200A1Publication number07648653, 648653, US 5214715 A, US 5214715A, US-A-5214715, US5214715 A, US5214715AInventorsGail A. Carpenter, Stephen Grossberg, John W. ReynoldsOriginal AssigneeTrustees Of Boston UniversityExport CitationBiBTeX, EndNote, RefManPatent Citations (6), Non-Patent Citations (6), Referenced by (67), Classifications (10), Legal Events (8) External Links: USPTO, USPTO Assignment, EspacenetPredictive self-organizing neural networkUS 5214715 AAbstract An A pattern recognition subsystem responds to an A feature representation input to select A-category-representation and predict a B-category-representation and its associated B feature representation input. During learning trials, a predicted B-category-representation is compared to that obtained through a B pattern recognition subsystem. With mismatch, a vigilance parameter of the A-pattern-recognition subsystem is increased to cause reset of the first-category-representation selection. Inputs to the pattern recognition subsystems may be preprocessed to complement code the inputs.
As shown in FIG. 1, ART networks encode new input patterns received at 20, in part, by changing the weights or long term memory (LTM) traces of a bottom-up adaptive filter 22. This filter is contained in pathways leading from a feature representation field (F.sub.1) to a category representation field (F.sub.2) of short term memory. Generally, the short term memory (STM) fields hold new patterns relative to each input pattern. The long term memory (LTM), on the other hand, defines patterns learned from some number of input patterns, that is, over a relatively longer period of time. This bottom-up filtering property is shared by many other models of adaptive pattern recognition and associative learning. In an ART network, however, it is a second, top-down adaptive filter 24 that leads to the crucial property of code self-stabilization. The top-down filtered inputs to F.sub.1 form a template pattern and enable the network to carry out attentional priming, pattern matching, and self-adjusting parallel search.
The orienting subsystem is one of the means by which an ART network carries out active regulation of the learning process. Attentional gain control 28 and 30 at F.sub.1 and F.sub.2 also contributes to this active regulation. Gain control acts to adjust overall sensitivity to patterned inputs and to coordinate the separate, synchronous functions of the ART system.
In accordance with particular features of the present invention, the A pattern recognition system comprises a feature representation field (F.sub.1) which receives input signals, defining an input pattern, and template signals. An A-category-representation in an A-category-representation field (F.sub.2) is selected based on a pattern from the feature representation field. Template signals are generated based on the selected A-category-representation. With a sufficient match between the template signals and the input signals, the A-category-representation selection and the template signals are adapted to the input signals. However, with an insufficient match the A-category-representation selection is reset. Preferably, a like pattern recognition subsystem responsive to a different set of input patterns defines the control B-category-representation.
More specifically, a new neural network architecture, called ARTMAP, autonomously learns to classify arbitrarily many, arbitrarily ordered vectors into recognition categories based on predictive success. This supervised learning system is built up from a pair of Adaptive Resonance Theory modules (ART.sub.a and ART.sub.b) that are capable of self-organizing stable recognition categories in response to arbitrary sequences of input patterns. During training trials, the ART.sub.a module receives a stream {a.sup.(p) } of input patterns, and ART.sub.b receives a stream {b.sup.(p) } of input patterns, where b.sup.(p) is the correct prediction given a.sup.(p). These ART modules are linked by an associative learning network and an internal controller that ensures autonomous system operation in real time. During test trials, the remaining patterns a.sup.(p) are presented without b.sup.(p), and their predictions at ART.sub.b are compared with b.sup.(p). Tested on a benchmark machine learning database in both on-line and off-line simulations, the ARTMAP system learns orders of magnitude more quickly, efficiently, and accurately than alternative algorithms, and achieves 100% accuracy after training on less than half the input patterns in the database. It achieves these properties by using an internal controller that conjointly maximizes predictive generalization and minimizes predictive error by linking predictive success to category size on a trial-by-trial basis, using only local operations. This computation increases the vigilance parameter ρ.sub.a of ART.sub.a by the minimal amount needed to correct a predictive error at ART.sub.b. Parameter ρ.sub.a calibrates the minimum confidence that ART.sub.a must have in a category, or hypothesis, activated by an input a.sup.(p) in order for ART.sub.a to accept that category, rather than search for a better one through an automatically controlled process of hypothesis testing. Parameter ρ.sub.a is compared with the degree of match between a.sup.(p) and the top-down learned expectation, or prototype, that is read-out subsequent to activation of an ART.sub.a category. Search occurs if the degree of match is less than ρ.sub.a. ARTMAP is hereby a type of self-organizing expert system that calibrates the selectivity of its hypotheses based upon predictive success. As a result, rare but important events can be quickly and sharply distinguished even if they are similar to frequent events with different consequences. Between input trials ρ.sub.a relaxes to a baseline vigilance ρ.sub.a. When ρ.sub.a is large, the system runs in a conservative mode, wherein predictions are made only if the system is confident of the outcome. Very few false-alarm errors then occur at any stage of learning, yet the system reaches asymptote with no loss of speed. Because ARTMAP learning is self-stabilizing, it can continue learning one or more databases, without degrading its corpus of memories, until its full memory capacity is utilized.
The architecture described herein forms part of Adaptive Resonance Theory, or ART, which was introduced in 1976.sup.3,4 in order to analyze how brain networks can autonomously learn in real time about a changing world in a rapid but stable fashion. Since that time, ART has steadily developed as physical theory to explain and predict ever larger data bases about cognitive information processing and its neural substrates.sup.5-8. A parallel development has described a series of rigorously characterized neural architectures--called ART 1, ART 2, and ART 3 --with increasingly powerful learning, pattern recognition, and hypothesis testing capabilities.sup.1,9-11.
The present class of architectures are called Predictive ART architectures because they incorporate ART modules into systems that can learn to predict a prescribed m-dimensional output vector b given a prescribed n-dimensional input vector a (FIG. 2). The present example of Predictive ART is called ARTMAP because its transformation from vectors in .sup.n to vectors in .sup.m defines a map that is learned by example from the correlated pairs {a.sup.(p), b.sup.(p) } of sequentially presented vectors, p=1, 2, . . . .sup.12. For example, the vectors a.sup.(p) may encode visual representations of objects, and the vectors b.sup.(p) may encode their predictive consequences, such as different tastes in the banana example above. The degree of code compression in memory is an index of the system's ability to generalize from examples.
An overview of the system is presented in FIG. 2. It includes two ART modules ART.sub.a and ART.sub.b. Each module includes the usual feature representation short term memory field F.sub.1 and category representation short term memory field F.sub.2. Thus, ART.sub.a includes short term fields F.sub.1.sup.a and F.sub.2.sup.a, while ART.sub.b includes short term memory fields F.sub.1.sup.b and F.sub.2.sup.b. Thus, a pattern represented by the Vector a selects, through a long term memory adaptive filter 22a, a category representation in field F.sub.2.sup.a. That category representation may itself be a pattern within F.sub.2.sup.a, but typically a single category is chosen. In the adaptive filter 22a, each element of the vector in F.sub.1.sup.a is weighted toward each category node of F.sub.2.sup.a. However, for clarification only a few weighted connections 22a of the bottom-up adaptive filter are illustrated. A template from the selected category which defines an expected pattern is generated by a top-down adaptive filter 24a. Top-down weights are provided from all nodes of F.sub.2.sup.a to all nodes of F.sub.1.sup.a but for clarification only a few weighted connections 24a are illustrated.
As in a conventional ART system, the top-down template is compared with the input vector a in F.sub.1.sup.a against a vigilance parameter. If the match is sufficient, as determined by an internal control system 24, the initial selection is maintained. If the match is insufficient, the previously selected category is no longer considered and another category is selected by the bottom-up adaptive filter. In this respect, ART.sub.a is conventional.
In accordance with the present invention, ART.sub.a is associated with another ART module ART.sub.b. ART.sub.b operates in the same fashion as ART.sub.a but receives different inputs. For example, ART.sub.a may receive a visual representation vector while ART.sub.b receives a taste representation vector. The internal control systems of the two ART systems are linked by control system 25 in a manner described below. The categories which are selected by the ART.sub.a and ART.sub.b modules from associated inputs a and b are associated in an associative memory 26. Operation of this memory as a map field is described below.
The resonant focus of attention is a consequence of a matching rule called the 2/3 Rule.sup.9. This rule clarifies how a bottom-up input pattern can supraliminally activate its feature detectors at the level F.sub.1 of an ART network, yet a top-down expectation can only subliminally sensitize, or prime, the level F.sub.1. Supraliminal activation means that F.sub.1 can automatically generate output signals that initiate further processing of the input. Subliminal activation means that F.sub.1 cannot generate output signals that initiate further processing of the input. Subliminal activation means that F.sub.1 cannot generate output signals, but its primed cells can more easily be activated by bottom-up inputs. For example, the verbal command "Look for the yellow banana" can prime visual feature detectors to respond more sensitively to visual inputs that represent a yellow banana, without forcing these cells to be fully activated, which would have caused a visual hallucination.
Carpenter and Grossberg.sup.6 have shown that the 2/3 Rule is realized by a kind of analog spatial logic. This logical operation computes the spatial intersection of bottom-up and top-down information. The spatial intersection is the focus of attention. It is of interest that sublimal top-down priming, which instantiates a type of "intentionality" in an ART system implies a type of matching law, which instatiates a type of "logic." Searle.sup.17 and others have criticized some AI models because they sacrifice intentionality for logic. In ART, intentionality implies logic.
As discussed above, the main elements of an ARTMAP system are shown in FIG. 2. Two modules, ART.sub.a and ART.sub.b read vector inputs a and b. If ART.sub.a and ART.sub.b were disconnected, each module would self-organize category groupings for the separate input sets. In the application described below, ART.sub.a and ART.sub.b are fast-learn ART 1 modules coding binary input vectors. ART.sub.a and ART.sub.b are here connected by an inter-ART module that in many ways resembles ART 1. This inter-ART module includes a Map Field 26 that controls the learning of an associative map from ART.sub.a recognition categories to ART.sub.b recognition categories. This map does not directly associate exemplars a and b, but rather associates the compressed and symbolic representations of families of exemplars a and b. The Map Field also controls match tracking of the ART.sub.a vigilance parameter. A mismatch at the Map Field between the ART.sub.a category activated by an input a and the ART.sub.b category activated by the input b increases ART.sub.a vigilance by the minimum amount needed for the system to search for and, if necessary, learn a new ART.sub.a category whose prediction matches the ART.sub.b category.
This inter-ART vigilance resetting signal is a form of "back propagation" of information, but one that differs from the back propagation that occurs in the Back Propagation network. For example, the search initiated by the inter-ART reset can shift attention to a novel cluster of visual features that can be incorporated through learning into a new ART.sub.a recognition category. This process is analogous to learning a category for "green bananas" based on "taste" feedback. However, these events do not "back propagate" taste features into the visual representation of the bananas, as can occur using the Back Propagation network. Rather, match tracking reorganizes the way in which visual features are grouped, attended, learned, and recognized for purposes of predicting an expected taste.
ART modules ART.sub.a and ART.sub.b Each ART module in FIG. 2 establishes compressed recognition codes in response to sequences of input patterns a and b. Associative learning at the Map Field 26 links pairs of pattern classes via these compressed codes. One type of generalization follows immediately from this learning strategy: If one vector a is associated with a vector b, then any other input that activates a's category node will predict the category of pattern b. Any ART module can be used to self-organize the ART.sub.a and ART.sub.b categories. In the poison mushroom application below, a and b are binary vectors, so ART.sub.a and ART.sub.b can be ART 1 modules. The main computations of the ART 1 module will here be outlined. A full definition of ART 1 modules, as systems of differential equations, along with an analysis of their network dynamics, can be found in Carpenter and Grossberg.sup.9. For other applications ART 2 and ART 3 modules may be more appropriate.
In an ART 1 module, an input pattern I is represented in field F.sub.1 and the recognition category for I is represented in field F.sub.2. We consider the case where the competitive field F.sub.2 makes a choice and where the system is operating in a fast-learn mode, as defined below. An algorithm for simulations is given below.
FIG. 3 illustrates the main components of an ART 1 module. The binary vector I forms the bottom-up input to the field F.sub.1 whose activity vector is denoted x. The competitive field F.sub.2 is designed to make a choice. Adaptive pathways 22 lead from each F.sub.1 node to all F.sub.2 nodes, and pathways 24 lead from each F.sub.2 node to all F.sub.1 nodes. Only sample pathways are shown. Reset occurs when the match between x and I fails to meet the criterion established by the vigilance parameter ρ. All paths are excitatory unless marked with a minus sign. A field of M nodes F.sub.1 with output vector x≡(x.sub.1, . . . , x.sub.M) registers the F.sub.0 →F.sub.1 input vector I≡(I.sub.1, . . . , I.sub.M). Each F.sub.1 node can receive input from three sources: the F.sub.0 →F.sub.1 bottom-up input; nonspecific gain controls 25 and 30; and top-down signals 24 from the N nodes of F.sub.2, via an F.sub.2 → F.sub.1 adaptive filter. A node is said to be active if it generates an output signal equal to 1. Output from inactive nodes equals 0. In ART 1 an F.sub.1 node is active if at least 2 of the 3 input signals are large. This rule for F.sub.1 activation is called the 2/3 Rule. The 2/3 Rule is realized in its simplest, dimensionless form as follows.
The ith F.sub.1 node is active if its net input exceeds a fixed threshold. Specifically, ##EQU1## where term I.sub.i is the binary F.sub.0 →F.sub.1 input, term g.sub.1 is the binary nonspecific F.sub.1 gain control signal, term Σy.sub.j z.sub.ji is the sum of F.sub.2 →F.sub.1 signals y.sub.j via pathways with adaptive weights z.sub.ji and z is a constant such that
0&amp;lt;z&amp;lt;1.                                                     (2)
The signal g.sub.1 from the F.sub.1 gain control 30 is defined by ##EQU2## Note that F.sub.2 activity inhibits F.sub.1 gain, as shown in FIG. 3. These laws for F.sub.1 activation imply that, if F.sub.2 is inactive, ##EQU3## If exactly one F.sub.2 node J is active, the sum Σy.sub.j z.sub.ji in (1) reduces to the single term z.sub.Ji, so ##EQU4##
Let T.sub.j denote the total input from F.sub.1 to jth F.sub.2 node, given by ##EQU5## where the Z.sub.ij denote the F.sub.1 →F.sub.2 adaptive weights. If some T.sub.j &gt;0, define the F.sub.2 choice index J by
In the typical case, J is uniquely defined. Then the F.sub.2 output vector y=(y.sub.1, . . . , y.sub.N) obeys ##EQU6## If two or more indices j share maximal input, then they equally share the total activity. This case is not considered here.
(a&#8745;b).sub.i =1 a.sub.i =1 and b.sub.i =1.          (10)
All ART 1 learning is gated by F.sub.2 activity; that is, the adaptive weights z.sub.Ji and Z.sub.iJ can change only when the Jth F.sub.2 node is active. Then both F.sub.2 →F.sub.1 and F.sub.1 →F.sub.2 weights are functions of the F.sub.1 vector x, as follows.
Top-down F.sub.2 →F.sub.1 weights in active paths learn x; that is, when the Jth F.sub.2 node is active
Z.sub.Ji &#8594;X.sub.i.                                  (11)
All other Z.sub.ji remain unchanged. Stated as a differential equation, this learning rule is ##EQU8## In (12), learning by Z.sub.ji is gated by y.sub.j. When the y.sub.j gate opens--that is, when y.sub.j &gt;0--then learning begins and Z.sub.ji is attracted to X.sub.i. In vector terms, if y.sub.j &gt;0, then Z.sub.j ≡(Z.sub.j1, Z.sub.j2, . . . , Z.sub.jM) approaches x. Such a law is therefore sometimes called learning by gated steepest descent. It is also called the outstar learning rule, and was introduced into the neural modelling literature in 1969.sup.23.
Thus with fast learning, the top-down weight vector Z.sub.j is a binary vector at the start and end of each input presentation. By (4), (5), (10), (11), and (13), the F.sub.1 activity vector can be described as ##EQU9## By (5) and (12), when node J is active, learning causes
Z.sub.J &#8594;I&#8745;Z.sub.j.sup.(old)                (15)
where Z.sub.J.sup.(old) denotes Z.sub.J at the start of the input presentation. By (11) and (14), x remains constant during learning, even though The first time an F.sub.2 node J becomes active, it is said to be uncommitted. Then, by (13)-(15),
Z.sub.J &#8594;I                                          (16)
In simulations it is convenient to assign initial values to the bottom-up F.sub.1 →F.sub.2 adaptive weights Z.sub.ij in such a way that F.sub.2 nodes first become active in the order j=1,2, . . . . This can be accomplished by letting
a.sub.1 &amp;gt;a.sub.2. . . &amp;gt;a.sub.N.                            (18)
Like the top-down weight vector Z.sub.J, the bottom-up F.sub.1 →F.sub.2 weight vector Z.sub.J ≡(Z.sub.1J . . . Z.sub.iJ . . . Z.sub.MJ) also becomes proportional to the F.sub.1 output vector x when the F.sub.2 node J is active. In addition, however, the bottom-up weights are scaled inversely to ##EQU10## where β&gt;0. This F.sub.1 →F.sub.2 learning law, called the Weber Law Rule.sup.9, realizes a type of competition among the weights Z.sub.J adjacent to a given F.sub.2 node J. This competitive computation could alternatively be transferred to the F.sub.1 field, as it is in ART 2. By (14), (15), and (19), during learning ##EQU11##
The Z.sub.ij initial values are required to be small enough so that an input I that perfectly matches a previously learned vector Z.sub.J will select the F.sub.2 node J rather than an uncommitted node. This is accomplished by assuming that ##EQU12## for all F.sub.0 →F.sub.1 inputs I. When I is first presented, x=I, so by (6), (15), (17), and (20), the F.sub.1 →F.sub.2 input vector T≡(T.sub.1,T.sub.2, . . . , T.sub.N) is given by ##EQU13## In the simulations below, β is taken to be so small that, among committed nodes, T.sub.j is determined by the size of taken to be so small that an uncommitted node will generate the maximum T.sub.j value in (22) only if all committed nodes. Larger values of a.sub.j and β bias the system toward earlier selection of uncommitted nodes when only poor matches are to be found among the committed nodes. A more complete discussion of this aspect of ART 1 system design is given by Carpenter and Grossberg.sup.9.
By (7), (21), and (22), a committed F.sub.2 node J may be chosen even if the match between I and z.sub.j is poor; the match need only be the best one available. If the match is too poor, then the ART 1 system can autonomously carry out hypothesis testing, or search, for a better F.sub.2 recognition code. This search process is mediated by the orienting subsystem, which can reset F.sub.2 nodes in response to poor matches at F.sub.1 (FIG. 3). The orienting subsystem is a type of novelty detector that measures system confidence. If the degree of match between bottom-up input I and top-down weight vector z.sub.j is too poor, the system's confidence in the recognition code labelled by J is inadequate. Otherwise expressed, the input I is too unexpected relative to the top-down vector z.sub.j, which plays the role of a learned top-down expectation.
An unexpected input triggers a novelty burst at the orienting subsystem, which sends a nonspecific reset wave r from the orienting subsystem to F.sub.2. The reset wave enduringly shuts off node j so long as input I remains on. With J off and its top-down F.sub.2 →F.sub.1 signals silent, F.sub.1 can again instate vector x=I, which leads to selection of another F.sub.2 node through the bottom-up F.sub.2 →F.sub.1 adaptive filter. This hypothesis testing process leads to activation of a sequence of F.sub.2 nodes until one is chosen whose vector of adaptive weights forms an adequate match with I, or until an uncommitted node is selected. The search takes place so rapidly that essentially no learning occurs on that time scale. Learned weights are hereby buffered against recoding by poorly matched inputs that activate unacceptable F.sub.2 recognition codes. Thus, during search, previously learned weights actively control the search for a better recognition code without being changed by the signals that they process.
As noted above, the degree of match between bottom-up input I and top-down expectation z.sub.j is evaluated at the orienting subsystem, which measures system confidence that category J adequately represents input I. A reset wave is triggered only if this confidence measure falls below a dimensionless parameter P that is called the vigilance parameter. The vigilance parameter calibrates the system's sensitivity to disconfirmed expectations.
Step 2--With J active, compare the F.sub.1 output vector x=I∩z.sub.J with the F.sub.0 →F.sub.1 input vector I at the orienting subsystem 26, 28 (FIG. 3).
Step 3A--Suppose that I∩z.sub.J fails to match I at the level required by the vigilance criterion, i.e., that ##EQU14## Then F.sub.2 reset occurs: node J is shut off for the duration of the input interval during which I remains on. The index of the chosen F.sub.2 node is reset to the value corresponding to the next highest F.sub.1 →F.sub.2 input T.sub.j. With the new node active, Steps 2 and 3A are repeated until the chosen nodes satisfies the resonance criterion in Step 3B. Note that reset never occurs if
Step 3B--Suppose that I∩z.sub.J meets the criterion for resonance; i.e., that
Then the search ceases and the last chosen F.sub.2 node J remains active until input I shuts off (or until ρ increases). In this state, called resonance, both the F.sub.1 →F.sub.2 and the F.sub.2 →F.sub.1 adaptive weights approach new values if I∩z.sub.J.sup.(old) ≠z.sub.J.sup.(old). Note that resonance cannot occur if ρ&gt;1.
If ρ≦1, search ceases whenever I .sub.J, as is the case if an uncommitted node J is chosen. If vigilance is close to 1, then reset occurs if F.sub.2 →F.sub.1 input alters the F.sub.1 activity pattern at all; resonance requires that I be a subset of z.sub.J. If vigilance is near 0, reset never occurs. The top-down expectation z.sub.J of the first chosen F.sub.2 node J is then recoded from z.sub.J.sup.(old) to I∩z.sub.J.sup.(old), even if I and z.sub.J.sup.(old) are very different vectors.
For simplicity, ART 1 is exposed to discrete presentation intervals during which an input is constant and after which F.sub.1 and F.sub.2 activities are set to zero. Discrete presentation intervals are implemented in ART 1 by means of the F.sub.1 and F.sub.2 gain control signals g.sub.1 and g.sub.2 (FIG. 5). The F.sub.2 gain signal g.sub.2 is assumed, like g.sub.1 in (3), to be 0 if F.sub.0 is inactive. Then, when F.sub.0 becomes active., g.sub.2 and F.sub.2 signal thresholds are assumed to lie in a range where the F.sub.2 node that receives the largest input signal can become active. When an ART 1 system is embedded in a hierarchy, F.sub.2 may receive signals from sources other than F.sub.1. This occurs in the ARTMAP system described below. In such a system, F.sub.2 still makes a choice and gain signals from F.sub.0 are still required to generate both F.sub.1 and F.sub.2 output signals. In the simulations, F.sub.2 nodes that are reset during search remain off until the input shuts off. A real-time ART search mechanism that can cope with continuously fluctuating analog or binary inputs of variable duration, fast or slow learning, and compressed or distributed F.sub.2 codes is described by Carpenter and Grossberg.sup.11.
The Map Field A Map Field module 26 links the F.sub.2 fields of the ART.sub.a and ART.sub.b modules. FIG. 4 illustrates the main components of the Map Field. We will describe one such system in the fast-learn mode with choice at the fields F.sub.2.sup.a and F.sub.2.sup.b. As with the ART 1 and ART 2 architectures themselves.sup.9,10, many variations of the network architecture lead to similar computations. In the ARTMAP hierarchy, ART.sub.a, ART.sub.b and Map Field modules are all described in terms of ART 1 variables and parameters. Indices a and b identify terms in the ART.sub.a and ART.sub.b modules, while Map Field variables and parameters have no such index. Thus, for example, ρ.sub.a, ρ.sub.b, and ρ denote the ART.sub.a, ART.sub.b, and Map Field vigilance parameters, respectively.
Both ART.sub.a and ART.sub.b are fast-learn ART 1 modules. With one optional addition, they duplicate the design described above. That addition, called complement coding, represents both the on-response to an input vector and the off-response to that vector. This ART coding strategy has been shown to play a useful role in searching for appropriate recognition codes in response to predictive feedback 24,25. To represent such a code in its simplest form, let the input vector a itself represent the on-response, and the complement of a, denoted by a.sup.c, represent the off-response, for each ART.sub.a input vector a. If a is the binary vector (a.sub.1, . . . ,a.sub.Ma), the input to ART.sub.a in the 2M.sub.a -dimensional binary vector.
(a,a.sup.c)&#8801;(a.sub.1, . . . , a.sub.Ma,a.sub.1.sup.c, . . . , a.sub.Ma.sup.c)                                           (26)
The utility of complement coding for searching an ARTMAP system will be described below. Conditions will also be given where complement coding is not needed. In fact, complement coding was not needed for any of the simulations described below, and the ART.sub.a input was simply the vector a.
In the discussion of the Map Field module below, F.sub.2.sup.a nodes, indexed by j=1 . . . N.sub.a, have binary output signals y.sub.j.sup.a ; and F.sub.2.sup.b nodes indexed by k=1 . . . N.sub.b, have binary output signals y.sub.k.sup.b. Correspondingly, the index of the active F.sub.2.sup.a node is denoted by J, and the index of the active F.sub.2.sup.b node is denoted by K. Because the Map Field is the interface where signals from F.sub.2.sup.a and F.sub.2.sup.b interact, it is denoted by F.sup.ab. The nodes of F.sup.ab have the same index k, k=1,2, . . . , N.sub.b as the nodes of F.sub.2.sup.b because there is a one-to-one correspondence between these sets of nodes.
Each node of F.sup.ab can receive input from three sources: F.sub.2.sup.a, F.sub.2.sup.b, and Map Field gain control 32 (signal G). The F.sup.ab output vector x obeys the 2/3 Rule of ART 1; namely, ##EQU15## where term y.sub.k.sup.b is the F.sub.2.sup.b output signal, term G is a binary gain control signal, term Σy.sub.j.sup.a w.sub.jk is the sum of F.sub.2.sup.a →F.sup.ab signals y.sub.j.sup.a via pathways with adaptive weights w.sub.jk, and w is a constant such that
0&amp;lt;w&amp;lt;1.                                                     (29)
Values of the gain control signal G and the F.sub.2.sup.a →F.sup.ab weight vectors w.sub.j ≡(w.sub.j1, . . . , w.sub.jNb), j=1 . . . N.sub.a, are specified below.
Comparison of (1) and (28) indicates an analogy between fields F.sub.2.sup.b, F.sup.ab, and F.sub.2.sup.a in a Map Field module and fields F.sub.0, F.sub.1, and F.sub.2, respectively, in an ART 1 module. Differences between these modules include the bidirectional non-adaptive connections between F.sub.2.sup.b and F.sup.ab in the Map Field module (FIG. 4) compared to the bidirectional adaptive connections between fields F.sub.1 and F.sub.2 in the ART 1 module (FIG. 3). These different connectivity schemes require different rules for the gain control signals G and g.sub.1.
If an active F.sub.2.sup.a node J has not yet learned a prediction, the ARTMAP system is designed so that J can learn to predict any ART.sub.b pattern if one is active or becomes active while J is active. This design constraint is satisfied using the assumption, analogous to (13), that F.sub.2.sup.a →F.sup.ab initial values
for j =1 . . . N.sub.a and k=1 . . . N.sub.b.
Rules governing G and w.sub.j (0) enable the following Map Field properties to obtain. If both ART.sub.a and ART.sub.b are active, then learning of ART.sub.a →ART.sub.b associations can take place at F.sup.ab. If ART.sub.a is active but ART.sub.b is not, then any previously learned ART.sub.a →ART.sub.b prediction is read out at F.sup.ab. If ART.sub.b is active but ART.sub.a is not, then the selected ART.sub.b category is represented at F.sup.ab. If neither ART.sub.a nor ART.sub.b is active, then F.sup.ab is not active. By (28)-(31), the 2/3 Rule realizes these properties in the following four cases.
1.) F.sub.2.sup.a active and F.sub.2.sup.b active--If both the F.sub.2.sup.a category node J and the F.sub.2.sup.b category node K are active, then G=0 by (30). Thus by (28), ##EQU17## All x.sub.k =0 for k≠K. Moreover x.sub.K =1 only if an association has previously been learned in the pathway from node J to node K, or if J has not yet learned to predict any ART.sub.b category. If J predicts any category other than K, then all x.sub.k 32 0.
2.) F.sub.2.sup.a active and F.sub.2.sup.b inactive--If the F.sub.2.sup.a node J is active and F.sub.2.sup.b is inactive, then G=1. Thus ##EQU18## By (31) and (33), if an input a has activated node J in F.sub.2.sup.a but F.sub.2.sup.b is not yet active, J activates all nodes k in F.sup.ab if J has learned no predictions. If prior learning has occurred, all nodes k are activated whose adaptive weights w.sub.Jk are still large.
3.) F.sub.2.sup.b active and F.sub.2.sup.a inactive--If the F.sub.2.sup.b node K is active and F.sub.2.sup.a is inactive, then G=1. Thus ##EQU19## In this case, the F.sup.ab output vector x is the same as the F.sub.2.sup.a output vector y.sup.b.
F.sub.2.sup.b choice and priming is as follows. If ART.sub.b receives an input b while ART.sub.a has no input, then F.sub.2.sup.b chooses the node K with the largest F.sub.1.sup.b →F.sub.2.sup.b input. Field F.sub.2.sup.b then activates the Kth F.sup.ab node, and F.sup.ab →F.sub.2.sup.b feedback signals support the original F.sub.1.sup.b →F.sub.2.sup.b choice. If ART.sub.a receives an input a while ART.sub.b has no input, F.sub.2.sup.a chooses a node J. If, due to prior learning, some w.sub.JK =1 while all other w.sub.Jk =0, we say that a predicts the ART.sub.b category K, as F.sup.ab sends its signal vector x to F.sub.2.sup.b. Field F.sub.2.sup.b is hereby attentionally primed, or sensitized, but the field remains inactive so long as ART.sub.b has no input from F.sub.0.sup.b. If then an F.sub.0.sup.b →F.sub.1.sup.b input b arrives, the F.sub.2.sup.b choice depends upon network parameters and timing. It is natural to assume, however, that b simultaneously activates the F.sub.1.sup.b and F.sub.2.sup.b gain control signals g.sub.1.sup.b and g.sub.2.sup.b (FIG. 3). Then F.sub.2.sup.b processes the F.sup.ab prime x as soon as F.sub.1.sup.2 processes the input b, and F.sub.2.sup.b chooses the primed node K. Field F.sub.1.sup.b then receives F.sub.2.sup.b →F.sub.1.sup.b expectation input z.sub.K.sup.b as well as F.sub.0.sup.b →F.sub.1.sup.b input b, leading either to match or reset.
F.sub.2.sup.a →F.sup.ab learning laws are as follows. The F.sub.2.sup.a →F.sup.ab adaptive weights w.sub.jk obey an outstar learning law similar to that governing the F.sub.2 →F.sub.1 weights z.sub.ji in (12); namely, ##EQU20## According to (35), the F.sub.2.sup.a →F.sup.ab weight vector w.sub.j approaches the F.sup.ab activity vector x if the Jth F.sub.2.sup.a node is active. Otherwise w.sub.j remains constant. If node J has not yet learned to make a prediction, all weights w.sub.Jk equal 1, by (31). In this case, if ART.sub.b receives no input b, then all x.sub.k values equal 1 by (33). Thus, by (35), all w.sub.jk values remain equal to 1. As a result, category choices in F.sub.2.sup.a do not alter the adaptive weights w.sub.jk until these choices are associated with category choices in F.sub.2.sup.b.
The Map Field provides the control that allows the ARTMAP system to establish different categories for very similar ART.sub.a inputs that make different predictions, while also allowing very different ART.sub.a inputs to form categories that make the same prediction. In particular, the Map Field orienting subsystem 34, 36 becomes active only when ART.sub.a makes a prediction that is incompatible with the actual ART.sub.b input. This mismatch event activates the control strategy, called match tracking, that modulates the ART.sub.a vigilance parameter ρ.sub.a in such a way as to keep the system from making repeated errors. As illustrated in FIG. 4, a mismatch at F.sup.ab while F.sub.2.sup.b is active triggers an inter-ART reset signal R to the ART.sub.a orienting subsystem. This occurs whenever ##EQU21## where ρ denotes the Map Field vigilance parameter. The entire cycle of ρ.sub.a adjustment proceeds as follows through time. At the start of each input presentation, ρ.sub.a equals a fixed baseline vigilance ρ.sub.a. When an input a activates an F.sub.2.sup.a category node J and resonance is established, ##EQU22## as in (25). Thus, there is no reset r.sub.a generated by the ART.sub.a orienting subsystem 38, 39 (FIG. 5). An inter-ART reset signal R is sent to ART.sub.a if the ART.sub.b category predicted by a fails to match the active ART.sub.b category, by (36). The inter-ART reset signal R raises ρ.sub.a to a value that is just high enough to cause (37) to fail, so that ##EQU23## Node J is therefore reset and an ART.sub.a search ensues. Match tracking continues until an active ART.sub.a category satisfies both the ART.sub.a matching criterion (37) and the analogous Map Field matching criterion. Match tracking increases the ART.sub.a vigilance by the minimum amount needed to abort an incorrect ART.sub.a →ART.sub.b prediction and to drive a search for a new ART.sub.a category that can establish a correct prediction. As shown by example below, match tracking allows a to make a correct prediction on subsequent trials, without repeating the initial sequence of errors. Match tracking hereby conjointly maximizes predictive generalization and minimizes predictive error on a trial-by-trial basis, using only local computations.
The operation of match tracking can be implemented in several different ways. One way is to use a variation on the Vector Integration to Endpoint, or VITE, circuit.sup.26 as follows. Let an ART.sub.a binary reset signal r.sub.a (FIG. 5) obey the equation ##EQU24## as in (23). The complementary ART.sub.a resonance signal r.sub.a.sup.c =1-r.sub.a. Signal R equals 1 during inter-ART rest; that is, when inequality (36) holds. The size of the ART.sub.a vigilance parameter ρ.sub.a at 38 is determined by the match tracking equation ##EQU25## where γ&gt;1. During inter-ART reset, R=r.sub.a =1, causing ρ.sub.a to increase until r.sub.a.sup.c =0. Then ρ.sub.a tracking (38). When r.sub.a.sup.c =0, ρ.sub.a relaxes to ρ.sub.a. This is assumed to occur at a rate slower than node activation, also called short term memory (STM), and faster than learning, also called long term memory (LTM). Such an intermediate rate is called medium term memory (MTM).sup.11. Thus the higher vigilance with continued input to F.sub.0.sup.a will be maintained for subsequent search.
An ART.sub.a search that is triggered by increasing ρ.sub.a according to (40) ceases if some active F.sub.2.sup.a node J satisfies
If no such node exists, F.sub.2.sup.a shuts down for the rest of the input presentation. In particular, if a z.sub.J.sup.a, match tracking makes ρ.sub.a &gt;1, so a cannot activate another category in order to learn the new prediction. The following anomalous case can thus arise. Suppose that a=z.sub.J.sup.a but the ART.sub.b input b mismatches the ART.sub.b expectation z.sub.K.sup.b previously associated with J. Then match tracking will prevent the recoding that would have associated a with b. That is, the ARTMAP system with fast learning and choice will not learn the prediction of an exemplar that exactly matches a learned prototype when the new prediction contradicts the previous predictions of the exemplars that created the prototype. This situation does not arise when all ART.sub.a inputs a have the same number of 1's, as follows.
When an ART.sub.a category node J becomes committed to input a, then the 2/3 Rule (15), z.sub.J.sup.a can be recoded only by decreasing its number of 1 entries, and thus its norm. Once this occurs, no input a can ever be a subset of z.sub.J.sup.a, by (42). In particular, the situation described in the previous section cannot arise.
In the simulations reported below, all ART.sub.a inputs have norm 22. Equation (42) can also be satisfied by using complement coding, since complement coding thus ensures that the system will avoid the case where some input a is a proper subset of the active ART.sub.a prototype z.sub.J.sup.a and the learned prediction of category J mismatches the correct ART.sub.b pattern.
Finally, note that with ARTMAP fast learning and choice, an ART.sub.a category node J is permanently committed to the first ART.sub.b category node K to which it is associated. However, the set of input exemplars that access either category may change through time, as in the banana example described in the introduction.
The role of match tracking is illustrated by the following example. The input pairs shown in Table 1 are presented in order a.sup.(1), b.sup.(1)), (a.sup.(2), b.sup.(2)), (a.sup.(3), b.sup.(3)). The problem solved by match tracking is created by vector a.sup.(2) lying "between" a.sup.(1) and a.sup.(3), with a.sup.(1) a.sup.(2) a.sup.(3), while a.sup.(1) and a.sup.(3) are mapped to the same ART.sub.b vector. Suppose that, instead of match tracking, the Map Field orienting subsystem merely activated the ART.sub.a reset system. Coding would then proceed as follows.
Choose ρ.sub.a ≦0.6 and ρ.sub.b &gt;0. Vectors a.sup.(1) then b.sup.(1) are presented, activate ART.sub.a and ART.sub.b categories J=1 and K =1, and the category J=1 learns to predict category K =1, thus associating a.sup.(1) with b.sup.(1). Next a.sup.(2) then b.sup.(2) are presented. Vector a.sup.(2) first activates J=1 without reset, since ##EQU26## However, node J=1 predicts node K=1. Since ##EQU27## ART.sub.b search leads to activation (selection) of a different F.sub.2.sup.b node, K=2. Because of the conflict between the prediction (K=1) made by the active F.sub.2.sup.a node and the currently active F.sub.2.sup.b node (K=2), the Map Field orienting subsystem resets F.sub.2.sup.a, but without match tracking. Thereafter a new F.sub.2.sup.a node (J=2) learns to predict the correct F.sub.2.sup.b node (K=2), associating a.sup.(2) with b.sup.(2).
Vector a.sup.(3) first activates J=2 without ART.sub.a reset, thus predicting K=2, with z.sub.2.sup.b =b.sup.(2). However, b.sup.(3) mismatches z.sub.2.sup.b, leading to activation of the F.sub.2.sup.b node K=1, since b.sup.(3) =b.sup.(1). Since the predicted node (K=2) then differs from the active node (K=1), the Map Field orienting subsystem again resets R.sub.2.sup.a. At this point, still without match tracking, the F.sub.2.sup.a node J=1 would become active, without subsequent ART.sub.a reset, since z.sub.1.sup.a =a.sup.(1) and ##EQU28## Since node J=1 correctly predicts the active node K=1, no further reset or new learning would occur. On subsequent prediction trials, vector a.sup.(3) would once again activate J=2 and then K=2. When vector b.sup.(3) is not presented, on a test trial, vector a.sup.(3) would not have learned its correct prediction; rather, b.sup.(2) would be incorrectly predicted.
With match tracking, when a.sup.(3) is presented, the Map Field orienting subsystem causes ρ.sub.a to increase to a value slightly greater than reset, node J=1 will also be reset because ##EQU29## The reset of node J=1 permits a.sup.(3) to choose an uncommitted F.sub.2.sup.a node (J=3) that is then associated with the active F.sub.2.sup.b node (K=1). Because vector a.sup.(3) is exactly learned at the new J=3 node, subsequent a.sup.(3) inputs will select J=3 directly at any vigilance; J=2 will not first be selected. Thereafter each ART.sub.a input predicts the correct ART.sub.b output without search or error unaffected by the lower base level of vigilance.
The utility of ART.sub.a complement coding is illustrated by the following example. Assume that the nested input pairs in Table 7 are presented to an ARTMAP system in order (a.sup.(3), b.sup.(3)), (a.sup.(2), b.sup.(2)), (a.sup.(1), b.sup.(1)), with match tracking but without complement coding. Choose ρ.sub.a &lt;0.5 and ρ.sub.b &gt;0.
Vectors a.sup.(3) and b.sup.(3) are presented and activate ART.sub.a and ART.sub.b categories J=1 and K=1. The system learns to predict b.sup.(3) given a.sup.(3) by associating the F.sub.2.sup.a node J=1 with the F.sub.2.sup.b node K=1.
Next a.sup.(2) and b.sup.(2) are presented. Vector a.sup.(2) first activates J=1 without reset, since ∩z.sub.1.sup.a ≧ρ.sub.a =ρ.sub.a. However, node J=1 predicts node K=1. As in the previous example, after b.sup.(2) is presented, the F.sub.2.sup.b node K=2 becomes active and leads to an inter-ART reset. Match tracking makes ρ.sub.a &gt;1, so F.sub.2.sup.a shuts down until the pair (a.sup.(2), b.sup.(2)) shuts off. Pattern b.sup.(2) is coded in ART.sub.b as z.sub.2.sup.b, but no learning occurs in the ART.sub.a and F.sup.ab modules.
Next a.sup.(1) activates J=1 without reset, since ∩z.sub.1.sup.a =1≧ρ.sub.a =ρ.sub.a. Since node J=1 predicts the correct pattern b.sup.(1) =z.sub.1.sup.b, no reset ensues. Learning does occur, however, since z.sub.a.sup.a shrinks to a.sup.(1). If each input can be presented only once, a.sup.(2) does not learn to predict b.sup.(2). However if the input pairs are presented repeatedly, match tracking allows ART.sub.a to establish 3 category nodes and an accurate mapping.
With complement coding, the correct map can be learned on-line for any ρ.sub.a &gt;0. The critical difference is due to the fact that than equaling 1 as before. Thus either ART.sub.a reset (if ρ.sub.a &gt;5/6) or match tracking (if ρ.sub.a ≦5/6) establishes a new ART.sub.a node rather than shutting down on that trial. On the next trial, a.sup.(1) also establishes a new ART.sub.a category that maps to b.sup.(1).
SIMULATION ALGORITHMS ART 1 Algorithm Fast-learn ART 1 with binary F.sub.0 →F.sub.1 input vector I and choice at F.sub.2 can be simulated by following the rules below. Fields F.sub.0 and F.sub.1 have M nodes and field F.sub.2 has N nodes.
Initially all F.sub.2 nodes are said to be uncommitted. Weights Z.sub.ij in F.sub.1 →F.sub.2 paths initially satisfy
Z.sub.ij (0)=&#945;.sub.j,                                (A1)
where Z.sub.j ≡(Z.sub.1j, . . . , Z.sub.Mj) denotes the bottom-up F.sub.1 →F.sub.2 weight vector. Parameters α.sub.j are ordered according to
&#945;.sub.1 &amp;gt;&#945;.sub.2 &amp;gt;. . . &amp;gt;&#945;.sub.N,        (A2)
where ##EQU30## for β&gt;0 and for any admissible F.sub.0 →F.sub.1 input I. In the simulations in this article, a.sub.j and β are small.
Weights z.sub.ji in F.sub.2 →F.sub.1 paths initially satisfy
The top-down, F.sub.2 →F.sub.1 weight vector (z.sub.j1, . . . , z.sub.jM) is denoted z.sub.j.
The binary F.sub.1 output vector x≡(x.sub.1, . . . , x.sub.M) is given by ##EQU31## The input T.sub.j from F.sub.1 to the jth F.sub.2 node obeys ##EQU32## The set of committed F.sub.2 nodes and update rules for vectors z.sub.j and Z.sub.j are defined iteratively below.
If F.sub.0 is active ( F.sub.2 is one node with index J satisfying ##EQU33## If more than one node is maximal, one of these is chosen at random. After an input presentation on which node J is chosen, J becomes committed. The F.sub.2 output vector is denoted by y≡(y.sub.1, . . . , y.sub.N).
where ρ is the ART 1 vigilance parameter. If such a node J exists, that node remains active, or in resonance, for the remainder of the input presentation. If no node satisfies (A8), F.sub.2 remains inactive after search, until I shuts off.
At the end of an input presentation the F.sub.2 →F.sub.1 weight vector Z.sub.J satisfies
Z.sub.J =I&#8745;z.sub.J.sup.(old)                       (A 9)
where z.sub.J.sup.(old) denotes z.sub.J at the start of the current input presentation. The F.sub.1 →F.sub.2 weight vector Z.sub.J satisfies ##EQU34##
ART.sub.a and ART.sub.b are fast-learn ART 1 modules. Inputs to ART.sub.a may, optionally, be in the complement code form. Embedded in an ARTMAP system, these modules operate as outlined above, with the following additions. First, the ART.sub.a vigilance parameter ρ.sub.a can increase during inter-ART reset according to the match tracking rule. Second, the Map Field F.sup.ab can prime ART.sub.b. That is, if F.sup.ab sends nonuniform input to F.sub.2.sup.b in the absence of an F.sub.0.sup.b →F.sub.1.sup.b input b, then F.sub.2.sup.b remains inactive. However, as soon as an input b arrives, F.sub.2.sup.b chooses the node K receiving the largest F.sup.ab →F.sub.2.sup.b input. Node K, in turn, sends to F.sub.1.sup.b the top-down input z.sub.K.sup.b. Rules for match tracking and complement coding are specified below.
Let x.sup.a ≡(x.sub.1.sup.a. . . x.sub.Ma.sup.a) denote the F.sub.1.sup.a output vector; let y.sup.a ≡(y.sub.1.sup.a. . . y.sub.Na.sup.a) denote the F.sub.2.sup.a output vector; let x.sup.b ≡(x.sub.1.sup.b. . . x.sub.Mb.sup.b) denote the F.sub.1.sup.b output vector; and let y.sup.b ≡(y.sub.1.sup.b. . . y.sub.Nb.sup.b) denote the F.sub.2.sup.b output vector. The Map Field F.sup.ab has N.sub.b nodes and binary output vector x. Vectors x.sup.a, y.sup.a, x.sup.b, y.sup.b, and x are set to 0 between input presentations.
Map Field learning is as follows. Weights w.sub.jk, where j=1 . . . N.sub.a and k=1 . . . N.sub.b, in F.sub.2.sup.a →F.sup.ab paths initially satisfy
Each vector (w.sub.j1, . . . w.sub.jNb) is denoted w.sub.j. During resonance with the ART.sub.a category J active, w.sub.j →x. In fast learning, once J learns to predict the ART.sub.b category K, that association is permanent; i.e., w.sub.JK =1 for all times.
Map Field activation is as follows. The F.sup.ab output vector x obeys ##EQU35##
Match tracking is as follows. At the start of each input presentation the ART.sub.a vigilance parameter ρ.sub.a equals a baseline vigilance ρ.sub.a. The Map Field vigilance parameter is ρ. If
then ρ.sub.a is increased until it is slightly larger than Then
where a is the current ART.sub.a input vector and J is the index of the active F.sub.2.sup.a node. When this occurs, ART.sub.a search leads either to activation of a new F.sub.2.sup.a node J with
(a,a.sup.c)&#8801;(a.sub.1. . . a.sub.Ma,a.sub.1.sup.c. . . a.sub.Ma.sup.a),(A17)
a.sub.i.sup.c &#8801;1- a.sub.i.                           (A18)
Complement coding may be useful if the following set of circumstances could arise: an ART.sub.a input vector a activates an F.sub.2.sup.a node J previously associated with an F.sub.2.sup.b node K; the current ART.sub.b input b mismatches z.sub.K.sup.b ; and a is a subset of z.sub.J.sup.a. These circumstances never arise if all constant. For the simulations in this article, ARTMAP processing The following nine cases summarize fast-learn ARTMAP system processing with choice at F.sub.2.sup.a and F.sub.2.sup.b and with Map Field vigilance ρ&gt;0. Inputs a and b could appear alone, or one before the other. Input a could make a prediction based on prior learning or make no prediction. If a does make a prediction, that prediction may be confirmed or disconfirmed by b. The system follows the rules outlined in the previous section assuming, as in the simulations, that all used. For each case, changing weight vectors z.sub.J.sup.a, z.sub.K.sup.b, and w.sub.K are listed. Weight vectors Z.sub.J.sup.a and Z.sub.K.sup.b change accordingly, by (A11). All other weights remain constant.
Case 1: a only, no prediction. Input a activates a matching F.sub.2.sup.a node J, possibly following ART.sub.a search. All F.sub.2.sup.a →F.sup.ab weights W.sub.Jk =1, so all x.sub.k =1. ART.sub.b remains inactive. With learning z.sub.J.sup.a →z.sub.J.sup.a(old) ∩a.
Case 2: a only, with prediction. Input a activates a matching F.sub.2.sup.a node J. Weight w.sub.JK =1 while all other w.sub.Jk =0, and x=w.sub.J. F.sub.2.sup.b is primed, but remains inactive. With learning, Z.sub.J.sup.a →z.sub.J.sup.a(old) ∩a.
Case 3: b only. Input b activates a matching F.sub.2.sup.b node K, possibly following ART.sub.b search. At the Map Field, x=y.sup.b. ART.sub.a remains inactive. With learning, z.sub.K.sup.b →z.sub.K.sup.b(old) ∩b.
Case 4: a then b, no prediction. Input a activates a matching F.sub.2.sup.a node J. All x.sub.k become 1 and ART.sub.b is inactive, as in Case 1. Input b then activates a matching F.sub.2.sup.b node K, as in Case 3. At the Map Field x→y.sup.b ; that is, x.sub.K =1 and other x.sub.k =0. With learning Z.sub.J.sup.a →z.sub.J.sup.a(old) ∩a, z.sub.K.sup.b →z.sub.K.sup.b(old) ∩b, and w.sub.J →y.sup.b ; i.e., J learns to predict K.
Case 5: a then b, with prediction confirmed. Input a activates a matching F.sub.2.sup.a node J, which in turn activates a single Map Field node K and primes F.sub.2.sup.b, as in Case 2. When input b arrives, the Kth F.sub.2.sup.b node becomes active and the prediction is confirmed; that is,
Note that K may not be the F.sub.2.sup.b node b would have selected without the F.sup.ab →F.sub.2.sup.b prime. With learning, z.sub.J.sup.a →z.sub.J.sup.a(old) ∩a and z.sub.K.sup.b →a.sub.K.sup.b(old) ∩b.
Case 6. a then b, prediction not confirmed. Input a activates a matching F.sub.2.sup.a node, which in turn activates a single Map Field node and primes F.sub.2.sup.b, as in Case 5. When input b arrives, (A19) fails, leading to reset of the F.sub.2.sup.b node via ART.sub.b reset. A new F.sub.2.sup.b node K that matches b becomes active. The mismatch between the F.sub.2.sup.a →F.sup.ab weight vector and the new F.sub.2.sup.b vector y.sup.b sends Map Field activity x to 0, by (A12), leading to Map Field reset, by (A13). By match tracking, ρ.sub.a grows until (A14) holds. This triggers an ART.sub.a search that will continue until, for an active F.sub.2.sup.a node J, w.sub.JK =1, and (A15) holds. If such an F.sub.2.sup.a node does become active, learning will follow, setting z.sub.J.sup.a →z.sub.J.sup.a(old) ∩a and z.sub.K.sup.b →z.sub.K.sup. b(old) ∩b. If the F.sub.2.sup.a node J is uncommitted, learning sets w.sub.J →y.sup.b. If no F.sub.2.sup.a node J that becomes active satisfies (A15) and (A16), F.sub.2.sup.a shuts down until the inputs go off. In that case, with learning, z.sub.K.sup.b →z.sub.K.sup.b(old) ∩b.
Case 7: b then a, no prediction. Input b activates a matching F.sub.2.sup.b node K, then X=y.sup.b, as in Case 3. Input a then activates a matching F.sub.2.sup.a node J with all w.sub.Jk =1. At the Map Field, x remains equal to y.sup.b. With learning, z.sub.J.sup.a →z.sub.J.sup.a (old) ∩a, w.sub.J →y.sup.b, and z.sub.K.sup.b →z.sub.K.sup.b(old) ∩b.
Case 8: b then a, with prediction confirmed. Input b activates a matching F.sub.2.sup.b node K, then x=y.sup.b, as in Case 7. Input a then activates a matching F.sub.2.sup.a node J with w.sub.JK =1 and all other w.sub.Jk =0. With learning z.sub.J.sup.a →z.sub.J.sup.a(old) ∩a and z.sub.K.sup.b →z.sub.K.sup.b(old) ∩b.
Case 9: b then a, with prediction not confirmed. Input b activates a matching F.sub.2.sup.b node K, then x=y.sup.b and input a activates a matching F.sub.2.sup.a node, as in Case 8. However (A16) fails and x→0, leading to a Map Field reset. Match tracking resets ρ.sub.a as in Case 6, ART.sub.a search leads to activation of an F.sub.2.sup.a node (J) that either predicts K or makes no prediction, or F.sub.2.sup.a shuts down. With learning z.sub.K.sup.b →z.sub.K.sup.b(old) ∩b. If J exists, z.sub.J.sup.a →z.sub.J.sup.a(old) ∩a; and if J initially makes no prediction, w.sub.j →y.sup.b, i.e., J learns to predict K.
ARTMAP Simulations: Distinguishing Edible And Poisonous Mushrooms The ARTMAP system was tested on a benchmark machine learning database that partitions a set of vectors a into two classes. Each vector a characterizes observable features of a mushroom as a binary vector, and each mushroom is classified as edible or poisonous. The database represents the 11 species of genus Agaricus and the 12 species of the genus Lepiota described in The Audubon Society Field Guide to North American Mushrooms.sup.19. These two genera constitute most of the mushrooms described in the Field Guide from the family Agaricaceae (order Agaricales, class Hymenomycetes, subdivision Basidiomycetes, division Eumycota). All the mushrooms represented in the database are similar to one another: "These mushrooms are placed in a single family on the basis of a correlation of characteristics that include microscopic and chemical features . . .".sup.19 (p.500). The Field Guide warns that poisonous and edible species can be difficult to distinguish on the basis of their observable features. For example, the poisonous species Agaricus californicus is described as a "dead ringer" for the Meadow Mushroom, Agaricus campestris, that "may be known better and gathered more than any other wild mushroom in North America" (p. 505). This database thus provides a test of how ARTMAP and other machine learning systems distinguish rare but important events from frequently occurring collections of similar events that lead to different consequences.
The database of 8124 exemplars describes each of 22 observable features of a mushroom, along with its classification as poisonous (48.2%) or edible (51.8%). The 8124 "hypothetical examples" represent ranges of characteristics within each species; for example, both Agaricus californicus and Agaricus campestris are described as having a "white to brownish cap," so in the database each species has corresponding sets of exemplar vectors representing their range of cap colors. There are 126 different values of the 22 different observable features. A list of the observable features and their possible values is given in Table 2. For example, the observable feature of "cap-shape" has six possible values. Consequently, the vector inputs to ART.sub.a are 126-element binary vectors, each vector having 22 1's and 104 0's, to denote the values of an exemplar's 22 observable features. The ART.sub.b input vectors are (1,0) for poisonous exemplars and (0,1) for edible exemplars.
The ARTMAP system learned to classify test vectors rapidly and accurately, and system performance compares favorably with results of other machine learning algorithms applied to the same database. The STAGGER algorithm reached its maximum performance level of 95% accuracy after exposure to 1000 training inputs.sup.20. The HILLARY algorithm achieved similar results.sup.21. The ARTMAP system consistently achieved over 99% accuracy with 1000 exemplars, even counting "I don't know" responses as errors. Accuracy of 95% was usually achieved with on-line training on 300-400 exemplars and with off-line training on 100-200 exemplars. In this sense, ARTMAP was an order of magnitude more efficient than the alternative systems. In addition, with continued training, ARTMAP predictive accuracy always improved to 100%. These results are elaborated below.
Four types of on-line simulations were carried out, using two different baseline settings of the ART.sub.a vigilance parameter ρ.sub.a : ρ.sub.a =0 (forced choice condition) and ρ.sub.a =0.7 (conservative condition); and using sample replacement or no sample replacement. With sample replacement, any one of the 8124 input samples was selected at random for each input presentation. A given sample might thus be repeatedly encountered while others were still unused. With no sample replacement, a sample was removed from the input pool after it was first encountered. The replacement condition had the advantage that repeated encounters tended to boost predictive accuracy. The no-replacement condition had the advantage of having learned from a somewhat larger set of inputs at each point in the simulation. The replacement and no-replacement conditions had similar performance indices, all other things being equal. Each of the 4 conditions was run on 10 independent simulations. With ρ.sub.a =0, the system made a prediction in response to every input. Setting ρ.sub.a =0.7 increased the number of "I don't know" responses, increased the number of ART.sub.a categories, and decreased the rate of incorrect predictions to nearly 0%, even early in training. The ρ.sub.a =0.7 condition generally outperformed the ρ.sub.a =0 condition, even when incorrect predictions and "I don't know" responses were both counted as errors. The primary exception occurred very early in training, when a conservative system gives the large majority of its no-prediction responses.
Results are summarized in Table 3. Each entry gives the number of correct predictions over the previous 100 trials (input presentations), averaged over 10 simulations. For example, with ρ.sub.a =0 in the no-replacement condition, the system made, on the average, 94.9 correct predictions and 5.1 incorrect predictions on trials 201-300. In all cases a 95% correct-prediction rate was achieved before trial 400. With ρ.sub.a =0, a consistent correct-prediction rate of over 99% was achieved by trial 1400, while with ρ.sub.a =0.7 the 99% consistent correct-prediction rate was achieved earlier, by trial 800. Each simulation was continued for 8100 trials. In all four cases, the minimum correct-prediction rate always exceeded 99.5% by trial 1800 and always exceeded 99.8% by trial 2800. In all cases, across the total of 40 simulations summarized in Table 2, 100% correct prediction was achieved on the last 1300 trials of each run.
Note the relatively low correct-prediction rate for ρ.sub.a =0.7 on the first 100 trials. In the conservative mode, a large number of inputs initially make no prediction. With ρ.sub.a =0.7 an average total of only 2 incorrect predictions were made on each run of 8100 trials. Note too that Table 3 underestimates prediction accuracy at any given time, since performance almost always improves during the 100 trials over which errors are tabulated.
The role of repeated training set presentations was examined by comparing simulations that used the 100% training set accuracy criterion with simulations that used only a single presentation of each input during training. With only a few exceptions, performance was similar. In fact for ρ.sub.a =0.7, and for small training sets with ρ.sub.a =0, 100% training-set accuracy was achieved with single input presentations, so results were identical. Performance differences were greatest for ρ.sub.a =0 simulations with mid-sized training sets (60-500 samples), when 2-3 training set presentations tended to add a few more ART.sub.a learned category nodes. Thus, even a single presentation of training-then-testing inputs, carried out on-line, can be made to work almost as well as off-line training that uses repeated presentations of the training set. This is an important benefit of fast learning controlled by a match tracked search.
The simulations summarized in Table 4 illustrate off-line learning with ρhd a=0. In this forced choice case, each ART.sub.a input led to a prediction of poisonous or edible. The number of test set errors with small training sets was relatively large, due to the forced choice. The table illustrates system performance after training on input sets ranging in size from 3 to 4000 exemplars. Each line shows average correct and incorrect test set predictions over 10 independent simulations, plus the range of learned ART.sub.a category numbers.
Table 4 summarizes the average results over 10 simulations at each size training set. For example, with very small, 5-sample training sets, the system established between 1 and 5 ART.sub.a categories, and averaged 73.1% correct responses on the remaining 8119 test patterns. Success rates ranged from chance (51.8%, one category) in one instance where all five training set exemplars happened to be edible, to surprisingly good (94.2%, 2 categories). The range of success rates for fast-learn training on very small training sets illustrates the statistical nature of the learning process. Intelligent sampling of the training set or, as here, good luck in the selection of representative samples, can dramatically alter early success rates. In addition, the evolution of internal category memory structure, represented by a set of ART.sub.a category nodes and their top-down learned expectations, is influenced by the selection of early exemplars. Nevertheless, despite the individual nature of learning rates and internal representations, all the systems eventually converge to 100% accuracy on test set exemplars using only approximately) 1/600 as many ART.sub.a categories as there are inputs to classify.
As in the case of poisonous mushroom identification, it may be important for a system to be able to respond "I don't know" to a novel input, even if the total number of correct classifications thereby decreases early in learning. For higher values of the baseline vigilance ρ.sub.a, the ARTMAP system creates more ART.sub.a categories during learning and becomes less able to generalize from prior experience than when ρ.sub.a equals 0. During testing, a conservative coding system with ρ.sub.a =0.7 makes no prediction in response to inputs that are too novel, and thus initially has a lower proportion of correct responses. However, the number of incorrect responses is always low with ρ.sub.a =0.7, even with very few training samples, and the 99% correct-response rate is achieved for both forced choice (ρ.sub.a =0) and conservative (ρ.sub.a =0.7) systems with training sets smaller than 1000 exemplars.
Table 5 summarizes simulation results that repeat the conditions of Table 4 except that ρ.sub.a =0.7. Here, a test input that does not make a 70% match with any learned expectation makes an "I don't know" prediction. Compared with the ρ.sub.a =0 case of Table 4, Table 5 shows that larger training sets are required to achieve a correct prediction rate of over 95%. However, because of the option to make no prediction, the average test set error rate is almost always less than 1%, even when the training set is very small, and is less than 0.1% after only 500 training trials. Moreover, 100% accuracy is achieved using only (approximately) 1/130 as many ART.sub.a categories as there are inputs to classify.
Each ARTMAP category code can be described as a set of ART.sub.a feature values on 1 to 22 observable features, chosen from 126 feature values, that are associated with the ART.sub.b identification as poisonous or edible. During learning, the number of feature values that characterize a given category is monotone decreasing, so that generalization within a given category tends to increase. The total number of classes can, however, also increase, which tends to decrease generalization. Increasing the number of training patterns hereby tends to increase the number of categories and decrease the number of critical feature values of each established category. The balance between these opposing tendencies leads to the final net level of generalization.
Table 6 illustrates the long term memory structure underlying the 125-sample forced-choice simulation ρ.sub.a =0. Of the nine categories established at the end of the training phase, 4 are identified as poisonous (P) and 5 are identified as edible (E). Categories 1, 5, 7 and 8 are identified as poisonous (P) and categories 2, 3, 4, 6, and 9 are identified as edible (E). Each ART.sub.a category assigns a feature value to a subset of the 22 observable features. For example, Category 1 (poisonous) specifies values for 5 features, and leaves the remaining 17 features unspecified. The corresponding ART.sub.a weight vector has 5 ones and 121 zeros. Note that the features that characterize category 5 (poisonous) form a subset of the features that characterize category 6 (edible). This category structure gave 96.4% correct responses on the 7999 test set samples, which are partitioned as shown in the last line of Table 5. When 100% accuracy is achieved, a few categories with a small number of specified features typically code large clusters, while a few categories with many specified features code small clusters of rare samples.
Table 7 illustrates the statistical nature of the coding process, which leads to a variety of category structures when fast learning is used. Test set prediction accuracy of the simulation that generated Table 7 was similar to that of Table 6, and each simulation had a 125-sample training set. However, the simulation of Table 7 produced only 4 ART.sub.a categories, only one of which (category 1) has the same long term memory representation as category 2 in Table 6. Note that, at this stage of coding, certain features are uninformative. For example, no values are specified for features 1, 2, 3, or 22 in Table 6 or Table 7; and feature 16 (veil-type) always has the value "partial." However, performance is still only around 96%. As rare instances from small categories later in the coding process, some of these features may become critical in identifying exemplars of small categories.
While the invention has been particularly shown and described with reference to a preferred embodiment thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. For example, with only two choices of poisonous or nonpoisonous as the second-category-representation in the simulation, a full ART pattern recognition system ART.sub.b would not be required. In that simple case, ART.sub.a could be reset as described in response to a simple comparison. However, the use of the second pattern recognition system ART.sub.b allows for a much more complex comparison. Further, any of the ART systems may be utilized in the present invention, and the invention may even be applied to other neural network systems.
TABLE 3______________________________________On-Line LearningAverage number of correct predictions onprevious 100 trials   --&#961;.sub.a = 0             --&#961;.sub.a = 0                       --&#961;.sub.a = 0.7                               --&#961;.sub.a = 0.7Trial   no replace             replace   no replace                               replace______________________________________ 100    82.9      81.9      66.4    67.3 200    89.8      89.6      87.8    87.4 300    94.9      92.6      94.1    93.2 400    95.7      95.9      96.8    95.8 500    97.8      97.1      97.5    97.8 600    98.4      98.2      98.1    98.2 700    97.7      97.9      98.1    99.0 800    98.1      97.7      99.0    99.0 900    98.3      98.6      99.2    99.01000    98.9      98.5      99.4    99.01100    98.7      98.9      99.2    99.71200    99.6      99.1      99.5    99.51300    99.3      98.8      99.8    99.81400    99.7      99.4      99.5    99.81500    99.5      99.0      99.7    99.61600    99.4      99.6      99.7    99.81700    98.9      99.3      99.8    99.81800    99.5      99.2      99.8    99.91900    99.8      99.9      99.9    99.92000    99.8      99.8      99.8    99.8______________________________________
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS5054093 *Apr 8, 1988Oct 1, 1991Cooper Leon NParallel, multi-unit, adaptive, nonlinear pattern class separator and identifierUS5058180 *Apr 30, 1990Oct 15, 1991National Semiconductor CorporationNeural network apparatus and method for pattern recognitionUS5060278 *May 15, 1990Oct 22, 1991Ricoh Company, Ltd.Pattern recognition apparatus using a neural network systemUS5095443 *Oct 4, 1989Mar 10, 1992Ricoh Company, Ltd.Plural neural network system having a successive approximation learning methodWO1986002553A1 *Oct 22, 1985May 9, 1986Nastech Pharm CoNovel method of administering anti-nausea and anti-emetic pharmaceutical agents and novel dosage forms containing sameWO1991000261A1 *Jul 2, 1990Jan 10, 1991Hoechst AgProcess for preparing 4,5-dichloro-2-nitroaniline* Cited by examinerNon-Patent CitationsReference1"Art 3: Hierarchical Search Using Chemical Transmitters in Self-Organizing Pattern Recognition Architectures," Gail A. Carpenter et al., Neural Networks, in press, Mar. 1990.2 *Art 3: Hierarchical Search Using Chemical Transmitters in Self Organizing Pattern Recognition Architectures, Gail A. Carpenter et al., Neural Networks, in press, Mar. 1990.3Gail A. Carpenter, "Neural Network Models for Pattern Recognition and Associative Memory," Neural Networks, vol. 2, No. 4, 1989, New York, pp. 243-257.4 *Gail A. Carpenter, Neural Network Models for Pattern Recognition and Associative Memory, Neural Networks, vol. 2, No. 4, 1989, New York, pp. 243 257.5Ryan, "The Resonance Correlation Network", IEEE Int'l Conf. on Neural Nets Jul. 1988.6 *Ryan, The Resonance Correlation Network , IEEE Int l Conf. on Neural Nets Jul. 1988.* Cited by examinerReferenced byCiting PatentFiling datePublication dateApplicantTitleUS5384895 *Aug 28, 1992Jan 24, 1995The United States Of America As Represented By The Secretary Of The NavySelf-organizing neural network for classifying pattern signatures with `a posteriori` conditional class probabilityUS5420939 *Dec 31, 1992May 30, 1995Intel CorporationMethod and apparatus for a focal neuron systemUS5438629 *Jun 19, 1992Aug 1, 1995United Parcel Service Of America, Inc.Method and apparatus for input classification using non-spherical neuronsUS5452399 *Dec 30, 1993Sep 19, 1995United Parcel Service Of America, Inc.Method and apparatus for input classification using a neuron-based voting schemeUS5555495 *Oct 25, 1993Sep 10, 1996The Regents Of The University Of MichiganMethod for adaptive control of human-machine systems employing disturbance responseUS5566092 *Dec 30, 1993Oct 15, 1996Caterpillar Inc.Machine fault diagnostics system and methodUS5566273 *Dec 30, 1993Oct 15, 1996Caterpillar Inc.Supervised training of a neural networkUS5602761 *Dec 30, 1993Feb 11, 1997Caterpillar Inc.Machine performance monitoring and fault classification using an exponentially weighted moving average schemeUS5638491 *Jan 23, 1995Jun 10, 1997United Parcel Service Of America, Inc.Method and apparatus for hierarchical input classification using a neural networkUS5640103 *May 22, 1996Jun 17, 1997Siemens Corporate Research, Inc.Radial basis function neural network autoassociator and method for induction motor monitoringUS5640493 *Apr 17, 1995Jun 17, 1997E. I. Du Pont De Nemours & Co., Inc.Historical database training method for neural networksUS5659666 *Oct 13, 1994Aug 19, 1997Thaler; Stephen L.Device for the autonomous generation of useful informationUS5664067 *Jan 11, 1995Sep 2, 1997United Parcel Service Of America, Inc.Method and apparatus for training a neural networkUS5694406 *Aug 5, 1996Dec 2, 1997Board Of Regents, The University Of Texas SystemParallel associative processor formed from modified dramUS5708727 *Dec 7, 1994Jan 13, 1998Fujitsu LimitedNeuroprocessing serviceUS5758148 *Dec 7, 1992May 26, 1998Board Of Regents, The University Of Texas SystemSystem and method for searching a data base using a content-searchable memoryUS5777608 *Mar 1, 1994Jul 7, 1998Board Of Regents, The University Of Texas SystemApparatus and method for in-parallel scan-line graphics rendering using content-searchable memoriesUS5826249 *Jun 6, 1997Oct 20, 1998E.I. Du Pont De Nemours And CompanyHistorical database training method for neural networksUS5845271 *Jan 26, 1996Dec 1, 1998Thaler; Stephen L.Non-algorithmically implemented artificial neural networks and components thereofUS5852815 *May 15, 1998Dec 22, 1998Thaler; Stephen L.Neural network based prototyping system and methodUS5852816 *May 15, 1998Dec 22, 1998Thaler; Stephen L.Neural network based database scanning systemUS5974404 *Jun 24, 1997Oct 26, 1999United Parcel Service Of America, Inc.Method and apparatus for input classification using a neural networkUS5991525 *Aug 22, 1997Nov 23, 1999Voyan TechnologyMethod for real-time nonlinear system state estimation and controlUS5999922 *May 26, 1995Dec 7, 1999Fujitsu LimitedNeuroprocessing serviceUS6009418 *Sep 13, 1996Dec 28, 1999Cooper; David L.Method and apparatus for neural networking using semantic attractor architectureUS6014653 *May 15, 1998Jan 11, 2000Thaler; Stephen L.Non-algorithmically implemented artificial neural networks and components thereofUS6018727 *Aug 13, 1997Jan 25, 2000Thaler; Stephen L.Device for the autonomous generation of useful informationUS6032142 *Nov 5, 1997Feb 29, 2000U.S. Philips CorporationAutonomous compound agentsUS6038556 *Aug 31, 1998Mar 14, 2000Hutchison; William R.Adaptive autonomous agent with verbal learningUS6041172 *Nov 26, 1997Mar 21, 2000Voyan TechnologyMultiple scale signal processing and control systemUS6078918 *Apr 2, 1998Jun 20, 2000Trivada CorporationOnline predictive memoryUS6115701 *Aug 9, 1999Sep 5, 2000Thaler; Stephen L.Neural network-based target seeking systemUS6148034 *Jun 4, 1998Nov 14, 2000Linden Technology LimitedApparatus and method for determining video encoding motion compensation vectorsUS6285971Aug 10, 1999Sep 4, 2001Voyan TechnologyMethod for real-time nonlinear system state estimation and controlUS6356884Jul 2, 1999Mar 12, 2002Stephen L. ThalerDevice system for the autonomous generation of useful informationUS6516005Dec 31, 1998Feb 4, 2003Mitsubishi Denki Kabushiki KaishaApparatus and method for data decodingUS6587845 *Feb 15, 2000Jul 1, 2003Benjamin B. BraunheimMethod and apparatus for identification and optimization of bioactive compounds using a neural networkUS6601049 *May 26, 2000Jul 29, 2003David L. CooperSelf-adjusting multi-layer neural network architectures and methods thereforUS6754644 *Mar 6, 2000Jun 22, 2004William R. HutchisonAdaptive autonomous agent with verbal learningUS7505921Mar 3, 2000Mar 17, 2009Finali CorporationSystem and method for optimizing a product configurationUS7526467Sep 10, 2003Apr 28, 2009Natural Selection, Inc.Intelligently interactive profiling system and methodUS7577631Mar 21, 2002Aug 18, 2009Feldhake Michael JCognitive image filteringUS7599861Mar 2, 2006Oct 6, 2009Convergys Customer Management Group, Inc.System and method for closed loop decisionmaking in an automated care systemUS7707148Oct 6, 2004Apr 27, 2010Natural Selection, Inc.Method and device for clustering categorical data and identifying anomalies, outliers, and exemplarsUS7783581 *Dec 26, 2005Aug 24, 2010Nec CorporationData learning system for identifying, learning apparatus, identifying apparatus and learning methodUS7809663May 22, 2007Oct 5, 2010Convergys Cmg Utah, Inc.System and method for supporting the utilization of machine languageUS7945627Sep 28, 2006May 17, 2011Bitdefender IPR Management Ltd.Layout-based electronic communication filtering systems and methodsUS7958079Mar 19, 2009Jun 7, 2011Natural Selection, Inc.Intelligently interactive profiling system and methodUS8010614Nov 1, 2007Aug 30, 2011Bitdefender IPR Management Ltd.Systems and methods for generating signatures for electronic communication classificationUS8090721Feb 27, 2010Jan 3, 2012Natural Selection, Inc.Method and device for clustering categorical data and identifying anomalies, outliers, and exemplarsUS8103615Apr 28, 2011Jan 24, 2012Natural Selection, Inc.Intelligently interactive profiling system and methodUS8131655May 30, 2008Mar 6, 2012Bitdefender IPR Management Ltd.Spam filtering using feature relevance assignment in neural networksUS8254669 *Sep 16, 2008Aug 28, 2012Ricoh Company, Ltd.Data processing apparatus, computer program product, and data processing method for predicting an optimum function based on a case database and image feature values calculated by a feature-value calculating unitUS8379085 *Aug 18, 2009Feb 19, 2013Behavioral Recognition Systems, Inc.Intra-trajectory anomaly detection using adaptive voting experts in a video surveillance systemUS8379830May 22, 2007Feb 19, 2013Convergys Customer Management Delaware LlcSystem and method for automated customer service with contingent live interactionUS8452668Aug 12, 2009May 28, 2013Convergys Customer Management Delaware LlcSystem for closed loop decisionmaking in an automated care systemUS8473449Dec 22, 2009Jun 25, 2013Neuric Technologies, LlcProcess of dialogue and discussionUS8484139Aug 7, 2008Jul 9, 2013Hitachi, Ltd.Data classification method and apparatusUS8572184Oct 4, 2007Oct 29, 2013Bitdefender IPR Management Ltd.Systems and methods for dynamically integrating heterogeneous anti-spam filtersUS20090074288 *Sep 16, 2008Mar 19, 2009Hirobumi NishidaData processing apparatus, computer program product, and data processing methodUS20110043626 *Aug 18, 2009Feb 24, 2011Wesley Kenneth CobbIntra-trajectory anomaly detection using adaptive voting experts in a video surveillance systemUS20120016829 *Jun 22, 2009Jan 19, 2012Hewlett-Packard Development Company, L.P.Memristive Adaptive Resonance NetworksWO1994014162A1 *Dec 6, 1993Jun 23, 1994G Jack LipovskyPattern search and refresh logic in dynamic memoryWO1995012171A2 *Oct 24, 1994May 4, 1995Univ MichiganMethod and apparatus for adaptive control of human-machine systems employing disturbance responseWO2000016212A1 *Sep 10, 1998Mar 23, 2000David L CooperMethod and apparatus for neural networking using semantic attractor architectureWO2001095253A2 *Jun 8, 2001Dec 13, 2001Medimage ApsQuality and safety assurance of medical applicationsWO2004025411A2 *Sep 10, 2003Mar 25, 2004David B FogelIntelligently interactive profiling system and method* Cited by examinerClassifications U.S. Classification382/157, 706/20, 706/27, 382/159, 706/25International ClassificationG06N3/04Cooperative ClassificationG06N3/0409, G06K9/6222European ClassificationG06N3/04A, G06K9/62B1P1ALegal EventsDateCodeEventDescriptionMay 22, 2005SULPSurcharge for late paymentYear of fee payment: 11May 22, 2005FPAYFee paymentYear of fee payment: 12Dec 8, 2004REMIMaintenance fee reminder mailedNov 17, 2000FPAYFee paymentYear of fee payment: 8Jan 7, 1997CCCertificate of correctionSep 30, 1996FPAYFee paymentYear of fee payment: 4May 17, 1994CCCertificate of correctionMay 13, 1991ASAssignmentOwner name: TRUSTEES OF BOSTON UNIVERSITY, MASSACHUSETTSFree format text: ASSIGNMENT OF ASSIGNORS INTEREST.;ASSIGNORS:CARPENTER GAIL A.;GROSSBERG STEPHEN;REYNOLDS JOHN H.;REEL/FRAME:005771/0549Effective date: 19910501RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services©2012 Google