Long-term memory in a video analysis system

A long-term memory used to store and retrieve information learned while a video analysis system observes a stream of video frames is disclosed. The long-term memory provides a memory with a capacity that grows in size gracefully, as events are observed over time. Additionally, the long-term memory may encode events, represented by sub-graphs of a neural network. Further, rather than predefining a number of patterns recognized and manipulated by the long-term memory, embodiments of the invention provide a long-term memory where the size of a feature dimension (used to determine the similarity between different observed events) may grow dynamically as necessary, depending on the actual events observed in a sequence of video frames.

CROSS REFERENCE TO RELATED APPLICATION

This application relates to commonly assigned co-pending, U.S. patent application Ser. No. 12/028,484 filed Feb. 8, 2008, entitled “Behavioral Recognition System”, which is herein incorporated by reference in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the invention provide a long-term memory used to store and retrieve percepts in a video analysis system. More specifically, embodiments of the invention relate to techniques for programmatically associating, learning, and recalling patterns of behavior depicted in a sequence of video frames.

2. Description of the Related Art

Some currently available video surveillance systems are provide simple object recognition capabilities. For example, a video surveillance system may be configured to classify a group of pixels in a given frame having certain specified characteristics (referred to as a “blob”) as being a particular object (e.g., a person or vehicle). Once identified, a “blob” may be tracked from frame-to-frame in order to follow the movement of the “blob” over time, e.g., a person walking across the field of vision of a video surveillance camera. Further, such systems may be able to determine when an object has engaged in certain predefined behaviors.

However, such surveillance systems typically require that the objects and/or behaviors which may be recognized by the system to be defined in advance. Thus, in practice, these systems simply compare recorded video to predefined definitions for objects and/or behaviors. In other words, unless the underlying system includes a description of a particular object or behavior, the system is generally incapable of recognizing that behavior (or at least instances of the pattern describing the particular object or behavior). Thus, to recognize additional objects or behaviors, separate software products may need to be developed. This results in surveillance systems with recognition capabilities that are labor intensive and prohibitively costly to maintain or adapt for different specialized applications. For example, monitoring airport entrances for lurking criminals and identifying swimmers who are not moving in a pool are two distinct situations, and therefore may require developing two distinct software products having their respective “abnormal” behaviors pre-coded. Thus, currently available video surveillance systems are typically incapable of recognizing new patterns of behavior that may emerge in a given scene or recognizing changes in existing patterns. Further, such systems are often unable to associate related aspects from different patterns of observed behavior, e.g., to learn to identify behavior being repeatedly performed by a criminal prior to breaking into cars parked in a parking lot.

Further, the static patterns that available video surveillance systems are able to recognize are frequently either under inclusive (i.e., the pattern is too specific to recognize many instances of a given object or behavior) or over inclusive (i.e., the pattern is general enough to trigger many false positives). In some cases, the sensitivity of may be adjusted to help improve the recognition process, however, this approach fundamentally relies on the ability of the system to recognize predefined patterns for objects and behavior. As a result, by restricting the range of objects that a system may recognize using a predefined set of patterns, many available video surveillance systems have been of limited usefulness.

SUMMARY OF THE INVENTION

One embodiment of the invention includes a computer-implemented method for storing an encoded percept in a long-term memory. The method may generally include initializing the long-term memory. The long term long-term memory includes a set of hard-locations, a weight matrix, and an activation function. The method may further include receiving a percept. The percept itself comprises a sub-graph of a neural network excited by an input stimulus. The method may further include encoding the percept as a bit pattern, generating a variable length entry in the set of hard locations, and storing the encoded percept as a hard location in the set of hard locations. The method may further include generating an entry in the weight matrix corresponding to the encoded percept.

Still another embodiment of the invention includes a computer-readable storage medium containing a program which, when executed by a processor, performs an operation for storing an encoded percept in a long-term memory. The operation may generally include initializing the long-term memory. The long-term memory itself includes a set of hard-locations, a weight matrix, and an activation function. The operation may also include receiving a percept. The percept itself may comprise a sub-graph of a neural network excited by an input stimulus. The operation may further include encoding the percept as a bit pattern, generating a variable length entry in the set of hard locations, and storing the encoded percept as a hard location in the set of hard locations. The operation may further include generating an entry in the weight matrix corresponding to the encoded percept.

Still another embodiment of the system having a video input source configured to provide a sequence of video frames, each depicting a scene. The system may also include a processor and a memory containing a machine learning application which when executed by the processor is configured to perform an operation for storing an encoded percept in a long-term memory. The operation may generally include initializing the long-term memory. The long-term memory itself includes a set of hard-locations, a weight matrix, and an activation function. The operation may also include receiving a percept. The percept itself may comprise a sub-graph of a neural network excited by an input stimulus. The operation may further include encoding the percept as a bit pattern, generating a variable length entry in the set of hard locations, and storing the encoded percept as a hard location in the set of hard locations. The operation may further include generating an entry in the weight matrix corresponding to the encoded percept.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the invention provide a long-term memory used to store and retrieve information learned while a video analysis system observes a stream of video frames. The video analysis system may include a computer vision engine and a machine learning engine. The computer vision engine may be configured to receive and process a sequence of video frames. The machine learning engine may be configured to process the output of the computer vision engine. In one embodiment, the machine learning engine may include multiple data structures, referred to as memories, used to store and process information related to what is observed and perceived to have occurred within the scene. For example, the machine learning engine may include a perceptual memory, an episodic memory, and a long-term memory.

In one embodiment, the long-term memory provides a memory with a capacity that grows in size gracefully, as events are observed over time. Additionally, the long-term memory may encode events, represented by sub-graphs of a neural network, using a run-length encoding scheme, minimizing the storage requirements of the long-term memory. Further, rather than predefining a number of patterns recognized and manipulated by the long-term memory, embodiments of the invention provide a long-term memory where the size of a feature dimension (used to determine the similarity between different observed events) may grow dynamically as necessary, depending on the actual events observed in a sequence of video frames. As a result, the dimensions of a weight matrix used to encode patterns (and to reinforce/decay memories stored in the long-term memory) expand dynamically in response to changes in the feature dimension. That is, the information stored in the long-term memory is driven by the actual events observed in a scene.

As events occur, and re-occur, patterns encoding the events are stored and reinforced in the long-term memory. Additionally, the overhead required to store and retrieve information from the long-term memory is minimized due to the ability of the weight matrix and the feature dimension to grow as necessary. The storage capacity of the long-term memory is limited only by the memory capacity of the computer.

FIG. 1illustrates components of a video analysis and behavior-recognition system100, according to one embodiment of the present invention. As shown, the behavior-recognition system100includes a video input source105, a network110, a computer system115, and input and output devices118(e.g., a monitor, a keyboard, a mouse, a printer, and the like). The network114may transmit video data recorded by the video input112to the computer system115. Illustratively, the computer system115includes a CPU120, storage125(e.g., a disk drive, optical disk drive, floppy disk drive, and the like), and a memory130containing a computer vision engine135and a machine learning engine140. The computer vision engine135and the machine learning engine140may provide software applications configured to analyze a sequence of video frames provided by the video input112.

Network110receives video data (e.g., video stream(s), video images, or the like) from the video input source105. The video input source105may be a video camera, a VCR, DVR, DVD, computer, web-cam device, or the like. For example, the video input source105may be a stationary video camera aimed at a certain area (e.g., a subway station, a parking lot, a building entry/exit, etc.), which records the events taking place therein. Generally, the area visible to the camera is referred to as the “scene.” The video input source105may be configured to record the scene as a sequence of individual video frames at a specified frame-rate (e.g., 24 frames per second), where each frame includes a fixed number of pixels (e.g., 320×240). Each pixel of each frame may specify a color value (e.g., an RGB value) or grayscale value. Further, the video stream may be formatted using known such formats e.g., MPEG2, MJPEG, MPEG4, H.263, H.264, and the like.

As described in greater detail below, the computer vision engine135may be configured to analyze this raw information to identify active objects in the video stream, classify the objects, derive a variety of metadata regarding the actions and interactions of such objects, and supply this information to a machine learning engine140. In turn, the machine learning engine140may be configured to evaluate, observe, learn and remember details regarding events (and types of events) that transpire within the scene over time.

In one embodiment, the machine learning engine140receives the video frames and the results generated by the computer vision engine135. The machine learning engine140may be configured to analyze the received data, build semantic representations of events depicted in the video frames, detect patterns, and, ultimately, to learn from these observed patterns to identify normal and/or abnormal events. In one embodiment, data describing whether a normal/abnormal behavior/event has been determined and/or what such behavior/event is may be provided to output devices118to issue alerts, for example, an alert message presented on a GUI interface screen. In general, the computer vision engine135and the machine learning engine140both process video data in real-time. However, time scales for processing information by the computer vision engine135and the machine learning engine140may differ. For example, in one embodiment, the computer vision engine135processes the received video data frame-by-frame, while the machine learning engine140processes data every N-frames. In other words, while the computer vision engine135analyzes each frame in real-time to derive a set of information about what is occurring within a given frame, the machine learning engine150is not constrained by the real-time frame rate of the video input.

Note, however,FIG. 1illustrates merely one possible arrangement of the behavior-recognition system100. For example, although the video input source105is shown connected to the computer system115via the network110, the network110is not always present or needed (e.g., the video input source105may be directly connected to the computer system115). Further, various components and modules of the behavior-recognition system100may be implemented in other systems. For example, in one embodiment, the computer vision engine135may be implemented as a part of a video input device (e.g., as a firmware component wired directly into a video camera). In such a case, the output of the video camera may be provided to the machine learning engine135for analysis. Similarly, the output from the computer vision engine135and machine learning engine140may be supplied over computer network110to other computer systems. For example, the computer vision engine135and machine learning engine140may be installed on a server system and configured to process video from multiple input sources (i.e., from multiple cameras). In such a case, a client application running on another computer system may request (or receive) the results of over network110.

FIG. 2further illustrates components of the computer vision engine135and the machine learning engine140first illustrated inFIG. 1, according to one embodiment of the present invention. As shown, the computer vision engine135includes a primitive event detection module205and a phase space partitioning module210. In one embodiment, the phase space partitioning module210may be configured to generate a variety of contextual and kinematic data related to the scene depicted in a sequence of video frames. For example, the computer vision engine135may, over a sequence of frames, develop a background/foreground model of the scene indicating which elements of a given frame depict scene background and which elements depict foreground objects.

Once the background and foreground of the scene is identified, objects in the scene foreground may be classified as being some type of thing, based on the features or characteristics of a given object. For example, a classifier may receive the pixels believed to depict the foreground object along with other data related to its kinematics, pose, position, etc. From this information, the classifier may classify the object as depicting, e.g., a car or a person, or something else, depending on the capabilities of the classifier. Further, in one embodiment, the phase space partitioning model210may be configured to identify regions of the background image and identify contextual information about each region, e.g., whether one region is in front of (or behind) another. From this information, the phase space partitioning module210may determine dimensionality and geometry of both background and foreground elements. For example, assume the phase space partitioning module210identifies a blob of pixels as depicting a human individual and that the blob is 30 pixels in height. This provides a convenient mechanism for estimating the size of objects in the scene, based on the average height of a person.

In one embodiment, the primitive event detection module205may be configured to identify what activity a given object is engaged in using the classification and other data generated by the computer vision engine135and the phase space partitioning module210. Further, the primitive event detection module205may generate a stream of primitive events describing what activity is observed to have occurred. For example, assume the computer vision engine135has identified a foreground object and classified that foreground object as being a car. In such a case, the primitive event detection module205may determine that the car has engaged in some specified action. For example, a sequence of primitive events related to events occurring at a shopping-mall parking lot could include: “car appears in scene,” “car moves to a given location,” “car stops moving,” “person appears proximate to car” “person moves,” person leaves scene” “person appears in scene,” “person moves proximate to car,” person disappears,” “car starts moving,” and “car disappears.” This information (labeled inFIG. 2as “primitive event stream”) along with other information generated by the computer vision engine135(labeled inFIG. 2“semantic representations”) is passed to the machine learning engine140.

Illustratively, the machine learning engine140includes a workspace215, a perceptual memory220, an episodic memory225, a long-term memory230, and codelets235. In one embodiment, the perceptual memory220, the episodic memory225, and the long-term memory230are used to identify patterns of behavior, evaluate events that transpire in the scene, and encode and store observations.

Generally, the perceptual memory220receives the output of the computer vision engine. The episodic memory225stores data representing observed events with details related to a particular episode, e.g., information describing time and space details related on an event. That is, the episodic memory225may include details describing “what and where” something occurred within a scene such as a particular car (car A) moved to a location believed to be a parking spot (parking spot5) at 9:43 AM. The long-term memory230may store data generalizing events observed in the scene.

To continue with the example of a car parking, the long-term memory230may encode information related to observations learned by an analysis of the behavior of objects in the scene such as “cars tend to park in a particular place,” “when parking cars tend to move a certain speed,” and “after a car parks, people tend to appear in the scene,” etc. Thus, the long-term memory230stores observations about what happens within a scene with much of the particular episodic details stripped away. In this way, when a new event occurs, memories from the episodic memory225and the long-term memory230may be used to relate and understand a current event, i.e., the new event may be compared with past experience, leading to both reinforcement, decay, and adjustments to the information stored in the long-term memory230, over time.

In one embodiment, the perceptual memory220may be implemented as a neural network configured to receive the primitive event stream and other information from the computer vision engine135. In such a case, the primitive event stream provides a stimulus be used to excite a group of nodes present in the neural network (referred as a percept). Once excited by input data from the computer vision engine135, the percept may be copied into the workspace215(illustrated inFIG. 2as an active percept218). When a percept is instantiated into the workspace215, memories may be retrieved from the episodic memory225and the long-term memory230(illustrated inFIG. 2as retrieved memories217) to relate and understand a current event as well as to compare the current event to past experience. In one embodiment, the memories in the episodic memory225and long-term memory230are themselves represented as percepts. That is, the memories may be represented as directed graphs (or sub-graphs) similar to the neural network built-up as the perceptual memory220observes the information from the computer vision engine135. Thus, each percept encodes information related to an observed event.

Generally, the workspace215provides a computational engine for the machine learning engine140. For example, the workspace215may be configured to copy percepts from the perceptual memory220(i.e., to copy a sub-graph of a neural network that was excited by events observed by the computer vision engine135), retrieve relevant memories from the episodic memory225and the long-term memory230, select and invoke the execution of one of codelets235(illustrated asFIG. 2as running codelet216). In one embodiment, each codelet235is a software program configured to evaluate different sequences of events and to determine how one sequence may follow (or otherwise relate to) another (e.g., a finite state machine). More generally, the codelet may provide a small software program configured to find interesting patterns. In turn, the codelet may create, reinforce, or modify memories in the episodic memory225and the long-term memory230(i.e., when a pattern is identified). By repeatedly scheduling codelets for execution, copying memories and percepts to/from the workspace215, the machine learning engine140performs a cognitive cycle used to observe, and learn, about patterns of behavior that occur within the scene.

FIGS. 3A-3Dillustrate a long-term memory140used to create, store, reinforce, and modify memories (represented as encoded percepts), according to one embodiment of the invention. First,FIG. 3Aillustrates the initial state of a long-term memory140. In a particular embodiment, the long-term memory is implemented as a sparse distributed memory data structure. As shown, the long-term memory230includes a null set of hard locations305, an activation function310, and a null weight matrix140. As illustrated in greater detail below in conjunction withFIGS. 3B-3Dthe hard locations305store encoded patterns of percepts—sorted into different classes or categories. That is, each hard location305corresponds to a category of event with different variations (e.g., a car parking stopping starting etc.). In one embodiment, hard locations305may be implemented as an adaptive resonance theory (ART) network and ART network locations (i.e., hard locations305) may be created and/or modified according to vigilance parameter and choice parameter. As memories (i.e., percepts) are selected to be stored in the long-term memory140(e.g., as a codelet operates on an excited percept), the hard locations305are used to determine whether to create a new memory in the long-term memory or to modify the weights in the weight matrix140associated with an existing hard location305. Thus, the size of the hard locations in the weight matrix140is dynamic.

FIG. 3Billustrates an example of a new percept325being encoded and stored in long-term memory140. As shown, the percept325includes a directed graph of three nodes, “A,” “B,” and “C,” and edges from node “A” to both nodes “B” and “C.” Assume for this example, that nodes “A,” “B,” and “C” form a percept related to a parking event observed in the scene and that these nodes were excited in the perceptual memory220by input from the computer vision engine135. In such a case, the percept325may be copied into the workspace215and evaluated using a codelet235. Further, the nodes in percept325could represent subject kinematics (i.e., the position and motion of an object classified as a car), a node in the neural network representing a parking event, and a node representing a parking space. Illustratively, the percept325is encoded as a matrix330. The first row of the matrix330is used to indicate whether a given node is present in the percept325. The remaining rows of the matrix330are used to present whether a directed link exists from one of the nodes (per row) to another one of the nodes (per column). For example, the second row encodes the links from A→B and A→C present in percept325. Once encoded, the percept325may be stored as a new hard location320. Illustratively, the percept is encoded as a hard location320in hard locations305. Alternatively, the hard location320may be compressed using a run-length encoding scheme. Doing so may allow the dynamic increase of the features and the weight matrix.

In one embodiment, the hard location for a percept is created as a bit pattern generated by copying the values of the encoded matrix, in row order. Note, this approach allows the hard locations305to include each feature dimension in the encoded percept (i.e., each node and link between nodes in a percept). Further, as this is the first time the percept325(or any percept) has been stored in long-term memory140, the matrix330is copied directly into the weight matrix315as entry340. That is, like the hard location320, the matrix300is copied to entry340in row order. Each entry in the weight matrix represents one of the feature dimensions for the memory. Subsequently, as similar memories (i.e., percepts that categorize to the same hard location320or are within a minimum distance to hard location320) are stored in the long-term memory140, the weights for each feature dimension in weight matrix315associated with hard location320are reinforced (positively or negatively).

For example,FIG. 3Cillustrates a second percept345being stored in the long-term memory140, according to one embodiment of the invention. Illustratively, the percept345includes the same nodes “A,” “B,” and “C,” as node325, but lacks the links from node “A” to nodes “B” and “C.” Thus, the percept345and235share some, but not all feature dimensions in common. In this case, the hard locations305(e.g., an ART network) has decided to modify existing memories for an existing hard location. (e.g., by using a vigilance and choice function). The percept345is encoded in a matrix350using the same encoding scheme of matrix330ofFIG. 3B. Note, in this example, the feature dimensions in percept345are the same as percept325(i.e., each includes nodes “A,” “B,” and “C,”), thus, the dimensionality of the hard locations305is unchanged by encoding this percept. Once the appropriate hard location is identified (in this example, hard location320), the activation function310is used to activate one or more of the memories encoded in weight matrix315. In one embodiment, the activation function310calculates a Hamming distance between the hard location and the encoded percept retrieve related experiences using one or more entries in the weight matrix315. In an alternative embodiment, where the hard locations are implemented as an adaptive resonance theory (ART) network, the choice function and vigilance function may be determined directly from an ART network. Once identified, the weights of each feature dimension in an activated entry are adjusted. That is, the current encoded percept is used to reinforce (positively and negatively) the feature dimension values for one or more entries of the weight matrix315. Illustratively, the encoded percept350is used to reinforce the weight matrix entry340. In one embodiment, the reinforcement occurs by converting each “0” value in the encoded percept to a −1 and then adding each feature dimension the percept to the weights. Doing so results in a match between values in the encoded percept and the weights to increase a weight value. For example, a match between two “1” values reinforces that weight (resulting in weights with a grater positive magnitude) and match between two 0 values (after being converted to a −1 values) results in weights with a greater negative magnitude. Table I, below, illustrates the reinforcement for the weights corresponding to hard location320after encoded percept350is applied.

TABLE IPositive and Negative memory reinforcement111011000000(weights as initially encoded from percept 325)+111−1−1−1−1−1−1−1−1−1(percept 345 as applied to reinforce memory)222−100−1−1−1−1−1−1(resulting values for this memory in weight matrix 315)
In one embodiment, the resulting values in weight matrix315may have a maximum magnitude. That is, the reinforcement (positive or negative) is cut-off after reaching a certain value. While such a cut-off may be tailored to suit the needs of a given case, a cut-off of +/−16 has proven to be an effective.

FIG. 3Dillustrates a third percept345being stored in the long-term memory140, according to one embodiment of the invention. Unlike the percept345ofFIG. 3C, the percept365is dissimilar to percept320ofFIG. 3A. For example, percept365does not include any of the nodes “A,” “B,” or “C,” and includes nodes “D” and “E.” That is, percept345includes different feature dimensions. Continuing with the example of a car parking, node “D” could represent a person appearing in the scene (e.g., when exiting a parked car) and node E could represent kinematics associated with the person. Because the encoded percept365represents a different general category of event from the encoded percepts325and345, a new hard location375is used to represent this percept in hard locations305. Note, matrix370now includes columns for the new nodes “D” and “E” not present in the matrices330and350. Accordingly, the hard locations305grow to accommodate the nodes not present in the other encoded percepts (i.e., percepts325and345). In one embodiment, this is determined by the ART network used to represent hard locations320. Thus, this example embodiment illustrates how the hard locations305in the long-term memory140can grow gracefully as new events are observed by the computer vision engine135, resulting in new nodes in the neural network of the perceptual memory220being encoded in the long-term memory230. Additionally, an entry360in the weight matrix corresponding to node375is initialized using the values of matrix370encoding percept365. Like the entry340used to initialize weight matrix for percept325, entry360copies the values from the matrix370, in row-order. Thereafter as other percept configurations are stored in long-term memory140, the values in entry360of the weight matrix315are reinforced (either positively or negatively), up to the magnitude cutoff (e.g., +/−16).

In addition to storing memories in long-term memory140(as encoded percepts), percepts may be retrieved from long-term memory. For example, one percept may be used to activate certain memories in the long-term memory140, and the activated memories may themselves be used to construct a percept. For example,FIGS. 4A-4Billustrate percepts being retrieved from the long-term memory140in response to an input stimulus, according to one embodiment of the invention. The activation function310is used to select one or more hard locations305to activate. In this example, which hard locations305are activated is determined using a Hamming distance (i.e., using activation function310). As is known, a Hamming distance specifies how distinct two input bit sequences are by counting the number of bit-wise differences between the inputs. By setting a Hamming distance threshold, only hard locations305with a Hamming distance below the threshold are activated. In one embodiment, this may be calculated from data was already stored in the hard locations320. That is, only hard locations with a sufficient similarity to the input string are activated. Of course the threshold may be tailored to suit the needs of an individual case. Further, although illustrated using a Hamming distance, one of skill in the art will recognize that an ART distance measure calculated between the input to the ART network and the existing hard locations may also be used. In such a case, hard locations within a specified threshold of the input (as determined using the distance measure between the input and a given hard location) are activated.

Additionally, as shown inFIGS. 4A-4B, encoded percepts are stored in the long term memory in a single pass, i.e., the results of the activation function are used to to determine a final activation. However, one of ordinary skill in the art will recognize that this process may be performed recursively. For example, a first final activation returned for a first input may itself be supplied as a second input to the long term memory, resulting in a second final activation itself also input to the long term memory. This process may continue until a reaching a steady state and the resulting steady-state final activation may be returned as an activated memory responsive to the original input stimulus.

As shown inFIG. 4A, a single hard location is to be activated based on an input, specifically, the hard location enclosed by dashed lines405. The selected location is copied as final activation410. In one embodiment, the final activation410is determined using the weights corresponding to the hard locations activated by the input. As only one hard location305is activated in the example ofFIG. 4A, the values for the corresponding weight matrix are used to generate an encoded matrix415. In one embodiment, each positive weight value is converted to a “1” and each negative (or 0) value is converted to a 0 in encoded matrix415. The encoded matrix is then decoded using the same encoding scheme used to encode percepts. That is, the matrix is then decoded to generate a percept returned as a response to a request to retrieve memories corresponding to an input percept. Illustratively, matrix415is decoded to create percept420.

FIG. 4Bshows a second example of a memory being retrieved from the long-term memory140, according to one embodiment of the invention. Unlike the example ofFIG. 4A,FIG. 4Bshows multiple hard locations being activated and combined as a response to an input percept. Illustratively, hard locations425are to be activated based on an input. Thus, in this example assume that a Hamming distance determined for each of the hard locations425are within the Hamming distance threshold. Once the hard locations to activate have been identified, the corresponding weight matrix values are used to create final activation430. In one embodiment, the final activation430may be determined by adding the weights associated with each activated hard location and converting the weights to an encoded percept matrix (i.e., matrix435) using the same decoding scheme discussed above (converting each positive weight to a 1 and each negative (or zero) weight to a −1). By adding the weights from multiple hard locations305, the resulting percept may capture and combine many different memories related to the input percept.

FIG. 5illustrates a method500for processing information received from a computer vision engine, according to one embodiment of the invention. As shown, the method500begins at step505where the machine learning engine140begins receiving information from a computer vision engine. As described above, the information may include a semantic event stream and a phase space symbol stream generated from a sequence of video frames. More generally, the computer vision engine converts image data to numerical values, and the machine learning engine converts these numerical values to symbolic representations. At step510, the input symbol stream received at step505is input to a perceptual memory. In response, at step515, the input symbols may excite certain nodes of the perceptual memory. The group of nodes excited by a given input stimulus in the perceptual memory (i.e., a sub graph of a neural network maintained by the perceptual memory) is copied to a workspace. At step520, memories similar to the input stimulus (i.e. the sub-graph copied to the work space) may be retrieved from the episodic memory225. As described above, memories in the episodic memory225may include time and space details related to a particular event. At the same time, memories similar to the input stimulus may also be retrieved from the long-term memory140. For example, as illustrated above inFIGS. 4A and 4B, hard locations similar to the input stimulus may be selected using an activation function (e.g., a Hamming distance), and combined using a set of weights corresponding to each activated hard location. Once the episodic memories and long-term memories relevant to a current event which exited the perceptual memory are retrieved, the workspace may invoke a codelet used to relate and understand a current event, i.e., the new event may be compared with past experience, leading to both reinforcement, decay, and adjustments to the information stored in both the episodic memory and the long-term memory230over time.

FIG. 6illustrates a method600for storing an encoded percept in a long-term memory, according to one embodiment of the invention. Generally, the data flow begins with information excited from the perceptual memory220(i.e., a percept) flowing to the workspace where it is evaluated by a codelet. A percept with successful codelet execution may be moved from the workspace to the episodic memory225. The percept may be stored in the episodic memory225in a manner which includes time and space details of the event perceived by the perceptual memory220(as encoded in the percept). When the percept in the episodic memory225is mature, i.e., has been in the episodic memory225for a sufficient period of time (which may require the percept to be reinforced), the percept may be moved to the long-term memory. In such cases, the percept may be used to create a new long-term memory or reinforce existing long-term memories or both.

As shown, the method600begins at step605where a percept is selected from the episodic memory to be stored in the long-term memory. For example, a codelet may select to encode a percept in long-term memory when that percept is excited by input from the computer vision engine with a relatively high frequency. That is, events that occur more frequently are likely to be encoded in the long-term memory (and reinforced), where infrequently occurring events are not. At step610, the percept is encoded. For example,FIGS. 4B-4Dillustrate different percepts being encoded into a matrix format. Further, as stated, the encoded format may be compressed using a run-length encoding scheme (or other compression scheme). Doing so may allow the size of the feature space to be variable. Once encoded, at step615it is determined whether to create or reinforce information in the long term memory. If a new hard location is to be used, at steps620-630, the percept is used to create a new hard location (i.e., a new memory) in the long-term memory. Specifically, at step620, the encoded percept is used to generate a new hard location value. And at step625, the weight matrix is initialized for the new hard location. For example,FIGS. 3B and 3Dillustrate an embodiment where a percept is encoded as a matrix, and the matrix itself is then encoded as a hard location using a row-order of the matrix. Returning to step615, if the encoded percept is used to reinforce one or more existing memories, then the percept is used to modify (i.e., to reinforce) the weight values associated with the hard location to which the encoded percept is similar. At step635, the corresponding weight matrix entries are identified. And at step640, the weight matrix values are updated. For example, in one embodiment, the encoded percept is added to the weight values, as described above in conjunction withFIG. 3C.

FIG. 7illustrates a method700for retrieving an encoded percept from the long-term memory in response to an input stimulus, according to one embodiment of the invention. As shown, the method beings at step705where the activation function is used to identify one or more hard locations to activate. As described, in one embodiment, hard locations with a Hamming distance below a specified threshold, relative the input stimulus (an encoded percept), are activated. At step710, the hard locations activated by the input stimulus are identified. At step715, a final activation is determined, based on the locations activated by the input stimulus. For example, as described above, the final activation may be determined as an additive combination of each feature dimension value from each activated weight matrix entry. At step720, a percept configuration is generated from the final activation. And at step725, the percept configuration is returned to the workspace in response to the input stimulus.

FIGS. 8A-8Billustrate examples of input percepts used to activate elements of a long-term memory as well as percepts returned in response, according to one embodiment of the invention. In the example ofFIG. 8A, a stimulus800is used supplied to long-term memory, resulting in response805being returned (e.g., according to the method700). Illustratively, stimulus800and response805are shown as percepts. Thus, the percept of stimulus800includes nodes representing a sub-graph of a neural network exited by an input from a computer vision engine. Illustratively, response805includes fewer nodes than stimulus800. This may occur as the stimulus800is generated by events input from the computer vision engine and includes nodes representing the detail of a current event being observed by the computer vision engine135(e.g., a percept stored in episodic memory) where the response800does not include some of the specific detail of the input stimulus (e.g., the node representing “parking space16”). This allows a codelet to evaluate the stimulus representing the current event to identify patterns of events from the numerical data output by the computer vision engine.

Similarly,FIG. 8Billustrates a stimulus810and response815. Illustratively, the stimulus810generates a response815that includes more nodes than the input. This may occur when one stimulus activates multiple hard locations in the long-term memory. For example, the actions represented by the nodes of stimulus810include “subject kinematics” a “person subject” and an event of “disappear.” Assume that these actions are highly correlated in the long-term memory with the actions of a person entering a car and leaving a parking lot. In such a case, the input stimulus may activate hard locations related to parking, resulting in weight values for hard locations associated with a person disappearing to be combined with weight values for hard locations associated with vehicle behavior, resulting in a final activation returned in response to the input stimulus815.

Advantageously, as described herein, embodiments of the invention provide a long-term memory used to store and retrieve information learned while a video analysis system observes a stream of video frames. The long-term memory provides a memory with a capacity that grows in size gracefully, as events are observed over time. Additionally, the long-term memory may encode events, represented by sub-graphs of a neural network. Further, rather than predefining a number of patterns recognized and manipulated by the long-term memory, embodiments of the invention provide a long-term memory where the size of a feature dimension (used to determine the similarity between different observed events) may grow dynamically as necessary, depending on the actual events observed in a sequence of video frames.