Patent Publication Number: US-11651210-B2

Title: Interpreting and improving the processing results of recurrent neural networks

Description:
TECHNICAL FIELD 
     This invention relates to recurrent neural networks, and more particularly to interpreting and/or improving the robustness of decision making in recurrent neural networks. 
     BACKGROUND 
     Artificial neural networks are devices that are inspired by the structure and functional aspects of networks of biological neurons. In particular, artificial neural networks mimic the information encoding and other processing capabilities of networks of biological neurons using a system of interconnected constructs called nodes. The arrangement and strength of connections between nodes in an artificial neural network determines the results of information processing or information storage by the artificial neural network. 
     Neural networks can be trained to produce a desired signal flow within the network and achieve desired information processing or information storage results. In general, training a neural network will change the arrangement and/or strength of connections between nodes during a learning phase. The training will be directed to achieving certain processing results. The processing results should be consistent with a set of examples, i.e., a training set. A neural network can be considered trained when sufficiently appropriate processing results are achieved by the neural network for given sets of inputs. 
     Because training is fundamental to the processing performed by neural networks, neural networks are generally unable to process data that deviates in form or in type from the data in the training set. Indeed, even when the same type of content is present, seemingly insignificant perturbations—at least in the opinion of humans—can lead to dramatically different processing results. 
     An example are the so-called “adversarial examples” in image classification. Many image classifiers are sensitive to small (once again, in the opinion of human observers) non-random perturbations of the input data. Although an image classifier may correctly classify one image, a small perturbation of that same image may cause the image classifier to misclassify the perturbed image. In other words, in the image space, the classes appear to intersect in the region of the adversarial examples—even if they are well-defined elsewhere. 
     SUMMARY 
     The present methods and apparatus interpreting decision making in recurrent neural networks and improving the robustness of decision making in recurrent neural networks. In brief, recurrent neural networks inherently exhibit temporal dynamic behavior. The activity in a recurrent neural network that is responsive to an input occurs over time. For example, the results of information processing can be fed back to nodes that have performed other processing operations. As another example, forward propagation through the network can include delays that coordinate the arrival of information. 
     Because of this temporal dynamic behavior, the response of a recurrent neural network to a given input can reflect prior input to the network. For example, a recurrent neural network that is quiescent may respond differently to a given input than it would if it were still responding to a previous input. 
     The present methods and apparatus exploit the temporal dynamic behavior of a recurrent neural network to provide improved information processing and a more robust output—and interpretation of that output. The temporal dynamic behavior of a recurrent neural network is interpreted as a process whereby relevant processing results are progressively reinforced or even amplified and irrelevant processing results are progressively attenuated or even discarded. The reinforcement and/or attenuation can reflect a decision being based on:
         a larger sample of an input that changes over time (such as, e.g., a longer sample of a video or an audio stream),   repeated input of a non-changing input,   different classes of input data (e.g., audio, video, image, physical parameters), and   an ensemble of results from non-specialized or even universal processing activity in the recurrent neural network.       

     Implementation of such features within a recurrent neural network can help improve the robustness of decision making in the recurrent neural network—as well as the interpretation of the output of a recurrent neural network. Information processing in the recurrent neural network can be progressively reinforced over time. Reliance upon different classes of input data and longer durations of input data prevent noise, failure, or even adversarial perturbation of one class from unduly disturbing information processing by the network. Non-specialized processing activity allows context to be used in decision making. 
     In one aspect, methods, systems, and apparatus, including computer programs encoded on a computer storage medium, are described. For example, a method includes defining a plurality of different windows of time in a recurrent artificial neural network, wherein each of the different windows has different durations, has different start times, or has both different durations and different start times, identifying occurrences of topological patterns of activity in the recurrent artificial neural network in the different windows of time, comparing the occurrences of the topological patterns of activity in the different windows, and classifying, based on a result of the comparison, a first decision that is represented by a first topological pattern of activity that occurs in a first of the windows as less robust than a second decision that is represented by a second topological pattern of activity that occurs in a second of the windows. 
     In another aspect, methods, systems, and apparatus, including computer programs encoded on a computer storage medium, are described. For example, a method includes defining a first window of time and a second window of time in a recurrent artificial neural network, wherein the first window of time starts before the second window of time, identifying a topological pattern of activity in the recurrent artificial neural network that occurs in the first window of time but not in the second window of time, and adjusting one or more characteristics of the recurrent artificial neural network to attenuate or eliminate the occurrence of the topological pattern in the first window of time. 
     These and other aspects can include one or more of the following features. The first window can start before the second window. Data can be successively input into the recurrent artificial neural network and occurrences of the topological patterns can be successively identified in different windows of time that are defined relative to the successive inputs of the data. Each of the different windows of time can define a plurality of start times, defines a plurality of durations, or defines both a plurality of start times and a plurality of durations for the identification of topological patterns. Each of the different windows of time can define at least two durations, with a longer of the durations defined for identification of a more complex topological pattern of activity and a shorter of the durations defined for identification of a less complex topological pattern of activity. Each of the different windows of time can define at least two start times, with a sooner of the start times defined for identification of a topological pattern of activity in a region of the recurrent neural network that is primarily perturbed by a single class of input data and a later of the start times defined for identification of a topological pattern of activity in a region of the recurrent neural network that fuses classes of input data. One or more characteristics of the recurrent artificial neural network can be adjusted to attenuate or eliminate the first decision that is represented by a first topological pattern of activity that occurs in a first window. Occurrences of the topological patterns of activity can be compared by subtracting a first collection of binary digits from a second collection of binary digits, wherein each binary digit indicates whether a respective topological pattern occurred. Occurrences of topological patterns of activity can be identified by identifying occurrences of simplex patterns of activity. For example, the simplex patterns can enclose cavities. The topological pattern of activity can be identified by comparing a collection of topological patterns of activity that occur in the first window of time with a collection of topological patterns of activity that occur in the second window of time. Collections of the topological patterns of activity can be compared by subtracting a first collection of binary digits from a second collection of binary digits, wherein each binary digit indicates whether a respective topological pattern occurred. First data can be input into the recurrent artificial neural network at a time such that the recurrent artificial neural network is perturbed by the first data during the first window of time. Second data can be input into the recurrent artificial neural network at a time such that the recurrent artificial neural network is perturbed by the second data during the second window of time. The first data and the second data can be either first and second images that feature a same subject or first and second text snippets that share a textual characteristic. Each of the first window of time and the second window of time can define a longer duration for identification of a more complex topological pattern of activity and a shorter duration for identification of a less complex topological pattern of activity. The first window of time can be defined for identification of a topological pattern of activity in a region of the recurrent neural network that is primarily perturbed by a single class of input data. The second window of time can be defined for identification of a topological pattern of activity in a region of the recurrent neural network that that fuses classes of input data. Occurrences of topological patterns of activity can be identified by identifying occurrences of simplex patterns of activity. The simplex patterns can enclose cavities. 
     The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims. 
    
    
     
       DESCRIPTION OF DRAWINGS 
         FIG.  1    is a schematic representation of an implementation of an artificial neural network system that abstracts and clusters data that originates from multiple, different sensors. 
         FIGS.  2  and  3    are representations of patterns of activity that can be identified and “read” to generate a collection of digits from neural network. 
         FIG.  4    is a graph that represents occurrences of patterns as a function of time. 
         FIG.  5    is a schematic representation of an approach for inputting data that originates from different sensors into neural network. 
         FIG.  6    is a schematic representation of how the windows that are defined by window definition unit can be used in various contexts. 
         FIG.  7    is a schematic representation of how windows that are defined by window definition unit to identify adversarial attacks can be used to improve the resistance of network to adversarial attacks. 
     
    
    
     Like reference symbols in the various drawings indicate like elements. 
     DETAILED DESCRIPTION 
     Recurrent artificial neural networks can be constructed to implement a variety of different connections that convey information across the network over time. The connections may feed information forward or backward within the network and can be implemented on a variety of different levels and time scales. For example, connections can be implemented on the level of a region or other collection or nodes that are primary perturbed by one type of input data. As another example, connections can be implemented between different regions that are primary perturbed by different types of input data. The time scales for information transmission under such diverse circumstances can also vary. 
     With this in mind, in some implementation, the response of a recurrent neural network to input can be viewed as a process of progressive certainty. An instantaneous perturbation that is responsive to one type of input data is not only fused or combined with perturbations that are responsive to other types of input data, but it is also fused or combined with perturbations that are responsive to same type of input data that occur at different times. The fusion or combination can progressively amplify relevant perturbations and/or progressively dampen irrelevant perturbations. Relevant subnetworks within the recurrent neural network can be activated. Even low-likelihood conclusions can be reached if enough input is received. Further, the conclusions are robust and insensitive to noise, fault, or even adversarial attack. 
       FIG.  1    is a schematic representation of an implementation of an artificial neural network system  100  that abstracts and clusters data  50  that originates from multiple, different sensors. Neural network system  100  includes a collection of inputs  105 , a recurrent neural network  110 , a collection of outputs  115 , and a window definition unit  125 . Window definition unit  130  can be used to determine that certain decisions are not as robust as others and, e.g., may be the result of noise, fault, or even adversarial attack. In some implementations, window definition unit  130  can be used to improve the robustness of recurrent neural network  110 , as discussed further below. 
     Data  50   
     In some implementations, recurrent neural network  110  can be coupled to receive data  50  that originates from multiple, different sensors. The sensors can be, e.g., transducers that convert different physical properties into data or devices that sense only data, such as, e.g., a device that senses the content of a document or data stream. Data  50  may have different formats or other characteristics. For example, certain classes of data  50  (e.g., video or audio data) may change relatively rapidly in time, whereas other classes of data  50  (e.g., a still image or temperature) may change relatively slowly or not at all. 
     In the illustrated implementation, data  50  includes one or more of sound data  60  that originates from, e.g., a microphone, still image data  62  that originates from, e.g., a still camera, video data  64  that originates from, e.g., a video camera, and temperature data  66  that originates from, e.g., a temperature sensor. This is for illustrative purposes only. Data  50  need not include one or more of sound data  60 , still image data  62 , video data  64 , temperature data  66 . Also, data  50  can include one or more of a variety of other different types of data including, e.g., pressure data, chemical composition data, acceleration data, electrical data, position data, or the like. Data  50  that originates from a sensor can undergo one or more processing actions prior to input into neural network  110 . Examples of such processing actions include, e.g., amplitude scaling, time coding, time or phase shifting, and/or non-linear processing in an artificial neural network device. 
     In other implementations, only a single type of input data is received. 
     Network Inputs  105   
     In the illustrated implementation, inputs  105  are schematically represented as a well-defined input layer of nodes that each passively relay the input to one or more locations in neural network  110 . However, this is not necessarily the case. For example, in some implementations, one or more of inputs  105  can scale, delay, phase shift or otherwise process some portion or all of the input data before data is conveyed to neural network  110 . As another example, data may be injected into different layers and/or edges or nodes throughout neural network  110 , i.e., without a formal input layer as such. For example, a user can specify that data is to be injected into specific nodes or links that are distributed throughout network  110 . As another example, neural network  110  need not be constrained to receiving input in a known, previously defined manner (e.g., always injecting a first bit into a first node, the second bit into a second node, . . . etc.). Instead, a user can specify that certain bits in the data are to be injected into edges rather than nodes, that the order of injection need not follow the order that the bits appear, or combinations of these and other parameters. Nevertheless, for the sake of convenience, the representation of inputs  105  as an input layer will be maintained herein. 
     Recurrent Neural Network  110   
     In recurrent neural networks, the connections between nodes form a directed graph along a temporal sequence and the network exhibits temporal dynamic behavior. In some implementations, recurrent neural network  110  is a relatively complex neural network that is modelled on a biological system. In other words, recurrent neural network  110  can itself model a degree of the morphological, chemical, and other characteristics of a biological system. In general, recurrent neural networks  110  that are modelled on biological systems are implemented on one or more computing devices with a relatively high level of computational performance. 
     In contrast with, e.g., traditional feedforward neural networks, recurrent neural networks  110  that are modelled on biological systems may display background or other activity that is not responsive to input data. Indeed, activity may be present in such neural networks  110  even in the absence of input data. However, upon input of data, a recurrent neural network  110  will be perturbed. Since the response of such a neural network  110  to a perturbation may depend, in part, on the state of neural network  110  at the time that data is input, the response of such a neural network  110  to the input of data may also depend on the background or other activity that is already present in neural network  110 . Nevertheless, even though such activity in a neural network is not responsive only to the input of data, it is responsive to input data. 
     The response of neural network  110  to the input data can be read as a collection of topological patterns. In particular, upon the input of data, neural network  110  will respond with a certain activity. That activity will include:
         activity that does not comport with defined topological patterns, and   activity that does comport with defined topological patterns.       

     The activity in neural network  110  that does not comport with defined topological patterns can in some cases be incorrect or incomplete abstractions of the characteristics of the input data, or other operations on the input data. The activity in neural network  110  that does comport with topological patterns can abstract different characteristics of the input data. Each of the abstracted characteristics may be more or less useful depending on the application. By limiting representation  120  to representation of certain topological patterns, both incorrect or incomplete abstractions and abstraction of characteristics that are not relevant to a particular application can be “filtered out” and excluded from representation  120 . 
     At times, neural network  110  will respond to the input of data that originates from different sensors with one or more topological patterns that are the same, even if other topological patterns are different. For example, neural network  110  may respond to either a temperature reading or a still image of a desert with a topological pattern that represents a qualitative assessment of “hot,” even if other topological patterns are also part of the response to each input. Similarly, neural network  110  can respond to the conclusion of a musical composition or a still image of a plate with crumbs with a topological pattern that represents a qualitative assessment of “done,” even if other topological patterns are also part of the response to each input. Thus, at times, the same characteristic may be abstracted from data that has different origins and different formats. 
     At times, neural network  110  will respond to the input of data that originates from different sensors with one or more topological patterns that represent the synthesis or fusion of the characteristics of the data from those sensors. In other words, a single such pattern can represent an abstraction of the same characteristic that is present in different types of data. In general, the fusion or synthesis of data from different sensors will act to cause such patterns to arise or the strength of the activity of such patterns to increase. In other words, data from different sensors can act as “corroborative evidence” that the same characteristic is present in the diverse input data. 
     In some cases, topological patterns that represent the synthesis or fusion of the characteristics of data from different sensors will only arise if certain characteristics are present in the data from different sensors. Neural network  110  can in effect act as an AND gate and require that certain characteristics in data from different sensors in order for certain patterns of activity to arise. However, this need not be the case. Instead, the magnitude of the activity that forms a pattern may increase or the timing of the activity may shorten in response to data from different sensors. In effect, the topological patterns of activity—and their representation in representation  120 —represent abstractions of the characteristics of the input data in a very rich state space. In other words, the topological patterns of activity and their representation are not necessarily the predefined “results” of processing input data in the sense that, e.g., a yes/no classification is the predefined result yielded by a classifier, a set of related inputs is the predefined result yielded by a clustering device, or a prediction is the predefined result yielded by a forecasting model. Rather, the topological patterns are abstractions of the characteristics of the input data. Although that state space may at times include abstractions such as a yes/no classification, the state space is not limited to only those predefined results. 
     Further, the topological patterns may abstract characteristics of only a portion (e.g., a particular region of an image or a particular moment in a video or audio stream or a particular detail of the input such as a pixel) of the input data, rather than the entirety of the input data. Thus, the state space of the abstractions is neither limited to either a predefined type of result (e.g., a classification, a cluster, or a forecast), nor to abstractions of the entirety of the input data. Rather, the topological patterns are a tool that allows the processing by a high-dimensional, non-linear, recurrent dynamic system (i.e., neural network  110 ) to be read. The topological patterns extract correlates of the input data that arise in neural network  110 , including correlates that fuse the data into a more complete “whole.” Further, by virtue of the recurrent nature of the neural network, the fusion occurs over time. As initial operations or abstractions are completed, the results of these initial operations or abstractions can be fused with other operations or abstractions that are completed at the same time or even later. The fusion thus occurs at a different, later time than the initial operations or abstractions. 
     Notwithstanding the different origins and formats, neural network  110  can still abstract characteristics from the data. For example, neural network  110  may abstract:
         physical traits (e.g., color, shape, orientation, speed),   categories (e.g., car, cat, dog), and/or   abstract qualitative traits (e.g., “alive” vs. “dead,” “smooth” vs. “rough,” “animate” vs. “inanimate,” “hot” vs. “cold,” “open” vs. “closed”).       

     If one were to constrain input data to originating from a small number of sensors, it may be unlikely that neural network  110  would abstract the data from that sensor in certain ways. By way of example, it may be unlikely that neural network  110  would abstract temperature data by itself into a pattern of activity that corresponds to a spatial trait like shape or orientation. However, as data from different sensors is input into neural network  110 , the perturbations provoked by diverse input data meet each other and can collectively influence the activity in neural network  110 . As a result, the neural network  110  may abstract input data into different or more certain patterns of activity. 
     For example, there may be a degree of uncertainty associated with the presence or absence of a pattern. If the input data includes data from diverse range of sensors, both the diversity of the patterns and the certainty of the patterns may increase as the data that originates from different sensors is synthesized or fused within the neural network  110 . By way of analogy, a passenger who is sitting in a train at a train station may look out the window and see an adjacent train that appears to be moving. That same passenger may also, e.g., feel forward pressure from the seat. The fusion or synthesis of this information increases the passenger&#39;s degree of certainty that the passenger&#39;s train is moving, rather than the adjacent train. When neural network receives diverse input data, the perturbations provoked by that data can collectively be abstracted into different or more certain patterns of activity. 
     The ability of recurrent neural network  110  to process input data from diverse sensors also provides a degree of robustness to the abstraction of that data. By way of example, one sensor of a group may become inaccurate or even inoperative and yet neural network  110  can continue to abstract data from the other sensors. Often, recurrent neural network  110  will abstract data from the other sensors into the same patterns of activity that would have arisen had all of the sensors been functioning as designed. However, in some instances, the certainty of those abstractions may decrease. Nevertheless, abstraction can continue even if such a problem should arise. 
     Moreover, there are several characteristics of links and nodes that form recurrent neural network  110  that can improve the robustness of a recurrent neural network. One example characteristic is a relatively large fan-out and/or large fan-in of the links that are connected to nodes. In this context, fan-out is the number of nodes or links that receive input from a single output of a node or link. Fan-in is the number of inputs that a node or link receives. The large fan-in and fan-out are schematically illustrated by the dashed-line links discussed above. 
     In some implementations, a single node may output signals to between 10 and 10{circumflex over ( )}6 other nodes, for example, between 10{circumflex over ( )}3 and 10{circumflex over ( )}5 other nodes. In some implementations, a single node may receive signals from between 10 and 10{circumflex over ( )}6 other nodes, for example, between 10{circumflex over ( )}3 and 10{circumflex over ( )}5 other nodes. Such a relatively large fan-out leads to a very dramatic distribution of the results of processing by each node. Further, such a relatively large fan-in allows each node to based processing on input that originates from a legion of different nodes. Any particular fault—be it in the input data or the nodes and links within the recurrent neural network itself—is unlikely to lead to catastrophic failure. 
     Another example characteristic that can improve the robustness of a recurrent neural network is the non-linear transmission of information within the neural network. For example, the links in recurrent neural network  110  can be spike-like transmissions that carry information, e.g., based on the number of spikes within a given time. As another example, the nodes and links in recurrent neural network  110  can have non-linear activation functions, including activation functions that resemble the activation functions of biological neurons. 
     Another example characteristic that can improve the robustness of a recurrent neural network are multi-link connections between individual nodes. In some cases, such multiple links may be purely redundant and convey the exact same information between the connected nodes in the exact same manner. However, in general, multiple links will not convey the exact same information in the exact same manner. For example, different processing results may be conveyed by different links. As another example, the multiple links may convey the same result such that the result arrives at the destination node at different times and/or with different consequences at the receiving node. 
     In some implementations, the links in a recurrent neural network can be either inhibitory or excitatory. Inhibitory links make it less likely that the receiving node outputs a particular signal whereas excitatory links make it more likely that the receiving node outputs a particular signal. In some implementations, nodes may be connected by multiple excitatory links (e.g., between 2 and 20 links or between 3 and 10 links). In some implementations, nodes may be connected by multiple inhibitory links (e.g., between 5 and 40 links or between 10 and 30 links). 
     Multi-link connections both provide a robust connectivity amongst the nodes and help avoid fully deterministic processing. As discussed further below, another characteristic that can contribute to robustness is non-deterministic transmission of information between nodes. Any particular fault—be it in the input data or the nodes and links within the recurrent neural network itself—is unlikely to lead to catastrophic failure because of the distributed transmission of non-deterministic information through multi-link connections. 
     Another example characteristic that can improve the robustness of a recurrent neural network is non-deterministic transmission between individual nodes. A deterministic system is a system that develops future states without randomness. For a given input, a deterministic system will always produce the same output. In the present context, non-deterministic transmission between nodes allows a degree of randomness in the signal that is transmitted to another node (or even output from the recurrent neural network) for a given set input data. The input data is not merely the data that is input to the recurrent neural network as a whole, but also encompasses the signals received by individual nodes within the recurrent neural network. 
     Such randomness can be introduced into the signal transmission in a variety of ways. For example, in some implementations, the behavior of nodes can be non-deterministic. Decision thresholds, time constants, and other parameters can be randomly varied to ensure that a given node does not respond identically to the same input signals at all times. As another example, the links themselves can be non-deterministic. For example, transmission times and amplitude attenuations can be randomly varied to ensure that a given link does not convey the same input signal identically at all times. 
     As yet another example, the behavior of the recurrent neural network as a whole can be non-deterministic and this behavior can impact the transmission of signals between nodes. For example, the recurrent neural network may display background or other activity that is not dependent on the input data, e.g., present even in the absence of input data. Such a background level of activity may lead to non-deterministic transmission between individual nodes even if the nodes and the links are themselves deterministically defined. 
     By introducing a degree of variability into the signal transmission, the processing within the recurrent neural network will inherently be tolerant of minor deviations. In particular, a recurrent neural network that can produce meaningful results notwithstanding a certain amount of variability in the signal transmission within the recurrent neural network will also be able to produce meaningful results if there is a fault—either in the input data or the nodes and links within the recurrent neural network itself. The performance of the recurrent neural network will degrade gracefully rather than catastrophically. 
     For the sake of completeness, a single recurrent neural network  110  need not possess all of these characteristic simultaneously in order to have an improved robustness. Rather, a combination of these characteristics or even individual one of such characteristics can improve robustness to some extent. 
     Network Outputs  115  and Representation  120   
     The abstraction of data by neural network  110  can be read from outputs  115  as, e.g., a collection of (generally binary) digits that each represent the presence or absence of a respective topological pattern of activity in neural network  110  responsive to input data. In some case, each digit in representation  120  represents the presence or absence of a respective pattern of activity in neural network  110 . Representation  120  is only schematically illustrated and representation  120  can be, e.g., one-dimensional vector of digits, a two-dimensional matrix of digits, or other collection of digits. In general, the digits in representation  120  will be binary and indicate in a yes/no manner whether a pattern of activity is present or not. However, this is not necessarily the case. Instead, in some implementations, the digits in representation  120  will be multi-valued. The values can denote characteristics of the presence or absence of a respective pattern of activity in neural network  110 . For example, the values can indicate the strength of the activity or a statistical probability that a specific pattern of activity is in fact present. By way of example, activity that is relatively large in magnitude or that occurs within a relatively short window of time can be considered as indicating that a specific decision has been reached or was likely to have been reached. In contrast, activity that is relatively small in magnitude or that occurs over a relatively longer time can be considered less likely to indicate that a specific decision has been reached. 
     In any case, the responsive patterns of activity represent a specific operation performed by the neural network  110  on the input data. The operation can be arbitrarily complex. A single digit can thus encode an arbitrarily complex operation and a set of digits can convey a set of operations, each with an arbitrary level of complexity. 
     Further, the topological patterns of activity—and their representation in representation  120 —can be “universal” in the sense that they are not dependent on the origin of the data being input into the neural network nor on the application to which representation  129  is applied. Rather, the topological patterns of activity express abstract characteristics of the data that is being input into neural network  110 —regardless of the origins of that data. 
     Typically, multiple topological patterns of activity will arise in response to a single input, whether the input is discrete (e.g., a still photo or a single reading from a transducer that measures a physical parameter) or continuous (e.g., a video or an audio stream). The output representation  120  can thus represent the presence or absence topological structures that arise in the patterns of activity responsive to the input data even in a relatively complex recurrent neural network that is modelled on biological systems. 
     In the illustrated implementation, outputs  115  are schematically represented as a multi-node output layer. However, outputs  115  need not be a multi-node output layer. For example, output nodes  115  can be individual “reader nodes” that identify occurrences of a particular pattern of activity at a particular collection of nodes in neural network  110  and hence read the output of neural network  110 . The reader nodes can fire if and only if the activity at a particular collection of nodes satisfies timing (and possibly magnitude or other) criteria. For example, output nodes  115  can be connected to a collection of nodes in neural network  110  and indicate the presence or absence topological structures based on, e.g., the activity levels of each individual node crossing a respective threshold activation level, a weighted sum of the activity levels of those nodes crossing a threshold activation level, or a non-linear combination of the activity levels of those nodes crossing a threshold activation level. 
     The information in representation  120  is holographically represented in the sense that information about the value of a single digit is distributed across the values of other digits in the representation  120 . In other words, random subsets of digits in representation  120  also contain information about the operations performed by the neural network  110  to input, just at lower resolution than would be present if all the digits in representation  120  were present. As discussed further below, different topological patterns have different degrees of complexity. Some relatively more complex patterns may include relatively less complex patterns. Further, simple patterns can be assembled to more complex patterns. Moreover, relatively high levels of abstraction that are represented by some digits may be correlated to some extent with other abstractions represented by other digits. For example, a digit that represents the occurrence of an abstract qualitative trait like “alive” may be correlated with a digit that represents a qualitative trait like “animate.” Information about the occurrence of some topological patterns thus inherently includes some information about the occurrence of other topological patterns. 
     For the sake of convenience, the remainder of the application will refer to the representation  120  as a collection of binary bits and the FIGS. will illustrate them as such. However, it is to be understood that in all cases the digits of representation  120  can also be multi-valued to encode various aspects of the operations performed by the network. 
     Window Definition Unit  125   
     As discussed above, the response of a recurrent neural network to input can be viewed as a process of progressive certainty in which the perturbations responsive to the instantaneous input is combined or fused with perturbations that are responsive to previous input. The perturbations give rise to topological patterns of activity that are represented by the digits in representation  120 . 
     Window definition unit  125  is a device that is configured to define different windows of time for the detection of topological patterns of activity and the corresponding representation in representation  120 . In general, window definition unit  125  can define both the duration of the windows and the timing of the windows. 
     As an aside, in artificial recurrent neural network devices, time is artificial and represented using mathematical constructs. For example, rather than requiring a real world passage of time for signals to transmit from node to node, such signals can be represented in terms of artificial units that are generally unrelated to the real world passage of time—as measured in computer clock cycles or otherwise. Nevertheless, the state of an artificial recurrent neural network device can be described as “dynamic” in that it changes with respect to these artificial units. For the sake of convenience, these artificial units are referred to herein as “time.” Nevertheless, it is to be understood that these units are artificial—even when they are scaled to real-world time in recurrent neural networks that are closely modelled on biological systems—and generally do not correspond to the real world passage of time. 
     Further, as used herein, a “window” for the detection of topological patterns of activity need not be constrained to a single duration of the time. As discussed further below, there are various factors that can influence the timing of topological patterns of activity in a recurrent neural network. These factors include, e.g., the complexity of the topological patterns and the regions in which topological patterns of activity are identified. With such factors in mind, a single “window” can be defined to have different durations and/or different start times for the identification of different patterns. By way of example, a single “window” can be defined to have relatively short durations for the identification of relatively simple topological patterns of activity but relatively longer durations for the identification of relatively complex topological patterns of activity. The durations can be defined based on factors such as, e.g., transmission delays, response times, and decay times associated with various parameters in the recurrent neural network. As another example, a single “window” can be defined to have a variety of different start times for identification of different patterns of activity in different regions of a neural network. For example, the start time(s) of a single window for a region that is primarily perturbed by only a single class of input data can be prior to the start time(s) of that same single window for a region that fuses multiple classes of input data. Once again, the start times can be defined based on factors such as, e.g., transmission delays, response times, and decay times associated with various parameters in the recurrent neural network. 
     Returning to window definition unit  125 , the different windows of time can be tailored to achieve a number of different objectives. For example: 
     Inter-pattern Window Definition: As discussed further below, different topological patterns of activity can have different complexities. In some implementations, window definition unit  125  can define different duration windows to accommodate the different complexities of different patterns. For example, when activity that comports with relatively complex topological patterns is to be identified, window definition unit  125  can define longer duration windows than when activity that comports with relatively simple topological patterns is to be identified.
 
Window Definition for Response to Input Duration: Some types of input inherently occur over relatively long durations (e.g., a video or an audio stream), whereas other types of input inherently occur over relatively short durations (e.g., a still image or a single data point). The topological patterns of activity that arise in response to different inputs can be scaled and shifted in time according to the duration of the input and the timing of the occurrence of the input. In some implementations, when activity that is responsive to relatively longer duration input is to be identified, window definition unit  125  can define longer duration windows and/or windows that occur later in time than when activity that is responsive to relatively shorter duration input is to be identified.
 
Window Definition for Capturing Abstractions of Fused Input: In some implementations, abstractions of input data that originates from different sensors can be made later in time and over relatively longer durations than abstractions of input data that originate from different sensors. For example, as illustrated in  FIG.  5    below, different regions of a recurrent neural network may be primarily perturbed by a single class of input data. Decisions and abstractions that fuse different classes of input data may occur later in time and require relatively longer durations than decisions and abstractions that are based on a single class of input data. When activity that is responsive to data that originates from different sensors is to be identified, window definition unit  125  can define longer duration windows and/or windows that occur later in time than when activity that is responsive to a single sensor is to be identified.
 
Window Definition for Measuring Certainty: In some implementations, windows can be defined to monitor changes in the certainty of decisions. In particular, initial abstractions of input data may be incomplete and/or inaccurate. Since a recurrent neural network processes information over time, with feedback loops and long-lived changes in the states of nodes and/or links, relevant processing results can be progressively reinforced or even amplified and irrelevant processing results can be progressively attenuated or even discarded. When activity that represents higher certainty results is to be identified, window definition unit  125  can define longer duration windows and/or windows that occur later in time than when activity that is less certain is to be identified.
 
Window Definition for Identifying Adversarial Attack: In some implementations, windows can be defined to identify an adversarial attack on the network. Image classifiers and other neural networks can be sensitive to small, non-random perturbations of the input data. The impact of such small, non-random perturbations is greatest on short times scales, i.e., before relevant processing results can be progressively reinforced, before irrelevant processing results can be progressively attenuated, before input that occur over relatively long durations has been input in its entirety, and/or before input data that originates from different sensors can be fused. When activity that arises soon after input or on short duration time scales is identified, window definition unit  125  can facilitate identification of (attempted) adversarial attacks on the recurrent neural network.
 
       FIG.  2    is a representation of patterns  400  of activity that can be identified and “read” to generate collection  120  from neural network  110  ( FIG.  1   ). 
     Patterns  400  are representations of activity within a recurrent artificial neural network. To read patterns  400 , a functional graph is treated as a topological space with nodes as points. Activity in nodes and links that comports with patterns  400  can be recognized as ordered regardless of the identity of the particular nodes and/or links that participate in the activity. In the illustrated implementation, patterns  400  are all directed cliques or directed simplices. In such patterns, activity originates from a source node that transmits signals to every other node in the pattern. In patterns  400 , such source nodes are designated as point 0 whereas the other nodes are designated as points 1, 2, . . . . Further, in directed cliques or simplices, one of the nodes acts a sink and receives signals transmitted from every other node in the pattern. In patterns  400 , such sink nodes are designated as the highest numbered point in the pattern. For example, in pattern  405 , the sink node is designated as point 2. In pattern  410 , the sink node is designated as point 3. In pattern  415 , the sink node is designated as point 3, and so on. The activity represented by patterns  400  is thus ordered in a distinguishable manner. 
     Each of patterns  400  has a different number of points and reflects ordered activity in a different number of nodes. For example, pattern  405  is a 2-simplex and reflects activity in three nodes, pattern  410  is a 3D-simplex and reflects activity in four nodes, and so on. As the number of points in a pattern increases, so does the degree of ordering and the complexity of the activity. For example, for a large collection of nodes that have a certain level of random activity within a window, some of that activity may comport with pattern  405  out of happenstance. However, it is progressively more unlikely that random activity will comport with the respective of patterns  410 ,  415 ,  420  . . . . The presence of activity that comports with pattern  430  is thus indicative of a relatively higher degree of ordering and complexity in the activity that the presence of activity that comports with pattern  405 . 
     Different duration windows can be defined for different determinations of the complexity of activity. For example, when activity that comports with pattern  430  is to be identified, longer duration windows can be used than when activity that comports with pattern  405  is to be identified. 
       FIG.  3    is a representation of patterns  300  of activity that can be identified and “read” to generate binary digit collection  120  from neural network  110  ( FIG.  1   ). 
     Patterns  300  are groups of directed cliques or directed simplices of the same dimension (i.e., have the same number of points) that define patterns involving more points than the individual cliques or simplices and enclose cavities within the group of directed simplices. 
     By way of example, pattern  305  includes six different three point, 2-dimensions patterns  405  that together define a homology class of degree two, whereas pattern  310  includes eight different three point, 2-dimensions patterns  405  that together define a second homology class of degree two. Each of the three point, 2-dimensions patterns  405  in patterns  305 ,  310  can be thought of as enclosing a respective cavity. The nth Betti number associated with a directed graph provides a count of such homology classes within a topological representation. 
     The activity represented by patterns such as patterns  300  represents a relatively high degree of ordering of the activity within a network that is unlikely to arise by random happenstance. Patterns  300  can be used to characterize the complexity of that activity. 
     In some implementations, only some patterns of activity are identified and/or some portion of the patterns of activity that are identified are discarded or otherwise ignored. For example, with reference to  FIG.  2   , activity that comports with the five point, 4-dimensional simplex pattern  415  inherently includes activity that comports with the four point, 3-dimensional and three point, 2-dimension simplex patterns  410 ,  405 . For example, points 0, 2, 3, 4 and points 1, 2, 3, 4 in 4-dimensional simplex pattern  415  of  FIG.  2    both comport with 3-dimensional simplex pattern  410 . In some implementations, patterns that include fewer points—and hence are of a lower dimension—can be discarded or otherwise ignored. As another example, only some patterns of activity need be identified. For example, in some implementations only patterns with odd number of points (3, 5, 7, . . . ) or even numbers of dimensions (2, 4, 6, . . . ) are identified. Notwithstanding the identification of only some patterns, information about the activity in the neural network can nevertheless be holographically represented, i.e., at lower resolution that if all patterns of identified and/or represented in an output. 
     As discussed above, the patterns of activity that are responsive to input data  50  represent a specific operation of arbitrary complexity performed by the neural network  110  on that input data  50 . In some implementations, the complexity of the operation will be reflected in the complexity of the topological pattern. For example, the operation or abstraction represented by the five point, 4-dimensional simplex pattern  415  may be more complex than the operations or abstractions represented by the four point, 3-dimensional and three point, 2-dimension simplex patterns  410 ,  405 . In such cases, digits that represent the presence of activity convey that a set operations or abstractions is performed in neural network  110 , where each of these operations or abstractions has an arbitrary level of complexity. 
       FIG.  4    is a schematic representation of a determination of the timing of activity patterns that have a distinguishable complexity. The determination represented in  FIG.  4    can be performed as part of an identification or “reading” of patterns of activity to generate binary digit collection  120  from neural network  110  ( FIG.  1   ). 
       FIG.  4    includes a graph  605  and a graph  610 . Graph  605  represents occurrences of patterns as a function of time along the x-axis. In particular, individual occurrences are represented schematically as vertical lines  606 ,  607 ,  608 ,  609 . Each row of occurrences can be instances where activity matches a respective pattern or class of pattern. For example, the top row of occurrences can be instances where activity matches pattern  405  ( FIG.  2   ), the second row of occurrences can be instances where activity matches pattern  410  ( FIG.  2   ), the third row of occurrences can be instances where activity matches pattern  415  ( FIG.  2   ), and so on. 
     Graph  605  also includes dashed rectangles  615 ,  620 ,  625  that schematically delineate different windows of time when the activity patterns have a distinguishable complexity. As shown, the likelihood that activity in the recurrent artificial neural network matches a pattern indicative of complexity is higher during the windows delineated by dashed rectangles  615 ,  620 ,  625  than outside those windows. 
     Graph  610  represents the complexity associated with these occurrences as a function of time along the x-axis. Graph  610  includes a first peak  630  in complexity that coincides with the window delineated by dashed rectangle  615  and a second peak  635  in complexity that coincides with the window delineated by dashed rectangles  620 ,  625 . As shown, the complexity represented by peaks  630 ,  635  is distinguishable from what can be considered to be a baseline level  640  of complexity. 
     In some implementations, the times at which the output of a recurrent artificial neural network is to be read coincide with the occurrences of activity patterns that have a distinguishable complexity. For example, in the illustrative context of  FIG.  4   , the output of a recurrent artificial neural network can be read at peaks  630 ,  635 , i.e., during the windows delineated by dashed rectangles  615 ,  620 ,  625 . 
     In some implementations, not only the content but also the timing of the activity patterns that have a distinguishable complexity can be output from the recurrent artificial neural network. In particular, not only the identity and activity of the nodes that participate in activity that comports with the activity patterns, but also the timing of the activity patterns can be considered the output of the recurrent artificial neural network. The identified activity patterns as well as the timing when this decision is to be read can thus represent the result of processing by the neural network. 
       FIG.  5    is a schematic representation of an approach for inputting data  50  that originates from different sensors into neural network  110 . In the illustrated implementation, different subsets  105 ′,  105 ″,  105 ″′ of network inputs  105  are dedicated to receiving different types of input data. For example, a first subset  105 ′ can be dedicated to receiving a first class of input data (e.g., data that originates from a first sensor) whereas a second subset  105 ″ can be dedicated to receiving a second class of input data (e.g., data that originates from a second sensor). 
     In some implementations, corresponding “regions”  505 ,  510  of neural network  110  receive different classes of input data from different subsets  105 ′,  105 ″,  105 ″′ of network inputs  105 . For example, in the schematic illustration, regions  505 ,  510  are shown spatially discrete collections of nodes and edges with relatively few node-to-node connections between each region. This is not necessarily the case. Rather, the nodes and edges of each region  505 ,  510  can be spatially distributed within neural network  110  but yet receive a particular class of input data. 
     Regardless the distribution of the nodes in each region  505 ,  510 , the processing in each region  505 ,  510  is primarily—but not necessarily exclusively—perturbed by the respectively received class of input data. The extent of perturbation can be measured based on the activity that occurs in a region with and without the respective class of input data being present. For example, a region that is primarily perturbed by a first class of input data may respond to the first class of input data in generally the same manner regardless of whether other classes of input data perturb network  110  at the same time. The processing and abstractions performed by each region  505 ,  510  are primarily influenced by the received class of input data. Nevertheless, the topological patterns of activity that arise in each region  505 ,  510  can be read as a digit collection  120 . The same is true for other regions of recurrent neural network  110 . 
     This schematically represented in neural network system  500  by separately designating different subsets  115 ′,  115 ″,  115 ″′ of network outputs  115 . In particular, subset  115 ′ can be dedicated to outputting digits that represent topological patterns of activity that arise in region  505  of neural network  110 , whereas subset  115 ″′ can be dedicated to outputting digits that represent topological patterns of activity that arise in region  5100  of neural network  110 . However, subset  115 ″ outputs digits that are not found in either of regions  505 ,  510 . Indeed, the digits that are output in subset  115 ″ may represent a fusion or further abstraction of the abstract representations and processing results that arise in regions  505 ,  510  to a higher level of complexity. 
     For example, a given digit in subset  115 ″ may arise if and only if both one or more digits in subset  115 ′ and one or more digit in subset  115 ″′ have certain values. The digit in subset  115 ″ can thus represent an arbitrarily higher level abstraction—both of the abstractions generated in regions  505 ,  510  but also of the input data itself. 
     When different regions are primarily perturbed by a single class of input data, the processing in those regions can be tailored to the nature of the input data. For example, the depth of connection and the topology of network loops can be tailored to the input data. In recurrent neural networks that are modelled on biological systems, neuronal dynamics and synaptic plasticity can also be tailored to the input data. The tailoring, e.g., capture different time scales. For example, the processing in a region that is tailored to processing classes of input data that changes relatively rapidly (e.g., video or audio data) can be faster than the processing in a region that is tailored to processing classes of input data that changes relatively slowly or not at all. 
     Further, when different regions of a recurrent neural network are primarily perturbed by a single class of input data, it is easier for humans to attribute the representations that arise in a recurrent neural network to particular input data. The representations that arise in a particular region can be attributed to the class of input data that primarily perturbs that region. Once the representations that arise in a particular region are attributed, higher level and more complex abstractions that arise in response to the representations in a particular region can also be more easily understood. 
     Also, training can be targeted to portions of a recurrent neural network that are not primarily perturbed by a single class of input data, i.e., targeted to the portions of a recurrent neural network that fuse the processing results of regions that are primarily perturbed by a single class of input data. In effect, the regions that are primarily perturbed by a single class of input data will generate representations of the input data that are universal—not only for output from the recurrent neural network but also for further abstraction and other operations with the recurrent neural network. 
       FIG.  6    is a schematic representation of how the windows that are defined by window definition unit  125  can be used in various contexts, such as identifying activity that is responsive to input of different durations, capturing abstractions of fused input, measuring certainty, and identifying adversarial attack. The illustrated implementation of system  100  includes a plurality of data buffers  605 ,  610  and a comparator  615 . Data buffers  605 ,  610  can be implemented as any of a number of different data storage devices. Comparator  615  is a device that is configured to compare a collection of binary or other digits and/or determine a difference therebetween. Comparator  615  can be implemented as hardware and/or software. Although only two data buffers  605 ,  610  are illustrated, more than two data buffers can be used in some contexts. 
     In operation, window definition unit  125  defines the duration and/or the timing of different windows for output of different representations  120  of topological patterns. The different representations  120  can represent topological patterns that occur, e.g., within windows that have a same duration but occur at different times, windows that have occur at overlapping times but have different durations, or windows that have occur at different times and different durations. Data buffer  605  is coupled to store a first set of representations  120  of topological patterns. Data buffer  610  is coupled to store a second set of representations  120  of topological patterns. In the illustrated implementation, the stored representations  120  are schematically illustrated as a two-dimensional matrix of binary digits. In other implementations, representations  120  can be stored, e.g., as one-dimensional vectors and/or as multi-valued digits. 
     Comparator  615  is coupled to receive the sets of representations  120  that are stored in data buffers  605 ,  610 , compare them, and output the results of the comparison. The comparison can take a number of different forms depending on the application context. The result of the comparison can also take multiple forms, but in general will be a collection of digits that represent topological patterns that satisfy certain criteria. Thus, although comparator  615  is illustrated as having a single output  620 , comparator  615  can also include several outputs to output multiple digits in parallel. 
     For example, in implementations where activity that is responsive to relatively long duration input is to be identified, comparator  615  can identify digits in representations  120  that occur at different times during the duration of the input. For example, digits that are common to different feature matrices or vectors can be identified by comparator  615 . The common digits can be output, e.g., in series or in parallel. 
     As another example, in implementations where activity that is responsive to relatively short duration input is to be identified, comparator  615  can identify digits in a representation  120  that occurs immediately after input of the short duration input but that are not present in other representations  120  that occur at other times. For example, comparator can subtract a feature matrix or vector that includes representations that occur shortly prior to the input of the short duration input from a feature matrix or vector that includes representations that occur shortly after the input. 
     As another example, in implementations where abstractions of fused input are to be identified, comparator  615  can be used to identify digits in a representation  120  that arise at a certain time after the input of the data. For example, a collection of remainder digits (i.e., digits that remain after subtraction of digits that arise a relatively short time after an input from digits that arise a relatively long time after an input) can be identified as abstractions that result from fusing the input from different sensors. 
     As another example, in implementations where the certainty of a decision or abstraction is to be measured, comparator  615  can be used to identify digits that reoccur or even become reinforced in different representations  120 . The reoccurring or reinforced digits can be identified, e.g., after successive input of relatively slow changing data such as, e.g., a series of images that feature the same subject, a series of text snippets that share a textual characteristic (e.g., a subject, a theme, a tense), or the like. In some implementations, the reinforcement of digits can be identified based on the value of a non-binary digit, where the value reflects, e.g., the level of activity that forms a topological pattern and/or the duration of the topological pattern. 
     As another example, in implementations where an adversarial attack is to be identified, comparator  615  can be used to identify digits that occur soon after input and are progressively attenuated in subsequent representations  120 . The attenuated digits can be identified, e.g., after successive input of relatively slow changing data such as, e.g., a series of images that may or may not feature the same subject, a series of text snippets, or the like. In some implementations, the attenuation of digits can be identified based on the value of a non-binary digit, where the value reflects, e.g., the level of activity that forms a topological pattern and/or the duration of the topological pattern. 
       FIG.  7    is a schematic representation of how windows that are defined by window definition unit  125  to identify adversarial attacks can be used to improve the resistance of network  110  to adversarial attacks. In addition to data buffers  605 ,  610  and comparator  615 , the illustrated implementation of system  100  includes a training unit  705 . 
     Training unit  705  is a device that is configured to iteratively alter one or more attributes of recurrent neural network  110  based on digits that are determined to represent less robust decisions or abstractions in neural network  110 . For example, topological patterns that occur soon after input can be progressively attenuated or even eliminated from subsequent representations  120 . As another example, topological patterns that do not reoccur after successive input of, e.g., a series of images that feature the same subject a series of text snippets that share a textual characteristic can be progressively attenuated or even eliminated from subsequent representations  120 . 
     In order to attenuate or eliminate the decisions or abstractions, training unit  705  can add or remove either nodes or links from neural network  110 , change the weights of links, change the topological patterns that are represented in digit collection  120 , or other modify recurrent neural network  110 . In implementations where recurrent neural network  110  is a relatively complex neural network that is modelled on a biological system, training unit  705  can alter, e.g., morphological, chemical, or other characteristics of the model. 
     The training of recurrent neural network  110  to produce a more robust collection of digits  120  that, e.g., resists adversarial attack can be beneficial in ways that extend beyond mere identification of an (attempted) adversarial attack. For example, as discussed above, in recurrent neural networks that are used for image classification, susceptibility to adversarial attack can be taken as a sign that classes intersect in the region of the adversarial examples. By training recurrent neural network  110  to resist adversarial examples, the classes can be better defined. 
     Training may also be useful even in the absence of an adversarial attack, e.g., to ensure that neural network  110  is sufficiently insensitive to noise and/or sufficiently sensitive or insensitive to the content of input data. 
     Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). 
     The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources. 
     The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures. 
     A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network. 
     The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). 
     Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. 
     To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user&#39;s client device in response to requests received from the web browser. 
     While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. 
     Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. 
     Thus, particular implementations of the subject matter have been described. Other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous. 
     A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made. Accordingly, other implementations are within the scope of the following claims.