Patent Publication Number: US-2021182654-A1

Title: Input into a neural network

Description:
TECHNICAL FIELD 
     This invention relates to artificial neural networks, and more particularly to abstracting data that originates from different sensors and transducers using artificial neural networks. 
     BACKGROUND 
     Artificial neural networks are devices that are inspired by the structure and functional aspects of networks of biological neurons. In particular, artificial neural networks mimic the information encoding and other processing capabilities of networks of biological neurons using a system of interconnected constructs called nodes. The arrangement and strength of connections between nodes in an artificial neural network determines the results of information processing or information storage by the artificial neural network. 
     Neural networks can be trained to produce a desired signal flow within the network and achieve desired information processing or information storage results. In general, training a neural network will change the arrangement and/or strength of connections between nodes during a learning phase. A neural network can be considered trained when sufficiently appropriate processing results are achieved by the neural network for given sets of inputs. 
     Because training is fundamental to the processing performed by neural networks, these neural networks are generally unable to process data that deviates in form or in content from the data in the training set. By way of example, a neural network image classifier that has been trained to classify images will likely produce meaningless results if audio data is input. As another example, a neural network that has been trained to cluster heart arrhythmias will generally not work if telecommunications signals are input. 
     SUMMARY 
     Abstracting data that originates from different sensors and transducers using artificial neural networks is described. 
     In one implementation, a method includes identifying topological patterns of activity in a recurrent artificial neural network and outputting a collection of digits. The topological patterns are responsive to an input, into the recurrent artificial neural network, of first data originating from a first sensor and second data originating from a second sensor. Each topological pattern abstracts a characteristic shared by the first data and the second data. The first and second sensors sense different data. Each digit represents whether one of the topological patterns of activity has been identified in the artificial neural network. Corresponding systems and apparatus, including computer programs encoded on a computer storage medium, are also possible. 
     In another implementation, a method includes identifying topological patterns of activity in a recurrent artificial neural network and outputting a collection of digits. Each of the topological patterns indicates performance of a specific operation in the recurrent artificial neural network on both first data originating from a first sensor and second data originating from a second sensor. The first and second sensors sense different data. Each digit represents whether one of the topological patterns of activity has been identified in the artificial neural network. Corresponding systems and apparatus, including computer programs encoded on a computer storage medium, are also possible. 
     In another implementation, a recurrent artificial neural network includes a first region that is configured to receive data originating from a first sensor, a second region that is configured to receive data originating from a second sensor, and a third region that is configured to receive results of processing by both the first region and by the second regions. The first region is primarily perturbed by data originating from the first sensor and the second region is primarily perturbed by data originating from the second sensor even when both regions are perturbed at the same time. The third region is configured to output indications of the presence of topological patterns of activity that are responsive to the results of the processing by the first region and by the second regions. Corresponding systems and apparatus, including computer programs encoded on a computer storage medium, are also possible. 
     In another implementation, a device includes a hierarchical system of recurrent artificial neural networks. A first level of the hierarchical system includes a first recurrent artificial neural network configured to receive data originating from a first sensor and to output first indications of the presence of topological patterns of activity responsive to the input of the data originating from the first sensor and a second recurrent artificial neural network configured to receive data originating from a second sensor and to output second indications of the presence of topological patterns of activity responsive to the input of the data originating from the second sensor, wherein the first and second sensors sense different data. A second level of the hierarchical system includes a third recurrent artificial neural network configured to receive the indications of the presence of topological patterns of activity in the first recurrent artificial neural network and the indications of the presence of topological patterns of activity in the second recurrent artificial neural network. The third recurrent artificial neural network is configured to abstract a characteristic of the shared by the first indications and the second indications. Corresponding systems and apparatus, including computer programs encoded on a computer storage medium, are also possible. 
     These and other implementations can include one or more of the following features. The first data originating from the first sensor and the second data originating from the second sensor can be input into the recurrent artificial neural network. The first and the second data can be input into the recurrent artificial neural network sufficiently close in time such that perturbations responsive to the input of the first and second data are present in the recurrent artificial neural network at the same time. The topological patterns of activity can be clique patterns. The clique patterns of activity can enclose cavities. A plurality of windows of time during which the activity of the artificial neural network is responsive to an input into the artificial neural network can be defined. The topological patterns of activity can be identified in each of the pluralities of windows of time. The first sensor can produce a stream of output data and the second sensor can produce slower changing or static output data. The slower changing or static output data can be rate coded. The rate coded data can be input into the recurrent artificial neural network at a same time as when the data that originates from the first transducer is input into the recurrent artificial neural network. Inputting the data originating from the first and second sensors can include delaying input of data originating from the first transducer to coordinate the input of the data originating from the first sensor with the input of the data originating from the second sensor. The input of data originating from the first sensor can be interrupted and identification of the topological patterns of activity and output the collection of digits can be continued during the interruption. Prior to the interruption, the identified topological patterns of activity can include a first pattern and during the interruption, the identified topological patterns of activity also includes the same first pattern. Inputting the data originating from the first and the second sensors can include scaling a magnitude of the data originating from the first sensor based on the data originating from the second sensor. Inputting the data originating from the first and the second sensors can include setting a parameter of the recurrent artificial neural network based on the data originating from the second sensor. Inputting the data originating from the first and the second sensors can include setting a parameter of an output of data from the recurrent artificial neural network based on the data originating from the second sensor. Inputting the data originating from the first and the second sensors can include inputting the data originating from the first sensor into a first region of the recurrent neural network, wherein the first region is primarily perturbed by data of the class that originates from the first sensor and inputting the data originating from the second sensor into a second region of the recurrent neural network, wherein the second region is primarily perturbed by data of the class that originates from the second sensor. The digits can be multi-valued and represent a probability that the topological pattern of activity is present in the artificial neural network. The first and second sensors can be first and second transducers that convert different physical properties into data. Third data originating from a third sensor can be input into the recurrent artificial neural network. The third sensor senses data that differs from the first and second data and the third data can be input into the recurrent artificial neural network such that perturbations responsive to the input of the first data, the second data, and third data are present in the recurrent artificial neural network at the same time. Topological patterns of activity that abstract a characteristic shared by the first data, the second data, and the third data can be identified. 
     Each of the first regions and the second regions can be an identifiably discrete collection of nodes and edges with relatively few node-to-node connections between each region. The first region can configured to output indications of the presence of topological patterns of activity that are primarily responsive to the input of the data originating from the first sensor. The first sensor can produce a stream of output data and the second sensor can produce slower changing or static output data. A rate coder can be coupled to rate code the slower changing or static output data and input the rate coded data into the second region at a same time as when the data that originates from the first sensor is input into the first region. A means for scaling can scale a magnitude of the data originating from the first sensor prior to receipt by the first region, wherein the scaling is based on the data originating from the second sensor. An input can be coupled to inject some of the data originating from the first sensor into a node or link of the recurrent neural network. The input can include a delay or a scaling element. A magnitude of the delay or of the scaling can be based on the data originating from the second sensor. A first level of the hierarchical system can include a third recurrent artificial neural network configured to receive data originating from a third sensor and to output third indications of the presence of topological patterns of activity responsive to the input of the data originating from the third sensor. The third recurrent artificial neural network can be configured to receive the third indications of the presence of topological patterns of activity in the third recurrent artificial neural network and to abstract a characteristic of the shared by the first indications, the second indications, and the third indications. 
     The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims. 
    
    
     
       DESCRIPTION OF DRAWINGS 
         FIG. 1  is a schematic representation of an implementation of an artificial neural network system that abstracts data that originates from multiple, different sensors. 
         FIGS. 2 and 3  are representations of patterns of activity that can be identified and “read” to generate a collection of digits from neural network. 
         FIG. 4  is a graph that represents occurrences of patterns as a function of time. 
         FIG. 5  is a schematic representation of a variety of processing actions that can be performed on data that originates from different sensors prior to input into a neural network. 
         FIGS. 6-10  are schematic representations of approaches for inputting data that originates from different sensors into neural network. 
         FIG. 11  is a schematic representation of an approach for outputting data from a neural network into which data that originates from different sensors is input. 
         FIG. 12  is a schematic illustration of the use of a binary digit collection that represents the occurrence of topological structures in the activity in a neural network. 
         FIG. 13  is a schematic illustration of a hierarchical system of recurrent neural networks that abstracts data that originates from multiple, different sensors. 
     
    
    
     Like reference symbols in the various drawings indicate like elements. 
     DETAILED DESCRIPTION 
       FIG. 1  is a schematic representation of an implementation of an artificial neural network system  100  that abstracts data  50  that originates from multiple, different sensors. Neural network system  100  includes a collection of inputs  105 , the neural network  110  itself, and a collection of outputs  115 . Neural network  110  can receive data  50  that originates from multiple, different sensors over inputs  105 . The sensors can be, e.g., transducers that convert different physical properties into data or devices that sense only data, such as, e.g., a device that senses the content of a document or data stream. The abstraction of data  50  by neural network  110  can be read from outputs  115  as, e.g., a collection of (generally binary) digits  120  that each represent the presence or absence of a respective topological pattern of activity in neural network  110  responsive to input data  50 . These responsive patterns of activity represent a specific operation performed by the neural network  110  on input data  50 . The operation can be arbitrarily complex. A single digit can thus encode an arbitrarily complex operation and a set of digits can convey a set operations, each with an arbitrary level of complexity. 
     Further, the topological patterns of activity—and their representation in digits  120 —can be “universal” in the sense that they are not dependent on the origin of the data being input into the neural network. Rather, the topological patterns of activity express abstract characteristics of the data  50  that is being input into neural network  110 —regardless of the origins of that data. 
     In more detail, data  50  may originate from different sensors and have different formats. For example, certain classes of data  50  (e.g., video or audio data) may change relatively rapidly in time, whereas other classes of data  50  (e.g., still image or temperature) may change relatively slowly or not at all. Notwithstanding the different origins and formats, neural network  110  can still abstract characteristics from the data. For example, neural network  110  may abstract:
         physical traits (e.g., color, shape, orientation, speed),   categories (e.g., car, cat, dog), and/or   abstract qualitative traits (e.g., “alive” vs. “dead,” “smooth” vs. “rough,” “animate” vs. “inanimate,” “hot” vs. “cold,” “open” vs. “closed”).       

     The topological patterns of activity—and their representation in digits  120 —are the response of neural network  110  to the diverse data  50 . Typically, multiple topological patterns of activity will arise in response to a single input, whether the input is discrete (e.g., a still photo or a single reading from a transducer that measures a physical parameter) or continuous (e.g., a video or an audio stream). 
     At times, neural network  110  will respond to the input of data  50  that originates from different sensors with one or more topological patterns that are the same, even if other topological patterns are different. For example, neural network  110  may respond to either a temperature reading or a still image of a desert with a topological pattern that represents a qualitative assessment of “hot,” even if other topological patterns are also part of the response to each input. Similarly, neural network  110  can respond to the conclusion of a musical composition or a still image of a plate with crumbs with a topological pattern that represents a qualitative assessment of “done,” even if other topological patterns are also part of the response to each input. Thus, at times, the same characteristic may be abstracted from data that has different origins and different formats. 
     At times, neural network  110  will respond to the input of data  50  that originates from different sensors with one or more topological patterns that represent the synthesis or fusion of the characteristics of the data from those sensors. In other words, a single such pattern can represent an abstraction of the same characteristic that is present in different types of data. In general, the fusion or synthesis of data from different sensors will act to cause such patterns to arise or the strength of the activity of such patterns to increase. In other words, data from different sensors can act as “corroborative evidence” that the same characteristic is present in the diverse input data. 
     In some cases, topological patterns that represent the synthesis or fusion of the characteristics of data from different sensors will only arise if certain characteristics are present in the data from different sensors. Neural network  110  can in effect act as an AND gate and require that certain characteristics in data from different sensors in order for certain patterns of activity to arise. However, this need not be the case. Instead, the magnitude of the activity that forms a pattern may increase or the timing of the activity may shorten in response to data from different sensors. In effect, the topological patterns of activity—and their representation in digits  120 —represent abstractions of the characteristics of data  50  in a very rich state space. In other words, the topological patterns of activity and their representation are not necessarily the predefined “results” of processing input data in the sense that, e.g., a yes/no classification is the predefined result yielded by a classifier, a set of related inputs is the predefined result yielded by a clustering device, or a prediction is the predefined result yielded by a forecasting model. Rather, the topological patterns are abstractions of the characteristics the input data. Although that state space may at times include abstractions such as a yes/no classification, the state space is not limited to only those predefined results. 
     Further, the topological patterns may abstract characteristics of only a portion (e.g., a particular region of an image or a particular moment in a video or audio stream or a particular detail of the input such as a pixel) of the input data, rather than the entirety of the input data. Thus, the state space of the abstractions is neither limited to either a predefined type of result (e.g., a classification, a cluster, or a forecast), nor to abstractions of the entirety of the input data. Rather, the topological patterns are a tool that allows the processing by a high-dimensional, non-linear, recurrent dynamic system (i.e., neural network  110 ) to be read. The topological patterns extract correlates of the data from the different sensors that arise in neural network  110 , including correlates that fuse the data into a more complete “whole.” 
     Upon input of data  50 , neural network  110  will respond with a certain activity. That activity will include:
         activity that does not comport with defined topological patterns, and   activity that does comport with defined topological patterns.       

     The activity in neural network  110  that does not comport with defined topological patterns can in some cases be incorrect or incomplete abstractions of the characteristics of the input data  50 . Further, the activity in neural network  110  that does comport with defined topological patterns can abstract different characteristics of the input data  50 . Each of the abstracted characteristics may be more or less useful depending on the application. By limiting digits  120  to representation of certain topological patterns, both incorrect or incomplete abstractions and abstraction of characteristics that are not relevant to a particular application can be “filtered out” and excluded from digits  120 . 
     In the illustrated implementation, data  50  includes one or more of sound data  60  that originates from, e.g., a microphone, still image data  62  that originates from, e.g., a still camera, video data  64  that originates from, e.g., a video camera, and temperature data  66  that originates from, e.g., a temperature sensor. This is for illustrative purposes only. Data  50  need not include one or more of sound data  60 , still image data  62 , video data  64 , temperature data  66 . Also, data  50  can include one or more of a variety of other different types of data including, e.g., pressure data, chemical composition data, acceleration data, electrical data, position data, or the like. As discussed further below, data  50  that originates from a sensor can undergo one or more processing actions prior to input into neural network  110 . Examples of such processing actions include, e.g., non-linear processing in an artificial neural network device. 
     As discussed above, the topological patterns that arise in neural network  110  are abstractions of the characteristics the input data  50  in a rich state space. If one were to constrain data  50  to originating from a small number of sensors, it may be unlikely that neural network  110  would abstract the data from that sensor in certain ways. By way of example, it may be unlikely that neural network  110  would abstract temperature data  66  by itself into a pattern of activity that corresponds to a spatial trait like shape or orientation. However, as data from different sensors is input into neural network  110 , the perturbations provoked by diverse input data  50  meet each other and can collectively influence the activity in neural network  110 . As a result, the neural network  110  may abstract input data  50  into different or more certain patterns of activity. 
     For example, there may be a degree of uncertainty associated with the presence or absence of a pattern. If data  50  includes data from diverse range of sensors, both the diversity of the patterns and the certainty of the patterns may increase as the data  50  that originates from different sensors is synthesized or fused within the neural network  110 . By way of analogy, a passenger who is sitting in a train at a train station may look out the window and see an adjacent train that appears to be moving. That same passenger may also, e.g., feel forward pressure from the seat. The fusion or synthesis of this information increases the passenger&#39;s degree of certainty that the passenger&#39;s train is moving, rather than the adjacent train. When neural network receives diverse input data  50 , the perturbations provoked by that data can collectively be abstracted into different or more certain patterns of activity. 
     The ability of neural network  110  to process input data  50  from diverse sensors also provides a degree of robustness to the abstraction of that data. By way of example, one sensor of a group may become inaccurate or even inoperative and yet neural network  110  can continue to abstract data  50  from the other sensors. Often, neural network  110  will abstract data  50  from the other sensors into the same patterns of activity that would have arisen had all of the sensors been functioning as designed. However, in some instances, the certainty of those abstractions may decrease. Nevertheless, abstraction can continue even if such a problem should arise. 
     In the illustrated implementation, inputs  105  are schematically represented as a well-defined input layer of nodes that each passively relay the input to one or more locations in neural network  110 . However, this is not necessarily the case. For example, in some implementations, one or more of inputs  105  can scale some portion or all of the input data or perform other preliminary processing before data is conveyed to neural network  110 . As another example, data  50  may injected into different layers and/or edges or nodes throughout neural network  110 , i.e., without a formal input layer as such. For example, a user can specify that data is to be injected into specific nodes or links that are distributed throughout network  110 . As another example, neural network  110  need not be constrained to receiving input in a known, previously defined manner (e.g., always injecting a first bit into a first node, the second bit into a second node, . . . etc.). Instead, a user can specify that certain bits in the data are to be injected into edges rather than nodes, that the order of injection need not follow the order that the bits appear, or combinations of these and other parameters. Nevertheless, for the sake of convenience, the representation of inputs  105  as an input layer will be maintained herein. 
     In the illustrated implementation, neural network  110  is schematically illustrated as a recurrent neural network. In recurrent neural networks, the connections between nodes form a directed graph along a temporal sequence and the network exhibits temporal dynamic behavior. In some implementations, neural network  110  is a relatively complex neural network that is modelled on a biological system. In other words, neural network  110  can itself model a degree of the morphological, chemical, and other characteristics of a biological system. In general, neural networks  110  that are modelled on biological systems are implemented on one or more computing devices with a relatively high level of computational performance. 
     In contrast with, e.g., traditional feedforward neural networks, neural networks  110  that are modelled on biological systems may display background or other activity that is not responsive to input data  50 . Indeed, activity may be present in such neural networks  110  even in the absence of input data  50 . 
     However, upon input of data  50 , a neural network  110  will be perturbed. Since the response of such a neural network  110  to a perturbation may depend, in part, on the state of neural network  110  at the time that data  50  is input, the response of such a neural network  110  to the input of data  50  may also depend on the background or other activity that is already present in neural network  110 . 
     Nevertheless, even though such activity in a neural network is not responsive only to the input of data  50 , it is responsive to input data  50 . Digits  120  can thus represent the presence or absence topological structures that arise in the patterns of activity responsive to the input data even in a relatively complex neural network that is modelled on biological systems. 
     In the illustrated implementation, outputs  115  are schematically represented as a multi-node output layer. However, outputs  115  need not be a multi-node output layer. For example, output nodes  115  can be individual “reader nodes” that identify occurrences of a particular pattern of activity at a particular collection of nodes in neural network  110  and hence read the output of neural network  110 . The reader nodes can fire if and only if the activity at a particular collection of nodes satisfies timing (and possibly magnitude or other) criteria. For example, output nodes  115  can be connected to a collection of nodes in neural network  110  and indicate the presence or absence topological structures based on, e.g., the activity levels of each individual node crossing a respective threshold activation level, a weighted sum of the activity levels of those nodes crossing a threshold activation level, or a non-linear combination of the activity levels of those nodes crossing a threshold activation level. 
     The output received over outputs  115  is collection of digits  120  that each represent the presence or absence of a respective pattern of activity in neural network  110 , which can be implemented as a relatively complex neural network that models characteristics of a biological system. Collection  120  is only schematically illustrated and collection  120  can be, e.g., one-dimensional vector of digits, a two-dimensional matrix of digits, or other collection of digits. In general, the digits in collection  120  will be binary and indicate in a yes/no manner whether a pattern of activity is present or not. However, this is not necessarily the case. Instead, in some implementations, the digits in collection  120  will be multi-valued. The values can denote characteristics of the presence or absence of a respective pattern of activity in neural network  110 . For example, the values can indicate the strength of the activity or a statistical probability that a specific pattern of activity is in fact present. By way of example, activity that is relatively large in magnitude or that occurs within a relatively short window of time can be considered as indicating that a specific operation has been performed or was likely to have been performed. In contrast, activity that is relatively small in magnitude or that occurs over a relatively longer time can be considered less likely to indicating that a specific operation has been performed. 
     The information in collection  120  is holographically represented in the sense that information about the value of a single digit is distributed across the values of other digits in the collection  120 . In other words, random subsets of digits in collection  120  also contain information about the operations performed by the neural network  110  in response to input  50 , just at lower resolution than would be present if all the digits in collection  120  were present. As discussed further below, different topological patterns have different degrees of complexity. Some relatively more complex patterns may include relatively less complex patterns and simple patterns can be assembled into more complex patterns. Moreover, relatively high levels of abstraction that are represented by some digits may be correlated to some extent with other abstractions represented by other digits. For example, a digit that represents the occurrence of an abstract qualitative trait like “alive” may be correlated with a digit that represents a qualitative trait like “animate.” Information about the occurrence of some topological patterns thus inherently includes some information about the occurrence of other topological patterns. 
     For the sake of convenience, the remainder of the application will refer to the digits of collection  120  as binary bits and the FIGS. will illustrate them as such. However, it is to be understood that in all cases the digits of collection  120  can also be multi-valued to encode various aspects of the operations performed by the network. 
       FIG. 2  is a representation of patterns  400  of activity that can be identified and “read” to generate collection  120  from neural network  110  ( FIG. 1 ). 
     Patterns  400  are representations of activity within a recurrent artificial neural network. To read patterns  400 , a functional graph is treated as a topological space with nodes as points. Activity in nodes and links that comports with patterns  400  can be recognized as ordered regardless of the identity of the particular nodes and/or links that participate in the activity. In the illustrated implementation, patterns  400  are all directed cliques or directed simplices. In such patterns, activity originates from a source node that transmits signals to every other node in the pattern. In patterns  400 , such source nodes are designated as point 0 whereas the other nodes are designated as points 1, 2, . . . . Further, in directed cliques or simplices, one of the nodes acts a sink and receives signals transmitted from every other node in the pattern. In patterns  400 , such sink nodes are designated as the highest numbered point in the pattern. For example, in pattern  405 , the sink node is designated as point 2. In pattern  410 , the sink node is designated as point 3. In pattern  415 , the sink node is designated as point 3, and so on. The activity represented by patterns  400  is thus ordered in a distinguishable manner. 
     Each of patterns  400  has a different number of points and reflects ordered activity in a different number of nodes. For example, pattern  405  is a 2D-simplex and reflects activity in three nodes, pattern  410  is a 3D-simplex and reflects activity in four nodes, and so on. As the number of points in a pattern increases, so does the degree of ordering and the complexity of the activity. For example, for a large collection of nodes that have a certain level of random activity within a window, some of that activity may comport with pattern  405  out of happenstance. However, it is progressively more unlikely that random activity will comport with the respective of patterns  410 ,  415 ,  420  . . . . The presence of activity that comports with pattern  430  is thus indicative of a relatively higher degree of ordering and complexity in the activity that the presence of activity that comports with pattern  405 . 
     Different duration windows can be defined for different determinations of the complexity of activity. For example, when activity that comports with pattern  430  is to be identified, longer duration windows can be used than when activity that comports with pattern  405  is to be identified. 
       FIG. 3  is a representation of patterns  300  of activity that can be identified and “read” to generate binary digit collection  120  from neural network  110  ( FIG. 1 ). 
     Patterns  300  are groups of directed cliques or directed simplices of the same dimension (i.e., have the same number of points) that define patterns involving more points than the individual cliques or simplices and enclose cavities within the group of directed simplices. 
     By way of example, pattern  305  includes six different three point, 2-dimensions patterns  405  that together define a homology class of degree two, whereas pattern  310  includes eight different three point, 2-dimensions patterns  405  that together define a second homology class of degree two. Each of the three point, 2-dimensions patterns  405  in patterns  305 ,  310  can be thought of as enclosing a respective cavity. The nth Betti number associated with a directed graph provides a count of such homology classes within a topological representation. 
     The activity represented by patterns such as patterns  300  represents a relatively high degree of ordering of the activity within a network that is unlikely to arise by random happenstance. Patterns  300  can be used to characterize the complexity of that activity. 
     In some implementations, only some patterns of activity are identified and/or some portion of the patterns of activity that are identified are discarded or otherwise ignored. For example, with reference to  FIG. 2 , activity that comports with the five point, 4-dimensional simplex pattern  415  inherently includes activity that comports with the four point, 3-dimensional and three point, 2-dimension simplex patterns  410 ,  405 . For example, points 0, 2, 3, 4 and points 1, 2, 3, 4 in 4-dimensional simplex pattern  415  of  FIG. 2  both comport with 3-dimensional simplex pattern  410 . In some implementations, patterns that include fewer points—and hence are of a lower dimension—can be discarded or otherwise ignored. As another example, only some patterns of activity need be identified. For example, in some implementations only patterns with odd number of points (3, 5, 7, . . . ) or even numbers of dimensions (2, 4, 6, . . . ) are identified. Notwithstanding the identification of only some patterns, information about the activity in the neural network can nevertheless be holographically represented, i.e., at lower resolution that if all patterns of identified and/or represented in an output. 
     As discussed above, the patterns of activity that are responsive to input data  50  represent a specific operation of arbitrary complexity performed by the neural network  110  on that input data  50 . In some implementations, the complexity of the operation will be reflected in the complexity of the topological pattern. For example, the operation or abstraction represented by the five point, 4-dimensional simplex pattern  415  may be more complex than the operations or abstractions represented by the four point, 3-dimensional and three point, 2-dimension simplex patterns  410 ,  405 . In such cases, digits that represent the presence of activity convey that a set operations or abstractions is performed in neural network  110 , where each of these operations or abstractions has an arbitrary level of complexity. 
       FIG. 4  is a schematic representation of a determination of the timing of activity patterns that have a distinguishable complexity. The determination represented in  FIG. 4  can be performed as part of an identification or “reading” of patterns of activity to generate binary digit collection  120  from neural network  110  ( FIG. 1 ). 
       FIG. 4  includes a graph  605  and a graph  610 . Graph  605  represents occurrences of patterns as a function of time along the x-axis. In particular, individual occurrences are represented schematically as vertical lines  606 ,  607 ,  608 ,  609 . Each row of occurrences can be instances where activity matches a respective pattern or class of pattern. For example, the top row of occurrences can be instances where activity matches pattern  405  ( FIG. 2 ), the second row of occurrences can be instances where activity matches pattern  410  ( FIG. 2 ), the third row of occurrences can be instances where activity matches pattern  415  ( FIG. 2 ), and so on. 
     Graph  605  also includes dashed rectangles  615 ,  620 ,  625  that schematically delineate different windows of time when the activity patterns have a distinguishable complexity. As shown, the likelihood that activity in the recurrent artificial neural network matches a pattern indicative of complexity is higher during the windows delineated by dashed rectangles  615 ,  620 ,  625  than outside those windows. 
     Graph  610  represents the complexity associated with these occurrences as a function of time along the x-axis. Graph  610  includes a first peak  630  in complexity that coincides with the window delineated by dashed rectangle  615  and a second peak  635  in complexity that coincides with the window delineated by dashed rectangles  620 ,  625 . As shown, the complexity represented by peaks  630 ,  635  is distinguishable from what can be considered to be a baseline level  640  of complexity. 
     In some implementations, the times at which the output of a recurrent artificial neural network is to be read coincide with the occurrences of activity patterns that have a distinguishable complexity. For example, in the illustrative context of  FIG. 4 , the output of a recurrent artificial neural network can be read at peaks  630 ,  635 , i.e., during the windows delineated by dashed rectangles  615 ,  620 ,  625 . 
     In some implementations, not only the content but also the timing of the activity patterns that have a distinguishable complexity can be output from the recurrent artificial neural network. In particular, not only the identity and activity of the nodes that participate in activity that comports with the activity patterns, but also the timing of the activity patterns can be considered the output of the recurrent artificial neural network. The identified activity patterns as well as the timing when this decision is to be read can thus represent the result of processing by the neural network. 
       FIG. 5  is a schematic representation of a variety of processing actions  500  that can be performed on data  50  that originates from different sensors prior to input into neural network  110 . As before, the particular types of data and particular structure of inputs  105  are only illustrative. 
     For example, data  50  that originates from an input sensor can be scaled  505  prior to input into inputs  105 . The magnitude of the scaling can be chosen based not only on the characteristics of neural network  110  but also based on the characteristics of the other data  50  that is input into neural network  110 . For example, if a relatively large amount of data  50  is to be input into neural network  110 , individual data points can be scaled by a relatively small scalar to maintain compatibility with, e.g., firing thresholds and other characteristics of neural network  110 . On the other hand, if a relatively small amount of data  50  is to be input into neural network  110 , individual data points can be scaled by a relatively larger scalar to maintain compatibility with, e.g., firing thresholds and other characteristics of neural network  110 . 
     As yet another example, scaling can be used to implement an amplitude-coding scheme. For example, rather than attempting to input a binary or other representation of parameters such as color and temperature into neural network  110 , the same parameters can be encoded using amplitude. 
     As another example, data  50  that originates from an input sensor can be delayed  510  prior to input into inputs  105 . The input of some data can be delayed so that data that originates from different sensors can be synchronized or otherwise coordinated in time. For example, the arrival of data  50  can be timed, e.g., to ensure that the results of processes that require different durations to complete can be input at the same time, to ensure that inhibitory or excitatory effects are properly timed, or to ensure that different stimuli arrive sufficiently close in time that activation thresholds are reached. Delays can also be used to implement logic functions, e.g., to time different stimuli so that one stimulus always arrives during any refractory period resulting from an earlier stimulus. As yet another example, delays can be used to implement network-wide policies that specify, e.g., the number of active nodes at a given time, the amount of input received during a particular window of time, or the like. 
     As yet another example, delays can be used to implement a phase-coding scheme. For example, rather than attempting to input a binary or other representation of parameters such as color and temperature into neural network  110 , the same parameters can be encoded using the timing of input into neural network  110 . 
     As another example, data  50  that originates from an input sensor can be rate coded  515  prior to input into inputs  105 . Rate coding converts the magnitude of a value into a firing rate, e.g., the frequency at which an input signal is presented on inputs  105 . 
     In some implementations, rate coding can be used to input different frequency data  50  into a single neural network  110 , e.g., in the same manner or at the same time. In other words, slow-changing or static data  50  can perturb neural network  110  over a relatively longer period of time. By extending the duration of an input, slow-changing or static data  50  can perturb neural network  110  in a manner that is more akin to the manner in which more dynamic data  50  that changes relatively quickly perturbs neural network  110 . 
     By way of example, still image data  62  is by definition “still,” i.e., the values of individual pixels in the image do not change over time. In contrast, video or audio data is dynamic and the values of individual pixels vary over time. Rather than, e.g., perturbing neural network  110  one time only with inputs that are scaled to the color or other characteristic(s) of individual pixels in still image data  62 , the same characteristic(s) can be rate coded into a series of inputs that perturb neural network  110  over time. Such a perturbation can be input, e.g., in the same manner or at the same time as other perturbations that change with time, such as an audio or video signal. 
     As another example, data  50  that originates from an input sensor can be processed  520  using one or more other neural networks prior to input into inputs  105 . The other neural network(s) can have been trained to arrive at a particular result such as, e.g., classifications of input data, clustering of inputs, or a forecast. 
     As another example, data  50  that originates from an input sensor can be processed  525  to convert dynamic input data into less-dynamic or even static input data. By way of example, video data  64  is dynamic in that it generally changes from one instant to the next, e.g., from frame to frame. 
     Rather than, e.g., perturbing neural network  110  at the same frame rate as video data  64 , video data  64  can be down-sampled in time and input into neural network  110  in the same manner as one would input a series of discrete still images. For example, a particular frame of the video data  64  can be sampled and held on the input of neural network  110  until neural network  110  re-enters a quiescent or other state indicating that abstraction of that particular sample of the video data  64  is complete. 
     In some implementations, such down-sampling can be used to input different frequency data  50  into a single neural network  110 , e.g., in the same manner or at the same time. For example, down-sampled video data  64  can be input in the same manner or at the same time as static still image data  62 . 
     As another example, different data  50  can be multiplexed  525  for input into neural network  110 . For example, a single node or edge in neural network  110  can receive data  50  that originates from different sensors. A number of different schemes can be used to multiplex the input. For example, different data from different sensors can be input into a single node or edge in neural network  110  at different times (time-division multiplexing). As another example, different data from different sensors can be input into a single node or edge in neural network  110  at different frequencies (frequency-division multiplexing). 
     As another example, different data  50  can be processed  530  according to a linear or other function prior to input into neural network  110 . For example, data  50  can be filtered, classified, encoded, or otherwise processed prior to input. The results of the processing can be input into neural network  110 . 
       FIG. 6  is a schematic representation of an approach for inputting data  50  that originates from different sensors into neural network  110 . As shown, each of inputs  105  has at least one projection that extends to, e.g., a node or edge in neural network  110 . In some cases, projections from different inputs  105  may extend to the same node or edge in neural network  110 . Each node or edge can receive a combined input that represents, e.g., a linear superposition or other combination of input received from different sensors. 
       FIG. 7  is a schematic representation of another approach for inputting data  50  that originates from different sensors into neural network  110 . In the illustrated implementation, different subsets  105 ′,  105 ″,  105 ′″ of network inputs  105  are dedicated to receiving different types of input data. For example, a first subset  105 ′ can be dedicated to receiving a first class of input data (e.g., data that originates from a first sensor) whereas a second subset  105 ′ can be dedicated to receiving a second class of input data (e.g., data that originates from a second sensor). 
     In some implementations, corresponding “regions”  705 ,  710  of neural network  110  receive different classes of input data from different subsets  105 ′,  105 ″,  105 ′″ of network inputs  105 . For example, in the schematic illustration, regions  705 ,  710  are shown spatially discrete collections of nodes and edges with relatively few node-to-node connections between each region. This is not necessarily the case. Rather, the nodes and edges of each region  705 ,  710  can be spatially distributed within neural network  110  but yet receive a particular class of input data. 
     Regardless the distribution of the nodes in each region  705 ,  710 , the processing in each region  705 ,  710  is primarily—but not necessarily exclusively—perturbed by the respectively received class of input data. The extent of perturbation can be measured based on the activity that occurs in a region with and without the respective class of input data being present. For example, a region that is primarily perturbed by a first class of input data may respond to the first class of input data in generally the same manner regardless of whether other classes of input data perturb network  110  at the same time. The processing and abstractions performed by each region  705 ,  710  are primarily influenced by the received class of input data. Nevertheless, the topological patterns of activity that arise in each region  705 ,  710  can be read as a digit collection  120 . The same is true for other regions of recurrent neural network  110 . 
     This schematically represented in neural network system  700  by separately designating different subsets  115 ′,  115 ″,  115 ′″ of network outputs  115 . In particular, subset  115 ′ can be dedicated to outputting digits that represent topological patterns of activity that arise in region  705  of neural network  110 , whereas subset  115 ′″ can be dedicated to outputting digits that represent topological patterns of activity that arise in region  710  of neural network  110 . However, subset  115 ″ outputs digits that are not found in either of regions  705 ,  710 . Indeed, the digits that are output in subset  115 ″ may represent a fusion or further abstraction of the abstract representations and processing results that arise in regions  705 ,  710  to a higher level of complexity. 
     For example, a given digit in subset  115 ″ may arise if and only if both one or more digits in subset  115 ′ and one or more digit in subset  115 ′″ have certain values. The digit in subset  115 ″ can thus represent an arbitrarily higher level abstraction—both of the abstractions generated in regions  705 ,  710  but also of the input data itself. 
     When different regions are primarily perturbed by a single class of input data, the processing in those regions can be tailored to the nature of the input data. For example, the depth of connection and the topology of network loops can be tailored to the input data. In recurrent neural networks that are modelled on biological systems, neuronal dynamics and synaptic plasticity can also be tailored to the input data. The tailoring, e.g., capture different time scales. For example, the processing in a region that is tailored to processing classes of input data that changes relatively rapidly (e.g., video or audio data) can be faster than the processing in a region that is tailored to processing classes of input data that changes relatively slowly or not at all. 
     Further, when different regions of a recurrent neural network are primarily perturbed by a single class of input data, it is easier for humans to attribute the representations that arise in a recurrent neural network to particular input data. The representations that arise in a particular region can be attributed to the class of input data that primarily perturbs that region. Once the representations that arise in a particular region are attributed, higher level and more complex abstractions that arise in response to the representations in a particular region can also be more easily understood. 
     Also, training can be targeted to portions of a recurrent neural network that are not primarily perturbed by a single class of input data, i.e., targeted to the portions of a recurrent neural network that fuse the processing results of regions that are primarily perturbed by a single class of input data. In effect, the regions that are primarily perturbed by a single class of input data will generate representations of the input data that are universal—not only for output from the recurrent neural network but also for further abstraction and other operations with the recurrent neural network. 
       FIG. 8  is a schematic representation of another approach for inputting data  50  that originates from different sensors into neural network  110 . The illustrated portion of neural network  110  includes nodes  805 ,  810  and links  815 ,  820 ,  825 ,  830 . In the illustrated implementation, links  815 ,  820  are unidirectional links from node  805  to node  810 . Links  815 ,  820  are also unidirectional links, but connect node  810  to other locations in neural network  110 . 
     Inputs  105  include a first input  835 , a second input  840 , and a third input  845 . In some implementations, first input  835  is dedicated to inputting data that originates from a first sensor, second input  840  is dedicated to inputting data that originates from a second sensor, and third input  845  is dedicated to inputting data that originates from a third sensor. 
     Inputs  835 ,  840 ,  845  can be nodes of an input layer. In the illustrated implementation, inputs  835 ,  840 ,  845  inject data into links of neural network  110 , rather than nodes. In some implementations, inputs  835 ,  840 ,  845  can, e.g., scale some portion or all of the input data or perform other preliminary processing before data is conveyed to neural network  110 . 
     In the illustrated implementation, input  835  injects data into a first position along link  815  over a link  850 . Input  840  injects data into a second position along link  815  over a link  855 . Input  845  injects data into the same first position along link  815  as input  835  over a link  860 . However, link  860  includes a delay element  865 . Delay element delays the arrival of data from input  845  at the first position along link  815 . By delaying data from input  845 , delay element  865  can help coordinate the arrival of data at the first position along link  815  in neural network  110 . 
     As discussed above, such a delay may be useful in a variety of different circumstances. By delaying the arrival of data from input  845  at the first position along link  815 , delay element  865  can help insure that the arrival of data from input  845  is properly synchronized with the arrival of data from input  845 . 
       FIG. 9  is a schematic representation of another approach for inputting data  50  that originates from different sensors into neural network  110 . The illustrated portion of neural network  110  includes a node  905  and links  910 ,  915 ,  920 . In the illustrated implementation, links  910 ,  915  are unidirectional links directed toward node  905  and link  920  is a unidirectional link directed away from node  905 . 
     Inputs  105  include a first input  925 , a second input  930 , and a third input  935 . In some implementations, first input  925  is dedicated to inputting data that originates from a first sensor, second input  930  is dedicated to inputting data that originates from a second sensor, and third input  935  is dedicated to inputting data that originates from a third sensor. 
     Inputs  925 ,  930 ,  935  can be nodes of an input layer. In some implementations, inputs  925 ,  930 ,  935  can, e.g., scale some portion or all of the input data or perform other preliminary processing before data is conveyed to neural network  110 . 
     In the illustrated implementation, input  930  injects data into a first position along link  910  over a link  940 . Inputs  935 ,  940  inject data elsewhere into neural network  110  over a respective of links  945 ,  950 . Further, the data conveyed over inputs  935 ,  940  can be used to scale the magnitude of the input over link  940 . This is schematically represented in  FIG. 9  by a variable amplifier  955 , which scales input over link  940  based on the inputs over links  945 ,  950 . 
     Please note that the illustration of variable amplifier  955  as a voltage controlled amplifier is schematic and for didactic purposes. For example, the data that inputs  935 ,  940  inject into neural network  110  over links  945 ,  950  need not be, e.g., a relatively slow changing voltage. Rather, for example, variable amplifier  955  can change the rate at which rate-coded data is input over link  940  based on the input on links  945 ,  950 . As another example, variable amplifier  955  can change the timing of the input of phase coded data over link  940  based on the input on links  945 ,  950 . 
     Regardless of how it is accomplished, the scaling of the data from input  930  based on data from inputs  925 ,  935  by variable amplifier  955  can help weigh the relative import of data that originates from one sensor based on data that originates from one or more other sensor(s). 
     Such a scaling may be useful in a variety of different circumstances. For example, data from input  930  may become more important if the sensors from which the data input over inputs  925 ,  935  originates are not present. As another example, data from input  930  may increase or decrease in relevance if the sensors from which the data input over inputs  925 ,  935  originates has certain values. As yet another example, the scaling of data from input  930  may reflect the training of neural network  110 . For example, neural network  110  may include a collection of variable amplifiers such as variable amplifier  955  that each scales different inputs over different links. A directed training process can be used to tailor the scaling. 
     As yet another example, the scaling of data from input  930  may reflect external, context-dependent factors rather than internal, state-dependent factors. For example, the scaling may change based on, e.g., the time of day or other factor that is not otherwise reflected in the activity of neural network  110 . 
       FIG. 10  is a schematic representation of another approach for inputting data  50  that originates from different sensors into a neural network  110 . The illustrated portion of neural network  110  includes a node  1005  and links  1010 ,  1015 ,  1020 . In the illustrated implementation, links  1010 ,  1015  are unidirectional links directed toward node  1005  and link  1020  is a unidirectional link directed away from node  1005 . 
     Inputs  105  include a first input  1025 , a second input  1030 , and a third input  1035 . In some implementations, first input  1025  is dedicated to inputting data that originates from a first sensor, second input  1030  is dedicated to inputting data that originates from a second sensor, and third input  1035  is dedicated to inputting data that originates from a third sensor. 
     Inputs  1025 ,  1030 ,  1035  can be nodes of an input layer. In some implementations, inputs  1025 ,  1030 ,  1035  can, e.g., scale some portion or all of the input data or perform other preliminary processing before data is conveyed to neural network  110 . 
     In the illustrated implementation, input  1030  injects data into a first position along link  1010  over a link  1040 . Inputs  1035 ,  1040  inject data elsewhere into neural network  110  over respective of link  1045 ,  1050 . Further, the data conveyed over inputs  1035 ,  1040  can be used to change one or more parameters of link  1010 . In the illustrated implementation, link  1010  includes a parameter setting element  1055  that is operable to vary one or more parameters of link  1010  based on the inputs over links  1045 ,  1050 . 
     Examples of parameters that may be varied by parameter setting element  1055  include, e.g., the magnitude of a signal conveyed along link  1010 , the timing of a signal conveyed along link  1010 , the weight of a signal conveyed along link  1010 , and even whether link  1010  can covey input  1030  to node  1005  at all. In implementations where neural network  110  is modelled on a biological system, one or more of morphological, chemical, and other characteristics of the biological system can be varied by parameter setting element  1055 . Examples would include thresholds for activation of link  1010 , the nature, number, and/or distribution of ion channels or receptors along link  1010 , or any of a host of other parameters. 
     By varying one or more parameters of link  1010  based on the inputs over links  1045 ,  1050 , parameter setting element  1055  can help weigh the relative import of data that originates from one sensor based on data that originates from one or more other sensor(s). As before, the scaling may reflect the training of neural network  110  and/or external, context-dependent factors. 
       FIG. 11  is a schematic representation of another approach for outputting data from neural network  110  into which data that originates from different sensors is input. The schematically represented neural network  110  includes a node  1105  that is read over a link  1110 . Inputs  105  of neural network  110  also include a first input  1115 , a second input  1120 , and a third input  1125 . In some implementations, first input  1115  is dedicated to inputting data that originates from a first sensor, second input  1120  is dedicated to inputting data that originates from a second sensor, and third input  1125  is dedicated to inputting data that originates from a third sensor. 
     In the illustrated implementation, inputs  1115 ,  1120 ,  1125  also inject data into neural network  110  over a respective link  1130 ,  1135 ,  1140 . Further, the data conveyed over inputs  1120 ,  1125  can be used to change one or more parameters of the link  1110  over which node  1105  is read. In the illustrated implementation, link  1110  includes a parameter setting element  1145  that is operable to vary one or more parameters of link  1110  based on the inputs over links  1135 ,  1140 . 
     Examples of parameters that may be varied by parameter setting element  1145  include, e.g., the value of a signal conveyed along link  1110  and the timing of a signal conveyed along link  1110 . In implementations where neural network  110  is modelled on a biological system, one or more of morphological, chemical, and other characteristics of the biological system can be varied by parameter setting element  1145 . Examples would include thresholds for activation of link  1110 , the nature, number, and/or distribution of ion channels or receptors along link  1110 , or any of a host of other parameters. 
     By varying one or more parameters of link  1110  based on the inputs over links  1135 ,  1140 , parameter setting element  1145  can perform a variety of different actions. For example, parameter setting element  1145  may hold a value for a relatively longer time or until reset. A collection of such “holding” parameter setting elements could thus indicate whether certain patterns of activity arose during a relatively long time period. The time period can be, e.g., the duration of an input that changes over time, such as an audio or a video stream. 
     As another example, in implementations where the digits in collection  120  are multi-valued, parameter setting element  1145  can scale a value. The scaling may reflect, e.g., the training of neural network  110  and/or external, context-dependent factors. As yet another example, parameter setting element  1145  may act to prevent conveyance of a value to outputs  115 , in effect, deleting the value. Once again, the deletion may reflect, e.g., the training of neural network  110  and/or external, context-dependent factors. 
       FIG. 12  is a schematic illustration of a use of a binary digit collection  120  that represents the occurrence of topological structures in the activity in a neural network. In particular, neural network  110  can be part of system  1200  that includes one or more processors  1205 ,  1210  that are dedicated to performing certain operations on binary digit collection  120 . 
     In the illustrated implementation, processor  1205  is a linear processor that performs operations based on a linear combination of the characteristics of the binary digits in collection  120 . For example, linear processor  1205  can be a device that classifies an object—namely, representations of the patterns of activity in a neural network. In other words, the representations of the patterns of activity is a feature vector that represents the characteristics of the input into neural network  1110  that are used by linear processor  1205  to classify data  50 . 
     Linear processor  1205  includes an input  1215  and an output  1220 . Input  1215  is coupled to receive representations of the patterns of activity in a neural network, namely, binary digit collection  120 . Linear processor  1205  can receive binary digit collection  120  in a variety of ways. For example, the digits in collection  120  can be received as collection of discrete events or as a continuous stream over a real time or non-real time communication channel. 
     Output  1220  is coupled to output the processing result from linear processor  1205 . In the illustrated implementation, output  1220  is schematically illustrated as a parallel port with multiple channels. This is not necessarily the case. For example, output  1220  can output the result over a serial port or a port with combined parallel and serial capabilities. 
     In some implementations, linear processor  1205  can be implemented on one or more computing devices with relatively limited computational performance. For example, linear processor  1205  can be implemented on a personal computer or a mobile computing device such as a smart phone or tablet. 
     In the illustrated implementation, processor  1210  is a neural network processor  1210 . Neural network processor  1210  is a neural network device that performs operations on data—namely, representations of the patterns of activity in a neural network—based on a non-linear combination of the data&#39;s characteristics. For example, neural network processor  1210  can be a device that classifies an object—namely, representations of the patterns of activity in a neural network. In other words, the representations of the patterns of activity is a feature vector that represents the characteristics of the input into neural network  1110  that are used by neural network processor  1210  to classify data  50 . 
     In the illustrated implementation, neural network processor  1210  is a feedforward network that includes an input layer  1230  and an output layer  1235 . As with linear classifier  1205 , neural network classifier  1210  can receive the binary digit collection  120  in a variety of ways. For example, the digits in collection  120  can be received as discrete events or as a continuous stream over a real time or non-real time communication channel. 
     In some implementations, neural network processor  1210  can perform inferences on one or more computing devices with relatively limited computational performance. For example, neural network processor  1210  can be implemented on a personal computer or a mobile computing device such as a smart phone or tablet, e.g., in a Neural Processing Unit of such a device. 
     In some implementations, neural network processor  1210  can be, e.g., a deep neural network such as a convolutional neural network that includes convolutional layers, pooling layers, and fully-connected layers. Convolutional layers can generate feature maps, e.g., using linear convolutional filters and/or nonlinear activation functions. Pooling layers reduce the number of parameters and control overfitting. The computations performed by the different layers in neural network processor  1210  can be defined in different ways in different implementations of neural network processor  1210 . 
       FIG. 13  is a schematic illustration of a hierarchical system  1300  of recurrent neural networks  110  that abstracts data  50  that originates from multiple, different sensors. Hierarchical system  1300  includes a first level  1305  of neural networks  110  and a second level  1310 . Each neural network  110  in first level  1305  abstracts data  50  from a respective sensor and outputs a digit collection  120  that represents the occurrence of topological structures in the responsive activity. The neural network  110  in second level  1310  abstracts the digit collections  120  that are output from the neural networks  110  in first level  1305  and outputs a digit collection  120  that represents the occurrence of topological structures in the responsive activity. The digit collection  120  that is output from neural network  110  in second level  1310  can thus include correlates that fuse diverse input data  50  into a more complete “whole.” 
     In the illustrated implementation, system  1300  is illustrated as having two levels. This is not necessarily the case. For example, a hierarchical system can include more levels. A hierarchical system of neural networks can be balanced or unbalanced, and ragged hierarchies are also possible. 
     Further, in the illustrated implementation, data  50  that is input into the neural network  110  in first level  1305  includes sound data  60 , still image data  62 , and video data  64 . This is not necessarily the case and fewer and/or other types of input data are possible. 
     As an aside, each of the above-described processing actions (e.g.,  FIG. 5 ), approaches for inputting data (e.g.,  FIGS. 6-10 ), and approaches for outputting data (e.g.,  FIG. 11  can be used in conjunction with any of the neural networks at any hierarchical level. 
     Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). 
     The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources. 
     The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures. 
     A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network. 
     The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). 
     Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. 
     To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user&#39;s client device in response to requests received from the web browser. 
     While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. 
     Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. 
     Thus, particular implementations of the subject matter have been described. Other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous. 
     A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made. Accordingly, other implementations are within the scope of the following claims.