Patent Publication Number: US-2003223619-A1

Title: Method and apparatus for visual motion recognition

Description:
CROSS REFERENCE TO RELATED APPLICATIONS  
       [0001] This application claims the priority of the European application 02 005 282.5, filed Mar. 12, 2002, the disclosure of which is incorporated herein by reference in its entirety.  
       BACKGROUND OF THE INVENTION  
       [0002] The invention relates to a method and apparatus for visual motion recognition.  
       [0003] Systems for visual motion recognition are described in U.S. Pat. No. 5,511,133 and in Stocker, A. and R. J. Douglas (1999), computation of smooth optical flow in a feedback connected analog network, in M. S. Kearns, S. A. Solla, and D. A. Cohn (Eds.), Advances in Neural Information Processing Systems 11, Cambridge, Mass., pp. 706-712, MIT Press.  
       [0004] Systems for visual motion recognition are used for obtaining information about moving patterns, such as velocity, position and configuration. The use of neural networks in such systems allows fast and in particular parallel processing of image data. Usually each neuron or unit of a neural network is assigned to a particular sampling position or pixel of the image. For motion detection generally two neural networks are employed, one network for x-directional motion and one network for y-directional motion. Both networks interact. The networks are designed such that the state of the neurons converge to an optical flow (u,v), which is indicative of the velocity vector at a respective point of a moving pattern. The term ‘indicative’ means that the state of a particular neuron is an estimate of the actual velocity component.  
       [0005] The optical flow {right arrow over (v)}=(u,v) can e.g. be obtained by minimizing the cost function  
         H ({right arrow over ( v )}( x,y,t );ρ,σ)=∫∫( F   2   +ρS+σB ) dxdy   (1)  
       [0006] where F, S and B are functions which are defined below. The weight of F, S and B is defined by the parameters ρ and σ. The boundary condition of such networks is usually defined by  
         {right arrow over (v)} ′( x,y,t )=0 (along the boundary)  (2)  
       [0007] The function F is defined by:  
         F ( {right arrow over (v)} ( x,y,t )) 2 =( E   x   ·u+E   y   ·v+E   t ) 2   (3)  
       [0008] where E x , E y , E t  are the partial derivatives of the brightness E at the respective points of the input image data with respect to the x- and y-components and the time, and u,v are the x- and y-components of the optical flow (u,v). The function S is defined by:  
         S ({right arrow over ( v )}′( x,y,t ))=( u   x   2   +u   y   2   +v   x   2   +v   y   2 )  (4)  
       [0009] where u x ,u y ,v x ,v y  are the spatial gradients of the optical flow vector field. The function B is defined by:  
         B ( {right arrow over (v)} ( x,y,t ))=( u−u   0 ) 2 +( v−v   0 ) 2   (5)  
       [0010] where (u 0 ,v 0 ) is a bias potential or reference motion.  
       [0011] The use of the term “input image data” in this document does not imply that the image information is supplied digitally. The information is preferably provided as an analog signal, for example as a voltage signal.  
       [0012] Existing systems are responsive to noise. Noise is erroneously interpreted as small moving objects. Existing systems often don&#39;t work reliably if there is more than one moving object. The velocity vector estimations are poor, particularly at the edges of moving patterns.  
       BRIEF SUMMARY OF THE INVENTION  
       [0013] Hence, it is a general object of the invention to provide a system of the kind described above which avoids the disadvantages described above at least partially.  
       [0014] Now, in order to implement these and still further objects of the invention, which will become more readily apparent as the description proceeds, the apparatus for visual motion recognition according to the invention is manifested by the features that it comprises an optical flow network to which input image data is applied and which generates an optical flow vector field, wherein said optical flow network comprises optical flow neurons which are connected to a reference motion potential over local bias conductances, and further comprises a motion selective network, which generates an attentional bias, wherein said attentional bias controls said local bias conductances of said optical flow network.  
       [0015] In yet another aspect, the invention relates to a method for determining an optical flow vector field based on input image data, characterized by the steps of:  
       [0016] Applying the input image data to an optical flow network, particularly a bi-directional neural network comprising pairs of optical flow neurons, where each of said optical flow neurons is connected to a reference motion potential over a local bias conductance, generating said optical flow vector field,  
       [0017] Applying said optical flow vector field to a motion selective network, which generates an attentional bias,  
       [0018] Using said attentional bias to control said local bias conductances of said optical flow network.  
       [0019] By employing said motion selective network which controls the local bias conductances of said optical flow network the system can be set up to put its attention on a certain expected motion. For example noise, which is usually not within the range of the expectation, is thereby eliminated. 
     
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
     [0020] The invention will be better understood and objects other than those set forth above will become apparent when consideration is given to the following detailed description thereof. Such description makes reference to the annexed drawings, wherein:  
     [0021]FIG. 1 is a schematic view of the networks and their interaction, input and output,  
     [0022]FIG. 2 shows a single unit of the bi-directional optical flow network,  
     [0023]FIG. 3 shows the interaction between the motion selective network and the x-directional optical flow network,  
     [0024]FIG. 4 shows an idealized optical flow field as generated by conventional optical flow networks,  
     [0025]FIG. 5 shows the idealized output of a motion selective network for an expected motion to the left,  
     [0026]FIG. 6 shows the idealized output of the system according to the invention for an expected motion to the left,  
     [0027]FIG. 7 shows the idealized output of a motion selective network for an expected motion to the right,  
     [0028]FIG. 8 shows the idealized output of the system according to the invention for an expected motion to the right, and  
     [0029]FIG. 9 shows a schematic view of the preferred embodiment of the invention, including a camera pointed towards moving sample objects. 
    
    
     DETAILED DESCRIPTION OF THE INVENTION  
     [0030]FIG. 1 is a schematic view of an embodiment of the invention. Continuous input image data  1  are sampled at discrete positions (i,j) to generate brightness or intensity values E ij (t). The spatial and temporal derivatives E x , E y , E t  of the brightness E are fed into an optical flow network  27 . Generally optical flow networks are networks that have image information as an input and optical flow or velocity information as an output. In the preferred embodiment of the invention the optical flow network  27  is bi-directional and consists of two neural networks: An x-directional optical flow network  6  and a y-directional optical flow network  7 . The neural networks are designed such that the states of the optical flow neurons  3 ,  4  converge to a value indicative of the components of a velocity vector of a moving pattern  30 . There is a pair of optical flow neurons  3 ,  4  for each image pixel or discrete sampling position. The arrangement of the neurons is retinotopical. The output of the optical flow network  27  is an optical flow vector field  28  with one optical flow vector  13  for each pair of optical flow neurons  3 ,  4 .  
     [0031] The bi-directional optical flow network  27  interacts with a motion selective network  5 .  
     [0032] Generally motion selective networks have an optical flow vector field or velocity vector field as an input and a field of values indicating to which extent a selection criterion was fulfilled as an output. In the preferred embodiment of the invention the selection criterion is the similarity of the optical flow input with an expected motion  19 ,  20 . The selection criterion can be changed dynamically if the motion selective network has an additional input  15  for parameters characterizing the selection criterion. For example if the selection criterion is the similarity with an expected motion  19 ,  20 , these parameters comprise the components of the velocity vector {right arrow over (v)} Model  of that expected motion  19 ,  20 . The expected motion can also be characterized by a field of velocity vectors. An other example for such a parameter is the expected size N max  of moving patterns as described below.  
     [0033] The motion selective network  5  is used to determine whether a certain moving pattern is of interest. The output of the motion selective network  5  provides a measure of the attention to be put on discrete positions of the image. In this manner the output provides an attentional bias  12 , which is used to control the optical flow network  27 , particularly to inhibit optical flow neurons  3 ,  4  at positions which are not of interest. Extending the conventional optical flow network  27  with a motion selective network  5  has the advantage that noise is not as easily misinterpreted as a moving object. Objects which are not of interest and are e.g. moving in a different direction than the objects of interest can be ignored by choosing the expected motion  19 ,  20  accordingly as described in the following on the basis of FIGS.  4  to  8 . The motion selective network  5  has the optical flow u, v generated by the optical flow network  27  as an input. The attentional bias  12  is therefor with respect to the optical flow network  27  a feedback signal.  
     [0034] On top of the motion selective network  5 , there can be one or more additional networks  37 , which are not shown in FIG. 1. These additional networks  37  influence the motion selective network  5  by the top-down input  35 . The additional networks  37  receive a bottom-up input  36 , for example the attentional bias A ij , from the motion selective network  5 . These additional networks  37  are higher level networks performing recognition and/or control tasks at a higher level of abstraction.  
     [0035]FIG. 2 shows one unit  29  of a bi-directional optical flow network  27 . There is a total of m×n such units  29  in the network  27 . A unit can be identified by its position (i,j). Each unit corresponds to a particular pixel of the input image data  1 . There are m different values for i and n different values for j. Each unit  29  comprises an x-directional optical flow neuron  3  and an y-directional optical flow neuron  4 . Each optical flow neuron  3 ,  4  is connected to four neighboring optical flow neurons  3 ,  4  with horizontal conductances  8 . Each optical flow neuron  3 ,  4  is connected to a reference motion potential  10 ,  11  with a local bias conductance  9 . The state u ij  or v ij  of each optical flow neuron  3 ,  4  converges to the reference motion potential  10 ,  11  with a rate depending among other factors on the local bias conductance  9 . Each optical flow neuron  3 ,  4  receives an input current from a controllable current source  32 ,  33 . The current of current sources  32 ,  33  depends on the spatial and temporal derivatives E x ,E y ,E t  of the brightness E of the image. It also depends on the state of both optical flow neurons  3 ,  4  of the unit  29 . Depending on the sign of the current the controllable current source  32 ,  33  can also act as a current sink.  
     [0036] In the preferred embodiment of the invention the current sources are controlled according to the following formulas:  
       l   Aij   ∝−E   x     ij   ( E   x     ij     u   ij   +E   y     ij     v   ij   +E   t     ij   )  (6)  
       l   Bij   ∝−E   y     ij   ( E   x     ij     u   ij   +E   y     ij     v   ij   +E   t     ij   )  (7)  
     [0037] The dynamics of an optical flow network  27  consisting of such units  29  can be described by the following equations:  
                       u   .       i                 j       =     -       1   C     [         E     x     i                 j              (         E     x     i                 j              u     i                 j         +       E     y     i                 j              v     i                 j         +     E     t   ij         )       -                                                 ρ        (       u       i   +   1     ,   j       +     u       i   -   1     ,   j       +     u     i   ,     j   +   1         +     u     i   ,     j   -   1         -     4        u   ij         )       +     σ        (       u     i                 j       -     u   0       )         ]                              (   8   )                         v   .       i                 j       =     -       1   C     [         E     y     i                 j              (         E     x     i                 j              u     i                 j         +       E     y     i                 j              v     i                 j         +     E     t   ij         )       -                                                 ρ        (       v       i   +   1     ,   j       +     v       i   -   1     ,   j       +     v     i   ,     j   +   1         +     v     i   ,     j   -   1         -     4        v   ij         )       +     σ        (       v     i                 j       -     v   0       )         ]                              (   9   )                       
 
     [0038] More general the equations can be written as  
       {dot over (u)}   ij   =F   U ( u   ij   ,v   ij   ,u   neighbors   ,E   x     ij     ,E   y     ij     ,E   t     ij   )  (10)  
       {dot over (v)}   ij   =F   V ( u   ij   ,v   ij   ,v   neighbors   ,E   x     ij     ,E   y     ij     ,E   t     ij   )  (11)  
     [0039] wherein i is the state of the x-directional optical flow neuron  3  at position (i,j). v ij  is the state of the y-directional optical flow neuron  3  at position (i,j). {dot over (u)} ij  is the temporal derivative of the state u ij . {dot over (v)} ij  is the temporal derivative of the state v ij . E x     ij   ,E y     ij   ,E t     ij    are the spatial and temporal derivatives of the brightness E at position (i,j). σ is the bias conductance  9  which connects the x-directional optical flow neuron  3  with a bias voltage u 0  and which connects the y-directional optical flow neuron  4  with a bias voltage v 0 . u 0  and v 0  are reference motion potentials. The reference motion is represented by a reference motion potential vector (u 0 ,v 0 ) and can be changed to adapt the system to a certain constant shift of the total image, for example if the camera recording E ij  is moved. Since u 0  and v 0  do not depend on the position (i,j) the reference motion potential vector (u 0 ,v 0 ) is a common reference motion potential vector, in particular common for all units of the optical flow network  27 . ρ is the horizontal conductance which connects the optical flow neuron  4  at position (i,j) with its four nearest neighbors at positions (i+1,j), (i−1,j), (i,j+1) and (i,j−1). C is the capacity by which each neuron is connected to a constant potential, for example u 0  or v 0 . Neurons at the boundaries of the m×n network have only three or two such nearest neighbors. F U  and F V  are functions. The conductances ρ and σ can be implemented using active or passive circuitry.  
     [0040]FIG. 3 shows the motion selective network and its interaction with the x-directional optical flow network  6 . The interaction with the v-directional optical flow network  7  is of the same nature and is not shown. The motion selective network  5  comprises a motion selective neuron  2  with state a ij  at each discrete position (i,j). From each motion selective neuron  2  there originates an activation  14 . The activation is directed to other motion selective neurons  2 , to a global inhibitory unit  22 , to control the bias conductance  9  of the corresponding optical flow neurons  3 ,  4 , or to influence one or more additional neural networks  37  by their bottom-up input  36 .  
     [0041] The activation  14  is generated as defined by a nonlinear activation function g:  
       A   ij   =g ( a   ij )  (12)  
     [0042] Wherein A ij  is the activation originating from the neuron at position (i,j) with state a ij .  
     [0043] In the preferred embodiment of the invention the nonlinear activation function is sigmoid, particularly as follows:  
               A     i                 j       =       g        (     a     i                 j       )       =       1   2          (       tanh        (       a     i                 j         a   0       )       +   1     )                 (   13   )                       
 
     [0044] In the preferred embodiment of the invention all motion selective neurons  2  take the same effect on the global inhibitory unit  22 . In the preferred embodiment of the invention the dynamics of the motion selective network  5  are defined as follows:  
                 a   .       i                 j       =     -       1   C          [         a     i                 j       R     +       (     α   +   β     )            ∑   kl          A     k                 l           -     α                   A     i                 j         -     β                   N   max       -     γ        
                       ∑   neighbors          A   neighbors         -     δ        (         v   _       i                 j       ·       v   _     Model       )         ]                 (   14   )                       
 
     [0045] wherein  
                 v   _       i                 j       =     (           u     i                 j                 v     i                 j             )             (   15   )                       
 
     [0046] and C is a constant, R is the input resistance and α, β, γ and δ are constants (α+β) is a measure of the weight of the global inhibitory unit  22 . α is a measure of the self-excitation of the motion selective neuron  2 . {right arrow over (v)} Model  is the expected motion  15 ,  20 .  
     [0047] The scalar product −δ({right arrow over (v)} ij ·{right arrow over (v)} Model ) in formula (14) causes the motion selective network  5  to select optical flow which is similar to the expected motion {right arrow over (v)} Model . The scalar product can be replaced by other terms, in particular a norm −δ∥{right arrow over (v)} ij ∥. In this case the motion selective network  5  selects fast patterns, independent of their direction.  
     [0048] The motion selective network  5  is a multiple-winner-take-all network (mWTA-network). Such networks are similar to conventional winner-take-all networks (WTA-networks), with the exception that the total activity is forced to N max , the number of winners. In this application the total activity N max  corresponds to the expected size of the moving pattern or to the size of attention. While {right arrow over (v)} Model  allows to direct the attention of the system to a certain direction and velocity of motion, N max  allows to adjust the system to focus on moving objects  31  of a certain size.  
     [0049] More generally the dynamics of the motion selective network  5  can be described by the formula:  
       a   ij   =Z ( g   1 ( a   ij )+ g   2 ( a   neighbors1 )+ g   3 ( a   neighbors2 ), u   ij   ,v   ij   ,l   ij )  (16)  
     [0050] Wherein g 1 ,g 2 ,g 3  and Z are different, non-constant functions. Z is such that the state of network converges for constant u and v. a neighbors1  is a set of the states a kl  of motion selective neurons  2  in a first defined neighborhood  21  around position (i,j). In a preferred embodiment of the invention the set comprises only the four nearest neighbors at positions (i+1,j), (i−1,j) (i,j+1) and (i,j−1). a neighbors2  is a set of the states a kl  of motion selective neurons  2  in a second defined neighborhood. The second defined neighborhood usually comprises all neurons of the first defined neighborhood and is therefore bigger. In a preferred embodiment of the invention the second neighborhood comprises all motion selective neurons  2  in the network  5 . u ij ,v ij  is with respect to the motion selective network a bottom-up input  34 , while l ij  is a top-down input  35 . In the preferred embodiment of the invention l ij  is dynamic and is a function of values provided by one or more additional networks  37 .  
     [0051] The bias conductances  9  at each position (i,j) of the optical flow network  6  and  7  are controlled by the activation  14  of the corresponding motion selective neuron  2  as defined by the formula:  
     σ ij =σ 1 +(1 −A   ij )σ 2   (17)  
     [0052] wherein σ ij  is the bias conductance  9  of a optical flow neuron  3 ,  4  at position (i,j), σ 1  is a minimal bias conductance and σ 1 +σ 2  is a maximal bias conductance. The relationship can generally be defined by any formula:  
     σ ij   =f ( A   ij )  (18)  
     [0053] In an implementation or circuit of the neural network  6 ,  7  the controllable bias conductance  9  is usually not a resistor. Instead it is an active element, controllable by a voltage and implemented as an arrangement of transistors, for example FETs, as well known to persons skilled in the art.  
     [0054]FIG. 4 shows an idealized optical flow vector field  28  as generated by conventional ideal optical flow networks without a particular focus of attention. A first pattern  17  is moving to the left, a second pattern  18  is moving to the right.  
     [0055]FIG. 5 shows the idealized output or activation of a motion selective network  5  for an expected motion  19  to the left. Black indicates that the optical flow in this region is to be suppressed. White indicates the detection of a moving pattern with the characteristics the system is tuned for. The output shown in the diagram is simplified in that it takes only two discrete values, illustrated in black and white. The output of the actual system takes continues values.  
     [0056]FIG. 6 shows the idealized output of the system according to the invention for an expected motion  19  to the left. The optical flow vector field  28  shows basically only the pattern  17  moving similar to the expected motion  19 . The pattern  18  which is moving in the opposite direction is ignored and without effect on the output optical flow vector field  28 .  
     [0057]FIG. 7 shows the idealized output of a motion selective network  5  for an expected motion  20  to the right. Black indicates that the optical flow in this region is to be ignored.  
     [0058]FIG. 8 shows the output of the system according to the invention for an expected motion  20  to the right. The optical flow vector field  28  shows only the pattern  18  moving similar to the expected motion  19 , while the pattern  17  is ignored.  
     [0059]FIG. 9 shows a schematic view of the preferred embodiment of the invention, including a camera  23  pointed towards moving sample objects  31 . The camera  23 , particularly a CCD-camera, produces a two-dimensional brightness signal  24  input stage  25  calculates spatial and temporal derivatives  26  of the brightness. The two dimensional optical flow network  27  generates an optical flow vector field  28  based on these derivatives  26 . The optical flow network  27  is biased by the selective network  5 , which generates an attentional bias  12  based on a dynamic bottom up input  34 , which is in particular an optical flow vector field, and an input  15 , which is a selection criterion. The selection criterion is for example provided by a vector {right arrow over (v)} Model  which characterizes an expected motion  19 ,  20  and a number N max  relating to the expected size of moving objects. The motion selective network can be enhanced by one or more additional networks  37  which influence the motion selective network  5  by a top-down input  35 , in particular an input l ij (t) for each motion selective neuron  2 . The additional neural networks  37  may receive a bottom-up input  36  from the motion selective network  5 , in particular an attentional bias. The additional neural networks  37  do not necessarily have the same number, density and topography of neurons as the motion selective network  5 . The input or activation l ij (t) of a motion selective neuron  2  at position (i,j) is in particular a function of values provided by one or more of the additional networks  37 .  
     [0060] In systems for visual motion recognition with optical neural networks the term “bottom-up” is generally used for “towards a higher level of abstraction. “top-down” is used for “towards a lower level of abstraction”. The lowest level of abstraction within the system is the input image data provided by the camera.  
     [0061] The neural networks  5 ,  27 ,  37  can be implemented as analog circuits. An example of such circuits is described in the publication Stocker, A. and R. J. Douglas (1999), computation of smooth optical flow in a feedback connected analog network, in M. S. Kearns, S. A. Solla, and D. A. Cohn (Eds.), Advances in Neural Information Processing Systems 11, Cambridge, Mass., pp 706-712, MIT Press.  
     [0062] The neural networks  5 ,  27  can also be implemented using a computer system. The behavior of the neural networks is simulated with a special software. Better performance in the simulation of neural networks can be achieved by using parallel computer systems.  
     [0063] The output of the system is the optical flow vector field  28  and attentional bias matrix  16 . This is an advantage over conventional systems which have usually only the optical vector field output. The outputs are used for example for recording, visualization or further processing by other components or systems such as controllers. Depending on the application only one of the two outputs  28 ,  16  may actually be used.  
     [0064] Even though the system is designed mainly for a certain number of m×n image pixels or units the number of units may be changed at certain stages of the system, in particular by resampling. Especially the input stage  25  will produce better values for the derivatives E x , E y  and E t  if the input provided by the camera  23  has a higher resolution than the output required for the optical flow network  27 .  
     [0065] Even though, in the following, the invention is primarily claimed in terms of an apparatus, it must be noted that the applicant reserves the right to claim the invention with a corresponding set of method claims.  
     [0066] While there are shown and described presently preferred embodiments of the invention, it is to be distinctly understood that the invention is not limited thereto but may be otherwise variously embodied and practised within the scope of the following claims.