Abstract:
Methods and systems for data segmentation include determining initial segmentation scores for each unit of an input data set using a neural network, with each unit being assigned an initial segmentation score for each of multiple segmentation classes. Final segmentation scores are determined for each unit of the input data set by enforcing a smoothness criterion. The input data set is segmented in accordance with the final segmentation scores.

Description:
RELATED APPLICATION INFORMATION 
       [0001]    This application claims priority to U.S. Application Ser. No. 62/291,076 filed on Feb. 4, 2016, incorporated herein by reference in its entirety. 
     
    
     BACKGROUND 
       [0002]    Technical Field 
         [0003]    The present invention relates to image segmentation and, more particularly, to modified neural network segmentation that uses a reaction-diffusion process obeying a variational principle (referred to as “variational reaction diffusion”). 
         [0004]    Description of the Related Art 
         [0005]    Semantic segmentation seeks to take incoming data, for example in the form of a graphical image, and divide the data into logical segments. In the case of processing an image, the segmented output may group together pixels that represent, e.g., people, roads, trees, or other distinctive image features. 
         [0006]    While different segmentation techniques have been used, existing semantic segmentation is either computationally myopic (e.g., taking into account only a small portion of the image at a time) or are computationally inefficient. 
       SUMMARY 
       [0007]    A method for data segmentation include determining initial segmentation scores for each unit of an input data set using a neural network, with each unit being assigned an initial segmentation score for each of multiple segmentation classes. Final segmentation scores are determined for each unit of the input data set by enforcing a smoothness criterion. The input data set is segmented in accordance with the final segmentation scores. 
         [0008]    A system for data segmentation includes a neural network configured to determine initial segmentation scores for each unit of an input data set, with each unit being assigned an initial segmentation score for each of multiple segmentation classes. A segmentation module is configured to determine final segmentation scores for each unit of the input data set by enforcing a smoothness criterion and to segment the input data set in accordance with the final segmentation scores. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0009]    The disclosure will provide details in the following description of preferred embodiments with reference to the following figures wherein: 
           [0010]      FIG. 1  is a block/flow diagram of a method/system for data segmentation in accordance with the present principles; 
           [0011]      FIG. 2  is a block/flow diagram of a method for data segmentation in accordance with the present principles; 
           [0012]      FIG. 3  is a block/flow diagram of a method for model and neural network training in accordance with the present principles; 
           [0013]      FIG. 4  is a block diagram of a monitoring system in accordance with the present principles; and 
           [0014]      FIG. 5  is a block diagram of a processing system in accordance with the present principles. 
       
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
       [0015]    Embodiments of the present invention provide image segmentation that makes use of a convolutional neural network (CNN) to identify image features and further uses variational reaction diffusion (VRD) to ensure smoothness in the segmentation output. VRD is used during CNN training as well to produce backpropagation derivatives that may be used to adjust weights in the CNN. The VRD processes described herein provide exact inference and loss derivatives in N log N time in the number of pixels. 
         [0016]    Inference in VRD may be interpreted as evolving evidence (or class scores) under the dynamics of a reaction-diffusion process. Evidence for one semantic class may be modeled as a unary potential that propagates across the image via diffusion and reacts with evidence of other semantic classes. Each of these processes may locally create or suppress evidence for each class. By restricting the model to the class of processes that generates solutions to convex, variational problems, a stable equilibrium can be ensured. 
         [0017]    Referring now in detail to the figures in which like numerals represent the same or similar elements and initially to  FIG. 1 , a semantic segmentation procedure is illustratively depicted in accordance with one embodiment of the present principles. An input image  102  is provided, being formed from a two-dimensional array of pixels. It should be understood that the present principles are not limited to graphical information, but may instead be applied to any type of information that has segmentable features. The input image  102  is used by CNN  104  to generate a set of feature outputs. Each pixel in the input image  102  may correspond to a single input of the CNN, with the number of per-pixel outputs of the CNN  104  corresponding to a number of different segmentation fields selected for the image input  102 . 
         [0018]    For each pixel of the input image  102 , the CNN  104  generates an output vector that includes numerical values for the pixel along each of a set of different segmentation fields. It should be understood that the segmentation fields do not necessarily correspond to intuitive features of the image input  102 , but are instead the result of the CNN&#39;s training process, representing automatically generated features that provide meaningful segmentation of the image. 
         [0019]    The output vectors are used by a VRD module  106  to guarantee a smoothness criterion in the image. In one example, smoothness may refer to a tendency for pixels in a particular segment of an image input  102  to be located adjacent to, or at least near to, one another. During training, the VRD module  106  also provides error information back to the CNN  104  based on an error signal. Both the VRD module  106  and the CNN  104  adjust internal parameters in response to the error signal to improve future operation. The VRD  106  outputs a set of vectors for each pixel that represent segmentation scores, with a best score for each pixel representing the segment associated with that pixel. 
         [0020]    As noted above, the image input may be represented as I ∈           2 , a rectangular subset of            2  representing the domain of the image. VRD may be defined as a function that maps a spatially varying set of N i  input features, represented as a function s i : I→           N     i   , to a set of N o  output scores s o : I→           N     o   . N o  is interpreted as the number of semantic classes, with s k   o (x) being a score associated with the k th  class at a pixel x ∈ I. A prediction is generated via 
         [0000]    
       
         
           
             arg 
              
             
                 
             
              
             
               
                 max 
                 k 
               
                
               
                 
                   
                     s 
                     k 
                     o 
                   
                    
                   
                     ( 
                     x 
                     ) 
                   
                 
                 . 
               
             
           
         
       
     
         [0021]    A set of output vectors from the CNN  104  is denoted herein as s i  (forming the input to the VRD  106 ) and a set of output vectors from the VRD  106  is denoted herein as s o . The two vectors can be combined to a vector s=(s oT  s iT ) T , denoting the concatenation of s i  and s o  into a single function I→           N     i     +N     o   . The VRD module  106  generates s o  by solving an optimization problem, using s i  as an input. For notational simplicity, the dependence of s on the specific pixel x in the image I is omitted. The optimization problem can be expressed as: 
         [0000]    
       
         
           
             
               arg 
                
               
                   
               
                
               
                 
                   min 
                   
                     s 
                     o 
                   
                 
                  
                 
                   
                     ∫ 
                     I 
                   
                    
                   
                     
                       S 
                       T 
                     
                      
                     Qs 
                   
                 
               
             
             + 
             
               
                 ∑ 
                 
                   d 
                   = 
                   1 
                 
                 2 
               
                
               
                 
                   
                     ∂ 
                     
                       s 
                       T 
                     
                   
                   
                     ∂ 
                     
                       x 
                       d 
                     
                   
                 
                  
                 B 
                  
                 
                   
                     ∂ 
                     s 
                   
                   
                     ∂ 
                     
                       x 
                       d 
                     
                   
                 
                  
                 dx 
               
             
           
         
       
     
         [0000]    where d represents the dimensions of the image. The parameters B and Q represent positive-definite parameter matrices that are independent of x. The result is an infinite-dimensional, convex, quadratic optimization problem in s o . The optimization problem can be discretized, with the derivatives being replaced with a finite difference approximation: 
         [0000]    
       
         
           
             
               
                 
                   ∂ 
                   s 
                 
                 
                   ∂ 
                   
                     x 
                     k 
                   
                 
               
               ≈ 
               
                 
                   ε 
                   
                     - 
                     1 
                   
                 
                  
                 
                   ( 
                   
                     
                       s 
                        
                       
                         ( 
                         
                           x 
                           ′ 
                         
                         ) 
                       
                     
                     - 
                     
                       s 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                   
                   ) 
                 
               
             
             , 
           
         
       
     
         [0000]    for some x′ a small distance ∈ away from x along the x k  axis. Intuitively, the term s T Qs can be interpreted as a unary potential relating s i  and s o  at each point, while 
         [0000]    
       
         
           
             
               
                 ∂ 
                 
                   s 
                   T 
                 
               
               
                 ∂ 
                 
                   x 
                   k 
                 
               
             
              
             B 
              
             
               
                 ∂ 
                 s 
               
               
                 ∂ 
                 
                   x 
                   k 
                 
               
             
           
         
       
     
         [0000]    represents a binary potential discouraging spatial changes in the score vector. 
         [0022]    The calculus of variations may be used to express the solution to the above optimization problem to express its solution as that of the following linear system of partial differential equations: 
         [0000]    
       
      
       B 
       o 
       Δs 
       o 
       −Q 
       o 
       s 
       o 
       =Q 
       i 
       s 
       i 
       −B 
       i 
       Δs 
       i  
      
     
         [0000]    where Δ represents the vector Laplacian ((Δf) i :=Σ j ∂ 2 f i /∂x j   2 ) and where B and Q have been partitioned into sub-matrices B o , Q o , B i , Q i , such that s T Qs=s oT Q o s o +2s oT Q i s i +f(s i ) and likewise for B. This system can be efficiently solved via a linear change of variables and a backsubstitution exactly analogous to the solution to a finite-dimensional quadratic. Specifically, Schur decomposition is used to write)(B o ) −1 Q o =VUV T , where V is orthonormal and U is upper-triangular. A change of variables z=V T s o  is performed and a new parameter is defined as s p :=Q i s i −B i Δs i . A solution for z is found using backsubstitution, first solving the following scalar partial differential equation for z N     o   , fixing z N     o   , solving for Z N     o     −1 , and proceeding backwards to z 1 : 
         [0000]    
       
         
           
             
               
                 Δ 
                  
                 
                     
                 
                  
                 
                   z 
                   k 
                 
               
               - 
               
                 
                   U 
                   kk 
                 
                  
                 
                   z 
                   k 
                 
               
             
             = 
             
               
                 
                   ( 
                   
                     
                       
                         
                           V 
                           T 
                         
                          
                         
                           ( 
                           
                             B 
                             o 
                           
                           ) 
                         
                       
                       
                         - 
                         1 
                       
                     
                      
                     
                       s 
                       p 
                     
                   
                   ) 
                 
                 k 
               
               + 
               
                 
                   ∑ 
                   
                     j 
                     = 
                     
                       k 
                       + 
                       1 
                     
                   
                   
                     N 
                     o 
                   
                 
                  
                 
                   
                     U 
                     kj 
                   
                    
                   
                     z 
                     j 
                   
                 
               
             
           
         
       
     
         [0000]    where k represents a particular semantic class. 
         [0023]    After solving for z, the output scores are obtained via s o =Vz. The scalar partial differential equations above may be discretized and solved either via fast Fourier transform or by the multigrid method, the complexity of which scales only as N log N in the number of pixels. 
         [0024]    Assuming unit distance between adjacent pixels, discretization yields the following finite system of linear equations ∀x ∈ I ∩           2 , where f denotes the right-hand side of the above equation: 
         [0000]    
       
         
           
             
               f 
                
               
                 ( 
                 x 
                 ) 
               
             
             = 
             
               
                 
                   - 
                   
                     ( 
                     
                       
                         U 
                         kk 
                       
                       + 
                       4 
                     
                     ) 
                   
                 
                  
                 
                   
                     z 
                     k 
                   
                    
                   
                     ( 
                     x 
                     ) 
                   
                 
               
               + 
               
                 
                   ∑ 
                   
                     
                       
                         
                            
                           δ 
                            
                         
                         1 
                       
                       = 
                       1 
                     
                     , 
                     
                       δ 
                       ∈ 
                       
                         ℤ 
                         2 
                       
                     
                   
                 
                  
                 
                   
                     z 
                     k 
                   
                    
                   
                     ( 
                     
                       x 
                       + 
                       δ 
                     
                     ) 
                   
                 
               
             
           
         
       
     
         [0000]    Assuming zero boundary conditions, this system can be solved by a discrete sine transform. Because the above expression can be written as a convolution of z k  with some filter F, z k  can be computed as the inverse discrete sine transform of the discrete sine transform of f divided by the discrete sine transform of F. 
         [0025]    Once a solution for z is found, the V matrix can be inverted to express the output in terms of s o , which forms the output of the VRD module  106 . This output, which provides for each pixel x in the image I a set of values corresponding to each of the k segmentation classes, is used to determine the output segmentation class according to the best score in s o . 
         [0026]    Referring now to  FIG. 2 , a segmentation method is shown. Block  202  processes an input dataset (e.g., an image) using CNN  104 , producing the feature scores s i . At block  204 , the VRD module  106  receives input VRD parameters (e.g., B and Q matrices as described above, processed with Schur decomposition to form the V and U matrices). 
         [0027]    Block  206  transforms the CNN output vector s i  to form an intermediate set of scores s p  via a linear transformation. Block  206  then uses the VRD parameters and s p  to solve the partial differential equations described above for z. As described above, the calculus of variations is used to reduce the optimization problem to such a set of equations. The Schur decomposition provides a change in variables that reduces the system of equations to an upper-triangular form that can be solved as the sequence of scalar-valued partial differential equations (e.g., changing from B o  and Q o  to V and U). Block  206  solves the system of equations for z k  in decreasing order from k=N to k=1. Once z k  is fully solved in this way, block  208  finds output class scores s o  based on z, reversing the linear change in variables, to provide the segment assignment for each pixel in the image. 
         [0028]    Referring now to  FIG. 3 , a method of model learning and CNN training is shown. During learning, an input error signal is generated by comparing an expected segmentation output with s o , with the difference between the two representing the error signal. The error signal is defined as a differentiable loss function L(s o ). Gradient-based learning computes the derivatives of L with respect to the parameter matrices B, Q, and potentially the inputs s i , allowing the model to be used in backpropagation. The backpropagation derivative is 
         [0000]    
       
         
           
             
               
                 
                   d 
                    
                   
                       
                   
                    
                   L 
                 
                 
                   ds 
                   p 
                 
               
                
               
                 : 
               
                
               I 
             
             → 
             
               
                 N 
                 o 
               
             
           
         
       
     
         [0000]    and can be solved by solving the same partial differential equation system as in the inference process described above, replacing s p  with 
         [0000]    
       
         
           
             
               
                 d 
                  
                 
                     
                 
                  
                 L 
               
               
                 d 
                  
                 
                     
                 
                  
                 
                   s 
                   o 
                 
               
             
             . 
           
         
       
     
         [0000]    Specifically, the following equation is solved for 
         [0000]    
       
         
           
             
               
                 d 
                  
                 
                     
                 
                  
                 L 
               
               
                 d 
                  
                 
                     
                 
                  
                 
                   s 
                   o 
                 
               
             
              
             
               : 
             
           
         
       
     
         [0000]    
       
         
           
             
               
                 
                   B 
                   o 
                 
                  
                 
                     
                 
                  
                 Δ 
                  
                 
                   
                     d 
                      
                     
                         
                     
                      
                     L 
                   
                   
                     d 
                      
                     
                         
                     
                      
                     
                       s 
                       p 
                     
                   
                 
               
               - 
               
                 
                   Q 
                   o 
                 
                  
                 
                   
                     d 
                      
                     
                         
                     
                      
                     L 
                   
                   
                     d 
                      
                     
                         
                     
                      
                     
                       s 
                       p 
                     
                   
                 
               
             
             = 
             
               
                 d 
                  
                 
                     
                 
                  
                 L 
               
               
                 d 
                  
                 
                     
                 
                  
                 
                   s 
                   o 
                 
               
             
           
         
       
     
         [0029]    Block  302  receives an error signal from the training data and block  304  receives the input VRD parameters (e.g., the same parameters as are used in block  204  above). Block  306  solves the partial differential equations for z. Block  308  uses the variable change relationship 
         [0000]    
       
         
           
             
               
                 d 
                  
                 
                     
                 
                  
                 L 
               
               
                 d 
                  
                 
                     
                 
                  
                 
                   s 
                   p 
                 
               
             
             = 
             Vz 
           
         
       
     
         [0000]    to produce the output backpropagation derivative and block  310  determines VRD parameter derivatives. The parameter derivatives can be expressed as simple functions of the backpropagation derivative as follows: 
         [0000]    
       
         
           
             
               
                 d 
                  
                 
                     
                 
                  
                 L 
               
               
                 d 
                  
                 
                     
                 
                  
                 
                   B 
                   o 
                 
               
             
             = 
             
               - 
               
                 
                   ∫ 
                   I 
                 
                  
                 
                   
                     
                       d 
                        
                       
                           
                       
                        
                       L 
                     
                     
                       d 
                        
                       
                           
                       
                        
                       
                         s 
                         p 
                       
                     
                   
                    
                   Δ 
                    
                   
                       
                   
                    
                   
                     
                       s 
                       o 
                     
                     T 
                   
                    
                   dx 
                 
               
             
           
         
       
       
         
           
             
               
                 d 
                  
                 
                     
                 
                  
                 L 
               
               
                 d 
                  
                 
                     
                 
                  
                 
                   Q 
                   o 
                 
               
             
             = 
             
               
                 ∫ 
                 I 
               
                
               
                 
                   
                     d 
                      
                     
                         
                     
                      
                     L 
                   
                   
                     d 
                      
                     
                         
                     
                      
                     
                       s 
                       p 
                     
                   
                 
                  
                 
                   
                     s 
                     o 
                   
                   T 
                 
                  
                 dx 
               
             
           
         
       
     
         [0030]    Block  312  then adjusts the parameters for the VRD module  106  and the CNN  104  according to the error signal, with the respective derivatives providing a degree of parameter change needed for a given error signal. 
         [0031]    Embodiments described herein may be entirely hardware, entirely software or including both hardware and software elements. In a preferred embodiment, the present invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc. 
         [0032]    Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. A computer-usable or computer readable medium may include any apparatus that stores, communicates, propagates, or transports the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. The medium may include a computer-readable storage medium such as a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk, etc. 
         [0033]    Each computer program may be tangibly stored in a machine-readable storage media or device (e.g., program memory or magnetic disk) readable by a general or special purpose programmable computer, for configuring and controlling operation of a computer when the storage media or device is read by the computer to perform the procedures described herein. The inventive system may also be considered to be embodied in a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein. 
         [0034]    A data processing system suitable for storing and/or executing program code may include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code to reduce the number of times code is retrieved from bulk storage during execution. Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I/O controllers. 
         [0035]    Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters. 
         [0036]    Referring now to  FIG. 400 , an exemplary monitoring system  400  is shown that provides one concrete example of image segmentation. In particular, the monitoring system  400  includes a hardware processor  402  and a memory  404 . The monitoring system  400  further includes one or more cameras  412  and/or other sensors that may be used to collect information in an environment. The monitoring system  400  further includes one or more functional modules that may, in one embodiment, be implemented as software that is stored in memory  404  and executed by hardware processor  402 . In an alternative embodiment, the functional modules may be implemented as one or more discrete hardware components, for example as application specific integrated chips or field programmable gate arrays. 
         [0037]    A CNN  406  takes as input an image I that may be captured by the camera  412  and stored in memory  404  or that may be provided by any other source. The CNN  406  assigns, for each pixel in the image, scores that correspond to a set of different segmentation classes that are learned by the CNN  406 . A VRD module  408  enforces a smoothness criterion on the scores output by the CNN, providing updated scores for each pixel of the image. A segmentation module  410  then determines which pixels belong to each segmentation class, with each pixel being assigned to a class in accordance with the best score (e.g., highest or lowest score, depending on how the scores are calculated) out of that pixels updated scores. 
         [0038]    If the segmentation module  410  indicates the existence in the image of, for example, a particular class or pattern of classes within the input image, an alert module  414  may be configured to provide a monitoring alert to an operator and, optionally, to initiate an automated action such as, e.g., locking doors or increasing a physical security level of a premises. Alternatively, the alert module  414  may trigger on a change in the segmentation scores beyond a predetermined threshold. Any appropriate condition may be implemented to capture, for example, motion or the presence of a particular kind of segmentation pattern within a video feed. A condition may be triggered by, e.g., a score for a given segmentation class exceeding a threshold or if a change in the segmentation scores for one or more of the segmentation classes exceeds a threshold. 
         [0039]    Referring now to  FIG. 5 , an exemplary processing system  500  is shown which may represent the monitoring system  400 . The processing system  500  includes at least one processor (CPU)  504  operatively coupled to other components via a system bus  502 . A cache  506 , a Read Only Memory (ROM)  508 , a Random Access Memory (RAM)  510 , an input/output (I/O ) adapter  520 , a sound adapter  530 , a network adapter  540 , a user interface adapter  550 , and a display adapter  560 , are operatively coupled to the system bus  502 . 
         [0040]    A first storage device  522  and a second storage device  524  are operatively coupled to system bus  502  by the  1 / 0  adapter  520 . The storage devices  522  and  524  can be any of a disk storage device (e.g., a magnetic or optical disk storage device), a solid state magnetic device, and so forth. The storage devices  522  and  524  can be the same type of storage device or different types of storage devices. 
         [0041]    A speaker  532  is operatively coupled to system bus  502  by the sound adapter  530 . A transceiver  542  is operatively coupled to system bus  502  by network adapter  540 . A display device  562  is operatively coupled to system bus  502  by display adapter  560 . 
         [0042]    A first user input device  552 , a second user input device  554 , and a third user input device  556  are operatively coupled to system bus  502  by user interface adapter  550 . The user input devices  552 ,  554 , and  556  can be any of a keyboard, a mouse, a keypad, an image capture device, a motion sensing device, a microphone, a device incorporating the functionality of at least two of the preceding devices, and so forth. Of course, other types of input devices can also be used, while maintaining the spirit of the present principles. The user input devices  552 ,  554 , and  556  can be the same type of user input device or different types of user input devices. The user input devices  552 ,  554 , and  556  are used to input and output information to and from system  500 . 
         [0043]    Of course, the processing system  500  may also include other elements (not shown), as readily contemplated by one of skill in the art, as well as omit certain elements. For example, various other input devices and/or output devices can be included in processing system  500 , depending upon the particular implementation of the same, as readily understood by one of ordinary skill in the art. For example, various types of wireless and/or wired input and/or output devices can be used. Moreover, additional processors, controllers, memories, and so forth, in various configurations can also be utilized as readily appreciated by one of ordinary skill in the art. These and other variations of the processing system  500  are readily contemplated by one of ordinary skill in the art given the teachings of the present principles provided herein. 
         [0044]    The foregoing is to be understood as being in every respect illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is not to be determined from the Detailed Description, but rather from the claims as interpreted according to the full breadth permitted by the patent laws. It is to be understood that the embodiments shown and described herein are only illustrative of the principles of the present invention and that those skilled in the art may implement various modifications without departing from the scope and spirit of the invention. Those skilled in the art could implement various other feature combinations without departing from the scope and spirit of the invention. Having thus described aspects of the invention, with the details and particularity required by the patent laws, what is claimed and desired protected by Letters Patent is set forth in the appended claims.