Abstract:
Learning processes for a single hidden layer neural network, including linear input units, nonlinear hidden units, and linear output units, calculate the lower-layer network parameter gradients by taking into consideration a solution for the upper-layer network parameters. The upper-layer network parameters are calculated by a closed form formula given the lower-layer network parameters. An accelerated gradient algorithm can be used to update the lower-layer network parameters. A weighted gradient also can be used. With the combination of these techniques, accelerated training with faster convergence, to a point with a lower error rate, can be obtained.

Description:
BACKGROUND 
     Single-hidden-layer neural networks (SHLNN) with least square error training are commonly used in data processing applications such as various types of pattern classification, including but not limited to image classification, text classification, handwriting recognition and speech recognition, due partly to their powerful modeling ability and partly due to the existence of efficient learning algorithms. 
     In a single hidden layer neural network, given the set of input vectors X=[x 1 , . . . , x i , . . . , x N ], each vector is denoted by x i =[x 1i , . . . , x ji , . . . , x Di ] T  where D is the dimension of the input vector and N is the total number of training samples. Also, L is the number of hidden units and C is the dimension of the output vector. The output of the SHLNN is y i =U T h i , where h i =σ(W T x i ) is the hidden layer output, U is an L×C weight matrix at the upper layer, W is an D×L weight matrix at the lower layer, and σ(·) is the sigmoid function. Bias terms are implicitly represented in the above formulation if x i  and h i  are augmented with 1&#39;s. 
     Given the target vectors T=[t 1 , . . . , t i , . . . , t N ], where each target t i =[t 1i , . . . , t ji , . . . , t Ci ] T , the parameters U and W are learned to minimize the square error E=∥Y−T∥ 2 =Tr[(Y−T)(Y−T) T ], where Y=[y 1 , . . . , y i , . . . , y N ]. After the lower layer weights W are fixed, the hidden layer values H=[h 1 , . . . , h i , . . . , h N ] are also determined uniquely. Subsequently, the upper layer weights U can be determined by setting the gradient 
                 ∂   E       ∂   U       =         ∂     Tr   ⁡     [       (         U   T     ⁢   H     -   T     )     ⁢       (         U   T     ⁢   H     -   T     )     T       ]           ∂   U       =     2   ⁢       H   ⁡     (         U   T     ⁢   H     -   T     )       T               
to zero, leading to the closed-form solution: U=(HH T ) −1 HT T , which defines an implicit constraint between the two sets of weights, U and W, via the hidden layer output H, in the SHLNN.
 
     In some systems, achieving good classification accuracy involves utilizing a large number of hidden units, which increases the model size and the test time. Tradeoffs designed to reduce model size, often are not efficient in finding good model parameters. 
     SUMMARY 
     This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. 
     The training process for a single hidden layer neural network calculates the lower-layer network parameter gradients by taking into consideration a solution for the upper-layer network parameters. An accelerated gradient algorithm can be used to update the lower-layer network parameters. A weighted gradient also can be used. With the combination of these techniques, accelerated training with faster convergence, to a point with a lower error rate, can be obtained. 
     Accordingly, in various aspects the subject matter can be embodied in a computer-implemented process, an article of manufacture and/or a computing machine. Data defining a single hidden layer neural network is received into memory. During training, a gradient of a square error with respect to lower layer network parameters is computed using upper layer network parameters as a function of the lower layer network parameters. The network parameters are updated according to the computed gradient. In one implementation, the lower layer network parameters are updated using a gradient history. In another implementation, the gradient uses a weight for each sample, wherein the weights are positively correlated to errors introduced by each sample. 
     In the following description, reference is made to the accompanying drawings which form a part hereof, and in which are shown, by way of illustration, specific example implementations of this technique. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the disclosure. 
    
    
     
       DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram of an example application environment in which the SHLNN can be used. 
         FIG. 2  is a flow chart describing an example implementation of the training process. 
         FIG. 3  is a flow chart describing another example implementation of the training process. 
         FIG. 4  is a flow chart describing another example implementation of the training process. 
         FIG. 5  is a flow chart describing another example implementation of the training process. 
         FIG. 6  is a block diagram of an example computing device in which such a system can be implemented. 
     
    
    
     DETAILED DESCRIPTION 
     The following section provides an example operating environment in which an SHLNN can be implemented. 
     Referring to  FIG. 1 , a single hidden layer neural network (SHLNN) is trained by a training process  100 . Input training data  102  is applied to the input layer  106  of an SHLNN and output data  112  is provided by the output layer  110 . The output data  112  is compared to output training data  104  to obtain error data, and the training process  100  continually adjusts (as indicated by arrows  114  and  116 ) the weights between the input layer  106  and the hidden layer  108  and the weights between the hidden layer  108  and the output layer  110  to minimize the error. 
     Such training may be performed to create a pattern classification tool. For example, two dimensional images including handwriting from an input device can be used as the inputs to the input layer, and the output layer can provide outputs classifying images as a letter. As another example, audio data including speech from an input device can be used as the inputs to the input layer, and the output layer can provide outputs classifying the speech into letters, phonemes or other components of speech. 
     After a neural network is trained, actual data can be input to the input layer, and the output layer outputs would indicate how the input data is classified. 
     Given this context, example implementations of the training process will be described in more detail in connection with  FIGS. 2-5 . 
       FIG. 2  illustrates an example implementation that is unaware of the upper-layer solution when calculating the gradient. In particular, given fixed current U (weight matrix of the upper layer) and W (weight matrix of the lower layer) as shown at  200 , the following gradient is computed ( 202 ): 
                             ∂   E       ∂   W       =       ⁢       ∂     Tr   ⁡     [       (         U   T     ⁢     σ   ⁡     (       W   T     ⁢   X     )         -   T     )     ⁢       (         U   T     ⁢     σ   ⁡     (       W   T     ⁢   X     )         -   T     )     T       ]           ∂   W                   =       ⁢     2   ⁢     X   ⁡     [     H   ∘     (     1   -   H     )     ∘       (         UU   T     ⁢   H     -   UT     )     T       ]                       (   1   )               
where ∘ is element-wise product.
 
     In this implementation, W is updated ( 204 ) first using the gradient defined directly as 
                     W     k   +   1       =       W   k     -     ρ   ⁢       ∂   E       ∂     W   ′                     (   2   )               
where ρ is the learning rate. Next, U is calculated ( 206 ) using the closed-form solution: U=(HH T ) −1 HT T .
 
     In another implementation, shown in  FIG. 3 , the training process relies on the fact that the upper layer weight matrix U completely depends on the lower layer weight matrix W. In this implementation, given fixed current U (weight matrix of the upper layer) and W (weight matrix of the lower layer) as shown at  300 , the gradient ∂E/∂W is computed  302  by considering W&#39;s effect on the upper-layer weights U and thus the effect on the square error as the training objective function. By treating U a function of W, and substituting the closed form solution of U=(HH T ) −1 HT T  into the equation for the squared error E, the gradient is defined as: 
                             ∂   E       ∂   W       =       ⁢       ∂     Tr   ⁡     [       (         U   T     ⁢   H     -   T     )     ⁢       (         U   T     ⁢   H     -   T     )     T       ]           ∂   W                   =       ⁢       ∂     Tr   ⁡     [       (           [         (     HH   T     )       -   1       ⁢     HT   T       ]     T     ⁢   H     -   T     )     ⁢       (           [         (     HH   T     )       -   1       ⁢     HT   T       ]     T     ⁢   H     -   T     )     T       ]           ∂   W                   =       ⁢       ∂     Tr   ⁡     [       TT   T     -           TH   T     ⁡     (     HH   T     )         -   1       ⁢     HT   T         ]           ∂   W                   =       ⁢           -     ∂       Tr   [     HH   T     )       -   1           ⁢     HT   T     ⁢     TH   T       ]       ∂   W                   =       ⁢       -     ∂     Tr   ⁡     [         (         σ   ⁡     (       W   T     ⁢   X     )       ⁡     [     σ   ⁡     (       W   T     ⁢   X     )       ]       T     )       -   1       ⁢     σ   ⁡     (       W   T     ⁢   X     )       ⁢     T   T     ⁢       T   ⁡     [     σ   ⁡     (       W   T     ⁢   X     )       ]       T       ]             ∂   W                   =       ⁢     2   ⁢       X   ⁡     [       H   T     ∘       (     1   -   H     )     T     ∘     [           H   †     ⁡     (     HT   T     )       ⁢     (     TH   †     )       -       T   T     ⁡     (     TH   †     )         ]       ]       .                     (   3   )             where                           H   †     =         H   T     ⁡     (     HH   T     )         -   1               (   4   )               
is the pseudo-inverse of H. In the derivation of (3), HH T  is symmetric and so is (HH T ) −1 . Also,
 
     
       
         
           
             
               
                 
                   
                     
                       ∂ 
                       
                         Tr 
                         ⁡ 
                         
                           [ 
                           
                             
                               
                                 ( 
                                 
                                   HH 
                                   T 
                                 
                                 ) 
                               
                               
                                 - 
                                 1 
                               
                             
                             ⁢ 
                             
                               HT 
                               T 
                             
                             ⁢ 
                             
                               TH 
                               T 
                             
                           
                           ] 
                         
                       
                     
                     
                       ∂ 
                       
                         H 
                         T 
                       
                     
                   
                   = 
                   
                     
                       
                         - 
                         2 
                       
                       ⁢ 
                       
                         
                           
                             H 
                             T 
                           
                           ⁡ 
                           
                             ( 
                             
                               HH 
                               T 
                             
                             ) 
                           
                         
                         
                           - 
                           1 
                         
                       
                       ⁢ 
                       
                         HT 
                         T 
                       
                       ⁢ 
                       
                         
                           
                             TH 
                             T 
                           
                           ⁡ 
                           
                             ( 
                             
                               HH 
                               T 
                             
                             ) 
                           
                         
                         
                           - 
                           1 
                         
                       
                     
                     + 
                     
                       2 
                       ⁢ 
                       
                         T 
                         T 
                       
                       ⁢ 
                       
                         
                           
                             
                               TH 
                               T 
                             
                             ⁡ 
                             
                               ( 
                               
                                 HH 
                                 T 
                               
                               ) 
                             
                           
                           
                             - 
                             1 
                           
                         
                         . 
                       
                     
                   
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
     
     Given this gradient, W is updated  304 , then U is updated  306 . In this implementation, because the effect of W on U is used, training tends to move W towards a direction that finds optimal points faster. However, due to the more complicated gradient calculation that involves a pseudo-inverse, each iteration takes a longer time than the iterations of the first implementation. The products of matrices are grouped in (3) to reduce memory usage when the number of samples is very large. 
     Another implementation, described now in connection with  FIG. 4 , relies on the principle that, for convex problems, convergence speed can be improved if gradient information over history is used when updating the weights. Although the increase in speed may not be guaranteed for non-convex problems, in practice, faster convergence, to a point where a lower error rate is achieved, has occurred. In particular, during initialization  400 , the process sets W 0  and then sets  W   1 =W 0  and m 1 =1. The current U and W are set and the gradient is computed as in the other implementations, as indicated at  402 . Then W and U are updated ( 404 ,  406 ). When updating  404  the lower weight matrix, W,  W  and t are updated by: 
     
       
         
           
             
               W 
               k 
             
             = 
             
               
                 
                   
                     W 
                     _ 
                   
                   k 
                 
                 - 
                 
                   ρ 
                   ⁢ 
                   
                     
                       ∂ 
                       E 
                     
                     
                       ∂ 
                       
                         
                           W 
                           _ 
                         
                         ′ 
                       
                     
                   
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   and 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   
                     m 
                     
                       k 
                       + 
                       1 
                     
                   
                 
               
               = 
               
                 
                   
                     1 
                     2 
                   
                   ⁢ 
                   
                     ( 
                     
                       1 
                       + 
                       
                         
                           1 
                           + 
                           
                             4 
                             ⁢ 
                             
                               m 
                               k 
                               2 
                             
                           
                         
                       
                     
                     ) 
                   
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   and 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   
                     
                       W 
                       _ 
                     
                     
                       k 
                       + 
                       1 
                     
                   
                 
                 = 
                 
                   
                     W 
                     k 
                   
                   + 
                   
                     
                       
                         m 
                         
                           k 
                           - 
                           1 
                         
                       
                       
                         m 
                         
                           k 
                           + 
                           1 
                         
                       
                     
                     ⁢ 
                     
                       
                         ( 
                         
                           
                             W 
                             k 
                           
                           - 
                           
                             W 
                             
                               k 
                               - 
                               1 
                             
                           
                         
                         ) 
                       
                       . 
                     
                   
                 
               
             
           
         
       
     
     In the foregoing implementations, each sample is weighted the same. The convergence speed can be increased by focusing on samples with the most errors, as described in connection with  FIG. 5 . This implementation makes use of a weight for each sample i, namely: 
     
       
         
           
             
               λ 
               ii 
             
             = 
             
               
                 ( 
                 
                   
                     
                       N 
                       E 
                     
                     ⁢ 
                     
                       
                          
                         
                           
                             y 
                             i 
                           
                           - 
                           
                             t 
                             i 
                           
                         
                          
                       
                       2 
                     
                   
                   + 
                   1 
                 
                 ) 
               
               / 
               2 
             
           
         
       
     
     where E is the square error over the whole training set and N is the training set size. The weights are so chosen that they are positively correlated to the errors introduced by each sample while being smoothed to make sure weights assigned to each sample is at least 0.5. Thus, after initializing values and W and U ( 500 ), these weights are initialized  502 . At each step, instead of minimizing E directly, the weighted error is minimized ( 504 ):
 
 Ë=Tr [( Y−T )Λ( Y−T ) T ],
 
     where Λ=diag[λ 11 , . . . , λ ii , . . . , λ NN ] is an N by N diagonal weight matrix. To minimize Ë, once the lower layer weights W are fixed ( 506 ) the upper layer weights U can be determined ( 506 ) by setting the following gradient to zero: 
     
       
         
           
             
               
                 
                   
                     
                       ∂ 
                       
                         E 
                         ¨ 
                       
                     
                     
                       ∂ 
                       U 
                     
                   
                   = 
                     
                   ⁢ 
                   
                     
                       ∂ 
                       
                         Tr 
                         ⁡ 
                         
                           [ 
                           
                             
                               ( 
                               
                                 Y 
                                 - 
                                 T 
                               
                               ) 
                             
                             ⁢ 
                             
                               
                                 Λ 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     Y 
                                     - 
                                     T 
                                   
                                   ) 
                                 
                               
                               T 
                             
                           
                           ] 
                         
                       
                     
                     
                       ∂ 
                       U 
                     
                   
                 
               
             
             
               
                 
                   = 
                     
                   ⁢ 
                   
                     2 
                     ⁢ 
                     H 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       
                         Λ 
                         ⁡ 
                         
                           ( 
                           
                             
                               
                                 U 
                                 T 
                               
                               ⁢ 
                               H 
                             
                             - 
                             T 
                           
                           ) 
                         
                       
                       T 
                     
                   
                 
               
             
           
         
       
     
     This gradient has the closed form solution of:
 
 U =( HΛH   T ) −1   HΛT   T  
 
     By substitution, the following is derived: 
     
       
         
           
             
               
                 
                   
                     
                       ∂ 
                       
                         E 
                         ¨ 
                       
                     
                     
                       ∂ 
                       W 
                     
                   
                   = 
                   
                     
                       ∂ 
                       
                         Tr 
                         ⁡ 
                         
                           [ 
                           
                             
                               ( 
                               
                                 
                                   
                                     U 
                                     T 
                                   
                                   ⁢ 
                                   H 
                                 
                                 - 
                                 T 
                               
                               ) 
                             
                             ⁢ 
                             
                               
                                 Λ 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     
                                       
                                         U 
                                         T 
                                       
                                       ⁢ 
                                       H 
                                     
                                     - 
                                     T 
                                   
                                   ) 
                                 
                               
                               T 
                             
                           
                           ] 
                         
                       
                     
                     
                       ∂ 
                       W 
                     
                   
                 
               
             
             
               
                 
                   = 
                   
                     
                       ∂ 
                       
                         Tr 
                         ⁡ 
                         
                           [ 
                           
                             
                               ( 
                               
                                 
                                   
                                     
                                       [ 
                                       
                                         
                                           
                                             ( 
                                             
                                               H 
                                               ⁢ 
                                               
                                                   
                                               
                                               ⁢ 
                                               Λ 
                                               ⁢ 
                                               
                                                   
                                               
                                               ⁢ 
                                               
                                                 H 
                                                 T 
                                               
                                             
                                             ) 
                                           
                                           
                                             - 
                                             1 
                                           
                                         
                                         ⁢ 
                                         H 
                                         ⁢ 
                                         
                                             
                                         
                                         ⁢ 
                                         Λ 
                                         ⁢ 
                                         
                                             
                                         
                                         ⁢ 
                                         
                                           T 
                                           T 
                                         
                                       
                                       ] 
                                     
                                     T 
                                   
                                   ⁢ 
                                   H 
                                 
                                 - 
                                 T 
                               
                               ) 
                             
                             ⁢ 
                             
                               
                                 Λ 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     
                                       
                                         
                                           [ 
                                           
                                             
                                               
                                                 ( 
                                                 
                                                   H 
                                                   ⁢ 
                                                   
                                                       
                                                   
                                                   ⁢ 
                                                   Λ 
                                                   ⁢ 
                                                   
                                                       
                                                   
                                                   ⁢ 
                                                   
                                                     H 
                                                     T 
                                                   
                                                 
                                                 ) 
                                               
                                               
                                                 - 
                                                 1 
                                               
                                             
                                             ⁢ 
                                             H 
                                             ⁢ 
                                             
                                                 
                                             
                                             ⁢ 
                                             Λ 
                                             ⁢ 
                                             
                                                 
                                             
                                             ⁢ 
                                             
                                               T 
                                               T 
                                             
                                           
                                           ] 
                                         
                                         T 
                                       
                                       ⁢ 
                                       H 
                                     
                                     - 
                                     T 
                                   
                                   ) 
                                 
                               
                               T 
                             
                           
                           ] 
                         
                       
                     
                     
                       ∂ 
                       W 
                     
                   
                 
               
             
             
               
                 
                   = 
                   
                     
                       ∂ 
                       
                         Tr 
                         ⁡ 
                         
                           [ 
                           
                             
                               T 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               Λ 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               
                                 T 
                                 T 
                               
                             
                             - 
                             
                               T 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               Λ 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               
                                 
                                   
                                     H 
                                     T 
                                   
                                   ⁡ 
                                   
                                     ( 
                                     
                                       H 
                                       ⁢ 
                                       
                                           
                                       
                                       ⁢ 
                                       Λ 
                                       ⁢ 
                                       
                                           
                                       
                                       ⁢ 
                                       
                                         H 
                                         T 
                                       
                                     
                                     ) 
                                   
                                 
                                 
                                   - 
                                   1 
                                 
                               
                               ⁢ 
                               H 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               Λ 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               
                                 T 
                                 T 
                               
                             
                           
                           ] 
                         
                       
                     
                     
                       ∂ 
                       W 
                     
                   
                 
               
             
             
               
                 
                   = 
                   
                     
                       - 
                       
                         ∂ 
                         
                           Tr 
                           ⁡ 
                           
                             [ 
                             
                               
                                 
                                   ( 
                                   
                                     H 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     Λ 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     
                                       H 
                                       T 
                                     
                                   
                                   ) 
                                 
                                 
                                   - 
                                   1 
                                 
                               
                               ⁢ 
                               H 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               Λ 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               
                                 T 
                                 T 
                               
                               ⁢ 
                               T 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               Λ 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               
                                 H 
                                 T 
                               
                             
                             ] 
                           
                         
                       
                     
                     
                       ∂ 
                       W 
                     
                   
                 
               
             
             
               
                 
                   
                     = 
                     
                       2 
                       ⁢ 
                       
                         X 
                         ⁡ 
                         
                           [ 
                           
                             
                               H 
                               T 
                             
                             ∘ 
                             
                               
                                 ( 
                                 
                                   1 
                                   - 
                                   H 
                                 
                                 ) 
                               
                               T 
                             
                             ∘ 
                             
                               [ 
                               
                                 
                                   
                                     
                                       H 
                                       ‡ 
                                     
                                     ⁡ 
                                     
                                       ( 
                                       
                                         H 
                                         ⁢ 
                                         
                                             
                                         
                                         ⁢ 
                                         Λ 
                                         ⁢ 
                                         
                                             
                                         
                                         ⁢ 
                                         
                                           T 
                                           T 
                                         
                                       
                                       ) 
                                     
                                   
                                   ⁢ 
                                   
                                     ( 
                                     
                                       TH 
                                       ‡ 
                                     
                                     ) 
                                   
                                 
                                 - 
                                 
                                   Λ 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   
                                     
                                       T 
                                       T 
                                     
                                     ⁡ 
                                     
                                       ( 
                                       
                                         TH 
                                         ‡ 
                                       
                                       ) 
                                     
                                   
                                 
                               
                               ] 
                             
                           
                           ] 
                         
                       
                     
                   
                   , 
                 
               
             
           
         
       
       
         
           
             
               where 
               ⁢ 
               
                   
               
               ⁢ 
               
                 H 
                 ‡ 
               
             
             = 
             
               Λ 
               ⁢ 
               
                   
               
               ⁢ 
               
                 
                   
                     H 
                     T 
                   
                   ⁡ 
                   
                     ( 
                     
                       H 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       Λ 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         H 
                         T 
                       
                     
                     ) 
                   
                 
                 
                   - 
                   1 
                 
               
             
           
         
       
     
     Because the weights are re-estimated after each iteration, this process will try to move the weights with a larger step toward the direction where the error can be most effectively reduced. After the error for a sample is reduced, the weight for that sample in the next iteration, which speeds up convergence and reduces the likelihood of local optima. 
     Having now described several example implementations, a computing environment in which such a system is designed to operate will now be described. The following description is intended to provide a brief, general description of a suitable computing environment in which this system can be implemented. The system can be implemented with numerous general purpose or special purpose computing hardware configurations. Examples of well known computing devices that may be suitable include, but are not limited to, personal computers, server computers, hand-held or laptop devices (for example, media players, notebook computers, cellular phones, personal data assistants, voice recorders), multiprocessor systems, microprocessor-based systems, set top boxes, game consoles, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. 
       FIG. 6  illustrates an example of a suitable computing system environment. The computing system environment is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of such a computing environment. Neither should the computing environment be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the example operating environment. 
     With reference to  FIG. 6 , an example computing environment includes a computing machine, such as computing machine  600 . In its most basic configuration, computing machine  600  typically includes at least one processing unit  602  and memory  604 . The computing device may include multiple processing units and/or additional co-processing units such as graphics processing unit  620 . Depending on the exact configuration and type of computing device, memory  604  may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two. This most basic configuration is illustrated in  FIG. 6  by dashed line  606 . 
     Additionally, computing machine  600  may also have additional features/functionality. For example, computing machine  600  may also include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in  FIG. 6  by removable storage  608  and non-removable storage  610 . Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer program instructions, data structures, program modules or other data. Memory  604 , removable storage  608  and non-removable storage  610  are all examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by computing machine  600 . Any such computer storage media may be part of computing machine  600 . 
     Computing machine  600  may also contain communications connection(s)  612  that allow the device to communicate with other devices. Communications connection(s)  612  is an example of communication media. Communication media typically carries computer program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal, thereby changing the configuration or state of the receiving device of the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. 
     Computing machine  600  may have various input device(s)  614  such as a display, a keyboard, mouse, pen, camera, touch input device, and so on. Output device(s)  616  such as speakers, a printer, and so on may also be included. All of these devices are well known in the art and need not be discussed at length here. 
     The system may be implemented in the general context of software, including computer-executable instructions and/or computer-interpreted instructions, such as program modules, being processed by a computing machine. Generally, program modules include routines, programs, objects, components, data structures, and so on, that, when processed by a processing unit, instruct the processing unit to perform particular tasks or implement particular abstract data types. This system may be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices. 
     The terms “article of manufacture”, “process”, “machine” and “composition of matter” in the preambles of the appended claims are intended to limit the claims to subject matter deemed to fall within the scope of patentable subject matter defined by the use of these terms in 35 U.S.C. §101. 
     Any or all of the aforementioned alternate embodiments described herein may be used in any combination desired to form additional hybrid embodiments. It should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific implementations described above. The specific implementations described above are disclosed as examples only.