Patent Publication Number: US-11645536-B2

Title: Parallel residual neural network architecture and system and method for training a residual neural network

Description:
TECHNICAL FIELD 
     The following relates generally to artificial neural networks and more specifically to training a residual neural network. 
     BACKGROUND 
     Deep Convolution Neural Networks (CNN) have been used in image recognition tasks with great success. Some conventional approaches include AlexNet, VGG, Inception, and Residual Networks (ResNet). 
     Training a deep neural network is not an easy task. As the gradient at each layer is dependent upon those in higher layers multiplicatively, the gradients in earlier layers can vanish or explode, ceasing the training process. The gradient vanishing problem is significant for neuron activation functions such as the sigmoid, where the gradient approaches zero exponentially away from the origin on both sides. The standard approach to combat vanishing gradient is to apply Batch Normalization (BN) followed by the Rectified Linear Unit (ReLU) activation. More recently, skip connections have been proposed to allow previous layers to propagate relatively unchanged. Using this approach, it may be possible to train extremely deep networks of hundreds of layers; for example, ResNets have been trained with about one thousand layers. 
     As the number of layers grows large, so does the training time. In operation of a neural network in a data processing task, to evaluate the neural network&#39;s output, one needs to propagate the input of the network layer by layer in a procedure known as forward propagation. Similarly, during training of the neural network, one needs to propagate the gradient of the loss function from the end of the network to update the model parameters, or weights, in each layer of the network using gradient descent. The complexity of forward and backward propagation is O(K), where K is the number of layers in the network. There is a need to speed up the training process in deep neural networks. 
     SUMMARY 
     In an aspect, there is provided a method for training a residual neural network executed on one or more processing units, the residual neural network comprising a plurality of warp units connected in series, each warp unit comprising an input, an output, a plurality of independent residual units connected in parallel from the input to the output, a direct connection from the input to the output, and at least one derivative unit connected from the input to the output in parallel to the residual units, each residual unit comprising one or more weights, the method comprising: initializing the one or more weights of each residual unit; inputting a plurality of training cases to the first warp unit in the series; using each training case to optimize the one or more weights for each residual unit in parallel in the first warp unit in the series; starting with the output of the first warp unit in the series, iteratively propagating the output of each warp unit to the input of the next respective warp unit in the series, for each respective warp unit, using each training case to optimize the one or more weights for each residual unit in parallel in the respective warp unit; and storing the output of the last warp unit in the series and the weights for each residual unit. 
     In a particular case, the at least one derivative unit is determined using at least one multiplication of one of the residual units to a derivative of another one of the residual units. 
     In another case, the at least one derivative unit is determined as a derivative of at least one of the residual units. 
     In yet another case, the at least one derivative unit is determined as a derivative of at least one of the residual units multiplied by the input. 
     In yet another case, the output is determined from the input using first-order Taylor Series Expansion of the residual units and the at least one derivative unit. 
     In yet another case, the method further comprising performing back propagation comprising propagating a weight gradient of a loss function from the last warp unit in the series to the first warp unit, in each warp unit, the one or more weights of each residual unit are independently updated in parallel. 
     In yet another case, the weight gradient for each warp unit is determined as a gradient of the output of the warp unit multiplied by a sum comprising the gradients of the residual units and gradients of the at least one derivative unit. 
     In yet another case, the gradient of each of the at least one derivative units comprising at least one multiplication of the gradient of one of the residual units to the gradient of another one of the residual units. 
     In yet another case, the sum further comprises the identity matrix. 
     In yet another case, at least two of the warp units comprising a differing quantity of residual units from each other. 
     In another aspect, there is provided a system for training a residual neural network, the system comprising one or more processors and one or more non-transitory computer storage media, the one or more non-transitory computer storage media causing the one or more processors to execute the residual neural network comprising: a plurality of warp units comprising an input and an output and a direct connection from the input to the output; for each of the warp units, a plurality of residual units connected in parallel, each residual unit comprising one or more weights; for each of the warp units, at least one derivative unit connected in parallel to the residual units; and a warp operator to receive the inputs and outputs from each of the warp units and to train the residual neural network, the training comprising optimizing the weights of each residual unit based on a plurality of training cases. 
     In a particular case, the weights for each residual unit in a warp unit is determined on a separate processing unit. 
     In another case, training of the residual neural network comprising; initializing the one or more weights of each residual unit; inputting a plurality of training cases to the first warp unit in the series; using each training case to optimize the one or more weights for each residual unit in parallel in the first warp unit in the series; starting with the output of the first warp unit in the series, iteratively propagating the output of each warp unit to the input of the next respective warp unit in the series, for each respective warp unit, using each training case to optimize the one or more weights for each residual unit in parallel in the respective warp unit; and storing the output of the last warp unit in the series and the weights for each residual unit in the one or more non-transitory computer storage media. 
     In yet another case, the at least one derivative unit is determined using at least one multiplication of one of the residual units to a derivative of another one of the residual units. 
     In yet another case, the at least one derivative unit is determined as a derivative of at least one of the residual units. 
     In yet another case, the at least one derivative unit is determined as a derivative of at least one of the residual units multiplied by the input. 
     In yet another case, the output is determined from the input using first-order Taylor Series Expansion of the residual units and the at least one derivative unit. 
     In yet another case, the warp operator further performing, with the warp units, back propagation, the back propagation comprising propagating a weight gradient of a loss function from the last warp unit in the series to the first warp unit, in each warp unit, the one or more weights of each residual unit are independently updated in parallel. 
     In yet another case, the weight gradient for each warp unit is determined as a gradient of the output of the warp unit multiplied by a sum comprising the gradients of the residual units and gradients of the at least one derivative unit. 
     In yet another case, the gradient of each of the at least one derivative units comprising at least one multiplication of the gradient of one of the residual units to the gradient of another one of the residual units. 
     These and other aspects are contemplated and described herein. It will be appreciated that the foregoing summary sets out representative aspects of a system and method for training a residual neural network and assists skilled readers in understanding the following detailed description. 
    
    
     
       DESCRIPTION OF THE DRAWINGS 
       A greater understanding of the embodiments will be had with reference to the Figures, in which: 
         FIG.  1    is an architecture diagram of a residual unit in a conventional residual neural network; 
         FIG.  2    is a block diagram of a conventional residual neural network; 
         FIG.  3    is a block diagram of an embodiment of a system for training a residual neural network; 
         FIG.  4    is a further embodiment of a system for training a residual neural network; 
         FIG.  5    is a table illustrating an exemplary structure of a system for training a residual neural network, in accordance with an embodiment of the presently disclosed system; 
         FIG.  6    and  FIG.  7    are tables illustrating differences in test error percentages and improvement percentages in training speed between a conventional residual neural network and a residual neural network trained using an embodiment of the presently disclosed system on the CIFAR10 and CIFAR100 benchmark datasets; 
         FIG.  8    is a flowchart illustrating a method for training a residual neural network, in accordance with an embodiment of the presently disclosed system; 
         FIG.  9    is a table illustrating differences in validation error percentages between a conventional residual neural network and a residual neural network trained using an embodiment of the presently disclosed system on the ImageNet benchmark dataset; and 
         FIG.  10    is a chart illustrating differences in validation error between a conventional residual neural network and a residual neural network trained using an embodiment of the presently disclosed system on the ImageNet benchmark dataset. 
     
    
    
     DETAILED DESCRIPTION 
     Embodiments will now be described with reference to the figures. For simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Also, the description is not to be considered as limiting the scope of the embodiments described herein. 
     Any module, unit, component, server, computer, terminal or device exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the device or accessible or connectable thereto. Any application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media. 
     To speed up the training process for a deep neural network, it may be advantageous to have a shallower network that accurately approximates the deep network. Applicant has found that residual neural network (ResNet) architectures permit such an approximation. However, Applicant has further established a faster approach to training K layer ResNets in O(1) time as opposed to O(K) time with forward and backward propagation. 
     In one aspect, an artificial neural network architecture is provided for training a ResNet. In another aspect, a system and method for training a ResNet are provided. The system may comprise a parallelizable warp operator for training the ResNet by restructuring residual units of the ResNet using a first-order Taylor series approximation. 
     The following can be used to significantly reduce the training times for a ResNet. This speedup in training may be achieved by decoupling the ResNet into individual residual units, or layers, and training them in parallel using the parallelizable warp operator. The number of parallelizable computations grows exponentially with the number of ResNet layers skipped; therefore, it is a significant improvement that most, or even all, of the ResNet layers can be trained in the time of just a few. 
     Referring now to  FIG.  1   , a single residual unit  100  in a conventional ResNet is shown. The residual unit  100  is composed of a stack of weight layers  102  and a shortcut connection  104 . The number of weight layers  102  can vary, e.g.  FIG.  1    shows two weight layers  102   a,    102   b  but there may be more; single-layer residual units are uncommon, however, due to a lack of advantages. The weight layers  102  can also vary in size and type, e.g. they may be fully-connected layers or convolutional layers. 
     Shortcut connection  104  skips one or more weight layers  102 . The purpose of shortcut connections is to perform an identity mapping, such that the shortcut connection  104  maps an input  106  that is given to the residual unit  100  to be added to an output  108  of a final layer  102   b  of the weight layers  102 . A person of skill will readily understand operation of a residual unit of a conventional ResNet as shown. 
       FIG.  2    illustrates a conventional ResNet  200 , which typically consists of a long chain of residual units  100 , each generally having the structure shown in  FIG.  1   , connected in sequence by connections  202 . Previous studies suggest that ResNets behave as an ensemble of shallow networks and that randomly deactivating residual units during training (similar to the drop-out technique) appears to improve performance. Applicant has found based on these observations that the output of a residual unit can be considered as a small perturbation of the input. 
     ResNets appear to exhibit an iterative estimation behavior as opposed to representation learning. Within one ResNet stage ( FIG.  2    illustrates a single stage, where a stage is a chain of residual units having outputs of the same dimension), which has shortcut connections in all but the first residual units, the feature map representation appears to steadily converge to one that minimizes the loss function at the end of the stage. ResNets appear to perform representation learning in the first few layers and iterative estimation in the rest within the same stage. 
     ResNets may be explained by applying the perturbation theory. Consider the input x i  of the i-th residual unit of the ResNet 200 and its output x i+1 , where
 
 x   i+1   =h ( x   i )+ F   i ( x   i   ,W   i )   (1)
 
     Typically, h(x i ) is taken to be an identity mapping, h i (x i )=x i , which represents the shortcut connection. When the feature maps are down sampled, h may be a 1×1 convolution layer with a stride of 2. The function F i  may a combination of convolution, normalization and non-linearity layers, so that W i  collectively represents the weights of all layers in 
     Applicant has determined that the interpretation of ResNets as an ensemble of subnetworks is accurate up to the first order in F with identity mapping. The identity mapping causes F to be small compared to the input x so one can approximate the output of a chain of residual units by a series expansion. For instance, the output of two residual units x 3  is related to the input of the first unit. When x k  is expressed in terms of x k+1 , which is referred to as an iteration. The following equations show two iterations.
 
 x   3   =x   2   +F   2 ( x   2   ,W*   2 )
 
 x   3   =x   1 +F 1 ( x   1   , W*   1 )+ F   2 ( x   1   ,W*   2 )+ F′   2 ( x   1   ,W*   2 ) F   1 ( x   1   ,W*   1 )+ O (∈ 2 )   (2)
 
where F′ 2  (x 1 , W* 2 ) denotes the partial derivative of F 2  with respect to x 1 . A Taylor series expansion in powers of F 1  was performed on F 2  in the second line above. The Taylor expansion for multivariate functions is f(x+a)=x+a∇ x f(x)+O(∈ 2 ). The first-order Taylor series expansion is almost exact when ReLU activations are used. Equation (2) can be interpreted as an ensemble sum of subnetworks. Although equation (2) relates to the expansion for two iterations, additional iterations could be applied for additional residual units.
 
     Extending equation (2) to apply across K residual units:
 
 x   K+1   =x   1 +Σ σ     c     ∈P(S     k     )\{ø} [Π i=2   K   F′   c(i) ( x   1   ,W*   c(i) )] F   c(1) ( x   1   ,W*   c(1) )+ O (∈ 2 ),   (3)
 
where S K ={1,2, . . . , K}, σ c  any subset of the power set P(S K ) and the last term arises from higher order terms from the Taylor expansion. The indices c are ordered such that c(i)&gt;c(i−1)&gt; . . . &gt;c(1). For a fixed cardinality |σ c |=k, there are
 
               (         K           k         )               
terms, where
 
               (         K           k         )     ⁢                     
is the corresponding binomial coefficient. The binomial nature of subnetworks is evident from this formulation.
 
     In Equation (3), the sum over all subsets of S K , σ c , can be rewritten as a sum over the cardinality k. Each k has a binomial number of terms 
               (         K           k         )     .         
The binomial nature of subnetworks is evident from this formulation. For each k, there are k factors of F in the sum. Thus k can be interpreted as the subnetwork depth.
 
     It may be observed that the gradient in a ResNet is the sum from all subnetwork contributions, including the identity network. Skilled persons are familiar with suitable magnitudes of subnetwork gradients for an 110-layer ResNet. If one takes F to have ReLU non-linearity. Then F″(x)=0. The non-trivial gradient can be expressed as just a product of F and (F′) k−1 . Thus, it can be seen that the gradient norm decreases exponentially with subnetwork depth k. 
     Applicant has further determined that the first order perturbations are sufficient for approximating the ResNet and that higher order perturbations can be discarded. For ReLU non-linearity, the second-order perturbations vanish almost exactly. The second-order perturbation terms all contain the Hessian F″(x)=0. However, after the network is trained, the only non-linear function in F, ReLU, is non-linear only at the origin; batch normalization layers are non-linear during training, due to the scaling by the sample variance. Therefore, all second-order perturbations vanish with almost exactly. The same argument applies to higher orders. 
       FIG.  3    illustrates the presently disclosed warped residual neural network (also referred to herein as WarpNet). The WarpNet  300  comprises a plurality of warp units  304  linked in series, each warp unit  304  comprising a non-overlapping subset of the plurality of ordered residual units  302  which have been decoupled, such that the connections  202  between residual units  100  in  FIG.  1    have been broken or removed. The residual units of any particular warp unit  304  are ordered, that is sequential, in the ResNet. Further, the residual units  302  are connected in parallel within each warp unit and each residual unit is independent of other residual units. A link from the input to the output of the warp unit  304 , and at least one F′, each a derivative unit computed based on the residual unit F, are also connected in parallel with the residual units  302 . The included derivative units are determined based on a Taylor Series expansion to compute the output from the input. 
     The input to the warp unit  304  is selected from the first input among the included residual units  302 , and the output to the warp unit  304  is selected from the last output among the included residual units  302 . For example, in  FIG.  2   , a first residual unit F 1  accepts input x 1  and produces output x 2 =x 1 +F 1  and a second residual unit F 2  accepts input x 2  (which was the output of the first unit) and produces output x 3 =x 2 +F 2 . By contrast, in the WarpNet  300 , a first warp unit  304   a,  comprising residual units  302   a  consisting of F 1  and F 2  accepts input x 1  and directly produces output x 3 , bypassing x 2  altogether. Skilled persons will appreciate the processing savings of such an approach. 
     More specifically, in  FIG.  3   , warp unit  302   a  comprises residual units F 1 , F 2 , derivative unit F′ 2 , and a link from the input x 1  to the output x 3 , connected in parallel. The output of the warp unit  304  is the summation of the outputs of the included residual units, derivative units and the input to the warp unit  304 . It is contemplated that a particular warp unit could comprise additional residual units, as described below, for bypassing additional points in the ResNet. If there are additional residual units, then there may be additional derivative units as well. 
     In  FIG.  3   , additional warp units  304  are shown for various parallel sets of residual units  302 . It is contemplated that a particular WarpNet need not have uniformity amongst warp units  304 ; namely, it is possible in specific implementations for one particular warp unit to comprise a particular number of residual units, and another warp unit to comprise a different number of residual units. 
     The computation in the warp unit  304  is different from that in the conventional ResNet  200 . As illustrated in  FIG.  4   , a system  400  is provided for training a ResNet using a warp operator  402  to compute the output (i.e., x K+1 ) of the warp unit  304  directly from the input (i.e., x 1 ), as shown in Equation (3). The number of weights in the warp unit  304  is the same as the one in the original ResNet  200  for K consecutive residual units. For instance, the weights W 1 , W 2  and W K  may be present in the warp unit  304 . However, these weights can be used and updated in parallel due to the use of the warp operator  402 . Also note that, in  FIG.  3   , F 2  shares the same weights with F′ 2 , as they come from the same residual unit in the original ResNet  200 . 
     The warp operator  402  may be first applied for forward propagation. A modified forward propagation rule for the WarpNet  300  can be described using warp operator  402 , T warp   K , which may, for example, be implemented by a graphics processing unit (GPU). The expression for T warp   K  is derived from Equation (3) by dropping the negligible O(∈ 2 ) term, that is, by using the Taylor series expansion to the first order: 
               x     K   +   1       =       x   1     +       ∑       σ   c     ∈       P   ⁡     (     S   k     )       ⁢   \   ⁢     {   ∅   }           ⁢       [       ∏     i   =   2     K     ⁢           ⁢       F     c   ⁡     (   i   )       ′     ⁡     (       x   1     ,     W     c   ⁡     (   i   )       *       )         ]     ⁢       F     c   ⁡     (   1   )         ⁡     (       x   1     ,     W     c   ⁡     (   1   )       *       )                   
Note that T warp   K  can be calculated in a parallelizable manner for all K. For example, where K=2,
 
 x   3   =x   1   +F   1 ( x   1   ,W   1 )+ F   2 ( x   1   ,W   2 )+ F′   2 ( x   1   ,W   2 ) F   1 ( x   1   ,W   1 )
 
where W i  corresponds to the weights in the i-th residual unit in the original ResNet  200 .
 
     The warp operator  402  may be applied for backpropagation as well. A modified backpropagation rule for the WarpNet  300  can also be described using the warp operator T warp   K . If the upstream gradient ∂L/∂x 5  is known and in order to compute ∂L/∂W 1  for gradient descent, the gradient is first backpropagated down from x 5  to x 3 . With x 5 =T warp   K (x 3 ), the backpropagated gradient can be derived as: 
                 ∂   L       ∂     x   3         =         ∂   L       ∂     x   5         ⁡     [     I   +       ∂       F   3     ⁡     (       x   3     ,     W   3       )           ∂     x   3         +       ∂       F   4     ⁡     (       x   3     ,     W   4       )           ∂     x   3         +         ∂       F   4     ⁡     (       x   3     ,     W   4       )           ∂     x   3         ⁢       ∂       F   3     ⁡     (       x   3     ,     W   3       )           ∂     x   3             ]             
where I is the identity matrix and the derivative of F″ 4  is set to zero for ReLU non-linearities. Applicant has shown that the same kind of parallelism in the warp operator  402  is present for backpropagation as for forward propagation.
 
     The weight gradient for updates can be evaluated as follows: 
                 ∂   L       ∂     W   1         =         ∂   L       ∂     x   3         ⁡     [         ∂       F   1     ⁡     (       x   1     ,     W   1       )           ∂     W   1         +         ∂       F   2     ⁡     (       x   1     ,     W   2       )           ∂     x   1         ⁢       ∂       F   1     ⁡     (       x   1     ,     W   1       )           ∂     W   1             ]             
The update rule is similar for W 2 . Rules for the all other weights in the WarpNet  300  can be obtained in the same way,
 
                 ∂   L       ∂     W   2         =         ∂   L       ∂     x   3         ⁡     [         ∂       F   2     ⁡     (       x   1     ,     W   2       )           ∂     W   2         +           ∂   2     ⁢       F   2     ⁡     (       x   1     ,     W   2       )             ∂     W   2       ⁢     ∂     x   1           ⁢       F   1     ⁡     (       x   1     ,     W   1       )           ]             
The weights W 1  and W 2  can be updated in parallel independently. The derivatives ∂F 2 /∂x 1  in ∂L/∂W 1  and other derivatives needed in ∂L/∂x 3  are already computed in the forward pass which could be saved and reused, for example by storing it in memory  404 . Furthermore, all derivatives needed in ∂L/∂x 3  can also be computed in the forward pass.
 
     In an aspect, as previously mentioned, the warp operator  402  may be implemented on one or more processing units such as CPUs or GPUs. In another aspect, the warp operator  402  may be part of another module, such as a learning module, that is responsible for training the WarpNet  300 . 
     Memory  404  may be provided for storing activations and learned weights for each weight layer  102 . The memory  404  may further store a training set comprising training data. The training data may, for example, be used for image classification in which case the training data may comprise images with known classifications. The memory  404  may further store a validation set comprising validation data. 
     During the training stage, the WarpNet  300  optimizes weights for each weight layer  102 . After learning, the optimized weight configuration can then be applied to test data. Exemplary applications of such a neural network include image classification, machine translation, object recognition, speech recognition and genomic-oriented applications. 
     During the training stage, a plurality of training cases are presented to the WarpNet  300 . In order to determine a final parameter setting for the parameters of the WarpNet  300 , the warp operator  402  can be applied to each warp unit  304  to perform a computation of the output of the residual units  302  using just the input to the warp unit  304 , as illustrated in  FIG.  4   . Each independent residual unit within the warp unit  304  is activated in parallel for the purpose of this computation. Advantageously, this produces comparable results to the original ResNet  200  while having a significant reduction in training time due to the parallelism. 
     Referring now to  FIG.  5   , an exemplary structure of a system for training a ResNet is shown. In order to ensure the validity of the series expansion, the 1×1 convolution layers on shortcut connections may be replaced by an average pooling layer and a concatenate layer to reduce the spatial dimensions of feature maps and multiply their channels. In an aspect, a wide residual architecture may be adopted. The average pooling and concatenate layers may be used so that the identity mapping is preserved across dimensionality changing units. The convolution blocks may be comprised of the following layers, from input to output, BN-Conv-BN-ReLU-Conv-BN, as illustrated in  FIG.  5   . The layers [T warp   (K) ×N warp ] represent forward propagating N warp  times, such that x i+K =T warp   (K) (x i ,W i ,W i+1 , . . . ,W i+K−1 ) and the indices i correspond to the indices in the original ResNet  200 . 
     Experimental results indicate that a computation bottleneck arises from the BN layers in F′ 2 . The reason being the gradient of BN layers contains an averaging operation that is expensive to compute. In an aspect, all BN layers in F′ 2  may be removed from the system. This results in a departure from the series approximation but experimental results indicate that the network still trains well. This is because the normalizing layers are still being trained in F 1,2 . To further improve the speedup, it may be beneficial to omit the F 1  block in the derivative term F′ 2 F 1 . 
     In experimentation conducted using Tensorflow, a network was implemented with parameters k w , K and N warp , where k w  is the widening factor, and K is the warp factor. Tensorflow&#39;s automatic differentiation for backpropagation may be employed, where the gradients are calculated by sweeping through the network through the chain rule. 
     Referring now to  FIG.  6    and  FIG.  7   , differences in test error percentages between a conventional residual neural network and a residual neural network trained using an embodiment of a system for training a residual neural network are shown. The corresponding relative speed-up with respect to residual neural network is shown in parentheses.  FIG.  6    corresponds to K=2 and  FIG.  7    corresponds to K=3. The conventional ResNet may have the same pooling and concatenate structure as the architecture shown in  FIG.  5   , but with each warp operator replaced by two residual units. Common data augmentation techniques, such as whitening, flipping and cropping may be used. Experiments may be performed for 80000 iterations, or 204 epochs; a training batch size of 128; and an initial learning rate is 0.1, which drops by a factor of 0.1 at epochs 60, 120, and 160, with a weight decay of 0.0005. 
     To exploit the parallelism of the WarpNet architecture and reduce the memory footprint at the same time, it may be advantageous to compute F 1  on one GPU, F 2  and F′ 2  at the same time on another GPU because they share the same weights. Experimental results indicate significant speed up in both forward and backward propagation. 
     To further speed up WarpNet, the modification where F 1  is replaced by x 1  was used and experiments showed that this modification does not affect the predictive performance of WarpNet. 
     Experimental results suggest that the computation time is relatively constant with respect to the network depth. The speedup appears to scale with increasing widening factor k w  and sizable with just a warp factor of K=2. 
       FIG.  6    and  FIG.  7    shows the predictive accuracy of WarpNets and their speed-up in parentheses for K=2 and K=3, respectively. As illustrated in  FIG.  6    and  FIG.  7   , a WarpNet may outperform a conventional ResNet in CIFAR-10 and CIFAR-100.  FIG.  6    and  FIG.  7    also illustrates exemplary relative speed-up of a WarpNet with respect to a conventional ResNet. The relative speed-up is defined as (t res −t warp )/t res , where t res  is the time taken for ResNet to process a batch and t warp  is the corresponding time for WarpNet. 
     The WarpNet shown in  FIG.  3    is a good approximation of the ResNet of  FIG.  2    and results in a significant savings in processing during training of the network. 
     Another example is shown in  FIG.  9    with reference to a down-sampled (32×32) ImageNet data set. The data set contains 1000 classes with 1281167 training images and 50000 validation images with 50 images in each class. The training batch size is 512, initial learning rate is 0.4 and drops by a factor of 0.1 at every 30 epochs. The weight decay is set to be 0.0001. We use the overall best performing warp operator in the CIFAR experiments, namely, the one containing F′x. Exemplary results are shown in  FIG.  9    and  FIG.  10   . It can be seen that for a given ResNet there is exists a WarpNet that obtains a higher validation accuracy with shorter training time. An increase in K from 2 to 3 with all other parameters fixed corresponds to WarpNet-109-2. The network has more residual units than WRN-73-2. It is observed that WarpNet-109-2 trains 12% faster than WRN-73-2 while resulting in a better validation accuracy. Second, WarpNet can achieve close to a benchmark validation error of 18.9% with WRN-28-10. It is notable that in the implementation used, it was not reasonable to train the corresponding WRN-73-4 on the dataset as the model requires too much memory on a single GPU. This shows that the weight distribution of WarpNet across GPUs allows a bigger network to be trained. Remarkably, the validation error curve for WRN-73-2 and its approximation WarpNet 73-2 (K=2; Nwarp=6) lie almost exactly on top of each other. This suggests that the utilized implementation of WarpNet is a good approximation of the corresponding WRN throughout training. 
       FIG.  8    illustrates a method for training a conventional ResNet using a WarpNet. At block  802 , a plurality of residual units  302  are arranged into one or more subsets corresponding to a plurality of warped units  304 , wherein each residual unit  100  comprises a subset of the plurality of weight layers  102  and a shortcut connection  104  mapping an input  106  of the residual unit  100  to be added to an output  108  of a final layer  102   b  of the subset of the plurality of weight layers  102 . The residual units  302  are decoupled from one another, such that each residual unit in a subset of the plurality of residual units is independent from other residual units. 
     At block  804 , a warp operator  402  is configured to perform a computation of an output of a warped unit  304  from an input to the warped unit using a first-order Taylor series approximation. Each independent residual unit in the subset of the plurality of residual units is activated in parallel for the computation. In other words, the output of each residual unit in the warp unit is calculated in parallel, not in sequence as in a conventional ResNet, which greatly speeds up the computation. 
     At block  806 , a final parameter setting for the plurality of parameters associated with the plurality of weight layers  102  is determined by training the residual neural network on a training set, wherein training the residual network comprises applying the warp operator  402  to each of the plurality of warped units  304 . At block  808 , the final parameter setting is stored on one or more non-transitory computer storage media, such as memory  404 . 
     Although the invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the claims appended hereto.