Title: DECOUPLING THE LAYERS IN RESIDUAL NETWORKS

Abstract: We propose a Warped Residual Network (WarpNet) using a parallelizable warp operator for forward and backward propagation to distant layers that trains faster than the original residual neural network. We apply a perturbation theory on residual networks and decouple the interactions between residual units. The resulting warp operator is a first order approximation of the output over multiple layers. The first order perturbation theory exhibits properties such as binomial path lengths and exponential gradient scaling found experimentally by Veit et al. (2016) . We demonstrate through an extensive performance study that the proposed network achieves comparable predictive performance to the original residual network with the same number of parameters, while achieving a significant speed-up on the total training time. As WarpNet performs model parallelism in residual network training in which weights are distributed over different GPUs, it offers speed-up and capability to train larger networks compared to original residual networks.

1. INTRODUCTION:

Deep Convolution Neural Networks (CNN) have been used in image recognition tasks with great success. Since AlexNet (Krizhevsky et al., 2012) , many other neural architectures have been proposed to achieve start-of-the-art results at the time. Some of the notable architectures include, VGG (Simonyan & Zisserman, 2015) , Inception (Szegedy et al., 2015) and Residual networks (ResNet) (He et al., 2015) .

Training a deep neural network is not an easy task. As the gradient at each layer is dependent upon those in higher layers multiplicatively, the gradients in earlier layers can vanish or explode, ceasing the training process. The gradient vanishing problem is significant for neuron activation functions such as the sigmoid, where the gradient approaches zero exponentially away from the origin on both sides. The standard approach to combat vanishing gradient is to apply Batch Normalization (BN) (Ioffe & Szegedy, 2015) followed by the Rectified Linear Unit (ReLU) (Hahnloser et al., 2000) activation. More recently, skip connections (Srivastava et al., 2015) have been proposed to allow previous layers propagate relatively unchanged. Using this methodology the authors in (Srivastava et al., 2015) were able to train extremely deep networks (hundreds of layers) and about one thousand layers were trained in residual networks (He et al., 2015) .

As the number of layers grows large, so does the training time. To evaluate the neural network's output, one needs to propagate the input of the network layer by layer in a procedure known as forward propagation. Likewise, during training, one needs to propagate the gradient of the loss function from the end of the network to update the model parameters, or weights, in each layer of the network using gradient descent. The complexity of forward and propagation is O(K), where K is the number of layers in the network. To speed up the process, one may ask if there exist a shallower network that accurately approximates a deep network so that training time is reduced. In this work we show that there indeed exists a neural network architecture that permits such an approximation, the ResNet.

Residual networks typically consist of a long chain of residual units. Recent investigations suggest that ResNets behave as an ensemble of shallow networks (Veit et al., 2016) . Empirical evidence supporting this claim includes one that shows randomly deactivating residual units during training (similar to drop-out (Srivastava et al., 2014) ) appears to improve performance (Huang et al., 2016) . The results imply that the output of a residual unit is just a small perturbation of the input. In this work, we make an approximation of the ResNet by using a series expansion in the small perturbation. We find that merely the first term in the series expansion is sufficient to explain the binomial distribution of path lengths and exponential gradient scaling experimentally observed by Veit et al. (2016) . The approximation allows us to effectively estimate the output of subsequent layers using just the input of the first layer and obtain a modified forward propagation rule. We call the corresponding operator the warp operator. The backpropagation rule is obtained by differentiating the warp operator. We implemented a network using the warp operator and found that our network trains faster on image classification tasks with predictive accuracies comparable to those of the original ResNet.

Our contributions in this work include:

• We analytically investigate the properties of ResNets. In particular, we show that the first order term in the Taylor series expansion of the layer output across K residual units has a binomial number of terms, which are interpreted as the number of paths in Veit et al. (2016) , and that for ReLU activations the second and higher order terms in the Taylor series vanish almost exactly.

• Based on the above-mentioned analysis, we propose a novel architecture, WarpNet, which employs a warp operator as a parallelizable propagation rule across multiple layers at a time. The WarpNet is an approximation to a ResNet with the same number of weights.

• We conduct experiments with WarpNet skipping over one and two residual units and show that WarpNet achieves comparable predictive performance to the original ResNet while achieving significant speed-up. WarpNet also compares favorably with data parallelism using mini-batches with ResNet. As opposed to data parallelized ResNet where nearly all the weights are copied to all GPUs, the weights in WarpNet are distributed over various GPUs which enables training of a larger network.

The organization of this paper is as follow. In Section 2 we analyze the properties of ResNet and show that the binomial path length arises from a Taylor expansion to first order. In Section 3 we describe Warped Residual Networks. In Section 4 we show that WarpNet can attain similar performence as the original ResNet while offering a speed-up.

2. PROPERTIES OF RESNETS:

In this section we show that recent numerical results (Veit et al., 2016) is explained when the perturbation theory is applied to ResNets. Consider the input x i of the i-th residual unit and its output x i+1 , where

EQUATION

Typically, h(x i ) is taken to be an identity mapping, h i (x i ) = x i . When the feature maps are down sampled, h is usually taken to be an 1 × 1 convolution layer with a stride of 2. The functions F i is a combination of convolution, normalization and non-linearity layers, so that W i collectively represents the weights of all layers in F i . In this work we only consider the case where the skip connection is the identity, h i (x i ) = x i .

Perturbative feature map flow First, we show that the interpretation of ResNets as an ensemble of subnetworks is accurate up to the first order in F with identity mapping. One can approximate the output of a chain of residual units by a series expansion. For instance, the output of two residual units x 3 is related to the input of the first unit by the following (we call the process where x k is expressed in terms of x k-1 an iteration. The following equations show two iterations).

x 3 = x 2 + F 2 (x 2 , W * 2 ) = x 1 + F 1 (x 1 , W * 1 ) + F 2 (x 1 + F 1 (x 1 , W * 1 ), W * 2 ) = x 1 + F 1 (x 1 , W * 1 ) + F 2 (x 1 , W * 2 ) + F 1 (x 1 , W * 1 )F 2 (x 1 , W * 2 ) + O( 2 ),

(2) where F 2 (x 1 , W *

2 ) denotes the partial derivative of F 2 with respect to x 1 and W * i denotes the weights at the loss minimum. A Taylor series expansion in powers of F 1 was performed on F 2 in the second line above. 1 The O(2 ) term arises from the Taylor series expansion, representing higher order terms. Equation ( 2) can be interpreted as an ensemble sum of subnetworks.

Below we show that the second and higher order terms are negligible, that is, the first order Taylor series expansion is almost exact, when ReLU activations are used. The second order perturbation terms all contain the Hessian F (x). But after the network is trained, the only non-linear function in F, ReLU, is only non-linear at the origin 2 . Therefore all second order terms vanish almost exactly. The same argument applies to higher orders. 1). The output across K residual units with ReLU non-linearity is

Theorem 1: Binomial Path Length Let the set of indices σ c = {c(1), c(2), . . . c(k)} obtained by choosing any subset of S K = {1, 2, • • • , K}, k < K, and then ordering such that c(k) > c(k -1) > • • • > c(

EQUATION

where the sum is over all subsets σ c and P(S K ) denotes the power set of S K . We have omitted the O( 2) term because the first order approximation is almost exact when ReLU is used as discussed above. The right hand side of Equation ( 3) is interpreted as the sum over subnetworks or paths in the sense of Veit et al. (2016) . The identity path corresponding to σ c = {∅} gives x 1 in the first term. If there is only one element in σ c , such that its cardinality |σ c | = 1, the product on the right hand side in parentheses is absent and only terms proportional to F c(1) appears in the sum, where c(1) ∈ {1, . . . , K}. We provide the proof of Equation ( 3) in Appendix A.

We can make the equation simpler, solely for simplicity, by setting all weights to be the same such that F c(i) = F and W * c(i) = W * for all i,

EQUATION

The binomial coefficients appear because the number of subsets of S K with cardinality k is K k . Note that the implementations of our proposed method (described in Section 3) do not use this simplification.

Exponential gradient scaling Similarly, one observes that the gradient is the sum from all subnetwork contributions, including the identity network. The magnitudes of subnetwork gradients for an 110 layer ResNet have been measured by (Veit et al., 2016) . If one takes F to have ReLU nonlinearity, then F (x, W * ) = 0 except at the origin. The non-trivial gradient can be expressed almost exactly as K k (F (x, W * )) k . This validates the numerical results that the gradient norm decreases exponentially with subnetwork depth as reported in (Veit et al., 2016) . Their experimental results indicate that the average gradient norm for each subnetwork of depth k is given by ||F (x, W * )|| k .

All aforementioned properties apply only after the ResNets are trained. However, if an approximation in the network is made, it would still give similar results after training. We show in the following sections that our network can attain similar performances as the original ResNet, validating our approximation.

3. WARPED RESIDUAL NETWORK:

The Warped Residual Network (WarpNet) is an approximation to the residual network, where K consecutive residual units are compressed into one warp layer. The computation in a warp layer is different from that in a conventional neural network. It uses a warp operator to compute the output (i.e., x K+1 ) of the layer directly from the input (i.e., x 1 ), as shown in Equation ( 4). The number of weights in a warped layer is the same of the one in the original residual network for K consecutive residual units. For instance, the weights W 1 , W 2 up to W K are present in a warped layer. But these weights can be used and updated in parallel due to the use of the warp operator. Below we first describe the forward and backward propagation rules used in warped residual network.

3.1. FORWARD PROPAGATION ACROSS WARP OPERATORS:

This section shows the propagation rules of the Warped Residual Network using the warp operator T warp .

The expression for T warp is derived from Equation 3, that is, by using the Taylor series expansion to the first order:

x K+1 = x 1 + σc∈P(S K )\{∅} k i=2 F c(i) (x 1 , W * c(i) ) F c(1) (x 1 , W * c(1) ),

Note that T warp can be calculated in a parallelizable manner for all K. This is shown in Figure 1 with K = 2, where3 

EQUATION

and W i corresponds to the weights in the i-th residual unit in the original ResNet. The formula for the K = 3 case is shown in Appendix A.

3.2. WARPED BACK-PROPAGATION:

Now we derive the backpropagation rules. Suppose that the upstream gradient ∂L/∂x 5 is known and we wish to compute ∂L/∂W 1 for gradient descent. We first back propagate the gradient down from x 5 to x 3 . With x 5 = T warp (x 3 ), we can derive the backpropagated gradient

∂L ∂x 3 = ∂L ∂x 5 • ∂x 5 ∂x 3 = ∂L ∂x 5 • I + ∂F 3 (x 3 , W 3 ) ∂x 3 + ∂F 4 (x 3 , W 4 ) ∂x 3 + ∂F 4 (x 3 , W 4 ) ∂x 3 ∂F 3 (x 3 , W 3 ) ∂x 3 ,

where I is the identity matrix and we have set the derivative of F 4 to zero for ReLU non-linearities. Note that we have removed all BN layers from F 4 in our implementation. One sees that the same kind of parallelism in the warp operator is also present for back propagation. Now we can evaluate the weight gradient for updates

∂L ∂W 1 = ∂L ∂x 3 • ∂x 3 ∂W 1 = ∂L ∂x 3 • ∂F 1 (x 1 , W 1 ) ∂W 1 + ∂F 2 (x 1 , W 2 ) ∂x 1 ∂F 1 (x 1 , W 1 ) ∂W 1 .

Similarly for the update rule for W 2 . Rules for the all other weights in WarpNet can be obtained in the same way,

∂L ∂W 2 = ∂L ∂x 3 • ∂x 3 ∂W 2 = ∂L ∂x 3 • ∂F 2 (x 1 , W 2 ) ∂W 2 + ∂ 2 F 2 (x 1 , W 2 ) ∂x 1 ∂W 2 F 1 (x 1 , W 1 ) .

The weights W 1 and W 2 can be updated in parallel independently. The derivative ∂F 2 (x 1 , W 2 )/∂x 1 (in ∂L/∂W 1 ) is already computed in the forward pass which could be saved and reused. Furthermore, derivatives other than F 3 needed in ∂L/∂x 3 can also be computed in the forward pass. For higher warp factors K, only the derivative F K+1 is not available after the forward pass. 

4. EXPERIMENTS:

In this section we discuss our implementation of the WarpNet architecture and the experimental results. In order to ensure the validity of the series expansion we replace the 1 × 1 convolution layers on skip connections by an average pooling layer and a concatenate layer before the residual unit to reduce the spatial dimensions of feature maps and multiply their channels. In this way all skip connections are identity mappings. We adopt a wide residual architecture (WRN) (Zagoruyko & Komodakis, 2016) . The convolution blocks F comprised of the following layers, from input to output, BN-Conv-BN-ReLU-Conv-BN (Han et al., 2016) . The neural architecture of Warped Residual Networks is shown in Table 1 . The layers [T

(K) warp ] ×N warp represent forward propagating N warp times, such that x i+K = T (K) warp (x i , W i , W i+1 , . . . , W i+K-1

) and the indices i correspond to the indices in the original residual network.

Using Tensorflow, we implemented a WarpNet with various parameters, k w , K and N warp . The widening factor (Zagoruyko & Komodakis, 2016 ) is k w , K is the warp factor and with the scheme shown in Figure 1 . We employ Tensorflow's automatic differentiation for backpropagation, where the gradients are calculated by sweeping through the network through the chain rule. Although the gradients computed in the forward pass can be re-used in the backward pass, we do not do so in our experiment and leave it to future work to potentially further speed up our method. Even so, the experimental results indicate that WarpNet can be trained faster than WRN with comparable predictive accuracy.

Consider the case K = 2, we found that the computation bottleneck arises from the BN layers in F 2 . The reason being the gradient of BN layers contains an averaging operation that is expensive to compute. In our final implementation we removed all BN layers in F 2 from our network. This results in a departure from our series approximation but it turns out the network still trains well. This is because the normalizing layers are still being trained in F 1,2 . To further improve the speed-up we replace the F 1 block in the derivative term F 2 F 1 with the input x 1 so that the term becomes F 2 x 1 . Similar approximations are made in cases where K > 2. We have conducted extensive experiments of this modification and found that it has similar predictive accuracies while improving speed-up. In the following, we refer to this modification of WarpNet as WarpNet1 and the one with F 2 F 1 as WarpNet2. For K = 3 we replace all F j F i by F j x 1 in WarpNet1. We also drop the term F 3 F 2 F 1 in computing x 4 in both versions of WarpNet due to the limited GPUs we have in the expriements.

To investigate the speed-up provided by WarpNet and its predictive performance with various approximations on the warp operators, we define the relative speed-up, RS, compared to the corresponding wide residual network (WRN) as

RS = t res -t warp t res ,

where t warp is the total time to process a batch for WarpNet during training, and t res is that for the baseline WRN. We study the performance of WarpNet with K = 2 and K = 3. The averaged results over two runs each are shown in Tables 2 and 3 . The first column in the Tables represents the methods including two versions of WarpNet with different modifications we have made in the warp operator. A wide ResNet (WRN) is obtained by replacing [T

(K):

warp ] ×N warp with K × N warp residual units in Table 1 . The total number of convolution layers (represented by n in WRN-n-k w ) is 6KN warp + 1, where the factor of 6 arise from two convolution layers in each residual unit and 3 stages in the network, plus 1 convolution layer at the beginning. The number of layers in WRN is always odd as we do not use the 1 × 1-convolution layer across stages.

We see that in most cases, WarpNet can achieve similar, if not better, validation errors than the corresponding wide ResNet while offering speed-up. The experiments also show that the modification of replacing F F by F x 1 , where x 1 is the input of the warp operator, achieves better accuracy most of the time while improving the speed-up. We observe that increasing from K = 2 to K = 3, using only one more GPU, significantly improves speed-up with only a slight drop in validation accuracy compared to the K = 2 case.

We have also performed experiments on the speed-up as the widening factor k w increases. We found that the speed-up increases as the WarpNet gets wider. For k w = 4, 8 and 16, the speed-up in total time for K = 2 is 35%, 40% and 42% respectively. The speed-up also increases with the warp factor K, for K = 3 using the F x modification, the speed-ups are 44%, 48% and 50% respectively.

4.2. RESULTS ON IMAGENET:

We also tested WarpNet on a down-sampled (32x32) ImageNet data set (Chrabaszcz & Hutter, 2017) . The data set contains 1000 classes with 1281167 training images and 50000 validation images with 50 images each class. The training batch size is 512, initial learning rate is 0.4 and drops by a factor of 0.1 at every 30 epochs. The weight decay is set to be 0.0001. We use the overall best performing warp operator in the CIFAR experiments, namely, the one containing F x.

The results are shown in Table 4 and Figure 2 . First, we show directly that for a given ResNet there exists a WarpNet that obtains a higher validation accuracy with shorter training time. We increase K from 2 to 3 and keep everything else fixed. This corresponds to WarpNet-109-2. The network has more residual units than WRN-73-2. We observed that WarpNet-109-2 trains 12% faster than WRN-73-2 while resulting in a better validation accuracy. Second, WarpNet can achieve close to the benchmark validation error of 18.9% with WRN-28-10 in (Chrabaszcz & Hutter, 2017) . Note that we were not able to train the corresponding WRN-73-4 on the dataset as the model requires too much memory on a single GPU. This shows that the weight distribution of WarpNet across GPUs for which we divide each batch into 2 or 4 mini-batches, respectively, and synchronization is done right after all GPUs finish their job on their mini-batch to avoid harming the accuracy. Table 5 shows the average result over 2 runs for each method. All methods see the same volume of data during training, which means that the number of epochs is the same for all methods. We chose the warp operators containing F x in this experiment, that is, WarpNet1 whose operations are specified in the first rows of each block in Tables 2 and 3 . We use the GPU assignment

[F 1 ], [F 2 , F 2 x 1 ], [F 3 , F 3 x 1 ]

for the case with 3 GPUs.

The results show that WarpNet is more accurate than data parallelism in both 2-GPU and 4-GPU cases. When 3 or 4 GPUs are used, WarpNet is much faster than data-parallelized ResNet with 4 GPUs. We believe this is because the data parallelism method needs to store all the weights of the model in all GPUs and its speed is slowed by the need to update all the weights across all GPUs at the time of synchronization. In comparison, WarpNet splits the weights among GPUs and each GPU only maintains and updates a subset of weights. Such weight distributions in WarpNet require less GPU memory, which allows it to train larger networks. Furthermore, data parallelism can be applied to WarpNet as well to potentially further speed up WarpNet, which is a topic beyond the scope of this paper.

5. CONCLUDING REMARKS:

In this paper, we proposed the Warped Residual Network (WarpNet) that arises from the first order Taylor series expansion with ReLU non-linearity. We showed analytically that the first order expansion is sufficient to explain the ensemble behaviors of residual networks (Veit et al., 2016) . The Taylor series approximation has the structure that allows WarpNet to train consecutive residual units in parallel while ensuring that the performance is similar to the corresponding ResNet. The weights of different residual units are distributed over the vairous GPUs which enables the training of bigger networks compared to ResNets given limited GPU memory. Experimental results show that Warp-Net can provide a significant speed-up over wide ResNets with similar predictive accuracy, if not better. We also show that WarpNet outperforms a data parallelism method on ResNet, achieving better predictive accuracies and a much better speed up when more than 2 GPUs are used.

power k + 1. There are two sources of such terms. First, those generated by an additional index reduction after K operations and the zeroth order Taylor expansion in terms of power k + 1, there are C(K, k + 1) such terms. Second, those generated by the first order Taylor expansion in terms of power k, there are C(K, k) such terms. Therefore the total number of terms with power k + 1 after K + 1 index reductions is

C(K + 1, k + 1) = C(K, k + 1) + C(K, k).

This is precisely the recursion formula satisfied by the binomial coefficients. We have explicitly shown earlier that for K = 3 and K = 4 the coefficients are binomial coefficients. Therefore the number of terms at any K and power k are the binomial coefficients, C(K, k) = K k . Note that the order of the indices in [Π K i=2 F c(i) ]F c(1) must be ordered, such that c(k) > c(k -1) > • • • > c(1). Where the indices c(i, k) are given by any subset of the integers S K = {1, 2, . . . , K}. Of course, the number of unordered subsets with cardinality k from a set of cardinality K is K k . To write down a term of power k explicitly in the first order Taylor expansion, we first choose a unordered subset of k indices from S K then we order the indices to form σ c = {c(k), . . . , c(1)}. Then the output after K residual units with input x i is the sum over all these subsets

EQUATION

where P(S K ) denotes the power set of S K . Note that when σ c is empty, the right hand side gives the identity mapping. This is the same as Equation ( 3). Setting all weights to be the same gives the form in Equation 4.

A.2 SECOND PROOF OF THEOREM 1, BERNOULLI PROCESS

The series of index reduction operations can be identified with a Bernoulli process with parameters K and p = 0.5. Each term in Equation (3) arises from a realization of the Bernoulli process. Summing over terms from all possible realizations results in Equation (3). Recall that to express x K+1 in terms of x 1 similar to Equation (3), we need K index reduction operations. Let X K:1 := {X K , X K-1 , . . . , X 1 } be a Bernoulli process, where X i ∼ B(K, p = 0.5). Then the realizations X i = 0 represents the power of a term remains the same after an index reduction, and X i = 1 denotes an increase in the power of a term by one. For example, consider K = 2, the terms corresponding to the realizations of the Bernoulli process X 2:1 = {X 2 , X 1 } are

{0, 0} → x 1 {1, 0} → F 2 {0, 1} → F 1 {1, 1} → F 2 F 1

One sees that x 3 can be obtained by summing over all terms corresponding to all realizations of X 3:1 . This generalizes to X K:1 for x K+1 . The probability of a term having power k is 2 -K K k . Since the total number of terms is 2 K , the number of terms having power k is the binomial coefficient K k . If we let σ c to be the term corresponding to a realization of X K:1 , then consecutive Taylor expansions corresponds to summing over all σ c and Equation (3) follows.

:

The Taylor expansion for multivariate functions is f(x + a) = f (x) + a • ∇xf (x) + • • •

Batch normalization layers are non-linear during training, due to the scaling by the sample variance.

This equation is from Equation (2) after dropping the negligible O( 2 ) terms.

Figures and Tables:

Figure 1: Forward propagation in WarpNet. Left, the original residual network where each block corresponds to a residual unit. Middle, the corresponding Warped Network with indices determined by the architecture of M . The diagram on the right shows the parallelism allowed in the forward pass of WarpNet. Note that F 2 shares the same weights with F 2 , as they come from the same residual unit in the original network M .

Figure 2: Top-5 validation error of the down-sampled ImageNet data set for k w = 2.

Network architecture of WarpNet.For the CIFAR-10 and CIFAR-100 data sets, we trained for 80000 iterations, or 204 epochs. We took a training batch size of 128. Initial learning rate is 0.1. The learning rate drops by a factor of 0.1 at epochs 60, 120, and 160, with a weight decay of 0.0005. We use common data augmentation techniques, namely, whitening, flipping and cropping.

Table content: <table><tr><td>H × W × C</td><td>Conv blocks</td></tr><tr><td>32 × 32 × 3</td><td>Input</td></tr><tr><td>32 × 32 × 16</td><td>Conv-BN-ReLU</td></tr><tr><td>Stage 1</td><td>Concatenate</td></tr><tr><td colspan="2">32 × 32 × 16k w [T warp ] ×N warp (K)</td></tr><tr><td>Stage 2</td><td>Avg pool(2x2)-Concatenate</td></tr><tr><td colspan="2">16 × 16 × 32k w [T warp ] ×N warp (K)</td></tr><tr><td>Stage 3</td><td>Avg pool(2x2)-Concatenate</td></tr><tr><td>8 × 8 × 64k w</td><td>[T warp ] ×N warp (K)</td></tr><tr><td>1 × 1 × 64k w</td><td>BN-ReLU-Average Pooling(8,8)</td></tr><tr><td/><td>Fully Connected(# of classes)</td></tr><tr><td colspan="2">4.1 RESULTS ON CIFAR-10 AND CIFAR-100</td></tr></table>

Comparison with data parallelism.

Table content: <table><tr><td/><td colspan="3"># of GPUs Validation error (%) Relative Speed-up</td></tr><tr><td>WRN-73-4</td><td>1</td><td>4.80</td><td>0%</td></tr><tr><td>WRN-73-4</td><td>2</td><td>4.86</td><td>29.6%</td></tr><tr><td>WRN-73-4</td><td>4</td><td>4.67</td><td>32.4%</td></tr><tr><td colspan="2">WarpNet1-73-4, K = 2, N warp = 6 2</td><td>4.51</td><td>23.0%</td></tr><tr><td colspan="2">WarpNet1-73-4, K = 3, N warp = 4 3</td><td>4.66</td><td>44.1%</td></tr><tr><td colspan="2">WarpNet1-73-4, K = 3, N warp = 4 4</td><td>4.66</td><td>44.6%</td></tr></table>

Peer review (iclr2018):
 The main contribution of this paper is a particular Taylor expansion of the outputs of a ResNet which is shown to be exact at almost all points in the input space. This expression is used to develop a new layer called a “warp layer” which essentially tries to compute several layers of the residual network using the Taylor expansion expression — however in this expression, things can be done in parallel, and interestingly, the authors show that the gradients also decouple when the (ResNet) model is close to a local minimum in a certain sense, which may motivate the decoupling of layers to begin with. Finally the authors stack these warp layers to create a “warped resnet” which they show does about as well as an ordinary ResNet but has better parallelization properties. To me the analytical parts of the paper are the most interesting, particularly in showing how the gradients approximately decouple. However there are several weaknesses to the paper (or maybe just things I didn’t understand). First, a major part of the paper tries to make the case that there is a symmetry breaking property of the proposed model, which I am afraid I simply was not able to follow. Some of the notation is confusing here — for example, presumably the rotations refer to image level rotations rather than literally multiplying the inputs by an orthogonal matrix, which the notation suggests to be the case. It is also never precisely spelled out what the final theoretical guarantee is (preferably the authors would do this in the form of a proposition or theorem). Throughout, the authors write out equations as if the weights in all layers are equal, but this is confusing even if the authors say that this is what they are doing, since their explanation is not very clear. The confusion is particularly acute in places where derivatives are taken, because the derivatives continue to be taken as if the weights were untied, but then written as if they happened to be the same. Finally the experimental results are okay but perhaps a bit preliminary. I have a few recommendations here: * It would be stronger to evaluate results on a larger dataset like ILSVRC. * The relative speed-up of WarpNet compared to ResNet needs to be better explained — the authors break the computation of the WarpNet onto two GPUs, but it’s not clear if they do this for the (vanilla) ResNet as well. In batch mode, the easiest way to parallelize is to have each GPU evaluate half the batch. Even in a streaming mode where images need to be evaluated one by one, there are ways to pipeline execution of the residual blocks, and I do not see any discussion of these alternatives in the paper. * In the experimental results, K is set to be 2, and the authors only mention in passing that they have tried larger K in the conclusion. It would be good to have a more thorough experimental evaluation of the trade-offs of setting K to be higher values. A few remaining questions for the authors: * There is a parallel submission (presumably by different authors called “Residual Connections Encourage Iterative Inference”) which contains some related insights. I wonder what are the differences between the two Taylor expansions, and whether the insights of this paper could be used to help the other paper and vice versa? * On implementation - the authors mention using Tensorflow’s auto-differentiation. My question here is — are gradients being re-used intelligently as suggested in Section 3.1? * I notice that the analysis about the vanishing Hessian could be applied to most of the popular neural network architectures available now. How much of the ideas offered in this paper would then generalize to non-resnet settings?