Title: BEYOND FINITE LAYER NEURAL NETWORKS: BRIDGING DEEP ARCHITECTURES AND NUMERICAL DIFFERENTIAL EQUATIONS

Abstract: Deep neural networks have become the state-of-the-art models in numerous machine learning tasks. However, general guidance to network architecture design is still missing. In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters. In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress (> 50%) the original networks while maintaining a similar performance. This can be explained mathematically using the concept of modified equation from numerical analysis. Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture. As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

1. INTRODUCTION:

Deep learning has achieved great success in may machine learning tasks. The end-to-end deep architectures have the ability to effectively extract features relevant to the given labels and achieve state-of-the-art accuracy in various applications (Bengio, 2009) . Network design is one of the central task in deep learning. Its main objective is to grant the networks with strong generalization power using as few parameters as possible. The first ultra deep convolutional network is the ResNet (He et al., 2015b) which has skip connections to keep feature maps in different layers in the same scale and to avoid gradient vanishing. Structures other than the skip connections of the ResNet were also introduced to avoid gradient vanishing, such as the dense connections (Huang et al., 2016a) , fractal path (Larsson et al., 2016) and Dirac initialization (Zagoruyko & Komodakis, 2017) . Furthermore, there has been a lot of attempts to improve the accuracy of image classifications by modifying the residual blocks of the ResNet. Zagoruyko & Komodakis (2016) suggested that we need to double the number of layers of ResNet to achieve a fraction of a percent improvement of accuracy. They proposed a widened architecture that can efficiently improve the accuracy. Zhang et al. (2017) pointed out that simply modifying depth or width of ResNet might not be the best way of architecture design. Exploring structural diversity, which is an alternative dimension in network design, may lead to more effective networks. In Szegedy et al. (2017) , Zhang et al. (2017) , Xie et al. (2017) , Li et al. (2017) and Hu et al. (2017) , the authors further improved the accuracy of the networks by carefully designing residual blocks via increasing the width of each block, changing the topology of the network and following certain empirical observations. In the literature, the network design is mainly empirical.It remains a mystery whether there is a general principle to guide the design of effective and compact deep networks.

Observe that each residual block of ResNet can be written as u n+1 = u n + ∆tf (u n ) which is one step of forward Euler discretization (AppendixA.1) of the ordinary differential equation (ODE) u t = f (u) (E, 2017) . This suggests that there might be a connection between discrete dynamic systems and deep networks with skip connections. In this work, we will show that many state-of-the-art deep network architectures, such as PolyNet (Zhang et al., 2017) , FractalNet (Larsson et al., 2016) and RevNet (Gomez et al., 2017) , can be consider as different discretizations of ODEs. From the perspective of this work, the success of these networks is mainly due to their ability to efficiently approximate dynamic systems. On a side note, differential equations is one of the most powerful tools used in low-level computer vision such as image denoising, deblurring, registration and segmentation (Osher & Paragios, 2003; Aubert & Kornprobst, 2006; Chan & Shen, 2005) . This may also bring insights on the success of deep neural networks in low-level computer vision. Furthermore, the connection between architectures of deep neural networks and numerical approximations of ODEs enables us to design new and more effective deep architectures by selecting certain discrete approximations of ODEs. As an example, we design a new network structure called linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method in numerical ODEs (Ascher & Petzold, 1997) . This architecture can be applied to any ResNet-like networks. In this paper, we apply the LM-architecture to ResNet and ResNeXt (Xie et al., 2017) and achieve noticeable improvements on CIFAR and ImageNet with comparable numbers of trainable parameters. We also explain the performance gain using the concept of modified equations from numerical analysis.

It is known in the literature that introducing randomness by injecting noise to the forward process can improve generalization of deep residual networks. This includes stochastic drop out of residual blocks (Huang et al., 2016b) and stochastic shakes of the outputs from different branches of each residual block (Gastaldi, 2017) . In this work we show that any ResNet-like network with noise injection can be interpreted as a discretization of a stochastic dynamic system. This gives a relatively unified explanation to the stochastic learning process using stochastic control. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the proposed LM-architecture. As an example, we introduce stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

1.1. RELATED WORK:

The link between ResNet (Figure 1 (a)) and ODEs were first observed by E (2017), where the authors formulated the ODE u t = f (u) as the continuum limit of the ResNet u n+1 = u n + ∆tf (u n ). Liao & Poggio (2016) bridged ResNet with recurrent neural network (RNN), where the latter is known as an approximation of dynamic systems. Sonoda & Murata (2017) and Li & Shi (2017) also regarded ResNet as dynamic systems that are the characteristic lines of a transport equation on the distribution of the data set. Similar observations were also made by Chang et al. (2017) ; they designed a reversible architecture to grant stability to the dynamic system. On the other hand, many deep network designs were inspired by optimization algorithms, such as the network LISTA (Gregor & LeCun, 2010) and the ADMM-Net (Yang et al., 2016) . Optimization algorithms can be regarded as discretizations of various types of ODEs (Helmke & Moore, 2012) , among which the simplest example is gradient flow.

Another set of important examples of dynamic systems is partial differential equations (PDEs), which have been widely used in low-level computer vision tasks such as image restoration. There were some recent attempts on combining deep learning with PDEs for various computer vision tasks, i.e. to balance handcraft modeling and data-driven modeling. Liu et al. (2010) and Liu et al. (2013) proposed to use linear combinations of a series of handcrafted PDE-terms and used optimal control methods to learn the coefficients. Later, Fang et al. (2017) extended their model to handle classification tasks and proposed an learned PDE model (L-PDE). However, for classification tasks, the dynamics (i.e. the trajectories generated by passing data to the network) should be interpreted as the characteristic lines of a PDE on the distribution of the data set. This means that using spatial differential operators in the network is not essential for classification tasks. Furthermore, the discretizations of differential operators in the L-PDE are not trainable, which significantly reduces the network's expressive power and stability. Chen et al. (2015) proposed a feed-forward network in order to learn the optimal nonlinear anisotropic diffusion for image denoising. Unlike the previous work, their network used trainable convolution kernels instead of fixed discretizations of differential operators, and used radio basis functions to approximate the nonlinear diffusivity function. More recently, Long et al. (2017) designed a network, called PDE-Net, to learn more general evolution PDEs from sequential data. The learned PDE-Net can accurately predict the dynamical behavior of data and has the potential to reveal the underlying PDE model that drives the observed data.

In our work, we focus on a different perspective. First of all, we do not require the ODE u t = f (u) associate to any optimization problem, nor do we assume any differential structures in f (u). The optimal f (u) is learned for a given supervised learning task. Secondly, we draw a relatively comprehensive connection between the architectures of popular deep networks and discretization schemes of ODEs. More importantly, we demonstrate that the connection between deep networks and numerical ODEs enables us to design new and more effective deep networks. As an example, we introduce the LM-architecture to ResNet and ResNeXt which improves the accuracy of the original networks.

We also note that, our viewpoint enables us to easily explain why ResNet can achieve good accuracy by dropping out some residual blocks after training, whereas dropping off sub-sampling layers often leads to an accuracy drop (Veit et al., 2016) . This is simply because each residual block is one step of the discretized ODE, and hence, dropping out some residual blocks only amounts to modifying the step size of the discrete dynamic system, while the sub-sampling layer is not a part of the ODE model. Our explanation is similar to the unrolled iterative estimation proposed by Greff et al. (2016) , while the difference is that we believe it is the data-driven ODE that does the unrolled iterative estimation.

2. NUMERICAL DIFFERENTIAL EQUATION, DEEP NETWORKS AND BEYOND:

In this section we show that many existing deep neural networks can be consider as different numerical schemes approximating ODEs of the form u t = f (u). Based on such observation, we introduce a new structure, called the linear multi-step architecture (LM-architecture) which is inspired by the well-known linear multi-step method in numerical ODEs. The LM-architecture can be applied to any ResNet-like networks. As an example, we apply it to ResNet and ResNeXt and demonstrate the performance gain of such modification on CIFAR and ImageNet data sets. Zhang et al. (2017) , introduced a PolyInception module in each residual block to enhance the expressive power of the network. The PolyInception model includes polynomial compositions that can be described as

2.1. NUMERICAL SCHEMES AND NETWORK ARCHITECTURES:

PolyNet (Figure 1(b)), proposed by

(I + F + F 2 ) • x = x + F (x) + F (F (x)).

We observe that PolyInception model can be interpreted as an approximation to one step of the backward Euler (implicit) scheme (AppendixA.1):

u n+1 = (I -∆tf ) -1 u n .

Indeed, we can formally rewrite (I -∆tf ) -1 as

I + ∆tf + (∆tf ) 2 + • • • + (∆tf ) n + • • • .

Therefore, the architecture of PolyNet can be viewed as an approximation to the backward Euler scheme solving the ODE u t = f (u). Note that, the implicit scheme allows a larger step size (Ascher & Petzold, 1997) , which in turn allows fewer numbers of residual blocks of the network. This explains why PolyNet is able to reduce depth by increasing width of each residual block to achieve state-of-the-art classification accuracy.

FractalNet (Larsson et al., 2016) (Figure 1(c) ) is designed based on self-similarity. It is designed by repeatedly applying a simple expansion rule to generate deep networks whose structural layouts are truncated fractals. We observe that, the macro-structure of FractalNet can be interpreted as the well-known Runge-Kutta scheme in numerical analysis. Recall that the recursive fractal structure of FractalNet can be written as

f c+1 = 1 2 k c * + 1 2 f c • f c .

For simplicity of presentation, we only demonstrate the FractalNet of order 2 (i.e. c ≤ 2). Then, every block of the FractalNet (of order 2) can be expressed as

x n+1 = k 1 * x n + k 2 * (k 3 * x n + f 1 (x n )) + f 2 (k 3 * x n + f 1 (x n

), which resembles the Runge-Kutta scheme of order 2 solving the ODE u t = f (u) (see Ap-pendixA.2).

RevNet(Figure 1(d) ), proposed by Gomez et al. (2017) , is a reversible network which does not require to store activations during forward propagations. The RevNet can be expressed as the following discrete dynamic system

X n+1 = X n + f (Y n ), Y n+1 = Y n + g(X n+1 ).

RevNet can be interpreted as a simple forward Euler approximation to the following dynamic system

Ẋ = f 1 (Y ), Ẏ = f 2 (X).

Note that reversibility, which means we can simulate the dynamic from the end time to the initial time, is also an important notation in dynamic systems. There were also attempts to design reversible scheme in dynamic system such as Nguyen & Mcmechan (2015) . 

u t = f (u) Runge-Kutta scheme RevNet Ẋ = f 1 (Y ), Ẏ = f 2 (X) Forward Euler scheme

2.2. LM-RESNET: A NEW DEEP ARCHITECTURE FROM NUMERICAL DIFFERENTIAL EQUATION:

We have shown that architectures of some successful deep neural networks can be interpreted as different discrete approximations of dynamic systems. In this section, we proposed a new struc-ture, called linear multi-step structure (LM-architecture), based on the well-known linear multi-step scheme in numerical ODEs (which is briefly recalled in Appendix A.3). The LM-architecture can be written as follows

EQUATION

where k n ∈ R is a trainable parameter for each layer n. A schematic of the LM-architecture is presented in Figure 2 . Note that the midpoint and leapfrog network structures in Chang et al. (2017) are all special case of ours. The LM-architecture is a 2-step method approximating the ODE u t = f (u). Therefore, it can be applied to any ResNet-like networks, including those mentioned in the previous section. As an example, we apply the LM-architecture to ResNet and ResNeXt. We call these new networks the LM-ResNet and LM-ResNeXt. We trained LM-ResNet and LM-ResNeXt on CIFAR (Krizhevsky & Hinton, 2009) and Imagenet (Russakovsky et al., 2014) , and both achieve improvements over the original ResNet and ResNeXt.

Implementation Details. For the data sets CIFAR10 and CIFAR100, we train and test our networks on the training and testing set as originally given by the data set. For ImageNet, our models are trained on the training set with 1.28 million images and evaluated on the validation set with 50k images. On CIFAR, we follow the simple data augmentation in Lee et al. (2015) for training: 4 pixels are padded on each side, and a 32×32 crop is randomly sampled from the padded image or its horizontal flip. For testing, we only evaluate the single view of the original 32×32 image. Note that the data augmentation used by ResNet (He et al., 2015b; Xie et al., 2017) is the same as Lee et al. (2015) . On ImageNet, we follow the practice in Krizhevsky et al. (2012) ; Simonyan & Zisserman (2014) . Images are resized with its shorter side randomly sampled in [256, 480] for scale augmentation (Simonyan & Zisserman, 2014) . The input image is 224×224 randomly cropped from a resized image using the scale and aspect ratio augmentation of Szegedy et al. (2015) . For the experiments of ResNet/LM-ResNet on CIFAR, we adopt the original design of the residual block in He et al. (2016) , i.e. using a small two-layer neural network as the residual block with bn-relu-conv-bn-reluconv. The residual block of LM-ResNeXt (as well as LM-ResNet164) is the bottleneck structure used by (Xie et al., 2017 ) that takes the form

1 × 1, 64 3 × 3, 64 1 × 1, 256

. We start our networks with a single We will justify this mathematically at the end of this section using the concept of modified equations from numerical analysis.

Explanation on the performance boost via modified equations. Given a numerical scheme approximating a differential equation, its associated modified equation (Warming & Hyett, 1974 ) is another differential equation to which the numerical scheme approximates with higher order of accuracy than the original equation. Modified equations are used to describe numerical behaviors of numerical schemes. For example, consider the simple 1-dimensional transport equation u t = cu x . Both the Lax-Friedrichs scheme and Lax-Wendroff scheme approximates the transport equation. However, the associated modified equations of Lax-Friedrichs and Lax-Wendroff are

u t -cu x = ∆x 2 2∆t (1 -r 2 )u xx and u t -cu x = c∆x 2 6 (r 2 -1)u xxx

respectively, where r = 2∆t ∆x . This shows that the Lax-Friedrichs scheme behaves diffusive, while the Lax-Wendroff scheme behaves dispersive. Consider the forward Euler scheme which is associated to ResNet, un+1-un

∆t = f (u n ). Note that u n+1 -u n ∆t = un + 1 2 ün ∆t + 1 6 ... u n ∆t 2 + O(∆t 3 ).

Thus, the modified equation of forward Euler scheme reads as

EQUATION

Consider the numerical scheme used in the LM-structure un+1-(1-kn)un-knun-1 ∆t = f (u n ). By Taylor's expansion, we have

u n+1 -(1 -k n )u n -k n u n-1 ∆t = (u n + ∆t un + 1 2 ∆t 2 ün + O(∆t 3 ))) -(1 -k n )u n -k n (u n -∆t un + 1 2 ∆t 2 ün + O(∆t 3 ))) ∆t = (1 + k n ) un + 1 -k n 2 ∆tü n + O(∆t 2 ).

Then, the modified equation of the numerical scheme associated to the LM-structure

EQUATION

Comparing ( 2) with (3), we can see that when k n ≤ 0, the second order term ü of ( 3) is bigger than that of (2). The term ü represents acceleration which leads to acceleration of the convergence of u n when f = -∇g Su & Boyd (2015) ; Wilson et al. (2016) . When f (u) = L(u) with L being an elliptic operator, the term ü introduce dispersion on top of the dissipation, which speeds up the flow of u n (Dong et al., 2017) . In fact, this is our original motivation of introducing the LM-architecture (1). Note that when the dynamic is truly a gradient flow, i.e. f = -∇g, the difference equation of the LM-structure has a stability condition -1 ≤ k n ≤ 1. In our experiments, we do observe that most of the coefficients are lying in (-1, 1) (Figure 4 ). Moreover, the network is indeed accelerating at the end of the dynamic, for the learned parameters {k n } are negative and close to -1 (Figure 4 ). 

3. STOCHASTIC LEARNING STRATEGY: A STOCHASTIC DYNAMIC SYSTEM PERSPECTIVE:

Although the original ResNet (He et al., 2015b) did not use dropout, several work (Huang et al., 2016b; Gastaldi, 2017) showed that it is also beneficial to inject noise during training. In this section we show that we can regard such stochastic learning strategy as an approximation to a stochastic dynamic system. We hope the stochastic dynamic system perspective can shed lights on the discovery of a guiding principle on stochastic learning strategies. To demonstrate the advantage of bridging stochastic dynamic system with stochastic learning strategy, we introduce stochastic depth during training of LM-ResNet. Our results indicate that the networks with proposed LM-architecture can also greatly benefit from stochastic learning strategies.

3.1. NOISE INJECTION AND STOCHASTIC DYNAMIC SYSTEMS:

As an example, we show that the two stochastic learning methods introduced in Huang et al. (2016b) and Gastaldi (2017) can be considered as weak approximations of stochastic dynamic systems.

Shake-Shake Regularization. Gastaldi (2017) introduced a stochastic affine combination of multiple branches in a residual block, which can be expressed as

X n+1 = X n + ηf 1 (X n ) + (1 -η)f 2 (X n ),

where η ∼ U(0, 1). To find its corresponding stochastic dynamic system, we incorporate the time step size ∆t and consider

EQUATION

which reduces to the shake-shake regularization when ∆t = 1. The above equation can be rewritten as

X n+1 = X n + ∆t 2 (f 1 (X n ) + f 2 (X n )) + √ ∆t(η - 1 2 )(f 1 (X n ) -f 2 (X n )).

Since the random variable (η-1 2 ) ∼ U(-1 2 , 1 2 ), following the discussion in Appendix B, the network of the shake-shake regularization is a weak approximation of the stochastic dynamic system

dX = 1 2 (f 1 (X) + f 2 (X))dt + 1 √ 12 (f 1 (X) -f 2 (X)) [1 N ×1 , 0 N,N -1 ]dB t ,

where dB t is an N dimensional Brownian motion, 1 N ×1 is an N -dimensional vector whose elements are all 1s, N is the dimension of X and f i (X), and denotes the pointwise product of vectors. Note from (4) that we have alternatives to the original shake-shake regularization if we choose ∆t = 1.

Stochastic Depth. Huang et al. (2016b) randomly drops out residual blocks during training in order to reduce training time and improve robustness of the learned network. We can write the forward propagation as

X n+1 = X n + η n f (X n ),

where P(η n = 1) = p n , P(η n = 0) = 1 -p n . By incorporating ∆t, we consider

X n+1 = X n + ∆tp n f (X n ) + √ ∆t η n -p n p n (1 -p n ) p n (1 -p n )f (X n ),

which reduces to the original stochastic drop out training when ∆t = 1. The variance of ηn-pn √ pn(1-pn)

is 1. If we further assume that (1 -2p n ) = O( √ ∆t), the condition(5) of Appendix B.2 is satisfied for small ∆t. Then, following the discussion in Appendix B, the network with stochastic drop out can be seen as a weak approximation to the stochastic dynamic system

dX = p(t)f (X)dt + p(t)(1 -p(t))f (X) [1 N ×1 , 0 N,N -1 ]dB t . Note that the assumption (1 -2p n ) = O( √ ∆t

) also suggests that we should set p n closer to 1/2 for deeper blocks of the network, which coincides with the observation made by Huang et al. (2016b, Figure 8) .

In general, we can interpret stochastic training procedures as approximations of the following stochastic control problem with running cost

min E X(0)∼data E(L(X(T )) + T 0 R(θ)) s.t. dX = f (X, θ)dt + g(X, θ)dB t

where L(•) is the loss function, T is the terminal time of the stochastic process, and R is a regularization term. In this section, we extend the stochastic depth training strategy to networks with the proposed LMarchitecture. In order to apply the theory of Itô process, we consider the 2nd order Ẍ + g(t) Ẋ = f (X) (which is related to the modified equation of the LM-structure (3)) and rewrite it as a 1st order ODE system Ẋ = Y, Ẏ = f (X) -g(t)Y.

Following a similar argument as in the previous section, we obtain the following stochastic process

Ẋ = Y, Ẏ = p(t)f (X)dt + p(t)(1 -p(t))f (X) [1 N ×1 , 0 N,N -1 ]dB t -g(t)Y dt,

which can be weakly approximated by

Y n+1 = X n+1 -X n ∆t , Y n+1 -Y n = ∆tp n f (X n ) + √ ∆t (η n -p n ) f (X n ) + g n Y n ∆t,

where P(η n = 1) = p n , P(η n = 0) = 1 -p n . Taking ∆t = 1, we obtain the following stochastic training strategy for LM-architecture

X n+1 = (2 + g n )X n -(1 + g n )X n-1 + η n f (X n ).

The above derivation suggests that the stochastic learning for networks using LM-architecture can be implemented simply by randomly dropping out the residual block with probability p.

Implementation Details. We test LM-ResNet with stochastic training strategy on CIFAR10. In our experiments, all hyper-parameters are selected exactly the same as in (Huang et al., 2016b) . The probability of dropping out a residual block at each layer is a linear function of the layer, i.e. we set the probability of dropping the current residual block as l L (1 -p L ), where l is the current layer of the network, L is the depth of the network and p L is the dropping out probability of the previous layer. In our experiments, we select p L = 0.8 for LM-ResNet56 and p L = 0.5 for LM-ResNet110.

During training with SGD, the initial learning rate is 0.1, and is divided by a factor of 10 after epoch 250 and 375, and terminated at 500 epochs. In addition, we use a weight decay of 0.0001 and a momentum of 0.9.

Results. Testing errors are presented in Table 4 . Training and testing curves of LM-ResNet with stochastic depth are plotted in Figure5. Note that LM-ResNet110 with stochastic depth training strategy achieved a 4.80% testing error on CIFAR10, which is even lower that the ResNet1202 reported in the original paper. The benefit of stochastic training has been explained from difference perspectives, such as Bayesian (Kingma et al., 2015) and information theory (Shwartz-Ziv & Tishby, 2017; Achille & Soatto, 2016) . The stochastic Brownian motion involved in the aforementioned stochastic dynamic systems introduces diffusion which leads to information gain and robustness.

4. CONCLUSION AND DISCUSSION:

In this paper, we draw a relatively comprehensive connection between the architectures of popular deep networks and discretizations of ODEs. Such connection enables us to design new and more effective deep networks. As an example, we introduce the LM-architecture to ResNet and ResNeXt which improves the accuracy of the original networks, and the proposed networks also outperform FractalNet and DenseNet on CIFAR100. In addition, we demonstrate that networks with stochastic training process can be interpreted as a weak approximation to stochastic dynamic systems. Thus, networks with stochastic learning strategy can be casted as a stochastic control problem, which we hope to shed lights on the discovery of a guiding principle on the stochastic training process. As an example, we introduce stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

As for our future work, if ODEs are considered as the continuum limits of deep neural networks (neural networks with infinite layers), more tools from mathematical analysis can be used in the study of neural networks. We can apply geometry insights, physical laws or smart design of numerical schemes to the design of more effective deep neural networks. On the other hand, numerical methods in control theory may inspire new optimization algorithms for network training. Moreover, stochastic control gives us a new perspective on the analysis of noise injections during network training. 

:



Figures and Tables:

Figure 1: Schematics of network architectures.

conv layer, followed by 3 residual blocks, global average pooling and a fully-connected classifier. The parameters k n of the LM-architecture are initialized by random sampling from U[-0.1, 0]. We initialize other parameters following the method introduced by He et al. (2015a). On CIFAR, we use SGD with a mini-batch size of 128, and 256 on ImageNet. During training, we apply a weight decay of 0.0001 for LM-ResNet and 0.0005 for LM-ResNeXt, and momentum of 0.9 on CIFAR. We apply a weight decay of 0.0001 and momentum of 0.9 for both LM-ResNet and LM-ResNeXt on ImageNet. For LM-ResNet on CIFAR10 (CIFAR100), we start with the learning rate of 0.1, divide it by 10 at 80 (150) and 120 (225) epochs and terminate training at 160 (300) epochs. For LM-ResNeXt on CIFAR, we start with the learning rate of 0.1 and divide it by 10 at 150 and 225 epochs, and terminate training at 300 epochs.

Figure 2: LM-architecture is an efficient structure that enables ResNet to achieve same level of accuracy with only half of the parameters on CIFAR10.

Figure 3: Training and testing curves of ResNext29 (16x64d, pre-act) and and LM-ResNet29 (16x64d, pre-act) on CIFAR100, which shows that the LM-ResNeXt can achieve higher accuracy than ResNeXt.

Figure 4: The trained parameters {k n } of LM-ResNet on CIFAR100.

Figure 5: Training and testing curves of ResNet56 (pre-act) and and LM-ResNet56 (pre-act) on CIFAR10 with stochastic depth.

In this table, we list a few popular deep networks, their associated ODEs and the numerical schemes that are connected to the architecture of the networks.

Table content: <table><tr><td>Network</td><td>Related ODE</td><td>Numerical Scheme</td></tr></table>

Comparisons of LM-ResNet/LM-ResNeXt with other networks on CIFAR Training and testing curves of LM-ResNeXt are plotted in Figure3. In Table2, we also present testing errors of FractalNet and DenseNet(Huang et al., 2016a)  on CIFAR 100. We can see that our proposed LM-ResNeXt29 has the best result. Comparisons between LM-ResNet and ResNet on ImageNet are presented in Table3. The LM-ResNet shows improvement over ResNet with comparable number of trainable parameters. Note that the results of ResNet on ImageNet are obtained from "https://github.com/KaimingHe/deep-residual-networks". It is worth noticing that the testing error of the 56-layer LM-ResNet is comparable to that of the 110-layer ResNet on CIFAR10; the testing error of the 164-layer LM-ResNet is comparable to that of the 1001-layer ResNet on CI-FAR100; the testing error of the 50-layer LM-ResNet is comparable to that of the 101-layer ResNet on ImageNet. We have similar results on LM-ResNeXt and ResNeXt as well. These results indicate that the LM-architecture can greatly compress ResNet/ResNeXt without losing much of the performance.

Table content: <table><tr><td>Model</td><td>Layer</td><td colspan="2">Error Params</td><td>Dataset</td></tr><tr><td>ResNet (He et al. (2015b))</td><td>20</td><td>8.75</td><td>0.27M</td><td>CIFAR10</td></tr><tr><td>ResNet (He et al. (2015b))</td><td>32</td><td>7.51</td><td>0.46M</td><td>CIFAR10</td></tr><tr><td>ResNet (He et al. (2015b))</td><td>44</td><td>7.17</td><td>0.66M</td><td>CIFAR10</td></tr><tr><td>ResNet (He et al. (2015b))</td><td>56</td><td>6.97</td><td>0.85M</td><td>CIFAR10</td></tr><tr><td>ResNet (He et al. (2016))</td><td>110, pre-act</td><td>6.37</td><td>1.7M</td><td>CIFAR10</td></tr><tr><td>LM-ResNet (Ours)</td><td>20, pre-act</td><td>8.33</td><td>0.27M</td><td>CIFAR10</td></tr><tr><td>LM-ResNet (Ours)</td><td>32, pre-act</td><td>7.18</td><td>0.46M</td><td>CIFAR10</td></tr><tr><td>LM-ResNet (Ours)</td><td>44, pre-act</td><td>6.66</td><td>0.66M</td><td>CIFAR10</td></tr><tr><td>LM-ResNet (Ours)</td><td>56, pre-act</td><td>6.31</td><td>0.85M</td><td>CIFAR10</td></tr><tr><td>ResNet (Huang et al. (2016b))</td><td>110, pre-act</td><td colspan="2">27.76 1.7M</td><td>CIFAR100</td></tr><tr><td>ResNet (He et al. (2016))</td><td>164, pre-act</td><td colspan="2">24.33 2.55M</td><td>CIFAR100</td></tr><tr><td>ResNet (He et al. (2016))</td><td>1001, pre-act</td><td colspan="3">22.71 18.88M CIFAR100</td></tr><tr><td>FractalNet (Larsson et al. (2016))</td><td>20</td><td colspan="2">23.30 38.6M</td><td>CIFAR100</td></tr><tr><td>FractalNet (Larsson et al. (2016))</td><td>40</td><td colspan="2">22.49 22.9M</td><td>CIFAR100</td></tr><tr><td>DenseNet (Huang et al., 2016a)</td><td>100</td><td colspan="2">19.25 27.2M</td><td>CIFAR100</td></tr><tr><td colspan="2">DenseNet-BC (Huang et al., 2016a) 190</td><td colspan="2">17.18 25.6M</td><td>CIFAR100</td></tr><tr><td>ResNeXt (Xie et al. (2017))</td><td>29(8×64d)</td><td colspan="2">17.77 34.4M</td><td>CIFAR100</td></tr><tr><td>ResNeXt (Xie et al. (2017))</td><td>29(16×64d)</td><td colspan="2">17.31 68.1M</td><td>CIFAR100</td></tr><tr><td>ResNeXt (Our Implement)</td><td colspan="3">29(16×64d), pre-act 17.65 68.1M</td><td>CIFAR100</td></tr><tr><td>LM-ResNet (Ours)</td><td>110, pre-act</td><td colspan="2">25.87 1.7M</td><td>CIFAR100</td></tr><tr><td>LM-ResNet (Ours)</td><td>164, pre-act</td><td colspan="2">22.90 2.55M</td><td>CIFAR100</td></tr><tr><td>LM-ResNeXt (Ours)</td><td>29(8×64d), pre-act</td><td colspan="2">17.49 35.1M</td><td>CIFAR100</td></tr><tr><td>LM-ResNeXt (Ours)</td><td colspan="3">29(16×64d), pre-act 16.79 68.8M</td><td>CIFAR100</td></tr></table>

Single-crop error rate on ImageNet (validation set)

Table content: <table><tr><td>Model</td><td>Layer</td><td colspan="2">top-1 top-5</td></tr><tr><td colspan="2">ResNet (He et al. (2015b)) 50</td><td>24.7</td><td>7.8</td></tr><tr><td colspan="2">ResNet (He et al. (2015b)) 101</td><td>23.6</td><td>7.1</td></tr><tr><td colspan="2">ResNet (He et al. (2015b)) 152</td><td>23.0</td><td>6.7</td></tr><tr><td>LM-ResNet (Ours)</td><td>50, pre-act</td><td>23.8</td><td>7.0</td></tr><tr><td>LM-ResNet (Ours)</td><td colspan="2">101, pre-act 22.6</td><td>6.4</td></tr></table>

Test on stochastic training strategy on CIFAR10

Table content: <table><tr><td>Model</td><td>Layer</td><td colspan="2">Training Strategy Error</td></tr><tr><td>ResNet(He et al. (2015b))</td><td>110</td><td>Original</td><td>6.61</td></tr><tr><td>ResNet(He et al. (2016))</td><td colspan="2">110,pre-act Orignial</td><td>6.37</td></tr><tr><td colspan="2">ResNet(Huang et al. (2016b)) 56</td><td>Stochastic depth</td><td>5.66</td></tr><tr><td>ResNet(Our Implement)</td><td>56,pre-act</td><td>Stochastic depth</td><td>5.55</td></tr><tr><td colspan="2">ResNet(Huang et al. (2016b)) 110</td><td>Stochastic depth</td><td>5.25</td></tr><tr><td colspan="2">ResNet(Huang et al. (2016b)) 1202</td><td>Stochastic depth</td><td>4.91</td></tr><tr><td>LM-ResNet(Ours)</td><td>56,pre-act</td><td>Stochastic depth</td><td>5.14</td></tr><tr><td>LM-ResNet(Ours)</td><td colspan="2">110,pre-act Stochastic depth</td><td>4.80</td></tr></table>

Peer review (iclr2018):
 Summary - This paper draws analogy from numerical differential equation solvers and popular residual network-like deep learning architectures. It makes connection from ResNet, FractalNet, DenseNet, and RevNet to different numerical solvers such as forward and backward Euler and Runge-Kunta. In addition, inspired by the Linear Multi-step methods (LM), the authors propose a novel LM-ResNet architecture in which the next residual block takes a linear combination of the previous two residual blocks’ activations. They also propose a stochastic version of LM-ResNet that resembles Shake-shake regularization and stochastic depth. In both deterministic and stochastic cases, they show a positive improvement in classification accuracy on standard object classification benchmarks such as CIFAR-10/100 and ImageNet. Pros - The intuition is good that connects differential equation and ResNet-like architecture, also explored in some of the related work. - Building upon the intuition, the author proposes a novel architecture based on a numerical ODE solver method. - Consistent improvement in accuracy is observed in both deterministic and stochastic cases. Cons - The title is a little bit misleading. “Beyond Finite Layer Neural Networks” sounds like the paper proposes some infinite layer neural networks but the paper only studies finite number of layers. - One thing that needs to be clarified is that, if the network is not targeted at solving certain ODEs, then why is the intuition from ODE matters? The paper does not motivate readers in this perspective. - Given the widespread use of ResNet in the vision community, the incremental improvement of 1% on ImageNet is less likely to push vision research to switch to a completely different architecture. Therefore, the potential impact of the this paper to vision community is probably limited. Conclusion - Based on the comments above, I think the paper is a good contribution which links ODE with Deep Networks and derivation is convincing. The proposed new architecture can be considered in future architecture designs. Although the increase in performance is small, I think it is good enough to accept.