Title: CONVOLUTIONAL NEURAL NETWORKS COMBINED WITH RUNGE-KUTTA METHODS

Abstract: A convolutional neural network for image classification can be constructed mathematically since it can be regarded as a multi-period dynamical system. In this paper, a novel approach is proposed to construct network models from the dynamical systems view. Since a pre-activation residual network can be deemed an approximation of a time-dependent dynamical system using the forward Euler method, higher order Runge-Kutta methods (RK methods) can be utilized to build network models in order to achieve higher accuracy. The model constructed in such a way is referred to as the Runge-Kutta Convolutional Neural Network (RKNet). RK methods also provide an interpretation of Dense Convolutional Networks (DenseNets) and Convolutional Neural Networks with Alternately Updated Clique (CliqueNets) from the dynamical systems view. The proposed methods are evaluated on benchmark datasets: CIFAR-10/100, SVHN and ImageNet. The experimental results are consistent with the theoretical properties of RK methods and support the dynamical systems interpretation. Moreover, the experimental results show that the RKNets are superior to the state-of-the-art network models on CIFAR-10 and on par on CIFAR-100, SVHN and ImageNet.

1. INTRODUCTION:

Residual Networks (ResNets) which are feed-forward network models with skip connections have achieved great success on several vision benchmarks (He et al., 2016a) . Recently, researchers have studied the relation between ResNets and dynamical systems (Liao & Poggio, 2016; E, 2017; Haber et al., 2017; Chang et al., 2018a; b; Lu et al., 2018) . Forward Euler method, a first-order RK method, has been employed to explain ResNets with full pre-activation (He et al., 2016b) from the dynamical systems view (Haber et al., 2017; Chang et al., 2018b) . Nevertheless, there is no firm evidence to prove that the residual block is just forward Euler method but not any other RK method. We regard the residual mapping as an approximation to the increment in a time-step. The accuracy of the approximation is determined by the structure of the convolutional network. Wide residual network (WRN) (Zagoruyko & Komodakis, 2016) has been proposed to improve the ability of the convolutional subnetwork. However, it is not very efficient only to widen the subnetwork. The new explanation of pre-activation ResNet and its variants which focus on improving residual mapping is one of our contributions.

In addition, some improvements on network architecture based on ordinary differential equations (ODEs) are proposed (Chang et al., 2018a; Lu et al., 2018; Chen et al., 2018) . Under the assumption that pre-activation ResNet is forward Euler method, Chang et al. (2018a) ; Lu et al. (2018) use special linear multi-step methods (LM methods) with low order to construct the network. Chen et al. ( 2018) utilize a third-party package which offers numerical ODE methods to replace residual block. There is no efficient network architecture for systematic generalization to high order till now. Nevertheless, a higher-order method can achieve a lower truncation error. Since a lower truncation error likely leads to a high accuracy, it is necessary to study an efficient network architecture with a high order.

If the process of image classification is deemed a sequence of time-dependent dynamical systems, there should be a series of ODEs to describe these systems. RK methods are widely-used procedures to solve ODEs in numerical analysis (Butcher, 2008) . They are also the building blocks of highorder LM methods. Consequently, these methods can be used to build network models for visual processing.

The neural network community has long been aware of the numerical methods for dynamical systems. Runge-Kutta Neural Network (RKNN) is proposed for identification of unknown dynamical systems in high accuracy (Wang & Lin, 1998) , but it has not been used to model the visual system nor been extended to convolutional networks. Moreover, RKNNs adopt a specific RK methods by indicating every coefficient for the RK methods. Thus, it is hard to apply high order RK methods in RKNNs. In addition, the time-step size need to be prespecified. Hence, RKNN cannot be used in tasks where the total time is unknown such as image classification. In contrast, we learn all the coefficients and time-step sizes implicitly by training in order to avoid these difficulties. As a result, one of the major contributions of the paper is a novel and effective neural network architecture inspired by the RK methods.

In order to apply RK methods to the image classification problem, the following assumptions are made throughout the paper. Firstly, the image classification procedure is multi-period and there are transitions between adjacent periods. Secondly, each period is modeled by a time-dependent firstorder dynamical system. Based on these assumptions, a novel network model called the RKNet is proposed.

In an RKNet, a period is composed of iterations of time-steps. A particular RK method is adopted throughout the time-steps in a period to approximate the increment in each step. The increment in each step is broken down to the increments in several stages according to the adopted RK method. Each stage is approximated by a convolutional subnetwork due to the versatility of neural networks on approximation.

Another contribution of this paper is a theoretical interpretation of DenseNets and CliqueNets from the dynamical systems view. The dense connections in DenseNet resemble the relationship among increments in the stages in explicit RK methods (ERK methods). Similarly, the clique blocks in CliqueNets resemble the relationship among increments in the stages in implicit RK methods (IRK methods). Under some conditions, DenseNets and CliqueNets can be formulated as approximating dynamical systems using multi-stage RK methods. We also propose a method to convert a DenseNet to an explicit RKNet (ERKNet) and a method convert a CliqueNet to an implicit RKNet (IRKNet). Furthermore, DenseNets and CliqueNets have only one time-step in each period, whereas RKNets are more general and can have multiple time-steps in each period.

We evaluate the performance of RKNets on benchmark datasets including CIFAR-10, CIFAR-100 (Krizhevsky, 2009) , SVHN (Netzer et al., 2011) and ILSVRC2012 classification dataset (Russakovsky et al., 2015) . Experimental results show that both ERKNets and IRKNets conform to the mathematical properties. Additionally, RKNets achieve higher accuracy than the state-of-the-art network models on CIFAR-10 and comparable accuracy on CIFAR-100, SVHN and ImageNet.

The rest of the paper is organized as follows. The related work is reviewed in Section 2. The architecture of RKNets, the dynamical systems interpretation of DenseNets and CliqueNets, and the conversion from them to RKNets are described in Section 3. The performance of RKNets is evaluated in Section 4. The conclusion and future work is described in Section 5.

2. RELATED WORK:

ResNets have gained much attention over the past few years since they have obtained impressive performance on many challenging image tasks, such as ImageNet (Russakovsky et al., 2015) and COCO object detection (Lin et al., 2014) . ResNets are deep feed-forward networks with the shortcuts as identity mappings. ResNets with pre-activation can be regarded as an unfolded shallow RNN, which implements a discrete dynamical system (Liao & Poggio, 2016) . It provides a novel point of view for explaining pre-activation ResNets from dynamical systems view.

combining dynamical system identification and RK methods with neural networks for scientific computing has emerged recently (Raissi et al., 2017a; b; Raissi, 2018) , introducing physics informed neural networks with automatic differentiation. Chen et al. ( 2018) utilize a third-party package which offers some numerical methods to compute the numerical solution in each time-step.

DenseNets (Huang et al., 2017) are the state-of-the-art network models after ResNets. The dense connection is the main difference from the previous models. There are direct connections from a layer to all subsequent layers in a dense block in order to allow better information and gradient flow. There is no interpretation of DenseNets from dynamical systems view yet. CliqueNets (Yang et al., 2018) are the state-of-the-art network models based on DenseNets. They adopt the alternately updated clique blocks to incorporate both forward and backward connections between any two layers in the same block. However, there is no interpretation of CliqueNets from dynamical systems view yet.

Given that the process of image classification is regarded as a sequence of time-dependent dynamical systems, there should be a set of ODEs that describes these systems. Consequently, mathematical tools could be employed to construct network models. RK methods are commonly used to solve ODEs in numerical analysis (Butcher, 2008) . Higher order RK methods can achieve lower truncation error. Moreover, these methods are usually the building blocks of high-order LM methods. Therefore, RK methods are ideal tools to construct network models from dynamical systems view.

RK methods have been adopted to construct neural networks, which are known as RKNN, for identification of unknown dynamical systems described by ODEs (Wang & Lin, 1998) . In that paper, neural networks are classified into two categories: (1) a network that directly learns the state trajectory of a dynamical system is called a direct-mapping neural network (DMNN); (2) a network that learns the rate of change of system states is called a RKNN. Hence, AlexNet (Krizhevsky et al., 2012) , VGGNet (Simonyan & Zisserman, 2015) , GoogLeNet (Szegedy et al., 2015) and ResNet (He et al., 2016a) all belong to DMNNs. Specifically, the original ResNet (He et al., 2016a ) is a DMNN because of the ReLU layer after the addition operation. As a result, the ResNet building block learns the state trajectory directly, not the rate of change of the system states. On the contrary, a ResNet with pre-activation (He et al., 2016b ) is an RKNN.

RKNNs are proposed to eliminate several drawbacks of DMNNs, such as the difficulty in obtaining high accuracy for the multi-step prediction of state trajectories. It has been shown theoretically and experimentally that the RKNN has higher prediction accuracy and better generalization capability than the conventional DMNN (Wang & Lin, 1998) .

Therefore, it is reasonable to believe that RK methods can be adopted to design effective network architectures for image classification problems. Additionally, the RK methods might improve the performance of image classification since the convolutional subnetworks are able to approximate the rate of change of the dynamical system states more precisely.

3. RKNETS:

The introduction to RK methods are in Section 3.1. We describe the overall structure of RKNet in Section 3.2. The structure of subnetwork for increment in each time-step is elaborated on in Section 3.3.

3.1. RUNGE-KUTTA METHODS:

An initial value problem for a time-dependent first-order dynamical system can be described by the following ODE (Butcher, 2008) :

EQUATION

)

where y is a vector representing the system state. The dimension of y should be equal to the dimension of the dynamical system. The ODE represents the rate of change of the system states.

The rate of change is a function of time and the current system state. RK methods utilize the rate of change calculated from the ODE to approximate the increment in each time-step, and then obtain the predicted final state at the end of each step. RK methods are numerical methods originated from r is the final state after r time-steps in period d. r is the total number of time-steps in a period. It can vary in different periods. Period 1 and time-step 1 in it are unfolded as an example. System state changes throughout a period. The final state of a step is estimated as the initial state of this step adding an increment. This operation originates from RK methods. To approximate the increment is the key point in RKNet. The dotted lines are for multiscale feature strategy.

Euler method. There are two types of RK methods: explicit and implicit. Both of them are employed in the RKNet. The family of RK methods is given by the following equations (Sli & Mayers, 2003) :

EQUATION

where

EQUATION

In equation 2, y n+1 is an approximation of the solution to equation 1 at time t n+1 , i.e. y(t n+1 ); y 0 is the input initial value; h s i=1 b i z i is the increment of system state y from t n to t n+1 ; s i=1 b i z i is the estimated slope which is the weighted average of the slopes z i computed in different stages. The positive integer s is the number of z i , i.e. the number of stages of the RK method. The equation 3 is the general formula of z i . h is the time-step size which can be adaptive for different time-steps but must be fixed across stages within a time-step.

In numerical analysis, s, a ij , b i and c i in equation 2 and equation 3 need to be prespecified for a particular RK method. These coefficients are displayed in a Butcher tableau. The ERK methods are those methods with a ij = 0 when 1 ≤ i ≤ j ≤ s. All the RK methods other than ERK methods are IRK methods. The algebraic relationships of the coefficients have to meet the order conditions to reach the highest possible order. Different RK methods have different truncation errors which are denoted by the order: an order p indicates that the local truncation error is O(h p+1 ). If a s-stage ERK method has order p, then s ≥ p; if p ≥ 5, then s > p (Butcher, 2008) . Furthermore, a s-stage IRK method can has order p = 2s when its coefficients are chosen under some conditions (Butcher, 2008) . Therefore, more stages may achieve higher orders, i.e. lower truncation errors. The Euler method is a one-stage first-order RK method with b 1 = 1 and c 1 = 0. In other words, high-order RK methods can be expected to achieve lower truncation errors than Euler method. Thus, the goal of our proposed RKNets is to improve the classification accuracy by taking advantage of high-order RK methods.

It is necessary to specify h in order to control the error of approximation in common numerical analysis. The varying time-step size can be adaptive to the regions with different rates of change. The truncation error is lower when the h is smaller.

3.2. FROM RK METHODS TO RKNETS:

There are three components of RKNets: the preprocessor, the multi-periods and the postprocessor. The preprocessor manipulates the raw images and passes the results to the first period. The postprocessor deals with the output from the last period or all the periods while adopting multiscale feature strategy (Yang et al., 2018) . Then, it passes the result to the classifier to make a decision. The periods between those two components are divided by the transition layers. These periods can be modeled by time-dependent dynamical systems. Each period of an RKNet is divided into r timesteps as shown in Figure 1 . RK methods approximate the final state of every time-step using the rate of change of the system state. Some guiding principles when applying RK methods to RKNets are listed as follows.

Firstly, dimensionality reduction is often carried out to simplify the system identification issue, when the dimension of real dynamical system is too high. The dimension of y in each period in RKNet is predefined as the multiplication of the size of feature map and the number of channels at the beginning of a period. The dimensions of y in the same periods of different RKNets can be different due to various degrees of dimensionality reduction. Nevertheless, the dimension of y is consistent within a period.

Secondly, given that there is no explicit ODE for image classification, a convolutional subnetwork is employed to approximate the increment in each time-step. The number of neurons in each hidden layer can be more than the dimension of y.

Thirdly, the number of stages s in each period is predefined in RKNet but the other coefficients, a ij , b i and c i in equation 2 and equation 3 are learned by training. Due to the order conditions (Butcher, 2008) , the relationship among the coefficients are more important than the specific value of any individual coefficient. Hence, the coefficients are learned implicitly but not as explicit parameters. The optimal relationship among the coefficients with a highest possible order is obtained after training. Lastly, the number of time-steps r in each period is predefined in RKNet, but the step size h is learned by training. n in equation 2 and equation 3 is limited to the range [0, r). The learned h is thus considered adaptive. In theory, the adaptive time-step size can achieve higher accuracy.

A variety of RK methods can be adopted in the different periods of RKNets, but the same RK method is used for all time-steps within one period in an RKNet. The network models are named after the specific method in each period, such as RKNet-3×2 4×1 2×5 1×1. The suffix in the name of an RKNet is composed of several s × r terms; each stands for the method in corresponding period. The number of such terms equals the total number of periods. s or r can vary in different periods. For example, RKNet-3×2 4×1 2×5 1×1 has four periods: period one has 2 time-steps and each step has 3 stages; period two has 1 time-step and it has 4 stages; period three has 5 time-steps and each step has 2 stages; period four has 1 time-step and it has 1 stage. We use this notation throughout this paper. In addition, ERKNets only adopt ERK methods and IRKNets only adopt IRK methods.

Given an RKNet model, s and r can be modified to construct more variants with the same dimensions in the corresponding periods. In other words, s and r control depth of the network while dimensionality reduction controls the width of the network. More stages, more time-steps and larger dimensions usually lead to higher classification accuracy. However, the complexity of an ODE increases with the increase of dimensions. As a result, the convolutional subnetwork which approximates the increment in a time-step need be more complex for larger dimensions. Hence, the accuracy is also associated with the matching degree of the dimension and the convolutional subnetwork. The unmatched high-dimensional network model may have lower accuracy. Additionally, the training method might affect the classification accuracy too.

3.3. ERKNETS AND IRKNETS:

In this section, we introduce the architecture of RKNets. As shown in equation 2, the sum of hb i z i represents the increment in a time-step. It is crucial to approximate this increment in RKNet. For y n is the approximation of y(t n ). A dense block grows every m times at a growth rate of k to form a convolutional subnetwork for generating each hb i z i . Here, h is time-step size, b i is coefficient of ERK method, and z i is the slope of each stage in ERK method. The total number of growth is ms in a dense block in order to generate hb i z i for i = 1, . . . , s. An explicit summation layer is added after a dense block to complete a time-step.

the purpose of constructing an RKNet, it is necessary to hide the time-step size and the coefficients in RK methods. hb i z i can be described as follows according to equation 3:

EQUATION

The above transformation first changes the explicit dependence on the time in equation 3 to an implicit one. Since the time parameter t n +c i h is different for the different stages, it can be absorbed into g i (•), which implicitly depends on time for stage i. Afterward, the summation in the input parameter of g i (•) is split into separate terms. F i (•) denotes the function of these terms for each stage. We verify that F i (•) can equal to g i (•) after training by experiment though F i (•) is more expressive than g i (•) in expression. Additionally, F i (•) is more memory efficient than g i (•) because of saving the storage for the summation inputted to g i (•).

3.3.1. CONNECT ERKNETS WITH DENSENETS:

In order to construct ERKNets, hb i z i can be described by the equation below, according to equation 4.

EQUATION

)

The above transformation first eliminates ha ij z j (i ≤ j) from F i (•) in equation 4 since a ij = 0 when 1 ≤ i ≤ j ≤ s for ERK methods (See 3.1). As a result, hb i z i is denoted by a function of y n and ha ij z j for j = 1, . . . , i -1. It is written as e i (•). After that, adjusting the coefficients of each parameter from a ij to b j yields another function E i (•). It is a function of y n and hb j z j for j = 1, . . . , i -1.

If a convolutional subnetwork is adopted to model E i (•) in equation 5, the most similar network structure is the dense connections in DenseNets. To be specific, a growth in a dense block concatenates all the preceding layers as the input of convolutional subnetwork just like that hb i z i uses y n and all the increments in preceding stages as the input of E i (•). For the purpose of adopting dense block in ERKNets, the dense blocks must conform to the following rules.

Rule 1 The number of channels of y n is in the form of mk, where m and k are positive integers and k is known as the growth rate in DenseNet literature. The dimension of y n is the multiplication of the size of feature map and mk.

Rule 2 Every m successive growth constructs a convolutional subnetwork for E i (•). Each subnetwork outputs mk channels which are regarded as a group according to the number of channels of y n . Each convolutional subnetwork concatenates y n and all the preceding groups as its input. The ith group generated by the ith subnetwork corresponds to hb i z i .

Rule 3 The total number of growth is ms, where s is number of stages of RK methods. Consequently, s groups representing hb i z i for i = 1, . . . , s are generated by s convolutional subnetworks modeling E i (•) for i = 1, . . . , s successively in a dense block.

Appending to a restricted dense block conforming to the above rules, y n and the groups hb i z i for i = 1, . . . , s are added to obtain y n+1 according to equation 2. Figure 2 illustrates one time-step of ERKNet.

In DenseNets, every dense block together with part of the subsequent computation can be regarded as a period using a s-stage ERK method with r = 1 time-step. The transition layers and the postprocessor contain the summation operation in equation 2. This gives an explanation of DenseNets from the dynamical systems view.

3.3.2. CONNECT IRKNETS WITH CLIQUENETS:

hb i z i for IRK methods can be described by the equation below, according to equation 4.

EQUATION

where v j = V j (hb j z j ) = J j (y n , v 1 , . . . , v j-1 ) .

The above transformation first adjusts the coefficients of each parameter of F i (•) in equation 4 from a ij to b j . It yields another function H i (•). As a result, every hb i z i is a function of y n . Thus, hb i z i can be denoted by a function of hb j z j for j = 1, . . . , s, j = i. This function is written as G i (•).

Inspired by Newton method which is used to implement IRK methods (Butcher, 2008) , hb i z i is initialized using all available information firstly and then updated alternately. Given v j is the initial value of hb j z j , the relationship between them is denoted by the function V j (•). Therefore, hb i z i can be denoted by a function of hb j z j for j = 1, . . . , i -1 and v j for j = i + 1, . . . , s. This function is written as I i (•). It is the update function of hb i z i . Since every hb j z j is a function of y n , every v j is also a function of y n . Thus, v j can be denoted by a function of y n and v q for q = 1, . . . , j -1. This function is written as J j (•). It is the initialization function of hb j z j .

The update process is a sequence of iterations till convergence in Newton method. In other words, v j is updated for many times to approach hb j z j . During updating, G i (•) with the biased input is used as the update function since I i (•) is unknown. If using convolutional subnetwork to model each I i (•), these functions can be learned under the help of training. As a result, each v j needs to be updated only once. Therefore, the computational cost is reduced remarkably.

If a convolutional subnetwork is adopted to model J j (•) in equation 7 and I i (•) in equation 6, the most similar network structure is the clique block in CliqueNets. To be specific, a clique block is composed of Stage-I and Stage-II in CliqueNet literature. Stage-I which initializes all layers in a clique block is regarded as a sequence of J j (•). Then, Stage-II for updating all layers alternately corresponds to all I i (•). In CliqueNet literature, all layers in a clique block except the top layer to be updated are concatenated as the bottom layer, i.e. the input of a convolutional subnetwork for updating. It is just like I i (•) uses hb j z j for j = 1, . . . , i -1 and v j for j = i + 1, . . . , s as input.

In order to adopt clique block in IRKNets, the clique blocks must conform to the following rules. y n is the approximation of y(t n ). A dense block, which is Stage-I of a clique block, grows k channels every time to generate the initial value of each hb i z i , written as v i . Here, h is time-step size, b i is coefficient of IRK method, and z i is the slope of each stage in IRK method. In Stage-II of a clique block, the convolutional subnetwork concatenating the current values of hb j z j for j = 1, . . . , 3, j = i to update every hb i z i alternately. An explicit summation layer is added after a clique block to complete a time-step.

Rule 1 The number of channels of y n is k, which is the growth rate in Stage-I since Stage-I is a dense block. The dimension of y n is the multiplication of the size of feature map and k.

Rule 2 Every growth in Stage-I constructs a convolutional subnetwork. Each subnetwork outputs k channels which are regarded as a group according to the number of channels of y n . Each convolutional subnetwork concatenates y n and all the preceding groups as its input. The ith group generated by the ith subnetwork is v i .

Rule 3 The total number of growth in Stage-I is s, which is number of stages of RK methods. Consequently, s groups representing v i for i = 1, . . . , s are generated by s convolutional subnetworks successively in Stage-I. s should be larger than 1 for updating alternately in Stage-II.

Appending to a restricted clique block conforming to the above rules, y n and the groups hb i z i for i = 1, . . . , s are added to obtain y n+1 according to equation 2. Figure 3 illustrates one time-step of IRKNet using a 3-stage IRK method as an example.

In CliqueNets, every clique block together with part of the subsequent computation can be regarded as a period using a s-stage IRK method with r = 1 time-step. The transition layers and the postprocessor contain the summation operation in equation 2. This gives an explanation of CliqueNets from the dynamical systems view.

4. EXPERIMENTS:

To verify the theoretical properties of RK methods and evaluate the performance of RKNets on image classification, experiments are conducted using the proposed network architectures. The experimental setup is described in Appendix A. Some extra techniques, including attentional transition, bottleneck and multiscale feature strategy, can be adopted in RKNets following CliqueNets. They are introduced in Appendix B. 3 . 2 . The top-1 and top-5 errors on ImageNet validation set with a single-crop (224×224) are shown in Table 3 . Figure 4 shows the single-crop top-1 validation errors of DenseNets, CliqueNets and RKNets as a function of the number of parameters (left) and FLOPs (right). According to the experimental results, RKNets are more efficient than the state-of-the-art models on CIFAR-10 and on par on CIFAR-100, SVHN and ImageNet.

5. CONCLUSION:

We propose to employ a type of numerical ODE methods, the RK methods, to construct convolutional neural networks for image classification tasks. The proposed network architecture can systematically generalize to high order. At the same time, we give a theoretical interpretation of the DenseNet and CliqueNet via the dynamical systems view. The model constructed using the RK methods is referred to as the RKNet, which can be converted from a DenseNet or CliqueNet by enforcing theoretical constraints.

The experimental results validate the theoretical properties of RK methods and support the dynamical systems interpretation. Moreover, the experimental results demonstrate that RKNets surpass the state-of-the-art models on CIFAR-10 and are on par on CIFAR-100, SVHN and ImageNet.

With the help of the dynamical systems view and various numerical ODE methods including RK methods, more general neural networks can be constructed. Many aspects of RKNets and the dynamical systems view still require further investigation. We hope this work inspires future research directions.

:



Figures and Tables:

Figure 1: Architecture of a 3-period RKNet. y (d) denotes the system state of period d. y (d) 0 is the initial state of period d. y (d)

Figure2: Architecture of one time-step in ERKNet using an s-stage ERK method. y n is the approximation of y(t n ). A dense block grows every m times at a growth rate of k to form a convolutional subnetwork for generating each hb i z i . Here, h is time-step size, b i is coefficient of ERK method, and z i is the slope of each stage in ERK method. The total number of growth is ms in a dense block in order to generate hb i z i for i = 1, . . . , s. An explicit summation layer is added after a dense block to complete a time-step.

Figure3: Architecture of one time-step in IRKNet using a 3-stage IRK method. y n is the approximation of y(t n ). A dense block, which is Stage-I of a clique block, grows k channels every time to generate the initial value of each hb i z i , written as v i . Here, h is time-step size, b i is coefficient of IRK method, and z i is the slope of each stage in IRK method. In Stage-II of a clique block, the convolutional subnetwork concatenating the current values of hb j z j for j = 1, . . . , 3, j = i to update every hb i z i alternately. An explicit summation layer is added after a clique block to complete a time-step.

Figure 4: Comparison of the DenseNets, CliqueNets and RKNets. The top-1 error rates (single-crop testing) on the ImageNet validation dataset are shown as a function of learned parameters (left) and FLOPs during test-time (right). RKNets compared here are the models shown in Table3.

on CIFAR-10, CIFAR-100 and SVHN while ERKNets are evaluated on Im-ageNet to compare with the state-of-the-art network models. The test errors of IRKNets on CIFAR-10, CIFAR-100 and SVHN are shown in Table

Classification errors on ImageNet validation set with a single-crop (224×224). The growth rate k is 32 and mk is the initial number of channels in each period in RKNets. m n stands for m in the nth period. For each RKNet in this table, m 0 is 2, m 1 is 4 and m 2 is 8. B represents bottleneck. The bottleneck layers which output 4k channels to the following layers are used in ERKNets.

Table content: <table><tr><td>Model</td><td colspan="2">m 3 FLOPs</td><td>Params</td><td>Top1 (%)</td><td>Top5 (%)</td></tr><tr><td/><td/><td>(G)</td><td>(M)</td><td/></tr><tr><td colspan="2">ERKNet-3×1 3×1 3×1 1×1-B 16</td><td>5.20</td><td>6.95</td><td>25.47</td><td>7.81</td></tr><tr><td colspan="2">ERKNet-3×1 3×1 4×1 2×1-B 20</td><td>6.35</td><td>14.49</td><td>24.12</td><td>7.17</td></tr><tr><td colspan="2">ERKNet-3×1 3×1 6×1 2×1-B 28</td><td>8.50</td><td>25.51</td><td>23.14</td><td>6.66</td></tr></table>

Peer review (iclr2019):
 The paper takes the perspective that a ResNet can be seen as a discrete time dynamical system approximation, and then uses that insight to introduce a Runge Kutta style continuous dynamical system. Results are achieved that outperform DenseNet and CliqueNet Quality - The connection between deep nets and dynamical systems is an important one, as a small but growing literature demonstrates. - The quality of this paper is difficult to extract, given rather long and unclear definitions of various key choices, at key points opting instead for a speculative connection to neurosciences (see clarity below). - The empirical results could make this paper compelling, but the comparisons are rather shallow, in that they compare on only a few datasets and don*t compare extensively to many of the state of the art methods. For example, in Table 2 DenseNet outperforms the IRKNets, but this fact is not highlighted. Overall this and other key omissions create the concern that the results are not particularly strong. - One key reference that is missing is the careful work Chen et al https://arxiv.org/abs/1806.07366. This work provides an excellent basis for how to present these ideas rigorously. Clarity - The connections to the brain -- such as "visual cortex", "ventral stream", etc -- are not rigorous and are unrelated to the paper at hand; really all connections to brain should be removed. - Figures 2 and 3 are the central architectural choices, but the writing around them does not clarify why all the choices have been made, and what the implications are. Originality - This is one of a small but growing number of papers connecting deep nets to ODE solvers. Chen et al should be cited (Chen et al https://arxiv.org/abs/1806.07366). - There appears to be sufficient originality. Significance - hard to determine with unclear exposition and rather incomplete empirical results.