Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the connection between ResNet and Neural ODE. Specifically, (1). this paper investigates the approximation error between the hidden state trajectory of the Resnet and the solution of the Neural ODE interpolating the hidden state of the Resnet. The approximation error can have a non-zero limit in general, and the authors specify a case (small enough initialization, linear blocks) where the error limit is zero. (2). The paper then views the hidden state trajectory of ResNet as a forward Euler method to solve the corresponding NODE and proposes to use the model obtained by the backward Euler method in the backpropagation, which has a lower memory cost. Such a practice is also transferred to the HeunNet with a theoretically lower approximation error bound. The authors further conduct experiments to verify the theoretical findings.
strengths_and_weaknesses: Strengths:
1.	This paper is well-written
2.	The theoretical analysis is solid, and the authors honestly report the experiment results.

Weakness:
1.	The settings of the theoretical analysis are far from the practice. This paper uses the forward propagation rule x_{t+1}=x_{t} +1/N f(x_t,\theta_t), where N is the depth of neural networks. However, in the latter analysis (e.g. Proposition 1,2), it is assumed that \theta_t has bounded norm (or f has bounded Lipschitz parameter with respect to theta_t). Therefore, if we absorb the 1/N factor into f (or more directly into \theta_t, as in neural networks, f is usually a piece-wise linear function of \theta_t), which is what we do in practice, the initialization scale of \theta can be rather small and decreases with the depth, while in practice the initialization scale usually depends on the width. Theorem 1 requires the parameters between adjacent layers to be close, which is also unrealistic.
2.	The experiment results are not promising even compared to the current state of the arts. For example, in table 2, the test accuracy on Imagenet for Resnet using the adjoint method is not high even compared to the vanilla Resnet [1]. Also, the performance of the adjoint method in Figure 4 is worse than the traditional backpropagation.

[1]. https://paperswithcode.com/sota/image-classification-on-imagenet?tag_filter=3

questions: 1. Why do we need to save the memory cost for the hidden state, which is relatively small? Specifically, the dimension of one hidden state (size m) is way smaller than the dimension of the corresponding layer’s parameter (size m^2).
2. What’s the point to approximate the NODE using Resnet (or study this regularization effect)? I expect the authors can comment on the advantage of NODE over ResNet and why we need to study this regularization effect.

limitations: Please refer to the "weakness" and “questions”.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the link between Neural ODEs and ResNets from a theoretical point of view.

The authors show that without extra regularity assumptions on the residual block, a ResNet might not converge to a Neural ODE in the limit when the depth goes to infinity. On the positive side, they then show how this issue can be resolved by assuming additional assumptions on the weight initialiatiation and study the case of linear residual blocks.

An additional contribution is a numerical technique allowing to perform backpropagation through ResNets similarly to the adjoint method for Neural ODEs; this has the advantage of not requiring to store activations, and therefore comes with constant memory footprint. The disadvantage is given by inaccurate gradients. Error bounds for recovering activations and gradients are provided, and even improved by using a modification of the proposed adjoint method analogue to the Heun's method for differential equations.





strengths_and_weaknesses: This paper sheds some light on the theoretical aspects relating Neural ODEs and ResNets, which has recently become a popular topic in the deep learning community. Several contributions have been provided in this direction prior to this paper, in particular highlighting the failure, in some cases, of ResNets to approximate ODEs in the infinite depth limit. Therefore, the level of originality of the paper is not very high, but the contributions build upon existing ones and the literature seems appropriately cited. The proposed technique based on the adjoint method to train ResNets without storing activations is interesting and yields improvements in performance on CIFAR-10 and ImageNet. 

The main weakness is that the authors only study the linear residual block case.


questions: How much do the results in section 3.2 depend on the choice of loss function?

What are the main theoretical bottlenecks preventing from extending the proofs on section 3.2 from the linear case to the non-linear case? Are there some non-linear activation functions for which this extension could be easier than others?
limitations: As mentioned above, the main limitation is that the study is limited to the linear case for the residual block, but the authors clearly mention this in their conclusion.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - The paper examines the relationship between and NeuralODEs and ResNets with $N$ layers whose residual functions have a multiplicative coefficient of $1/N$, i.e. with layers of the type $x_{n+1} = x_n + \frac{1}{N}f_{\theta_n}(x_n)$. 
- The paper shows that if the initialised weights of a _linear_ ResNet vary smoothly from one layer to the next and other assumptions at training time $t=0$ are satisfied, as $N\rightarrow \infty$, the map learned by the ResNet at any training time $t$ corresponds to the map induced by a (training time-dependent) ODE sending $x(0) \mapsto x(1)$, where $x(1)$ is the solution of this ODE at time $1$. 
- The paper shows that the smoothness assumption mentioned above is necessary and this convergence is not otherwise guaranteed. 
- The paper investigates how these sort of ResNets can be trained with $O(1)$ memory via a discrete adjoint method producing approximate gradients. It further investigates what the error of this approximation is as a function of $N$ and shows how this error can be reduced by using Heun's method. 
- Finally, the experimental section shows ResNets with rate $1/N$ work similarly well in practice as regular ResNets and study how the proposed discrete adjoint can be used to fine-tune deep ResNets with (initially) tied weights. 
strengths_and_weaknesses: # Strenghts

- Despite the well-known connections between ResNets and NeuralODEs, this paper shows that there are still many interesting aspects to uncover about their relationship. Usually, when this equivalence is discussed, it is typically assumed that $\theta_n = \theta$.  This paper investigates this equivalence in the linear regime when the weights $\theta_n$ vary at each layer $n$, which is much closer to how ResNets are actually constructed and used in practice. Therefore, this improves our understanding of a practically-relevant situation and it is likely that this result will also be relevant to practitioners and not just the theoretically inclined. 
- The proposed discrete adjoint method comes with a useful theoretical analysis providing theoretical guarantees regarding the error potential users should expect. 
- Although not spectacular or surprising, the successful experiments on reproducing the performance of regular ResNets using ResNets with rate $1/N$ are an important result. It shows that this model that is convenient theoretically is also relevant in practice and essentially it does not affect the usual performance we expect from ResNets. This also makes the theory more relevant for practitioners. 
- I like that the paper also explores failure modes in the experimental section for when the assumptions of the results do not hold. 
- In the era of "large models", the presented fine-tuning experiments are very relevant and could be a useful tool in the arsenal of anyone training deep ResNets. 

# Weaknesses

- I believe the main problem of the paper is the (too) many notational errors or omissions for a theoretical paper. While each in itself is minor, together they accumulate and they do tend to show up in important parts of the paper. So I am adding a list below: 
    1. In the bound of Proposition 1, if $L=0$, then $\frac{e^L - 1}{L} = \frac{0}{0}$, which is undefined. So I do not understand the calculations done below where $L=0$. Am I missing something here? 
    2. What does the notation $[\cdot]$ mean in the definition of $C_N$ (line 142). The $[\cdot]$ notation is also used in other places. And what does the superscript $K \times [0, 1]$ signify?
    3. Please explain the notation $C^k$ for classes of smooth functions. Many people with ML background might not be familiar with it. 
    4. Before Proposition 1, it says ``We now consider a smooth interpolation $\phi_\Theta$ for ResNet''. Which interpolation do you actually use? Two were introduced in the paragraph above. Does it matter? 
    5. Below Proposition 1 you mention that the bound is tight. But you show this only for the case when the ODE function depends only on time, which makes L superfluous (ignoring the undefined issue). But is the bound actually tight in general? This question is probably not super relevant for the discussion in the implications paragraph below, but I think it's still an important thing to clarify. 
    6. In line 188, $\nabla_n^N(t) = \nabla_{\theta_n^N}L$, there is no $t$ on the right hand side. Should that be $\nabla_{\theta_n^N}L(t)$?
    7. What is the difference between $L(t)$ and $l^N(t)$. Both are used and as far as I understand they denote the same thing. 
    8. In Assumption 2, what is $D_x$? As far as I can see, $D_x$ is never defined. Should that be $\partial_x$ instead?
    9. In Assumption 2, $C_{df}$ is defined and never used. Instead, an $L_f$ shows up, that was never introduced. Should $L_f$ be $C_{df}$?
    10. Proposition 6, No need to mention $L_{df}$ if it doesn't show up in the final bound. 
    11. Please clarify what you mean by "tied weights". I checked many ResNet papers and it doesn't seem to be standard terminology. I take it to mean $\theta_n = \theta$. But this can be a confusing term because the smoothness assumption you often employ also "ties" the weights of different layers together. Constant weights is a better term IMO. 
- Figure 2 is not particularly useful and could show something much more interesting and visualise it much better. First of all, a sequential colour scale is much more appropriate for what it is trying to show because this shows a progression of training steps. A categorical colouring scale does not convey that information. Second, plotting just a few weights in the matrices loses a lot of information. The plot could show instead the distance from the limit $\psi$ as a function of the training time or another aggregated metric of interest. Perhaps the distance between consecutive layers and how it evolves over time could be another idea. 
- In the first plot of Figure 3, since this is the median over only 3 runs, it's better to just include all 3 runs (6 runs in total) since it's not a lot. The median drops a lot of information and doesn't say much about the variation. Alternatively, better to show the mean and a confidence interval. 
- In line 377, you mention that "the HeunNet ... performs better than the ResNet". This claim is not really backed up by data. It's impossible to deduce that from Figure 4 and we would need to see some numerical data. There are two things that are needed to support that claim: scientific significance (if the difference is just 0.00001%, and it looks very small on the plot, then it's not really better) and statistical significance (what is the standard error?). The claim should either be dropped or more data should be supplied.  

======= EDIT ===========

Raised my score to accept after discussion. 
questions: - Most importantly, everything in the first bullet point of the weaknesses section. 
- In line 328, what do you mean by "the classical ResNet forward rule is not compatible with the usual training procedure of CIFAR"? 
limitations: Yes.
review_title: null

Meta-review: metareview: The paper provided an error bound for the approximation between the hidden trajectory of ResNet and its Neural ODE. The trajectory value of ResNet is different for each layer, which is more close to the practice. Authors find a case (linear residual functions, small initial loss) where the error bound can converge to zero. By leveraging the connection in the special case, an adjoin method is designed  to estimate the gradient with a memory-free manner. This work can push our understanding about the connection between deep NN models (i.e., ResNet) and Neural ODE. The designed algorithm also called for our attention that, the connection can facilitate "memory-free" gradient estimation.  
confidence: Certain
Decision: Accept