Two Sides of The Same Coin: Bridging Deep Equilibrium Models and Neural ODEs via Homotopy Continuation
Authors: Shutong Ding
Tianyu Cui
Jingya Wang
Ye Shi
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Equilibrium Models
Neural Ordinary Differential Equations
Homotopy Continuation
TL;DR: We bridge Neural ODEs and Deep Equilibrium Models via homotopy continuation and propose a new implicit model called HomoODE that inherits the property of high accuracy from DEQs and the property of stability from Neural ODEs.

Abstract: Deep Equilibrium Models (DEQs) and Neural Ordinary Differential Equations (Neural ODEs) are two branches of implicit models that have achieved remarkable success owing to their superior performance and low memory consumption. While both are implicit models, DEQs and Neural ODEs are derived from different mathematical formulations. Inspired by homotopy continuation, we establish a connection between these two models and illustrate that they are actually two sides of the same coin. Homotopy continuation is a classical method of solving nonlinear equations based on a corresponding ODE. Given this connection, we proposed a new implicit model called HomoODE that inherits the property of high accuracy from DEQs and the property of stability from Neural ODEs. Unlike DEQs, which explicitly solve an equilibrium-point-finding problem via Newton's methods in the forward pass, HomoODE solves the equilibrium-point-finding problem implicitly using a modified Neural ODE via homotopy continuation. Further, we developed an acceleration method for HomoODE with a shared learnable initial point. It is worth noting that our model also provides a better understanding of why Augmented Neural ODEs work as long as the augmented part is regarded as the equilibrium point to find. Comprehensive experiments with several image classification tasks demonstrate that HomoODE surpasses existing implicit models in terms of both accuracy and memory consumption.

Two Sides of The Same Coin: Bridging Deep Equilibrium Models and Neural ODEs via Homotopy Continuation
Shutong Ding ShanghaiTech University dingsht@@hanghaittcch edu. cn
Tianyu Cui ShanghaiTech University cutty022@shanhaatecccc edu. cn
Jingya Wang ShanghaiTech University wangjingya@Ghanghattehh edu.
Ye Shi t ShanghaiTech University
shiye@shanghaitechh edu.
Abstract
Deep Equilibrium Models (DEQs) and Neural Ordinary Differential Equations (Neural ODEs) are two branches of implicit models that have achieved remarkable success owing to their superior performance and low memory consumption. While both are implicit models, DEQs and Neural ODEs are derived from different mathematical formulations. Inspired by homotopy continuation, we establish a connection between these two models and illustrate that they are actually two sides of the same coin. Homotopy continuation is a classical method of solving nonlinear equations based corresponding ODE. Given this connection, we proposed new implicit model called HomoODE that inherits the property of high accuracy from DEQs and the property of stability from Neural ODEs. Unlike DEQs, which explicitly solve an equilibrium- coan---oott-iiitt findifniingg problem via Newton's 's methods in the forward pass, HomoODE solves the equilibrium point--ointttitte faini-finiing problem implicitly using a modified Neural ODE via homotopy continuation. Further, we developed an acceleration method for HomoODE with shared learnable initial point. It worth noting that our model also provides a better understanding of why Augmented Neural ODEs work long the augmented part is regarded the equilibrium point to find. Comprehensive experiments with several image classification tasks demonstrate that HomoODE surpasses existing implicit models in terms of both accuracy and memory consumption.
Introduction
Recent studies of implicit models have certified that such models meet surpass the perfor- mance of traditional deep neural networks. Instead of specifying the explicit computation process of the output, implicit models define the joint conditions that the layer's input and output satisfy. Instances of such models include the Neural Ordinary Differential Equations (Neural ODEs) [T0]
which treat ODEs learnable component and can be viewed continuous Residual Networks 255
(ResNets), the Deep Equilibrium Models (DEQs) 6 which compute the equilibrium point of nonlinear transformation corresponding to an infinite-depth weight-tied network; and optimization layers 4 45), which leverage the optimization techniques as layers of neural networks.
Although DEQs and Neural ODEs, as popular implicit models in recent years, have garnered much attention in terms of theoretical analysis and application, an insightful connection between these two
"Equal contribution. +Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

branches of the implicit model has not been established. DEQs involve an equilibrium foint-fnnding problem, which nonlinear equation system. Generally, this can be solved via the homotopy continuation method [3]] classical method that solves nonlinear equations along the zero path of the homotopy mapping and can be further formulated as an ODE. This motivated us to consider whether we could bridge DEQs and Neural ODEs via the theory of homotopy continuation.
In this paper, we show that Neural ODEs can also be viewed as procedure for solving an equilibrium- point-finding problem. However, while both of the two models can be considered as solving equilibrium-pont paletppoointniieeeee finding problems, they differ in how the input information is used. On the one hand, DEQs regard the input information as the condition that determines the equilibrium-pon fiiiin-f problem to solve via injecting it into the equilibrium function in the forward pass. On the other hand, Neural ODEs generate the initial points with different input information and expect them to converge to different equilibrium points. Therefore, we claim that DEQs and Neural ODEs are actually two sides of the same coin.
Inspired by the theoretical connection between the two models, we developed a new implicit model called HomoODE, which inherits the advantages of both. Specifically, HomoODE injects the input information into the underlying dynamic of an conliiriuu-ppon--fnddie as a DEQ does, but then obtains the output from an ODE solver just as a Neural ODE does. The connection between DEQ and Neural ODE means that HomoODE avoids DEQ's 's problem of unstable convergence in DEQs and the weak representation capability of Neural ODEs. Further, common issue for implicit models is the trade-off between computational speed and precision. Therefore, a natural intuition to accelerate such models is to find a good initial point that is close to the solution. We observed that, as the distance between the initial point and the solution drops to some range, the numbers of function evaluations (NFE) almost stop decreasing when applying homotopy continuation. Hence, we introduced an extra loss instead of zero vector initialization in DEQ to train a shared initial point. In this way, the distance from the initial point to each solution is within appropriate ranges. A series of experiments with several image classification tasks verify that HomoODE is able to converge stably and that it outperforms existing implicit models with better memory efficiency on several image classification tasks, and our acceleration method significantly reduces the NFE of HomoODE. In summary, our contributions are follows:
problem in the same way
(1) A connection between DEQs and Neural ODEs. We establish a connection between DEQs and Neural ODEs via homotopy continuation, which illustrates that DEQs and Neural ODEs are actually the two sides of the same coin. We believe this new perspective provides novel insights into the mechanisms behind implicit models.
(2) A New Implicit Model: HomoODE. We propose new implicit model called HomoODE, that inherits the advantages of both DEQs and Neural ODEs. HomoODE implicitly solves equilibrium- point-finding problems using homotopy continuation, unlike DEQs which explicitly solve these problems via Newton's 's methods. Additionally, we have accelerated HomoODE with a learnable initial point that is shared among all samples.
(3) Understanding Augmented Neural ODE, We demonstrate that Augmented Neural ODE can be treated special case of HomoODE based homotopy continuation. Hence Augmented Neural ODE enjoys better representation ability and outperforms Neural ODE.
(4) Better Performance. We conduct experiments on image classification datasets and confirm our HomoODE outperforms DEQ, Neural ODE, and variants of them both in accuracy and memory usage. Furthermore, we also perform the sensitivity analysis on the hyper-parameters to research the characters of our model.
2 Related Works
Deep Equilibrium Models. DEQs 6 have shown competitive performance on range of tasks, such as language modeling( graph-related tasks (20). image classification or segmentation [7], image generation [4]] inverse problems in imaging 16). image denoising [17] and optical flow estimation
[5] DEQs find an equilibrium point a nonlinear dynamical system corresponding to an effectively infinite-depth weight-tied network. However, training such models requires careful consideration of both initializations and the model structure 6, 9 3 and often consumes long training times. Many studies have been devoted to solving these problems. For example, the Monotone Operator
2

---

Equilibrium Network (monDEQ) [6 ensures stable convergence to unique equilibrium point by involving monotone operator theory. Bai et al. propose an explicit regularization scheme for DEQs that stabilizes the learning of DEQs by regularizing the Jacobian matrix of the fixed-point update equations. Kawaguchi et al. 30 prove that DEQs converge to global optimum at a linear rate for a general class of loss functions by analyzing its gradient dynamics. Agarwala et al. [ show that DEQs are sensitive to the higher-order statistics of their initial matrix family and consequently propose practical prescription for initialization. To reduce the computational expense of DEQ, Pal et al. 40 developed continuous DEQs utilizing an "infinity time" neural ODE but did not continue to explore the inherent connection between general DEQs and Neural ODEs. From the perspective of optimization, Optimization Induced Equilibrium Networks (OptEq)(47) theoretically connect their equilibrium point to the solution of a convex optimization problem with explicit objectives. Instead of regularizing the structures or involving parameterizations of the implicit layer design, Bai al. [8] propose a model-specific equilibrium solver, which both guesses an initial value of the optimization and performs iterative updates. However, unlike DEQs, which explicitly solve equilibrium-poiin poiiiuummmmo--iiiinngg problems via Newton's 's methods, our HomoODE solves these problems based on homotopy continuation implicitly. Accordingly, HomoODE does not suffer from the issue of unique equilibrium like DEQ and thus can avoid the stability issue. Moreover, we accelerate HomoODE using a good initial point learned with a corresponding loss function. Unlike Bai et al.'s 's approach [8) HomoODE learns an initial point shared among all samples without involving a network-based initializer.
Neural Ordinary Differential Equations. Neural ODEs have been applied to time series modeling continuous normalizing flows [10, [8] and modeling or controlling physical environments
44
32
50
48
19
14). Neural ODEs treat ODEs as learnable component and produce their outputs by
solving the Initial Value Problem [10]
12 29 However, Neural ODEs are often characterized by
long training times and sub-optimal results when the length of the training data increases [3] 15). Prior works have tried to tackle these problems by placing constraints on the Jacobian 13 or high derivatives of the differential equation 31. Conversely, Augmented Neural ODEs 12 learn the flow from the input to the features in an augmented space with better stability and generalization. Ghosh et al. [15] treat the integration time points as stochastic variables without placing any constraints. With diffeomorphism. the complexity of modeling the Neural ODEs can be offloaded onto the invertible neural networks 49, and training Neural ODEs with the adaptive checkpoint adjoint method [5T] can be accurate, fast, and robust to initialization. The symplectic adjoint method [36] finds the exact gradient via symplectic integrator with appropriate checkpoints and memory consumption that is competitive to the adjoint method. The advantage of HomoODE is that it inherits the property of high accuracy from DEQs and the property of stability from Neural ODEs. In addition, HomoODE provides an explanation of why Augmented Neural ODEs achieve better performance than Neural ODEs. Notably, Augmented Neural ODEs can be viewed as a special case of HomoODE.
Homotopy Continuation. Homotopy continuation [39 is a numerical technique that traces the solution path of given problem as the parameter changes from an initial value final value. Homotopy methods have been successfully applied to solving pattern formation problems arising from computational mathematics and biology including computing multiple solutions of differential equations 21 22), state problems of hyperbolic conservation laws (21]. computing bifurcation points diffusion equations 23 Recent advances in deep learning have also seen the homotopy continuation method fused into learning processes. For instance, Ko et al. (33 adapt homotopy optimization in Neural ODEs to gain better performance with less more training epochs. HomPINNs 27 traces observations in an approximate manner to identify multiple solutions, then solves the inverse problem via the homotopy continuation method. To reach good solution to the original geometrical problem, Hruby et al. (26] learn single starting point for a real homotopy continuation path. In this work, we establish a connection between DEQs and Neural ODEs from the perspective of homotopy continuation and develop a new implicit model called HomoODE based on this theoretical relationship.
of nonlinear systems (24 and solving reaction-diffusion
Background on Homotopy Continuation
Homotopy continuation has been broadly applied to solve nonlinear equations. The first step to solving specific problem r(z) = 0 is to construct a homotopy mapping.
3

---

N)g(2) is said to be homotopy mapping from g(2) to r()), if is a scalar parameter from 0 1, and g(z) a smooth function. The equation [2z, X) = 0 is the zero puth of this homotopy mapping.
Definition 1 (Homotopy mapping (39)) The function
Homotopy mapping provides continuous transformation by gradually deforming g(z) into r(=) while À increases from 0 to 1 in small increments. Hence, the solution to r(z) can be found by following the zero path of the homotopy mapping (z, A) = O. Usually, one can choose g(z) as an
artificial function with an easy solution. Here, we specifically consider Fixed Point Homotopy which
chooses g(2)
Zy-
(I) where 20 E R is fixed vector, and is the initial point of the homotopy continuation method. In one practical trick, we can follow the zero path by allowing both x and À to be functions of an independent variable s, which represents are length along the path. In other words, (=(s), A(s)) is the point arrived at by traveling a distance 8 along the zero path from the initial point (=(0), A(O)) = (zo, 0). In the zero path, we have (= A(s)) = 0, for all 2
-2 - Ar(=) + () 20),
Take the derivative for this equation with respect to s lead to: (H A) Oz
d=
(H X) dd
ds +
ds O.
(2)
0%
The vector dd=_ dA) E TR++1 is the tangent vector to the zero path, and it lies in the null space of matrix (ff(z.A) (H(().)) E [x To complete the definition de a normalization 0X
ds dA
dz
condition is imposed to fix the length of the tangent vector, i.c
dz
dd
11
ds
ds
Given the tangent vector and the initial point, we can trace the zero path and obtain the solution of F(2) = 0 by solving the ODE (
4 Bridging DEQs& & Neural ODEs via Homotopy Continuation
Here we briefly review DEQs and Neural ODEs, then bridge these two models via homotopy continu- ation. DEQs aim to solve the equilibrium point of the function f(; x 8), which parameterized by 0 and the input injection æ. The underlying equilibrium-poine point- finditg problem of DEQs is defined as follows:
x' = fa, 0).
Usually, we choose (=;;r, 0) as shallow stacked neural layer or block. Hence, the process of solving the equilibrium point can be viewed as modeling the "infinite-depth" representation of a shallow stacked block. One can any black-box root-finding solver fixed point iteration to obtain the equilibrium point =
Unlike the underlying equilibrium- crrrrriiimmmiiitt-iieee point- problem of DEQs, Neural ODEs view its underlying problem as an ODE, whose derivative parameterized by the network. Specifically, Neural ODEs map data point x into set of features by solving the Initial Value Problem 22 to some time T. The underlying ODE of Neural ODEs is defined as follows:
dt) = ((=(t),t;)) 2(tn) = x.
(5)
dt
where z(t) represents the hidden state at time f, (= 0) is neural networks parameterized by 0.
The same coin. DEQs apply Newton's methods solve the underlying equilibrium coltllum---oot--fiiiii finding problem = =f(;; xr 0). By defining r(2) = 2 f(2; z, O). one can alternatively solve this problem based on homotopy continuation. Now we will show that the problem.
coullibruum-poin-fiiie
underlying dynamics in Neural ODEs can also be treated as gquilibrium-poin-ffiie
Firstly, we apply the Fixed Point Homotopy solve the cowaliibruumm-iinn-fnnnii problem = = f(z; 0) and obtain the homotopy mapping (2, A) A(z f(2;0) + (1 A)z. Taking the partial derivative of (2, A) with respect to z and X, respectively, we obtain
0H 0H A)
Av)((z;0) (H(
-2(;8).
(6)
Oz
a

---

By substituting the partial derivative in 66 into we can obtain:
dz
dx
ds
ds
Based on the normalization condition equation:
we can reformulate
as the following differential
dz = (1 -1-
d= ds
(8)
ds
As Neural ODEs do, ODE (8. However, the norm of neural network output likely to exceed the unit length, i.e. violating the normalization condition (3.. To address this issue, we introduce v == dt as the velocity of the point (z, A) traveling along the zero path, and modify the normalization condition by introducing into
neural networks approximate the underlying dynamics of such an
dz
+ dd
=².
(9)
dt
dt
Note that the convergence of the homotopy continuation is not affected by the value of v. The underlying dynamics of (7) becomes
d=
dz
- -T- ([T=)))V=(::00)))((((
(10)
dt
dt
Following Neural ODEs, the differential equation (TO) can be approximated by neural networks, i.e. ds F=)(t) 0). However, we still need to ensure the existence of corresponding equilibrium- di point-finding problem for Neural ODE, Ingeniously, the modified normalization can also ensure the existence of the cetliiiriumm-oiin-fffitefe the neural networks F(z(t), t; .), we can compute the changing process of X(t) and the velocity by solving the following equations:
problem. When we obtain the Neural ODEs (5) by training
dA
= 02 (t; x(0) = 0, X(1) =
(11)
dt
Note that the modified normalization condition 9 provides the dynamic with another degree of freedom, which guarantees the existence A()). Otherwise, there might be no solution for A()) as there are two initial conditions (i.e., A(0) = 0, A(1) 1) for the system. Hence, the equilibrium- point-finding problem z = f(2; 9) is implicitly determined by the following partial differential equation:
F F2((),t:0) = (1 XOV=/(=:)) (F00011((((((())01) 0)1²
(12)
Therefore, Neural ODEs can be regarded as the procedure of solving an equilibrium-pont point-finding problem with homotopy continuation, and the hidden state at 0, =(4n) is the initial point of homotopy continuation.
Two sides. We have shown that both DEQs and Neural ODEs can be considered as solving citiibrium-pointffiddin problems through homotopy continuation. Now we discuss the difference in underlying qquilibruum-piint-fine finding problem between DEQs and Neural ODBs.
On the one hand, DEQs solve the problem from the same initial point 2n- The equilibrium- pointt finding problem of DEQs is parameterized by the input injection The underlying equilibrium-
point-finde fint-finding problem of DEQs is defined as follows:
z" f(z" xz 0), (0) 20
(13)
The input injection æ can be viewed as the condition to fuse the information of input to the underlying cantorruumm-oot-tiiniii problem. The underlying problem of DEQs varies depending on different conditions x. Therefore, DEQs are able to map inputs to diverse representations, which is crucial for achieving superior performance.
On the other hand, unlike DEQs, Neural ODEs solve an equilibrium-poine -oint--iiiitttn--inggg problem with different initial points =(lo). The underlying equilibrium-point- finding problem of Neural ODEs is defined as follows:
z f(z=00)
(14)
5

---

and Neural ODEs solve the problem through homotopy continuation:
dz(tt
(()(());;
(15)
dt
where f is determined by F in equation features z(to) æ and regard them as the initial points of the ODE. The fixed underlying problem ensures the stability of Neural ODEs but loses diversified representation capabilities. Therefore, we claim that DEQs and Neural ODEs are actually two sides of the same coin from the perspective of homotopy continuation.
12 Concretely, Neural ODEs map data into set of
5 HomoODE: an efficient and effective implicit model
=(4)) is equilibrium point of 2" f(=";0)
z((1) equilibrium point of f(z",2,0)
is equilibrium point of
x" f==;;, 8)
Neural ODE
=(t1)
HomoODE
2(11)
DEQ
dz
dz F(z(1),t;2,))
(()
f(= x2,",,,,
dt
Broyden Equation Solver
ODE Solver
ODE Solver
2(to)
condition
z(fo)
condition
Z0
Feature Extractor
Initial point
Feature Extractor
Initial point
Extractor
Connecting F(2) and f(z) ((=(),4zz,0 0) - (1- AV, 0) f( 0)
dd
Homotopy Continuation:
di
Figure 1: Comparison hetween different running mechanisms of implicit models.
As we show above, both DEQs and Neural ODEs can be considered as solving equilibrium-por pointtt- finding problems through homotopy continuation. Two well-known approaches for solving nonlinear equations are Newton's 's Method (2] and homotopy continuation [39]. DEQs solve an equilibrium- point-finding problem r(=) = via Newton's Methods which are local in the sense that a good estimate of the solution is required for the convergence. Unlike Newton's 's Method, homotopy continuation is global in the sense that solutions g(z) = 0 may not need to be anywhere close to the solution of r(2) 0 (IT) Inspired by the connection between DEQs and Neural ODEs, very natural thought is that we can apply homotopy continuation to solve the underlying nonlincar equation of DEQs. By replacing the equilibrium-poine point-pooodii problem f(2; 0) f(2; 0), we can obtain the following differential equation:
dz dt (I M(2) (()) f(;= 0)
dz dt
02
(16)
Hence, we proposed a new implicit model called HomoODE, which models an equilibrium problem [(2; x, 0) implicitly. Specifically, we employ neural networks to approximate the differential equation (16). In the design of the network structure, we introduce the condition x: into the underlying dynamic of HomoODE as DEQ does and obtain the output from the same initial point through the ODE solver Neural ODE does. In this way, HomoODE not only has the ability of diversified

---

representation of DEQs but also has the property of stable convergence of Neural ODEs. In addition. the time information t is not explicitly formulated by the dynamic of HomoODE (T6). So unlike Neural ODEs, HomoODE does not require the input of time information t. Figure iillstrates the structure of HomoODE as well as Neural ODE and DEQ. Besides, it worth noting that mentioned in (T6) is just an auxiliary variable for the theoretical analysis. Therefore, we do not need to include it as hyperparamette or compute its value in the forward pass because it has been implicitly contained in the neural network.
Forward Pass. In HomoODE, the raw data x is first input to a feature extractor g(æ; and then injected into an ODE solver. Suppose 2(t) represents the intermediate state of HomoODE, calculating z(t) involves an integration starting from the initial point =(ta) = 0 to the solution =(t2), ). Notably, the output z(t;) is equivalent to the solution 2" of the implicit qquilibriumppoiii findiig problem = f(2 x. 0). Then we can use the ODE solvers to obtain the solution =* of the origin equilibrium problem and this representation can be used for downstream tasks, such as classification, regression, etc.
2(1) OOESolle(316))
0), (1)
(17)
Backward Pass. In the backward pass of HomoODE, we can apply the adjoint sensitivity method, or straightly differentiate through the operations of the forward pass. The condition traces back to another gradient flow. More details related to the construction of HomoODE dynamics and the computation of the gradients based on the adjoint method can be referred to in the supplementary materials.
Acceleration for HomoODE
In practice, we observe that HomoODE needs more func- tion evaluations in the ODE-solving process, which results from bad initialization (c.g., zero vector initialization). To address this issue, we referred to [8), which makes an input-based initial point guess with an extra neural net- work to accelerate the equation-solving procedure in DEQ. However, this method poses extra cost in both memory and computation.
1000
relation tion
850
limit
700
z
550
400
250
To avoid these drawbacks, we further investigate the rela- tionship between the number of iterations and the distance from the initialization to the solution =' in homotopy continuation. Specifically, we perform the homotopy con- tinuation method on nontrivial nonlinear equation using ode45 in Matlab (28). As is shown in Figure 2 it does not bring any reduction in the number of iterations when the initial point is close enough to the equilibrium solution. This means it is unnecessary to approximate the specific initial point very accurately for specific sample x in HomoODE, and that is different from that in [8)
100
2
4
6
10
12
1/d
Figure 2: Relationship between the dis- tance from 20 to z' and the iteration number of ODE solver. The x-axis is the inversion of the distance 1/d, and the y-axis is the iteration number N.
Hence, we share the initial point for all the samples and just maintain a scalar value for the averaged value the whole feature map in one channel. Briefly, we just need to store a tensor with the shape of (1, 1, e) as the shared initial information Eg. and broadcast it into the initial point zo with the shape of (h, w. c) when taking it into the ODE solver. Here h, w, c means the height, width, and channel number of the feature map. The loss function of 20 is defined as follows:
((x)) := Eg-D (2* (x) 2
(18)
where = (ar) denotes the equilibrium solution of the sample x and D denotes the distribution of æ. In fact, this update on Eg is equivalent to maintaining the dynamic geometrical center of the cquilibrium points of all the samples.
7 Understanding Augmented Neural ODE by HomoODE
Augmented Neural ODEs, as a simple extension of Neural ODEs, are more expressive models and outperform Neural ODEs. Augmented Neural ODEs allow the ODE flow to lift points into the extra
7

---

dimensions to avoid trajectories crossing each other [2] However, more theoretical analysis of how and why augmentation improves Neural ODEs is lacking. Our work provides another perspective on understanding the effectiveness of Augmented Neural ODEs. Augmented Neural ODEs formulate the augmented ODE problem as:
hh(t) dt [(t)]
(h(t) [a(t)
h(O) a(0)
F(
.1;0),
x: 0
(19)
where a(l) E IR denotes point in the augmented part, and h(() E IR is the hidden state at time t.
Specifically, Augmented Neural ODEs track back the ODE flow recover the original input by using the hidden state h(t) and the time information t. The input z is the injection of the dynamics in HomoODE. The augmented part a(t) can be viewed the intermediate =(t) in HomoODE. In Augmented Neural ODEs, we can treat the recovered input * as the condition in HomoODE, improving its representation ability. Hence, Augmented Neural ODEs outperform Neural ODEs. However, the origin input r computed by Augmented Neural ODEs may not be accurate enough. This probably is the reason why the performance of Augmented Neural ODEs is not competitive to HomoODE.
8 Experiments
To confirm the efficiency of HomoODE, we conduct experiments on several classical image clas- sification datasets compare model with the previous implicit model, including DEQ (6), monDEQ [46], Neural ODE [10) and Augmented Neural ODE [12]. For ODE datasets, DEQs inherently are not designed handle simulation tasks of continuous physical processes DEQs do not involve the continuous time series in the models. Sequential datasets are not suitable as well due the inconsistent backbones. For sequential datasets, DEQ models are based on transformer, while Neural ODEs are based on fully-connected layers. Therefore, we consider the image classification task suitable for the fair comparison of HomoODE and baselines. Concretely, we evaluate the stability of the training process via the learning curve of accuracy in the test dataset and exhibit the performance of different implicit models in terms accuracy, memory consumption, and inference time. It is worth noting that we also perform HomoODE with & without the data augmentation and the adjoint backpropagation technique to check their impacts on our model. Besides, we also contrast HomoODE with zero vector initialization and learnable initialization assess the capability of our acceleration method. Our code is available at: https: : / tigghu...... com/vaadx209/hhooood
3x3 Conv2d
Com2dds
Condition
3x3 (x-))
3x3 (onv2d
GroupNorm
Group Normp Norm
ReLU
Dropout
=(t)
da
F(z(t),t;a ar
Figure 3: The deployed neural network architecture in HomoODE. Here, denotes the stride and the channel number of all convolutional layers is 64.
Experimental setup. HomoODE is performed on several standard datasets, including CIFAR- 34]. SVHN [38], and MNIST [35] As shown in Figure 3 HomoODE contains several simple convolutional layers. Its network structure is not specially designed for image classification tasks like MDEQ 7 Notably, the memory consumption of HomoODE, is less than that of other implicit models as reported in [46] 7.. As we discussed in Section the time information not fused into the input of HomoODE, unlike Neural ODE, Besides, optimize the shared initial information

---

20 using SGD optimizer with the learning rate 0.02 and perform the update once every 20 updates for HomoODE.
Comparison with former implicit models. Table 1 presents the performance of HomoODE with differ- ent settings and other implicit models in the CIFAR- 10 dataset. It can be observed that HomoODE out- performs the previous implicit models in terms of both accuracy and memory consumption. Moreover, the inference time of HomoODE is much faster than DEQ and its variants. Notably, we also find that DEQ runs slower with larger test batch sizes. This may be due to the inefficiency of DEQ for paral- lel computation with large test batch sizes. In con- trast, HomoODE does not exhibit this problem. Ad- ditionally, there is a large increase in accuracy in model when the data augmentation technique is ap- plied. It suggests that HomoODE learns better and more complex representations through augmented datasets. Furthermore, given that adopt the dopri5 ode solver with an adaptive step, this complex repre- sentation may result in increased NFE and inference time. Overall, these phenomenons validate that Ho- moODE has more powerful representation ability compared other implicit models with similar model capacity. Extensive experiments in SVHN and MNIST also confirm these properties of our model shown in Table 2
0.8
0.7
Test/cc
BEG
ANOCS
0.6
0.5
0.4
0
10
15
20
25
Epoch
Figure 4: Learning curve of different implicit models on CIFAR-10 datasets across 5 runs without data augmentation. The x-axis de- notes the epoch during training and the y-axis denotes the accuracy of models on the test datasets.
Besides, we also plot the learning curves of different algorithms Figure The results demonstrate the stability of training HomoODE compared with other methods and exhibit that HomoODE is not prone to over-fitting, whereas other bDE-b models may suffer from that.
Efficiency of the learnable initialization. Figure illustrates that the learnable initialization trick can improve HomoODE with about 2 speedup in the inference time than before. This impact is obvious in both cases with & without the adjoint backpropagation technique. In addition, the corresponding test accuracy during the training process also reflects this acceleration technique does not bring loss in the performance of our model. Surprisingly, it even brings a slight improvement in the case of the adjoint backpropagation technique. This probably because a good initial point can decrease the total length of zero path s, which reduces the gradient error induced by using the adjoint method.
Conclusion
In this paper, we show that both DEQs and Neural ODEs can be viewed as a procedure for solving an squilibrium-ponne finding problem via the theory of homotopy continuation. Motivated by this observation, we illustrate that these two implicit models are actually the two sides of the same coin. Specifically, DEQs inject the input information as the condition into the cettiibriuu-oiintffinee problem z* = [== xc 0) while Neural ODEs fuse the input information into the initial point. Further, we propose a novel implicit model called HomoODE, which inherits the advantages of both DEQS and Neural ODEs. Our experiments indeed verify that HomoODE outperforms both DEQs and Neural ODEs while avoiding the instability of the training process, that is often observed with DEQs. Moreover, we developed a method to speed up HomoODE and the ODE-solving operation by almost three times by using a shared learnable initial point. Overall, the experimental results several classical image classification datasets demonstrate the efficiency of HomoODE in terms of both accuracy and memory consumption.
Although this paper offers a brand new perspective on implicit models, we also want to highlight a limitation of this idea, Actually, we do not present explicit form of the equilibrium transformation function, which implicitly determined by a modified neural ODE, Besides, while HomoODE has powerful representation ability, the qquiiibrium-poin-soolle procedure of it is implicit, which

---

Method
Model size
Inference Time
Accuracy
80-
monDEQ
DEQ
172K
170K
5.8x
82.2 1 0.3%
zero no adi
no od
Neural ODE 10
172K
3.2x
55.3 + 0.3%
60
Aug. Neural ODE
HomoODE
T3
172K
1.7%
58.9 2.8%
NFE-
HomoODE
132K
1.0%
83.2 0..%
HomoODE
132K
0.2%
HomoODE"
132K
1.0%
88.4
0.1%%
30
Table 1: Performance of HomoODE compared to pre- vious implicit models on CIFAR-10. with data aug- mentation; with adjoint method. The inference time expressed as a multiple the inference time of Ho- moODE with the adjoint method and the inference batch size is 400. Each result is obtained with 5 ran- dom runs.
230
260
300
0.25
0.80
Teat
0.75
zero adj
26. od.
0.70
Dataset
Method
Model
Accuracy
DEQ 0
170K
93.6 = 0.3%
0.65
monDEQ
170K
92.4 = 0.1%
SVHN
Neural ODE 10
172K
81.0 0.6%
0.00
Aug. Neural ODE LP
172K
83.5 = 0.5%
HomoODE
132K
0.1%
0.58
100
150
200
250
300
DEQ 6
80K
99.5 0.1%
Figure 5: Learning curve of HomoODE with different settings on CIFAR-IO across 5 runs without data augmentation. The axis denotes the epoch during training. The y-axis (up) denotes the accuracy of models on the test dataset and the y-axis (down) denotes the NFE of them in the forward pass.
Epoch
monDEQ
84K
99.1 0.1%
MNIST
Neural ODE 10
84K
96.4 0.5%
Aug. Ncural ODE 12
84K
98.2 = 0.1%
HomoODE
0.1%%
Table 2: Performance of HomoODE compared to pre- vious implicit models on SVHN and MNIST. Each result is obtained with random runs.
weakens its interpretability. Hence, exploring interpretable approach for the forward pass and backpropagation of HomoODE is under consideration in our future work.
Acknowledgement
This work was supported by NSFC (No.62303319), Shanghai Sailing Program (22YF1428800 21YF1429400), Shanghai Local College Capacity Building Program (23010503100, Shanghai Fron- tiers Science Center of Kluman-centent Artificial Intelligence (ShangHAD, MoE Key Laboratory of Intelligent Perception and Haman--aachne Collaboration (ShanghaiTech University), and Shanghai Engineering Research Center of Intelligent Vision and Imaging.
References
[1] Ralph Abraham and Joel Robbin. Transversal mappings and WA Benjamin New York, 1967.
[2] Milton Abramowitz, Irene A Stegun, and Robert H Romer. Handbook of mathematical functions with formulas, graphs, and mathematical tables, 1988.
[3] Atish Agarwala and Samuel Schoenholz. Deep equilibrium networks are sensitive to initial- ization statistics. In International Conference Machine Learning, pages 136-160. PMLR, 2022.
[4] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference Machine Learning, pages 136-145. PMLR, 2017.
[5] Shaojie Bai, Zhengyang Geng, Yash Savani, and Zico Kolter. Deep equilibrium optical flow estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 620-630, 2022.
10

---

[6] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
[7] Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Multiscale deep equilibrium models. Advances in Neural Information Processing Systems, 15:5238-5250 2020.
[8] Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Neural deep equilibrium solvers, In International Conference on Learning Representations, 2021.
[9] Shaojie Bai, Vladlen Kolun, and Zico Kolter. Stabilizing equilibrium models by jacobian regularization. In International Conference on Machine Leurning, pages 554-565. PMLR,
2021.
[10] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[11] Tianran Chen and Tien- Yien Li. Homotopy continuation method for solving systems of nonlinear and polynomial equations. Communications in Information and Systems, 15(2):119 19-3077, 2015.
[12]
Emilien Dupont, Arnaud Doucet, and Yee Whye Tch. Augmented neural odes. Advances in neural information processing systems, 32, 2019.
[13] Chris Finlay. Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ode: the world of jacobian and kinetic regularization. In International conference on machine learming, pages 3154-3164. PMLR, 2020.
[14] Mare Finzi, Alexander Wang, and Andrew G Wilson. Simplifying hamiltonian and lagrangian neural networks via explicit constraints, Advances in neural information processing systems, 33:13880-18889 2020.
[15] Arnab Ghosh, Harkirat Behl, Emilien Dupont, Philip Torr, and Vinay Namboodiri. Steer: Simple temporal regularization for neural ode. Advances in Neural Information Processing Systems, 33:14831-14844,, 2020.
[16] Davis Gilton, Gregory Ongie, and Rebecca Willett. Deep equilibrium architectures for inverse problems imaging. IEEE Transactions on Computational Imaging, 7:1123-1133, 2021.
[17] Alexandros Gkillas, Dimitris Ampeliotis, and Kostas Berberidis. Connections between deep cquilibrium and sparse representation models with application to hyperspectral image denoising. IEEE Transactions on Image Processing, 32:1513-1522 2023.
[18] Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv: 180000.01 2018.
[19] Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in neural information processing systems, 32, 2019.
[20] Fangda Gu, Heng Chang, Wenwu Zhu, Somayeh Sojoudi, and Laurent El Ghaoui. Implicit graph neural networks. Advances in Neural Information Processing Systems, 33:1 1994- 2020.
[21] Wenrui Hao, Jonathan Hauenstein, Chi-Wang Shu, Andrew J Sommese, Zhiliang Xu, and Yong-Tao Zhang. A homotopy method based on weno schemes for solving steady state problems of hyperbolic conservation laws. Journal of Computationall Physics, 250:332-346, 2013.
[22] Wenrui Hao, Jan Hesthaven, Guang Lin, and Bin Zheng. A homotopy method with adaptive basis selection for computing multiple solutions of differential equations. Journal of Scientific Computing, 82:1 2020.
[23] Wenrui Hao and Chuan Xue. Spatial pattern formation in reaction- difffusooff models: compu- tational approach. Journal of Mathematical Biology, 80:521-543, 2020.
[24] Wenrui Hao and Chunyue Zheng. An adaptive homotopy method for computing bifurcations of nonlincar parametric systems. Journal of Scientific Computing, 82:1-19, 2020.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[26] Petr Hruby, Timothy Duff, Anton Leykin, and Tomas Pajdla. Learning to solve hard minimal problems. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 5532-5542, 2022.
11

---

[27] Yao Huang, Wenrui Hao, and Guang Lin. Hompinns: Homotopy physics-informed networks for learning multiple solutions of nonlinear elliptic differential equations. Computers & Mathematics with Applications, 121:62-73, 2022.
informed neural
[28] The MathWorks Inc. Matlab version: 9.13.0 (r2022b), 2022.
[29] Edward L Ince. Ordinary differential equations. Courier Corporation, 1956.
[30] Kenji Kawaguchi. On the theory of implicit deep learning: Global convergence with implicit layers. In International Conference on Learning Representations (ICLR), 2021.
[31] Jacob Kelly, Jesse Bettencourt, Matthew Johnson, and David K Duvenaud. Learning differ- ential equations that are casy to solve. Advances in Neural Information Processing Systems, 33:4370-4388, 2020.
[32] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-6707, 2020.
[33] Joon-Hyuk Ko, Hankyul Koh, Nojun Park, and Wonho Jhe. Homotopy-based training of neuralodes for accurate dynamics discovery. arXiv preprint arXiv:2210.01400) 2022.
[34] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[35] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 16217)2227822324 1998.
[36] Takashi Matsubara, Yuto Miyatake, and Takaharu Yaguchi. Symplectic adjoint method for exact gradient of neural ode with minimal memory. Advances in Neural Information Processing Systems, 34:20772-20784, 2021.
[37] Mohammed Ali mnmoustafa. Tiny imagenet, 2017.
[38] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[39] Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999.
[40] Avik Pal, Alan Edelman, and Christopher Rackauckas. Continuous deep equilibrium models: Training neural odes faster by integrating them to infinity. arXiv preprint 2rXhv:2201.12240 2022.
[41] Ashwini Pokle, Zhengyang Geng, and Zico Kolter. Deep equilibrium approaches to diffusion models. Advances in Neural Information Processing Systems, 35:37975-37990 2022.
[42] Lev Semenovich Pontryagin. Muthematical theory of optimal processes. CRC press, 1987.
[43] Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.04488: 2020.
[44] Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems, 32, 2019.
[45] Haixiang Sun, Ye Shi, Jingya Wang, Hoang Duong Tuan, Vincent Poor, and Dacheng Tao. Alternating differentiation for optimization layers. International Conference on Learning Representations, 2023.
[46] Ezra Winston and Zico Kolter. Monotone operator equilibrium networks. Advances in neural 2020.
information processing systems,
[47] Xingyu Xie, Qiuhao Wang, Zenan Ling, Xia Li, Guangcan Liu, and Zhouchen Lin. Opti- mization induced equilibrium networks: An explicit optimization perspective for understand- ing equilibrium models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4564):3604-3616, 2023.
[48] Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Tbrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dy- namies forecasting. Journal of Statistical Mechanics: Theory and Experiment, 1024(12) 2021.
12

---

[49] Weiming Zhi, Tin Lai, Lionel Ott, Edwin V Bonilla, and Fabio Ramos. Learning efficient and robust ordinary differential equations via invertible neural networks. International Conference Machine Learning, pages 27060-270777 PMLR, 2022.
[50] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. In International Conference Learning Reprexentations.
[51] Juntang Zhuang, Nicha Dvomek, Xiaoxiao Li. Sekhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ode. In International Conference Machine Learning, pages 1 11639-11 11649. PMLR, 2020.
13

---

A Background of Homotopy Continuation
A)g(z) provides a continuous transformation by gradually deforming g(2) into r(z) while increases from 0 to in small increments. The solution r(2) can be found by following the zero path of the homotopy mapping (2 X) = Usually, one can choose an artificial function g(2) with an easy solution. Figure 6 shows the transformation of
Homotopy mapping (z, Ar(z)
homotopy mapping with À increasing from 0 to the homotopy function goes from an artificial,
"easy" problem to the nonlinear problem in which we are interested.
=
0.4
A = 0.7
==1
Figure 6: Transformation of homotopy mapping along with X.
A.1
Global Convergence of Homotopy Continuation
Here, we briefly recall the theoretical foundation of homotopy methods to show its global convergence with probability one.
Let E" denote re-dimensional real Euclidean space, let U C E" and V En be open sets, and let H. : U x V 10. - E" be a C² mapping. H is said to be transversal to zero if the Jacobian matrix V has full rank on H 10).
Definition
Theorem 1. (Parametrized Sard's Theorem (TT) If H 2, X) is transversal zero, then for almost all 2g E U the mapping
HoO (z, X) H (z0,2,, 2,
(20)
is also transversal to zero; ie., with probubility one the Jucobian matrix V (h, has full rank on H (0)
The method for constructing a homotopy algorithm to solve the nonlinear system r(2) 0 with global convergence is as follows: 1) H (20, 2, A) is transversal to zero; 2) H (z. 0) = H( 2, 0) is
trivial to solve and has a unique solution 20: 3) H (z, 1) = r(2): 4) H (0) is bounded.
Then for almost all zg E U there exist zero path s of Hz. along with the Jacobian matrix V H has rank n. The zero path starts from (0, =n) and reaching =* at X =1 This zero path does not intersect itself, and disjoint from any other zero paths of Hz Furthermore, Vr(z) nonsingular, then the zero path has a finite arc length.
A.2 Fixed Point Homotopy Continuation
One commonly used homotopy function to find solutions of r(2) is the Fixed Point Homotopy given by:
H A) >r(2)) + (1 A)(z 20),
(21)
where 2p E R and in unit interval [0. 1]. At À 0, the starting system is H(2, 0)
2p 0
for which the only solution is = zg. At = 1, the system (=,, 1) = r(=) = 0 is the system of equations of interest.
For C "R"" we use int to denote the interior U. And we say is boundary-free at Ao E [0, 1] if
z & OU for any z H| ({0}). Generally, we say is boundary-free for subset C [0, H is boundary-free for all À E S. The following theorem provides the fixed point of f() under the existence of Fixed Point Homotopy.
14

---

11 which is defined as Theorem 2. (Fixed Point Theorem (II)) Given smooth function f U - R", let U E R be compact and intU * ②. For some 20 intU, H [0, + R boundary-fred for
OSAS10 where
H(2, X(= f(=)) +1 A)(= 20).
(22)
f haas fixed point, i.e. there exists an z E U such that f(=*) = z".
A.3 Newton Homotopy
Another commonly used homotopy function is the Newton homotopy follows:
Hz,, Ar(2) (1
(23)
= -1 (1-)))(a)))
-1
where R IR is the smooth system of interest, and 2g is generically chosen point in R
Notably, there is close connection between the Newton homotopy and the well-known Newton's method [2] for solving nonlinear equations. Given Vr(z) is nonsingular, we apply the differ- entiation on the zero path of the homotopy mapping, i.e. Hz, A) = 0, yielding the initial value problem:
dz
- (Vr(2())))
dA z(0) Zu-
(24)
Applying Euler's method at À = 0 with step size to the above ODE (24) from the initial point 2(),, the approximation next iteration x1) becomes:
211) = 20) - (Vr(2(0)))"
(25)
Apparently, (25 is a single iteration of Newton's method. Hence, Newton's iteration can considered as the application of Euler's method with step size I on the solution curve given by the Newton homotopy. However, in contrast to Newton's method, which is generally a local method, the Newton homotopy exhibits certain global convergence properties.
v=0.1
v=1
v=10
v=100
6.5
6.5
6.5
6.5
5
5
5
5.5
5.5
5.5
5.5
5
5
5
4.5
4.5 0
4.5
4.5
0.5
1
0.5
1
0
0.5
0.5
A
A
X
À
Figure Iterative curve of homotopy continuation under different
B The impact of different values of velocity v for the homotopy continuation
ds the velocity of the point (2, traveling along the zero path, In Section we introduce v := de and modity the normalization condition by introducing v into (3). Here we demonstrate that the convergence of the homotopy method is not affected by such modification by numerical experiments.
We choose a nontrivial nonlinear equation and solve it via homotopy continuation under different velocities v. Figure shows the zero paths of homotopy continuation under different velocities Apparently, the zero paths are all on the same trajectory and heading for the same solution. Hence, the convergence of the homotopy method doesn't 't depend on the change in the value of velocity. With modified normalization condition, the underlying dynamics become:
dz
dz
= -T
02
(26)
dt
15

---

Following Neural ODEs, we can employ neural networks to approximate the differential equation above. The existence of corresponding equilibrium point-finding problem for Neural ODE is also guaranteed by the modified normalization condition. Specifically, one can refer to (11) (2). Hence, Neural ODEs can be regarded as the procedure of solving an equiliriuumppoin--nnee finding problem with homotopy continuation.
C Bridging DEQ and Neural ODE from another type of Homotopy method
We have established the connection between DEQs and Neural ODEs via Fixed Point Homotopy in Section 4 Here, we also show that we can show a similar connection using another type of Homotopy method, L.e., Newton Homotopy.
By defining r=z) = z - [zz 0), and setting =0 for simplification, from 23 we have:
((2zA)) = z - f(; 0) + (1 - x)
(27)
Different from Fixed Point Homotopy. we can associate z directly with without introducing v. From 24 have:
dz dA = (T- =
=(0) = 0.
(28)
Here we can view as the time information for the differential equation, and x(1) the solution of the corresponding equilbrium-ppint-ffndnng problem when À = 1. We can also employ neural
networks to approximate the differential equation as Neural ODEs do. Therefore, Neural ODEs
can also be regarded as the procedure of solving gquilibruum-poin-ffnddii problem via Newton Homotopy.
D Stability of HomoODE
This section demonstrates the stability of HomoODE based on Picard-Lindeof Theorem 29
Theorem 3. (Picard-Linneëßf Theorem (29) Let T: [a, b] be an interval, let f : IR + R be function, and let
(t) 2tt)
(29)
be the associated ordinary differential equation. If is Lipschitz continuous in the second argument 2, then this ODE possesses unique solution on [a, a fe each possible initial value z(0) = E R", where +. I. the Lipschitz constant of the second argument of J.
HomoODE solves the equilibrium-poin finding problem implicitly using modified Neural ODE via homotopy continuation. Hence it also has the stability of Neural ODE, which can be explained by the Picard- Lindeöf Theorem. Assuming the underlying dynamic of HomoODE is Lipschitz continuous in 2, then both existence and uniqueness can be guaranteed by the Picard-Lindeöf Theorem.
E Accelearation for HomoODE
In Section it is mentioned that the update on the shared initial point 2g is equivalent to maintaining dynamic geometrical center of the equilibrium points all the samples. We will show the correctness of this proposition and illustrate the relationship between the learning rate and the step of the dynamic update.
The update on the shared initial point zg is equivalent to maintaining dynamic geometrical center of the equilibrium points of all the samples.
Proposition 1.
Proof. Recall that the loss function of is defined as L(zo) == E.nD (2*(x) 105-2p1 20)3]. According to the definition of the variance, we have
Var(=* - =0) E (=*(x) - -00) E [xx²(x)) -20]²
(30)
16

---

Define := (x)) we obtain
E
(x²) - 2n)²
= -Var (x*(2)) =o) +[(==(a) z0
Var (="()) z0) + =
(31)
= Var(=*(z)) + (z* - 220)
Here Var (x)) is irrelevant to 2p- Hence, mingo
(2 20)²] equivalent to minimizing
millz, (2* - =o) which means the update 2o is equivalent to maintaining a geometrical center of the equilibrium points of all the samples.
According to Proposition 1, we also write the update the form of 2g a²* + (1 a-)=0 which more straightforward. Therefore, we also want to further explore the relationship between the learning rate Tiiit and update step Consider the gradient descent 20
= - 1/ EODD [ (z*(x)) =00)²
= 20 + 2n (3" (z) - 2n)
(32)
2n (x) + 2n)) zi.-
Since zo is the broadcast tensor from initial information Eg. the actual gradient descent has the form of
2
2
E0 hw Thmi 2² (x) + (1
hw Mmmn) Z0-
(33)
where h, w are the height and width of the feature map respectively. Finally, we obtain the relationship between Moxt and is Miiit hw a. This means we can set large learning rate for Thnit even greater than 1, especially when the feature map is large.
F Adjoint Method for HomoODE
Algorithm 1 Adjoint Method for HomoODE
Input: initial point 2to)), condition z, parameter 6, start time tg, stop time t OL
so = [=(tt), Ba(t,) Og 0g]
>> Define initial augmented state
def aag_auygamaaa (og__dnnaiis(((())) ((1,)())()),,,,1..)))))
> Define dynamics on augmented state Compute Jector-Jacobian products
return (F(=(t), 1; 0); aat) T
0 -a() Ter - 00
[a(tu), Bz(fa) 0z 80
OL OL AL
ODESolve(so: aug dynamics; to; 0)
>> Solve reverse-time ODE > Return gradients with respect to a and
return OL OL 0z 09
The adjoint method 10 is an efficient backpropagation method that can save the memory footprint in Neural ODE during training. However, there is minor difference when apply the adjoint method in the training of HomoODE. Unlike Neural ODE which computes the gradient with respect =to)), our HomoODE calculates the gradient with respect the condition æ instead. Accordingly, we modify the adjoint method in Neural ODE and present the computation procedure in Algorithm
G Addiioona Experimental Results
We conducted additional experiments in CIFAR-100 34 and Tiny ImageNet [37 to validate the potential of applying HomoODE to difficult image classification tasks with larger model sizes. Specifically, we extend the channel numbers of the convolutional layers to 128 in HomoODE and increase the model size of the compared implicit models correspondingly for faimesss. The experiments CIFAR- 100 and Tiny ImageNet are all implemented with data augmentation. The concrete operations in data augmentation involve zero- prro-padding the 32x32 images to 40x40 and then performing random horizontal flips. As shown Table 4 the performance of HomoODE is much better than other implicit models in terms of both memory consumption and test accuracy.
17

---

Method
Model size
Accuracy
DEQ 6
770K
64.5 0.7%
monDEQ 46
1M
59.8 + 0.3%
Neural ODE
10
874K
31.7 0.6%
Aug. Neural ODE
12
857K
36.2 1 0.9%
HomoODE
565K
69.30 0.1%
HomoODE
565K
71.57 0.2%
Table 3: Additional experiments CIFAR- HomoODE and HomoODE represent the training with and without the adjoint method, respectively. The experiments are all implemented with data augmentation.
Method
Model size
Accuracy
DEQ
810K
47.6 0.4%
monDEQ
46
783K
24.83 + 0.1%
HomoODE
617K
54.14 + 0.2%
HomoODE
617K
56.1 0.1%
Table 4: Additional experiments on Tiny-ImageNet. HomoODE and HomoODE represent the training with and without the adjoint method, respectively. The experiments are all implemented with data augmentation.
Besides, we also performed the ablation study on the learning rate for the shared initial point as shown in Figure Notably, the ablation study is implemented with the adjoint method. Given the practicality the adjoint method in handling the backpropagation of large deep models, we are inclined to explore wider range of characters when utilizing It can be observed that HomoODE robust to the learning rate for the shared initial point and there exists a slight improvement when Ir 0.1. We believe the reason is the same the slight improvement in contrast experiments of HomoODE with/without acceleration. That is, the larger learning rate for the shared initial point makes the initial point =o closer to the current geometrical center of the equilibrium points of all samples, which further leads to the shorter zero path that the adjoint method goes through. Finally, reduces the error of backpropagated gradient.
H Hyper-parameter Settings
Our experiments are implemented on GPU NVIDIA GeForce RTX 3090 with 24GB. The hyper-parameters applied in HomoODE are shown in Table Experiments related to other implicit
28
0.9
25
0.8
24
NFE.F
Text Apc
23
22
1e-1
0.6
1e-1
21
I 10-2
1e-2
1e-3
1e-3
20
10-4
10-4
100
150 Epoch
200
250
300
50
150
200
250
300
Epoch
Figure Ablation study on the learning rate for the shared initial point.
18

---

models are based on deg monotone _o__ net and nngmented-neuralee neural-odes Notably, the dropout because the traditional
layer in our experiments follows the variational dropout operation dropout operation hurts the stability of convergence to the equilibrium.
Parameter
MNIST
SVHN
CIEAR-10
CIFAR-100
Ratch Size
64
64
64
64
Optimizer
Adam
Adam
Adam
Adam
Learning Rate
0.001
0.001
0.001
0.001
Prequency of Tnitial Point Update
20
20
20
5
Optimizer for Initial Point
SGD
SGD
SGD
SGD
Learning Rate for Initial Point
0.02
0.02
0.02
0.01
Variational Drogout
Number of Channels
32
64
64
128
Absolute Tolerance for ODE Solver
1E-3
1E-3
1B-3
1E-3
Relative Tolerance ODE Solver
IE-3
IF-3
IF.3
IE.3
Table 5: parameerrsseterss used in HomoODE under different image classification tasks.
I Details on Tested Equation in Section 6
With respect to the details on the tested nonlinear equation mentioned in Section 6 we randomly choose equation with high nonlinearity that is
fa) 2x exp(-0.1x) + 5sin(4r)- 16 0,
and one solution of it is 6.4217. Our experiment shown in Figure 2 is based on the initial points
around this solution. Actually, we found that even if we modify the parameters in this function arbitrarily, similar result can be obtained as well.
J More Discussions on Learnable Initial Point
is worth noting that we also carried out some trials with an input-based initial point predictor [ to accelerate HomoODE. However, the performance of the initial point predictor in HomoODE is not desirable in terms of both inference time and test accuracy. Based on the comprehensive analysis we provided in Section B C this is probably because too frequent or too large updates on the initial point will destroy the stable link between ODE function F(z(t), f; xz 9) and the equilibrium transformation function x2; 0). In particular, the additionally-introduce. variable will change sharply when large change is applied to the initial point. In this case, the function X(t) will change dramatically, eventually ruining the trained equilibrium transformation function f(z; xz, 0). This phenomenon can also be illustrated from the perspective of Newton homotopy. Note that the ODE function is actually
determined by the initial point. Therefore, once the initial point is changed, the ODE function will change and then the trained ODE function will be ruined.
http::///iithhbbcc///uuusss////
19


Reviews:
[{"review_id": "A8UvDamCjG", "reviewer": "Reviewer_rMvZ", "summary": "The work shows the connection between Deep Equilibrium Moldes (DEQs) and Neural Ordinary Differential Equations (NODEs) using homotopy continuation, a method to solve nonlinear equations. Based on the analysis, the authors suggest a new implicit model, HomoODE, which has high expressible power from DEQs and stability from NODEs. The method is demonstrated to obtain the best performance for image classification tasks between the models tested.", "questions": "* How does one determine $v$? Is it a hyperparameter or computed using some equation?\n* What do \"explicit\" and \"implicit\" mean in, e.g., \"unlike DEQs, which explicitly solve equilibrium-point-finding problems via Newton's methods, our HomoODE solves these problems based on homotopy continuation implicitly\"? It could be unclear and require some explanation because the meaning of \"implicit\" seems different from that of \"implicit NN.\"\n* What are \"two initial conditions\" in \"there might be no solution for $\\lambda(t)$ as there are two initial conditions for the system\"?\n* What do $h$, $w$, and $c$ stand for in line 265?\n* Why is the inference time of HomoODEs with data augmentation longer than those without data augmentation in Table 1 (e.g., HomoODE vs. HomoODE$^*$, or HomoODE$^\\dagger$ vs. HomoODE$^{*\\dagger}$)? Is it due to increased iterations due to training with the augmented dataset?\n* Could you please elaborate on the meaning of \"fixed condition\" in \"Neural ODEs solve an equilibrium-point-finding problem with fixed condition $x$ and different initial points $z(t_0)$\"? The statement seems to contradict because Equation (5) states that $z(t_0) = x$, so different $z(t_0)$ implies different $x$. In addition, Neural ODE can take $t$ as an input, which evolves in time.", "limitations": "* The method to optimize the shared initial information seems tricky and could be stated as a part of the limitation.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The connection between DEQ and NODE is presented clearly and seems mathematically solid, which delivers a lot of insights to the community (in particular, in the field of implicit models).\n* In addition to mathematical notions, the paper covers practical ideas, e.g., the learnable initial condition. The effect of the adjoint method is also investigated.", "weaknesses": "* The reason why the authors choose image classification is not very clear. With the current manuscript, the practical contribution of the paper looks quite limited because DEQ is designed for sequential data, and NODE is for ODEs. Thus, the reviewer cannot assess the effectiveness of HomoODE because image classification is not a main field of the baseline methods. The reviewer strongly recommends the authors run experiments on ODEs to show real improvement compared to DEQ and NODE. \n* Related to the first point, the reason why the HomoODE works better in the image classification task is not clear. The reviewer did not find out what the authors would like to state with these experiments. I believe there are a lot of excellent models to compare in the field of image classification if the authors would like to claim the model's effectiveness in the image classification tasks. It could be understandable if the image classification task is to show the adaptability of the model, and there is another main experiment (e.g., on an ODE dataset).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the Reviewer rMvz for the detailed and constructive comments. In the following, we have provided an item-by-item response to the comments.\n\n> **Q1**: The reason why the authors choose image classification is not very clear. The practical contribution looks quite limited because DEQ is designed for sequential data, and NODE is for ODEs. The reviewer strongly recommends the authors run experiments on ODEs to show real improvement compared to DEQ and NODE. \n\n**A1**: Thanks for the comments. Although DEQs were first designed for sequential data and Neural ODEs were for ODEs, image datasets have been frequently used in the subsequential works [7, 43, 10, 12, R1, R2, R3] of DEQs and Neural ODEs. The main target of this paper is to establish the connection between DEQs and Neural ODEs via homotopy continuation. **Choosing suitable datasets for both DEQs and Neural ODEs is important**. Since ODE datasets are not suitable for DEQs, so we cannot compare on this dataset. Sequential datasets are not suitable as well due to the inconsistent backbones. For sequential datasets, DEQ models are based on transformer, while Neural ODEs are based on fully-connected layers. For image datasets, both DEQs and Neural ODEs use standard convolutional layers. Therefore, we consider image classification datasets as a reasonable and appropriate choice. \n\n[R1] Li, Mingjie, et al. \"Optimization inspired multi-branch equilibrium models.\" ICLR 2021.\n\n[R2] Pokle, Ashwini, Zhengyang Geng, and J. Zico Kolter. \"Deep equilibrium approaches to diffusion models.\" NeurIPS 2022.\n\n[R3] Zhuang, Juntang, et al. \"Adaptive checkpoint adjoint method for gradient estimation in neural ode.\" ICML 2020. \n\n> **Q2 (1)**: Why the HomoODE works better in the image classification task is not clear. \n\n**A2(1)**: We believe the superiority of HomoODE comes from that HomoODE inherits the property of high accuracy from DEQs and avoid the instability issues of DEQs. Specifically, the structure of HomoODE enable it to solve the equilibrium point through the ODE solver, so it inherits the expressible representation capability from DEQs, and the property of stable convergence from Neural ODEs (line 224-227). The performance of DEQs is compromised by instability. Therefore, after handling the instability issue of DEQs, the performance of HomoODE improves accordingly. \n\n> **Q2(2)**: The reviewer believes there are a lot of excellent models to compare for image classification if the authors would like to claim the model's effectiveness in this task. It could be understandable if the image classification task is to show the adaptability of the model, and there is another main experiment (e.g., on an ODE dataset). \n\n**A2(2)**: We would like to clarify that we do not want to claim the model's effectiveness and adaptability, specifically in image classification tasks. Instead, we want to bridge DEQs and Neural ODEs via homotopy continuation. Therefore, we need to choose suitable datasets for both DEQs and Neural ODEs. As we illustrated in **A1**, image classification tasks play a fair role for both DEQs and Neural ODEs and the ODE datasets are unsuitable for DEQ models.  \n\n\n>**Q3**: How does one determine $v$? Is it a hyperparameter or computed using some equation? \n\n**A3**: $v$ is neither a hyperparameter nor computed using some equation. Instead, it is an auxiliary variable for the theoretical analysis to bridge the DEQs and Neural ODEs via Fixed Point Homotopy (line 178-184).  \n\n>**Q4**: What do \"explicit\" and \"implicit\" mean in \"DEQs explicitly solve equilibrium-point-finding problems via Newton's methods, our HomoODE solves these problems based on homotopy continuation implicitly\"? Require explanation since \"implicit\" seems different from \"implicit NN\".  \n\n**A4**: The \"implicitly\" in this sentence is indeed different from that of \"implicit NN\" as DEQs and Neural ODEs. The \"explicit\" in this sentence means DEQs solve the underlying equilibrium-point-finding problem $z^* = f(z^*;x,\\theta)$ explicitly via Newton’s methods. In HomoODE, we cannot directly obtain the form of its underlying equilibrium-point-finding problem $f(z;x,\\theta)$, so we claim \"our HomoODE solves these problems based on homotopy continuation implicitly”. We will add these analyses in the final version. \n\n>**Q5**: What are \"two initial conditions\" in \"there might be no solution for $\\lambda(t)$ as there are two initial conditions for the system\"?\n\n**A5**: The two initial conditions are $\\lambda(0) = 0$ and $\\lambda(1) = 1$ (Equation 11), which come from the theory of homotopy continuation.\n\n>**Q6**: What do $h$, $w$, and $c$ stand for in line 265?\n\n**A6**: Here $h, w, c$ means the height, width and channel number of the feature map. We will add a more detailed description for them in our final version.\n\n>**Q7**: Why is the inference time of HomoODE with data augmentation longer than those without data augmentation in Table 1 (e.g., HomoODE vs. HomoODE$^*$, or HomoODE$^\\dagger$ vs. HomoODE $^{\\dagger*}$)? Is it due to increased iterations due to training with the augmented dataset?\n\n**A7**: Yes, the number of function evaluations (NFE) of HomoODE increased with the augmented dataset. HomoODE learns better representations through augmented datasets, resulting in increased NFE and higher test accuracy, which leads to longer inference time.  \n\n>**Q8**: Elaborate the meaning of \"fixed condition\" in \"Neural ODEs solve an equilibrium-point-finding problem with fixed condition $x$ and different initial points $z(t_0)$\"? The statement seems to contradict .... \n\n**A8**: Thank you for pointing out this. There is a typo in this sentence, and it should be \"Neural ODEs solve the fixed equilibrium-point-finding problem with different initial points $z(t_0)$\". There is no condition in Neural ODEs, and here we want to state the underlying equilibrium-point-finding problem of Neural ODEs does not vary with the input. We will modify this statement in the final version. "}, {"review_id": "30v5e0sXtU", "reviewer": "Reviewer_aBa7", "summary": "The paper uses homotopy continuation to show that Deep Equilibrium Models and Neural ODEs are effectively equivalent. They use the best properties of both worlds: DEQs for their higher accuracy and Neural ODEs for their stability to propose a new implicit method HomoODE.", "questions": "See the section on weaknesses", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Theoretical explorations of the relationship between Neural ODEs and DEQs are limited, and the paper does a good job of linking the two ideas.", "weaknesses": "* Reformulating DEQs as Neural ODEs have been done in a prior work [1], which has not been mentioned or compared against\n* MNIST and SVHN are too small-scale experiments and can lead to incorrect extrapolations on the method's benefits. I would recommend including larger problems like ImageNet.\n* Some information on the baseline would be useful: Typically, solving discrete dynamical systems (DEQs) is faster than a continuous dynamical system. So the 8x inference time might be an artifact of poor solver choice for the DEQ.\n\n[1] Pal, Avik, Alan Edelman, and Christopher Rackauckas. \"Continuous Deep Equilibrium Models: Training Neural ODEs faster by integrating them to Infinity.\" arXiv preprint arXiv:2201.12240 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your valuable feedback and constructive comments! We itemize the weaknesses or comments you mentioned and answer to them.\n\n> **Q1**: Reformulating DEQs as Neural ODEs have been done in a prior work [R4], which has not been mentioned or compared against\n\n**A1**: Thank you for your constructive suggestion. We have read this paper carefully and will cite it in our final version. However, we would like to point out that the goal and the studied problem in our work is different from those in [R4]. Our work attempts to establish an inherent connection between **general DEQ** and **general neural ODE** via homotopy continuation and then develop the intermediate model HomoODE to confirm our theory, while [R4] aims to reduce the computational expense of DEQ in training and inference via utilizing an **\"infinity time\" neural ODE**. \n\nIn other words, our paper is for the theoretical analysis of the **inherent connection** rather than just reformulating DEQs or Neural ODEs to a special Neural ODE or DEQ to improve some aspects of their performance. \n\nBesides, based on the performance reported in [R4], HomoODE achieves better accuracy than Skip DEQ and Skip Reg. DEQ with even smaller model size. Here is the comparison in CIFAR-10 dataset. \n\n|   Method          |  Moel Size |   Accuracy    |\n|:-----------------:|:----------:|:-------------:|\n|   Skip DEQ [R4]    |  200K      |   82.0 ± 0.3% |\n| Skip Reg. DEQ [R4] |  164K      |   81.1 ± 0.3% |\n|**HomoODE (adj.)** |  132K      |  88.4 ± 0.1%  |\n|   **HomoODE**     |  132K      |   90.1 ± 0.2% |\n\n\n[R4] Pal, Avik, Alan Edelman, and Christopher Rackauckas. \"Continuous Deep Equilibrium Models: Training Neural ODEs faster by integrating them to Infinity.\" arXiv preprint arXiv:2201.12240 (2022).\n\n> **Q2**: MNIST and SVHN are too small-scale experiments and can lead to incorrect extrapolations on the method's benefits. I would recommend including larger problems like ImageNet.\n\n**A2**: Apart from MNIST and SVHN, **we did conducted experiments on CIFAR-10 and CIFAR-100** (Please refer to Appendix G). All these experiments show the superiority of our HomoODE. Due to time limitations, the experiments on ImageNet are still in progress. **We chose the Tiny-ImageNet dataset, a subset of ImageNet, as supplementary experiments during the rebuttal phase. The results shown in the following table also verify the superiority of our model HomoODE.** \n\nThe following table shows the result of experiments on Tiny-ImageNet. \n|  Method          | Model Size  |    Accuracy     |\n|:------------------:|:-----------:|:---------------:|\n| DEQ                | 810k        | 47.60 ± 0.4%    |\n| monDEQ             | 783k        | 24.83 ± 0.1%    |\n| **HomoODE**        | 617k        | 56.1 ± 0.1%     |\n| **HomoODE (adj.)** | 617k        | 54.14 ± 0.2%    |\n\n\n> **Q3 (1)**: Some information on the baseline would be useful: Typically, solving discrete dynamical systems (DEQs) is faster than a continuous dynamical system. \n\n**A3 (1)**: We respectively disagree with this comment. The iteration number required by solving discrete dynamical systems (DEQs) may be smaller than a continuous dynamical system, but the total running time may be still larger. This is because DEQs need nontrivial efforts in calculating the approximation of inverse Jacobian $B$ at each iteration, while this procedure is not needed in ODE-based models.  \n\nIn our experiments of DEQs, we followed the hyperparameter setting of DEQ on CIFAR-10 and chose Broyden solver for both forward and backward, and the threshold for the forward process is 18 and 20 for the backward process. Our experiments are implemented on a GPU of NVIDIA GeForce RTX 3090 with 24GB. \n\n**Q3 (2)**: The 8x inference time might be an artifact of poor solver choice for the DEQ. \n\n**A3 (2)**: Thank you for raising the concern. We have doubled checked the inference time in Table I, and there is a mistake as we ran multiple programs in parallel on the same GPU before. Now we have fixed this issue and updated the correct inference time in the following table. The new experiments are conducted with two different settings with test batch size 400 (used in our previous experiment) and test batch size 32 (newly added). We can see that DEQ still runs slow and even slower with large test batch sizes. This may be due to the inefficiency of DEQ for parallel computation with large test batch sizes. \n\n|  Method          | Inference Time (Batch size 400) | Inference Time (Batch size 32)|\n|:----:|:----:|:----:|\n| DEQ  | 5.8$\\times$  | 2.2$\\times$ |\n| **HomoODE**  |  1.2$\\times$| 1.2$\\times$ |\n| **HomoODE (aug.)**  | 1.4$\\times$ | 1.4$\\times$ |\n| **HomoODE (adj.)**  |  1.0$\\times$  | 1.0$\\times$ |\n| **HomoODE (adj.,aug.)**  |  1.0$\\times$ | 1.0$\\times$ |\n| monDEQ  | 1.6$\\times$ | 1.7$\\times$ |\n| NODE  | 3.2$\\times$ | 1.8$\\times$ |\n| ANODE  | 1.7$\\times$ | 1.2$\\times$ |"}, {"review_id": "aoqX5zUeDO", "reviewer": "Reviewer_iBob", "summary": "This paper unifies DEQs and Neural ODEs via homotopy continuation. The authors propose a novel implicit model, HomoODE, which inherits the property of high accuracy from DEQs and the property of stability from Neural ODEs. Moreover, they develop an acceleration method by borrowing the idea from the DEQ-solver.", "questions": "Please see weaknesses.", "limitations": "The authors have adequately addressed the limitations, and there is no potential negative societal impact of their work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1 This paper reveals the inherent connection between two typical implicit models, i.e., DEQs and ODEs.  This connection is really insightful. This paper, in my opinion, contributes significantly to our knowledge of implicit models.\n\n2 The proposed HomoODE combines the advantages of DEQs and ODEs. The authors leverage homotopy continuation, and thus HomoODEs do not suffer from the issue of unique equilibrium like DEQs. The proposed acceleration method is also nice. The authors provide a new design paradigm for implicit models.\n\n", "weaknesses": "1 This paper argures that \"DEQs and Neural ODEs are two sides of the same coin\", but compared with the analysis on  \"two sides\" is less than that on \"the same coin.\"   \n\n2 More experiments on large scale datasets, e.g., ImageNet, would be nice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully appreciate the reviewer for recommending acceptance of our submission and thank his/her valuable suggestions to help improve it. We will answer all the questions that the reviewers concern about.\n\n> **Q1**: This paper argues that \"DEQs and Neural ODEs are two sides of the same coin\", but compared with the analysis on \"two sides\" is less than that on \"the same coin.\"\n\n**A1**: Thanks for pointing out the lack of more detail about the \"two sides\". \n\nDEQs involve an underlying equilibrium-point-finding problem, which varies on different conditions $x$. As a result, DEQs can map inputs to diverse representations and thus enjoys good performance. However, DEQs suffer from stability issues since the existence of unique fixed points cannot be guaranteed. \n\nUnlike DEQs, the equilibrium-point-finding problem in Neural ODEs does not vary with the initial points $z(t_0)$. The structure of the Neural ODEs can guarantee stability, but the same underlying problems will limit its representation capability. Therefore, we claim that DEQs and Neural ODEs are actually two sides of the same coin. \n\nWe will add more analyses in the camera-ready version. \n\n> **Q2**: More experiments on large-scale datasets, e.g., ImageNet, would be nice. \n\n**A2**: Thanks for your valuable suggestion. Due to time limitations, our experiments on ImageNet are still in progress. However, to confirm the capability of our model in larger and more difficult datasets, we conducted the experiments on Tiny-ImageNet, which include a subset of the ImageNet dataset. Here we show the results of HomoODE compared with other implicit models. \n\n|  Method  | Model Size  |    Accuracy     |\n|:------------------:|:-----------:|:---------------:|\n| DEQ                | 810k        | 47.60 ± 0.4%    |\n| monDEQ             | 783k        | 24.83 ± 0.1%    |\n| **HomoODE**        | 617k        | 56.1 ± 0.1%     |\n| **HomoODE (adj.)** | 617k        | 54.14 ± 0.2%    |\n\n\nMoreover, the results on CIFAR-100 presented in Table 3 of the Appendix can also demonstrate certain capabilities of HomoODE on large-scale datasets."}, {"review_id": "Mrh2Aw5q8s", "reviewer": "Reviewer_ZDnx", "summary": "The paper shows that the fixed point equation solved by a Deep Equilibrium Model (DEQ) can be written as a Neural ODE.This is achieved by first introducing the equation of the solution path for the Homotopy equation, which is referred to as the Fixed Point Homotopy equation, and then deriving the appropriate homotopy mapping for the fixed point equation solved by a DEQ. Hence, they introduce an ODE that solves for a fixed point of a system (using homotopy continuation) and therefore claim that DEQs and neural ODEs are “essentially two-sides of the same coin”.\n\nThe authors then introduced HomoODE, that uses the input injection from a DEQ and the derived fixed-point-ODE to solve for a task, however instead of solving for the fixed point using newton-method in a DEQ homoODE uses a normal ode-solver similar to neural ODEs. The input injection helps in getting better performance than neural ODE and the implicit solving using ODE solver results in a more memory efficient methodology", "questions": "- In lines 205-210 the authors mention that neural ODE is a fixed-point problem with a “fixed-condition” x with different initial point z(t_0). I am not sure if I get that point. Presumably the fixed point used to derived the fixed point ODE could be conditioned on the input $x$ as well.\n- The derived equation in equation 12 has $F(z(t), t; x; \\theta)$ under the square root, is that a typo? or, there is some kind of “input injection” of $x$ that is implicitly assumed in Neural ODE that solves an equilibrium-point-finding problem.\n- How is $v$, the velocity in equation 9 chosen in the final neural ODE?\n- In Section 6, the authors mention the homotopy continuation on a nonlinear equation in using matlab. The details on which equation and what was the purpose of that experiment is missing. I am unsure about the conclusion of that section, do that authors learn a common initialization for all samples $x$ in the dataset, and is that learned or just an average over the entire dataset?", "limitations": "Authors have discussed the limitations of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written, and effectively establishes the connection between neural ODEs and DEQs. Furthermore, they introduce HomoODE, which inherits the better performance capabilities of a DEQ, and has a lesser memory footprint (thanks to the ODEsolver instead of newton method based solver, used in Neural ODEs). \n\nThe experiments show on various classification tasks that HomoODE matches or beats DEQs in various benchmarks with fewer model parameters and with faster inference time.", "weaknesses": "There are some details missing in some parts of the paper, I have pointed them out in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and suggestions. Here we address your detailed comments as below:\n\n> **Q1**: In lines 205-210 the authors mention that neural ODE is a fixed-point problem with a “fixed-condition” $x$ with different initial point $z(t_0)$. I am not sure if I get that point. Presumably the fixed point used to derive the fixed point ODE could be conditioned on the input $x$ as well.\n\n**A1**: Thank you for pointing out the lack of clarity here. There is a typo here and it should be \"Neural ODEs solve the fixed equilibrium-point-finding problem with different initial points $z(t_0)$\". There is no condition in Neural ODEs and here we want to state the underlying equilibrium-point-finding problem in Neural ODEs does not vary with the input. We will modify this statement in the final version. \n\n> **Q2**: The derived equation in equation 12 has $F(z(t),t;x;\\theta)$  under the square root, is that a typo? or, there is some kind of “input injection” of  $x$ that is implicitly assumed in Neural ODE that solves an equilibrium-point-finding problem.\n\n**A2**: Yes, it is a typo. Thank you for your careful reading. $F(z(t),t;x;\\theta)$ should be $F(z(t),t;\\theta)$. We will fix this issue in the final version. \n\n> **Q3**: How is $v$, the velocity in equation 9 chosen in the final neural ODE?\n \n**A3**: We would like to clarify that $v$ is an auxiliary variable for the theoretical analysis to bridge the DEQs and Neural ODEs via Fixed Point Homotopy (line 178-184). In the experiments, we do not need to include $v$ as a hyperparameter or compute its value because it has been implicitly contained in the neural network. \n\n> **Q4(1)**: In Section 6, the authors mention the homotopy continuation on a nonlinear equation in using matlab. The details on which equation and what was the purpose of that experiment is missing. \n\n\n**A4(1)**: Thank you for raising the concern. As illustrated in line 257-261, we would like to explain via Figure 2 that it is **not necessary** to train the initial points for its corresponding image very accurately. This is because homotopy continuation is different from methods like Newton's method where even if we set a initial point very close to the solution, it will still need **certain iteration times** to obtain the solution. That's why we only maintain a common initial point for all samples $x$ rather than calculate a sample-conditioned initial point like [8]. \n\nWith respect to the detail on the tested nonlinear equation, we randomly choose an equation with high nonlinearity that is \n$$\nf(x) = 2x+\\exp(-0.1x)+5\\sin(4x)-16 = 0,\n$$\nand one of the solutions is 6.4217. Our experiments in homotopy continuation are based on the initial points around this solution. Actually, you can modify the parameters in this function arbitrarily and can obtain similar conclusion as well. We will add these details in the final version. \n\n> **Q4(2)**: I am unsure about the conclusion of that section, do that authors learn a common initialization for all samples $x$ in the dataset, and is that learned or just an average over the entire dataset?\n\n**A4(2)**: Yes, we learn a common initialization for all samples $x$ in the dataset. With the training process, the fixed-point solutions of all samples are changing as well. To ensure a stable training process, the common initialization is learned with stochastic gradient descent instead of directly averaging among the sample solutions. "}]

Author rebuttal:
We thank the reviewers for their careful reading and constructive and detailed suggestions to help us improve our manuscript.  We sincerely appreciate that the reviewers consider unanimously that our work "effectively" (ZDnx) establishes an "inherent and insightful" (iBob) and "mathematically solid" (rMvZ) connection in the absence of theoretical explorations between Neural ODEs and DEQs (aBa7). We are further glad that the reviewers agree our work provides "nice" and "practical" (rMvZ, iBob) ideas like the learnable initial point, and that the proposed HomoODE inherits the advantages of DEQs and Neural ODEs (iBob, ZDnx), which delivers a lot of insights and contributes significantly to the community of implicit model (iBob, rMvZ).

In the following, we will try to address the concerns/questions of the reviewers and provide a detailed item-by-item response to your comments.

Meta-review: The reviewers unanimously find the paper interesting and result worth publication. The area chair agrees with the evaluation after reading the discussion and the manuscript.
Decision: Accept (poster)