Title: Two Sides of The Same Coin: Bridging Deep Equilibrium Models and Neural ODEs via Homotopy Continuation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Equilibrium Models

Neural Ordinary Differential Equations

Homotopy Continuation
Abstract: Deep Equilibrium Models (DEQs) and Neural Ordinary Differential Equations (Neural ODEs) are two branches of implicit models that have achieved remarkable success owing to their superior performance and low memory consumption. While both are implicit models, DEQs and Neural ODEs are derived from different mathematical formulations. Inspired by homotopy continuation, we establish a connection between these two models and illustrate that they are actually two sides of the same coin. Homotopy continuation is a classical method of solving nonlinear equations based on a corresponding ODE. Given this connection, we proposed a new implicit model called HomoODE that inherits the property of high accuracy from DEQs and the property of stability from Neural ODEs. Unlike DEQs, which explicitly solve an equilibrium-point-finding problem via Newton's methods in the forward pass, HomoODE solves the equilibrium-point-finding problem implicitly using a modified Neural ODE via homotopy continuation. Further, we developed an acceleration method for HomoODE with a shared learnable initial point. It is worth noting that our model also provides a better understanding of why Augmented Neural ODEs work as long as the augmented part is regarded as the equilibrium point to find. Comprehensive experiments with several image classification tasks demonstrate that HomoODE surpasses existing implicit models in terms of both accuracy and memory consumption.

Two Sides of The Same Coin:

Abstract

Deep Equilibrium Models (DEQs) and Neural Ordinary Differential Equations (Neural ODEs) are two branches of implicit models that have achieved remarkable success owing to their superior performance and low memory consumption. While both are implicit models, DEQs and Neural ODEs are derived from different mathematical formulations. Inspired by homotopy continuation, we establish a connection between these two models and illustrate that they are actually two sides of the same coin. Homotopy continuation is a classical method of solving nonlinear equations based on a corresponding ODE. Given this connection, we proposed a new implicit model called HomoODE that inherits the property of high accuracy from DEQs and the property of stability from Neural ODEs. Unlike DEQs, which explicitly solve an equilibrium-point-finding problem via Newton's methods in the forward pass, HomoODE solves the equilibrium-point-finding problem implicitly using a modified Neural ODE via homotopy continuation. Further, we developed an acceleration method for HomoODE with a shared learnable initial point. It is worth noting that our model also provides a better understanding of why Augmented Neural ODEs work as long as the augmented part is regarded as the equilibrium point to find. Comprehensive experiments with several image classification tasks demonstrate that HomoODE surpasses existing implicit models in terms of both accuracy and memory consumption.

1 Introduction

Recent studies of implicit models have certified that such models can meet or even surpass the performance of traditional deep neural networks. Instead of specifying the explicit computation process of the output, implicit models define the joint conditions that the layer's input and output satisfy. Instances of such models include the Neural Ordinary Differential Equations (Neural ODEs) [10], which treat ODEs as a learnable component and can be viewed as continuous Residual Networks [25] (ResNets), the Deep Equilibrium Models (DEQs) [6], which compute the equilibrium point of a nonlinear transformation corresponding to an infinite-depth weight-tied network; and optimization layers [4, 45], which leverage the optimization techniques as layers of neural networks.

Although DEQs and Neural ODEs, as popular implicit models in recent years, have garnered much attention in terms of theoretical analysis and application, an insightful connection between these two

branches of the implicit model has not been established. DEQs involve an equilibrium-point-finding problem, which is a nonlinear equation system. Generally, this can be solved via the homotopy continuation method [39], a classical method that solves nonlinear equations along the zero path of the homotopy mapping and can be further formulated as an ODE. This motivated us to consider whether we could bridge DEQs and Neural ODEs via the theory of homotopy continuation.

In this paper, we show that Neural ODEs can also be viewed as a procedure for solving an equilibriumpoint-finding problem. However, while both of the two models can be considered as solving equilibrium-point-finding problems, they differ in how the input information is used. On the one hand, DEQs regard the input information as the condition that determines the equilibrium-point-finding problem to solve via injecting it into the equilibrium function in the forward pass. On the other hand, Neural ODEs generate the initial points with different input information and expect them to converge to different equilibrium points. Therefore, we claim that DEQs and Neural ODEs are actually two sides of the same coin.

Inspired by the theoretical connection between the two models, we developed a new implicit model called HomoODE, which inherits the advantages of both. Specifically, HomoODE injects the input information into the underlying dynamic of an equilibrium-point-finding problem in the same way as a DEQ does, but then obtains the output from an ODE solver just as a Neural ODE does. The connection between DEQ and Neural ODE means that HomoODE avoids DEQ's problem of unstable convergence in DEQs and the weak representation capability of Neural ODEs. Further, a common issue for implicit models is the trade-off between computational speed and precision. Therefore, a natural intuition to accelerate such models is to find a good initial point that is close to the solution. We observed that, as the distance between the initial point and the solution drops to some range, the numbers of function evaluations (NFE) almost stop decreasing when applying homotopy continuation. Hence, we introduced an extra loss instead of zero vector initialization in DEQ to train a shared initial point. In this way, the distance from the initial point to each solution is within appropriate ranges. A series of experiments with several image classification tasks verify that HomoODE is able to converge stably and that it outperforms existing implicit models with better memory efficiency on several image classification tasks, and our acceleration method significantly reduces the NFE of HomoODE. In summary, our contributions are as follows:

(1) A connection between DEQs and Neural ODEs. We establish a connection between DEQs and Neural ODEs via homotopy continuation, which illustrates that DEQs and Neural ODEs are actually the two sides of the same coin. We believe this new perspective provides novel insights into the mechanisms behind implicit models.

(2) A New Implicit Model: HomoODE. We propose a new implicit model called HomoODE, that inherits the advantages of both DEQs and Neural ODEs. HomoODE implicitly solves equilibriumpoint-finding problems using homotopy continuation, unlike DEQs which explicitly solve these problems via Newton's methods. Additionally, we have accelerated HomoODE with a learnable initial point that is shared among all samples.

(3) Understanding Augmented Neural ODE. We demonstrate that Augmented Neural ODE can be treated as a special case of HomoODE based on homotopy continuation. Hence Augmented Neural ODE enjoys better representation ability and outperforms Neural ODE.

(4) Better Performance. We conduct experiments on image classification datasets and confirm our HomoODE outperforms DEQ, Neural ODE, and variants of them both in accuracy and memory usage. Furthermore, we also perform the sensitivity analysis on the hyper-parameters to research the characters of our model.

2 Related Works

Deep Equilibrium Models . DEQs [6] have shown competitive performance on a range of tasks, such as language modeling[6], graph-related tasks [20], image classification or segmentation [7], image generation [41], inverse problems in imaging[16], image denoising [17] and optical flow estimation [5]. DEQs find an equilibrium point of a nonlinear dynamical system corresponding to an effectively infinite-depth weight-tied network. However, training such models requires careful consideration of both initializations and the model structure [6, 9, 3], and often consumes long training times. Many studies have been devoted to solving these problems. For example, the Monotone Operator

Equilibrium Network (monDEQ) [46] ensures stable convergence to a unique equilibrium point by involving monotone operator theory. Bai et al. [9] propose an explicit regularization scheme for DEQs that stabilizes the learning of DEQs by regularizing the Jacobian matrix of the fixed-point update equations. Kawaguchi et al.[30] prove that DEQs converge to global optimum at a linear rate for a general class of loss functions by analyzing its gradient dynamics. Agarwala et al.[3] show that DEQs are sensitive to the higher-order statistics of their initial matrix family and consequently propose a practical prescription for initialization. To reduce the computational expense of DEQ, Pal et al. [40] developed continuous DEQs utilizing an "infinity time" neural ODE but did not continue to explore the inherent connection between general DEQs and Neural ODEs. From the perspective of optimization, Optimization Induced Equilibrium Networks (OptEq)[47] theoretically connect their equilibrium point to the solution of a convex optimization problem with explicit objectives. Instead of regularizing the structures or involving parameterizations of the implicit layer design, Bai et al. [8] propose a model-specific equilibrium solver, which both guesses an initial value of the optimization and performs iterative updates. However, unlike DEQs, which explicitly solve equilibrium-point-finding problems via Newton's methods, our HomoODE solves these problems based on homotopy continuation implicitly . Accordingly, HomoODE does not suffer from the issue of unique equilibrium like DEQ and thus can avoid the stability issue. Moreover, we accelerate HomoODE using a good initial point learned with a corresponding loss function. Unlike Bai et al.'s approach [8], HomoODE learns an initial point shared among all samples without involving a network-based initializer.

Neural Ordinary Differential Equations . Neural ODEs have been applied to time series modeling [44, 32], continuous normalizing flows [10, 18], and modeling or controlling physical environments [50, 43, 48, 19, 14]. Neural ODEs treat ODEs as a learnable component and produce their outputs by solving the Initial Value Problem [10, 12, 29]. However, Neural ODEs are often characterized by long training times and sub-optimal results when the length of the training data increases [13, 15]. Prior works have tried to tackle these problems by placing constraints on the Jacobian[13] or high derivatives of the differential equation [31]. Conversely, Augmented Neural ODEs [12] learn the flow from the input to the features in an augmented space with better stability and generalization. Ghosh et al. [15] treat the integration time points as stochastic variables without placing any constraints. With diffeomorphism, the complexity of modeling the Neural ODEs can be offloaded onto the invertible neural networks [49], and training Neural ODEs with the adaptive checkpoint adjoint method [51] can be accurate, fast, and robust to initialization. The symplectic adjoint method [36] finds the exact gradient via a symplectic integrator with appropriate checkpoints and memory consumption that is competitive to the adjoint method. The advantage of HomoODE is that it inherits the property of high accuracy from DEQs and the property of stability from Neural ODEs. In addition, HomoODE provides an explanation of why Augmented Neural ODEs achieve better performance than Neural ODEs. Notably, Augmented Neural ODEs can be viewed as a special case of HomoODE.

Homotopy Continuation. Homotopy continuation [39] is a numerical technique that traces the solution path of a given problem as the parameter changes from an initial value to a final value. Homotopy methods have been successfully applied to solving pattern formation problems arising from computational mathematics and biology including computing multiple solutions of differential equations [21, 22], state problems of hyperbolic conservation laws [21], computing bifurcation points of nonlinear systems [24] and solving reaction-diffusion equations [23]. Recent advances in deep learning have also seen the homotopy continuation method fused into learning processes. For instance, Ko et al.[33] adapt homotopy optimization in Neural ODEs to gain better performance with less more training epochs. HomPINNs [27] traces observations in an approximate manner to identify multiple solutions, then solves the inverse problem via the homotopy continuation method. To reach a good solution to the original geometrical problem, Hruby et al. [26] learn a single starting point for a real homotopy continuation path. In this work, we establish a connection between DEQs and Neural ODEs from the perspective of homotopy continuation and develop a new implicit model called HomoODE based on this theoretical relationship.

3 Background on Homotopy Continuation

Homotopy continuation has been broadly applied to solve nonlinear equations. The first step to solving a specific problem r ( z ) = 0 is to construct a homotopy mapping.

Definition 1 (Homotopy mapping [39]) The function H ( z, λ ) = λr ( z ) + (1 -λ ) g ( z ) is said to be a homotopy mapping from g ( z ) to r ( z ) , if λ is a scalar parameter from 0 to 1 , and g ( z ) is a smooth function. The equation H ( z, λ ) = 0 is the zero path of this homotopy mapping.

Homotopy mapping provides a continuous transformation by gradually deforming g ( z ) into r ( z ) while λ increases from 0 to 1 in small increments. Hence, the solution to r ( z ) can be found by following the zero path of the homotopy mapping H ( z, λ ) = 0 . Usually, one can choose g ( z ) as an artificial function with an easy solution. Here, we specifically consider Fixed Point Homotopy which chooses g ( z ) = z -z 0 :

where z 0 ∈ R n is a fixed vector, and is the initial point of the homotopy continuation method. In one practical trick, we can follow the zero path by allowing both z and λ to be functions of an independent variable s , which represents arc length along the path. In other words, ( z ( s ) , λ ( s )) is the point arrived at by traveling a distance s along the zero path from the initial point ( z (0) , λ (0)) = ( z 0 , 0) . In the zero path, we have H ( z ( s ) , λ ( s )) = 0 , for all s ≥ 0 .

Take the derivative for this equation with respect to s lead to:

The vector ( dz ds , dλ ds ) ∈ R n +1 is the tangent vector to the zero path, and it lies in the null space of matrix [ ∂H ( z,λ ) ∂z , ∂H ( z,λ ) ∂λ ] ∈ R n × ( n +1) . To complete the definition of ( dz ds , dλ ds ) , a normalization condition is imposed to fix the length of the tangent vector, i.e.

Given the tangent vector and the initial point, we can trace the zero path and obtain the solution of F ( z ) = 0 by solving the ODE (2).

4 Bridging DEQs & Neural ODEs via Homotopy Continuation

Here we briefly review DEQs and Neural ODEs, then bridge these two models via homotopy continuation. DEQs aim to solve the equilibrium point of the function f ( z ; x, θ ) , which is parameterized by θ and the input injection x . The underlying equilibrium-point-finding problem of DEQs is defined as follows:

Usually, we choose f ( z ; x, θ ) as a shallow stacked neural layer or block. Hence, the process of solving the equilibrium point can be viewed as modeling the 'infinite-depth' representation of a shallow stacked block. One can use any black-box root-finding solver or fixed point iteration to obtain the equilibrium point z ⋆ .

Unlike the underlying equilibrium-point-finding problem of DEQs, Neural ODEs view its underlying problem as an ODE, whose derivative is parameterized by the network. Specifically, Neural ODEs map a data point x into a set of features by solving the Initial Value Problem [29] to some time T . The underlying ODE of Neural ODEs is defined as follows:

where z ( t ) represents the hidden state at time t , F ( z ( t ) , t ; θ ) is neural networks parameterized by θ .

The same coin. DEQs apply Newton's methods to solve the underlying equilibrium-point-finding problem z = f ( z ; x, θ ) . By defining r ( z ) = z -f ( z ; x, θ ) , one can alternatively solve this equilibrium-point-finding problem based on homotopy continuation. Now we will show that the underlying dynamics in Neural ODEs can also be treated as an equilibrium-point-finding problem.

Firstly, we apply the Fixed Point Homotopy to solve the equilibrium-point-finding problem z = f ( z ; θ ) and obtain the homotopy mapping H ( z, λ ) = λ ( z -f ( z ; θ )) + (1 -λ ) z . Taking the partial derivative of H ( z, λ ) with respect to z and λ , respectively, we obtain

By substituting the partial derivative in (6) into (2), we can obtain:

Based on the normalization condition (3), we can reformulate (7) as the following differential equation:

As Neural ODEs do, we can use neural networks to approximate the underlying dynamics of such an ODE(8). However, the norm of neural network output is likely to exceed the unit length, i.e. violating the normalization condition (3). To address this issue, we introduce v := ds dt as the velocity of the point ( z, λ ) traveling along the zero path, and modify the normalization condition by introducing v into (3):

Note that the convergence of the homotopy continuation is not affected by the value of v . The underlying dynamics of (7) becomes

Following Neural ODEs, the differential equation (10) can be approximated by neural networks, i.e. dz dt = F ( z ( t ) , t ; θ ) . However, we still need to ensure the existence of a corresponding equilibriumpoint-finding problem for Neural ODE. Ingeniously, the modified normalization can also ensure the existence of the equilibrium-point-finding problem. When we obtain the Neural ODEs (5) by training the neural networks F ( z ( t ) , t ; θ ) , we can compute the changing process of λ ( t ) and the velocity v by solving the following equations:

Note that the modified normalization condition (9) provides the dynamic with another degree of freedom, which guarantees the existence of λ ( t ) . Otherwise, there might be no solution for λ ( t ) as there are two initial conditions (i.e., λ (0) = 0 , λ (1) = 1 ) for the system. Hence, the equilibriumpoint-finding problem z = f ( z ; θ ) is implicitly determined by the following partial differential equation:

Therefore, Neural ODEs can be regarded as the procedure of solving an equilibrium-point-finding problem with homotopy continuation, and the hidden state at t = 0 , z ( t 0 ) is the initial point of homotopy continuation.

Two sides. We have shown that both DEQs and Neural ODEs can be considered as solving equilibrium-point-finding problems through homotopy continuation. Now we discuss the difference in underlying equilibrium-point-finding problem between DEQs and Neural ODEs.

On the one hand, DEQs solve the problem from the same initial point z 0 . The equilibrium-pointfinding problem of DEQs is parameterized by the input injection x . The underlying equilibriumpoint-finding problem of DEQs is defined as follows:

The input injection x can be viewed as the condition to fuse the information of input to the underlying equilibrium-point-finding problem. The underlying problem of DEQs varies depending on different conditions x . Therefore, DEQs are able to map inputs to diverse representations, which is crucial for achieving superior performance.

On the other hand, unlike DEQs, Neural ODEs solve an equilibrium-point-finding problem with different initial points z ( t 0 ) . The underlying equilibrium-point-finding problem of Neural ODEs is defined as follows:

and Neural ODEs solve the problem through homotopy continuation:

where f is determined by F in equation (12). Concretely, Neural ODEs map raw data into a set of features z ( t 0 ) = x and regard them as the initial points of the ODE. The fixed underlying problem ensures the stability of Neural ODEs but loses diversified representation capabilities. Therefore, we claim that DEQs and Neural ODEs are actually two sides of the same coin from the perspective of homotopy continuation.

5 HomoODE: an efficient and effective implicit model

is equilibrium

point of

ODE Solver

is equilibrium

point of

Feature

Extractor

Neural ODE

Connecting             and            via

Homotopy Continuation

:

ODE Solver

Feature

Extractor

Initial point

HomoODE

Broyden

Equation Solver

Feature

Extractor

Initial point

DEQ

condition

condition

is equilibrium

point of

As we show above, both DEQs and Neural ODEs can be considered as solving equilibrium-pointfinding problems through homotopy continuation. Two well-known approaches for solving nonlinear equations are Newton's Method [2] and homotopy continuation [39]. DEQs solve an equilibriumpoint-finding problem r ( z ) = 0 via Newton's Methods which are local in the sense that a good estimate of the solution is required for the convergence. Unlike Newton's Method, homotopy continuation is global in the sense that solutions of g ( z ) = 0 may not need to be anywhere close to the solution of r ( z ) = 0 [11]. Inspired by the connection between DEQs and Neural ODEs, a very natural thought is that we can apply homotopy continuation to solve the underlying nonlinear equation of DEQs. By replacing the equilibrium-point-finding problem z = f ( z ; θ ) to z = f ( z ; x, θ ) , we can obtain the following differential equation:

Hence, we proposed a new implicit model called HomoODE, which models an equilibrium problem z = f ( z ; x, θ ) implicitly. Specifically, we employ neural networks to approximate the differential equation (16). In the design of the network structure, we introduce the condition x into the underlying dynamic of HomoODE as DEQ does and obtain the output from the same initial point through the ODE solver as Neural ODE does. In this way, HomoODE not only has the ability of diversified

representation of DEQs but also has the property of stable convergence of Neural ODEs. In addition, the time information t is not explicitly formulated by the dynamic of HomoODE (16). So unlike Neural ODEs, HomoODE does not require the input of time information t . Figure 1 illustrates the structure of HomoODE as well as Neural ODE and DEQ. Besides, it is worth noting that v mentioned in (16) is just an auxiliary variable for the theoretical analysis. Therefore, we do not need to include it as a hyperparameter or compute its value in the forward pass because it has been implicitly contained in the neural network.

Forward Pass. In HomoODE, the raw data x is first input to a feature extractor g ( x ; ω ) and then injected into an ODE solver. Suppose z ( t ) represents the intermediate state of HomoODE, calculating z ( t ) involves an integration starting from the initial point z ( t 0 ) = 0 to the solution z ( t 1 ) . Notably, the output z ( t 1 ) is equivalent to the solution z ⋆ of the implicit equilibrium-point-finding problem z = f ( z ; x, θ ) . Then we can use the ODE solvers to obtain the solution z ⋆ of the origin equilibrium problem and this representation can be used for downstream tasks, such as classification, regression, etc.

Backward Pass. In the backward pass of HomoODE, we can apply the adjoint sensitivity method, or straightly differentiate through the operations of the forward pass. The condition traces back to another gradient flow. More details related to the construction of HomoODE dynamics and the computation of the gradients based on the adjoint method can be referred to in the supplementary materials.

6 Acceleration for HomoODE

In practice, we observe that HomoODE needs more function evaluations in the ODE-solving process, which results from bad initialization (e.g., zero vector initialization). To address this issue, we referred to [8], which makes an input-based initial point guess with an extra neural network to accelerate the equation-solving procedure in DEQ. However, this method poses an extra cost in both memory and computation.

To avoid these drawbacks, we further investigate the relationship between the number of iterations and the distance from the initialization z 0 to the solution z ⋆ in homotopy continuation. Specifically, we perform the homotopy continuation method on a nontrivial nonlinear equation using ode45 in Matlab [28]. As is shown in Figure 2, it does not bring any reduction in the number of iterations when the initial point is close enough to the equilibrium solution. sample x

0

2

4

6

8

10

12

1

=d

100

250

400

550

700

850

1000

relation

limit

N

This means it is unnecessary to approximate the specific initial point very accurately for a specific in HomoODE, and that is different from that in [8].

Hence, we share the initial point for all the samples x and just maintain a scalar value for the averaged value of the whole feature map in one channel. Briefly, we just need to store a tensor with the shape of (1 , 1 , c ) as the shared initial information ˜ z 0 , and broadcast it into the initial point z 0 with the shape of ( h, w, c ) when taking it into the ODE solver. Here h, w, c means the height, width, and channel number of the feature map. The loss function of ˜ z 0 is defined as follows:

where z ⋆ ( x ) denotes the equilibrium solution of the sample x and D denotes the distribution of x . In fact, this update on ˜ z 0 is equivalent to maintaining the dynamic geometrical center of the equilibrium points of all the samples.

7 Understanding Augmented Neural ODE by HomoODE

Augmented Neural ODEs, as a simple extension of Neural ODEs, are more expressive models and outperform Neural ODEs. Augmented Neural ODEs allow the ODE flow to lift points into the extra

dimensions to avoid trajectories crossing each other [12]. However, more theoretical analysis of how and why augmentation improves Neural ODEs is lacking. Our work provides another perspective on understanding the effectiveness of Augmented Neural ODEs. Augmented Neural ODEs formulate the augmented ODE problem as:

where a ( t ) ∈ R p denotes a point in the augmented part, and h ( t ) ∈ R d is the hidden state at time t .

Specifically, Augmented Neural ODEs can track back the ODE flow to recover the original input x by using the hidden state h ( t ) and the time information t . The input x is the injection of the dynamics in HomoODE. The augmented part a ( t ) can be viewed as the intermediate z ( t ) in HomoODE. In Augmented Neural ODEs, we can treat the recovered input x as the condition in HomoODE, improving its representation ability. Hence, Augmented Neural ODEs outperform Neural ODEs. However, the origin input x computed by Augmented Neural ODEs may not be accurate enough. This probably is the reason why the performance of Augmented Neural ODEs is not competitive to HomoODE.

8 Experiments

To confirm the efficiency of HomoODE, we conduct experiments on several classical image classification datasets to compare our model with the previous implicit model, including DEQ [6], monDEQ [46], Neural ODE [10] and Augmented Neural ODE [12]. For ODE datasets, DEQs inherently are not designed to handle simulation tasks of continuous physical processes as DEQs do not involve the continuous time series in the models. Sequential datasets are not suitable as well due to the inconsistent backbones. For sequential datasets, DEQ models are based on transformer, while Neural ODEs are based on fully-connected layers. Therefore, we consider the image classification task is suitable for the fair comparison of HomoODE and baselines. Concretely, we evaluate the stability of the training process via the learning curve of accuracy in the test dataset and exhibit the performance of different implicit models in terms of accuracy, memory consumption, and inference time. It is worth noting that we also perform HomoODE with & without the data augmentation and the adjoint backpropagation technique to check their impacts on our model. Besides, we also contrast HomoODE with zero vector initialization and learnable initialization to assess the capability of our acceleration method. Our code is available at: https://github.com/wadx2019/homoode .

+

ReLU

Dropout

+

3

×

3 Conv2d (s=1)

3

3 Conv2d (s=1)

3

3 Conv2d (s=1)

3

3 Conv2d (s=2)

+

Condition

2(t)

d

dt

F(z(t),t;x,0)

Experimental setup. HomoODE is performed on several standard datasets, including CIFAR-10 [34], SVHN [38], and MNIST [35]. As shown in Figure 3, HomoODE contains several simple convolutional layers. Its network structure is not specially designed for image classification tasks like MDEQ [7]. Notably, the memory consumption of HomoODE is less than that of other implicit models as reported in [46, 48, 7]. As we discussed in Section 5, the time information t is not fused into the input of HomoODE, unlike Neural ODE. Besides, we optimize the shared initial information

˜ z 0 using SGD optimizer with the learning rate 0 . 02 and perform the update once every 20 updates for HomoODE.

Comparison with former implicit models. Table 1 presents the performance of HomoODE with different settings and other implicit models in the CIFAR10 dataset. It can be observed that HomoODE outperforms the previous implicit models in terms of both accuracy and memory consumption. Moreover, the inference time of HomoODE is much faster than DEQ and its variants. Notably, we also find that DEQ runs slower with larger test batch sizes. This may be due to the inefficiency of DEQ for parallel computation with large test batch sizes. In contrast, HomoODE does not exhibit this problem. Additionally, there is a large increase in accuracy in our model when the data augmentation technique is applied. It suggests that HomoODE learns better and more complex representations through augmented datasets. Furthermore, given that we adopt the dopri5 ode solver with an adaptive step, this complex representation may result in increased NFE and inference time. Overall, these phenomenons validate that Ho-











































moODE has a more powerful representation ability compared to other implicit models with similar model capacity. Extensive experiments in SVHN and MNIST also confirm these properties of our model as shown in Table 2.

Besides, we also plot the learning curves of different algorithms in Figure 4. The results demonstrate the stability of training HomoODE compared with other methods and exhibit that HomoODE is not prone to over-fitting, whereas other ODE-based models may suffer from that.

Efficiency of the learnable initialization. Figure 5 illustrates that the learnable initialization trick can improve HomoODE with about 2 . 5 × speedup in the inference time than before. This impact is obvious in both cases with & without the adjoint backpropagation technique. In addition, the corresponding test accuracy during the training process also reflects this acceleration technique does not bring a loss in the performance of our model. Surprisingly, it even brings a slight improvement in the case of the adjoint backpropagation technique. This is probably because a good initial point can decrease the total length of zero path s , which reduces the gradient error induced by using the adjoint method.

9 Conclusion

In this paper, we show that both DEQs and Neural ODEs can be viewed as a procedure for solving an equilibrium-point-finding problem via the theory of homotopy continuation. Motivated by this observation, we illustrate that these two implicit models are actually the two sides of the same coin. Specifically, DEQs inject the input information as the condition into the equilibrium-point-finding problem z ⋆ = f ( z ⋆ ; x, θ ) while Neural ODEs fuse the input information into the initial point. Further, we propose a novel implicit model called HomoODE, which inherits the advantages of both DEQs and Neural ODEs. Our experiments indeed verify that HomoODE outperforms both DEQs and Neural ODEs while avoiding the instability of the training process, that is often observed with DEQs. Moreover, we developed a method to speed up HomoODE and the ODE-solving operation by almost three times by using a shared learnable initial point. Overall, the experimental results on several classical image classification datasets demonstrate the efficiency of HomoODE in terms of both accuracy and memory consumption.

Although this paper offers a brand new perspective on implicit models, we also want to highlight a limitation of this idea, Actually, we do not present an explicit form of the equilibrium transformation function, which is implicitly determined by a modified neural ODE. Besides, while HomoODE has a powerful representation ability, the equilibrium-point-solving procedure of it is implicit, which

















































































weakens its interpretability. Hence, exploring a more interpretable approach for the forward pass and backpropagation of HomoODE is under consideration in our future work.

References

Ralph Abraham and Joel Robbin. Transversal mappings and flows . WABenjamin New York, 1967.

Milton Abramowitz, Irene A Stegun, and Robert H Romer. Handbook of mathematical functions with formulas, graphs, and mathematical tables, 1988.

Atish Agarwala and Samuel S Schoenholz. Deep equilibrium networks are sensitive to initialization statistics. In International Conference on Machine Learning , pages 136-160. PMLR, 2022.

Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning , pages 136-145. PMLR, 2017.

Shaojie Bai, Zhengyang Geng, Yash Savani, and J Zico Kolter. Deep equilibrium optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 620-630, 2022.

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems , 32, 2019.

Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Multiscale deep equilibrium models. Advances in Neural Information Processing Systems , 33:5238-5250, 2020.

Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Neural deep equilibrium solvers. In International Conference on Learning Representations , 2021.

Shaojie Bai, Vladlen Koltun, and Zico Kolter. Stabilizing equilibrium models by jacobian regularization. In International Conference on Machine Learning , pages 554-565. PMLR, 2021.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Tianran Chen and Tien-Yien Li. Homotopy continuation method for solving systems of nonlinear and polynomial equations. Communications in Information and Systems , 15(2):119-307, 2015.

Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. Advances in neural information processing systems , 32, 2019.

Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ode: the world of jacobian and kinetic regularization. In International conference on machine learning , pages 3154-3164. PMLR, 2020.

Marc Finzi, Ke Alexander Wang, and Andrew G Wilson. Simplifying hamiltonian and lagrangian neural networks via explicit constraints. Advances in neural information processing systems , 33:13880-13889, 2020.

Arnab Ghosh, Harkirat Behl, Emilien Dupont, Philip Torr, and Vinay Namboodiri. Steer: Simple temporal regularization for neural ode. Advances in Neural Information Processing Systems , 33:14831-14843, 2020.

Davis Gilton, Gregory Ongie, and Rebecca Willett. Deep equilibrium architectures for inverse problems in imaging. IEEE Transactions on Computational Imaging , 7:1123-1133, 2021.

Alexandros Gkillas, Dimitris Ampeliotis, and Kostas Berberidis. Connections between deep equilibrium and sparse representation models with application to hyperspectral image denoising. IEEE Transactions on Image Processing , 32:1513-1528, 2023.

Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367 , 2018.

Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in neural information processing systems , 32, 2019.

Fangda Gu, Heng Chang, Wenwu Zhu, Somayeh Sojoudi, and Laurent El Ghaoui. Implicit graph neural networks. Advances in Neural Information Processing Systems , 33:11984-11995, 2020.

Wenrui Hao, Jonathan D Hauenstein, Chi-Wang Shu, Andrew J Sommese, Zhiliang Xu, and Yong-Tao Zhang. A homotopy method based on weno schemes for solving steady state problems of hyperbolic conservation laws. Journal of Computational Physics , 250:332-346, 2013.

Wenrui Hao, Jan Hesthaven, Guang Lin, and Bin Zheng. A homotopy method with adaptive basis selection for computing multiple solutions of differential equations. Journal of Scientific Computing , 82:1-17, 2020.

Wenrui Hao and Chuan Xue. Spatial pattern formation in reaction-diffusion models: a computational approach. Journal of Mathematical Biology , 80:521-543, 2020.

Wenrui Hao and Chunyue Zheng. An adaptive homotopy method for computing bifurcations of nonlinear parametric systems. Journal of Scientific Computing , 82:1-19, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Petr Hruby, Timothy Duff, Anton Leykin, and Tomas Pajdla. Learning to solve hard minimal problems. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5532-5542, 2022.

Yao Huang, Wenrui Hao, and Guang Lin. Hompinns: Homotopy physics-informed neural networks for learning multiple solutions of nonlinear elliptic differential equations. Computers &Mathematics with Applications , 121:62-73, 2022.

The MathWorks Inc. Matlab version: 9.13.0 (r2022b), 2022.

Edward L Ince. Ordinary differential equations . Courier Corporation, 1956.

Kenji Kawaguchi. On the theory of implicit deep learning: Global convergence with implicit layers. In International Conference on Learning Representations (ICLR) , 2021.

Jacob Kelly, Jesse Bettencourt, Matthew J Johnson, and David K Duvenaud. Learning differential equations that are easy to solve. Advances in Neural Information Processing Systems , 33:4370-4380, 2020.

Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems , 33:6696-6707, 2020.

Joon-Hyuk Ko, Hankyul Koh, Nojun Park, and Wonho Jhe. Homotopy-based training of neuralodes for accurate dynamics discovery. arXiv preprint arXiv:2210.01407 , 2022.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Takashi Matsubara, Yuto Miyatake, and Takaharu Yaguchi. Symplectic adjoint method for exact gradient of neural ode with minimal memory. Advances in Neural Information Processing Systems , 34:20772-20784, 2021.

Mohammed Ali mnmoustafa. Tiny imagenet, 2017.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Jorge Nocedal and Stephen J Wright. Numerical optimization . Springer, 1999.

Avik Pal, Alan Edelman, and Christopher Rackauckas. Continuous deep equilibrium models: Training neural odes faster by integrating them to infinity. arXiv preprint arXiv:2201.12240 , 2022.

Ashwini Pokle, Zhengyang Geng, and J Zico Kolter. Deep equilibrium approaches to diffusion models. Advances in Neural Information Processing Systems , 35:37975-37990, 2022.

Lev Semenovich Pontryagin. Mathematical theory of optimal processes . CRC press, 1987.

Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.04385 , 2020.

Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems , 32, 2019.

Haixiang Sun, Ye Shi, Jingya Wang, Hoang Duong Tuan, H Vincent Poor, and Dacheng Tao. Alternating differentiation for optimization layers. International Conference on Learning Representations , 2023.

Ezra Winston and J Zico Kolter. Monotone operator equilibrium networks. Advances in neural information processing systems , 33:10718-10728, 2020.

Xingyu Xie, Qiuhao Wang, Zenan Ling, Xia Li, Guangcan Liu, and Zhouchen Lin. Optimization induced equilibrium networks: An explicit optimization perspective for understanding equilibrium models. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):3604-3616, 2023.

Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124012, 2021.

Weiming Zhi, Tin Lai, Lionel Ott, Edwin V Bonilla, and Fabio Ramos. Learning efficient and robust ordinary differential equations via invertible neural networks. In International Conference on Machine Learning , pages 27060-27074. PMLR, 2022.

Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. In International Conference on Learning Representations .

Juntang Zhuang, Nicha Dvornek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ode. In International Conference on Machine Learning , pages 11639-11649. PMLR, 2020.

A Background of Homotopy Continuation

Homotopy mapping H ( z, λ ) = λr ( z ) + (1 -λ ) g ( z ) provides a continuous transformation by gradually deforming g ( z ) into r ( z ) while λ increases from 0 to 1 in small increments. The solution to r ( z ) can be found by following the zero path of the homotopy mapping H ( z, λ ) = 0 . Usually, one can choose an artificial function g ( z ) with an easy solution. Figure 6 shows the transformation of homotopy mapping with λ increasing from 0 to 1 , the homotopy function goes from an artificial, "easy" problem to the nonlinear problem in which we are interested.

A.1 Global Convergence of Homotopy Continuation

Here, we briefly recall the theoretical foundation of homotopy methods to show its global convergence with probability one.

Definition 2 Let E n denote n -dimensional real Euclidean space, let U ⊂ E n and V ⊂ E m be open sets, and let H : U × V × [0 , 1) → E n be a C 2 mapping. H is said to be transversal to zero if the Jacobian matrix ∇ H has full rank on H -1 (0) .

Theorem 1. (Parametrized Sard's Theorem [1]) If H ( z 0 , z, λ ) is transversal to zero, then for almost all z 0 ∈ U the mapping

is also transversal to zero; i.e., with probability one the Jacobian matrix ∇ H z 0 ( λ, z ) has full rank on H -1 z 0 (0) .

The method for constructing a homotopy algorithm to solve the nonlinear system r ( z ) = 0 with global convergence is as follows: 1) H ( z 0 , z, λ ) is transversal to zero; 2) H z 0 ( z, 0) = H ( z 0 , z, 0) is trivial to solve and has a unique solution z 0 ; 3) H z 0 ( z, 1) = r ( z ) ; 4) H -1 z 0 (0) is bounded.

Then for almost all z 0 ∈ U there exist a zero path s of H z 0 , along with the Jacobian matrix ∇ H z 0 has rank n . The zero path starts from (0 , z 0 ) and reaching z ⋆ at λ = 1 . This zero path s does not intersect itself, and it is disjoint from any other zero paths of H z 0 . Furthermore, if ∇ r ( z ) is nonsingular, then the zero path s has a finite arc length.

A.2 Fixed Point Homotopy Continuation

One commonly used homotopy function to find solutions of r ( z ) = 0 is the Fixed Point Homotopy [11] given by:

where z 0 ∈ R n and λ in unit interval [0 , 1] . At λ = 0 , the starting system is H ( z, 0) = z -z 0 = 0 for which the only solution is z = z 0 . At λ = 1 , the system H ( z, 1) = r ( z ) = 0 is the system of equations of interest.

For U ⊂ R n , we use int U to denote the interior of U . And we say H is boundary-free at λ 0 ∈ [0 , 1] if z / ∈ ∂U for any z ∈ H | -1 λ = λ 0 ( { 0 } ) . Generally, we say H is boundary-free for λ in a subset S ⊂ [0 , 1] if H is boundary-free for all λ ∈ S . The following theorem provides the fixed point of f ( z ) under the existence of Fixed Point Homotopy.

̸

Theorem 2. (Fixed Point Theorem [11]) Given smooth function f : U → R n , let U ∈ R n be compact and int U = ⊘ . For some z 0 ∈ int U , if H : U × [0 , 1] → R n is boundary-free for 0 ≤ λ ≤ 1 , where

then f has a fixed point, i.e., there exists an z ⋆ ∈ U such that f ( z ⋆ ) = z ⋆ .

A.3 Newton Homotopy

Another commonly used homotopy function is the Newton homotopy [11], which is defined as follows:

where r : R n → R n is the smooth system of interest, and z 0 is a generically chosen point in R n .

Notably, there is a close connection between the Newton homotopy and the well-known Newton's method [2] for solving nonlinear equations. Given ∇ r ( z ) is nonsingular, we can apply the differentiation on the zero path of the homotopy mapping, i.e. H ( z, λ ) = 0 , yielding the initial value problem:

Applying Euler's method at λ = 0 with step size 1 to the above ODE (24) from the initial point z = z (0) , the approximation of next iteration z (1) becomes:

Apparently, (25) is a single iteration of Newton's method. Hence, Newton's iteration can be considered as the application of Euler's method with step size 1 on the solution curve given by the Newton homotopy. However, in contrast to Newton's method, which is generally a local method, the Newton homotopy exhibits certain global convergence properties.

0

0.5

1

4.5

5

5.5

6

6.5

z

v=0.1

0

0.5

1

4.5

5

5.5

6

6.5

v=1

0

0.5

1

4.5

5

5.5

6

6.5

0

0.5

1

4.5

5

5.5

6

6.5

v=100

B The impact of different values of velocity v for the homotopy continuation

In Section 4 we introduce v := ds dt as the velocity of the point ( z, λ ) traveling along the zero path, and modify the normalization condition by introducing v into (3). Here we demonstrate that the convergence of the homotopy method is not affected by such modification by numerical experiments.

We choose a nontrivial nonlinear equation and solve it via homotopy continuation under different velocities v . Figure 7 shows the zero paths of homotopy continuation under different velocities v . Apparently, the zero paths are all on the same trajectory and heading for the same solution. Hence, the convergence of the homotopy method doesn't depend on the change in the value of velocity. With the modified normalization condition, the underlying dynamics become:

Following Neural ODEs, we can employ neural networks to approximate the differential equation above. The existence of a corresponding equilibrium-point-finding problem for Neural ODE is also guaranteed by the modified normalization condition. Specifically, one can refer to (11) (12). Hence, Neural ODEs can be regarded as the procedure of solving an equilibrium-point-finding problem with homotopy continuation.

C Bridging DEQ and Neural ODE from another type of Homotopy method

We have established the connection between DEQs and Neural ODEs via Fixed Point Homotopy in Section 4. Here, we also show that we can show a similar connection using another type of Homotopy method, i.e., Newton Homotopy.

By defining r ( z ) = z -f ( z ; θ ) , and setting z 0 = 0 for simplification, from (23) we have:

Different from Fixed Point Homotopy, we can associate z directly with λ without introducing v . From (24) we have:

Here we can view λ as the time information for the differential equation, and z (1) is the solution z ⋆ of the corresponding equilibrium-point-finding problem when λ = 1 . We can also employ neural networks to approximate the differential equation as Neural ODEs do. Therefore, Neural ODEs can also be regarded as the procedure of solving an equilibrium-point-finding problem via Newton Homotopy.

D Stability of HomoODE

This section demonstrates the stability of HomoODE based on Picard-Lindeöf Theorem [29].

Theorem 3. (Picard-Lindeöf Theorem [29]) Let I : [ a, b ] be an interval, let f : I × R n → R n be a function, and let

be the associated ordinary differential equation. If f is Lipschitz continuous in the second argument z , then this ODE possesses a unique solution on [ a, a + ϵ ] for each possible initial value z (0) = z 0 ∈ R n , where ϵ < 1 L , L is the Lipschitz constant of the second argument of f .

HomoODE solves the equilibrium-point-finding problem implicitly using a modified Neural ODE via homotopy continuation. Hence it also has the stability of Neural ODE, which can be explained by the Picard-Lindeöf Theorem. Assuming the underlying dynamic of HomoODE is Lipschitz continuous in z , then both existence and uniqueness can be guaranteed by the Picard-Lindeöf Theorem.

E Accelearation for HomoODE

In Section 6, it is mentioned that the update on the shared initial point z 0 is equivalent to maintaining a dynamic geometrical center of the equilibrium points of all the samples. We will show the correctness of this proposition and illustrate the relationship between the learning rate and the step of the dynamic update.

Proposition 1. The update on the shared initial point z 0 is equivalent to maintaining a dynamic geometrical center of the equilibrium points of all the samples.

Proof. Recall that the loss function of z 0 is defined as L ( z 0 ) := E x ∼D [ ( z ⋆ ( x ) -z 0 ) 2 ] . According to the definition of the variance, we have

Define ¯ z ⋆ := E ( z ⋆ ( x )) , we obtain

Here Var( z ⋆ ( x )) is irrelevant to z 0 . Hence, min z 0 E [ ( z ⋆ ( x ) -z 0 ) 2 ] is equivalent to minimizing min z 0 (¯ z ⋆ -z 0 ) 2 , which means the update on z 0 is equivalent to maintaining a geometrical center of the equilibrium points of all the samples.

According to Proposition 1, we can also write the update as the form of z 0 = α ¯ z ⋆ +(1 -α ) z 0 , which is more straightforward. Therefore, we also want to further explore the relationship between the learning rate η init and update step α . Consider the gradient descent on z 0

Since z 0 is the broadcast tensor from initial information ˜ z 0 , the actual gradient descent has the form of

where h, w are the height and width of the feature map respectively. Finally, we obtain the relationship between η init and α is η init = hw 2 α . This means we can set a large learning rate for η init even greater than 1, especially when the feature map is large.

F Adjoint Method for HomoODE

Algorithm 1 Adjoint Method for HomoODE

The adjoint method [10, 42] is an efficient backpropagation method that can save the memory footprint in Neural ODE during training. However, there is a minor difference when we apply the adjoint method in the training of HomoODE. Unlike Neural ODE which computes the gradient with respect to z ( t 0 ) , our HomoODE calculates the gradient with respect to the condition x instead. Accordingly, we modify the adjoint method in Neural ODE and present the computation procedure in Algorithm 1.

G Additional Experimental Results

We conducted additional experiments in CIFAR-100 [34] and Tiny ImageNet [37] to validate the potential of applying HomoODE to difficult image classification tasks with larger model sizes. Specifically, we extend the channel numbers of the convolutional layers to 128 in HomoODE and increase the model size of the compared implicit models correspondingly for fairness. The experiments in CIFAR-100 and Tiny ImageNet are all implemented with data augmentation. The concrete operations in data augmentation involve zero-padding the 32×32 images to 40×40 and then performing random horizontal flips. As shown in Table 4, the performance of HomoODE is much better than other implicit models in terms of both memory consumption and test accuracy.

Besides, we also performed the ablation study on the learning rate for the shared initial point as shown in Figure 8. Notably, the ablation study is implemented with the adjoint method. Given the practicality of the adjoint method in handling the backpropagation of large deep models, we are inclined to explore a wider range of characters when utilizing it. It can be observed that HomoODE is robust to the learning rate for the shared initial point and there exists a slight improvement when lr = 0 . 1 . We believe the reason is the same as the slight improvement in contrast experiments of HomoODE with/without acceleration. That is, the larger learning rate for the shared initial point makes the initial point z 0 closer to the current geometrical center of the equilibrium points of all samples, which further leads to the shorter zero path that the adjoint method goes through. Finally, it reduces the error of backpropagated gradient.

H Hyper-parameter Settings

Our experiments are implemented on a GPU of NVIDIA GeForce RTX 3090 with 24GB. The hyper-parameters applied in HomoODE are shown in Table 5. Experiments related to other implicit













































































models are based on deq 2 , monotone_op_net 3 and augmented-neural-odes 4 . Notably, the dropout layer in our experiments follows the variational dropout operation in [7] because the traditional dropout operation hurts the stability of convergence to the equilibrium.

I Details on Tested Equation in Section 6

With respect to the details on the tested nonlinear equation mentioned in Section 6, we randomly choose an equation with high nonlinearity that is

and one solution of it is 6 . 4217 . Our experiment shown in Figure 2 is based on the initial points around this solution. Actually, we found that even if we modify the parameters in this function arbitrarily, a similar result can be obtained as well.

J More Discussions on Learnable Initial Point

It is worth noting that we also carried out some trials with an input-based initial point predictor [8] to accelerate HomoODE. However, the performance of the initial point predictor in HomoODE is not desirable in terms of both inference time and test accuracy. Based on the comprehensive analysis we provided in Section B, C, this is probably because too frequent or too large updates on the initial point will destroy the stable link between ODE function F ( z ( t ) , t ; x, θ ) and the equilibrium transformation function f ( z ; x, θ ) . In particular, the additionally-introduced variable v will change sharply when a large change is applied to the initial point. In this case, the function λ ( t ) will change dramatically, eventually ruining the trained equilibrium transformation function f ( z ; x, θ ) . This phenomenon can also be illustrated from the perspective of Newton homotopy. Note that the ODE function is actually determined by the initial point. Therefore, once the initial point is changed, the ODE function will change and then the trained ODE function will be ruined.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that the fixed point equation solved by a Deep Equilibrium Model (DEQ) can be written as a Neural ODE.This is achieved by first introducing the equation of the solution path for the Homotopy equation, which is referred to as the Fixed Point Homotopy equation, and then deriving the appropriate homotopy mapping for the fixed point equation solved by a DEQ. Hence, they introduce an ODE that solves for a fixed point of a system (using homotopy continuation) and therefore claim that DEQs and neural ODEs are “essentially two-sides of the same coin”.\n\nThe authors then introduced HomoODE, that uses the input injection from a DEQ and the derived fixed-point-ODE to solve for a task, however instead of solving for the fixed point using newton-method in a DEQ homoODE uses a normal ode-solver similar to neural ODEs. The input injection helps in getting better performance than neural ODE and the implicit solving using ODE solver results in a more memory efficient methodology\n\nSTRENGTHS:\nThe paper is well written, and effectively establishes the connection between neural ODEs and DEQs. Furthermore, they introduce HomoODE, which inherits the better performance capabilities of a DEQ, and has a lesser memory footprint (thanks to the ODEsolver instead of newton method based solver, used in Neural ODEs). \n\nThe experiments show on various classification tasks that HomoODE matches or beats DEQs in various benchmarks with fewer model parameters and with faster inference time.\n\nWEAKNESSES:\nThere are some details missing in some parts of the paper, I have pointed them out in the Questions section.\n\nQUESTIONS:\n- In lines 205-210 the authors mention that neural ODE is a fixed-point problem with a “fixed-condition” x with different initial point z(t_0). I am not sure if I get that point. Presumably the fixed point used to derived the fixed point ODE could be conditioned on the input $x$ as well.\n- The derived equation in equation 12 has $F(z(t), t; x; \\theta)$ under the square root, is that a typo? or, there is some kind of “input injection” of $x$ that is implicitly assumed in Neural ODE that solves an equilibrium-point-finding problem.\n- How is $v$, the velocity in equation 9 chosen in the final neural ODE?\n- In Section 6, the authors mention the homotopy continuation on a nonlinear equation in using matlab. The details on which equation and what was the purpose of that experiment is missing. I am unsure about the conclusion of that section, do that authors learn a common initialization for all samples $x$ in the dataset, and is that learned or just an average over the entire dataset?\n\nLIMITATIONS:\nAuthors have discussed the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper unifies DEQs and Neural ODEs via homotopy continuation. The authors propose a novel implicit model, HomoODE, which inherits the property of high accuracy from DEQs and the property of stability from Neural ODEs. Moreover, they develop an acceleration method by borrowing the idea from the DEQ-solver.

STRENGTHS:
1 This paper reveals the inherent connection between two typical implicit models, i.e., DEQs and ODEs.  This connection is really insightful. This paper, in my opinion, contributes significantly to our knowledge of implicit models.

2 The proposed HomoODE combines the advantages of DEQs and ODEs. The authors leverage homotopy continuation, and thus HomoODEs do not suffer from the issue of unique equilibrium like DEQs. The proposed acceleration method is also nice. The authors provide a new design paradigm for implicit models.

WEAKNESSES:
1 This paper argures that "DEQs and Neural ODEs are two sides of the same coin", but compared with the analysis on  "two sides" is less than that on "the same coin."   

2 More experiments on large scale datasets, e.g., ImageNet, would be nice.

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
The authors have adequately addressed the limitations, and there is no potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

comments: SUMMARY:
The paper uses homotopy continuation to show that Deep Equilibrium Models and Neural ODEs are effectively equivalent. They use the best properties of both worlds: DEQs for their higher accuracy and Neural ODEs for their stability to propose a new implicit method HomoODE.

STRENGTHS:
* Theoretical explorations of the relationship between Neural ODEs and DEQs are limited, and the paper does a good job of linking the two ideas.

WEAKNESSES:
* Reformulating DEQs as Neural ODEs have been done in a prior work [1], which has not been mentioned or compared against
* MNIST and SVHN are too small-scale experiments and can lead to incorrect extrapolations on the method's benefits. I would recommend including larger problems like ImageNet.
* Some information on the baseline would be useful: Typically, solving discrete dynamical systems (DEQs) is faster than a continuous dynamical system. So the 8x inference time might be an artifact of poor solver choice for the DEQ.

[1] Pal, Avik, Alan Edelman, and Christopher Rackauckas. "Continuous Deep Equilibrium Models: Training Neural ODEs faster by integrating them to Infinity." arXiv preprint arXiv:2201.12240 (2022).

QUESTIONS:
See the section on weaknesses

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work shows the connection between Deep Equilibrium Moldes (DEQs) and Neural Ordinary Differential Equations (NODEs) using homotopy continuation, a method to solve nonlinear equations. Based on the analysis, the authors suggest a new implicit model, HomoODE, which has high expressible power from DEQs and stability from NODEs. The method is demonstrated to obtain the best performance for image classification tasks between the models tested.\n\nSTRENGTHS:\n* The connection between DEQ and NODE is presented clearly and seems mathematically solid, which delivers a lot of insights to the community (in particular, in the field of implicit models).\n* In addition to mathematical notions, the paper covers practical ideas, e.g., the learnable initial condition. The effect of the adjoint method is also investigated.\n\nWEAKNESSES:\n* The reason why the authors choose image classification is not very clear. With the current manuscript, the practical contribution of the paper looks quite limited because DEQ is designed for sequential data, and NODE is for ODEs. Thus, the reviewer cannot assess the effectiveness of HomoODE because image classification is not a main field of the baseline methods. The reviewer strongly recommends the authors run experiments on ODEs to show real improvement compared to DEQ and NODE. \n* Related to the first point, the reason why the HomoODE works better in the image classification task is not clear. The reviewer did not find out what the authors would like to state with these experiments. I believe there are a lot of excellent models to compare in the field of image classification if the authors would like to claim the model's effectiveness in the image classification tasks. It could be understandable if the image classification task is to show the adaptability of the model, and there is another main experiment (e.g., on an ODE dataset).\n\nQUESTIONS:\n* How does one determine $v$? Is it a hyperparameter or computed using some equation?\n* What do \"explicit\" and \"implicit\" mean in, e.g., \"unlike DEQs, which explicitly solve equilibrium-point-finding problems via Newton's methods, our HomoODE solves these problems based on homotopy continuation implicitly\"? It could be unclear and require some explanation because the meaning of \"implicit\" seems different from that of \"implicit NN.\"\n* What are \"two initial conditions\" in \"there might be no solution for $\\lambda(t)$ as there are two initial conditions for the system\"?\n* What do $h$, $w$, and $c$ stand for in line 265?\n* Why is the inference time of HomoODEs with data augmentation longer than those without data augmentation in Table 1 (e.g., HomoODE vs. HomoODE$^*$, or HomoODE$^\\dagger$ vs. HomoODE$^{*\\dagger}$)? Is it due to increased iterations due to training with the augmented dataset?\n* Could you please elaborate on the meaning of \"fixed condition\" in \"Neural ODEs solve an equilibrium-point-finding problem with fixed condition $x$ and different initial points $z(t_0)$\"? The statement seems to contradict because Equation (5) states that $z(t_0) = x$, so different $z(t_0)$ implies different $x$. In addition, Neural ODE can take $t$ as an input, which evolves in time.\n\nLIMITATIONS:\n* The method to optimize the shared initial information seems tricky and could be stated as a part of the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}