SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shutong Ding*
ShanghaiTech University  
dingsht@shanghaitech.edu.cn
Tianyu Cui*
ShanghaiTech University cuity2022@shanghaitech.edu.cn
ShanghaiTech University wangjingya@shanghaitech.edu.cn
Ye Shi†  
ShanghaiTech University  
shiye@shanghaitech.edu.cn
Deep Equilibrium Models (DEQs) and Neural Ordinary Differential Equations (Neural ODEs) are two branches of implicit models that have achieved remarkable success owing to their superior performance and low memory consumption. While both are implicit models, DEQs and Neural ODEs are derived from different mathematical formulations. Inspired by homotopy continuation, we establish a connection between these two models and illustrate that they are actually two sides of the same coin. Homotopy continuation is a classical method of solving nonlinear equations based on a corresponding ODE. Given this connection, we proposed a new implicit model called HomoODE that inherits the property of high accuracy from DEQs and the property of stability from Neural ODEs. Unlike DEQs, which explicitly solve an equilibrium-point-finding problem via Newton's methods in the forward pass, HomoODE solves the equilibrium-point-finding problem implicitly using a modified Neural ODE via homotopy continuation. Further, we developed an acceleration method for HomoODE with a shared learnable initial point. It is worth noting that our model also provides a better understanding of why Augmented Neural ODEs work as long as the augmented part is regarded as the equilibrium point to find. Comprehensive experiments with several image classification tasks demonstrate that HomoODE surpasses existing implicit models in terms of both accuracy and memory consumption.
Recent studies of implicit models have certified that such models can meet or even surpass the performance of traditional deep neural networks. Instead of specifying the explicit computation process of the output, implicit models define the joint conditions that the layer's input and output satisfy. Instances of such models include the Neural Ordinary Differential Equations (Neural ODEs) [10], which treat ODEs as a learnable component and can be viewed as continuous Residual Networks [25] (ResNets), the Deep Equilibrium Models (DEQs) [6], which compute the equilibrium point of a nonlinear transformation corresponding to an infinite-depth weight-tied network; and optimization layers [4, 45], which leverage the optimization techniques as layers of neural networks.
Although DEQs and Neural ODEs, as popular implicit models in recent years, have garnered much attention in terms of theoretical analysis and application, an insightful connection between these two
branches of the implicit model has not been established. DEQs involve an equilibrium-point-finding problem, which is a nonlinear equation system. Generally, this can be solved via the homotopy continuation method [39], a classical method that solves nonlinear equations along the zero path of the homotopy mapping and can be further formulated as an ODE. This motivated us to consider whether we could bridge DEQs and Neural ODEs via the theory of homotopy continuation.
In this paper, we show that Neural ODEs can also be viewed as a procedure for solving an equilibrium-point-finding problem. However, while both of the two models can be considered as solving equilibrium-point-finding problems, they differ in how the input information is used. On the one hand, DEQs regard the input information as the condition that determines the equilibrium-point-finding problem to solve via injecting it into the equilibrium function in the forward pass. On the other hand, Neural ODEs generate the initial points with different input information and expect them to converge to different equilibrium points. Therefore, we claim that DEQs and Neural ODEs are actually two sides of the same coin.
Inspired by the theoretical connection between the two models, we developed a new implicit model called HomoODE, which inherits the advantages of both. Specifically, HomoODE injects the input information into the underlying dynamic of an equilibrium-point-finding problem in the same way as a DEQ does, but then obtains the output from an ODE solver just as a Neural ODE does. The connection between DEQ and Neural ODE means that HomoODE avoids DEQ's problem of unstable convergence in DEQs and the weak representation capability of Neural ODEs. Further, a common issue for implicit models is the trade-off between computational speed and precision. Therefore, a natural intuition to accelerate such models is to find a good initial point that is close to the solution. We observed that, as the distance between the initial point and the solution drops to some range, the numbers of function evaluations (NFE) almost stop decreasing when applying homotopy continuation. Hence, we introduced an extra loss instead of zero vector initialization in DEQ to train a shared initial point. In this way, the distance from the initial point to each solution is within appropriate ranges. A series of experiments with several image classification tasks verify that HomoODE is able to converge stably and that it outperforms existing implicit models with better memory efficiency on several image classification tasks, and our acceleration method significantly reduces the NFE of HomoODE. In summary, our contributions are as follows:
(1) A connection between DEQs and Neural ODEs. We establish a connection between DEQs and Neural ODEs via homotopy continuation, which illustrates that DEQs and Neural ODEs are actually the two sides of the same coin. We believe this new perspective provides novel insights into the mechanisms behind implicit models.
(2) A New Implicit Model: HomoODE. We propose a new implicit model called HomoODE, that inherits the advantages of both DEQs and Neural ODEs. HomoODE implicitly solves equilibrium-point-finding problems using homotopy continuation, unlike DEQs which explicitly solve these problems via Newton's methods. Additionally, we have accelerated HomoODE with a learnable initial point that is shared among all samples.
(3) Understanding Augmented Neural ODE. We demonstrate that Augmented Neural ODE can be treated as a special case of HomoODE based on homotopy continuation. Hence Augmented Neural ODE enjoys better representation ability and outperforms Neural ODE.
(4) Better Performance. We conduct experiments on image classification datasets and confirm our HomoODE outperforms DEQ, Neural ODE, and variants of them both in accuracy and memory usage. Furthermore, we also perform the sensitivity analysis on the hyper-parameters to research the characters of our model.
Deep Equilibrium Models. DEQs [6] have shown competitive performance on a range of tasks, such as language modeling[6], graph-related tasks [20], image classification or segmentation [7], image generation [41], inverse problems in imaging[16], image denoising [17] and optical flow estimation [5]. DEQs find an equilibrium point of a nonlinear dynamical system corresponding to an effectively infinite-depth weight-tied network. However, training such models requires careful consideration of both initializations and the model structure [6, 9, 3], and often consumes long training times. Many studies have been devoted to solving these problems. For example, the Monotone Operator
Equilibrium Network (monDEQ) [46] ensures stable convergence to a unique equilibrium point by involving monotone operator theory. Bai et al. [9] propose an explicit regularization scheme for DEQs that stabilizes the learning of DEQs by regularizing the Jacobian matrix of the fixed-point update equations. Kawaguchi et al.[30] prove that DEQs converge to global optimum at a linear rate for a general class of loss functions by analyzing its gradient dynamics. Agarwala et al.[3] show that DEQs are sensitive to the higher-order statistics of their initial matrix family and consequently propose a practical prescription for initialization. To reduce the computational expense of DEQ, Pal et al. [40] developed continuous DEQs utilizing an "infinity time" neural ODE but did not continue to explore the inherent connection between general DEQs and Neural ODEs. From the perspective of optimization, Optimization Induced Equilibrium Networks (OptEq)[47] theoretically connect their equilibrium point to the solution of a convex optimization problem with explicit objectives. Instead of regularizing the structures or involving parameterizations of the implicit layer design, Bai et al. [8] propose a model-specific equilibrium solver, which both guesses an initial value of the optimization and performs iterative updates. However, unlike DEQs, which explicitly solve equilibrium-point-finding problems via Newton's methods, our HomoODE solves these problems based on homotopy continuation implicitly. Accordingly, HomoODE does not suffer from the issue of unique equilibrium like DEQ and thus can avoid the stability issue. Moreover, we accelerate HomoODE using a good initial point learned with a corresponding loss function. Unlike Bai et al.'s approach [8], HomoODE learns an initial point shared among all samples without involving a network-based initializer.
Neural Ordinary Differential Equations. Neural ODEs have been applied to time series modeling [44, 32], continuous normalizing flows [10, 18], and modeling or controlling physical environments [50, 43, 48, 19, 14]. Neural ODEs treat ODEs as a learnable component and produce their outputs by solving the Initial Value Problem [10, 12, 29]. However, Neural ODEs are often characterized by long training times and sub-optimal results when the length of the training data increases [13, 15]. Prior works have tried to tackle these problems by placing constraints on the Jacobian[13] or high derivatives of the differential equation [31]. Conversely, Augmented Neural ODEs [12] learn the flow from the input to the features in an augmented space with better stability and generalization. Ghosh et al. [15] treat the integration time points as stochastic variables without placing any constraints. With diffeomorphism, the complexity of modeling the Neural ODEs can be offloaded onto the invertible neural networks [49], and training Neural ODEs with the adaptive checkpoint adjoint method [51] can be accurate, fast, and robust to initialization. The symplectic adjoint method [36] finds the exact gradient via a symplectic integrator with appropriate checkpoints and memory consumption that is competitive to the adjoint method. The advantage of HomoODE is that it inherits the property of high accuracy from DEQs and the property of stability from Neural ODEs. In addition, HomoODE provides an explanation of why Augmented Neural ODEs achieve better performance than Neural ODEs. Notably, Augmented Neural ODEs can be viewed as a special case of HomoODE.
Homotopy Continuation. Homotopy continuation [39] is a numerical technique that traces the solution path of a given problem as the parameter changes from an initial value to a final value. Homotopy methods have been successfully applied to solving pattern formation problems arising from computational mathematics and biology including computing multiple solutions of differential equations [21, 22], state problems of hyperbolic conservation laws [21], computing bifurcation points of nonlinear systems [24] and solving reaction-diffusion equations [23]. Recent advances in deep learning have also seen the homotopy continuation method fused into learning processes. For instance, Ko et al.[33] adapt homotopy optimization in Neural ODEs to gain better performance with less more training epochs. HomPINNs [27] traces observations in an approximate manner to identify multiple solutions, then solves the inverse problem via the homotopy continuation method. To reach a good solution to the original geometrical problem, Hruby et al. [26] learn a single starting point for a real homotopy continuation path. In this work, we establish a connection between DEQs and Neural ODEs from the perspective of homotopy continuation and develop a new implicit model called HomoODE based on this theoretical relationship.
3 Background on Homotopy Continuation
Homotopy continuation has been broadly applied to solve nonlinear equations. The first step to solving a specific problem  $r(z) = 0$  is to construct a homotopy mapping.
Definition 1 (Homotopy mapping [39]) The function  $H(z, \lambda) = \lambda r(z) + (1 - \lambda)g(z)$  is said to be a homotopy mapping from  $g(z)$  to  $r(z)$ , if  $\lambda$  is a scalar parameter from 0 to 1, and  $g(z)$  is a smooth function. The equation  $H(z, \lambda) = 0$  is the zero path of this homotopy mapping.
Homotopy mapping provides a continuous transformation by gradually deforming  $g(z)$  into  $r(z)$  while  $\lambda$  increases from 0 to 1 in small increments. Hence, the solution to  $r(z)$  can be found by following the zero path of the homotopy mapping  $H(z, \lambda) = 0$ . Usually, one can choose  $g(z)$  as an artificial function with an easy solution. Here, we specifically consider Fixed Point Homotopy which chooses  $g(z) = z - z_0$ :
$$
H (z, \lambda) = \lambda r (z) + (1 - \lambda) \left(z - z _ {0}\right), \tag {1}
$$
where  $z_0 \in \mathbb{R}^n$  is a fixed vector, and is the initial point of the homotopy continuation method. In one practical trick, we can follow the zero path by allowing both  $z$  and  $\lambda$  to be functions of an independent variable  $s$ , which represents arc length along the path. In other words,  $(z(s), \lambda(s))$  is the point arrived at by traveling a distance  $s$  along the zero path from the initial point  $(z(0), \lambda(0)) = (z_0, 0)$ . In the zero path, we have  $H(z(s), \lambda(s)) = 0$ , for all  $s \geq 0$ .
Take the derivative for this equation with respect to  $s$  lead to:
$$
\frac {\partial H (z , \lambda)}{\partial z} \frac {d z}{d s} + \frac {\partial H (z , \lambda)}{\partial \lambda} \frac {d \lambda}{d s} = 0. \tag {2}
$$
The vector  $\left(\frac{dz}{ds},\frac{d\lambda}{ds}\right)\in \mathbb{R}^{n + 1}$  is the tangent vector to the zero path, and it lies in the null space of matrix  $\left[\frac{\partial H(z,\lambda)}{\partial z},\frac{\partial H(z,\lambda)}{\partial\lambda}\right]\in \mathbb{R}^{n\times (n + 1)}$ . To complete the definition of  $(\frac{dz}{ds},\frac{d\lambda}{ds})$ , a normalization condition is imposed to fix the length of the tangent vector, i.e.
$$
\left\| \frac {d z}{d s} \right\| ^ {2} + \left| \frac {d \lambda}{d s} \right| ^ {2} = 1. \tag {3}
$$
Given the tangent vector and the initial point, we can trace the zero path and obtain the solution of  $F(z) = 0$  by solving the ODE (2).
4 Bridging DEQs & Neural ODEs via Homotopy Continuation
Here we briefly review DEQs and Neural ODEs, then bridge these two models via homotopy continuation. DEQs aim to solve the equilibrium point of the function  $f(z;x,\theta)$ , which is parameterized by  $\theta$  and the input injection  $x$ . The underlying equilibrium-point-finding problem of DEQs is defined as follows:
$$
z ^ {\star} = f \left(z ^ {\star}; x, \theta\right). \tag {4}
$$
Usually, we choose  $f(z; x, \theta)$  as a shallow stacked neural layer or block. Hence, the process of solving the equilibrium point can be viewed as modeling the "infinite-depth" representation of a shallow stacked block. One can use any black-box root-finding solver or fixed point iteration to obtain the equilibrium point  $z^*$ .
Unlike the underlying equilibrium-point-finding problem of DEQs, Neural ODEs view its underlying problem as an ODE, whose derivative is parameterized by the network. Specifically, Neural ODEs map a data point  $x$  into a set of features by solving the Initial Value Problem [29] to some time  $T$ . The underlying ODE of Neural ODEs is defined as follows:
$$
\frac {d z (t)}{d t} = F (z (t), t; \theta), \quad z \left(t _ {0}\right) = x, \tag {5}
$$
where  $z(t)$  represents the hidden state at time  $t$ ,  $F(z(t),t;\theta)$  is neural networks parameterized by  $\theta$ .
The same coin. DEQs apply Newton's methods to solve the underlying equilibrium-point-finding problem  $z = f(z;x,\theta)$ . By defining  $r(z) = z - f(z;x,\theta)$ , one can alternatively solve this equilibrium-point-finding problem based on homotopy continuation. Now we will show that the underlying dynamics in Neural ODEs can also be treated as an equilibrium-point-finding problem.
Firstly, we apply the Fixed Point Homotopy to solve the equilibrium-point-finding problem  $z = f(z; \theta)$  and obtain the homotopy mapping  $H(z, \lambda) = \lambda (z - f(z; \theta)) + (1 - \lambda)z$ . Taking the partial derivative of  $H(z, \lambda)$  with respect to  $z$  and  $\lambda$ , respectively, we obtain
$$
\frac {\partial H (z , \lambda)}{\partial z} = I - \lambda \nabla_ {z} f (z; \theta), \frac {\partial H (z , \lambda)}{\partial \lambda} = - f (z; \theta). \tag {6}
$$
By substituting the partial derivative in (6) into (2), we can obtain:
$$
\frac {d z}{d s} = \left(I - \lambda \nabla_ {z} f (z; \theta)\right) ^ {- 1} f (z; \theta) \frac {d \lambda}{d s}. \tag {7}
$$
Based on the normalization condition (3), we can reformulate (7) as the following differential equation:
$$
\frac {d z}{d s} = \left(I - \lambda (z) \nabla_ {z} f (z; \theta)\right) ^ {- 1} f (z; \theta) \sqrt {1 - \left\| \frac {d z}{d s} \right\| ^ {2}}. \tag {8}
$$
As Neural ODEs do, we can use neural networks to approximate the underlying dynamics of such an ODE (8). However, the norm of neural network output is likely to exceed the unit length, i.e. violating the normalization condition (3). To address this issue, we introduce  $v \coloneqq \frac{ds}{dt}$  as the velocity of the point  $(z,\lambda)$  traveling along the zero path, and modify the normalization condition by introducing  $v$  into (3):
$$
\left\| \frac {d z}{d t} \right\| ^ {2} + \left| \frac {d \lambda}{d t} \right| ^ {2} = v ^ {2}. \tag {9}
$$
Note that the convergence of the homotopy continuation is not affected by the value of  $v$ . The underlying dynamics of (7) becomes
$$
\frac {d z}{d t} = (I - \lambda (z) \nabla_ {z} f (z; \theta)) ^ {- 1} f (z; \theta) \sqrt {v ^ {2} - \left\| \frac {d z}{d t} \right\| ^ {2}}. \tag {10}
$$
Following Neural ODEs, the differential equation (10) can be approximated by neural networks, i.e.  $\frac{dz}{dt} = F(z(t), t; \theta)$ . However, we still need to ensure the existence of a corresponding equilibrium-point-finding problem for Neural ODE. Ingeniously, the modified normalization can also ensure the existence of the equilibrium-point-finding problem. When we obtain the Neural ODEs (5) by training the neural networks  $F(z(t), t; \theta)$ , we can compute the changing process of  $\lambda(t)$  and the velocity  $v$  by solving the following equations:
$$
\frac {d \lambda}{d t} = \sqrt {v ^ {2} - \| F (z (t) , t ; \theta) \| ^ {2}}, \quad \lambda (0) = 0, \quad \lambda (1) = 1. \tag {11}
$$
Note that the modified normalization condition (9) provides the dynamic with another degree of freedom, which guarantees the existence of  $\lambda(t)$ . Otherwise, there might be no solution for  $\lambda(t)$  as there are two initial conditions (i.e.,  $\lambda(0) = 0$ ,  $\lambda(1) = 1$ ) for the system. Hence, the equilibrium-point-finding problem  $z = f(z; \theta)$  is implicitly determined by the following partial differential equation:
$$
F (z (t), t; \theta) = (I - \lambda (t) \nabla_ {z} f (z; \theta)) ^ {- 1} f (z; \theta) \sqrt {v ^ {2} - \| F (z (t) , t ; \theta) \| ^ {2}}. \tag {12}
$$
Therefore, Neural ODEs can be regarded as the procedure of solving an equilibrium-point-finding problem with homotopy continuation, and the hidden state at  $t = 0$ ,  $z(t_0)$  is the initial point of homotopy continuation.
Two sides. We have shown that both DEQs and Neural ODEs can be considered as solving equilibrium-point-finding problems through homotopy continuation. Now we discuss the difference in underlying equilibrium-point-finding problem between DEQs and Neural ODEs.
On the one hand, DEQs solve the problem from the same initial point  $z_0$ . The equilibrium-point-finding problem of DEQs is parameterized by the input injection  $x$ . The underlying equilibrium-point-finding problem of DEQs is defined as follows:
$$
z ^ {\star} = f \left(z ^ {\star}; x, \theta\right), z ^ {(0)} = z _ {0} \tag {13}
$$
The input injection  $x$  can be viewed as the condition to fuse the information of input to the underlying equilibrium-point-finding problem. The underlying problem of DEQs varies depending on different conditions  $x$ . Therefore, DEQs are able to map inputs to diverse representations, which is crucial for achieving superior performance.
On the other hand, unlike DEQs, Neural ODEs solve an equilibrium-point-finding problem with different initial points  $z(t_0)$ . The underlying equilibrium-point-finding problem of Neural ODEs is defined as follows:
$$
z ^ {\star} = f \left(z ^ {\star}; \theta\right), \tag {14}
$$
and Neural ODEs solve the problem through homotopy continuation:
$$
\frac {d z (t)}{d t} = F (z (t), t; \theta), z \left(t _ {0}\right) = x \tag {15}
$$
where  $f$  is determined by  $F$  in equation (12). Concretely, Neural ODEs map raw data into a set of features  $z(t_0) = x$  and regard them as the initial points of the ODE. The fixed underlying problem ensures the stability of Neural ODEs but loses diversified representation capabilities. Therefore, we claim that DEQs and Neural ODEs are actually two sides of the same coin from the perspective of homotopy continuation.
5 HomoODE: an efficient and effective implicit model
Figure 1: Comparison between different running mechanisms of implicit models.
As we show above, both DEQs and Neural ODEs can be considered as solving equilibrium-point-finding problems through homotopy continuation. Two well-known approaches for solving nonlinear equations are Newton's Method [2] and homotopy continuation [39]. DEQs solve an equilibrium-point-finding problem  $r(z) = 0$  via Newton's Methods which are local in the sense that a good estimate of the solution is required for the convergence. Unlike Newton's Method, homotopy continuation is global in the sense that solutions of  $g(z) = 0$  may not need to be anywhere close to the solution of  $r(z) = 0$  [11]. Inspired by the connection between DEQs and Neural ODEs, a very natural thought is that we can apply homotopy continuation to solve the underlying nonlinear equation of DEQs. By replacing the equilibrium-point-finding problem  $z = f(z; \theta)$  to  $z = f(z; x, \theta)$ , we can obtain the following differential equation:
$$
\frac {d z}{d t} = \left(I - \lambda (z) \nabla_ {z} f (z; x, \theta)\right) ^ {- 1} f (z; x, \theta) \sqrt {v ^ {2} - \left\| \frac {d z}{d t} \right\| ^ {2}} \tag {16}
$$
Hence, we proposed a new implicit model called HomoODE, which models an equilibrium problem  $z = f(z;x,\theta)$  implicitly. Specifically, we employ neural networks to approximate the differential equation (16). In the design of the network structure, we introduce the condition  $x$  into the underlying dynamic of HomoODE as DEQ does and obtain the output from the same initial point through the ODE solver as Neural ODE does. In this way, HomoODE not only has the ability of diversified
representation of DEQs but also has the property of stable convergence of Neural ODEs. In addition, the time information  $t$  is not explicitly formulated by the dynamic of HomoODE (16). So unlike Neural ODEs, HomoODE does not require the input of time information  $t$ . Figure 1 illustrates the structure of HomoODE as well as Neural ODE and DEQ. Besides, it is worth noting that  $v$  mentioned in (16) is just an auxiliary variable for the theoretical analysis. Therefore, we do not need to include it as a hyperparameter or compute its value in the forward pass because it has been implicitly contained in the neural network.