SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Forward Pass. In HomoODE, the raw data  $x$  is first input to a feature extractor  $g(x; \omega)$  and then injected into an ODE solver. Suppose  $z(t)$  represents the intermediate state of HomoODE, calculating  $z(t)$  involves an integration starting from the initial point  $z(t_0) = 0$  to the solution  $z(t_1)$ . Notably, the output  $z(t_1)$  is equivalent to the solution  $z^{\star}$  of the implicit equilibrium-point-finding problem  $z = f(z; x, \theta)$ . Then we can use the ODE solvers to obtain the solution  $z^{\star}$  of the origin equilibrium problem and this representation can be used for downstream tasks, such as classification, regression, etc.
$$
z \left(t _ {1}\right) = \operatorname {O D E S o l v e} \left(z \left(t _ {0}\right), F \left(z (t), t; x, \theta\right), t _ {0}, t _ {1}\right) \tag {17}
$$
Backward Pass. In the backward pass of HomoODE, we can apply the adjoint sensitivity method, or straightly differentiate through the operations of the forward pass. The condition traces back to another gradient flow. More details related to the construction of HomoODE dynamics and the computation of the gradients based on the adjoint method can be referred to in the supplementary materials.
6 Acceleration for HomoODE
In practice, we observe that HomoODE needs more function evaluations in the ODE-solving process, which results from bad initialization (e.g., zero vector initialization). To address this issue, we referred to [8], which makes an input-based initial point guess with an extra neural network to accelerate the equation-solving procedure in DEQ. However, this method poses an extra cost in both memory and computation.
To avoid these drawbacks, we further investigate the relationship between the number of iterations and the distance from the initialization  $z_0$  to the solution  $z^{\star}$  in homotopy continuation. Specifically, we perform the homotopy continuation method on a nontrivial nonlinear equation using ode45 in Matlab [28]. As is shown in Figure 2, it does not bring any reduction in the number of iterations when the initial point is close enough to the equilibrium solution. This means it is unnecessary to approximate the specific sample  $x$  in HomoODE, and that is different from that in
Figure 2: Relationship between the distance from  $z_0$  to  $z^{\star}$  and the iteration number of ODE solver. The x-axis is the inversion of the distance  $1 / d$ , and the y-axis is the iteration number  $N$ .
Hence, we share the initial point for all the samples  $x$  and just maintain a scalar value for the averaged value of the whole feature map in one channel. Briefly, we just need to store a tensor with the shape of  $(1, 1, c)$  as the shared initial information  $\tilde{z}_0$ , and broadcast it into the initial point  $z_0$  with the shape of  $(h, w, c)$  when taking it into the ODE solver. Here  $h, w, c$  means the height, width, and channel number of the feature map. The loss function of  $\tilde{z}_0$  is defined as follows:
$$
\mathcal {L} \left(\tilde {z} _ {0}\right) := \mathbb {E} _ {x \sim \mathcal {D}} \left[ \left(z ^ {\star} (x) - \tilde {z} _ {0}\right) ^ {2} \right], \tag {18}
$$
where  $z^{\star}(x)$  denotes the equilibrium solution of the sample  $x$  and  $\mathcal{D}$  denotes the distribution of  $x$ . In fact, this update on  $\tilde{z}_0$  is equivalent to maintaining the dynamic geometrical center of the equilibrium points of all the samples.
7 Understanding Augmented Neural ODE by HomoODE
Augmented Neural ODEs, as a simple extension of Neural ODEs, are more expressive models and outperform Neural ODEs. Augmented Neural ODEs allow the ODE flow to lift points into the extra
dimensions to avoid trajectories crossing each other [12]. However, more theoretical analysis of how and why augmentation improves Neural ODEs is lacking. Our work provides another perspective on understanding the effectiveness of Augmented Neural ODEs. Augmented Neural ODEs formulate the augmented ODE problem as:
$$
\frac {d}{d t} \left[ \begin{array}{l} h (t) \\ a (t) \end{array} \right] = F \left(\left[ \begin{array}{l} h (t) \\ a (t) \end{array} \right], t; \theta\right), \quad \left[ \begin{array}{l} h (0) \\ a (0) \end{array} \right] = \left[ \begin{array}{l} x \\ 0 \end{array} \right], \tag {19}
$$
where  $a(t) \in \mathbb{R}^p$  denotes a point in the augmented part, and  $h(t) \in \mathbb{R}^d$  is the hidden state at time  $t$ .
Specifically, Augmented Neural ODEs can track back the ODE flow to recover the original input  $x$  by using the hidden state  $h(t)$  and the time information  $t$ . The input  $x$  is the injection of the dynamics in HomoODE. The augmented part  $a(t)$  can be viewed as the intermediate  $z(t)$  in HomoODE. In Augmented Neural ODEs, we can treat the recovered input  $x$  as the condition in HomoODE, improving its representation ability. Hence, Augmented Neural ODEs outperform Neural ODEs. However, the origin input  $x$  computed by Augmented Neural ODEs may not be accurate enough. This probably is the reason why the performance of Augmented Neural ODEs is not competitive to HomoODE.
To confirm the efficiency of HomoODE, we conduct experiments on several classical image classification datasets to compare our model with the previous implicit model, including DEQ [6], monDEQ [46], Neural ODE [10] and Augmented Neural ODE [12]. For ODE datasets, DEQs inherently are not designed to handle simulation tasks of continuous physical processes as DEQs do not involve the continuous time series in the models. Sequential datasets are not suitable as well due to the inconsistent backbones. For sequential datasets, DEQ models are based on transformer, while Neural ODEs are based on fully-connected layers. Therefore, we consider the image classification task is suitable for the fair comparison of HomoODE and baselines. Concretely, we evaluate the stability of the training process via the learning curve of accuracy in the test dataset and exhibit the performance of different implicit models in terms of accuracy, memory consumption, and inference time. It is worth noting that we also perform HomoODE with & without the data augmentation and the adjoint backpropagation technique to check their impacts on our model. Besides, we also contrast HomoODE with zero vector initialization and learnable initialization to assess the capability of our acceleration method. Our code is available at: https://github.com/wadx2019/homoode.
Figure 3: The deployed neural network architecture in HomoODE. Here,  $s$  denotes the stride and the channel number of all convolutional layers is 64.
Experimental setup. HomoODE is performed on several standard datasets, including CIFAR-10 [34], SVHN [38], and MNIST [35]. As shown in Figure 3, HomoODE contains several simple convolutional layers. Its network structure is not specially designed for image classification tasks like MDEQ [7]. Notably, the memory consumption of HomoODE is less than that of other implicit models as reported in [46, 48, 7]. As we discussed in Section 5, the time information  $t$  is not fused into the input of HomoODE, unlike Neural ODE. Besides, we optimize the shared initial information
$\tilde{z}_0$  using SGD optimizer with the learning rate 0.02 and perform the update once every 20 updates for HomoODE.
Comparison with former implicit models. Table 1 presents the performance of HomoODE with different settings and other implicit models in the CIFAR-10 dataset. It can be observed that HomoODE outperforms the previous implicit models in terms of both accuracy and memory consumption. Moreover, the inference time of HomoODE is much faster than DEQ and its variants. Notably, we also find that DEQ runs slower with larger test batch sizes. This may be due to the inefficiency of DEQ for parallel computation with large test batch sizes. In contrast, HomoODE does not exhibit this problem. Additionally, there is a large increase in accuracy in our model when the data augmentation technique is applied. It suggests that HomoODE learns better and more complex representations through augmented datasets. Furthermore, given that we adopt the dopri5 ode solver with an adaptive step, this complex representation may result in increased NFE and inference time. Overall, these phenomenons validate that Ho
moODE has a more powerful representation ability compared to other implicit models with similar model capacity. Extensive experiments in SVHN and MNIST also confirm these properties of our model as shown in Table 2.
Figure 4: Learning curve of different implicit models on CIFAR-10 datasets across 5 runs without data augmentation. The x-axis denotes the epoch during training and the y-axis denotes the accuracy of models on the test datasets.
Besides, we also plot the learning curves of different algorithms in Figure 4. The results demonstrate the stability of training HomODE compared with other methods and exhibit that HomODE is not prone to over-fitting, whereas other ODE-based models may suffer from that.
Efficiency of the learnable initialization. Figure 5 illustrates that the learnable initialization trick can improve HomoODE with about  $2.5 \times$  speedup in the inference time than before. This impact is obvious in both cases with & without the adjoint backpropagation technique. In addition, the corresponding test accuracy during the training process also reflects this acceleration technique does not bring a loss in the performance of our model. Surprisingly, it even brings a slight improvement in the case of the adjoint backpropagation technique. This is probably because a good initial point can decrease the total length of zero path  $s$ , which reduces the gradient error induced by using the adjoint method.
In this paper, we show that both DEQs and Neural ODEs can be viewed as a procedure for solving an equilibrium-point-finding problem via the theory of homotopy continuation. Motivated by this observation, we illustrate that these two implicit models are actually the two sides of the same coin. Specifically, DEQs inject the input information as the condition into the equilibrium-point-finding problem  $z^{\star} = f(z^{\star};x,\theta)$  while Neural ODEs fuse the input information into the initial point. Further, we propose a novel implicit model called HomoODE, which inherits the advantages of both DEQs and Neural ODEs. Our experiments indeed verify that HomoODE outperforms both DEQs and Neural ODEs while avoiding the instability of the training process, that is often observed with DEQs. Moreover, we developed a method to speed up HomoODE and the ODE-solving operation by almost three times by using a shared learnable initial point. Overall, the experimental results on several classical image classification datasets demonstrate the efficiency of HomoODE in terms of both accuracy and memory consumption.
Although this paper offers a brand new perspective on implicit models, we also want to highlight a limitation of this idea. Actually, we do not present an explicit form of the equilibrium transformation function, which is implicitly determined by a modified neural ODE. Besides, while HomoODE has a powerful representation ability, the equilibrium-point-solving procedure of it is implicit, which
Method	Model size	Inference Time	Accuracy
DEQ [6]	170K	5.8×	82.2 ± 0.3%
monDEQ [46]	172K	1.6×	74.0 ± 0.1%
Neural ODE [10]	172K	3.2×	55.3 ± 0.3%
Aug. Neural ODE [12]	172K	1.7×	58.9 ± 2.8%
HomoODE	132K	1.2×	85.8 ± 0.1%
HomoODE†	132K	1.0×	83.2 ± 0.4%
HomoODE*	132K	1.4×	90.1 ± 0.2%
HomoODE*†	132K	1.0×	88.4 ± 0.1%
Table 1: Performance of HomoODE compared to previous implicit models on CIFAR-10.  $\star$  with data augmentation;  $^\dagger$  with adjoint method. The inference time is expressed as a multiple of the inference time of HomoODE with the adjoint method and the inference batch size is 400. Each result is obtained with 5 random runs.
Dataset	Method	Model size	Accuracy
SVHN	DEQ [6]	170K	93.6 ± 0.5%
monDEQ [46]	170K	92.4 ± 0.1%
Neural ODE [10]	172K	81.0 ± 0.6%
Aug. Neural ODE [12]	172K	83.5 ± 0.5%
HomoODE	132K	95.9 ± 0.1%
MNIST	DEQ [6]	80K	99.5 ± 0.1%
monDEQ [46]	84K	99.1 ± 0.1%
Neural ODE [10]	84K	96.4 ± 0.5%
Aug. Neural ODE [12]	84K	98.2 ± 0.1%
HomoODE	34K	99.6 ± 0.1%
Table 2: Performance of HomoODE compared to previous implicit models on SVHN and MNIST. Each result is obtained with 5 random runs.
Figure 5: Learning curve of HomoODE with different settings on CIFAR-10 across 5 runs without data augmentation. The x-axis denotes the epoch during training. The y-axis (up) denotes the accuracy of models on the test dataset and the y-axis (down) denotes the NFE of them in the forward pass.
weakens its interpretability. Hence, exploring a more interpretable approach for the forward pass and backpropagation of HomoODE is under consideration in our future work.
[1] Ralph Abraham and Joel Robbin. Transversal mappings and flows. WA Benjamin New York, 1967.
[2] Milton Abramowitz, Irene A Stegun, and Robert H Romer. Handbook of mathematical functions with formulas, graphs, and mathematical tables, 1988.
[3] Atish Agarwala and Samuel S Schoenholz. Deep equilibrium networks are sensitive to initialization statistics. In International Conference on Machine Learning, pages 136-160. PMLR, 2022.
[4] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, pages 136-145. PMLR, 2017.
[5] Shaojie Bai, Zhengyang Geng, Yash Savani, and J Zico Kolter. Deep equilibrium optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 620-630, 2022.
[49] Weiming Zhi, Tin Lai, Lionel Ott, Edwin V Bonilla, and Fabio Ramos. Learning efficient and robust ordinary differential equations via invertible neural networks. In International Conference on Machine Learning, pages 27060-27074. PMLR, 2022.
[50] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. In International Conference on Learning Representations.
[51] Juntang Zhuang, Nicha Dvornek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ode. In International Conference on Machine Learning, pages 11639-11649. PMLR, 2020.
A Background of Homotopy Continuation
Homotopy mapping  $H(z, \lambda) = \lambda r(z) + (1 - \lambda)g(z)$  provides a continuous transformation by gradually deforming  $g(z)$  into  $r(z)$  while  $\lambda$  increases from 0 to 1 in small increments. The solution to  $r(z)$  can be found by following the zero path of the homotopy mapping  $H(z, \lambda) = 0$ . Usually, one can choose an artificial function  $g(z)$  with an easy solution. Figure 6 shows the transformation of homotopy mapping with  $\lambda$  increasing from 0 to 1, the homotopy function goes from an artificial, "easy" problem to the nonlinear problem in which we are interested.
Figure 6: Transformation of homotopy mapping along with  $\lambda$ .
A.1 Global Convergence of Homotopy Continuation
Here, we briefly recall the theoretical foundation of homotopy methods to show its global convergence with probability one.
Definition 2 Let  $E^n$  denote  $n$ -dimensional real Euclidean space, let  $U \subset E^n$  and  $V \subset E^m$  be open sets, and let  $H: U \times V \times [0,1) \to E^n$  be a  $\mathcal{C}^2$  mapping.  $H$  is said to be transversal to zero if the Jacobian matrix  $\nabla H$  has full rank on  $H^{-1}(0)$ .
Theorem 1. (Parametrized Sard's Theorem [1]) If  $H(z_0, z, \lambda)$  is transversal to zero, then for almost all  $z_0 \in U$  the mapping
$$
H _ {z _ {0}} (z, \lambda) = H \left(z _ {0}, z, \lambda\right), \tag {20}
$$
is also transversal to zero; i.e., with probability one the Jacobian matrix  $\nabla_{H_{z_0}}(\lambda, z)$  has full rank on  $H_{z_0}^{-1}(0)$ .
The method for constructing a homotopy algorithm to solve the nonlinear system  $r(z) = 0$  with global convergence is as follows: 1)  $H(z_0, z, \lambda)$  is transversal to zero; 2)  $H_{z_0}(z, 0) = H(z_0, z, 0)$  is trivial to solve and has a unique solution  $z_0$ ; 3)  $H_{z_0}(z, 1) = r(z)$ ; 4)  $H_{z_0}^{-1}(0)$  is bounded.
Then for almost all  $z_0 \in U$  there exist a zero path  $s$  of  $H_{z_0}$ , along with the Jacobian matrix  $\nabla H_{z_0}$  has rank  $n$ . The zero path starts from  $(0, z_0)$  and reaching  $z^{\star}$  at  $\lambda = 1$ . This zero path  $s$  does not intersect itself, and it is disjoint from any other zero paths of  $H_{z_0}$ . Furthermore, if  $\nabla r(z)$  is nonsingular, then the zero path  $s$  has a finite arc length.
A.2 Fixed Point Homotopy Continuation
One commonly used homotopy function to find solutions of  $r(z) = 0$  is the Fixed Point Homotopy [11] given by:
$$
H (z, \lambda) = \lambda r (z) + (1 - \lambda) \left(z - z _ {0}\right), \tag {21}
$$
where  $z_0 \in \mathbb{R}^n$  and  $\lambda$  in unit interval  $[0,1]$ . At  $\lambda = 0$ , the starting system is  $H(z,0) = z - z_0 = 0$  for which the only solution is  $z = z_0$ . At  $\lambda = 1$ , the system  $H(z,1) = r(z) = 0$  is the system of equations of interest.
For  $U \subset \mathbb{R}^n$ , we use int  $U$  to denote the interior of  $U$ . And we say  $H$  is boundary-free at  $\lambda_0 \in [0,1]$  if  $z \notin \partial U$  for any  $z \in H|_{\lambda = \lambda_0}^{-1}(\{0\})$ . Generally, we say  $H$  is boundary-free for  $\lambda$  in a subset  $S \subset [0,1]$  if  $H$  is boundary-free for all  $\lambda \in S$ . The following theorem provides the fixed point of  $f(z)$  under the existence of Fixed Point Homotopy.
Theorem 2. (Fixed Point Theorem [11]) Given smooth function  $f: U \to \mathbb{R}^n$ , let  $U \in \mathbb{R}^n$  be compact and  $\operatorname{int} U \neq \varnothing$ . For some  $z_0 \in \operatorname{int} U$ , if  $H: U \times [0,1] \to \mathbb{R}^n$  is boundary-free for  $0 < \lambda < 1$ , where
$$
H (z, \lambda) = \lambda (z - f (z)) + (1 - \lambda) (z - z _ {0}), \tag {22}
$$
then  $f$  has a fixed point, i.e., there exists an  $z^{\star} \in U$  such that  $f(z^{\star}) = z^{\star}$ .
A.3 Newton Homotopy
Another commonly used homotopy function is the Newton homotopy [11], which is defined as follows:
$$
H (z, \lambda) = \lambda r (z) + (1 - \lambda) [ r (z) - r \left(z _ {0}\right) ] \tag {23}
$$
$$
= r (z) - (1 - \lambda) r (z _ {0}),
$$
where  $r:\mathbb{R}^n\to \mathbb{R}^n$  is the smooth system of interest, and  $z_{0}$  is a generically chosen point in  $\mathbb{R}^n$
Notably, there is a close connection between the Newton homotopy and the well-known Newton's method [2] for solving nonlinear equations. Given  $\nabla r(z)$  is nonsingular, we can apply the differentiation on the zero path of the homotopy mapping, i.e.  $H(z,\lambda) = 0$ , yielding the initial value problem:
$$
\frac {d z}{d \lambda} = - (\nabla r (z (\lambda))) ^ {- 1} r \left(z _ {0}\right), \tag {24}
$$
$$
z (0) = z _ {0}.
$$
Applying Euler's method at  $\lambda = 0$  with step size 1 to the above ODE (24) from the initial point  $z = z(0)$ , the approximation of next iteration  $z(1)$  becomes:
$$
z (1) = z (0) - \left(\nabla r (z (0))\right) ^ {- 1} r (z (0)). \tag {25}
$$
Apparently, (25) is a single iteration of Newton's method. Hence, Newton's iteration can be considered as the application of Euler's method with step size 1 on the solution curve given by the Newton homotopy. However, in contrast to Newton's method, which is generally a local method, the Newton homotopy exhibits certain global convergence properties.
$\lambda$
$\lambda$
Figure 7: Iterative curve of homotopy continuation under different  $v$ .
B The impact of different values of velocity  $v$  for the homotopy continuation
In Section 4 we introduce  $v \coloneqq \frac{ds}{dt}$  as the velocity of the point  $(z, \lambda)$  traveling along the zero path, and modify the normalization condition by introducing  $v$  into (3). Here we demonstrate that the convergence of the homotopy method is not affected by such modification by numerical experiments.
We choose a nontrivial nonlinear equation and solve it via homotopy continuation under different velocities  $v$ . Figure 7 shows the zero paths of homotopy continuation under different velocities  $v$ . Apparently, the zero paths are all on the same trajectory and heading for the same solution. Hence, the convergence of the homotopy method doesn't depend on the change in the value of velocity. With the modified normalization condition, the underlying dynamics become:
$$
\frac {d z}{d t} = \left(I - \lambda (z) \nabla_ {z} f (z; \theta)\right) ^ {- 1} f (z; \theta) \sqrt {v ^ {2} - \left\| \frac {d z}{d t} \right\| ^ {2}}. \tag {26}
$$
Following Neural ODEs, we can employ neural networks to approximate the differential equation above. The existence of a corresponding equilibrium-point-finding problem for Neural ODE is also guaranteed by the modified normalization condition. Specifically, one can refer to (11) (12). Hence, Neural ODEs can be regarded as the procedure of solving an equilibrium-point-finding problem with homotopy continuation.
C Bridging DEQ and Neural ODE from another type of Homotopy method
We have established the connection between DEQs and Neural ODEs via Fixed Point Homotopy in Section 4. Here, we also show that we can show a similar connection using another type of Homotopy method, i.e., Newton Homotopy.
By defining  $r(z) = z - f(z;\theta)$ , and setting  $z_0 = \mathbf{0}$  for simplification, from (23) we have:
$$
H (z, \lambda) = z - f (z; \theta) + (1 - \lambda) f (0; \theta), \tag {27}
$$
Different from Fixed Point Homotopy, we can associate  $z$  directly with  $\lambda$  without introducing  $v$ . From (24) we have:
$$
\frac {d z}{d \lambda} = (I - \nabla_ {z} f (z (\lambda); \theta)) ^ {- 1} f (z (\lambda); \theta), \quad z (0) = \mathbf {0}. \tag {28}
$$
Here we can view  $\lambda$  as the time information for the differential equation, and  $z(1)$  is the solution  $z^{\star}$  of the corresponding equilibrium-point-finding problem when  $\lambda = 1$ . We can also employ neural networks to approximate the differential equation as Neural ODEs do. Therefore, Neural ODEs can also be regarded as the procedure of solving an equilibrium-point-finding problem via Newton Homotopy.
D Stability of HomoODE
This section demonstrates the stability of HomoODE based on Picard-Lindef Theorem [29].
Theorem 3. (Picard-Lindeöf Theorem [29]) Let  $I: [a, b]$  be an interval, let  $f: I \times \mathbb{R}^n \to \mathbb{R}^n$  be a function, and let
$$
z ^ {\prime} (t) = f (t, z (t)), \tag {29}
$$
be the associated ordinary differential equation. If  $f$  is Lipschitz continuous in the second argument  $z$ , then this ODE possesses a unique solution on  $[a, a + \epsilon]$  for each possible initial value  $z(0) = z_0 \in \mathbb{R}^n$ , where  $\epsilon < \frac{1}{L}$ ,  $L$  is the Lipschitz constant of the second argument of  $f$ .
HomoODE solves the equilibrium-point-finding problem implicitly using a modified Neural ODE via homotopy continuation. Hence it also has the stability of Neural ODE, which can be explained by the Picard-Lindef Theorem. Assuming the underlying dynamic of HomoODE is Lipschitz continuous in  $z$ , then both existence and uniqueness can be guaranteed by the Picard-Lindef Theorem.
E Acceleration for HomoODE
In Section 6, it is mentioned that the update on the shared initial point  $z_0$  is equivalent to maintaining a dynamic geometrical center of the equilibrium points of all the samples. We will show the correctness of this proposition and illustrate the relationship between the learning rate and the step of the dynamic update.
Proposition 1. The update on the shared initial point  $z_0$  is equivalent to maintaining a dynamic geometrical center of the equilibrium points of all the samples.
Proof. Recall that the loss function of  $z_0$  is defined as  $\mathcal{L}(z_0) \coloneqq \mathbb{E}_{x \sim \mathcal{D}}\left[(z^{\star}(x) - z_0)^2\right]$ . According to the definition of the variance, we have
$$
\operatorname {V a r} \left(z ^ {\star} (x) - z _ {0}\right) = \mathbb {E} \left[ \left(z ^ {\star} (x) - z _ {0}\right) ^ {2} \right] - \mathbb {E} \left[ z ^ {\star} (x) - z _ {0} \right] ^ {2}. \tag {30}
$$