SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Define  $\bar{z}^{\star} \coloneqq \mathbb{E}(z^{\star}(x))$ , we obtain
$$
\begin{array}{l} \mathbb {E} \left[ (z ^ {\star} (x) - z _ {0}) ^ {2} \right] = \operatorname {V a r} \left(z ^ {\star} (x) - z _ {0}\right) + \mathbb {E} \left[ z ^ {\star} (x) - z _ {0} \right] ^ {2} \\ = \operatorname {V a r} \left(z ^ {\star} (x) - z _ {0}\right) + \left(\bar {z} ^ {\star} - z _ {0}\right) ^ {2} \tag {31} \\ = \operatorname {V a r} \left(z ^ {\star} (x)\right) + \left(\bar {z} ^ {\star} - z _ {0}\right) ^ {2} \\ \end{array}
$$
Here  $\operatorname{Var}(z^{\star}(x))$  is irrelevant to  $z_0$ . Hence,  $\min_{z_0} \mathbb{E}\left[(z^{\star}(x) - z_0)^2\right]$  is equivalent to minimizing  $\min_{z_0} (\bar{z}^{\star} - z_0)^2$ , which means the update on  $z_0$  is equivalent to maintaining a geometrical center of the equilibrium points of all the samples.
According to Proposition 1, we can also write the update as the form of  $z_0 = \alpha \bar{z}^\star + (1 - \alpha)z_0$ , which is more straightforward. Therefore, we also want to further explore the relationship between the learning rate  $\eta_{\mathrm{init}}$  and update step  $\alpha$ . Consider the gradient descent on  $z_0$
$$
\begin{array}{l} z _ {0} = z _ {0} - \eta \nabla_ {z _ {0}} \mathbb {E} _ {x \sim \mathcal {D}} \left[ (z ^ {\star} (x) - z _ {0}) ^ {2} \right] \\ = z _ {0} + 2 \eta (\bar {z} ^ {\star} (x) - z _ {0}) \tag {32} \\ = 2 \eta \bar {z} ^ {\star} (x) + (1 - 2 \eta) z _ {0}. \\ \end{array}
$$
Since  $z_0$  is the broadcast tensor from initial information  $\tilde{z}_0$ , the actual gradient descent has the form of
$$
\tilde {z} _ {0} = \frac {2}{h w} \eta_ {\text {i n i t}} \bar {z} ^ {\star} (x) + \left(1 - \frac {2}{h w} \eta_ {\text {i n i t}}\right) \tilde {z} _ {0}. \tag {33}
$$
where  $h, w$  are the height and width of the feature map respectively. Finally, we obtain the relationship between  $\eta_{init}$  and  $\alpha$  is  $\eta_{init} = \frac{hw}{2}\alpha$ . This means we can set a large learning rate for  $\eta_{init}$  even greater than 1, especially when the feature map is large.
F Adjoint Method for HomoODE
Algorithm 1 Adjoint Method for HomoODE
Input: initial point  $z(t_0)$ , condition  $x$ , parameter  $\theta$ , start time  $t_0$ , stop time  $t_1$
$$
s _ {0} = [ z (t _ {1}), \frac {\partial L}{\partial z (t _ {1})}, 0 _ {x}, 0 _ {\theta} ]
$$
$\triangleright$  Define initial augmented state
def aug_dynamics([z(t), a(t), \(\cdot, \cdot\)], t, \(\theta)):
$\triangleright$  Define dynamics on augmented state
return  $[F(z(t),t;x,\theta); - a(t)^{\top}\frac{\partial f}{\partial z}, - a(t)^{\top}\frac{\partial f}{\partial x}, - a(t)^{\top}\frac{\partial f}{\partial\theta}]$  Compute vector-Jacobian products
$\left[z\left(t_{0}\right), \frac{\partial L}{\partial z\left(t_{0}\right)}, \frac{\partial L}{\partial x}, \frac{\partial L}{\partial \theta}\right] = \mathrm{ODESolve}\left(\mathrm{s}_{0}; \mathrm{aug\_dynamics}; \mathrm{t}_{1}; \mathrm{t}_{0}; \theta\right) \quad \triangleright$  Solve reverse-time ODE
return  $\frac{\partial L}{\partial x}, \frac{\partial L}{\partial \theta}$
$\triangleright$  Return gradients with respect to  $x$  and  $\theta$
The adjoint method [10, 42] is an efficient backpropagation method that can save the memory footprint in Neural ODE during training. However, there is a minor difference when we apply the adjoint method in the training of HomoODE. Unlike Neural ODE which computes the gradient with respect to  $z(t_0)$ , our HomoODE calculates the gradient with respect to the condition  $x$  instead. Accordingly, we modify the adjoint method in Neural ODE and present the computation procedure in Algorithm 1.
We conducted additional experiments in CIFAR-100 [34] and Tiny ImageNet [37] to validate the potential of applying HomoODE to difficult image classification tasks with larger model sizes. Specifically, we extend the channel numbers of the convolutional layers to 128 in HomoODE and increase the model size of the compared implicit models correspondingly for fairness. The experiments in CIFAR-100 and Tiny ImageNet are all implemented with data augmentation. The concrete operations in data augmentation involve zero-padding the  $32 \times 32$  images to  $40 \times 40$  and then performing random horizontal flips. As shown in Table 4, the performance of HomoODE is much better than other implicit models in terms of both memory consumption and test accuracy.
Method	Model size	Accuracy
DEQ [6]	770K	64.5 ± 0.7%
monDEQ [46]	1M	59.8 ± 0.3%
Neural ODE [10]	874K	31.7 ± 0.6%
Aug. Neural ODE [12]	857K	36.2 ± 0.9%
HomoODE†	565K	69.30 ± 0.1%
HomoODE	565K	71.57 ± 0.2%
Table 3: Additional experiments on CIFAR-100. HomoODE† and HomoODE represent the training with and without the adjoint method, respectively. The experiments are all implemented with data augmentation.
Method	Model size	Accuracy
DEQ [6]	810K	47.6 ± 0.4%
monDEQ [46]	783K	24.83 ± 0.1%
HomoODE†	617K	54.14 ± 0.2%
HomoODE	617K	56.1 ± 0.1%
Table 4: Additional experiments on Tiny-ImageNet. HomoODE† and HomoODE represent the training with and without the adjoint method, respectively. The experiments are all implemented with data augmentation.
Besides, we also performed the ablation study on the learning rate for the shared initial point as shown in Figure 8. Notably, the ablation study is implemented with the adjoint method. Given the practicality of the adjoint method in handling the backpropagation of large deep models, we are inclined to explore a wider range of characters when utilizing it. It can be observed that HomoODE is robust to the learning rate for the shared initial point and there exists a slight improvement when  $\mathrm{lr} = 0.1$ . We believe the reason is the same as the slight improvement in contrast experiments of HomoODE with/without acceleration. That is, the larger learning rate for the shared initial point makes the initial point  $z_0$  closer to the current geometrical center of the equilibrium points of all samples, which further leads to the shorter zero path that the adjoint method goes through. Finally, it reduces the error of backpropagated gradient.
H Hyper-parameter Settings
Our experiments are implemented on a GPU of NVIDIA GeForce RTX 3090 with 24GB. The hyper-parameters applied in HomoODE are shown in Table 5. Experiments related to other implicit
Figure 8: Ablation study on the learning rate for the shared initial point.
models are based on deq  $^{2}$ , monotone_op_net  $^{3}$  and augmented-neural-odes  $^{4}$ . Notably, the dropout layer in our experiments follows the variational dropout operation in [7] because the traditional dropout operation hurts the stability of convergence to the equilibrium.
Parameter	MNIST	SVHN	CIFAR-10	CIFAR-100
Batch Size	64	64	64	64
Optimizer	Adam	Adam	Adam	Adam
Learning Rate	0.001	0.001	0.001	0.001
Frequency of Initial Point Update	20	20	20	5
Optimizer for Initial Point	SGD	SGD	SGD	SGD
Learning Rate for Initial Point	0.02	0.02	0.02	0.01
Variational Dropout Rate	0.1	0.1	0.1	0.15
Number of Channels	32	64	64	128
Absolute Tolerance for ODE Solver	1E-3	1E-3	1E-3	1E-3
Relative Tolerance for ODE Solver	1E-3	1E-3	1E-3	1E-3
Table 5: Hyper-parameters used in HomoODE under different image classification tasks.
I Details on Tested Equation in Section 6
With respect to the details on the tested nonlinear equation mentioned in Section 6, we randomly choose an equation with high nonlinearity that is
$$
f (x) = 2 x + \exp (- 0. 1 x) + 5 \sin (4 x) - 1 6 = 0,
$$
and one solution of it is 6.4217. Our experiment shown in Figure 2 is based on the initial points around this solution. Actually, we found that even if we modify the parameters in this function arbitrarily, a similar result can be obtained as well.
J More Discussions on Learnable Initial Point
It is worth noting that we also carried out some trials with an input-based initial point predictor [8] to accelerate HomoODE. However, the performance of the initial point predictor in HomoODE is not desirable in terms of both inference time and test accuracy. Based on the comprehensive analysis we provided in Section B, C, this is probably because too frequent or too large updates on the initial point will destroy the stable link between ODE function  $F(z(t), t; x, \theta)$  and the equilibrium transformation function  $f(z; x, \theta)$ . In particular, the additionally-introduced variable  $v$  will change sharply when a large change is applied to the initial point. In this case, the function  $\lambda(t)$  will change dramatically, eventually ruining the trained equilibrium transformation function  $f(z; x, \theta)$ . This phenomenon can also be illustrated from the perspective of Newton homotopy. Note that the ODE function is actually determined by the initial point. Therefore, once the initial point is changed, the ODE function will change and then the trained ODE function will be ruined.
$^{2}$ https://github.com/locuslab/deq
<sup>3</sup>https://github.com/locuslab/monotone_op_net
<sup>4</sup>https://github.com/EmilienDupont/augmented-neural-odes