SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where CE denotes the cross entropy loss and  $\sigma$  denotes sigmoid function. We are given 5000 training data with corruption rate 0.5, 5000 clean validation data and 10000 clean testing data. The existing
methods for nonconvex-strongly-convex BLO [21, 32] often adopts a single fully-connected layer with a regularization as the LL problem, but the regularization and the simple network structure always degenerate the model performance. As our algorithm is able to tackle the nonconvex LL problem with multiple solutions, we can consider more complex neural network structure such as two layer MLP without any regularization. Since training samples is fewer than the model parameters, the LL training problem is an overparameterized neural network so that satisfies the PL condition.
Algorithm implementation details. Instead of calculating Hessian matrix explicitly which is time-consuming, we compute the Hessian-vector product via efficient method [53]. Specifically, we can auto differentiate  $\nabla_y g(x,y)^\top w$  with respect to  $x$  and  $y$  to obtain  $\nabla_{xy}^2 g(x,y)w$  and  $\nabla_{yy}^2 g(x,y)w$ . Also, let  $v = \nabla_y f(x,y) + \nabla_{yy}^2 g(x,y)w$  which detaches the dependency of  $v$  over  $x$  and  $y$ , the Hessian-vector product  $\nabla_{yy}^2 g(x,y)v$  can be calculated by auto differentiation of  $\nabla_y g(x,y)^\top v$ .
Parameter choices. The dimension of the hidden layer of MLP model is set as 50. We select the stepsize from  $\alpha \in \{1,10,50,100,200,500\}$ ,  $\gamma \in \{0.1,0.3,0.5,0.8\}$  and  $\beta \in \{0.001,0.005,0.01,0.05,0.1\}$ , while the number of loops is chosen from  $T \in \{5,10,20,30,50\}$  and  $N \in \{5,10,30,50,80\}$ .
(a) MNIST, linear model
(b) MNIST, 2-layer MLP model
(c) FashionMNIST, 2-layer MLP model
Figure 7: Test loss v.s. iteration.
Method	MNIST	FashionMNIST
Linear model	MLP model	MLP model
RHG [21]	87.83 ± 0.17	87.95 ± 0.21	81.87 ± 0.31
BOME	88.76 ± 0.15	90.01 ± 0.18	83.81 ± 0.26
IAPTT-GM	90.13 ± 0.13	90.86 ± 0.17	83.50 ± 0.28
V-PBGD	90.24 ± 0.13	92.16 ± 0.15	84.18 ± 0.22
GALET	90.20 ± 0.14	91.48 ± 0.16	84.03 ± 0.25
Table 3: Comparison of the test accuracy of different methods on two dataset with different network structure. The results are averaged over 10 seeds and  $\pm$  is followed by the variance.
The test loss v.s. iteration of different methods is shown in Figure 7, it can be seen that GALET converges fast in all of regimes. Moreover, we report the test accuracy of different methods in Table 3. It shows that GALET achieves comparable test accuracy with other nonconvex-nonconvex bilevel methods, which improves that of nonconvex-strongly-convex bilevel method RHG.