SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $(X_{1},Y_{1})$  and  $(X_{2},Y_{2})$  are validation and training data sets with sizes  $D_{X_1}$  and  $D_{X_2}$ , respectively;  $\tilde{f} (\cdot ;w)$  are the embedding layers of the model parameterized by  $w$ ; and,  $c$  is the classification layer. Moreover, we use conjugate gradient methods (CG) [17, 18] to solve the linear system when computing the hypergradient for hyper-representation learning experiments.
Reset While Algorithm 2 (reset) can be applied to both upper and lower-level problems, we focus our discussions here on the upper-level learning rate  $(\alpha_{k})$ . This is because we empirically find it to be more critical for the convergence performance (see Figure 6a). As shown in Algorithm 2, reset has 3 options. Options 1, 2, and 3 search starting from  $\alpha_{b,0}$ ,  $\alpha_{k-1}$ , and  $\eta \alpha_{k-1}$ , at iteration  $k$  respectively. Option 1 has the highest search cost as it always starts from the same initial upper bound  $(\alpha_{b,0})$ . Option 2 ensures the monotonicity of the learning rate due to  $\alpha_{k} \leq \alpha_{b,k} = \alpha_{k-1}$ . Option 3 chooses the search starting point at iteration  $k$  ( $\alpha_{b,k}$ ) by multiplying the previous learning rate  $(\alpha_{k-1})$  by a factor  $\eta \geq 1$ . As in the single-level convex case where monotonicity in the step size can potentially lead to slow convergence (see Figure 3), we again observe that monotonicity in the upper learning rate (i.e. option 2) leads to poorer performance when compared against options 1 or 3 as shown in Figure 10. Finally, we compare the performance of different  $\eta s$  in option 3 (note that  $\eta = 1$  in option 3 is equivalent to option 2). We observe in Figure 11 that different  $\eta s$  perform equally well. This shows the robustness of our algorithm to the choice of  $\eta$ . As mentioned previously, the choice of option 3 over option 1 are due to 2 reasons: (a) reduced search cost; (b) provides an overall non-increasing and non-monotonic trend of upper bound  $\alpha_{b,k}$ . We discuss search cost of different  $\eta s$  in option 3 below.
Search Cost Based on the results in Figure 7b, we observe the use of option 3 in reset can significantly reduce the search cost for both BiSLS-Adam and BiSLS-SGD. Figure 10 further suggests that options 1 and 3 have nearly the same performance. Therefore, option 3 is an efficient algorithm that maintains good performance while reducing computation cost. Option 2 has the lowest search cost ( $\sim$ 4 rounds per iteration). However, its performance is not as good as option 1 or 3 as
(a) Different Opt (BiSLS-SGD)
(b) Different Opt (BiSLS-Adam)
Figure 10: Validation loss against iterations with search options 1, 2, and 3 for the upper-level learning rate. The results for BiSLS-SGD and BiSLS-Adam are in (a) and (b), respectively. For the lower-level search, we fix it option 1 with  $\beta_{b,0} = 100$ . Results are based on hyper-representation learning.
(a) Different  $\eta s$  BiSLS-SGD
Figure 11: Validation loss against iterations for different  $\eta$ s based on reset option 3. Results for BiSLS-SGD are given in (a) and for BiSLS-Adam are given in (b). Note that  $\eta = 1$  in reset option 3 is equivalent to reset option 2. For the lower-level search, we fix it option 1 with  $\beta_{b,0} = 100$ . Results are based on hyper-representation learning.
(b) Different  $\eta_{\mathrm{S}}$  (BiSLS-Adam)
observed in Figure 10. Moreover, the average lower-level search cost is only 1 round per iteration when option 1 is used (see Figure 12).
Sensitivity on  $\delta$  As mentioned in Sec 2, due to the stochastic error in hypergradient computation, further complicated by the approximation error of  $y^{*}(x)$  (see (14)), a learning rate is not guaranteed to be found in the bi-level case. Specifically, this is in contrast to the single-level convex problems. To avoid this, we introduce in (14) a  $\delta$  slack to give some tolerance to such errors. Here, we give a thorough investigation of the effects of  $\delta$  on performance. We vary its magnitude across 6 orders for both reset options 1 and 3 (see Algorithm 2 and discussions on reset above). We observe that despite a large difference on the magnitudes of  $\delta$ , they all share very similar performance for both BiSLS-SGD and BiSLS-Adam: see Figures 13 and 14. We summarize the key fins in this section as follows: ① The option 3 in reset has good empirical performance (outperforms option 2) and is an effective way to reduce search cost (Figure 10, ??); ② BiSLS is highly robust to different choices of  $\eta$  in option 3 and  $\delta$  in (14) (Figure 11, 13, 14).
Figure 12: Lower-level search cost measured in the same way as upper-level against different lower-level search starting points  $(\beta_{b,0})$ . The lower-level search is done with option 1 (see above for discussions about these options).
(a) Different  $\delta \mathrm{s} / \mathrm{Opt - 1}$  (BiSLS-SGD)
(b) Different  $\delta s$  /Opt-1 (BiSLS-Adam)
Figure 13: Validation loss against iterations for different  $\delta$ s based on reset option 1. Results for BiSLS-SGD (a) and for BiSLS-Adam (b). For the lower-level search, we fix it option 1 with  $\beta_{b,0} = 100$ . Results are based on hyper-representation learning.
B.3 Data distillation objective and additional results
We let  $\mathcal{L}_S(w)$  denote the loss evaluated on dataset  $S$  with model weights  $w$ . The objective of data distillation can be expressed as a BO problem as follows:
$$
D ^ {*} = \underset {D} {\operatorname {a r g m i n}} \mathcal {L} _ {\tilde {V}} (w ^ {*} (D)) \text {s . t .} w ^ {*} (D) = \underset {w} {\operatorname {a r g m i n}} \mathcal {L} _ {D} (w),
$$
where  $\tilde{V}$  is of the same size as  $D$  and subsampled from the entire (original) dataset  $V$ . The solution  $D^{*}$  is the distilled data, e.g. 9 MNIST digits each corresponding to a different label. In figure 15a, we show the performance of BiSPS for different values of  $\alpha_{b,0}$  in comparison with BiSLS-SGD, and observe that BiSLS-SGD has better performance. In 15b, we show the results when we increase the number of lower-level iterations (T) from 20 to 50. As observed for  $T = 20$  (in Figure 8), BiSLS-SGD here also outperforms a fine-tuned Adam or SGD.
B.4 1-sample or 2-samples versions of algorithms for convex and bi-level optimization
We provide additional results to compare the performance of 1-sample and 2-samples (one for computing the gradient and the other for computing the step size) versions of our algorithms for
(a) Different  $\delta \mathrm{s} / \mathrm{Opt - 3}$  (BiSLS-SGD)
(b) Different  $\delta s / \mathrm{Opt - 3}$  (BiSLS-Adam)
Figure 14: Validation loss against iterations for different  $\delta$ s based on reset option 3 ( $\eta = 10$ ). Results for BiSLS-SGD (a) and for BiSLS-Adam (b). For the lower-level search, we fix it option 1 with  $\beta_{b,0} = 100$ . Results are based on hyper-representation learning.
(a) Different  $\alpha_{b,0}\mathrm{s}$  (BiSPS)
Figure 15: Validation loss against iterations. (a) Comparison between BiSPS with different  $\alpha_{b,0}\mathrm{s}$  and BiSLS-SGD. (b) Comparison between BiSLS-SGD to fine-tuned Adam/SGD ( $\beta_{k}$  fixed at  $10^{-4}$ ). Inverse Hessian in (2) treated as the Identity [30] when computing the hypergradient. Recall that T is the total number of lower-level iterations and we have shown the results for  $T = 20$  in Figure 8b.
(b)  $\mathrm{T} = {50}$  ,Identity
SPSB and BiSPS used for single-level and bi-level optimization, respectively. In the single-level case (Figure 16), we observe that 2-samples SPSB performs just as well as 1-sample SPSB. Interestingly, we observe that their step sizes also follow a similar pattern. That is: an initial increase followed by a regime where  $\gamma_{k} = \frac{f_{i_{k}}(x^{k}) - l_{i_{k}}^{*}}{c\|\nabla f_{i_{k}}(x^{k})\|^{2}}$  is frequently used, and eventually changes to decaying-step SGD. This seems to also match with Theorem 2 where a transition point for SPSB ( $k_{0} = \max \{1,\lceil \gamma_{0} / \omega \rceil -1\}$ ,  $w = \frac{1}{2cL_{\max}},\gamma_0\geq \frac{1}{\mu}$ ) is predicted. At the same time, we also note that (perhaps, unsurprisingly) the 2-samples version seems to have a slightly more oscillatory behavior than the 1-sample version as shown in Figure 16. SLSB with either 1-sample or 2-samples also result in a similar performance and step size. Overall, despite the requirements of Theorems 1 and 2 for a 2-samples assumption, the empirical performance of 1-sample and 2-samples for either SPSB or SLSB appears to be very similar. Moving on to the bi-level case, recall that Theorems 3 and 4 require the 2-samples assumption (i.e.,  $\alpha_{k}$  independent of  $h_f^k$ ) for the upper-level learning rate. We empirically verify this assumption with both hyper-representation learning and data distillation experiments. For hyper-representation learning experiments in Figure 17, BiSPS with either 1-sample or 2-samples for different values of  $\alpha_{b,0}$  show similar performance. In fact, for  $\alpha_{b,0} = 0.1$  we even observe that the 2-samples variant outperforms the 1-sample BiSPS. For data distillation experiments in Figure 18, the performances of 1-sample and 2-samples BiSPS are similar to each other when
$\alpha_{b,0} = 10.0$  or  $\alpha_{b,0} = 50.0$ . In general, the performance difference between 1-sample and 2-samples in the single-level or bi-level settings is small.
(a) Train loss
(b) Step size
Figure 16: Binary linear classification on w8a dataset using logistic loss [3]. Train loss (left) and step size (right) against iterations. We choose  $\gamma_{b,0} = 1000$  for all algorithms. The upper bound for either SPSB or SLSB decays as  $\gamma_{b,k} = \frac{\gamma_{b,0}}{\sqrt{k + 1}}$ . For decaying-step SGD, the learning rate schedule is  $\frac{\gamma_{b,0}}{\sqrt{k + 1}}$ .
Figure 17: Comparison between BiSPS (2-samples), BiSPS (1-sample) and decaying-step SGD. Experiments are based on hyper-representation learning. For either version of BiSPS, the lower-level learning rate  $(\beta_{k})$  is fixed at 10. The hypergradient is computed using conjugate gradient [17].
Figure 18: Comparison between BiSPS (2-samples), BiSPS (1-sample) and decaying-step SGD. Experiments are based on data distillation. For either version of BiSPS, the lower-level learning rate  $(\beta_{k})$  is fixed at  $10^{-4}$ . The Inverse Hessian in (2) is treated as the Identity when computing the hypergradient [30].
Decay SGD - αb,0 = 10.0	BiSPS - αb,0 = 10.0	BiSPS (2 samples) - αb,0 = 10.0
Decay SGD - αb,0 = 50.0	BiSPS - αb,0 = 50.0	BiSPS (2 samples) - αb,0 = 50.0
Decay SGD - αb,0 = 100.0	BiSPS - αb,0 = 100.0	BiSPS (2 samples) - αb,0 = 100.0