SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ f \left(\mathbf {x} _ {t}\right) - f ^ {\star} \right] \leq \left(1 + \frac {2 L}{(1 - a) \mu_ {F}}\right) \tau^ {2}. \tag {B.59}
$$
Dividing both sides by  $T$  and applying Jensen's inequality concludes the proof.
B.6 Proof of Corollary 8
Proof. Algorithm 1 calls the stochastic gradient oracle in expectation  $\mathcal{O}(1 + p_t n)$  times at iteration  $t$ . Therefore, the total number of gradient evaluations is upper bounded by  $\mathcal{O}(\sum_{t=0}^{T-1} p_t n + T)$ . By our choice of  $p_t$ , it holds that  $\sum_{t=0}^{T-1} p_t \leq \frac{1}{a} \sum_{t=0}^{T-1} \frac{1}{t+2} \leq \frac{1}{a} (\log(T) + 1 - 1) = \frac{1}{a} \log(T)$ . Due to the sublinear convergence rate of Algorithm 1, we conclude that the total number of stochastic gradient calls is  $\mathcal{O}(\log(1/\varepsilon)n + 1/\varepsilon)$ .
C Pseudo-codes for AdaSPS and AdaSLS
In this section, we provide formal pseudo codes for AdaSPS (AdaSPS) and AdaSLS (AdaSLS).
To implement AdaSPS, a lower bound of optimal function value for each minibatch function is required. For machine learning problems where the individual loss functions are non-negative, we can use zero as an input. Apart from that, we need to provide a constant  $c_p$  to adjust the magnitude of the stepsize. Theoretically suggested  $c_p$  for robust convergence satisfies  $c_p^{\mathrm{scale}} = c_p\sqrt{f_{i_0}(\mathbf{x}_0) - \ell_{i_0}^\star} \geq \frac{1}{2}$ . Therefore, a common choice is to set  $c_p = \frac{1}{\sqrt{f_{i_0}(\mathbf{x}_0) - \ell_{i_0}^\star}}$ .
Algorithm 2 AdaSPS
Require:  $\mathbf{x}_0\in \mathbb{R}^d,T\in \mathbb{N}^+,c_p > 0$
1: set  $\eta_{-1} = +\infty$
2: set  $\varepsilon = 10^{-10}$
3: for  $t = 0$  to  $T - 1$  do
4: uniformly sample  $i_t \subseteq [n]$
5: provide a lower bound  $\ell_{i_t}^{\star}\leq f_{i_t}^{\star}$
6: set  $\eta_t = \min \left\{\frac{f_{i_t}(\mathbf{x}_t) - \ell_{i_t}^*}{c_p||\nabla f_{i_t}(\mathbf{x}_t)||^2}\frac{1}{\sqrt{\sum_{s=0}^t f_{i_s}(\mathbf{x}_s) - \ell_{i_s}^* + \varepsilon}}, \eta_{t-1}\right\}$
7:  $\mathbf{x}_{t + 1} = \Pi_{\mathcal{X}}(\mathbf{x}_{t - }\eta_{t}\nabla f_{i_{t}}(\mathbf{x}_{t}))$
return  $\bar{\mathbf{x}}_T = \frac{1}{T}\sum_{t = 0}^{T - 1}\mathbf{x}_t$
To implement AdaSLS (AdaSLS), a line-search sub-solver 4 and an input constant  $c_{l} > 0$  are required. Similar to (AdaSPS), we can set  $c_{l} = \frac{1}{\rho\sqrt{\gamma_{0}||\nabla f_{i_{0}}(\mathbf{x}_{0})||^{2}}}$  according to the theory.
Algorithm 3 AdaSLS
Require:  $\mathbf{x}_0\in \mathbb{R}^d,T\in \mathbb{N}^+$ $c_{l} > 0$
1: set  $\eta_{-1} = +\infty$
2: set  $\varepsilon = 10^{-10}$
3: for  $t = 0$  to  $T - 1$  do
4: uniformly sample  $i_t \subseteq [n]$
5: obtain  $\gamma_{t}$  via backtracking line-search (4)
6: set  $\eta_t = \min \left\{\frac{\gamma_t}{c_l\sqrt{\sum_{s=0}^t\gamma_s||\nabla f_{i_s}(\mathbf{x}_s)||^2 + \varepsilon}}, \eta_{t-1}\right\}$
7:  $\mathbf{x}_{t + 1} = \Pi_{\mathcal{X}}(\mathbf{x}_t - \eta_t\nabla f_{i_t}(\mathbf{x}_t))$
return  $\bar{\mathbf{x}}_T = \frac{1}{T}\sum_{t = 0}^{T - 1}\mathbf{x}_t$
D Line-search procedure
In this section, we introduce the classical Armijo line-search method [3, 41]. Given a function  $f_{i_t}(\mathbf{x})$ , the Armijo line-search returns a stepsize  $\gamma_t$  that satisfies the following condition:
$$
f _ {i _ {t}} \left(\mathbf {x} _ {t} - \gamma_ {t} \nabla f _ {i _ {t}} \left(\mathbf {x} _ {t}\right)\right) \leq f _ {i _ {t}} \left(\mathbf {x} _ {t}\right) - \rho \gamma_ {t} \left\| \nabla f _ {i _ {t}} \left(\mathbf {x} _ {t}\right) \right\| ^ {2}, \tag {D.1}
$$
where  $\rho \in (0,1)$  is an input hyper-parameter. If  $f_{i_t}(\mathbf{x})$  is a smooth function, then backtracking line-search 4 is a practical implementation way to ensure that D.1 is satisfied.
Algorithm 4 Backtracking line-search
Require:  $\beta \in [\frac{1}{2}, 1)$ ,  $\rho \in (0, 1)$ ,  $\gamma_{\max} > 0$  (We fix  $\beta = 0.8$  and  $\rho = 0.5$  for AdaSLS)
1:  $\gamma = \gamma_{\mathrm{max}}$
2: while  $f_{i_t}(\mathbf{x}_t - \gamma \nabla f_{i_t}(\mathbf{x}_t)) > f_{i_t}(\mathbf{x}_t) - \rho \gamma ||\nabla f_{i_t}(\mathbf{x}_t)||^2$  do
3:  $\gamma = \beta \gamma$
return  $\gamma_{t} = \gamma$
To implement Algorithm 4, one needs to provide the decreasing factor  $\beta$ , the maximum stepsize  $\gamma_{\mathrm{max}}$ , and the condition parameter  $\rho$ . Starting from  $\gamma_{\mathrm{max}}$ , Algorithm 4 decreases the stepsize iteratively by a constant factor  $\beta$  until the condition D.1 is satisfied. Note that checking the condition requires additional minibatch function value evaluations. Fortunately, note that the output  $\gamma$  cannot be smaller than  $\frac{1 - \rho}{L}$  (Lemma 16), and thus the number of extra function value evaluations required is at most  $\mathcal{O}\left(\max \{\log_{1 / \beta}^{L\gamma_{\mathrm{max}} / (1 - \rho)}, 1\}\right)$ . In practice, Vaswani et al. [52] suggests dynamic initialization of  $\gamma_{\mathrm{max}}$  to reduce the algorithm's running time, that is, setting  $\gamma_{\mathrm{max}_t} = \gamma_{t-1}\theta^{1/n}$  where a common choice for  $\theta$  is 2. This strategy initializes  $\gamma_{\mathrm{max}}$  by a slightly larger number than the last output and thus is usually more efficient than keeping  $\gamma_{\mathrm{max}}$  constant or always using  $\gamma_{\mathrm{max}_t} = \gamma_{t-1}$ . In all our experiments, we use the same  $\gamma_{\mathrm{max}}$  at each iteration for AdaSLS to show its theoretical properties.
Goldstein line-search is another line-search method that checks D.1 and an additional curvature condition [41]. We do not study this method in this work and we refer to [52] for more details.
E Counter examples of SPS and its variants for SVRG
We provide two simple counterexamples where SVRG with the SPS stepsize and its intuitive variants fail to converge. For simplicity, consider the update rule  $\mathbf{x}_{t + 1} = \mathbf{x}_t - \eta_t\nabla f(\mathbf{x}_t)$ , i.e.  $\mathbf{w}_t = \mathbf{x}_t$  for all  $t\geq 0$ . Consider the function  $f(\mathbf{x}) = \frac{f_1(\mathbf{x}) + f_2(\mathbf{x})}{2}$  where  $f_{1}(\mathbf{x}) = a_{1}(\mathbf{x} - 1)^{2}$  and  $f_{2}(\mathbf{x}) = a_{2}(\mathbf{x} + 1)^{2}$  with  $a_1,a_2 > 0$ .
Example 19. Individual curvature is not representative. Consider the standard stochastic Polyak stepsize:  $\eta_t = \frac{f_i(\mathbf{x}_t) - f_i^\star}{||\nabla f_i(\mathbf{x}_t)||^2}$  where  $i$  is randomly chosen from  $\{1,2\}$ . We now let  $a_1 = 1$  and  $a_2 < 1$ . Note that  $\nabla^2 f(\mathbf{x}) = a_1 + a_2 \in (1,2)$  while  $\mathbb{E}_i[\eta_t] = \frac{1}{8} + \frac{1}{8a_2} \to +\infty$  as  $a_2 \to 0$ , which leads to divergence. The reason behind this is that individual curvature does not match the global curvature.
Example 20. Mismatching quantity. Consider a variant of stochastic Polyak stepsize:  $\eta_t = \frac{f_i(\mathbf{x}_t) - f_i^\star}{||\nabla f_i(\mathbf{x}_t) - \nabla f_i(\mathbf{w}_t) + \nabla f(\mathbf{w}_t)||^2}$  where  $i$  is randomly chosen from  $\{1,2\}$ . Let  $a_1 = a_2 = 1$ . We note  $\mathbb{E}_i[\eta_t\nabla f(\mathbf{x}_t)] = \frac{x_t^2 + 1}{2x_t} \neq 0$  and thus no stationary point exists. Similar reasoning can exclude a number of other variants such as:  $\eta_t = \frac{f_i(\mathbf{x}_t) - f_i(\mathbf{w}_t) + f(\mathbf{w}_t) - f_i^\star}{||\nabla f_i(\mathbf{x}_t) - \nabla f_i(\mathbf{w}_t) + \nabla f(\mathbf{w}_t)||^2}$ . Indeed, the numerator is not the proper function value difference of a valid function with the gradient defined in the denominator.
F Experimental details and additional experiment results
In this section, we provide a detailed setup of the experiments presented in the main paper.
In practice, we can use a lower bound of  $F_{i_t}^*$  for running AdaSVRPS since convergence is still guaranteed thanks to the property of AdaSPS. By default, we use  $\ell_{i_t}^* + \min_{\mathbf{x}} \{ \mathbf{x}^T (\nabla f(\mathbf{w}_t) - \nabla f_{i_t}(\mathbf{w}_t)) + \frac{\mu_F}{2} ||\mathbf{x} - \mathbf{x}_t||^2 \}$  for all the experiments, where  $\ell_{i_t}^*$  is a lower bound for  $f_{i_t}^*$ .
F.1 Synthetic experiment
We consider the minimization of a quadratic of the form:  $f(\mathbf{x}) = \frac{1}{n}\sum_{i=1}^{n}f_{i}(\mathbf{x})$  where  $f_{i}(\mathbf{x}) = \frac{1}{2}(\mathbf{x} - \mathbf{b}_{i})^{T}A_{i}(\mathbf{x} - \mathbf{b}_{i}), \mathbf{b}_{i} \in \mathbb{R}^{d}$  and  $A_{i} \in \mathbb{R}^{d \times d}$  is a diagonal matrix. We use  $n = 50, d = 1000$ . We control the interpolation by either setting all  $\{b_{i}\}$  to be identical or different. Each component of  $\{b_{i}\}$  is generated according
to  $\mathcal{N}(0,10^2)$ . We control the complexity of the problems by choosing different matrices  $A_{i}$ . For the strongly-convex case, we first generate a matrix  $A^{N} = \mathrm{clip}(\left( \begin{array}{ccc}a_{11} & \ldots & a_{1d}\\ \ldots \\ a_{n1} & \ldots & a_{nd} \end{array} \right),1,10)$  where each  $a_{ij}\sim N(0,15^2)$  and the clipping operator clips the elements to the interval between 1 and 10. Then we compute:
$$
A = \left( \begin{array}{c c c c c} 1 & 1 & \ldots & \frac {n}{\sum_ {i = 1} ^ {n} A _ {i (d - 1)} ^ {N}} & \frac {1 0 n}{\sum_ {i = 1} ^ {n} A _ {i d} ^ {N}} \\ \ldots & & \ldots & & \ldots \\ 1 & 1 & \ldots & \frac {n}{\sum_ {i = 1} ^ {n} A _ {i (d - 1)} ^ {N}} & \frac {1 0 n}{\sum_ {i = 1} ^ {n} A _ {i d} ^ {N}} \end{array} \right) \bigodot   A _ {N}  ,
$$
where  $\odot$  denotes the Hadamard product. We set the diagonal elements of each  $A_{i}$  using the corresponding row stored in the matrix  $A$ . Note that  $\nabla^2 f(\mathbf{x}) = \frac{1}{n}\sum_{i = 1}^{n}A_{i}$  has the minimum and the largest eigenvalues being 1 and 10. For the general convex case, we use the same matrix  $A_{N}$  to generate a sparse matrix  $A_{M}$  such that  $A_{M} = A_{N}\odot M$  where  $M$  is a mask matrix with  $M_{ij}\sim B(1,p)$  and  $(1\ldots 1)\cdot M_{:j}\geq 1$ ,  $\forall j\in [1,d]$ . We then compute the matrix  $A$  and set each  $A_{i}$  in the same way.
$$
A = \left( \begin{array}{c c c c c c c c} \frac {2 ^ {- 2 0} n}{\sum_ {i = 1} ^ {n} A _ {i 1} ^ {M}} & \frac {2 ^ {- 1 9} n}{\sum_ {i = 1} ^ {n} A _ {i 2} ^ {M}} & \dots & \frac {2 ^ {- 1} n}{\sum_ {i = 1} ^ {n} A _ {i 2 0} ^ {M}} & 1 & \dots & 1 & \frac {1 0 n}{\sum_ {i = 1} ^ {n} A _ {i d} ^ {M}} \\ & \dots & & \dots & & \dots & & \dots \\ \frac {2 ^ {- 2 0} n}{\sum_ {i = 1} ^ {n} A _ {i 1} ^ {M}} & \frac {2 ^ {- 1 9} n}{\sum_ {i = 1} ^ {n} A _ {i 2} ^ {M}} & \dots & \frac {2 ^ {- 1} n}{\sum A _ {i 2 0} ^ {M}} & 1 & \dots & 1 & \frac {1 0 n}{\sum_ {i = 1} ^ {n} A _ {i d} ^ {M}} \end{array} \right) \odot A _ {M}.
$$
Through the construction, the smallest eigenvalues of  $\nabla^2 f(\mathbf{x})$  are clustered around zero, and the largest eigenvalue is 10. Additionally, each  $\nabla^2 f_i(\mathbf{x})$  is positive semi-definite.
We set the batch size to be 1 and thus we have  $f_{i_t}^{\star} = 0$  with interpolation and  $\ell_{i_t}^{\star} = 0$  without interpolation.
For AdaSPS/AdaSVRPS we fix  $c_p^{\mathrm{scale}} = 1$ , and for AdaSVRPS we further use  $\mu_F = 10$  and  $p_t = \frac{1}{0.1t + 1}$ . We compare against DecSPS [44], SPS [34] and SVRG [22] and tune the stepsize for SVRG by picking the best one from  $\{10^i\}_{i = -4,\dots,3}$ .
In addition to these optimizers, we further evaluate the performance of SLS [52], AdaSLS, SGD,  $\mathrm{SPS}_{\mathrm{max}}$  [34] and AdaSVRLS. We fix  $c_{l}^{\mathrm{scale}} = 1$ ,  $\gamma_{\mathrm{max}} = 10$ ,  $\beta = 0.8$  and  $\rho = 1/2$  for both algorithms and for AdaSVRLS, we further use  $\mu_{F} = 10$  and  $p_{t} = \frac{1}{0.1t + 1}$ . For SGD, we use the best learning rate schedules in different scenarios. Specifically, for both interpolation problems, we keep the stepsize constant and for non-interpolation problems, we apply  $\Theta(1/\sqrt{t})$  and  $\Theta(1/t)$  decay schedules for convex and strongly-convex problems respectively. We further pick the best stepsize from  $\{10^{i}\}_{i=-4,\dots,3}$ . For  $\mathrm{SPS}_{\mathrm{max}}$ , we use  $\gamma_{b} = 10^{-3}$  and we only showcase its performance in non-interpolated settings. We report the results in Figure F.1. We observe that AdaSLS is comparable if no faster than the best-tuned vanilla SGD.  $\mathrm{SPS}_{\mathrm{max}}$  is reduced to the vanilla SGD with constant stepsize. AdaSVRLS provides similar performance to AdaSVRPS but due to the cost of additional function evaluations, it is less competitive than AdaSVRPS.
Figure F.1: Comparison of the considered optimizers on synthetic data set with quadratic loss. The left block of the label illustrates the variance-reduced methods and the right represents SGD with different step sizes. (Repeated 3 times. The solid lines and the shaded area represent the mean and the standard deviation.)
optimizers	hyper-parameters used for synthetic experiments
st-convex+ip	st-convex+non-ip	convex+ip	convex+non-ip
AdaSPS	cpc=1
AdaSLS	clscale=1,β=0.8,ρ=0.5,γmax=10
SPS	c=0.5
SPSmax	c=0.5,γb=10-3
SLS	ρ=0.1,β=0.9,γmax=10
DecSPS	c0=1,γb=10
SGD	constant,η=10-2	O(1/t),η=1	constant,η=10-2	O(1/√t),η=10-1
AdaSVRPS	cpc=1,μF=10,pt=1/0.1t+1
AdaSVRLS	clscale=1,β=0.8,ρ=0.5,γmax=0.1,μF=10,pt=1/0.1t+1
SVRG	η=10-2
Table F.1: Hyper-parameters of the considered optimizers used in synthetic experiments. st-convex stands for strongly-convex and ip stands for interpolation.
F.2 Binary classification
Following the binary classification experiments presented in the main paper, we provide additional experiments for VR algorithms. The chosen hyper-parameters for each algorithm can be found in Table F.3. In particular, we fix  $c_{l}^{\mathrm{scale}} = 1$ ,  $\gamma_{\mathrm{max}} = 10^{3}$ ,  $\beta = 0.8$  and  $\rho = 1/2$  for AdaSLS and AdaSVRLS. We report the best  $\mu_{F} \in \{10^{-4}, 10^{2}\}$ . In Figure F.2, we observe that AdaSVRLS/AdaSVRPS provides similar performance to the other two variance-reduction methods. The details of the four considered datasets are summarized in Table F.2.
We next investigate the impact of the probability schedule on the convergence behaviours. We pick w8a as the dataset and run AdaSVRPS (Alg. 1) with and without probability decay. Specifically, we set  $p_t = B / n$  and  $p_t = \frac{1}{0.1t + 1}$  to separate the cases. We control the level of the interpolation by using  $B = 32$  and  $B = 128$  since  $\sigma_{f,128} \leq \sigma_{f,32}$ . From Figure F.3, we observe that decreasing probability schedule is more efficient when the problem is more non-interpolated. This is because for interpolated problems, the frequent computation of the full gradients at the beginning provides no additional convergence benefits.
duke	rcv1	ijCNN	w8a
n	44	20242	49990	49749
d	7129	47236	22	300
B	1	64	64	128
Table F.2: Number of datapoints, dimension of features, used batch size of four datasets from LIBSVM [10]
Figure F.2: Comparison of the considered optimizers on four LIBSVM datasets with regularized logistic loss. The left block of the label illustrates the variance-reduced methods and the right represents SGD with different stepsizes. (Repeated 3 times. The solid lines and the shaded area represent the mean and the standard deviation.)
Figure F.3: Comparison of different probability schedules for AdaSVRPS on the w8a dataset with regularized logistic loss. Decreasing probability is more efficient when optimizing highly non-interpolated convex problems (Repeated 3 times. The solid lines and the shaded area represent the mean and the standard deviation.)
optimizers	hyper-parameters used for binary classification tasks
duke	rcv1	ijCNN	w8a	for all
AdaSPS	\( c_p^{\text{scale}}=1 \)
AdaSLS	\( c_l^{\text{scale}}=1, \gamma_{\text{max}}=10 \)	\( c_l^{\text{scale}}=1, \gamma_{\text{max}}=10^3 \)	\( c_l^{\text{scale}}=1, \gamma_{\text{max}}=10^3 \)	\( c_l^{\text{scale}}=1, \gamma_{\text{max}}=10^3 \)	\( \beta = 0.8, \rho = 0.5 \)
SPS	\( c = 0.5 \)
SLS	\( \gamma_{\text{max}}=10 \)	\( \gamma_{\text{max}}=10^3 \)	\( \gamma_{\text{max}}=10^3 \)	\( \gamma_{\text{max}}=10^3 \)	\( \beta = 0.9, \rho = 0.1 \)
DecSPS	\( \gamma_b=200 \)	\( \gamma_b=100 \)	\( \gamma_b=100 \)	\( \gamma_b=100 \)	\( c_0=1 \)
SGD	constant, \( \eta=10^{-1} \)	\( \Theta(1/\sqrt{t}), \eta=100 \)	\( \Theta(1/\sqrt{t}), \eta=100 \)	\( \Theta(1/\sqrt{t}), \eta=100 \)
AdaNorm	\( c_g=1 \)	\( c_g=10 \)	\( c_g=10 \)	\( c_g=10 \)	\( b_0=10^{-10} \)
Adam	\( \mathrm{lr}=10^{-3} \)	\( \mathrm{lr}=10^{-2} \)	\( \mathrm{lr}=10^{-2} \)	\( \mathrm{lr}=10^{-2} \)	\( \beta_1=0.9, \beta_2=0.999 \)
AdaSVRPS	\( c_p^{\text{scale}}=1, \mu_F=100 \)	\( c_p^{\text{scale}}=1, \mu_F=10^{-4} \)	\( c_p^{\text{scale}}=1, \mu_F=10^{-4} \)	\( c_p^{\text{scale}}=1, \mu_F=10^{-4} \)	\( p_t=\frac{B}{N} \)
AdaSVRLSa	\( c_l^{\text{scale}}=1, \mu_F=100 \)	\( c_l^{\text{scale}}=1, \mu_F=10^{-4} \)	\( c_l^{\text{scale}}=1, \mu_F=10^{-4} \)	\( c_l^{\text{scale}}=1, \mu_F=10^{-4} \)	\( \beta = 0.8, \rho = 0.5, p_t=\frac{B}{N} \)
SVRG	\( \eta=10^{-2} \)	\( \eta=100 \)	\( \eta=10 \)	\( \eta=10 \)
AdaSVRG	We use the heuristic method provided in Section 5 from [14].
aγmax= 1
Table F.3: Hyper-parameters of the considered optimizers used in binary classification.
G Deep learning task
In this section, we provide a heuristic extension of AdaSPS to over-parameterized non-convex optimization tasks. When training modern deep learning models, Loshchilov and Hutter [35] observe that a cyclic behaviour of the stepsize, i.e., increasing at the beginning and then decreasing up to a constant, can help with fast training and good generalization performance. Since AdaSPS is a non-increasing stepsize, it excludes such a cyclic behaviour. To address this issue, we provide a non-convex version of AdaSPS which incorporates a restart mechanism that allows an increase of the stepsize according to the local curvature. The full algorithm is summarized in Algorithm 5. In practice, we can set  $u = \frac{B}{n}$ . Algorithm 5 updates the stepsize and  $c_{p}$  at the beginning of each epoch and uses AdaSPS (AdaSPS) for the rest of the epoch.
Following [34, 52], we benchmark the convergence and generalization performance of AdaSPS (DL) 5 for the multi-class classification tasks on CIFAR10 [28] and CIFAR100 [29] datasets using ResNet-34 [21]. We compare against SPS [34], Adam [26], AdaGrad [15], DecSPS [44] and SGD with momentum. We use the smoothing technique and pick  $c = 0.02$  for SPS as suggested in [34]. We use the official implementations of Adam, AdaGrad, and SGD with momentum from https://torch.org/docs/stable/optim.html. We choose  $\mathrm{lr} = 10^{-3}$ ,  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$  for Adam. We choose  $\mathrm{lr} = 0.01$  for AdaGrad. We choose  $\mathrm{lr} = 0.01$  and  $\beta = 0.9$  for SGD with momentum. Finally, we pick  $c_{p}^{\mathrm{scale}} = 0.02$  for Algorithm 5. In Figure G.1, AdaSPS (DL) shows competitive performance on both datasets. We leave the study of its theoretical properties to future work.
Figure G.1: Comparison of the considered optimizers on multi-class classification tasks with CIFAR10 and CIFAR100 datasets using ResNet34 with softmax loss. AdaSPS (DL) 5 and SPS provide remarkable performance on both datasets.
optimizers	hyper-parameters used for multi-classification tasks
AdaSPS (DS)	\( c_p^{\text{scale}} = 0.2 \)
SPS	c=0.2 + smoothing technique [34]
DecSPS	\( c_0 = 1, \gamma_b = 1000 \)
SGD-M	lr = 0.01, β = 0.9
AdaGrad	lr = 0.01
Adam	lr = 10-3, \( \beta_1 = 0.9, \beta_2 = 0.999 \)
Table G.1: Hyper-parameters of the considered optimizers used in multi-classification tasks.
Algorithm 5 AdaSPS (DL)
Require:  $\mathbf{x}_0\in \mathbb{R}^d,T\in \mathbb{N}^+$ $c_{p}^{\mathrm{scale}} > 0$  , update frequency  $u\in \mathbb{N}^{+}$
1: set  $\eta_{-1} = +\infty$
2: set  $\varepsilon = 10^{-10}$
3: for  $t = 0$  to  $T - 1$  do
4: uniformly sample  $i_t \subseteq [n]$
5: provide a lower bound  $\ell_{i_t}^{\star}\leq f_{i_t}^{\star}$
6: if  $t \mod u$  is 0 then
7: set  $c_p = \frac{c_p^{\mathrm{scale}}}{\sqrt{\sum_{s=0}^t f_{i_s}(\mathbf{x}_s) - \ell_{i_s}^\star}}$
8: set  $\eta_{t} = \frac{f_{i_{t}}(\mathbf{x}_{t}) - \ell_{i_{t}}^{\star}}{c_{p}||\nabla f_{i_{t}}(\mathbf{x}_{t})||^{2}}\frac{1}{\sqrt{\sum_{s = 0}^{t}f_{i_{s}}(\mathbf{x}_{s}) - \ell_{i_{s}}^{\star} + \varepsilon}}$
9: else
10: set  $\eta_{t} = \min \left\{\frac{f_{i_{t}}(\mathbf{x}_{t}) - \ell_{i_{t}}^{\star}}{c_{p}||\nabla f_{i_{t}}(\mathbf{x}_{t})||^{2}}\frac{1}{\sqrt{\sum_{s = 0}^{t}f_{i_{s}}(\mathbf{x}_{s}) - \ell_{i_{s}}^{\star} + \varepsilon}},\eta_{t - 1}\right\}$
11:  $\mathbf{x}_{t + 1} = \mathbf{x}_t - \eta_t\nabla f_{i_t}(\mathbf{x}_t)$
return  $\mathbf{x}_T$