SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Initially,  $J$  comprises  $n$  blocks. Hence, the minimizer needs to be computed no more than  $O(n)$  times throughout the algorithm. Furthermore, before each merge, we perform up to  $O(n)$  comparisons. However, this complexity is reduced to  $O(1)$  for top-  $k$  loss and AoRR loss due to the structures of these losses. Overall, the time complexity of our PAVA is  $O(n^2 + nT)$ . Notably, for top-  $k$  loss and AoRR loss, the time complexity is reduced to  $O(n + nT)$ .
Moreover, in the empirical human risk minimization with CPT-weight in (4),  $\theta_i(z_i)$  is nonconvex with regard to  $z_{i}$  because  $\sigma_{i}$  is dependent on the value of  $z_{i}$ . Specifically,  $\theta_{i}(z_{i})$  takes the form of a two-piece function for  $z_{i} \in (-\infty, B]$  and  $(B,\infty)$ , where  $B$  represents a certain threshold. However, despite its nonconvexity,  $\theta_{i}(z_{i})$  remains convex within each piece. Exploiting this property, we can determine the minimizer of such a function by comparing the minimizers of the two separate pieces. Considering this observation, the overall time complexity of the PAVA for the empirical human risk minimization is still  $O(n^{2} + nT)$ .
B.4 Datasets Description
Datasets for our experiments are generated in two ways.
Synthetic data. We construct synthetic datasets where the data matrix  $X$  and label vector  $y$  are generated artificially. We utilize the datasets.make_classification() function from the Python package scikit-learn [38] to generate two-class classification problem datasets of various dimensions.
Real data. In the case of real data, the data matrix  $X$  and label vector  $y$  are derived from existing datasets. The 'SVMguide' [26] dataset, frequently used in support vector machines, is included in our experiments. We also employ 'AD' [29], which comprises potential advertisements for Internet pages, and 'Monks' [50], the dataset based on the MONK's problem that served as the first international comparison of learning algorithms. The 'Splice' dataset from the UCI [18] is used for the task of recognizing DNA sequences as exons or introns. We additionally include 'Australian', 'Phoneme', and 'Titanic' dataset from [17]. Lastly, the 'UTKFace' dataset [53] is used to predict gender based on facial images. Detailed statistics for each dataset are presented in Table 5.
Table 5: Statistical details of eight real datasets.
Datasets	# Classes	# Samples	# Features	Class Ratio
AD	2	2,359	1,558	0.20
SVMguide	2	3,089	4	1.84
Monks	2	432	6	1.12
Australian	2	690	14	0.80
Phoneme	2	5,404	5	0.41
Titanic	2	2,201	3	0.48
Splice	2	3,190	60	0.93
UTKFace	2	9,778	136	1.24
B.5 Details of Our Algorithm Setting
In the experiments, we set the maximum iteration limit in our algorithm to 300. We utilize FISTA to solve the  $w$ -subproblem in our algorithm and L-BFGS for the smoothed  $w$ -subproblem. For varying frameworks, we adopt different choices of  $\{\rho_k\}$ , which varies when the iteration number  $k$  increases, in Algorithm 1. To enable the replication of our experiments, we provide a suggested selection for  $\{\rho_k\}$ . The specifics regarding these choices for  $\{\rho_k\}$  are detailed in Table 6. We set the  $\gamma$  in Section 4 to  $\gamma^k = \max \{10^{-5} \times 0.9^k, 10^{-9}\}$ .
B.6 Details of Experiments Setting
Spectral Risk Measures. Our objective is to demonstrate the versatility of our algorithm and its ability to converge to a globally optimal solution in the convex problem. Consequently, we utilize two real binary classification problem datasets and several synthetic datasets to highlight the advantages of our algorithm. We compare our method with the algorithms in [36]. We adapt their implementation
Table 6: The choices for  $\left\{  {\rho }_{k}\right\}$
Framework	ρ0	ρk
SRM	10-5	1.2kρ0
AoRR	2 × 10-7	5[(k-7)/3]ρ0
EHRM	10-4	if ||zk - Dkw||2 > 10-2then ρk = 1.02ρk-1else ρk = 1.07ρk-1
by replacing the gradient of the  $\ell_2$  norm with the soft threshold operator to incorporate the  $\ell_1$  regularization. For simplicity, we set the regularization parameter  $\mu$  as 0.01 and  $q$  as 0.8. The datasets are divided into  $60\%$  for training and  $40\%$  for testing. Both SGD and LSVRG are run for 2000 epochs and the batch size of SGD is set to 64, and the epoch lengths are set to 100 as recommended in [36]. We give an example of how we choose the learning rate. For comparative purposes, we have chosen the learning rate that yields the lowest objective value in a single randomized experiment among five different learning rates from  $\{10^{-2}, 10^{-3}, 10^{-4}, 1/N_{\text{samples}}, 1/(N_{\text{samples}} \times N_{\text{features}})\}$ . We illustrate this approach using the AD dataset under the SRM superquantile framework as an example. The results are compiled in Table 7. The learning rate corresponding to the bold values, indicating the minimal objective value, is subsequently chosen.
Table 7: Objective value in different learning rates for SGD and LSVRG.
Logistic Loss	Hinge Loss
Learning Rate	LSVRG(NU)	LSVRG(U)	SGD	LSVRG(NU)	LSVRG(U)	SGD
g(w) = μ/2||w||2
0.01	0.49612	0.19169	0.16200	0.18016	0.22529	0.11009
0.001	0.15757	0.15753	0.16210	0.08600	0.09591	0.08702
0.0001	0.17936	0.17930	0.27038	0.09270	0.09261	0.11568
1/Nsamples	0.15761	0.15758	0.16755	0.08609	0.08940	0.08911
1/(Nsamples × Nfeatures)	0.65849	0.65849	0.68518	0.85333	0.85333	0.96771
g(w) = μ/2||w||1
0.01	0.34731	0.37191	0.27573	0.56240	0.45764	0.17956
0.001	0.26991	0.26986	0.28041	0.16954	0.16959	0.15089
0.0001	0.30385	0.30385	0.38138	0.15621	0.15471	0.18250
1/Nsamples	0.27084	0.27078	0.28763	0.16805	0.15828	0.15198
1/(Nsamples × Nfeatures)	0.66741	0.66743	0.68723	0.87358	0.87358	0.97219
Average of Ranked Range Aggregate Loss. We employ the same real datasets as those used in [27]. Consistent with their experimental setup, we randomly split each dataset into a  $50\%$  training set, a  $25\%$  validation set, and a  $25\%$  test set. As [27] only considered  $\ell_2$  regularization with  $\mu = 10^{-4}$ , our experiments in this section also utilize the AoRR aggregate loss with  $\ell_2$  regularization. The choice of hyper-parameters  $k$  and  $m$  and the selection of inner and outer epochs for each dataset follows [27]. Table 8 presents the hyper-parameters for individual logistic loss, while Table 9 displays those for individual hinge loss.
Empirical Human Risk Minimization. We set  $\gamma = 0.61$  and  $\delta = 0.69$  in (5) based on the recommendations in [45]. We set  $B = \log(1 + \exp(-5))$  in (4). For simplicity, we only use  $\ell_2$  regularization. We employ the 'UTKFace' dataset [53], also used in [31], for gender prediction. The experiments are performed five times with different random seeds. In this experimental segment, we exclusively use the logistic loss as the individual loss since hinge loss has most of the zeros so that it is not easy to determine the corresponding  $B$  in (4) which means that its distribution is not suitable for this framework. The cumulative distribution function of loss  $F$  is derived from synthetic data, and  $F(B)$  is approximately equal to 0.05. The datasets are divided into  $60\%$  for training and  $40\%$  for testing. Other settings are similar to those in [31]. In accordance with [31], we divide the population into two groups based on race (white  $G_1$  and other race  $G_2$ ). However, for simplicity, we only obtain the parameters  $w$  for gender prediction by minimizing the problem (1) without using a neural network. The parameter settings mirror those in Section 5.1. We use the same method to get the learning rate, and the learning rate for all three algorithms is set to  $1 / (N_{\text{samples}} \times N_{\text{features}})$ .
Let TPR denote the true positive rate, FPR the false positive rate, and FNR the false negative rate. Additionally, let  $X \in \mathcal{X}$  represent the dataset and  $Y \in \{0,1\}$  the label. We utilize the same fairness metrics suggested by [5], as followed in [31]. Considering the privileged group  $G_{1} \subseteq X$  and the unprivileged group  $G_{2} \subseteq X$ , the definitions of the following metrics are from [31]:
- Statistical Parity Difference (SPD):  $P\left( {Y = 1 \mid  X \in  {G}_{2}}\right)  - P\left( {Y = 1 \mid  X \in  {G}_{1}}\right)$  .
- Disparate Impact (DI):  $\frac{P(Y = 1|X\in G_2)}{P(Y = 1|X\in G_1)}$
- Equal Opportunity Difference (EOD): TPR  $(G_{2}) - \mathrm{TPR}(G_{1})$
- Average Odds Difference (AOD):  $\frac{1}{2}$  ( $\mathrm{FPR}(G_2) - \mathrm{FPR}(G_1) + (\mathrm{TPR}(G_2) - \mathrm{TPR}(G_1))$ ).
- Theil Index (TI):  $\frac{1}{n}\sum_{i=1}^{n}\frac{b_i}{\mu}\ln\left(\frac{b_i}{\mu}\right)$  where  $b_i = \widehat{Y}_i - Y_i + 1$  and  $\mu = \frac{1}{n}\sum_{i=1}^{n}b_i$ . Here,  $\widehat{Y}_i$  is the prediction of  $X_i$  and  $n$  represents the number of samples.
- False Negative Rate Difference (FNRD): FNR  $(G_{2}) - \mathrm{FNR}(G_{1})$
Table 8: AoRR hyper-parameters on real datasets for individual logistic loss.
Datasets	k	m	# Outer epochs	# Inner epochs	Learning rate
Monk	70	20	5	2000	0.01
Australian	80	3	10	1000	0.01
Phoneme	1400	100	10	1000	0.01
Titanic	500	10	10	1000	0.01
Splice	450	50	10	1000	0.01
Table 9: AoRR hyper-parameters on real datasets for individual hinge loss.
Datasets	k	m	# Outer epochs	# Inner epochs	Learning rate
Monk	70	45	5	1000	0.01
Australian	80	3	5	1000	0.01
Phoneme	1400	410	10	500	0.01
Titanic	500	10	5	500	0.01
Splice	450	50	10	1000	0.01
C Additional Experimental Results
In this section, synthetic datasets labeled 1, 2, 3, and 4 are utilized. Detailed descriptions of datasets 1, 2, 3, and 4 can be found in Table 10.
Table 10: Details of the synthetic datasets.
Datasets	num_sample	num_feature	Datasets	num_sample	num_feature
1	1000	500	3	5000	1000
2	2000	1000	4	10000	1000
C.1 Experiments of Synthetic Datasets with ERM
The experimental settings implemented in this section are consistent with those discussed in Section 5.1. The datasets are replaced with the synthetic datasets. We use the same method to get the learning rate for other algorithms. Tables 11, 12, and 13 enumerate the mean and standard deviation of the results, primarily focusing on the objective value and test accuracy. 'sADMM' stands for the ADMM applied to the smoothed problem. Tables 11, 12, and 13 show that under most scenarios in the ERM framework, our ADMM algorithm exhibits superior performance in terms of objective
values when compared to existing methods. This superiority is particularly evident in instances of hinge loss, with the algorithm demonstrating a test accuracy that is analogous to other methods.
Figure 2 graphically demonstrates the correlation between time and sub-optimality for each algorithm within the superquantile framework and hinge loss, which represents a convex problem whose global optimum is achievable. The figure indicates that ADMM displays a more rapid convergence towards the minimum relative to other algorithms. While sADMM does not match the performance of ADMM, it does achieve a commendable rate of convergence when compared to other algorithms.
Table 11: Results in synthetic datasets with SRM ERM framework and  $\ell_2$  regularization. 'objval' denotes the objective value of problem (1).
Logistic Loss	Hinge Loss
Datasets	ADMM	LSVRG(NU)	LSVRG(U)	SGD	ADMM	LSVRG(NU)	LSVRG(U)	SGD
1	objval	0.0923 (0.0057)	0.0923 (0.0057)	0.0923 (0.0057)	0.0923 (0.0057)	0.0088 (0.0011)	0.0095 (0.0013)	0.0096 (0.0012)	0.0091 (0.0012)
Accuracy	0.7875 (0.0113)	0.7875 (0.0113)	0.7875 (0.0113)	0.7875 (0.0113)	0.7590 (0.0185)	0.7770 (0.0143)	0.7765 (0.0153)	0.7760 (0.0171)
2	objval	0.0800 (0.0012)	0.0800 (0.0012)	0.0800 (0.0012)	0.08 (0.0012)	0.0068 (0.0002)	0.0075 (0.0004)	0.0075 (0.0003)	0.0071 (0.0002)
Accuracy	0.8483 (0.0115)	0.8485 (0.0111)	0.8485 (0.0111)	0.8485 (0.0111)	0.8193 (0.0079)	0.8330 (0.0072)	0.8325 (0.0068)	0.8303 (0.0060)
3	objval	0.1704 (0.0037)	0.1704 (0.0037)	0.1704 (0.0037)	0.1704 (0.0037)	0.0732 (0.0045)	0.0807 (0.0047)	0.0809 (0.0044)	0.0756 (0.0046)
Accuracy	0.8347 (0.0039)	0.8347 (0.0039)	0.8347 (0.0039)	0.8341 (0.0035)	0.7999 (0.0091)	0.8138 (0.0075)	0.8135 (0.0053)	0.8091 (0.0090)
4	objval	0.1395 (0.0014)	0.1395 (0.0014)	0.1395 (0.0014)	0.1395 (0.0014)	0.0664 (0.0020)	0.0716 (0.0021)	0.0716 (0.0021)	0.0667 (0.0020)
Accuracy	0.9400 (0.0014)	0.9401 (0.0014)	0.9401 (0.0014)	0.9402 (0.0014)	0.9250 (0.0046)	0.9330 (0.0029)	0.9330 (0.0028)	0.9261 (0.0049)
Table 12: Results in synthetic datasets with SRM ERM framework,  $\ell_1$  regularization, and logistic loss. 'objval' denotes the objective value of problem (1).
Datasets	ADMM	sADMM	LSVRG(NU)	LSVRG(U)	SGD
1	objval	0.1829 (0.012)	0.1829 (0.012)	0.1829 (0.012)	0.1829 (0.012)	0.1845 (0.0118)
Accuracy	0.8490 (0.0076)	0.8495 (0.0087)	0.8490 (0.0076)	0.8495 (0.008)	0.8500 (0.0079)
2	objval	0.1693 (0.0037)	0.1693 (0.0037)	0.1693 (0.0037)	0.1693 (0.0037)	0.1729 (0.0037)
Accuracy	0.9190 (0.0061)	0.9193 (0.0061)	0.9190 (0.0061)	0.9190 (0.0061)	0.9193 (0.0056)
3	objval	0.2584 (0.0049)	0.2584 (0.0049)	0.2585 (0.0049)	0.2585 (0.0049)	0.2596 (0.0049)
Accuracy	0.9005 (0.0053)	0.9005 (0.0053)	0.9005 (0.0053)	0.9005 (0.0053)	0.9005 (0.0051)
4	objval	0.1475 (0.0037)	0.1476 (0.0037)	0.1479 (0.0037)	0.1479 (0.0037)	0.1495 (0.0037)
Accuracy	0.9599 (0.0026)	0.9599 (0.0027)	0.9599 (0.0027)	0.9599 (0.0027)	0.9599 (0.0025)
Table 13: Results in synthetic datasets with SRM ERM framework,  $\ell_1$  regularization, and hinge loss. 'objval' denotes the objective value of problem (1).
Datasets	ADMM	sADMM	LSVRG(NU)	LSVRG(U)	SGD
1	objval	0.0777 (0.0075)	0.0782 (0.0074)	0.0856 (0.0087)	0.0856 (0.0088)	0.0802 (0.0076)
Accuracy	0.8095 (0.0200)	0.8085 (0.0181)	0.8000 (0.0181)	0.7995 (0.0181)	0.8045 (0.0192)
2	objval	0.0786 (0.0023)	0.0790 (0.0023)	0.0877 (0.0022)	0.0876 (0.0023)	0.0836 (0.0023)
Accuracy	0.8880 (0.0143)	0.8895 (0.0148)	0.8823 (0.0121)	0.8833 (0.0105)	0.8848 (0.0130)
3	objval	0.1983 (0.0058)	0.1985 (0.0058)	0.2038 (0.0057)	0.2037 (0.0058)	0.2006 (0.0058)
Accuracy	0.8783 (0.0056)	0.8776 (0.0057)	0.8770 (0.0061)	0.8780 (0.0057)	0.8783 (0.0048)
4	objval	0.1192 (0.0031)	0.1193 (0.0031)	0.1229 (0.0042)	0.1221 (0.0027)	0.1206 (0.0030)
Accuracy	0.9563 (0.0018)	0.9565 (0.0015)	0.9564 (0.0020)	0.9564 (0.0021)	0.9561 (0.0019)
C.2 Experiments of Synthetic Datasets with AoRR
In the application of AoRR to synthetic data, we have configured the parameters  $k$  and  $m$  to be 0.8 and 0.2 times the size of the training set, respectively. In other words,  $k = \lfloor 0.8N_{\mathrm{train}}\rfloor$  and  $m = \lceil 0.2N_{\mathrm{train}}\rceil$ . We maintain the learning rate in the DCA at 0.01, and set the outer epoch at 10, and the inner epoch at 2000. The parameters for SGD and LSVRG are identical to those employed in the preceding experiment (the learning rate is chosen from the same method). Tables 14 and 15
(g)
Figure 2: Time vs. Sub-optimality gap in synthetic datasets with superquantile framework. (a-d) for  $\ell_2$  regularization, and (e-f) for  $\ell_1$  regularization. Sub-optimality is defined as  $F^{k} - F^{*}$ , where  $F^{k}$  represents the objective function value at the  $k$ -th iteration or epoch and  $F^{*}$  denotes the minimum value obtained by all algorithms. Plots are truncated when  $F^{k} - F^{*} < 10^{-8}$ .
Enumerate the mean and standard deviation of results for the objective value, accuracy, and time. A careful examination of Tables 14 and 15 suggests that our algorithm, in most scenarios, achieves lower objective values and comparable accuracy in a shorter time span relative to existing methods.
Table 14: Comparison with AoRR framework,  $k = \lfloor 0.8 \times N_{\mathrm{train}} \rfloor$ ,  $m = \lceil 0.2 \times N_{\mathrm{train}} \rceil$  and logistic loss. 'objval' denotes the objective value of problem (1).
Datasets	ADMM	DCA	LSVRG(NU)	LSVRG(U)	SGD
1	objval	\( 1.75 \times 10^{-3} (9.52 \times 10^{-5}) \)	\( 7.12 \times 10^{-1} (3.94 \times 10^{-2}) \)	\( 2.10 \times 10^{-3} (1.56 \times 10^{-4}) \)	\( 2.14 \times 10^{-3} (1.17 \times 10^{-4}) \)	\( 4.65 \times 10^{-3} (3.32 \times 10^{-4}) \)
Accuracy	0.7641 (0.0429)	0.7729 (0.0325)	0.7514 (0.0329)	0.7394 (0.0151)	0.7737 (0.0312)
Time	11.43 (0.87)	125.7 (21.88)	110.24 (8.13)	94.86 (7.58)	285.43 (1.17)
2	objval	\( 1.41 \times 10^{-3} (2.99 \times 10^{-5}) \)	\( 7.22 \times 10^{-1} (2.73 \times 10^{-2}) \)	\( 1.71 \times 10^{-3} (3.85 \times 10^{-5}) \)	\( 1.81 \times 10^{-3} (7.96 \times 10^{-5}) \)	\( 2.21 \times 10^{-3} (5.86 \times 10^{-5}) \)
Accuracy	0.8283 (0.0073)	0.8315 (0.0159)	0.8248 (0.0158)	0.808 (0.0094)	0.8427 (0.0139)
Time	19.33 (1.98)	107.27 (20.81)	124.14 (12.98)	104.22 (4.69)	399.86 (11.15)
3	objval	\( 2.75 \times 10^{-3} (6.88 \times 10^{-5}) \)	\( 7.49 \times 10^{-1} (1.22 \times 10^{-2}) \)	\( 3.00 \times 10^{-3} (9.22 \times 10^{-5}) \)	\( 3.07 \times 10^{-3} (1.14 \times 10^{-4}) \)	\( 3.33 \times 10^{-3} (1.09 \times 10^{-4}) \)
Accuracy	0.8149 (0.0109)	0.7936 (0.0112)	0.8083 (0.0095)	0.8075 (0.012)	0.8222 (0.0058)
Time	56.95 (2.07)	127.18 (19.86)	161 (14.01)	149.78 (8.62)	746.75 (72.28)
4	objval	\( 3.24 \times 10^{-3} (3.13 \times 10^{-5}) \)	\( 7.43 \times 10^{-1} (2.30 \times 10^{-2}) \)	\( 3.23 \times 10^{-3} (3.05 \times 10^{-5}) \)	\( 3.27 \times 10^{-3} (3.99 \times 10^{-5}) \)	\( 3.20 \times 10^{-3} (3.39 \times 10^{-5}) \)
Accuracy	0.9417 (0.0039)	0.9302 (0.0045)	0.9325 (0.0056)	0.9273 (0.0018)	0.9385 (0.0044)
Time	222.35 (19.30)	154.46 (43.08)	222.37 (62.44)	205.76 (64.40)	525.84 (237.37)
C.3 Experiments about Figure 1
We increased the sample size to further observe the performance of our algorithm and the experimental results are shown in Figure 3. It can be seen that both ADMM and sADMM exhibit relatively fast convergence compared to other algorithms. An interesting phenomenon is that in the case of a large amount of data, random algorithms are able to achieve decent results in the early stages of optimization. The underlying reason for this phenomenon may be that existing methods use mini-batch samples that are independent of the sample size to update model parameters, thereby reducing the computational cost per iteration. This allows existing methods to update parameters more frequently, leading to faster convergence in the early stages of optimization. In contrast, the proposed algorithm uses the entire batch of samples in each iteration, resulting in slower iteration speeds. This leads to suboptimal solutions in the early stages compared to existing methods. In Appendix B.3, we provide a detailed explanation of the time complexity of the PAVA, which is  $O(n + nT)$  for top-k loss and AoRR loss, where  $T$  represents the maximum number of iterations when solving each PAVA subproblem, and  $n$  represents the sample size. Therefore, with an increase in
Table 15: Comparison with AoRR framework,  $k = \lfloor 0.8 \times N_{\mathrm{train}} \rfloor$ ,  $m = \lceil 0.2 \times N_{\mathrm{train}} \rceil$  and hinge loss. 'objval' denotes the objective value of problem (1).
Datasets	ADMM	DCA	LSVRG(NU)	LSVRG(U)	SGD
1	objval	2.97×10-5(1.92×10-6)	9.99×10-1(2.66×10-2)	1.71×10-3(1.49×10-4)	1.72×10-3(1.38×10-4)	3.68×10-5(2.34×10-6)
Accuracy	0.7700 (0.0371)	0.7750 (0.0409)	0.7600 (0.0427)	0.7620 (0.0415)	0.7680 (0.0315)
Time	4.37 (0.66)	137.22 (28.90)	116.39 (3.64)	114.37 (2.93)	286.19 (1.35)
2	objval	2.12×10-5(6.51×10-7)	1.00×100(2.66×10-2)	7.05×10-4(5.90×10-5)	1.21×10-3(3.85×10-4)	2.42×10-5(5.20×10-7)
Accuracy	0.8320 (0.0093)	0.8270 (0.0099)	0.8420 (0.0113)	0.6940 (0.0277)	0.8380 (0.0093)
Time	5.54 (0.64)	172.5 (24.82)	104.79 (2.87)	93.35 (3.00)	352.67 (4.98)
3	objval	4.82×10-5(1.49×10-6)	9.89×10-1(1.90×10-2)	2.94×10-4(9.49×10-6)	2.78×10-4(4.43×10-5)	6.30×10-5(2.01×10-6)
Accuracy	0.8130 (0.0113)	0.8060 (0.0084)	0.8100 (0.0101)	0.8070 (0.0135)	0.8180 (0.0066)
Time	15.36 (0.70)	290.7 (26.15)	112.68 (8.08)	99.52 (4.88)	540.62 (66.63)
4	objval	8.53×10-4(1.59×10-4)	9.82×10-1(2.51×10-2)	1.22×10-4(3.17×10-6)	1.08×10-4(3.85×10-6)	8.16×10-5(1.00×10-6)
Accuracy	0.9430 (0.0053)	0.9360 (0.0046)	0.9310 (0.0049)	0.9240 (0.0028)	0.9350 (0.0048)
Time	31.42 (3.34)	396.03 (31.66)	129.22 (14.09)	115.06 (8.46)	633.69 (82.42)
(b)
Figure 3: Time vs. Sub-optimality gap in synthetic dataset with ERM framework and  $\ell_1$  regularization. The datasets with the same number of samples are generated by different random number seeds. Sub-optimality is defined as  $F^{k} - F^{*}$ , where  $F^{k}$  represents the objective function value at the  $k$ -th iteration or epoch and  $F^{*}$  denotes the minimum value obtained by all algorithms. Plots are truncated when  $F^{k} - F^{*} < 10^{-8}$ .
sample size, the time required for our proposed algorithm will also increase. Nevertheless, compared to existing algorithms, we are still able to achieve higher accuracy within a reasonable time frame.