Stochastic Douglas-Rachford Splitting for Regularized Empirical Risk Minimization: Convergence, Mini-batch, and Implementation
Authors: Aysegul Bumin
Kejun Huang
Venue: Accepted by TMLR ()

Abstract: In this paper, we study the stochastic Douglas-Rachford splitting (SDRS) for general empirical risk minimization (ERM) problems with regularization. Our first contribution is to prove its convergence for both convex and strongly convex problems; the convergence rates are $O(1/\sqrt{t})$ and $O(1/t)$, respectively. Since SDRS reduces to the stochastic proximal point algorithm (SPPA) when there is no regularization, it is pleasing to see the result matches that of SPPA, under the same mild conditions. We also propose the mini-batch version of SDRS that handles multiple samples simultaneously while maintaining the same efficiency as that of a single one, which is not a straight-forward extension in the context of stochastic proximal algorithms. We show that the mini-batch SDRS again enjoys the same convergence rate. Furthermore, we demonstrate that, for some of the canonical regularized ERM problems, each iteration of SDRS can be efficiently calculated either in closed form or in close to closed form via bisection---the resulting complexity is identical to, for example, the stochastic (sub)gradient method. Experiments on real data demonstrate its effectiveness in terms of convergence compared to SGD and its variants.

Published in Transactions on Machine Learning Research (11/2022)
Stochastic Douglas-Rachford Splitting for Regularized Empirical Risk Minimization: Convergence, Mini-batch, and Implementation
Aysegul Bumin
aysegul. www.buuufl..cc.. edu
Deportment of Computer and Information Science and Engineering University of Florida
Kejun Huang
kejun.kwang@uuff edu
Department of Computer and Information Science and Engineering University of Florida
Reviewed on OpenReview: https: optp//vee.....ee...emeee net/forun?s id= wwDD9rN662
Abstract
In this paper, we we study the stochastic Douglas-Rachhord splitting (SDRS) for general empirical risk minimization (ERM) problems with regularization. Our first contribution to prove its convergence for both convex and strongly convex problems; the convergence rates are 0(1/VB) and O(1/t), respectively. Since SDRS reduces to the stochastic proximal point algorithm (SPPA) when there is no regularization, it is pleasing to see the result matches that of SPPA, under the same mild conditions. We also propose the mini-batch version of SDRS that handles multiple samples simultancously while maintaining the same efficiency as that of single one, which is not straight- forward extension in the context of stochastic proximal algorithms. We show that the mini-batch SDRS again enjoys the same convergence rate. Furthermore, we demonstrate that, for some of the canonical regularized ERM problems, each iteration of SDRS can be efficiently calculated either in closed form or in close to closed form via bisection-the resulting complexity identical to, for example, the stochastic (sub)gradien method. Experiments real data demonstrate its effectiveness in terms of convergence compared to SGD and its variants.
1 Introduction
Regularized empirical risk minimization (ERM) the foundation for supervised machine learning Vapnik 1991). It formulates the learning problem as the following optimization problem
minimize wERd
fi(w)+ + +g(w),
(1)
where each fi is the loss function defined on a data sample over a training set, and g is regularization term to help improve the generalization ability of the model.
Large-scale machine learning models ideally trained in sample-by-sampll manner in order to reduce computational and memory overhcad. The majority of existing stochastic algorithms use the stochastic gradient descent (SGD) framework (Robbins & Monro 1951 Bottou et al. 2018), possibly with an additional proximal step to handle the (often nonsmooth) regularization function g. Convergence is often ((1/Vt) for convex functions and O(1/t) for strongly convex functions. Some follow-up work focus on improving the theoretical convergence rate, such as SVRG (Johnson Zhang 2013). SDCA (Shalev-Shware & Zhang
2013), SAGA (Defazio et al. 2014). and many more (Defazio 2016 Lei et al. 2017 Schmidt et al. 2017

---

Published in Transactions on Machine Learning Research (11/2022)
Allen- Zhu
2017
2018a b). at the expense of additional computational or memory overhead at the order of
n. On the other hand, some practical variations have been proposed, such as AdaGrad (Duchi et al. 2011),
Adam (Kingma & Ba 2014). ; AdaBelief (Zhuang et al.) 2020), and mom-mmmmntu (Liu et al. 2020 even
though their thcorctical convergence have not been shown to be better.
More recently, there emerges a family of stochastic algorithms that does not rely on just first-order derivatives but the full information of the stochastic functions (in some literature they are called model-based stochastic algorithms). This is most representative by the stochastic proximal point algorithm (SPPA) that tries to solve I without the regularization term g(w) by the update Wt+1 = ProxA, (wy), where i, is randomly sampled
from the index set (1.....n) n [Bertsckas] 2011; Ryu & Boyd 2014 Bianchi 2016 Pätraşcu 2020 Toulis et al.
2021 Bumin & Huang 2021). Perhaps due to the fact that the algorithu relies on implementing the proximal
operator of somewhat arbitrary functions, which could be as hard as solving the problem itself, follow-up up
works focus on variations of SPPA that typically involve a (subigradient update (Wang & Bertsekas 2013
Duchi & Ruan 2018 Asi & Duchi 2019; Davis & Drusvyatskly 2019). As explained by [Toulis al. (2021) and later in this paper as well, in the context of stochastic oplimization the pertinent proximal operators
could be calculated efficiently, despite that they seem lack of structures to be exploited. There remains two challenges for applying SPPA in practice:
1. How to handle the regularization term in a more systematic manner? A naive approach would simply treat it as part of the stochastic function and cope with it stochastically, which is certainly not ideal. On the other hand, since most existing regularization terms have efficient proximal operators, it does not seem appropriate to use gradient- based update to handle it as suggested by some of the
SPPA variants such (Wang & Bertsekas 2013 Duchi & Ruan 2018:
Asi & Duchi 2019 Davis &
Drusvyatskiy 2019).
2. How to allow the algorithm to do mini-batch, i.e., using more than single data sample to apply the updates? This is trivial for SGD-based methods since any (oub-)gradient is additive, so their stochastic gradients can be calculated in parallol and summed together. For SPPA, however, mini- batch seerns not only complicated but also hard to parallelize. One attempt was made in (Chadha et al. 2022) but parallelizable only if the approximation of the proximal update boils down to a stochastic gradient step.
In this paper, we address the two issues with similar convergence guarantees. First we revisit the stochastic Douclas-Rachford splitting (SDRS) to solve ① with the regularization term. Then we propose SDRS mini-batch, which is inspired by consensus Bouglas-Rachfrrd The mini-batch version of SDRS also provides a principled way of doing SPPA mini-batch that can be fully parallelized by essentially dropping the regularization term. We show that for both cases the convergence guarantee of SDRS exactly the same as that of SPPA, i.e. with rate ((IvVt) for convex problems and ((1/t) for strongly convex problems.
1.1 Stochastic Douglas- Rachffrd splitting (SDRS)
The stochastic Douglas-Rachford
As can see, the algorithm involves two proximal operators in each iteration. The proximal operator of function ʃ at point v is defined as Proxy(w)) = arg min f(w) + (1/2)llw - wl2. Comparing SDRS with some well-known stochastic algorithm, we notice that:
Rachford splitting (SDRS) is described in Algorithm 1
if line 4 is replaced by a simple stochastic gradient update, then becomes the proximal SGD
algorithm;
if there is no regularization g, line 2 becomes w 1 w., and the algorithum becomes the stochastic
proximal point algorithm (SPPA) (Rockafellar) 1976);
if the stochastic step in line 3 is deterministic, then it is the (deterministic) Douglas-Rachford Rachford splitting
Lions & Mercier 1979: Eckstein & Bertsekas 1992), which is essentially equivalent to the celebrated
alternating direction method of multipliers (Gabay & Mercier 1976 Boyd et al. 2011
There have boen attempts to analyze the convergence of SDRS (Shi & Liu 2016 Salim et al. 2018 but they both make the assumption that fr are differentiable, which not ideal since differentiability is typically not required by any proximal algorithm. We will provide convergence analysis for SDRS that does not require any
2

---

Published in Transactions on Machine Learning Research (11/2022)
Algorithm Stochastic Douglas- Douglas-Rachford splitting Algorithm (SDRS)
initialize W0
for t = 0.1..... do
3: Wx+1 f- Proxxen((0s)
4: randomly draw is uniformly from fl.....n) n}
5: W1+11 < -- + ProxAis, (2w++1 - w.: - w++11
6: end for
return ④ = -E=(T \ Et= d
function to be differentiable, as well as several additional results such as convergence under strong convexity and a mini-batch extension.
1.2 SDRS with mini-batch
One of the important reasons why SGD-type algorithms are so widely used is that it effortlessly include multiple samples in one stochastic update, called mini-batch. Due to linearity of differentiation, the mini-batch stochastic gradient is simply the average of the ft's that are sampled. Furthermore, each stochastic gradient can be computed in parallel, and a central node only need collect their average, making extremoly casy to parallelize.
For SDRS, if each stochastic update involve multiple data samples, evaluating the proximal operator becomes significantly more computationally demanding. Suppose we consider samples in each stochastic step, not only is the complexity be in general Ogrdd + y²), but it is also hard to exploit parallelization. To address this issue, we propose Algorithm [ that inspired by consensus optimization using Douglas-Rachford splitting, which generates p+11 sequences {wo J....,,.,, (w)!". (wi!"),
and {w.}. At iteration t, ,(k) is the sample index obtained by processor k; in other words, each processor is in charge of computing only ProxA.S,uv) which is assumed
to be casy to compute will be discussed in $3 Then the central node collects their average and computes Proxx.- The deterministic counterpart of A Algorithm
when p = n and each i(4) k; which would be equivalent consensus ADMM (Boyd al. 2011 after reorganization of the algorithm.
Rachford splitting Algorithm (SDRS) with mini-batch Algorithm Stochastic Rachfor
initialize (
for = 0, 1..,,.., T do
w (k)
3: W++1 < Prox
A=
4: for 1: = 11...... do
5: randomly draw x(())
uniformly from (1....,n}
6: (k) - (()) + Proxxss(0)) 22w - w (k)
W1+1
end for
8: end for
9: return
i = IET \ /EE,> d
An interesting observation is that if Problem [ does not involve regularization term 9, then line 2 of the algorithm reduces to simple averaging of the individual proximal outputs. This also suggests that one way of doing mini-batch for SPPA should be Algorithm 2 with line being the average, which is not the same as any the proposed mini-batch methods in (Chadha et al. 2022). The mini-batch approach proposed by (Chadha et al. 2022) allows approximate proximal operators, but is fully parallelizable only if the approximation reduces to stochastic gradient update. Our proposed mini-batch is the first variant of SDRS and SPPA that is fully proximal and parallelizable.
3

---

Published in Transactions on Machine Learning Research (11/2022)
2 Convergence Analysis
In this section, we provide convergence analysis of SDRS for general convex loss functions with regularizer ① to a global minimum in expectation. In recent years there have been some work tackling the convergence
analysis of stochastic proximal point algorithms, e.g., [Bertsekas] 2011 Pätraşcu 2020 Toulis et al. 2021 and stochastic relevant. work, but they both make additional assumptions that /4 dilferentiable, which is uncommon for proximal algorithms. Our result resembles that of SPPA by Bertsekas
MauglasRRachfrd splitting (Shi & Liu 2016 Salim
2018). The latter two are the most
(2011), which is satisfying since Douglas-Rachford Rachford splitting can be considered a natural extension to the proximal point algorithm to combine two or more proximal operators.
In what follows, we make the assumption that each fi(.) is Lipschitz contimnous with constant at least I, L.e.:
Assumption 2.1. There is a constant L such that for all, w, tö, and f(()), [f(w)) [i(ma] L| u
We note that this assumption is equivalent to the common assumption in analyses of stochastic (sub)gradient methods that all stochastic (sub)gradicnts are upperbounded (Vandenberghe) 2020 pp. 3.3). This is also the only assumption (other than convexily) used establishing the convergence SPPA (and some of its variants) (Bertsekas] 2011p. Since the g function not handled in a stochastic fashion, it is pleasing see that it is not required to satisfy the Lipschitz contimuous assumption.
For simplicity, in the main paper we provide convergence analysis in expeclation. In the appendix utilize the supermartingale convergence theorem to establish convergence in probability. However, the key inequalities are cstablishod in this section, and the rest of the stops are somewhat standardizod, which we relegate to the appendix.
2.1 Generic convex case without strong convexity
In this subsoction show that for general convex functions that not necessarily strongly convex, SDRS converges in expectation with rate 0(IVVI). This is the same rate as SPPA under the same assumption [erttsekas 2011), as well as SGD under the same assumption (Bottou et al. 2018).
Theorem 2.2. Suppose all fI,...,Ja and are conver, and Assumption holds. Tf solution w. erists, then with initialization wo the the sequence WI...., wT generated by SDRS
(Algorithm i satisfies
- Σ f((.) + +g(w.)) +
Di w.ll2 ΣT XL
E
f((2)) g((i)
(2)
2 2Σ'11 X
Proof. As per the update rules defined in Algorithm we have that
1 (w, - w141) E 0g(w(+1) and X (w141 - w(+1) E 0. - -w;; + +w,+1).
X
Since g and fú, are convex, their first-order conditions imply
g(w.) 2 g(W(+1) 1 X, (w: (044))(W, wt+1)
(3)
1 L [W0+1 - (.. 2.+1 + w; - w.+1).
f. (...) 2 f., - w. + wr+1) +
(4)
X
4

---

Published in Transactions on Machine Learning Research (11/2022)
Adding 3
and 4
together and rearrange, have
ft, (00+41 w; + wx+1) + g(wr+1) fú, (w.) - g(w.)
(wi+1 "i,)((u. w(+1)
[Bb+11 -1(11 (w. 21+1 wi, w(+1)
X
(wo+1 (iw((. w(+1) + X²
[11+1 w(+1)"((e)) - wi+1)
X
(wt+1
(ow) w(+1)
A
1 || w=11²
1
| w2112
|100 10++111
(5)
2
2d
2x.
We now invoke Assumption 2.1 to have
Ju (we+1) - L.|2s, - wr++1| 1|| S Si (t - aū, + w+++)
(6)
Combining (5) and
6
we have
2A, (fi, (W++1) +g(w++1) fi, (w,) g(w,))
|10; w.ll |100+11 w.ll 100, 20.1112 22,Ll/u0; :.++1l-
Furthermore, notice that
-10,; W 112 22 wz+1ll -(1 20.4111 ->,1) XZ12 1R12 As result,
24g (f.(4))))) + g(w)+1)) - fú, (2.)) - g(ww)) S 11:0; w.112 -1|2,,+11 - w-11 2
+ X2
(7)
Now we take conditional expectation of the random variable it, conditioned w and wi; according to the update rule in Algorithm| D only the value of u.+1 but not w++1 w, depends on it obtain the inequality
1 n n
2λ,
Σ
f((w))1) g(w(+1)
Σ fi(w.) g(w.) s W. w.ll2 E,, || w.11 XZ (8)
72
Taking total expectation of
S and summing the inequalities with t 0,1, T, we have
T
1 Σ
fi ((w.)) ) olwv.))
Σ
2\
E
f((w++1) + g(w(-1)
1=0
aml
iml
|1:0 w. 1| I
E |0r+1
w.ll
X²I.² < 180 w.11 Σ
1=0
t=
Dividing both sides by 2 de and lowerbouding the hand-handdside by Jensen's inequality (since the objective is convex),
function
[1 + g(0)
Σ d
Σ
n
filw.) + slw.))
72
ΣT
71
obtain
5

---

Published in Transactions on Machine Learning Research (11/2022)
Remarks. The proof is somewhat reminiscent to that of SPPA by Bertsekas 2011 Indeed, if function g is absent, SDRS reduces to SPPA since uh++1 th, and it is pleasing to see that the same result of SPPA holds for the more general SDRS. The fundamental observation in (Bertsekas) 2011] is that inequality (4 gives an upperbound on fú. (8b++) - fú, (w.): however, when taking expectation over the random variable i, the variable wh++1 also depends on ie; which would not lead to a meaningful expression. This is why we need to invoke Assumption 2.1 to bound fi, (1i))) - fú, (w.) instead. In the case of SDRS ( not only is Ji evaluated at a point that dependent on it, but also Si, and g are evaluated different points; when changing the argument of fú, to be the same 9, also managed to make the argument independent of it, thus similar steps can be made in the sequel.
Theorem 2.2 establishes inequality 12) for any choice of A We now detail some specific choices of the step sizes. The basic arguments are essentially the same as those that have been well-studied for SGD.
Corollary 2.3 (Diminishing step sizes). Suppose all fI,..., and 9 convez, and Assumption 2.1 holds. If a solution w, erists, then with initialization wo, the sequence wi,...; WT generated by SDRS (Algorithm with diminishing step sizes such that x + O and Ei=l d = oo satisfies
n
E
((00)
fi(w.) + g(ww).
(9)
Proof. Suppose the step sizes square summable, i.c., E, co, then it is obvious that the right-hand-side of ( the numerator is finite while the denominator goes to infinity, which proves the result. Even if the step sizes are not square summable either, it can be shown that the quotient Σ, Σ. + 0. A proof can be found in the supplementary, even though it is not new.
Corollary 2.3 suggests that some well-known diminishing rules such as x T/t d T/vi all guarantee expected convergence of SDRS.
Corollary 2.4 (Constant step sizes). Suppose all fiy..., In and g are convex, and Assumplion 2.1 holds. If solution w, erists, then with initialization wo the sequence wi,..., wr generated by SDRS (Algorithm with constant step size d X salisfies
n Σ fi(10) g((0) n
n Σ n2
f((wx) + g(wo.) 5 11000 w.
L
+
A.
(10)
2TA
This means:
Letting T - 00, the expected optimality gop is upperbounded by constant times X.
For a given T, the right-hand-side is minimized by letting À = =11000 - 1ooo-ww1l//a a0.l/ITV/ and substituting it
back in 10 shows
1 Σ Mww.)) + sl(w.)) <
1 w. IL
E
(((1) + g(00)
n
VT
The proof is straight-forward and thus omitted. This shows that the expected convergence rate O(1/Vl), which is the same SPPA (Bertsekas] 2011) and SGD.
2.2 Strongly convex case
We now provide improved convergence analysis when g is strongly convex. This is particularly useful when the regularization function the Tikhonov regularization (Fuclidean norm squared).

---

Published in Transactions on Machine Learning Research (11/2022)
Proposition 2.5. If all f1,...: fn are convez, 9 is strongly convez with parameter μ. and Assumption 2.1 holds, then with inilialization woo the sequence wi...., ur generated by SDRS (Algorilhm step size X satisfies
7 with constand
2 Σ
E ((..))))((.))) -
1ul 2D w.l1 (1+ A4/2)" + AL2 + &2I
E
f(w)) + (
-
(11)
2
This means SDRS converyes, in expectation, linearly to a suboptimal point with gap proportional to À + uZ² If X is small, which is usually the case in practice then the gop is dominated by constant times X.
Proof. Similar to the first few steps of the proof of Theorem 2.2 but replacing 3 with the strongly convex form
g(w.) 2 g(w(+1) -20; wi+1) w w(+1) ) [lw. w++1l12
we get
2dc (Ji, (W++1) + g(wr+1) Sú, (w,) - ((w))) S ||00 - w.1l 1| - ||00 w. 112 + X3 Aull| - we+ 1|?
Using the inequality ||a + b1l2 = 2 2||a|| 1|2 + 2 2|bb| we can further upperbound the right-hand-side by 24x (fi, (W++1) + g(w,+1) ) - fú, (w,) - g(w.))
Aepl l|0, - r1,4111 112 + Ae mo+1ll -w.,-w,1-"
i w,l12 - ||10 -112+
Aepf
[0 -.,--,,--" 1 +
w.ll2
||0++1 - 12 + 1²1,2 + N2
2
where we upperbound |, w < X2 because (1i+41 w(+1) is subgradient at (w:+1
ú. + W(+1), of which the norm is no more than I. due to Assumption 2.1
Now we take conditional expectation of the random variable it, conditioned on wr41 and w. according to the update rule in Algorithm ! only the value of u but not w/+1 or w depends on i, we obtain
n
1 Σ f((w.) -g(w.)
2d,
f(((+1) + g(wr+1)
71
n
(12)
dep
S |1 u.l12
1+
Ei, 1|00411 - w.11 X²I. + A2L.?
2
Letting the step sizes be constant A X, we multiply both sides by (1 + Ap/22), take total expectation, and sum over O,..., T get
T
Ap
n
f((w+1) + g(we+1)
1
n f((w.) +g(w.))
2x
1
E
2
n
71
T
$ ||00 - w 2 + (22L² + 23 p2
1+
2
Applying the geometric series O1 (1 Ap/2)² = (11 + 22/2)T 1)/())(((/ppp (Ap//)) and lowerbound all the expectod function values on seft-hand-sid by Jensen's inequality, we get
We now show that with diminishing step sizes, the expected optimality gap converges to zero ((1(l) rate. Theorem 2.6. If all fi are conver, g is strongly conver with parameter μ,, and Assumption 2. holds. If a solution w. erists, then with initialization wo the sequence wI...., wr generated by SDRS (Algorithm with diminishing step sizes d = B/t where B > 4/p, then
2/ w0 - w.ll2 2L23(1 + m1)
f((()) + g(0))
<
+
(13)
12
ST(T + +1)
T+1
7

---

Published in Transactions on Machine Learning Research (11/2022)
When goes relatively large, the right-hand-side is dominated by the second term, which yields the O(1/t)
conneryence rale.
Proof. Taking total expectation of
12 and plug in the step sizes d B/t, we have
E
[((w++1) ((w++1)
Si(wx) g(w+))
7
1
21 + 118
L23
B² B²ul?
< E Elli, w.12
E 200+1
w211
+
28
48
2t
212
t
++2
L 3(1
3p}
$ 23
E 1|
w. 11
E
@1101111 w.11
2
21
where the last incquality stems from p and 1/t2 1/t for 2 1. Multiplying both sides by t 1 yields
n - -g(w.)
(t 1)+
f((++)) + g(w(++))
fe(w.)
71
71
11
+ 1) + 2)
1ii+1 - w. 1| L23 + 3p)( 1)
E 12:0
w. 11?
23
23
2t
t(t + 1)
+ 1) 2)
E E
w. w.ll 11
[00+1 w. 112 + LL3 +Bp),
23
2,3
where, again, the last incquality is duc to (t + 1)/2t for t2 1. Now let t = 1,2, inequalities, get
2, .... T and over all
1 Σ
t
f((w)) + g(w))
fi(w.) - g((w.)
n
n i=1
1
T(T 11
E
|01
w.ll2
E 1 w.ll
+TL w
Bp)
8
23
E
|0 w.12
BI2 3p).
Lowerbounding the left-hand-side by Jensen's inequality and dividing both sides by 1 1 T(T+1)/2
yields 13
2.3 Convergence of mini-batch SDRS
Thcorem 2.7.
The converyence of mini-batch SDRS as in Algorithm eractly the same as SDRS, i.e.,
for generic conver functions, we have
-1' f((w.) + g(w.))
|io ur, 1|2 + ΣT X21. 2I d,
E Σ M(() - 0(10)
(14)
which leads to a convergence rale of ((IVV1) with oppropriately chosen step sizes;
if g is strongly conver with parameter then with diminishing slep sizes A = B/t where B > 4/u, then
< 2| w.ll 2T.2 + 113)
n Σ f((10) + g(w))
ST(T+ +
(15)
T+1
When T goes relatively large, the right-hand-side is dominated by the second term, which gields the (1/t)) convergence rale.
8

---

Published in Transactions on Machine Learning Research (11/2022)
Proof.
As per the update rules defined in Algorithm 2
we have that
1
w (k)
W
0g(we+1)
X
and
(wi+1
(k) ((++)
E af uw) (w0 (k)
wi; (K)
+ wi+1), ,....,...
d
Due to (strong) convexity, their first-order conditions imply
1 P
g(w.)= 2 g(w++1) +
W
wot1 (w. - w(+1) + 1 lw., - w 12,
d
P
2
k=1
1,00 2 f 100 (10
(k)
wi
(k)
+ wl+1)
1
(10241 wi(h) T
20/11 wi
(b) (k)
wo+1), -I....P.
A,
Summing them up gives
p
P
24,
f am [+
(k)
(+t)) + w(+1) + ((w(+1))
fa (w.) -g(ww)
P
1 p
Σ
(0! (w.-wi41
2
[1(+1 wc+1) w)+1) (w. r0/4] + v0!1)) w(+1)
(k)
2
W++1
+
p k=1
p k=1
- -eeelles - wr+1l1
2 p
2 1
i 1+1
(k)
u
nm(1))
(w+
10++1
(10411 W641) (Ti (
(k)
p
p k=1
-udellw>> - w
(k)
w0( (1)) (w. - (1)b) - Mdel120. w1++ll2
w i+1
k=1
[w - w/ 11 l|., (k) 112 [|00
(K) -0, - pd:eleos - --,
w1++1l?
(16)
The rest of the steps follows almost identical to those in the proof of Theorem 2.2 after 5
Invoking
Assumption 2.1 to each saas we have
(K) fao 2|| w0 s f,00 (20 (k)
wi(h) (b) + ww-)), I....,P.
Plugging it into 16 gives
2λ,
f 00 (w(+1) + g(W(+1)
1
Σ
fal (w.) - g(w.)
p k=1
k=
(lw,- - wi 112 1lw.
(k)
1|2
|| 112 + 2h, [1( ,0("))) - -deellw, w
(K)
11+11
p
Also notice that
[| - u8 12 + (2x (k) = (11x0(4) (K) (x) 211) 2 + X < X
This implies
1
p
Σ
f,09 (Wi+1) g(W(41)
f,g0 (wx) - -g(w.))
p 1=1
k=1
[|w- - (t) -lw.. w(+2112) + 1 -dellw. w.1112
9

---

Published in Transactions on Machine Learning Research (11/2022)
Using the incquality ||a + bll2 x 2||a|12 + 2||b[]? we can further upperbound the right-hand-side by
1 P
2
(00 (w0+1) + -((0++))=
Σ
1 1,00 (w.) -g(wo))
p k=1
uade P
| 10!20112 1|0, u ( 11)
171,2
[
w 11?
|
we+1 112)
p
p [w 20,² 112 (1 1+
Aepl
||00 20/411 113
1² 2 + x2221.
(17)
The next step is take conditional expectation of the random variables i((x) in 17 and each of ūi (k), according to the update rule in Algorithm 2 only the values of w of (8) depends on o/ Specifically, on the left-hand-side, we have that for each
conditioned on w141
but not Wr+1 any
+++
Bug f000 (wL+1) = Σ ((w++1) and E,g0 faw =
1 - Ji(w));
n i=1
S0
1
a
1 n f((e+1) + +g(W(41))
Σ f((w,) g(ww))
2
n
n
1
[|w, 10/ (1
dupl
Fuges 1100
20/411 12) 2 + A22u2.
(k)
(18)
P
Taking total expectation of
18 gives
1 Σ fi(w.) -g(ww)
2A E
f(w(+1) + +g(we+1)= -
E Ellw, -wi 12.1
Aspl
F. F.l. - 10/4211) 122 + 221. 2 + X21.?
(K)
(19)
2
Now we separate the cases of μ = 0 and p > 0:
p 1 0, then taking summation of 19 with L 0,..., gives
2
X E
1
fi( + g(w++1)
fi(w.) - -s(w.)
E
l.
w
(8)
Σ
X222.
p k=
t=
1=0
Since each whth) initialized at the same point wo then we have T, if constant step sizes d |1 w.,l// w.ll/IvT. T, then
14 in Theorem 2.7
For given
F
n E1(0) n
(()) + g(())
n
Si S(ww)) + s(wrx))
5 |wo w. 1
/T
which shows the 0(1/VI) convergence rate in expectation.
10

---

Published in Transactions on Machine Learning Research (11/2022)
If μs O, then plugging in the step size X 3/t into 19 gives
fi(w(+1)= + g(W++1) 1 n
Σ
s((w+) - g(ww))
12
i=1
1
p
E
(1)
w.ll w|l1?-
+ pB
P
E
W [1 -w.1l +
(k)
L
+
8 p.L
23p
43p
2t
2ta
k=1
P
E
|1000
(1:)
-w.
++ 2
P
E ||00
w. 10.12=-
L2 + Bp)
23p
233
21
k=1
where the last inequality stems from 4B > and 1/12 < 1/t for 2 1. Multiplying both sides by + yields
1 Σ fi(w.) ( - g(w.)
1I+
f((w++1) + g(we+1)
1=1
i=1
1(1 + 1)
(1+ + 1)(1 + 2)
w. 123(1 + BB))(L +
E 100
Ee ||
2,3p
2,3p
2t
k=1
k=1
1(1 1)
(t + 1)( + 2)
P
Σ
w-1
(R) 2+1
w. 112 + +L23(11 + 3p))
23p
28p
where, again, the last inequality is duc to (t 1)/2t < 1 for t2 Now let t 1.2.... . and sum over all inequalities, we get
Σ IB
1 Σ fi(w.) - g(ww)
fi(w)) - g(w,)
i=1
T(T + 1)
E W
w.11
Σ
E
ww,1 + TL23(1 + Bp) 117
3p k=1
2,8p
k=1
<
w.ll + TLL23(1 + 34). E 1
3y
k=
Since each W is initialized at the same point Rv. then lowerbounding the left- sidd- by Jensen's 's inequality and dividing both sides by +
(k)
T(T + 1)/2 yields 15
3 Efficient Computation of the Proximal Operators
In this section, we consider how efficiently implement SDRS for regularized ERM problems 1 In each iteration of Algorithm [ it involves computing two proximal operators, one for the regularization function g and one for the loss of one sample fi. It is well-known that most regularization functions admit proximal operator that is efficient to compute, such as the well-known soft soft-thresholding for Lı norm regularization to promote sparsc solutions, or block soft-thresholding for sum of Euclidean norms to promote group sparsity. More comprehensive surveys of similar proximal operators have been studied in many works, for example by Parikh & Boyd (2014). As for the proximal operator for fa in line 4 of Algorithm ! it may seem just as challenging as solving the batch problem. However, we will see that there are some interesting properties when fr involves the loss evaluation of just one or few data points in a data fitting scenario.
In ERM formulation for supervised learning, we can often write the one-sample loss function as fi(w) [wTw y1) for regression or fi(w) ((yaTw) for binary classification, which means it is scalar function composed with a. w.w. Suppose it is differentiable, then by applying the chain rule, we have Vfi(w)) and (aTw),
11

---

Published in Transactions on Machine Learning Research (11/2022)
where E' is the derivative of the scalar function (()) This is also true when fa is nonsmooth that any subgradient is some scaled version of Ztz. Plugging it into the optimality condition, we get that
Proxxs, (w)) = w 0.2,
(20)
for some scalar a. The proximal update further reduces to finding the scalar a that solves the following singlo-variate convex problem,
a² | 2
minimize f(,] aall=;1))) +
(21)
This can be done by simply setting the derivative equal to zero. Due to convexily, its derivative is a monotonic function, one naive way to find the root is via bisection. In some cases the choice of even has closed-ford expression. In the supplementary we include the specific proximal operator for the least squares loss, logistic loss, hinge loss, and absolute error loss, which are some of the most widely used losses in ERM.
The special form of the proximal update (20) when the one-sample loss is the composition of a scalar convex function and linear function also gives an interesting explicit form of SDRS. If line of Algorithm [ is in the form of 20 then line 4 simplifies to w W++1 -
aæs; plugging it into line 2 further simplifies SDRS to
w++1
Proxx,9 (w: ax,),
which looks surprisingly similar to the proximal stochastic gradient descent, except that the step size a is chosen more delicately via solving (21). As we will see very soon in the next section, this delicate choice often makes a big difference in torms of convergence in practice. Similar arguments can be made for the mini-batch version, making the iterates having the form
Wr+1 f- PrOXAce
W:
(k)
where each (*) is calculated individually according to its own 21 This makes mini-batch SDRS more sophisticated than mini-batch SGD, as the search direction takes weighted sum of each samples to properly reflect their contribution in the minimization.
All of the aforementioned discussion on the specific implementation of SDRS only applies when the loss function is in the form fu(w)) f(x:w - yi) or ((y=Tw) However, even for gemeric nonlinear programming problems, it is still possible to efficiently evaluate the proximal update. If acquiring first-order information of fi is not computationally expensive, one can use methods such as L-BFGS (Nocedal & Wright, (0066) or
accelerated gradient descent [Nesterov 1983) to compute ProxAς (:) with approximately O(d) complexity.
4 Experiments
In this section, we show the performance of SDRS with proposed efficient implementation on both classification and regression tasks, and compare it with some widely used alternatives such as SGD-momentum (Liu et al. 2020), Adam (Kingma & Ba 2014 and AdaBelief (Zhuang et al. 2020 All formulations include an Li norm regularization, and each SGD-type algorithm is followed by soft toft-thresholding to handle it. More real data experiments can be found in the appendix.
4.1 Classification
We perform binary classification by using SVM and logistic regression with L, norm regularization on the bank note authentication data seliuua & Grafl 2017) This includes images of genuine and counterfeit banknotes. There are 1372 images in total. There are five attributes in each image, out of which four are features and one is the target attribute. The target attribute contains and 1, where 0 indicates genuine notes and indicates fake notes. The ratio between the two classes is 55/45 (genuine/conin cconterfeit) We present Figures and 2 illustrating SDRS's performance in loss-time and accuracy-time respectively. Figure T illustrates how
12

---

Published in Transactions on Machine Learning Research (11/2022)
SCD
90
SCD
1.0
00
Accurecy
0.8
0.6
60
2065
940
250
Nomartun
0.5
1.5 Seconds
0.5
1.5
2.5
Seconds
Figure 2: SVM with L1 regularization Bank Note Authentication: classification accuracy per seconds, batch size 1
Figure 1: SVM with L1 regularization Bank Note Authentication: loss per seconds, batch size 1
PCC
80
0.6
Accuracy
70
0.4
60
5BRS
SGD
560 & Momanium
0.5
1,5
2.5
0.0
0.5
1.0
Seconds
Seconds
Figure 4: Logistic regression with L, regularization on Bank Note Authentication: classification accuracy per seconds
Figure 3: Logistic regression with L1 regularization on Bank Note Authentications loss per seconds
the regularized hinge loss value decreases with time (seconds) by comparing different algorithms. Figures and 2 demonstrate that using our efficient implementation, SDRS takes less time to achieve a smaller loss and greater accuracy despite its complexity per iteration.
In Figures and we present results on logistic regression model with L1 regularization on Bank Note Authentication dataset, representing loss/time and accuracy /time respectively. As seen in Figure 3 SDRS achieves lower loss value in less time than all other algorithms; the closest performance is achieved by Stochastic Gradient with Momentum. With reference to Figure 4 SDRS achieves the highest accuracy rate in the shortest amount of time, and it has the lowest variance of all of the algorithms.
4.2 Classification using mini batch
We also run the classification experiments using the Bank Note Authentication data set. Figures 5 and D show the loss and accuracy values of SDRS on Bank Note Authentication dataset. for SVM approach. Our modification allows SDRS to handle larger batches. We observe that the SDRS still outperforms all the other methods. We also use Logistic Regression on the same Bank Note Authentication datasct. The Figure 7 and represent the loss and accuracy values respectively. We observe that SDRS reaches smaller value in the first few iterations.
13

---

Published in Transactions on Machine Learning Research (11/2022)
5015
1.0
80
Accuracy
La85
0.4
0.2
50
500
0.8
1.0
Seconds
Seconds
Figure 5: SVM on Bank Note Authentication data set: loss per seconds, batch size
Figure 6: SVM on Bank Note Authentication data
set: accuracy per seconds, batch size 4
80
Accuracy
0.5
70
0.4
60
0.3
Adm
50
5CD
0.2
0.6
1.0
12
0.3
Seconds
Seconds
Figure 8: Logistic Regression on Bank Note Authen- tication data set: accuracy per seconds, batch size 16
Figure 7: Logistic Regression on Bank Note Authen- tication data set: loss per seconds, batch size 16
4.3 Neural network training using mini-batch SPPA
As suggested in 1.2 the proposed mini-batch SDRS in Algorithm also suggests fully parallelizable mini-batch SPPA when the regularization term g does not exist, as opposed to the one that cannot be fully parallelized by Chadha et al. (2022). In this section we apply mini-batch SPPA to the nonconvex problem of neural network training. We apply on two famous image classification data sots, CIFAR10 and MNIST, with two different network architectures, CNN and ResNet.
CIFAR10 (Krizhevsky et al. 2010 consists of 50,000 training and 10,000 test images of size 32 32 in 10 classes. The performance of CNN training CIFAR10 is shown in Figures g and 10] The network consists of 5 layers, each of the first 2 being a combination of 5 x 5 convolutional filters and 2 x 2 max pooling, and the last three being fully connected. The loss function is cross-entropy loss, and the activation function is ReLU, The batch size is 4. To show the behavior of the algorithms more in detail, we recorded the loss values at every 500 stochastic updates, the experiment run for five epochs, and in total, 125 updates are presented (25 x 500 giving the number of training samples). The accuracy calculated on the test data at every 500 stochastic update overall for five epochs, using the accuracy calculation in the PyTorch tutorial.
5 Conclusion
We studiod stochastic Rachford splitting (SDRS) in this paper. Our first contribution to provide convergence analysis of SDRS, thus closing the theoretical gap. We also discussed various implementation
14

---

Published in Transactions on Machine Learning Research (11/2022)
SPOLAGO
Adagied Adam
60
2.0
SGD
1.8
Cross Entropy
Accuracy
1.6
1.4
30
1.2
SPBA AGC
Acam
1.0
Adidulla
520 whth Voreest.m
0.8
520
200
400
600
800
1000
0
200
400
600
800
1000
Seconds
Seconds
Figure 9: CNN on CIFARIO: cross entropy loss on test data
Figure 10: CNN on CIFAR10: prediction accuracy on test data
issues to make it more applicable for practical use of training regularized ERM problems, including a mini-batch strategy that keeps the simplicity of pertinent proximal operators. Our experiments showed that SDRS able to perform better than many well-known stochastic algorithms for various data sets.
References
Zeyuan Allen-Zhu. Katyusha; The First Direct Acceleration of Stochastic Gradient Methods, In STOC, 2017.
Zeyuan Allen-Zhu. Katyusha X: Practical Momentum Method for Stochastic Sum-of-NNnoonvex Optimization. In Proceedings of the 35th International Conference on Machine Learning. ICML '18, 2018a.
Zeyuan Allen-Zhu. How To Make the Gradients Small Stochastically. In Proceedings of the 32nd Conference on Neural Information Processing Systems, NeurIPS '18, 2018b.
Nuno Antonio, Ana de Almeida, and Luis Nunes. Hotel booking demand datasets. Dala in brief, 22:41 49, 2019.
Hilal Asi and John C Duchi. Stochastic (approximate) proximal point methods: Convergence, optimality, and adaptivity. SIAM Journal on Optimization, 29(3):2257-228) 2019.
Dimitri P Bertsekas. Incremental proximal methods for large scale convex optimization. Mathematical
Programming, 129(2):163, 2011.
Pascal Bianchi. Ergodic convergence of stochastic proximal point algorithm. STAM Journal on Oplimizalion, 26(4):2235-20) 2260, 2016.
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for largo-scale machine learning. STAM Review, 60(2):223 311, 2018.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Fekstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trendso in Machine learning, 3(1):1-122, 2011.
Aysegul Bumin and Kejun Huang. Efficient implementation of stochastic proximal point algorithm for matrix and tensor completion. In 2021 29th European Signal Processing Conference (EUSIPCO), pp. 1050-1054. IEEE, 2021.
Karan Chadha, Gary Cheng, and John Duchi. Accelerated, optimal and parallel: Some results on model-basod stochastic optimization. In International Conference on Machine Learning, pp. PMT.R, 2022.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Damek Davis and Dmitriy Drusvyatskiy. Stochastic model-based minimization of weakly convex functions. STAM Journal on Oplimization, 29(1):207-23) 239, 2019.
Aaron Defazio. A simple practical accelerated method for finite sums. Advances in Neural Information Processing Systems, 29:676-600 684, 2016.
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with
support for non-strongly convex composite objectives. In Advances in Neural Information Processing Systems, pp. 1646 1654, 2014.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archlvee.ccscok
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121 2159,, 2011.
John C Duchi and Feng Ruan. Stochastic methods for composite and weakly convex optimization problems. STAM Journal on Oplimization, 28(4):3229 -2599, 2018.
Jonathan Eckstein and Dimitri P Bertsekas. On the douglas douglas-rachfort rachford splitting method and the proximal point algorithm for maximal monotone operators. Mathematical Programming, 55(1):293-3)) 318, 1992.
Hadi Fanace-T T and Joao Gama. Event labeling combining ensemble detectors and background knowledge. Progress in Artificial Intelligence, pp. 1-15, 2013. ISSN 2192-6352. doi: 10.1007/s133748-0133 -13-4488--11-00---
Daniel Gabay and Bertrand Mercier. A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Computers 8 Mathematics with Applications, 2(1):17-40, 1976.
Simon Haykin. Adaptive Filtering Theory. Prentice Hall, 2002.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pp. 770 778, 2016.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems, pp. 315 323, 2013.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2014.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http://wwwe cs. toronto. edu/kriz/ aauu/kriz/ifury hlmll, 5, 2010.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. 2010. URL http://yann. lecun. com/eadl caadb/mmiisl, 7:23, 2010.
Lihua Lei, Cheng Ju, Jianbo Chen, and Michael Jordan. Non-convex finite-sum optimization via sesg methods. In Advances in Neural Information Processing Systems, pp. 2348 2358, 2017.
Picrre-Louis Lions and Bertrand Mercier. Splitting algorithms for the sum of two nonlinear operators. SIAM Journal on Numerical Analysis, 16(6):964-975 1979.
Yanli Liu, Yuan Gao, and Wotao Yin. An improved analysis of stochastic gradient descent with momentum. Advances in Newral Information Processing Systems, 8861-11111 2020.
Andrew Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Arnual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150. Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:://ww awwwwb.ww............ org hwtpee/.ww./...aooooo///11--001-
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate o(1/k²). In Dokl. akad. nauk Sssr, volume 269, pp. 543 543-547, 1983.
16

---

Published in Transactions on Machine Learning Research (11/2022)
Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in Optimization, 1(3): 127-239, 2014.
Andrei Pätraşeu. New nonasymptotic convergence rates of stochastic proximal point algorithm for stochastic convex optimization. Optimization, pp. 1-29, 2020.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics, pp. 400-407, 1951.
R Tyrrell Rockafellar. Monotone operators and the proximal point algorithm. SIAM Journal on Control and Optimization, 14(6):877-898, 1976.
Ernest K Ryu and Stephen Boyd. Stochastic proximal iteration: a non-asymptotic improvement upon stochastic gradient descent. Preprint, 2014.
Adil Salim, Pascal Bianchi, and Walid Hachem. A constant step stochastic rouglas-rachford algorithm with application to non separable regularizations. In 2018 IEEE International Conference on Acoustics, Speech IEEE, 2018.
and Signal Processing (ICASSP), pp. 2886 2890.
Mark Schmidt, Nicolas Le Roux, and Francis Bach, Minimizing finite sums with the stochastic average gradient. Mathematical Programming, 162((-2)::3 112, 2017.
Shai Shalev-Shwartz and Tong Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research, 14(2), 2013.
Ziqiang Shi and Rujie Liu. Online and stochastic dooulaa--rchhfrrd splitting method for large scale machine learning. In ACML workshop on Learning on Big Data, 2016.
Panos Toulis, Thibaut Horel, and Edoardo M Airoldi. The proximal Robbins Monno method. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 83(1:1888 212, 2021.
Lieven Vandenberghe. Lecture notes of E.FF236C Optimization Methods for Large-Scale Systems, 2020.
Vladimir Vapnik.
Principles of risk minimization for learning theory. Advances in Neural Information
Processing Systems, 4, 1991.
Mengdi Wang and Dimitri P Bertsekas. Incremental constraint regjection-roxiingg methods for nonsmooth convex optimization. STAM J. Optim.(to appear), 2013.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in Neural Information Processing Systems, 33:18795-18866, 2020.
A Efficient Implementation
In this section, we introduce several efficient methods to calculate the proximal operator update in Algorithm line 5, At first glance, may seem as hard as solving the batch problem itself, but we will see that there are some interesting properties when f, involves the loss evaluation of only one or a few data points data fitting scenario. On the other hand, is not clear how parallelize SPPA with room for future work.
which leaves
17

---

Published in Transactions on Machine Learning Research (11/2022)
A.1 Least squares loss
For the least squares loss f: (y: x. the proximal update is linear least squares problem with closed-form solution prox (w) (I + ) - + mays). It can be efficiently calculated by invoking the Sherman-Morrioon formula and avoid directly inverting d x d matrix for E Rd as
x. V4
Prox (w)
> 1/²
The update rule looks like, but is not exactly the same as, the normalized least mean squares (NLMS) algorithm
(Haykin 2002). It has appeared in (Pätraşcu 2020 but include here for completeness.
A.2 Logistic (cross-entropy) loss
Consider the logistic loss fi(w) log(1 + exp( w)) with yi +1. According to the arguments made in 3 we need to solve the nonlinear equation
a yıdr exp(x w al1x;111
+ exp(x al 23
Notice that the right-hand-side is number between and =b which gives the initial upper and lowerbound on a. Furthermore, we see that a. w and ||æ;;|? only need to be calculated once, with O(d) flops for w e R each bisoction step takes constant time and ncods no more than 20 30 scalar computations render an accurale-enough solution.
A.3 Hinge loss
Regarding nonsmooth optimization problems, it turns out many of the widely used loss functions admit closed form updates. The main idea is to consider the subgradient calculus, and find the point where 0 is in the subdifferential. Take support vector machine (SVM) as an example, in which the loss function is the hinge loss f(() = [1 y. +- Its subdifferential
(0}
yar w 0,
(f:(w) = [-
mawl 0,
ayya 00 <a
y:or w 0.
Added with the gradient of the proximal term and letting 0 to be element of the subdifferential set, we get. the update rule
1
y: w > 1, y w. otherwise.
Prox Bonn,,(()) (w) =
+ Aayia
A |x
w + Aayaa, 1-
An interesting observation here is that it looks like the perceptron algorithm with adaptively chosen step-size.
A.4 Absolute error loss
Robust regression using absolute error loss fi(w) = |y - aTwwl, another example to nousmooth convex loss function. The derivation similar that SVM: the subdifferential for the one-sample loss is
{x.}}
Vt x. w < 0,
0f((w) =
x! > 0,
(am:
1 11
Di x.
18

---

Published in Transactions on Machine Learning Research (11/2022)
After adding the proximal term, the update rule is
X
yb x. w - ->/122111?
Proxs,(w)
w + xæ;
x. > >|;3;
+ Vi v--w:w
otherwisc.
|
A.5 Generic losses
Even for generic nonlinear programming problems, it is still possible to efficiently evaluate the proximal update beyond merely simple gradient step. The idea is to apply the Amited-memory BFGS algorithm (Nocodal & Wright 2006), or L-BFGS for short. L-BFGS is a memory-cficciet implementation of the famous quasi-Newton algorithm BFGS. In a nut shell, L-BFGS is an iterative algorithm that makes use of the second-order information from the optimization loss function, but does not require solving matrix inverses and only requires explicitly evaluating first-order gradients. For a prescribed number of iterations, it requires one matrix-vecti vector multiplication and multiple vector multiplications. As result, the overall complexity is again ((d) if the initial guess of the Hessian matrix is diagonal.
While there are certain limitations for applying L-BFGS to general nonlinear programming problems, we reckon that it fits perfectly in the context of SPPA implementations.
L-BFGS has to specify a good initial guess of the Hessian approximation matrix. While in many cases people simply use the identity matrix to start, it may result in very poor approximation. Fortunately, thanks to the proximal term, the identity matrix is in fact a very good initial guess for the Hessian matrix for SPPA updates.
In order to save memory consumption, L-BFGS has to prescribe the number of iterations before rumning the algorithm. Obviously, if the prescribed number of iteration is too large, we incur unnecessary computations, while if it is too small need to invoke another round with new estimated Hessian matrix. However, again in the context of SPPA, the proximal term naturally provides a good initialization wy Our experience show that prescribing 10 iterations of L-BFGS updates is more than enough to obtain accurate solutions.
On the other hand, it perhaps makes more sense to simply use some more advanced first-order methods such as Nesterov's accelerated gradient descent (1983) to calculate the proximal update. Both L-BFGS and accelerated gradient descent evaluates the gradient of the loss function with (dd complexity, and the question is how to leverage convergence rate versus sophistication.
B Miscellaneous proofs
B.1 Any diminishing step size rule guarantees expected convergence
It is our observation that lot of the convergence analysis using diminishing step sizes requires that the step sizes are square summable. Take equation 2 as an example, on the right-hand-side, the second term in the numerator typically assumed to be finite, while the denominator goes to infinity, therefore the entire right hand side goes to zero. This requirement would rule out step size rules such as X = 33/vi since it is not square summable. The purpose of this subsection is to show that the common square summable requirement is not in fact necossary as the quotient still goes to zero. We not claiming to be the first notice this, it is only included for completeness.
Proposition B.1. For a nonnegative sequence A2,.., that satisfies de + 0 and E d = 00, we have thot.
Σ X2
0.
-t=1 A
x
19

---

Published in Transactions on Machine Learning Research (11/2022)
Proof. Let > O, since d + there exists an integer T1 such that d < for all t>> Tr There also exists
integer 72 such that
To
T
X
1=1
1=1
since the right- saatt-hand-idee is fixed number and
Et=1
d
Then for any T max(Ti,T2), we have
Σ x
Ti
X
Σi=T++ d
1=1
E A
Σ
T2
d
L1=1++1 At
E1=1 de Σ Di=T++1
21
1=
21
X
Ei=T++
A
c ci d
Sct
S 2c.
t= >
Ei= Ae
Σi A
Since the inequality holds for any > 0, we can let be arbitrarily close to zero, and there will be T large enough to make the inequality hold. This proves the proposition.
B.2 Convergence in probability
In this convergence analysis, will use the following well-known theorem [Bertsekas 2011
Theorem B.2 (Supermartingale Convergence Theorem). Let Xt, Y,, and Ze, = 0, 0,..... be three sequences of random variables and let F,, = =,,1.... be sels of random variables such that F. C Fet1 for all t. Suppose that:
1. The random variables Xx, Y,, and Ze are nonnegalive, and are funceiios of the random variables in Ft.
2. For each we have
F[X++1| S X, Y2 + Z4-
There holds, with probabilily 1, Ei= Zo 80.
Then we have Ei Y, s co, and the sequence X; converges to nonnegative random variable X, with probability 1.
Corollary B.3 (Convergence in probability with constant step sizes). Suppose all fi...., In and g are conver, and Assumption 2.1 holds. solution w, exists, then with inilialization wo, with probability the sequence wi....; wr generated by SDRS (Algorithm D mini-bou botch SDRS (Algorithm 2 with constant step size d = À satisfies
inf
fa(w.) + +((w)) <
1
fi(w.) + +.(ww) (...) + AL2
2
Proof. Rewriting g as
E; |10+11 - w. 2 < [; -
w.|l2 -w,
21
-
1
Σ
f((w++1) + g(w,+1) Σ
f((w.) - -(ww.) + +2212
i=1
i=1
Let X, =1 1= 2, 7x = 0, and
Y, = 2A
Σ
fi(wa+)) + g(w++1)
Σ f((.)) -((.)))((u))) p(uc.) (((c.)) X2I2
well as filtration F. [1/+1, Wo, w.
Do). which would imply F, C Fe+1 then according to the
Supermartingale Convergence Theorem
B.2
Σ t=1
21
1 n
f((w)+1) + g(w(+1)
Σ f((w.) - g((w)) X2I22 < <0,
20

---

Published in Transactions on Machine Learning Research (11/2022)
with probability 1. Divide both sides by 2A gives,
1
AL2
fi(w:+1) + g(we+11)
(.w.)) g(w,)
<00.
i=1
Since this infinite sum is finite, must have
AL2
inf fi(w++1) + g(wr+1)
fi(w.)) g(w,)
I 0.
This is because if the infirmum is strictly positive, say e, then the infinite sum would be bigger than times infinite, which contradicts the summable conclusion from the supermartingale convergence theorem. ]
Corollary B.4 (Convergence in probability with diminishing step sizes). Suppose all fi....: fn and g are convex, and Assumption 2.1 holds. If solution w, exists, then with inétialization wo, with probability the sequence wi,..., wr generated by SDRS (Algorithm 7 mini-batch SDRS (Algorithm) 3 with diminishing slep sizes such that X Et d 00, and EE < oo salisfies
inf
f(m)) g((wi)
S((w)) + g(w.).
Proof. Following the first few steps in the proof Corollary B.3 but replacing with de we have
1 n
Ei, ||00411 - w.112 S ||00 w.11 2 2hc
11 Σ
f((w)+1) + g(w++1)
Σ n
f(lw)) - + X8L2.
This time we let X1 = ||00; w.ll? Ze = 22 and
Js (( + ((w++))) 1 Σ fi(w.)) -((wr))
== 2d:
well filtration F, [We+1....- We: ....... w Since assume de is square summable, invoke
the Supermartingale Convergence Theorem B.2 to conclude that
f(( g(w(+1) 1 _ filw)) -((w))--le -g(w.) 3o
Σ 2Xc 1=1
1=1
with probability 1. This means
inf Σ f((w+11) + g(w(+1)) Σ f((w)) - -g(ww)) = 0:
otherwise, if the infiruum is 0, then the infinite bigger than E d co, since we assume A, is not summable, which contradicts the conclusion from the supermartingale convergence theorem.
C Additional Experiments
C.1 Classification on IMDB
IMDB large movie review data used to analyze binary sentiment (Maas et al. 2011). We have 25,000 movie reviews for training and 25,000 movie reviews for testing. To conduct our experiment, we used a bag of words format. Positive to negative sentiment. ratio is 50/50.
21

---

Published in Transactions on Machine Learning Research (11/2022)
1.0
80
Adam
Acala
75
0.8
Accuracy
70
0.5
60
55
Adan
0.3
125
200
50
75
100
125
150
175
200
Seconds
Seconds
Figure 11: SVM with L1 regularization on IMDB:
Figure 12: SVM with I regularization IMDB:
loss per seconds
classification accuracy per seconds
11
85
80
0.9
WMM MMM
Accuracy
Mil Homoroum
0.6
AdaDelie
0.4
93
50
Seconds
Seconds
Figure 13: Logistic regression with Li regularization on IMDB: loss per seconds
Figure 14: Logistic regression with L1 regularization on IMDB: classification accuracy per seconds
We use SVM and logistic regression for LMDB classification, as we did for Bank Note Authentication. SDRS outperforms the other algorithms when we use logistic regression to classify. Surprisingly, SGD Moment diverges with this data set in the logistic regression problem. The experiment shows that SGD Momentum is less stable than SDRS, despite performing similarly in most cases. All algorithms are tested using different. parameters, and the results presented here are the best results for each algorithm. The logistic loss is shown in Figure 13 and the accuracy per second is shown in Figure 14 Figure 13 shows that logistic loss decreases with SDRS, whereas it does not decrease quite as much with other algorithms. Furthermore, SDRS achieves a higher test accuracy than other algorithms in Figure 14. The performance SDRS is similar but not notably better than the state-of-the-art methods when we SVM.
C.2 Regression
We perform linear regression with L1 regularization and show the results of different algorithms including our efficient implementation both Bike Sharing and Hotel Average Daily Rates (ADR) datasets (Antonio
ct al.
2019
Bike Sharing. Bike Sharing is publicly available data set containing hourly daily count of rental bikes as well as environmental and seasonal settings. The data set we are using is from UC Irvine Machine Learning Repository. The core data set consists of two years of historical logs for years 2011 and 2012 from the Capital Bike share system, Washington, DC, USA (Fanaee-T & Gama 2013). There are 17379 samples in this dataset (12512 training samples, 3476 test samples, 1391 validation samples). with 16 features predict the hourly rental bike count.
https://archive.ft htt.://www........aaaaaaaaaaaaaa
22

---

Published in Transactions on Machine Learning Research (11/2022)
175000
XORA
0.16
150000
0.34
125000
100000
MSE Loss
0.06
50000
0.04
25000
0.07
0.0
Seconds
Seconds
Figure 15: Linear regression with 1 regularization on Bike Sharing: mean squared error loss per seconds.
Figure 16: Linear regression with L regularization on Hotel ADR: mean squared error loss per seconds.
Adre
0.36
100000
SGD Notaeal
AAD NarWERA
80000
Loos
0.00
0.06
20000
3.04
0.02
Seconds
Seconds
Figure 17: Linear regression on Bike Sharing: loss per second, batch size
Figure 18: Linear regression on Hotel ADR data set: loss per second, batch size 10
On the bike sharing data set, we perform linear regression with L1 regularization and compare the performance of different optimization algorithms. Figure 15 illustrates the decrcase in the loss values for different algorithms over time. The SDRS algorithm outperforms the other algorithms by reaching a smaller value in a shorter amount of time. We tried different settings for the algorithms, and the results presented in Figure [5 show the SDRS learning rate to be 1/Vi, where t is the number of iterations. Similarly, SDRS also offers the benefit of being stable, as demonstrated in Figure 15 Since the dimension is very low for this data set, SDRS is not significantly faster than SGD or SGD momemtum. With mini-batch, as shown in Figure 17 that SDRS takes fewer number of iterations to reach smaller loss value compared to other methods.
see
Hotel Average Daily Rates (ADR). Hotel ADR is publicly available data set consisting of hotel demand data for two different types of hotels; resort hotels and city hotels. Each hotel data set consists of 40,060 samples with 31 features each to predict average daily rate values (Antonio et al. 2019). Tn our experiments, we used resort hotels with 40,060 samples (30045 training, 10015 test.) with 8 fcatures.
We perform linear regression with L, regularization the Hotel ADR dataset and compare the performances of different optimization algorithms. Figure 16 shows how loss values have decreased over time for multiple algorithms. It takes SDRS less than a second to converge to very small loss value. We tried different settings for the algorithums, and the results are represented in Figure [6 where the learning rate for SDRS is 1/t, where is the number of iterations. With mini-batch, as in Figure 18 we can see the decreasing loss value over iterations. SDRS outperforms all the mothods and provides an almost negligible error bar. The performance can be further showcased using parallelization over different batches. Nevertheless, the current. results good enough to celebrate.
23

---

Published in Transactions on Machine Learning Research (11/2022)
80
Adaoree
2.0
Plonentur
70
9012
Cross Entropy Loss
60
Accuracy
50
0.5
SPRA-GGO AGO
Adben
2000
4000 6000 8000 10000 12000 14000
2500
5000 7500 10000 12500 15000 17500 Update
17500 20000
Stochastic Update
Figure 19: ResNet CIFAR10: cross entropy loss,
Figure 20: ResNet CIFAR10: prediction accuracy, batch size 32
batch size 32
100
80
Cross Entropy Loss
(1))
Accuracy
60
Aeleeed Adapted ZCO
Alhorrd
1.0
520
0.5
20
0.0
200
400
600
200
600
800
Update
Stochastic Update
Figure 21: CNN MNIST: cross entropy loss, batch
Figure 22: CNN on MNIST: prodiction accuracy,
size 64
batch size 64
C.3
Additional neural network training
To show that mini-batch SPPA works not only on the CNN architocture, we also tried training a residual neural network (ResNet) on CIFARIO. For the details about the network, one can refer to the 20 layer ResNet architecture on CIFAR10 (He al. 2016). In our settings, the batch size is 32. The results are shown the CNN case.
Figures 19 and 20
As we can see, the performance is indeed consistent
Finally, we show the training performance the MNIST handwritten digits data set (LeCun et al. 2010 It consists of 60,000 training, 10,000 test images of size 28 x 28 in 10 classes. The network we used for this experiment is based on convolutional deep neural network. It consists of 5 layers, each of the first 3 is a combination of 3 x 3 convolutional filters and x 2 max pooling with a stride of 2 and the last two fully connected. The loss function is cross-entropy loss, and the activation function is RELU, The batch size 64. To show the behavior of the algorithms more in detail, we recorded the loss values at every 25 stochastic update, the experiment run for one epoch, and in total, 38 updates are presenled (25 38 x 64 giving the number of training samples). The accuracy is calculated on the test data at every 500 stochastic update, using the accuracy calculation in PyTorch tutorial. As observed in Figures 21 and 22 SPPA-based methods outperform all SGD-based algorithms in terms of both the cross-cntropy loss and prediction accuracy on the test set.
24


Reviews:
[{"review_id": "c9NpO9vFjQ", "reviewer": "Reviewer_D6YK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe authors provide a solid analysis for the SDRS method in the single same setting. The proof is clear and correct. \n\nWeakness:\nThe presentation of the mini-batch SDRS method is not clear:\n1. How are $\\tilde w_t^{(k)}$'s initialized?\n2. In line 2 of Algorithm 2, a mini-batch is used, how is this different from the mini-batch freshly sampled in line 3? The conditional expectation used in the analysis requires that $w_{t+1}$ to be independent of the randomness in line 3. So I guess the mini-batch in lines 2 and 3 are different. If so, the notation needs to be changed as it is currently misleading.\n2. Do we need to maintain $\\tilde w_t^{(k)}$ for $k \\in { 1, \\ldots, n }$? If not, what is $\\tilde w_t^{(k)}$ in line 5?\n3. The proof of Theorem 2.7 is incomplete as the authors claim that the omitted part resembles the proof of Theorem 2. I tried to check this but it does not go through trivially. Please provide a complete proof.\n4. In Theorem 2.7, the convergence of mini-batch SDRS is exactly the same as the one of SDRS. In that case, what is the point of using a mini-batch of samples per iteration? I would expect certain speed-up that depends on the batch size $p$. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the stochastic Douglas-Rachford splitting (SDRS) for general empirical risk minimization (ERM) problems with regularization. It is proved that when a single sample is used per iteration, the convergence rates of SDRS are $O(1/\\sqrt{t})$ and $O(1/t)$ under convex and strongly convex settings, with appropriate choices of diminishing step sizes. The authors further study the convergence of mini-batch SDRS where a mini-batch of $p$ samples are used per SDRS iteration and show that the convergence rate stays the same. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please address my concerns listed in the weakness section.", "broader_impact_concerns": "This is a theoretical paper, so I do not see any concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aDdmBJZAFl", "reviewer": "Reviewer_toTG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-The problem and the contributions are very clear. \n\n-The convergence rates are exactly the ones that one would expect. There is no surprise here. The rates recover the rates of the stochastic proximal point as a special case.\n\n-Section 3 provides an interesting discussion of application of SDRS in ML. In particular, the equation after Eq 15 reminds me the difference between two of the most famous signal processing algorithms: LMS (which actually corresponds to SGD) and normalized LMS (which corresponds to SPPA). However, I think that Section A is covered in www.proximity-operator.net\n\n\nWeaknesses: \n\n- The paper does not do a good job in the literature review. In particular, there are few comments which are not appropriate for a research paper in my opinion. Examples:\n \"This means that the correct way of doing mini-batch for SPPA should be Algorithm 2 with line 2 being the average, which is not the same as any of the proposed mini-batch methods in (Chadha et al., 2022).\"\nWhy would your algorithm be the correct one and the other not being correct? At the stage of the paper, the authors haven't explained what Algorithm 2 is. In particular, does Algorithm 2 corresponds to any known algorithm in the deterministic case? I suspect that, in the deterministic case (i.e, n=1), Algorithm 2 is equivalent to Douglas Rachford applied to a lifted problem. Or maybe it is a known primal dual algorithm that can handle a consensus constraint? By the way, is Algorithm 2 converging with a constant step if n=1?\n\"The latter two [papers] are either preprint or abridged work without full convergence analysis\"\n\"The latter two\" papers seem to be the most related papers to this one, i.e. they are the papers which considered SDRS previously. Therefore, it is not fair to rule them out this way, even more in a section which is not dedicated to related work. In a related work section, one would need to compare the results, this is what I would have expected. Moreover, the way they are ruled out is incorrect: with a small research effort, I found out that Shi & Liu was published in ACML workshop on Learning on big data 2016, and that the full convergence analysis of Salim et al is in the supplementary material available on the website of the author for example.\n\n-Why is Assumption 2.1 (Lipschitzness of all f_i) needed, since this is not needed in the deterministic case? DRS is a proximal method, it should rely on milder assumptions than subgradient methods because we have to compute proximity operators instead of subgradients, which is more computationally demanding. Otherwise, what is the point of using a stochastic proximal method compared to a subgradient? Finally, I don't think that \"this [Lipschitz] assumption is so mild that most functions do satisfy it\". Because of this assumption, the authors cannot handle strong convexity in the stochastic term F.\n\n\n-[COMMENT IRRELEVANT FOR PUBLICATION IN TMLR]: The contributions might be of limited interest to the ML community. I think that the theoretical gap closed in this paper (the convergence rates) remained a gap mostly because the community has a limited interest in this problem. Although the analysis is quite standard, it is quite clean, and it is nice to see a paper presenting the convergence rates of SDRS with the correct values.\n\n\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the stochastic Douglas Rachford splitting (SDRS) algorithm to minimize F+g, where F is a sum of n terms f_i. They consider two variants of the algorithm. One vanilla SDRS where a randomly chosen term of the sum is considered per iteration, and one minibatch version which works on a lifted space (p sequences of iterates, where p is the minibatch size).\n\nThe main contribution is to establish convergence rates in the convex case and in the case where g is strongly convex. The paper always assumes that f_i is Lipschitz.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "MAJOR:\n\n-What is Algorithm 2 with n=1? It should a known primal dual algorithm applied to the consensus constraints.\n-Add a related works section, and discuss the results of concurrent works\n-Adjust the rhetoric (the points that I mentioned above). \n\nMINOR:\n\n-Change the title of Section B.1. What is the point of Section B.1?\n-Page 2: \"Suppose we consider p samples in each stochastic step, not only is the complexity be in general p^2 d + p^3, but [...]\" Could you explain this statement please?\n", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aimvje6lcJ", "reviewer": "Reviewer_oBE3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n1. The setting studied in the paper is relevant: implementing stochastic proximal updates is in many cases tractable (as the authors show), and not much work has been devoted to investigating accuracy guarantees for these types of oracles. \n2. The analyses in the paper are quite natural, so they can be useful for further work on the topic.\n3. Numerical experiments are very encouraging.\n\nWeaknesses:\n1. The main weakness of this paper is the lack of quantitative improvements in complexity, compared to more classical results. For example, Theorem 2.2 shows exactly the same convergence rate as that of SGD. In fact, I believe that the stochastic proximal gradient method would also have the same convergence rate under the same assumptions made by the authors, but please correct me if I am wrong. The natural question is then why one would prefer the proposed methods of this paper. Further, mini-batching usually would boost the rates by its variance reduction, but this is not at all reflected in Theorem 2.7. Somehow, while it is nice that the results of the work reproduce existing rates in the literature, shouldn't one expect better theoretical results from these stronger oracles?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides analyses for finite-sum (randomized, incremental and mini-batch) variants of the Douglas-Rachford splitting method. Assuming convexity and Lipschitzness of the individual functions in the finite sum, these methods attain rates $O(1/\\sqrt T)$ and $O(1/T)$ for the convex and strongly convex cases (regarding the regularization term) respectively. The paper also provides some numerical experiments, where the proposed algorithms compare favorably to existing alternatives in the literature.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. The guarantees of the algorithm are stated for the best iterate, which seems inconvenient, as it would be too expensive to implement. I recommend instead stating the accuracy guarantees for an iterate selected at random (which would lead to entirely analogous upper bounds).\n2. In the proof of theorem 2.2 (as well as the other convergence proofs of the paper), there is one step I couldn't understand. This is the equality before eqn. (5): $=\\frac{1}{\\lambda_t}(\\tilde w_{t+1}-\\tilde{w}_t)^{\\top}(w_{\\ast}-\\tilde{w}_{t+1})$. In particular, for this equality to hold, the last term of the preceding equation must be zero, which I couldn't follow why it is true. So please add a comment about this step.\n3. In the proof of Theorem 2.7, at some point it is said that the rest of the steps follow analogously as in the proof of Theorem 2.2. I would encourage the authors to: (i) Improve the analysis to leverage variance reduction for faster rates, and (ii) add the details of the rest of the proof.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers are now satisfied with the revision of the paper. I am also satisfied after seeing that my requested changes, and that of the three reviewers, have been implemented in the revision. Including precise pseudo code of the algorithms, a more practical output (the average iterate) and a proper literature review. As such, and given these methods are of interest to some of TMLR's audience, I now recommend the paper be accepted as is.
Decision: Accept (poster)