Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimization
Authors: Liang Zhang
Kiran Koshy Thekumparampil
Sewoong Oh
Niao He
Venue: NeurIPS 2022 Accept (2022)

Abstract: We study differentially private (DP) algorithms for smooth stochastic minimax optimization, with stochastic minimization as a byproduct. The holy grail of these settings is to guarantee the optimal trade-off between the privacy and the excess population loss, using an algorithm with a linear time-complexity in the number of training samples. We provide a general framework for solving differentially private stochastic minimax optimization (DP-SMO) problems, which enables the practitioners to bring their own base optimization algorithm and use it as a black-box to obtain the near-optimal privacy-loss trade-off. Our framework is inspired from the recently proposed Phased-ERM method [22] for nonsmooth differentially private stochastic convex optimization (DP-SCO), which exploits the stability of the empirical risk minimization (ERM) for the privacy guarantee. The flexibility of our approach enables us to sidestep the requirement that the base algorithm needs to have bounded sensitivity, and allows the use of sophisticated variance-reduced accelerated methods to achieve near-linear time-complexity. To the best of our knowledge, these are the first near-linear time algorithms with near-optimal guarantees on the population duality gap for smooth DP-SMO, when the objective is (strongly-)convex--(strongly-)concave. Additionally, based on our flexible framework, we enrich the family of near-linear time algorithms for smooth DP-SCO with the near-optimal privacy-loss trade-off.

Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimization
Liang Zhang
Kiran Koshy Thekumparampil Amazon Search & UIUC chekump2e1llinoss.coo edu
ETH Zurich & Max Planck Institute liang. 2ananghhannnnnn www.f..th.ccco
Sewoong Oh University of Washington sewoong8cs. washington edu
Niao He
ETH Zurich niao.he@i wwwinffffff.... wew.ttt.c.. ch
Abstract
We study differentially private (DP) algorithms for smooth stochastic minimax optimization, with stochastic minimization byproduct. The holy grail of these settings is to guarantee the optimal trade-off between the privacy and the excess population loss, using algorithm with linear time-complexity in the number of training samples. We provide a general framework for solving differentially private stochastic minimax optimization (DP-SMO) problems, which enables the practitioners to bring their own base optimization algorithm and use it as a black-box obtain the near-optimal privacy- trade-off. Our framework is inspired from the recently proposed Phased- Phased-ERM method [22] for nonsmooth differentially private stochastic convex optimization (DP-SCO), which exploits the stability of the empirical risk minimization (ERM) for the privacy guarantee. The flexibility of our approach enables us to sidestep the requirement that the base algorithm needs to have bounded sensitivity, and allows the use of sophisticated variance-reduced accelerated methods to achieve near-linear tome-complexity. To the best of our knowledge, these are the first near-linear time algorithms with near-optimal guarantees on the population duality gap for smooth DP-SMO, when the objective is flexible framework, we enrich the family of near- near-linear time algorithms for smooth DP-SCO with the near-optimal privacy-loss trade-off.
Additionally, based on our
1
Introduction
Machine learning models are nowadays trained using large corpora of data samples collected from many different entities, e.g., from users of large software service 28). However, has been empirically shown that these models can be exploited to reveal private information about these contributing entities. For example, Carlini al. B attacked the large language model, GPT-2, to reveal hundreds of verbatim text samples used to train these models. These attacks violate the privacy of the contributing entities, but naturally they expect that no private information about them can be revealed through these models. Over the last decade, this expectation such as GDPR EU [[5]. There are many mathematical frameworks formalizing this expectation of privacy, but the most widely accepted one is that of Differential Privacy (DP) [1]]. With high probability, models satisfying DP cannot be attacked by any adversary to identify that a particular training sample used in its training. Hence, DP provides any entity plausible deniability that they contributed to the training set. However, optimization algorithms for training such models under DP require careful design choices to ensure privacy while preventing the degradation of convergence
even legislated into laws
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

speed and data efficiency. This has led to the burgeoning field of differentially private optimization
algorithms 6 [22] which considers stochastic convex minimization as the canonical problem.
For differentially private stochastic convex optimization (DP-SCO) with (e, 6)DD guarantees, the optimal excess population risk is ( dl log(1 /5) (1n²²)nn)))) for p-strongly convex objectives, where n is the number of participants and d is the dimension of the variable. If the objective is also smooth, this can be achieved with linear-time (e, 6 algorithm 22 using O(n) stochastic gradient evaluations. This analysis critically relies on the concept of algorithmic stability 12.. which measures how much the population loss of an algorithm's output changes when a single data point in the input dataset is perturbed. This is also known as sensitivity of an algorithm in DP, which determines how much noise needs to be added in order to achieve ()-D) (see Definition D. We will use stability and sensitivity interchangeably when referring to optimization algorithms. As the sensitivity of stochastic gradient descent (SGD) known 124. Feldman et al. 22 were able to add an appropriate amount of optimal risk in linear time. However, such tight analysis of sensitivity is generally intractable for more complex optimization routines that practitioners might want to use. Further, the stability analysis of SGD only holds when the smoothness parameter E of the problem is upper-bounded by O(m)). In the first part of the paper, we show that we can achieve similar guarantees using a wider choice of, potentially more practical, "base" algorithms, without any restrictions on the smoothness parameter.
to this sensitivity to an SGD--based algorithm to achieve
On another front, many emerging practical machine learning applications are formulated as stochastic minimax optimization problems, e.g., generative adversarial networks [3] adversarially robust machine learning (35), and reinforcement learning [[6]. Designing DP algorithms for solving these minimax problems is of paramount importance. For example, private generative adversarial networks provide promising new direction to synthetic data generation 47 such as in networked time-series data [32). Motivated by such applications, we study the differentially private stochastic minimax optimization (DP-SMO) problem of the form:
min max F(x,y) A (elf(a, ZEX yEy
where the objective [(x, vi {) is smooth and convex-concave for any random vector &, and we given access to n i.i.d. samples [- In contrast to DP-SCO, where linear-time algorithms have been proposed to achieve optimal risk guarantees, existing private algorithms [ 10 for DP-SMO achieving optimal guarantees have time-complexity which scales super-lineary in the number of samples n (summarized in Table D. In the second part of this paper, we close this gap by introducing a new class of output perturbation- based DP algorithms for both strongly-com connve--trroglly and convex-concay settings, which can achieve near-optimal population risk bounds using ((n) stochastic gradient computations, where Ô hides logarithmic factors.
One of the main bottlenecks for (i) using wider variety of algorithms solve the DP-SCO problem, and (ii) solving the DP-SMO problem using linear-time algorithms, is the lack of known stability results for fast and sophisticated non-private algorithms, such as variance-reduced and accelerated methods. One can sidestep the sooorthh-sspciife stability requirement by utilizing the stability of the optimal solution to a strongly-conve -onveex and Lipschitz empirical risk minimization (ERM) problem
42 I.. Particularly, if the output of an algorithm for such an ERM problem is close enough to its empirical solution, then the sensitivity of the algorithm is automatically guaranteed. Exploiting this observation, Feldman et al. 22 proposed the phased-ERM algorithm for nonsmooth DP-SCO, which solves a series of strongly- convex ERM subproblems to sidestep the stability analysis of SGD for nonsmooth functions. This algorithm achieves a quadratic time-complexity for nonsmooth DP-SCO.
We observe that when the problem is additionally smooth, or when it allows smooth minimax reformulation 37 there exist plethora of fast algorithms to solve the resulting ERM subproblems. Combining these algorithms with (phased) output perturbation gives rise to a class of near-optimal pear-lineat comelccoyyyxity private algorithms for both smooth DP-SCO and DP-SMO problems, without any additional effort on their stability analysis. Our contributions are summarized below:
We introduce a flexible framework for solving smooth DP-SMO and smooth DP-SCO problems, utilizing the (phased) output perturbation mechanism. The black-box framework enables to bypass the need to prove algorithm-specific stability and transform off-the-shelf optimization algorithms
We only claim near near-optiman optimaliyy of our rate because of its dependence on the condition number or additional logarithmic terms. We call algorithm line-r time when it is linear-timg up some logarithmic factors.

---

Table 1: Among (s, 6D-DP smooth minimax algorithms that achieve the near-optimal utility bound on the population strong duality gap for the μl strongly-convex werex---tooggycooee concave case and the population weak duality gap for the convex-concave case, the proposed framework achieves the best gradient complexity. Here O hides logarithmic terms, É is the smoothness parameter, L/ and d max da dy). nlower-bounde are also summarized in the table.
Settings
Lower-bound
Algorithm
Utility
Complexity
SC-SC
1 p,
+
de (16( Ha
Thm. 4.3
o
6²
x
Ö(n + Vik)
pm
DP-SGDA 49
+
(dlog(1/6)
O VE)
vn
(dlog(1/8)
NSEG
1
+
O(n²)
C-C
2 a(th +
1 1og(2/6) ne
vn
n1
NISPP
1
+
(1/6
Ö(13/2)
vn
Thm. 4.5
vn +
1
d d'log( 116)
(()
into DP algorithms with optimmllimall guarantees. This is attractive as there are currently no stability algorithms for both SMO and SCO.
analyses for accelerated 2 39 and variance-reduced [26] 39
Using the framework, we provide the first near-linear time private algorithms for smooth DP-SMO with near-optimal bound on the population duality gap, under both and convex-concave cases (see Table Among other things, this implies that if a (primal) nonsmooth minimization problem can be reformulated problem, then we can solve it in near-linear time instead of the best known super-linear time 4 30) In prior work, near-linear time DP algorithms with optimal rates for solving nonsmooth convex objectives only existed for generalized linear losses 19.
smooth convex-concave minimax optimization
The framework also enriches the cohort of near-lineat time near-optimal private algorithms for smooth DP-SCO settings, which only contained SGD previously. Moreover, existing optimal DP algorithms for smooth DP-SCO rely on stability result of SGD,, which only holds for restricted range of the smoothness parameter 24). Such restrictions are avoided our framework.
Differentially private optimization has been an active research field for the past few years, and early works focused on the empirical problems 66 46 perturbation used in this paper, many existing works applied gradient perturbation to guarantee privacy. This method adds noise to each iteration of the algorithm and then applies moments accountant M advanced composition P to analyze the overall privacy. Although gradient perturbation does not need the smoothness convexity assumptions of the function and works for most iterative algorithms, requires larger mini-batches 7 or longer training time X if one resorts to the privacy amplification via subsampling [TO] to reduce the DP noise, resulting in super-lineat gradient queries in existing methods 7 49). Here we mainly review previous results that achieve the optimal utility bound
Related Works:
50 44 In addition to the output
0(1/V7 n2 + dd log((1/))/ on the population loss-occosee according to a lower-bound in Bassily et al.
6 when solving d dimensional DP-SCO problem using samples with (c, 8DP guarantees.
In the smooth convex case, Bassily et al. 7 is the first to derive optimal rates for DP-SCO with complexity (n3 VE) by gradient perturbation and stability of SGD 24 Feldman al. [2
provided two SGD-based linear- time algorithms: one uses privacy amplification by iteration [2] that only works for contractive updates; the second one uses phased output perturbation and stability of SGD. In the nonsmooth convex case, Bassily et al. [ established the stability of SGD for nonsmooth functions and obtained quadratic-ting time algorithm. Feldman et (2 leveraged the stability of ERM solutions this phased framework, Asi et al. 4 and Kulkarni et al. (00 used gradient perturbation and improved the complexity to O Omin (min(n332 n² /d)) and Omin (n/ /23/2/d/// respectively; Asi et al.
42 and achieved O log(1 1 ()))) complexity with phased output perturbation. Based on
B introduced hypothetical time algorithm assuming the existence of low-biased estimator.
3

---

Moreover, Bassily et al. [ gave near-linear time algorithm for nonsmooth convex generalized linear losses using phased SGD with smoothing techniques [37
To the best of our knowledge, only few papers studied DP-SMO and all of them used gradient perturbation to guarantee privacy. Boob and Guzmán [11] analyzed stability of Extragradient [%] and proximal point methods [4 for smooth convex-concave functions, and their DP versions run with time On²)) and (n1//2 respectively. Yang et al.
4 used the stability of SGDA 31 and obtained
DP algorithms with complexity (nn Vε) for the smooth convex-concare case and (n²²) for the
nonsmooth case, mirroring the guarantees of SGD for DP-SCO 7 g Kang et al. 29 only focused on stability and generalization analysis of gradient perturbed SGDA and provided high-probability results, See Table | for a brief comparison.
The idea of using the stability of ERM for smooth convex optimization has also been exploited before. Attia and Koren [5] developed black-box framework for smooth convex objectives that produces uniformly -stable algorithms while maintaining fast convergence rates. Lowy and Razaviyayn [3]] considered DP-SCO by output perturbation but only provided lear-linear time near-optimal algorithms for smooth strongly-convex losses. For smooth convex case, they achieved a sub-optimal rate O(1/Vn + (vdllo(11/) (22)))/3) in near-linear time. In the setting of smooth strongly-conve (strongly-Jconean)
DP-SMO, they directly utilized its DP-SCO reformulation in the primal function to obtain the final guarantees. Therefore, their utility bound is simply on the primal risk, which is weaker than the primal-dual gap considered in this work. The algorithms run in near-linear time for the strongly-concave case and super-linear time (n25)2 for the concave case. In contrast, provide near-optimal near-linear time algorithms in all aforementioned settings.
Notations: We use ||-II for the Euclidean norm of vector and ||| for the absolute value or the R + R is I. L--iipscitz if (9(x1) - ( < L|[z1 xz for æi, 22 in the domain of g. A function h : Rd + IR is E-smooth if it is differentiable and h(x2) $ h(z1) + Vh(x1) (I2 21) + (e/2)][z1 - x2
cardinality of a set. A function
A function p Rd + R is convex if p(az1 + (1 - 2)22)= S ap(z1) + (1 a)p(22) for all a E [0, 1], and is p-strongly convex if p(z) (14/ == is convex with > 0. A function q : IRd -> R is concave -q is convex and j--ttronly concave if qq is j--ttronlyy convex. For a vector x E Rd the notation x + (0, o²Ia) means + for a random vector ~ (0, Ig)) sampled from the Gaussian distribution.
2 Preliminaries
We first provide some background on differential privacy and stochastic minimax optimization.
2.1 Differential Privacy
Differential Privacy (DP), introduced in Dwork et al. [1], measures privacy leakage of an algorithm.
Definition 1. For two datasets S = {&r) and S' = (C)) we say the pair (S. S') is neighboring if max (IS S' |s' \ $|} = and we denote neighboring datasets with S ~ S'. For an algorithm A and some privacy parameters
0 and ô E (0, 1), we say A satisfies (s, -ddifferrett privacy if ((A(S) E A) I F(A(S") E A) + for all ~ S² and all subset in the range of A.
In this work, we focus on the settings when & E 10, and 6 E (0, 1/n) given dataset of size n. Sensitivity is important concept that makes designing (s, 6DP mechanisms straightforward.
Let A be some randomized algorithm operating on S and outputting a vector in Rd A has sensitivity A sups-ss [|A((S) A(S')) with probability at least 1 6, then the Gaussian mechanism outputs A(S) + (O, (AAV 2 (1g5//2/ 25) and achieves (s, -DD
Definition 2.
The following basic composition rule of differential privacy will be used in the analysis.
Lemma 2.1. Ai is (e,,8i)-DP and Az is (E2, 82) then A2) is (ei + E2, 61 + DDP ATZY. For a sequence of interactive algorithums ( each satisfying (ER, dDP and operat- ing on a subset Ss, if S& 's are disjoint then the composition A2 .............................. AK (k())) is (maxke[k] Ek, maxxe(K] 6R -DP (known as parallel composition in McSherry (3)))

---

Algorithm Output Perturbation for Strongly-Convex Minimization
Input:
Dataset {E:)==>> algorithm A, DP parameters (e, strong strong-convexity parameter p. Run the algorithm A on the smooth strongly-convex finite-sum problem mingex Fs(x) (1/n) Σx=1 fx; {,) to obtain the output A(S) such that [[A(S) -'sll (/mm) with proba- bility least 8/4, where 2'S- = arg minge Fg( the empirical optimal solution. Output: z- = A(5) + NN 12 with o =
(une).
2.2 Stochastic Minimax Optimization
The stochastic minimax (a.k.a. saddle point) optimization problem has the form:
min max F(z. y) a Eglf(xr E)),
(1)
#EX yEy
where the population-level expectation of the stochastic continuous objective (;) Rd x R + IR with closed convex domains X' IR and y [R whose stochasticity is captured by the random vector ç. We are interested in the population saddle point (x" .y" E X' x y of the above problem such that F(x", .,²"")) S F(x", y²) S Fzz, y²) for all (z. E X y
For some randomized algorithm with output (2: ÿ). we measure its convergence rates by the popula- tion strong duality gap E Elmaxyey F(2. y) minzzex F(x, 7) or the population weak duality gup maxyey Fi. v)] minzex Far, 7) Note that both duality gaps are always larger than 0, and deterministic (#: 5) is the saddle point if and only if the duality gaps O.
In practice, we usually do not have access to the distribution Pe of the random vector Ç. Instead we are given dataset S = (E J with random vectors independently sampled from the distribution Pg. We define the empirical minimax optimization problem as
min max Fs(zz y) a TI Σ f(. xeX yey
Similarly, we can define the empirical saddle point (& (g) and the empirical duality gap w.r.t. the empirical function Fs. Zhang et al. [5I] established the stability and generalization properties of the empirical saddle point, which is essential to the design of our DP-SMO algorithms.
3 Differentially Private Stochastic Convex Optimization
As warm-up, we start with a simpler problem of differentially private stochastic convex optimization (DP-SCO) to showcase our main ideas. We consider the following stochastic optimization:
min F(x) (x]
where the stochastic function f(..() : Rd + R is defined on convex domain X' C Rdd and & is random vector from an unknown distribution Pç. Given a dataset S with i.i.d. samples from Pç, we develop a generic output perturbation framework with both privacy and population loss guarantees.
3.1 Near-Linear Time Algorithms for Smooth Strongly-Convex Functions
First study the case when the objective function is strongly-convex with the following assumptions. Assumption 3.1. For any g, the function [(x;; E) is L-Lipschitz, E-smooth, and corvex on the closed convex domain X Rdd.
Assumption 3.2. For any &, f(;; {) satisfies Assumption 3.1 and it is p-strongly convex on X.
Shalev-Shwartz et al. 42 proved that the empirical optimal solution has bounded stability if the objective {:: is strongly-conver and Lipschitz w.r.t. its domain (see Lemma A.2 in the appendix). As long as the output of some algorithm .A is close enough to the empirical solution, we can show that has bounded sensitivity, and thus standard Gaussian mechanism (Definition 2 can be applied to ensure differential privacy. This is formalized in Algorithm[]] where A is any algorithm for solving smooth strongly-conve finite-sum sum minimization problems. The theorem below shows that Algorithm is (s, -DP with optrmaltimal guarantees the excess risk. A proof is provided in Appendix A
5

---

Algorithm 2 Phased Output Perturbation for Convex Minimization
Input: Dataset {E:)=
algorithm A, DP parameters (e, regularizer μa, initializer 2o- 1: Set K = log. n = n/K K and un = Z0-
2: for k 1..................
do
3: Set μ4 = μ -=
4: Run the algorithm A on the smooth strongly-convex finite-sum minimization problem minzex (x( _ (1/n) Ei=(k- (i=(k-1)n+111 - + (2x/2)|lz - to obtain the out- put I&k such that || æt I (/ with probability at least - 6/4, where 2t = arg mingex (x) is the empirical optimal solution.
5: 26 = 2g WO, ofta) with o% 4L (10g(2.5/0)/((ure))
Output: žk.
Theorem 3.3. Under Assumption
3.2
Algorithm
iis (e, 8DDD and its oulput ãt satisfies
dlog(2.5/6)
(excess empirical risk)
T8 1BFs (8) ( 331.2
un?
((F F(x*) = L2 48d (2.5
(excess population risk)
unn
un?=?
where K. = p. is the condition number and assume x² and 4'ss are interior points of X.
Remark 1. Many algorithms for smooth strongly-convex finite-sum minimization problems guarantee
(J)))) - ( S with ( k) log(1/7)) gradient evaluations, where n is the sample size and the condition number. Setting 1/7 32um² /(82 L?) satisfies the requirements of
A by Markov's 's inequality (see Appendix A The gradient complexity of Algorithm is O
x) ())) for SVRG (26] or SARAH 38
and O((n + vπx) log(n/8) (8)) for Katyusha
As a comparison, most state-of-the-un private algorithms
for smooth strongly
functions only focus empirical problems. The linear-time algorithms in Feldman et al. 22 achieve optimal population guarantees but only work for I O(n) since they utilize the stability of SGD. Instead, we provide a flexible framework that includes various base methods without the necessity to show their algorithm-speeific stability. Further, near-linear time-complexity can be attained using fast variance seduction-based methods. Similar results existed in 333, but their base algorithm A only has in-expectation guarantees, which brings a critical challenge the design of private mechanisms.
3.2 Near-Linear Time Algorithms for Smooth Convex Functions
Next, we study the convex setting. direct method to first reduce the convex problem strongly-convex one by adding regularizer (2 to the objective and then apply Algorithm However, this approach only achieves a sub-optimal rate 50 [3). Inspired by the phased phased-ERM 22 method for nonsmooth convex losses, we show that a more sophisticated multi-phase Algorithm using an increasing sequence regularization parameters {px} can achieve near-optimal guarantees on the population loss for smooth DP-SCO. The increasing regularization parameters ensure that both added DP noise and approximation errors coming from the regularizer can be properly controlled. The output of Algorithm 2 satisfies the following guarantees, as proved in Appendix A
Theorem 3.4. Let Assumption 3.7 hold. Suppose there exists at least one optimal solution x² e
arg min x F F(x:) such that a|x" D. Algorithm
2 is (e,8)-DP and its output ö& satisfies
1
7Vdlog(2.5/0)
BIF(Tx)-FCC < 4LD log(n)
Vn
ne
for the excess population risk when setting μ = DL/ max (1/vn, 14 logg)() /
Remark 2. By Remark
1 and the fact that ΣK 1 1/m4 & (((DLL) the total gradient com-
plexity of Algorithm 2 is O(n + VneD/ (og(n/))) using SVRG (2 or SARAH (38, and O(nn + n (oD//)log(n/E)) for Katyusha 2 (see Appendix A We also point out that the smoothness assumption here is not necessary derive the utility bound, but allows the use of fast accelerated algorithms for solving the regularized ERM problems.

---

Algorithm 3 Output Perturbation for Stonngl-Coneex-StonnlyyCCooooo Minimax Problems
Input: Dataset {E:] algorithm A, DP parameters (e, SC parameters (Mar My).
I: Run algorithm A on smooth strongly-conven setpeeex-strongyycooeeee
finite-sum saddle point problem m maxyey: (s(x,)) y) m/n) Zi=1 [a, y; {i) to obtain the output (A) Ay()) such that with probability at least 8/8.
L
Mz (S) - f-ll + (y)lAA 0/1l -
unn
where (2g. (5) is the saddle point of Fs y) and we let fr := min Myy. and By (2lg)))) (mgl). Output: I (S) + (0, oātd,) and y (s) oV
2: Set xx (8L/(n))) log(3/))/ (
Bassily et al. 7 first proved optimal population guarantees for smooth convex functions, but their algorithm needs ((n2/2 gradient queries. The algorithms in Feldman et al. [2 achieve linear time-complexity, and require that l v O((L/ D) max (V,, analysis of SGD. In contrast, our framework can achieve near-optimal population guarantees using any algorithm without additional restrictions to the smoothness parameter. By equipping with variance reduction-based methods, near-linear time-complexity can also be attained when e < <O(VLL/ D). To the best of our knowledge, this is the first time that sophisticated optimization algorithms besides SGD are proven to obtain near-optimal population guarantees in near- -inerr time, Additionally, in the regime that Di D(VLL/D) D) < E < O(LL/ D) where previous smooth DP-SCO algorithms [ [2] fail to provide optimal guarantees, our framework still achieves near-optimal rate with better gradient complexity (n3//4 VED/L. compared to the state-of-the -he- nonsmooth DP-SCO algorithms 4 30
rooted in the stability
Differentially Private Stochastic Minimax Optimization
Using the same ideas as the minimization case in the previous section, we develop differentially private algorithms for stochastic minimax optimization in
4.1 Near-Linear Time Algorithms for Smooth Sonngl---CCoee-ttoogyy--CCCC
Functions
(SC-SC) case with the following assumptions. Assumption 4.1. For any vector &, fxx g) is T--lipschizz and E-smooth on the closed convex domain X x y C Idd x Rdy, Moreover, f( E) is convex on X for any y E y, and f(x, is concave on for any x E X.
First, we study the strougly-ooveee-stooolyyyooomo
Assumption 4.2. For any vector &, fxx y; {) satisfies Assumption and f(, y: E) is -a-strongly convex on X for any E Y. and fxx is Hy-slrongly concave on for any E X.
In a manner similar to the minimization case, Zhang et al. 51 showed that the empirical saddle point is stable, which also implies its generalization error (see Lemma B.2 in the appendix). As direct consequence, any algorithm AA whose output is sufficiently close to the empirical saddle point has bounded sensitivity. This observation leads to Algorithm 3 for SC-SC DP-SMO with guarantees given in the theorem below. Here A can be any method for smooth SC-SC finite-sum minimax problems, and smoothness allows us obtain efficient algorithms.
Theorem 4.3. Under Assumption 4.2 Let saddle points (ø'. (G) and (z".y") be interior points of the domain X x y. Then Algorithun S is (E.6)-D1 and its outpus (ī, 9) satisfies the following utility
bounds on the empirical and population strong duality gap:
dlog(5/8)
(empirical)
E
max (5( yey
min Fs 8)
257L (Kz + +k)
xEX
um?g2
(population)
E
max F(ū, v) min DEy
Flz, 8)]
< 332 (Kn + k)
3
128d log((/0)))))
nEX
un
un2=2
where we let μ min( Hy Ke e/p By l/ K = e/j. and d max(de, d
7

---

Remark 3. Existing methods for sinite-sum saddle point problems output solutions such that
Blmax (s,, y)
minzex (1, Ay()) I with complexity ((T(n, k) log(( />)).
Thus setting 1/7 16um² ( satisfies the requirements of AA (see Appendix B The gradient + n²) log(n/5)) for
complexity of Algorithm
O(nk log(n/6)) for Extragradient 4 (
SVRG/SAGA 39 ( + / (n)( for Acc-SVRG/SAGA
39 and [ + MKkKy
n²/4 log(n/)) for AL-SVRE 3 or Catalys-Ace-SVRG
48
The minimax problem is equivalent to minimization problem z when the domain ) is restricted to a singleton. As a result, the lower-bound ([1/(uzn) + dr log(1/6) (=²,n==))) (og(1/))//m_n²²') of SC DP-SCO [ trivially holds for SC-SC DP-SMO, and our results are near-optimal w.r.t. n and (a. 8). The flexible framework allows the use of off uff-tf optimization algorithms for smooth SC-SC finite-sum minimax problems from well-studied research community 48 34
without being aware of the
algorithn-specific stability bound. Based on this framework, we can produce the first near- mear-linear time algorithms for smooth SC-SC DP-SMO with near-optimal guarantees the duality gap. Similar rates were obtained 3 on the population primal risk through the reduction to SC DP-SCO. However, the generalization error on the primal function does not always apply to the original minimax problem when the expectation and maximization cannot be exchanged.
4.2 Near-Linear Time Algorithms for Smooth Convex-Concave Functions
In this section, we focus on the more general case when the objective is smooth and convex-concar We continue to build private algorithms using the stability and generalization results of empirical solutions. When the function is convex-concawe
the following results for the regularized empirical problems can be applied by adding SC-SC regularizer. Here, the stability bound is slightly tighter than the one in [51] and a proof is provided in Appendix B for completeness.
Lemma 4.4. Consider a stochastic minimax problem such that f(x y: {) is convex-concave and L- Lipschitz with p2 Pu-strongly convex pyy syy-sronggyy concave regularizer G y). Let μ min Hyy and denote the empirical saddle point of function Fs(x, y) + G(r,, y) as (#'s U's) given dataset with n i.i.d samples. Then for any neighboring datasets S ~ S', we have
4L2
Me Mell 112 + Pullos
is 11?
um?
The stability result implies the generalization error of the empirical solution be bounded
2/21.
max IE F(f5. + G( w)] min E F(x, 03) + G(z, (G) < zEX
BEY
um
measured by the population weak duality gap
Unlike DP-SCO, the interaction between the primal variable x and the dual variable y puts additional challenge in the analysis. In order to derive near-optimal algorithm for convex-concave objectives, we need to carefully design the regularization parameters for the primal æ and the dual y to control both approximation errors and DP noise. Based on the phased output perturbation framework, we propose a novel algorithm for convex-concan DP-SMO in Algorithm F
The small fixed parameter ensures the approximation error from adding regularization for the dual properly bounded. The role of the increasing parameter grg the same Algorithm [ We only add noise to the primal solutions and output ãx in Algorithm g A natural way to derive a corresponding dual solution is to solve the smooth concave maximization problem maxyey R[F(@x, y)] with DP constraints. We instead provide an alternative method that is symmetric to Algorithm by switching the role of the primal and the dual. This phased algorithm iteratively solves
km
min max ZE vEy
Σ
P ||011
DR
2
at phase h and only perturbs the dual variables. The detailed Algorithm S is provided in Appendix B Its output ÿK together with the output æx of Algorithm g satisfy (e, DD by basic composition in Lemma 2.1 The guarantees are shown in the following theorem with detailed proofs in Appendix
8

---

Algorithm 4 Phased Output Perturbation for Convex-Concave Minimax Problems
Input: Dataset S { algorithm A, DP Parameters (e, 0), regularizer μ., initializer zo-
1: Set K = (n). n = n/K K and un = z0-
2: for k == 1.............
do
3: Set f4 = f 2*
4: Run the algorithm A on the smooth SC-SC finiee-smm saddle point problem
ka Σ
min max FA 3)
f(x vi
+ |
Mk 2
1
14 $1v/1?
#EX yey
2
(k-1))=+
to obtain the output (rk, Uk) such that with probability 1 55/8.
1,2
Ms - #I12 + pllyE - Stll? -
where (ātz V8)) is the saddle point of the regularized empircaa function Fic v).
5:
3K = 2% + NNO (ft,) with oR
Output: σK.
Theorem 4.5. Let Assumption 4.1 hold. Suppose max ( [| (ulll < for all x E X and y E y.
Then the composition of Algorithm and i (DP and the output (ØK, (K) satisfies the following bound on the population weak duality gap:
GVdlog((/0)
max (TK 8)1 yEy
min E uk)] s 16LD 1GL (n)
Vn +
ne
/log (allog(/////// (log(5/))/(ne) where d = max (dz, d when setting (L/D) max 2/ /n, 13 logg()) /
Remark 4. By Remark we can set 1/7 16 (0L2) to satisfy the requirements of A in
each phase. For example, if we Extragradient + the total complexity is 2 Σ (((m(e+
(( + n²) log(n/6)). After similar calculations, the complexity becomes O(nnL)/L + n//44) log(///)))))) if A is AL-SVRE 3 or Catalyst-Ace-VRR
pK)/p) log(n/))
48 noticing
that the condition number (e+ pk)/p μ can be as large as (n).
Remark 5. To achieve near-linear comellxpy,,,,ty, we utilize the special structures of our reg- ularized problems in Algorithm 4 and D (i) Objective functions of the regularized problems are separable with different smoothness parameters w.r.t and y. Applying the algorithm in Jin et al. (25] that is optimized for such special case, we can achieve the overall complexity O(n + mLD/LL L) log(n/5) (()) (ii) The regularization terms with potentially large smoothness pa- rameters are prox- wro--fieenll π SVRG/SAGA and its accelerated version 39 which leverage this property can be used, and the complexity only depends on the smoothness parameter of f(x, ys () The total complexity is O(nn O(n 2 log(n/5)) for SVRG/SAGA, and O(nn + nLDDLL log(n/5)) for AcCSSVRG/SAGA, noticing that now the condition number f/U is only O(ynkD/L) L). More
details are provided in Appendix B
Algorithm can be extended the smooth C-SC case the SC-C case, where near-linear time near-optimal algorithms are also provided (see Appendix BB. According to the lower-bound discussed in [ЛЛ) our utility guarantee is optimal up to logarithmic terms. Although the primal problem min (xx) 2 max F(x,, y) not necessarily smooth when f(, y; {) is smooth and convex- concave, we provide several instances of the flexible framework that achieve near-optimal rates in near-linear time. This improves upon current results for DP-SMO II 49 and gives a new example where near- leneer time algorithms are available for nonsmooth DP Our results suggest that for nonsmooth DP-SCO problems, if there exists some smooth convex-concave minimax reformulation, then near- opttiml rates can be attained in near-lineas time. For example, the nonsmooth convex problem minzer*= || Ar b//1 given E IR Xd, and b RRd, equivalent to the smooth convex- concave minimax problem min max -|vll> V (Ax b). However, remains an open question whether optimal utility bound can be obtained in linear time for general nonsmooth problems.
"Thi means the proximal operator easy compute. See Palaniappan and Bach 39 for more details.

---

5 Conclusion
We provide a general framework for both smooth DP-SCO and DP-SMO problems with the near- optimal privacy-utility trade-off. The flexible framework allows to bring various off-the-shelf, fast convergent non-private optimization algorithms into the DP domain. Using the framework, we enrich the class of near-linear time algorithms for smooth DP-SCO and provide the first near-linear time algorithms for smooth DP-SMO. For future work, it is interesting study whether the logarithmic terms in gradient complexity and the final utility bound of our algorithms can be further removed. Another direction to derive simpler private algorithms for smooth convex-conees DP-SMO based on the special structure of the regularized problems. Moreover, we believe our framework also opens the door simpler derivatives with possibly improved complexities for nonsmooth DP-SCO, nonsmooth DP-SMO, and even more general forms of optimization problems such as variational inequalities and games. We will leave these for future investigation.
Acknowledgments and Disclosure of Funding
We would like to thank anonymous reviewers for the insightful feedback. I.7.. gratefully acknowl- edges funding by the Max Planck ETH Center for Learning Systems (CLS). This work does not relate to the current position of K. T. at Amazon. S.O. is supported in part by NSF grants CNS-2002664, 118-1929955, and CCF-2019844 (IFML). N.H. is supported by ETH research grant funded through ETH Zurich Foundations and NCCR Automation funded through Swiss National Science Foundations; part of the work was done while N.H. was visiting the Simons Institute for the Theory of Computing.
part of NSF Institute for Foundations of Machine Learning
References
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[2] Zeyuan Allen-Zhu. Katyusha: The first direct acceleration of stochastic gradient methods. The Journal of Machine Learning Research, 18(1):8194-822 2017.
[3] Hilal Asi, Yair Carmon, Arun Jambulapati, Yujia Jin, and Aaron Sidford. Stochastic bias-reduced gradient methods. Advances in Neural Information Processing Systems, 34, 2021.
[4] Hilal Asi, Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimiza- tion: Optimal rates in 11 geometry. In International Conference on Machine Learning, pages 393-403, PMLR, 2021.
[5] Amit Attia and Tomer Koren. Uniform stability for first-order empirical risk minimization. In Conference Learning Theory, pages
PMLR, 2022.
[6] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 55th Annual Symposium on Foundations of Computer Science, pages 464 473. IEEE, 2014.
[7] Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. Private stochastic convex optimization with optimal rates. Advances in Neural Information Processing Systems, 32, 2019.
[8] Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems, 33, 2020.
[9] Raef Bassily, Cristóbal Guzmán, and Michael Menart. Differentially private stochastic opti- mization: New results in convex and non-convex settings. Advances in Neural Information Processing Systems, 34, 2021.
[10] Amos Beimel, Hai Brenner, Shiva Prasad Kasiviswanathan, and Kobbi Nissim. Bounds on the sample complexity for private learning and private data release. Machine learning, 94(3): 401-437, 2014.
10

---

[11] Digvijay Boob and Cristóbal Guzmán. Optimal algorithms for differentially private stochastic monotone variational inequalities and saddle-point problems. arXiv preprint (rXiv:214400298: 2021.
[12] Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
[13] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Frlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium, pages 2633-2650, 2021.
[14] Rishav Chourasia, Jiayuan Ye, and Reza Shokri. Differential privacy dynamics of langevin diffusion and noisy gradient descent. Advances in Neural Information Processing Systems, 34, 2021.
[15] Rachel Cummings and Deven Desai. The role of differential privacy in gdpr compliance. In Proceedings of the Conference on Fairness, Accountability, and Transparency, 2018.
[16] Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. Sbeed: Convergent reinforcement learning with nonlinear function approximation. In Interna- rional Conference Machine Learning, pages 1125-1121 1134. PMLR, 2018.
[17] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):211-400) 2014.
[18] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theorr of cryptography conference, pages 265-284. Springer, 2006.
[19] Farzan Farnia and Asuman Ozdaglar. Train simultaneously, generalize better: Stability of gradient-baser minimax learners. In International Conference on Machine Learning, pages 3174-3185. PMLR, 2021.
[20] Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas. Efficient and accurate estimation of lipschitz constants for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[21] Vitaly Feldman, Ilya Mironov, Kunal Talwar, and Abhradeep Thakurta. Privacy amplification by iteration. In 59th Anmual Symposium on Foundations of Computer Science, pages 521-532. IEEE, 2018.
[22] Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: optimal rates in linear time. Proceedings of the 52nd Anmual ACM SIGACT Symposium on Theory of Computing, pages 439-449, 2020.
[23] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[24] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pages 1225- 1234. PMLR, 2016.
[25] Yujia Jin, Aaron Sidford, and Kevin Tian. Sharper rates for separable minimax and finite sum optimization via primal-dual extragradient methods. arXiv preprint arXiv:2202.046400 2022.
[26] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in newral information processing systems, 26, 2013.
[27] Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International conference machine learning, pages 1376-1385. PMLR, 2015.

---

[28] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji. Kallista Bonawitz, Zachary Charles, Graham Cornode. Rachel Cummings. et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2):1 2021.
[29] Yilin Kang, Yong Liu, Jian Li, and Weiping Wang. Stability and generalization of differentially private minimax problems. arXiv preprint arXiv:2204.0455 2022.
[30] Janardhan Kulkarni, Yin Tat Lee, and Daogao Liu. Private non-smooth erm and sco in sub- quadratic steps. Advances in Neural Information Processing Systems, 34, 2021.
[31] Yunwen Lei, Zhenhuan Yang, Tianbao Yang. and Yiming Ying. Stability and generalization of stochastic gradient methods for minimax problems. In International Conference on Machine Learning, pages 6175-6186 PMLR, 2021.
[32] Zinan Lin, Alankar Jain, Chen Wang, Giulia Fanti, and Vyas Sekar. Using gans for sharing networked time series data: Challenges, initial promise, and open questions. In Proceedings of the ACM Internet Measurement Conference, pages 464 483, 2020.
[33] Andrew Lowy and Meisam Razaviyayn. Output perturbation for differentially private convex optimization with improved population loss bounds, runtimes and applications to private adversarial training. arXiv preprint arXiv:2102.04704 2021.
[34] Luo Luo, Guangzeng Xie, Tong Zhang, and Zhihua Zhang. Near optimal stochastic al- gorithms for finite-sum unbalanced convex convex-concave minimax optimization. arXiv preprint arXiv:2106.0176,, 2021.
[35] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
[36] Frank D McSherry. Privacy integrated queries: an extensible platform for privacy-preservig data analysis. Proceedings of ACM SIGMOD International Conference on Management of data, pages 19-30, 2009.
[37] Yu Nesterov. Smooth minimization of non-smooth functions. Muthematical progrumming, 103 (1):127-152, 2005.
[38] Lam M Nguyen, Jie Liu, Katya Scheinberg, and Martin Takáč. Sarah: novel method for machine learning problems using stochastic recursive gradient. In International Conference Machine Learning, pages 2613-2621. PMLR, 2017.
[39] Balamurugan Palaniappan and Francis Bach. Stochastic variance reduction methods for saddle- point problems. Advances in Neural Information Processing Systems, 29, 2016.
[40] Tyrrell Rockafellar. Monotone operators associated with saddle-functions and minimax problems. Nonlinear functionaa analysis, 18(part 1):397-407, 1970.
[41] Tyrrell Rockafellar. Monotone operators and the proximal point algorithm. SIAM journal on control and optimization, 14(5):877-898, 1976.
[42] Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Stochastic convex optimization. In Conference on Learning Theory, volume 2, page 5, 2009.
[43] Paul Tseng, On linear convergence of iterative methods for the variational inequality problem. Journal of Computational and Applied Mathematics, 60(1-2):237-252. 1995.
[44] Di Wang, Minwei Ye, and Jinhui Xu. Differentially private empirical risk minimization revisited: Faster and more general. Advances in Neural Information Processing Systems, 30, 2017.
[45] GR Wood and Zhang. Estimation of the lipschitz constant of function. Journal of Global Optimization, 8(1):91-03,, 1996.
12

---

[46] Xi Wu, Fengan Li, Arun Kumar. Kamalika Chaudhuri, Somesh Jha, and Jeffrey Naughton. Bolt- on differential privacy for scalable stochastic gradient descent- bessent-based analytics. In Proceedings of ACM International Conference on Management of Data, pages 1307-1322,,01
[47] Liyang Xie, Kaixiang Lin, Shu Wang. Fei Wang, and Jiayu Zhou. Differentially private 2018.
generative adversarial network. arXiv preprint
[48] Junchi Yang, Siqi Zhang, Negar Kiyavash, and Niao He. A catalyst framework for minimax optimization. Advances in Neural Information Processing Systems, 33:5667-5678, 2020.
[49] Zhenhuan Yang. Shu Hu, Yunwen Lei, Kush R Vashney, Siwei Lyu, and Yiming Ying. Dif- ferentially private sgda for minimax problems. In Uncertainty in Artificial Intelligence, pages PMLR, 2022.
[50] Jiaqi Zhang, Kai Zheng, Wenlong Mou, and Liwei Wang. Efficient private erm for smooth objectives. Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 3922-3928, 2017.
[51] Junyu Zhang, Mingyi Hong, Mengdi Wang, and Shuzhong Zhang. Generalization bounds for stochastic saddle point problems. In International Conference on Artificial Intelligence and Statistics, pages 568-576. 2021,
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section for contributions in DP-SCO and Section
4 for contributions in DP-SMO.
(b) Did you describe the limitations of your work? [Yes] See Section .
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This paper purely theoretical and has no immediate societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of theoretical results? [Yes] See Assumption
5.1 and 3.2 for DP-SCO and Assumption 4. and for DP-SMO.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A for DP-SCO and Appendix B for DP-SMO.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
13

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "TWYaVlqVUCO", "reviewer": "Reviewer_Jh8C", "summary": "The paper is one more link in the ever growing chain of results on DP ERM-based optimization algorithms, which dates all the way back to CMT08 and BST14. In this paper, the authors up the ante on two main axes, as follows. First, rather than looking at classic minimization problem, they deal with a min-max problem (a setting that does arise in practice, such as adversarial GANs), see Eqs. (1)-(2). Second, rather than giving their own algo. for such an optimization, the authors assume some off-the-shelf algorithm for which they can apply output perturbation since - by assumption - the optimization function is smooth and stongly-convex/strong-concave, a setting where an approximated loss must imply approximated models. Lastly, the authors achieve near-optimal loss in near-linear time (\"near\"=up to poly-log factors).\n", "questions": "The runtime complexity's dependency on \\epsilon for DP-SGDA is cited correctly?", "limitations": "I wish they had a line saying \"while having the loss-function strongly-convex / strongly-concave and smooth, we make rather a strong assumption, but this is ok as this is the first paper to ...\"", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Ultimately this is an interesting paper that makes solid progress on one of the more canonical and current problems (or venues) studied in the DP-ML literature. While the novelty factor isn't that great (the assumption of a loss-function which is strongly-convex strong-concave and smooth does imply the result) I still believe it is a problem worth mentioning in NeurIPS, in particular as it may \"lead the way\" for min-max DP-optimization algorithms that require less on the loss-function.\n\nStrengths: well written, solid progress on a very relevant problem.\nWeaknesses: rather strong assumption on the loss-function", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LpA5I-VYCnU", "reviewer": "Reviewer_s8NK", "summary": "The paper provides a general framework for solving smooth DP-SMO and smooth DP-SCO problems by utilizing the (phased) output perturbation mechanism, so there is no need for modifying the algorithms themself to have a DP result.", "questions": "1. Shouldn't the input of the proposed algorithms (i.e., Algorithm 1) include the parameter $\\mu$?\n\n2. How could one decide $L$ in Algorithm 1 in practice? \n\n", "limitations": "Yes.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: theoretically sounded and working for a wide range of objective functions.\n\nWeaknesses: 1. Lack of numerical results. The reviewer is curious about how to apply it to some popular algorithms and their performance compared with existing DP algorithms.\n2. The presentation of this paper is hard to follow for the reviewer.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FtTU31HvhqZ", "reviewer": "Reviewer_XKZT", "summary": "The paper studies DP stochastic convex optimization (SCO) and stochastic minmax optimization (SMO). They provide near-linear time algorithms under (strong) convexity assumptions that attain optimal or near-optimal rates. ", "questions": "I think most of my questions were asked in the above. Two more below:\n\n-Could you elaborate on the reason for the asymmetry in the x and y regularizers in Algorithm 4? \n\n-You claim in Remark 2 that smoothness is not necessary for the utility bound to hold. Can you explain why? Also, what gradient complexity would be achievable without smoothness? ", "limitations": "N/A ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n-The phased output perturbation idea is nice and simple, but apparently also effective. \n\n-Apparently improved (faster) algorithms for convex-concave and SC-SC min-max are the main contributions. \n\nWeaknesses (and borderline strengths/contributions): \n\n-Missing related work: \n\nTheorem 3.3 (excess risk bounds with black box \"bring your own algorithm\" output perturbation for strongly convex loss) seems to have already been provided in prior work: see Corollary 3.4 and Corollary 4.2 of https://arxiv.org/pdf/2102.04704.pdf. The idea of \"bring your own algorithm\" output perturbation is also a theme in this work and should be credited. \n\n-Theorem 3.4 is a somewhere between a good contribution and a marginal contribution: it is essentially optimal excess risk and near-linear time: existing works (e.g. FKT20) give optimal excess risk in linear time with some mild smoothness restrictions that this excess risk bound removes, but smoothness is still required here. Furthermore, a restriction on the smoothness parameter seems to be required here in order to be linear time. Also, optimal algorithms for non-smooth losses require a bit more computation (e.g. $n^{11/8}$ is the smallest gradient complexity I believe) vs. Theorem 3.4 is $n + \\sqrt{n} \\ell$. On a related note, the units in the gradient complexity expression are not matching, as $\\ell$ is not unit-free. What is the precise unit-free gradient complexity (e.g. with SVRG)? \n\n-**Discussion of optimality of the min-max bounds seems to be lacking. Lower bound for convex-concave was provided in Boob & Guzman, but I'm not sure its obvious that the the SC-SC lower bound follows immediately from SC SCO lower bound? It would be great to complete Table 1 with lower bounds. \n\n-Some imprecision/slight overclaiming of \"optimal\"  and \"linear time\" in places where this should be qualified with \"near\" (e.g. due to dependence on condition number). Also, \"smooth\" should be included  for emphasis whenever the problem class is smooth--e.g. Section 3.2 heading should be \"near-linear time algorithms for smooth convex functions\" since I don't think you are claiming near-linear time algorithms for general (non-smooth) convex functions. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FroStP_x764", "reviewer": "Reviewer_mZUV", "summary": "The paper proposes a framework for DP-SCO and DP-SMO, such that any non-private algorithm with a certain convergence guarantee can be plugged in. For smooth DP-SCO, it achieves the first near optimal guarantee without for algorithms apart from SGD; for smooth DP-SMO, it gives the first near linear time private algorithm.", "questions": "It all looks good to me.", "limitations": "N/A", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper provides the first near linear bound for DP-SMO.\nThe paper provides an algorithm-independent framework, thus extending the near linear time guarantee to algorithms other than DP-SGD.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept