Title: Private Algorithms for Stochastic Saddle Points and Variational Inequalities: Beyond Euclidean Geometry
Venue: NeurIPS 2024
Accepted: yes
Keywords: Differential Privacy

Stochastic Saddle Point Problem

Stochastic Variational Inequality

Strong Gap

Stochastic Minimax Optimization

Algorithmic Stability
Abstract: In this work, we conduct a systematic study of stochastic saddle point problems (SSP) and stochastic variational inequalities (SVI) under the constraint of $(\epsilon,\delta)$-differential privacy (DP) in both Euclidean and non-Euclidean setups. We first consider Lipschitz convex-concave SSPs in the $\ell_p/\ell_q$ setup, $p,q\in[1,2]$. That is, we consider the case where the primal problem has an $\ell_p$-setup (i.e., the primal parameter is constrained to an $\ell_p$ bounded domain and the loss is $\ell_p$-Lipschitz with respect to the primal parameter) and the dual problem has an $\ell_q$ setup. Here, we obtain a bound of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$ on the strong SP-gap, where $n$ is the number of samples and $d$ is the dimension. This rate is nearly optimal for any $p,q\in[1,2]$. Without additional assumptions, such as smoothness or linearity requirements, prior work under DP has only obtained this rate when $p=q=2$ (i.e., only in the Euclidean setup). Further, existing algorithms have each only been shown to work for specific settings of $p$ and $q$ and under certain assumptions on the loss and the feasible set, whereas we provide a general algorithm for DP SSPs whenever $p,q\in[1,2]$. Our result is obtained via a novel analysis of the recursive regularization algorithm. In particular, we develop new tools for analyzing generalization, which may be of independent interest. Next, we turn our attention towards SVIs with a monotone, bounded and Lipschitz operator and consider $\ell_p$-setups, $p\in[1,2]$. Here, we provide the first analysis which obtains a bound on the strong VI-gap of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$. For $p-1=\Omega(1)$, this rate is near optimal due to existing lower bounds. To obtain this result, we develop a modified version of recursive regularization. Our analysis builds on the techniques we develop for SSPs as well as employing additional novel components which handle difficulties arising from adapting the recursive regularization framework to SVIs.

Private Algorithms for Stochastic Saddle Points and Variational Inequalities: Beyond Euclidean Geometry

Abstract

In this work, we conduct a systematic study of stochastic saddle point problems (SSP) and stochastic variational inequalities (SVI) under the constraint of ( ϵ, δ ) -differential privacy (DP) in both Euclidean and non-Euclidean setups. We first consider Lipschitz convex-concave SSPs in the ℓ p /ℓ q setup, p, q ∈ [1 , 2] . That is, we consider the case where the primal problem has an ℓ p -setup (i.e., the primal parameter is constrained to an ℓ p bounded domain and the loss is ℓ p -Lipschitz with respect to the primal parameter) and the dual problem has an ℓ q setup. Here, we obtain a bound of ˜ O ( 1 √ n + √ d nϵ ) on the strong SP-gap, where n is the number of samples and d is the dimension. This rate is nearly optimal for any p, q ∈ [1 , 2] . Without additional assumptions, such as smoothness or linearity requirements, prior work under DP has only obtained this rate when p = q = 2 (i.e., only in the Euclidean setup). Further, existing algorithms have each only been shown to work for specific settings of p and q and under certain assumptions on the loss and the feasible set, whereas we provide a general algorithm for DP SSPs whenever p, q ∈ [1 , 2] . Our result is obtained via a novel analysis of the recursive regularization algorithm. In particular, we develop new tools for analyzing generalization, which may be of independent interest. Next, we turn our attention towards SVIs with a monotone, bounded and Lipschitz operator and consider ℓ p -setups, p ∈ [1 , 2] . Here, we provide the first analysis which obtains a bound on the strong VI-gap of ˜ O ( 1 √ n + √ d nϵ ) . For p -1 = Ω(1) , this rate is near optimal due to existing lower bounds. To obtain this result, we develop a modified version of recursive regularization. Our analysis builds on the techniques we develop for SSPs as well as employing additional novel components which handle difficulties arising from adapting the recursive regularization framework to SVIs.

1 Introduction

Stochastic saddle point problems (SSP), are an increasingly important part of the machine learning toolkit. These problems model optimization settings with an inherent min-max structure, and for this reason are also referred to as stochastic min-max optimization problems. Concretely, the goal is to find an approximate solution of the following problem defined over a convex-concave loss,

where D is an unknown distribution for which we have access to an i.i.d. sample S . Problems of this kind have important applications in stochastic optimization [NJLS09, JNT11, ZL15], federated learning [MSS19], distributionally robust learning [YLMJ22, ZZZ + 24a, ZB24], reinforcement learning [DSL + 18], and algorithmic fairness [ABD + 18, WM19].

Closely related to saddle point problems are stochastic variational inequalities (SVIs). Given a monotone operator, G D ( z ) := E x ∼D [ g ( z ; x )] , the objective is to approximate the point z ∗ ∈ Z , where

Stochastic saddle point problems can be easily related to variational inequalities by observing that the saddle operator (an operator closely related to the gradient) of a convex-concave function is monotone. While SSPs and SVIs are closely related, it can be the case that a problem which can be formulated as a monotone SVI is not easily cast as a convex-concave SSP [JN19].

Parallel to the above, the problem of privacy has become increasingly important in the big data era. In this regard, the notion of differential privacy has arisen as the premier standard. Stochastic optimization problems are a natural target for privacy concerns due to the fact that they are frequently formulated using a dataset of (potentially sensitive) individual data records. For many such problems, the constraint of differential privacy necessitates fundamentally new rates and techniques, and as such the formal characterization of these problems is an important task.

Thus far, work on differentially private SSPs and SVIs has focused primarily on Euclidean settings. However, a number of important, including many of those referenced at the start of this section, are naturally formulated in other geometries. Prior to this work, the optimal utility rate for DP SSPs was known only in Euclidean and polyhedral settings. For SVIs, the best achievable utility was unknown in any geometry (including Euclidean), at least under canonical utility measures. In this work, we provide the first systematic study of SSPs and SVIs in general geometries. The new analysis tools we develop lead to optimal rates for a number of these important setups.

1.1 Contributions

In this work, we provide the first systematic study of stochastic saddle point problems and variational inequalities in both Euclidean and non-Euclidean geometries. Our first results pertain to stochastic saddle point problems where the primal problem has an ℓ p -setup and the dual problem has an ℓ q -setup, where p, q ∈ [1 , 2] . Here we assume the convex-concave loss is Lipschitz. We generalize the recursive regularization framework developed in previous works to more handle non-Euclidean geometries [AZ18, BGM23]. At the heart of this extension is a fundamentally new analysis of the generalization properties of this algorithm. The issue of generalization has in fact been a key issue at the heart of many other works studying SSPs [LYYY21, OPZZ22, BGM23], as the presence of a supremum in the strong SP-gap accuracy measure (see Eqn. (4)) breaks more traditional generalization techniques. In contrast to prior work, our generalization technique works by avoiding entirely any generalization bound for the strong gap itself. Rather, we introduce a new accuracy measure measure which, when used in conjunction with the recursive regularization algorithm, eventually translates into a strong gap guarantee. Our technique stands in particular contrast to [BGM23], which is thus far the only work in the DP literature to obtain optimal strong SP-gap rates in the Euclidean setting and also uses recursive regularization. However, their technique fundamentally relies on a McDiarmids style concentration bound that is worse by a poly ( d ) factor in non-Euclidean setups such as the ℓ 1 setting [Pan08]. Using these new techniques, we provide the first analysis which obtains the near optimal rate of ˜ O ( 1 √ n + √ d nϵ ) on the strong SP-gap for any p, q ∈ [1 , 2] . Our algorithm achieves this rate in ˜ O ( min { n 2 ϵ 1 . 5 √ d , n 3 / 2 }) number of gradient evaluations. We note that the near optimality of this rate

is established by lower bounds for DP stochastic convex optimization, which is a special case of DP SSPs [BGN21, AFKT21]. Previously, comparable rates on the strong gap had only been obtained in the case where p = q = 2 or under strong additional assumptions [BGM23, ZB24, GGP24].

Next, we consider DP stochastic variational inequalities with a monotone, bounded, and Lipschitz operator. We adapt the recursive regularization framework even further and again leverage a novel generalization analysis. Here, we obtain the rate ˜ O ( 1 √ n + √ d nϵ ) on the strong VI-gap (see Eqn. (5)) in the ℓ p -setting, p ∈ [1 , 2] . Our algorithm again achieves this rate in ˜ O ( min { n 2 ϵ 1 . 5 √ d , n 3 / 2 }) number of gradient evaluations. This is the first result to obtain the near optimal convergence rate on the strong VI-gap for p -1 = Ω(1) , which notably includes the Euclidean case. The corresponding lower bound for p = 2 was established in [BG23], and we provide a simple extension of their technique to the case where p -1 = Ω(1) . See Appendix E. Finally, for the setting p = 2 , we show that our rate can be achieved in a near linear number of gradient evaluations by leveraging acceleration techniques for Lipschitz and strongly monotone variational inequalities.

1.2 Related Work

Differentially private stochastic optimization now has a broad body of work spanning over a decade [JKT12, BST14, JT14, TTZ15, BFTT19, FKT20, AFKT21]. Such work has rigorously characterized the problem of stochastic convex optimization in a variety of geometries. In ℓ p setups, for p ∈ [1 , 2] , it is now known that the optimal rate is ˜ O ( 1 √ n + √ d nϵ ) for such problems [AFKT21, BGN21]. It has also been shown that additional improvements are possible in the ℓ 1 setting under smoothness assumptions. The study of stochastic saddle point problems under differential privacy is much less developed, but has nonetheless attracted a surge of recent interest [YHL + 22, ZTOH22, BGM23, GGP24]. Without privacy, optimal O (1 / √ n ) guarantees on the strong SP-gap have long been known [NY78]. With privacy, the (near) optimal ˜ O ( 1 √ n + √ d nϵ ) rate was obtained only recently, and then only in the Euclidean setting [BGM23]. In fact, work on DP-SSPs has focused largely on the case where p = q = 2 , despite the fact that important problems are naturally formulated in other geometries. In particular, the case where q = 1 has important applications in distributionally robust optimization, federated learning, and algorithmic fairness. In this regard, the works [GGP24, ZB24] have recently studied DP SSPs with q = 1 . The work [GGP24] studies the ℓ 1 /ℓ 1 setting when the loss is additionally assumed to be smooth and the constraint set is polyhedral; they achieved the rate ˜ O ( 1 √ n + 1 ( nϵ ) 1 / 2 ) . We note that smoothness is fundamentally necessary in achieving this dimension independent rate, as otherwise existing lower bounds of ˜ Ω ( 1 √ n + √ d nϵ ) hold for such problems [AFKT21]. The work [ZB24] studied the problem of differentially private worst-group risk minimization, which is closely related to DP-SSPs in the ℓ 1 /ℓ 2 setting, but requires the loss to have a specific linear structure with respect to the dual parameter. For ℓ 1 /ℓ 2 saddle point problems having this structure, their result implies a rate of O ( 1 √ n + √ d nϵ ) on the strong gap.

Work on SVIs is less developed. Non-privately, the optimal strong VI-gap rate of O ( 1 √ n ) was established in [JNT11] (although related techniques trace back to [NY78]). Work on differentially private variational inequalities is limited to the work [BG23]. In the Euclidean setup, this work achieved a rate of ( 1 n 1 / 3 + √ d n 2 / 3 ϵ ) on the strong VI-gap under DP.

2 Preliminaries

In this section, we detail preliminaries for stochastic saddle point problems and differential privacy. Both SSPs and SVIs share a similar structure, which we detail first. Throughout, we use [ w,θ ] to denote the concatenation of the vectors w and θ . For a function f , we let ∇ f denote an arbitrary subgradient selection of f . Finally, we let Unif ( U ) denote the uniform distribution over the set U .

Stochastic Monotone Operators. Let X be some abstract data domain and let S ∼ D n for n > 0 and D some unknown distribution over X . Let ∥ · ∥ be some norm and ∥ · ∥ ∗ its dual. We consider some compact convex parameter space Z ⊆ R d of diameter B with respect to ∥ · ∥ . Let B d ∥·∥ ( r ) denote the d -dimensional ball of radius r > 0 w.r.t. ∥ · ∥ centered on zero. We assume there exists

L > 0 such that g : Z × X ↦→ B d ∥·∥ ∗ ( L ) is a bounded operator and that for any x ∈ X , g ( · ; x ) is monotone. That is, ∀ z 1 , z 2 ∈ Z it holds that ⟨ g ( z 1 ; x ) -g ( z 2 ; x ) , z 1 -z 2 ⟩ ≥ 0 . We define the empirical and population operators as G S ( z ) = 1 n ∑ n i =1 g ( z ; x i ) and G D ( z ) = E x ∼D [ g ( z ; x )] .

Stochastic Saddle Point Problems. SSPs have the following structure in addition to the above. Let d w , d θ ≥ 0 such that d w + d θ = d . We assume Z is the product of the convex compact sets W ⊆ R d w and Θ ⊆ R d θ equipped with norms ∥ · ∥ w and ∥ · ∥ θ and having diameters B w and B θ respectively. Then Z = W × Θ . We let ∥ [ w,θ ] ∥ = √ ∥ w ∥ 2 w + ∥ θ ∥ 2 θ , and thus the diameter of Z satisfies B ≤ √ B 2 w + B 2 θ ; note the geometric mean of two norms is always a norm.

In SSPs, we consider the case where the monotone operator is the saddle operator of a convexconcave loss function f : W × Θ × X ↦→ R . The saddle operator is defined as g ( w,θ ; x ) = [ ∇ w f ( w,θ ; x ) , -∇ θ f ( w,θ ; x )] and is always monotone if f is convex-concave. We also define the corresponding population loss and empirical loss functions as F D ( w,θ ) = E x ∼D [ f ( w,θ ; x )]

and F S ( w,θ ) = 1 n ∑ x ∈ S f ( w,θ ; x ) respectively. The boundedness assumption on g means f is L -Lipschitz. Concretely, ∀ w 1 , w 2 ∈ W and ∀ θ 1 , θ 2 ∈ Θ :

Under such assumptions, a solution for problem (1) always exists [Sio58], and is referred to as the saddle point . Further, given an SSP (1), we will denote a saddle point as [ w ∗ , θ ∗ ] .

The utility of an approximation to the saddle point is characterized by the strong SP-gap. Given a (randomized) algorithm A with output [ A w ( S ) , A θ ( S )] ∈ W × Θ , this is defined as

For notational convenience, we define the following function which is closely related to the SP-gap, ̂ Gap SP ( ¯ w, ¯ θ ) = max θ ∈ Θ { F D ( ¯ w,θ ) } -min w ∈W { F D ( w, ¯ θ ) } . Usefully, this function is known to be Lipschitz whenever f is Lipschitz.

Fact 1. ([BGM23]) If f is L -Lipschitz then ̂ Gap SP is √ 2 L -Lipschitz.

Finally, we define ℓ p /ℓ q saddle point problems as those which, in addition to the above, also have the following structure. Assume ∥ · ∥ w = ∥ · ∥ p and ∥ · ∥ θ = ∥ · ∥ q and that W and Θ have diameters bounded by B w and B θ with respect to ∥ · ∥ p and ∥ · ∥ q respectively. We assume that for any x ∈ X that f ( · , · ; x ) is L w Lipschitz in its first parameter w.r.t. ∥·∥ p and L θ Lipschitz in its second parameter w.r.t. ∥ · ∥ q . Note this implies an overall Lipschitz parameter, as per Eqn. (3), of L ≤ √ L 2 w + L 2 θ .

Stochastic Variational Inequalities. For SVIs, in addition to the assumption that the monotone operator is L -bounded, we will also assume it is β -Lipschitz. The objective for stochastic variational inequalities is to find an approximation of the (population) equilibrium point z ∗ , where z ∗ is characterized by Eqn. (2). We refer to such a solution as an equilibrium point. For approximate solutions, the quality of the approximation is characterized by the strong VI-gap:

Note that it is not true in general that the VI-gap bounds the SP-gap even when the monotone operator in question is the saddle operator of some convex-concave loss (see Fact 3 in Appendix A). However, in such a case it is true that the equilibrium point of the of the SVI is the saddle point of the corresponding SSP.

We say that an operator g is µ -strongly monotone if for any z 1 , z 2 ∈ Z that ⟨ g ( z 1 ; x ) -g ( z 2 ; x ) , z 1 -z 2 ⟩ ≥ µ 2 ∥ z 1 -z 2 ∥ .

Stability. Important to our analysis will be the notion of uniform stability [BE02].

Definition 1. A randomized algorithm A : X n ↦→W× Θ satisfies ∆ -uniform argument stability (UAS) if for any pair of adjacent datasets S, S ′ ∈ X n it holds that E A [ ∥A ( S ) -A ( S ′ ) ∥ ] ≤ ∆ .

The notion of strong convexity is often important in analyzing stability. A function ψ : Z ↦→ R is µ -strongly convex w.r.t. ∥·∥ if for any z, z ′ ∈ Z one has ψ ( z ) -ψ ( z ′ ) ≥ ⟨∇ ψ ( z ′ ) , z -z ′ ⟩ + µ 2 ∥ z -z ′ ∥ 2 . We call a function F : W× Θ ↦→ R µ -strongly-convex/strongly-concave (SC/SC) if for any θ ∈ Θ and w ∈ Θ the functions F ( · , θ ) and -F ( w, · ) are µ -strongly-convex.

Notably, adding a SC/SC regularizer leads to stability properties. The following fact results from a more general result for regularized SVIs; see Lemma 5 in Appendix A.

Lemma 1. Let ψ : W× Θ ↦→ R be µ -strongly-convex/strongly-concave w.r.t. ∥ · ∥ w and ∥ · ∥ θ . Then the algorithm which returns the saddle point of ( w,θ ) ↦→ 1 n ∑ z ∈ S f ( w,θ ; z ) + ψ ( w,θ ) is ( 2 L µn ) -UAS.

Differential Privacy (DP) [DMNS06]. An algorithm A is ( ϵ, δ ) -differentially private if for all datasets S and S ′ differing in one data point and all events E in the range of the A , we have, P ( A ( S ) ∈ E ) ≤ e ϵ P ( A ( S ′ ) ∈ E ) + δ .

2.1 Example of Non-Euclidean SSP

One important example of non-Euclidean SSPs arises from the problem of minimizing the worst case risk over multiple populations. This problem has arisen in group distributionally robust optimization to name just one of many applications [SGJ22, ZZZ + 24b, NMG24]. Let f : W×X ↦→ R and W have a standard ℓ p setup. Consider k distributions, D 1 , . . . , D k , and the goal of selecting a model w ∈ W which guarantees the lowest worst-case risk for the k distributions above:

where here ∆ denotes the standard k -dimensional simplex, and the equality above holds by the maximum principle for convex functions [Bau58]. Given that the feasible set for θ is a simplex, it is natural to endow this space with the ℓ 1 -geometry. We thus end up with a SSP problem in ℓ p /ℓ 1 setup.

3 A New Analysis for Recursive Regularization

Algorithm 1 Recursive Regularization: R SSP

Require: Dataset S ∈ X n , loss function f , subroutine A emp , regularization parameter λ ≥ Lκ B √ n

(where ∥ · ∥ 2 w and ∥ · ∥ 2 θ are 1 κ strongly convex), constraint set diameter B , Lipschitz constant L .

2: Let S 1 , ..., S T be a disjoint partition of S with each S t of size n ′ (which is always possible due to the condition on λ )

1: Let n ′ = n/ log 2 ( n ) , and T = log 2 ( L Bλ ) .

3: Let [ ¯ w 0 , ¯ θ 0 ] be any point in W× Θ

4: Define function ( w,θ, x ) ↦→ f (1) ( w,θ ; x ) = f ( w,θ ; x ) + 2 λ ∥ w -¯ w 0 ∥ 2 -2 λ ∥ ∥ θ -¯ θ 0 ∥ ∥ 2

5: for t = 1 to T do

w θ

7: Define ( w,θ, x ) ↦→ f ( t +1) ( w,θ ; x ) = f ( t ) ( w,θ ; x ) + 2 t +1 λ ∥ w -¯ w t ∥ 2 w -2 t +1 λ ∥ ∥ θ -¯ θ t ∥ ∥ 2 θ

6: [ ¯ w t , ¯ θ t ] = A emp ( S t , f ( t ) , [ ¯ w t -1 , ¯ θ t -1 ] , B 2 t )

8: end for

9: Output: [ ¯ w T , ¯ θ T ]

In this section, we present our modified recursive regularization algorithm, first developed in [AZ18] and extended to Euclidean SSPs in [BGM23]. We then discuss the key components of our analysis needed to obtain our results for non-Euclidean geometries. We conclude the section by applying our general result for recursive regularization to DP ℓ p /ℓ q -SSPs.

Algorithm Overview. As in [BGM23], our recursive regularization implementation, Algorithm 1, solves a series of regularized saddle point problems defined by f (1) , ..., f ( T ) . The saddle point problem defined in each round of Algorithm 1 is solved using some empirical subroutine, A emp . This subroutine takes as input a subset of the dataset, S t , the regularized loss function for that round, f ( t ) , a starting point, [ ¯ w t -1 , ¯ θ t -1 ] , and an upper bound on the expected distance to the empirical saddle

point of the problem defined by S t and f ( t ) . The exact implementation of A emp , Algorithm 3, will be discussed in the next section. Here, we focus on the guarantees of Recursive Regularization given that A emp satisfies a certain accuracy condition. At each round, the empirical subroutine A emp is required to find a point, [ ¯ w t , ¯ θ t ] , which is close (under ∥ · ∥ ) to the empirical saddle point. Because the scale of regularization doubles each round, this task becomes easier each round. Specifically, [BGM23] observed that implementations of A emp which satisfy the notion of relative accuracy succeed at finding such points.

Definition 2 ( ˆ α -relative accuracy) . Given a dataset S ′ ∈ X n ′ , loss function f ′ , and an initial point [ w ′ , θ ′ ] , we say that A emp satisfies ˆ α -relative accuracy w.r.t. the empirical saddle point [ w ∗ S ′ , θ ∗ S ′ ] of F ′ S ′ ( w,θ ) = 1 n ′ ∑ x ∈ S ′ f ′ ( w,θ ; x ) if, ∀ ˆ D > 0 , whenever E [ ∥ [ w ′ , θ ′ ] -[ w ∗ S ′ , θ ∗ S ′ ] ∥ ] ≤ ˆ D , the output [ ¯ w, ¯ θ ] of A emp satisfies E [ F ′ S ′ ( ¯ w,θ ∗ S ′ ) -F ′ S ′ ( w ∗ S ′ , ¯ θ ) ] ≤ ˆ D ˆ α .

In contrast to [BGM23], our algorithm uses more general regularization to ensure strongconvexity/strong-concavity with respect to the appropriate norm.

Guarantees of Recursive Regularization. Our general result for recursive regularization is stated as follows, and its full proof is given in Appendix B.2.

Theorem 1. Let A emp satisfy ˆ α -relative accuracy for any (5 L ) -Lipschitz loss function and dataset of size n ′ = n log( n ) and assume ∥·∥ 2 w and ∥·∥ 2 θ are 1 κ -strongly convex under ∥·∥ w and ∥·∥ θ respectively.

Then Algorithm 1, run with A emp as a subroutine and λ = 48 B ( ˆ ακ 2 + Lκ 3 / 2 √ n ′ ) , satisfies

The similarity of this result to [BGM23, Theorem 5] and our exposition thus far belies the difficulty of adapting their result to non-Euclidean setups. The key challenge addressed by the analysis of [BGM23] was that of generalization . In this regard, their key insight was to use McDiarmid style concentration bounds to show that the empirical saddle point obtains non-trivial guarantees on the strong gap. However, such concentration results critically rely on the fact that the underlying norm is Euclidean. A generalization of this concentration to, for example, the ℓ 1 setup, necessarily incurs an additional √ d factor [Pan08]. Thus, a fundamentally new analysis is needed. One should also note that the squared norms ∥·∥ 2 w may not be strongly convex for certain norms, such as ∥·∥ 1 . Regardless, in some such cases, we can still leverage this result by modifying the underlying problem, as we will show in Section 4.

Key Proof Ideas. We circumvent the above issues by providing a fundamentally new generalization analysis for the intermediate iterates of recursive regularization. Specifically, our analysis avoids entirely any analysis of the strong gap at intermediate stages of the algorithm. Instead, we introduce two new functions, which are similar in nature to the quantity used in the definition of relative accuracy, but are taken with respect to the population saddle point. For t ∈ [ T ] , define F ( t ) D ( w,θ ; x ) := E x ∼D [ f ( t ) ( w,θ ; x ) ] and let [ w ∗ t , θ ∗ t ] be its saddle point; define F ( t ) S := 1 n ′ ∑ x ∈ S t f ( t ) ( w,θ ; x ) . We are interested in the functions,

Notably, the strong-convexity/strong-concavity of F ( t ) D means that a bound on H ( t ) D ([ w,θ ]) yields a bound on ∥ [ w,θ ] -[ w ∗ t , θ ∗ t ] ∥ . Ultimately, finding a point sufficiently close to [ w ∗ t , θ ∗ t ] at each round is all recursive regularization needs to succeed. The question then, is how to obtain guarantees on H ( t ) D . We accomplish this via a stability-implies-generalization argument.

Lemma 2. Let f : Z × X ↦→ R be L -Lipschitz. Let [ w ∗ , θ ∗ ] ∈ Z be the population saddle point. For any x ∈ X define h ([ w,θ ]; x ) = f ( w,θ ∗ ; x ) -f ( w ∗ , θ ; x ) . For S ∼ D n , let H S ( z ) = 1 n ∑ x ∈ S h ( z ; x ) and H D ( z ) = E x ∼D [ h ( z ; x )] . Then for any ∆ -UAS algorithm, A , one has E S,A [ H D ( A ( S )) -H S ( A ( S ))] ≤ 2∆ L .

The proof relies on two main observations. First, it is easy to see that because f is Lipschitz, then h is also Lipschitz. Then, because H D and H S can be written as the expectation of f w.r.t. z ∼ D and z ∼ Unif ( S ) respectively, we can apply standard stability-implies-generalization results to h to obtain the claimed result [BE02]. We provide a full proof in Appendix B.1. This analysis bypasses difficulties of working directly with ̂ Gap SP experienced by [OPZZ22, BGM23] and other works since, in general, there is no function h such that ̂ Gap SP ( z ) = E x ∼D [ h ( z ; x )] . Note also it is important that we have defined h ( z ; x ) w.r.t. to the data independent point [ w ∗ t , θ ∗ t ] . Were [ w ∗ t , θ ∗ t ] to depend on S , this result would not hold.

Because H ( t ) S uses the population saddle point in its definition, it may not be immediately clear how one could first minimize H ( t ) S . Direct access to H ( t ) S is in fact not possible without knowledge of [ w ∗ t , θ ∗ t ] , which the algorithm does not have. In this regard, we observe that algorithms which minimize the empirical gap are powerful enough to minimize H ( t ) S , even without knowledge of [ w ∗ t , θ ∗ t ] , since

Particular to our analysis, we will leverage the fact that the exact empirical saddle point is ( L λn ) -stable and has an empirical gap of 0 .

4 Optimal Rates for Private ℓ p /ℓ q Saddle Point Problems

Setup. In this section, we apply Theorem 1 to obtain results for ℓ p /ℓ q saddle point problems. In order to do this, we will apply recursive regularization using norms slightly different than the ℓ p and ℓ q norms. Specifically, to solve an ℓ p /ℓ q SSP, we define ¯ p = max { p, 1 + 1 log( d ) } and ¯ q = max { q, 1 + 1 log( d ) } and will apply recursive regularization with ∥ · ∥ w = 1 B w ∥ · ∥ ¯ p and ∥ · ∥ θ = 1 B w ∥ · ∥ ¯ p . We also have ∥ · ∥ ¯ p ≤ ∥ · ∥ 1 ≤ d 1 -1 / ¯ p ∥ · ∥ ¯ p ≤ 2 ∥ · ∥ ¯ p . Thus, under these norms we have diameter bound B 2 = 1 and Lipschitz constant L 2 ≤ 4 B 2 w L 2 w +4 B 2 θ L 2 θ [NJLS09]. Further the strong convexity assumption needed by Theorem 1 is satisfied with κ = max { 1 ¯ p -1 , 1 ¯ q -1 } . This is because for any p > 1 , 1 2 ∥ · ∥ 2 p is ( p -1) -strongly convex w.r.t. ∥ · ∥ p [Bec17].

Private Algorithm Satisfying Relative Accuracy. To apply Theorem 1, we must construct an algorithm satisfying relative accuracy and ( ϵ, δ ) -DP. For this, we use the stochastic mirror prox algorithm of [JNT11], Algorithm 2. This algorithm will also have application in our analysis of SVIs later on.

Algorithm 2 Stochastic Mirror Prox

Require: Learning rate η , Operator oracle O , Initial point z 0 ∈ Z , Regularization function ψ

minimized at z 0 , Iterations T 1: for t = 1 . . . T do 2: ˜ z t = arg min u ∈Z { ψ ( u ) + ⟨ η O ( z t -1 ) -∇ ψ ( z t -1 ) , u ⟩} 3: z t = arg min u ∈Z { ψ ( u ) + ⟨ η O (˜ z t ) -∇ ψ ( z t -1 ) , u ⟩} 4: end for 5: SSP output: 1 T ∑ T t =1 z t 6: SVI output: z t ∗ for t ∗ ∼ Unif ([ T ])

This algorithm takes as input a stochastic oracle for the saddle operator of f , O , and a strongly convex regularizer, ψ . We leverage this algorithm by constructing a differentially private version of the operator oracle O , and taking as output the average iterate ¯ z = 1 T ∑ T t =1 z t . We make the oracle private by adding Gaussian noise to minibatch estimates of the saddle operator. It is then easy to show the whole algorithm is private by composition results and the post processing properties of differential privacy. We defer these details to Appendix C.2, and here state the final relative accuracy bound.

Algorithm 3 Recursive Regularization for SVIs: R SVI

Require: Dataset S , monotone operator g , Subroutine A emp , regularity parameter κ > 0 , regularization parameter λ ≥ L √ κ B √ n , constraint set diameter B , Strongly monotone operator ρ

1: Let n ′ = n/ log 2 ( n ) , and T = log 2 ( L κBλ ) .

2: Let S 1 , ..., S T be a disjoint partition of S with each S t of size n ′ (always possible due to the condition on λ )

3: ¯ z 0 be any point in Z

4: Define function ( z, x ) ↦→ g (1) ( z ; x ) = g ( z ; x ) + 2 λ · ρ ( z -¯ z 0 )

5: for t = 1 to T do

6: ¯ z t = A emp ( S t , g ( t ) , ¯ z t -1 , B 2 t )

7: Define function ( z, x ) ↦→ g ( t +1) ( z ; x ) = g ( t ) ( z ; x ) + 2 t +1 λ · ρ ( z -¯ z t )

8: end for

9: Output: ¯ z T

Lemma 3. Under the setup described above, there exists an ( ϵ, δ ) -DP algorithm which satisfies ˆ α -relative accuracy with parameter ˆ α = O ( √ κ ( B 2 w L 2 w + B 2 θ L 2 θ ) ( √ d log(1 /δ )˜ κ nϵ + 1 √ n ) ) and runs in O ( min { √ κn 2 ϵ 1 . 5 log 2 ( n ) √ d log(1 /δ )˜ κ , √ κn 3 / 2 log 3 / 2 ( n ) }) number of gradient evaluations, where ˜ κ = 1 + 1 { p < 2 ∨ q < 2 } · log( d ) .

Main Result for DP ℓ p /ℓ q SSPs. Applying now the result of recursive regularization, Theorem 1, we obtain the optimal rate for ℓ p /ℓ q SSPs (up to logarithmic factors). Recall under our chosen norm that B ≤ 1 and L 2 ≤ 4 B 2 w L 2 w +4 B 2 θ L 2 θ . Further, the privacy of Algorithm 1 follows from the privacy of A emp and the parallel composition and post processing properties of differential privacy.

Corollary 1. There exists an Algorithm, R , which is ( ϵ, δ ) -DP, has number of gradient evaluations bounded by O ( min { √ κn 2 ϵ 1 . 5 log( n ) √ d log(1 /δ )˜ κ , √ κn 3 / 2 √ log( n ) }) , and satisfies (up to log( n ) factors),

( κ is at most log( d ) .)

Note that in the ℓ 2 /ℓ 2 -setting κ = 1 and the above exactly recovers the result of [BGM23]. We further recall that the near optimality of this result is established by existing lower bounds for stochastic minimization, which is a special case of SSPs [BFTT19, BGN21].

5 Extension to Variational Inequalities

In this section, we start by discussing the modifications that must be made to the recursive regularization algorithm to handle the more general structure of SVIs. We then discuss key ideas in the analysis and how to apply the algorithm to SVIs in the ℓ p setting. We recall that we here assume the operator g is monotone, L -bounded and β -Lipschitz.

5.1 Recursive Regularization Algorithm for SVIs

Algorithm 3 bears many similarities to Algorithm 1. Most notable among the differences is that we here regularize with a strongly monotone operator ρ instead of a strongly-convex/strongly-concave function. In our eventual application we will use ρ = ∇ ( 1 2 ∥ · ∥ 2 ) , but strictly we only require that ρ satisfies the following.

Assumption 1. For κ > 0 , ρ : Z ↦→ R d is 1 κ -strongly monotone w.r.t. ∥·∥ and satisfy ∥ ρ ( z ) ∥ ∗ ≤ ∥ z ∥ for all z ∈ Z .

When ρ = ∇ ( 1 2 ∥ · ∥ 2 ) , the second half of the condition is always guaranteed by the properties of the dual norm (see Fact 2). When ρ satisfies Assumption 1, we obtain the following guarantee.

Theorem 2. Let A emp satisfy ˆ α -relative stationarity for any (5 L ) -bounded monotone operator and dataset of size n ′ = n log( n ) . Let ρ satisfy Assumption 1. Then Algorithm 1, run with A emp as a subroutine and λ = 48 B ( ˆ ακ 3 + ( βB + L ) κ 2 √ n ′ ) , satisfies

The full proof is in Appendix B.2. We discuss the key ideas in the following subsection.

5.2 Analysis Idea

Unfortunately, the analysis used for stochastic saddle point problems does not easily extend to variational inequalities due to the fact that the VI-gap and SP-gap behave in fundamentally different ways. Even though SSPs are a special case of SVIs (when the operator in question is the saddle operator of the loss function), a bound on the VI-gap does not imply a bound on the SSP-gap. Further, a natural attempt to extend the notion of relative accuracy (Definition 3) to SVIs by asking A emp to bound ⟨ G D ( z ∗ S ) , A ( S ) -z ∗ S ⟩ does not work, because such a term does not yield an upper bound on the distance ∥A ( S ) -z ∗ S ∥ . A similar problem holds for generalization measures in Eqn. (6).

A New Empirical Accuracy Measure. Motivated by the above issues, we introduce a new relative accuracy measure for our analysis of SVIs. Importantly, this notion will allow us to bound the distance between the output of A emp and the empirical equilibrium point of the strongly monotone operator created at each round of the recursive regularzation algorithm.

Definition 3 ( ˆ α -relative stationarity) . Given a dataset S ′ ∈ X n ′ , operator g ′ , and an initial point z ′ , we say that A emp satisfies ˆ α -relative stationarity w.r.t. to the empirical equilibrium z ∗ S ′ of G S ′ ( z ) = 1 n ′ ∑ x ∈ S ′ g ′ ( z ; x ) , if, ∀ ˆ D > 0 , whenever E [ ∥ z ′ -z ∗ S ′ ∥ ] ≤ ˆ D , the output ¯ z of A emp satisfies E [ ⟨ G (¯ z ) , ¯ z -z ∗ S ′ ⟩ ] ≤ ˆ D ˆ α .

Modified Generalization Measure. The generalization measure we use can be modified in a similar fashion.

Lemma 4. Let g ( z ; x ) = g 1 ( z ; x ) + g 2 ( z ) such that g 1 : Z × X ↦→ R d is L -bounded and β -Lipschitz with respect to z and g 2 : Z ↦→ R d is any (data indepedent) operator. Let z ∗ ∈ Z be its population equilibrium point. For any x ∈ X , define h ( z ; x ) = ⟨ g ( z ; x ) , z -z ∗ ⟩ . For S ∼ D n , let H S ( z ) = 1 n ∑ x ∈ S h ( z ; x ) and H D ( z ) = E x ∼D [ h ( z ; x )] . Then for any ∆ -UAS algorithm, A , one has E [ H D ( A ( S )) -H S ( A ( S ))] ≤ ∆( βB + L ) .

The proof is similar to that of Lemma 2, but must account for additional complications. First, the function h may not be Lipschitz if the regularizer, represented by g 2 above, is not Lipschitz. This happens, for example, in the ℓ 1 setting. Thus, we must decompose h in the stability-impliesgeneralization analysis and handle the non-Lipschitz, but data-independent, term g 2 separately. Then, the Lipschitzness of the remainder is established using that fact that g 1 is both bounded and Lipschitz. The full proof is in Appendix D.1.

5.3 Application to DP variational inequalities in the ℓ p setting

In order to apply Theorem 2 to SVIs with an ℓ p setup, we pick ∥ · ∥ = 1 B ∥ z ∥ ¯ p where ¯ p = max { p, 1 + 1 log( d ) } 2 . We will use the regularizer,

Note the above operator is uniquely defined since ¯ p > 1 , and thus ∥ · ∥ 2 ¯ p is differentiable [Gui09]. This choice of ρ satisfies Assumption 1 with parameter κ = 1 ¯ p -1 . To see this, first observe that

for any ¯ p > 1 , 1 2 ∥ · ∥ 2 ¯ p is (¯ p -1) -strongly convex w.r.t. ∥ · ∥ ¯ p and the gradient operator of a differentiable µ -strongly convex function is µ -strongly monotone. Second, for any norm and it's dual ∥∇ ( 1 2 ∥ z ∥ 2 ) ∥ ∗ ≤ ∥ z ∥ for all z (see Fact 2 in Appendix A).

To obtain relative stationarity guarantees, we again apply Algorithm 2 with a differentially private oracle for the empirical operator G S . The proof falls out of our existing analysis for stochastic mirror prox given in Appendix C.2. Specifically, Theorem 4 in the case where Θ is the empty set implies there exists an ( ϵ, δ ) -DP implementation of mirror prox which satisfies ˆ α -relative stationarity with

We here highlight one notable difference that arises with the algorithm. Typically, after running an algorithm like stochastic mirror prox for t iterations, one obtains a bound on the quantity E [ 1 T ∑ T t =1 ⟨ G S ( z t ) , z t -z ⟩ ] . Analysis then proceeds to bound the (empirical) VI-gap of the average iterate by leveraging monotonicity of the operator. However, this application of monotonicity does not help for the purposes of relative stationarity. For this reason, we instead select an iterate uniformly at random. We note this step is nonstandard as such a selection does not necessarily yield any bound on the (empirical) VI-gap.

Main Result for ℓ p DP SVIs. Using Algorithm 3 and the implementation of A emp described above, we ultimately obtain the following result as a corollary of Theorem 2 and Theorem 4. Recall under our choice of norm we have diameter bound 1 and operator bound BL . Further, we can leverage existing lower bounds to show that this rate is near optimal; more details are available in Appendix E.

Corollary 2. There exists an Algorithm, R , which is ( ϵ, δ ) -DP, has gradient evaluations bounded by

where κ = 1 max { p, 1+ 1 log( d ) } -1 is at most log( d ) and ˜ κ = 1 + 1 { p < 2 } · log( d ) .

Near Linear Time Algorithm for the ℓ 2 Setting. Because we assume the operator is Lipschitz, in the ℓ 2 setting, we can leverage existing accelerated optimization techniques to achieve a near linear time version of A emp , in a similar fashion to [ZTOH22, BGM23]. Specifically, using the accelerated SVRG algorithm of [PB16] and Gaussian noise it is possible to obtain the rate in Eqn. (8) (with κ = ˜ κ = 1 ) in O ( n + βn log( n/δ )) gradient evaluations. We provide full details in Appendix D.3.

Acknowledgments and Disclosure of Funding

R. Bassily's and M. Menart's research was supported by NSF CAREER Award 2144532 and, in part, by NSF Award 2112471. C. Guzmán's research was partially supported by INRIA Associate Teams project, ANID FONDECYT 1210362 grant, ANID Anillo ACT210005 grant, and National Center for Artificial Intelligence CENIA FB210017, Basal ANID.

References

[ABD + 18] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reductions approach to fair classification. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 60-69. PMLR, 10-15 Jul 2018.

[ACG + 16] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. CCS '16, page 308-318, New York, NY, USA, 2016. Association for Computing Machinery.

[AFKT21] Hilal Asi, Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: Optimal rates in ℓ 1 geometry. In International Conference on Machine Learning , 2021.

[AZ18] Zeyuan Allen-Zhu. How to make the gradients small stochastically: Even faster convex and nonconvex sgd. Advances in Neural Information Processing Systems , 31, 2018.

[Bau58] Heinz Bauer. Minimalstellen von funktionen und extremalpunkte. Archiv der Mathematik , 9(4):389-393, 1958.

[BE02] Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research , 2:499-526, 2002.

[Bec17] Amir Beck. First-Order Methods in Optimization . SIAM-Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2017.

[BFTT19] Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. Private stochastic convex optimization with optimal rates. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 11279-11288, 2019.

[BG23] Digvijay Boob and Cristóbal Guzmán. Optimal algorithms for differentially private stochastic monotone variational inequalities and saddle-point problems. Mathematical Programming , pages 1-43, 2023.

[BGM23] Raef Bassily, Cristóbal Guzmán, and Michael Menart. Differentially private algorithms for the stochastic saddle point problem with optimal rates for the strong gap. In Gergely Neu and Lorenzo Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory , volume 195 of Proceedings of Machine Learning Research , pages 2482-2508. PMLR, 12-15 Jul 2023.

[BGN21] Raef Bassily, Cristobal Guzman, and Anupama Nandi. Non-euclidean differentially private stochastic convex optimization. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 474-499. PMLR, 15-19 Aug 2021.

[BST14] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In IEEE 55th Annual Symposium on Foundations of Computer Science (FOCS 2014). (arXiv preprint arXiv:1405.7085) , pages 464-473. 2014.

[DMNS06] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pages 265-284. Springer, 2006.

[DSL + 18] Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. SBEED: Convergent reinforcement learning with nonlinear function approximation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1125-1134. PMLR, 10-15 Jul 2018.

[FKT20] Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 439-449, 2020.

[GGP24] Tomás González, Cristóbal Guzmán, and Courtney Paquette. Mirror descent algorithms with nearly dimension-independent rates for differentially-private stochastic saddle-point problems. CoRR , abs/2403.02912, 2024.

[Gui09] A Guirao. Uniformly convex functions on banach spaces. Proceedings of the American Mathematical Society , 137(3):1081-1091, 2009.

[JKT12] Prateek Jain, Pravesh Kothari, and Abhradeep Thakurta. Differentially private online learning. In 25th Annual Conference on Learning Theory (COLT) , pages 24.1-24.34, 2012.

[JN19] Anatoli B. Juditsky and Arkadii S. Nemirovski. Signal Recovery by Stochastic Optimization. Automation and Remote Control / Avtomatika i Telemekhanika , 80(10):1878-1893, October 2019.

[JNT11] Anatoli Juditsky, Arkadi Nemirovski, and Claire Tauvel. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems , 1(1):17 - 58, 2011.

[JT14] Prateek Jain and Abhradeep Thakurta. (near) dimension independent risk bounds for differentially private learning. In ICML , 2014.

[KLL21] Janardhan Kulkarni, Yin Tat Lee, and Daogao Liu. Private non-smooth erm and sco in subquadratic steps. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 4053-4064. Curran Associates, Inc., 2021.

[LYYY21] Yunwen Lei, Zhenhuan Yang, Tianbao Yang, and Yiming Ying. Stability and generalization of stochastic gradient methods for minimax problems. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 6175-6186. PMLR, 18-24 Jul 2021.

[MSS19] Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 4615-4625. PMLR, 09-15 Jun 2019.

[NJLS09] Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and And Shapiro. Robust stochastic approximation approach to stochastic programming. Society for Industrial and Applied Mathematics , 19:1574-1609, 01 2009.

[NMG24] Quan Nguyen, Nishant A. Mehta, and Cristóbal Guzmán. Beyond minimax rates in group distributionally robust optimization via a novel notion of sparsity, 2024.

[NY78] Arkadi Nemirovski and D Yudin. On cezari's convergence of the steepest descent method for approximating saddle point of convex-concave functions. In Soviet Mathematics. Doklady , volume 19, pages 258-269, 1978.

[OPZZ22] Asuman Ozdaglar, Sarath Pattathil, Jiawei Zhang, and Kaiqing Zhang. What is a good metric to study generalization of minimax learners? In Advances in Neural Information Processing Systems , volume 35. Curran Associates, Inc., 2022.

[Pan08] Liam Paninski. A coincidence-based test for uniformity given very sparsely sampled discrete data. IEEE Transactions on Information Theory , 54:4750-4755, 2008.

[PB16] Balamurugan Palaniappan and Francis Bach. Stochastic variance reduction methods for saddle-point problems. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

[SGJ22] Tasuku Soma, Khashayar Gatmiry, and Stefanie Jegelka. Optimal algorithms for group distributionally robust optimization and beyond. arXiv preprint arXiv:2212.13669 , 2022.

[Sio58] Maurice Sion. On general minimax theorems. Pacific Journal of Mathematics , 8(1):171 - 176, 1958.

[TTZ15] Kunal Talwar, Abhradeep Thakurta, and Li Zhang. Nearly optimal private lasso. In NIPS , 2015.

[WM19] Robert Williamson and Aditya Menon. Fairness risk measures. In International Conference on Machine Learning , pages 6786-6797. PMLR, 2019.

[YHL + 22] Zhenhuan Yang, Shu Hu, Yunwen Lei, Kush R Vashney, Siwei Lyu, and Yiming Ying. Differentially private sgda for minimax problems. In James Cussens and Kun Zhang, editors, Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence , volume 180 of Proceedings of Machine Learning Research , pages 21922202. PMLR, 01-05 Aug 2022.

[YLMJ22] Yaodong Yu, Tianyi Lin, Eric Mazumdar, and Michael I. Jordan. Fast distributionally robust learning with variance reduced min-max optimization. AISTATS , 2022.

[ZB24] Xinyu Zhou and Raef Bassily. Differentially private worst-group risk minimization. CoRR , abs/2402.19437, 2024.

[ZHWZ21] Junyu Zhang, Mingyi Hong, Mengdi Wang, and Shuzhong Zhang. Generalization bounds for stochastic saddle point problems. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 568-576. PMLR, 13-15 Apr 2021.

[ZL15] Yuchen Zhang and Xiao Lin. Stochastic primal-dual coordinate method for regularized empirical risk minimization. In International Conference on Machine Learning , pages 353-361. PMLR, 2015.

[ZTOH22] Liang Zhang, Kiran Koshy Thekumparampil, Sewoong Oh, and Niao He. Bring your own algorithm for optimal differentially private stochastic minimax optimization. In Advances in Neural Information Processing Systems , volume 35. Curran Associates, Inc., 2022.

[ZZZ + 24a] Lijun Zhang, Peng Zhao, Zhen-Hua Zhuang, Tianbao Yang, and Zhi-Hua Zhou. Stochastic approximation approaches to group distributionally robust optimization. CoRR , 2024.

[ZZZ + 24b] Lijun Zhang, Peng Zhao, Zhen-Hua Zhuang, Tianbao Yang, and Zhi-Hua Zhou. Stochastic approximation approaches to group distributionally robust optimization, 2024.

A Supplementary Lemmas

Fact 2. For any z , it holds that ∥∇ ( 1 2 ∥ z ∥ 2 ) ∥ ∗ ≤ ∥ z ∥ .

Proof. By the chain rule we have ∇ ( 1 2 ∥ z ∥ 2 ) = ∥ z ∥ · ∇ ( ∥ z ∥ ) and so ∥∇ ( 1 2 ∥ z ∥ 2 ) ∥ ∗ = ∥ z ∥ · ∥∇ ( ∥ z ∥ ) ∥ ∗ . Thus, it only remains to show that ∥∇ ( ∥ z ∥ ) ∥ ∗ ≤ 1 . By the definition of the dual norm we have ∥∇ ( ∥ z ∥ ) ∥ ∗ = max v : ∥ v ∥≤ 1 {⟨∇ ( ∥ z ∥ ) , v ⟩} . Further, by the definition of the subgradient we have for any z ′ that ∥ z ′ ∥ - ∥ z ∥ ≥ ⟨∇ ( ∥ z ∥ ) , z ′ -z ⟩ . Substituting v = z ′ -z , we have

as desired.

Lemma 5. Let µ, λ > 0 and ρ be µ -strongly monotone w.r.t. ∥ · ∥ . Then for any point z 0 ∈ Z , the algorithm which outputs the unique equilibrium of G S ( z ) + λ µ ( ρ ( z ) -ρ ( z 0 )) is ( 2 L µλn ) -uniform argument stable w.r.t. S .

Note without loss of generality we can always choose z 0 to be the point such that ρ ( z 0 ) = 0 , which is guaranteed to exist since ρ is strongly monotone. Thus this result also holds for regularization of the form G S ( z ) + λ · ρ ( z ) . Further, since the saddle operator of a strongly-convex/strongly-concave loss is strongly monotone, and the resulting SSP shares its equilibrium point with the corresponding SVI, Lemma 1 given in the preliminaries is also established through this result.

Proof. Let z λ and z ′ λ denote the equilibrium points of the regularized operators w.r.t. to adjacent datasets S and S ′ respectively. By the equilibrium condition we have for any z ∈ Z

Since ρ is a 1 -strongly monotone operator we have

Now we can use the monotonicity of G to derive,

Above ( i ) comes from the monotonicity of G , step ( ii ) comes from Hölder's inequality, and step ( iii ) comes from the fact that S and S ′ differ in at most one point. Simple algebra now obtains ∥ z λ -z ′ λ ∥ ≤ 2 L µλn .

Fact 3. There exists a differentiable convex-concave loss, distribution D , and point z ′ such that when the operator is the saddle operator of the loss, Gap VI ( z ) < Gap SP ( z ) .

Proof. We show that the VI-gap is upper bounded by the excess risk of some convex function with operator g ( z ) = ∇ f ( z ) . Since stochastic convex optimization is a special case of SSPs, this shows there exist scenarios where Gap SP ≰ Gap VI . Specifically, when the dual player space Θ is singleton, the SP-gap becomes the excess risk.

Let f ( z ; x ) = z 2 be defined over z ∈ [0 , 1] , and so it does not matter what the distribution or data domain is. Note that Gap VI becomes

Note that in this case the SP-gap is just the excess risk, Gap SP ( z ) = z 2 -min z ∈ [0 , 1] { z 2 } = z 2 . Consider the point z = 1 . It is easy to show that Gap VI (1) = 0 . 5 but Gap SP (1) = 1 , proving the claim.

B Missing Results from Section 3

B.1 Proof of Lemma 2

Lemma 6. (Restatement of Lemma 2) Let f : Z × X ↦→ R be L -Lipschitz. Let [ w ∗ , θ ∗ ] ∈ Z be the population saddle point. For any x ∈ X define h ([ w,θ ]; x ) = f ( w,θ ∗ ; x ) -f ( w ∗ , θ ; x ) . For S ∼ D n , let H S ( z ) = 1 n ∑ x ∈ S h ( z ; x ) and H D ( z ) = E x ∼D [ h ( z ; x )] . Then for any ∆ -UAS algorithm, A , one has E [ H D ( A ( S )) -H S ( A ( S ))] ≤ 2∆ L .

Proof. This result follows simply from two facts. First, because f is an L -Lipschitz function for any x ∈ X , we can show that h is L -Lipschitz. For any [ w,θ ] , [ w ′ , θ ′ ] ∈ Z observe,

The rest of the proof essentially follows from the stability implies generalization proof of [BE02] since H D and H S have a statistical form w.r.t. h . In more detail, for any i ∈ [ n ] denote S ( i ) as the dataset which replaces the i 'th datapoint of S , x i , with a fresh sample from D , x ′ . We have the following:

The last step follows from the previously established Lipschitzness property of h .

B.2 Convergence of Recursive Regularization for SSPs

We will first prove the following more general version statement of Theorem 1.

Theorem 3. Let λ ≥ 48 Lκ 3 / 2 B √ n ′ and A emp be such that for all t ∈ [ T ] it holds that E [ ∥ ∥ ¯ z t -z ∗ S,t ∥ ∥ 2 ] ≤ B 2 12 · 2 2 t κ 3 / 2 . Then Recursive Regularization satisfies

To prove this result, it will be helpful to first show several intermediate results. We start by defining several useful quantities. Define {F t } T t =0 as the filtration where F t is the sigma algebra induced by all randomness up to ¯ z t . For notational convenience we denote f (0) ( w,θ ; x ) = f ( w,θ ; x ) . Then for every t ∈ { 0 , 1 , ..., T } we define

z ∗ t = [ w ∗ t , θ ∗ t ] : saddle point of F ( t ) D ( w,θ ) := E x ∼D [ f ( t ) ( w,θ ; x ) ] ;

H ( t ) D ([ w,θ ]) := F ( t ) D ( w,θ ∗ t ) -F ( t ) D ( w ∗ t , θ ) : the relative accuracy function w.r.t. the population loss and population saddle point; and,

H ( t ) S ([ w,θ ]) := F ( t ) S ( w,θ ∗ t ) -F ( t ) S ( w ∗ t , θ ) : the relative accuracy function w.r.t. the empirical loss and population saddle point

We recall the generalization properties of H ( t ) S and H ( t ) D shown in Lemma 6. Crucially, the power of H ( t ) D is that it bounds the distance of a point to z ∗ t .

Fact 4 ([ZHWZ21], Theorem 1) . Let F : Z ↦→ R d be a γ -SC/SC function and let [ w ∗ , θ ∗ ] be the saddle point. Then ∥ [ w,θ ] -[ w ∗ , θ ∗ ] ∥ 2 ≤ 2( F ( w,θ ∗ ) -F ( θ ∗ ,w )) λ .

We now establish two distance inequalities which will be used when analyzing the final gap bound in Theorem 3. The first inequality below bounds the distance of the output of the t -th round to the equilibrium of G ( t ) D . The second inequality bounds how far the population equilibrium moves after another regularization term is added.

Lemma 7. Assume the conditions of Theorem 3 hold. Then for every t ∈ [ T ] , the following holds

Wenote that in contrast to [BGM23] and other analyses of recursive regularization, we define Property P.2 to bound E [∥ ∥ z ∗ t -z ∗ t -1 ∥ ∥ ] instead of E [ ∥ z ∗ t -¯ z t -1 ∥ ] . In [BGM23], the latter choice led to a need to bound E [ ̂ Gap SP (¯ z t -1 ) ] for all t ∈ [ T ] , which our analysis avoids. In particular, one can observe how the derivation of Eqn. (11) in the proof changes when z ∗ t -1 is replaced with ¯ z t -1 .

Proof of Lemma 7. We will prove both properties via induction on B 1 , ..., B T . Specifically, for each t ∈ [ T ] we will introduce two terms E t and F t , , and show that these terms are bounded if the bound on B t holds and that B t holds if E t -1 and F t -1 are bounded. Property P.1 is then established as a result of the fact that E [ ∥ ¯ z t -z ∗ t ∥ 2 ] ≤ 2( E t + F t ) . Note that B 1 holds as the base case because E [ ∥ z ∗ 1 -z ∗ 0 ∥ 2 ] ≤ B 2 .

Property P.1: We here prove that if B t is sufficiently bounded, then E t and F t are bounded where for t ∈ [ T ] we define

Additionally, this will establish property P.1 because for any t ∈ [ T ] it holds that,

The second inequality comes from the strong monotonicity of the operator (see Fact 4).

Note that E t is bounded by the assumption made in the statement of the theorem statement. We thus turn our attention towards bounding F t . We have

Inqeuality ( i ) comes from the fact that stability implies generalization for H ( t ) , Lemma 2. Note the algorithm which outputs this exact equilibrium point is L 2 2 t λn ′ stable (see Lemma 5/Assumption 1). Step ( ii ) uses the fact that z ∗ S,t is the exact saddle point of the regularized objective, and so for any [ w,θ ] ∈ Z , F ( t ) S ( w ∗ S,t , θ ) -F ( t ) S ( w,θ ∗ S,t ) ≤ 0 . The final inequality uses the setting of λ .

We thus have a final bound 2( E t + F t ) ≤ B 2 2 2 t .

Property P.2: Now assume B t -1 holds. We have

Inequality ( i ) above comes from removing negative terms and the fact that z ∗ t -1 is the saddle point w.r.t. F ( t -1) D . Using the induction argument we then complete the bound with the following:

We now turn to analyzing the utility of the algorithm to complete the proof.

proof of Theorem 3. Using the fact that ̂ Gap SP is √ 2 L -Lipschitz and property P.1 , we have

What remains is showing E [ ̂ Gap SP ( w ∗ T , θ ∗ T ) ] is ˜ O ( B ˆ α + BL √ n ′ ) . Let w ′ = arg min θ ∈ Θ F D ( w,θ ∗ T ) and

θ ′ = arg max w ∈W F D ( w ∗ T , θ ) . Using the fact that F D is convex-concave we have

where G D is the population loss saddle operator. Further by the definition of F ( T ) and denoting G ( T ) D as the saddle operator for F ( T ) D we have

Thus plugging the above into Eqn. (13) we have

Above, the second inequality comes from the first order optimally conditions for [ w ∗ T , θ ∗ T ] , the third from Cauchy Schwartz and a triangle inequality. The last inequality comes from the relationship between a norm and its dual, see Fact 2.

Taking the expectation on both sides of the above we have the following derivation,

Above, ( i ) and the following inequality both come from the triangle inequality. Equality ( ii ) is obtained by rearranging the sums. Inequality ( iii ) comes from applying properties P.1 and P.2 proved above. The last equality comes from the setting of λ and T .

Now using this result in conjunction with Eqn. (12) we have

Finally, we prove Theorem 1 leveraging the relative accuracy assumption.

Proof of Theorem 1. First, observe that under the setting of λ = 48 B ( ˆ ακ 2 + Lκ 3 / 2 √ n ′ ) used in the theorem statement that log( n ) B 2 λ = O ( log( n ) B ˆ ακ 2 + log 3 / 2 ( n ) BLκ 3 / 2 √ n ) . Thus what remains is to show that the distance condition required by Theorem 3 holds. That is, we now show that if A emp satisfies ˆ α -relative accuracy, then for all t ∈ [ T ] it holds that E [ ∥ ∥ ¯ z t -z ∗ S,t ∥ ∥ 2 ] ≤ B 2 12 · 2 2 t κ .

To prove this property, we must leverage the induction argument made by Lemma 7. Specifically, to prove the condition holds for some t ∈ [ T ] , assume B 2 t = E [∥ ∥ z ∗ t -z ∗ t -1 ] ∥ ∥ ] 2 ≤ B 2 2 2( t -1) (recall the base case for t = 1 trivially holds). As shown in the proof of Lemma 7, this implies that the quantities E t , F t (as defined in 9) are bounded by B 2 1152 · 2 2 t . We thus have

where B t is as defined in property P.2 . Inequality ( i ) comes from the strong monotonicity of G ( t ) S , Fact 4. Inequality ( iii ) comes from the setting λ ≥ 48ˆ ακ 2 /B . Inequality ( ii ) comes from the ˆ α -relative accuracy assumption on A emp , which holds so long as the expected distance is sufficiently bounded and each regularized loss is ( 5 L )-Lipschitz. In this regard, note that

Further, each f ( t ) is 5 L -Lipschitz. We can see that,

C Missing Results from Section 4

C.1 General Guarantee for Stochastic Mirror Prox

We start with the follow general statement regarding the stochastic mirror prox algorithm applied to monotone operators. Notable for our purpose of solving SSPs is that the saddle operator of a convex-concave function is a monotone operator, but the following holds for any monotone operator.

Lemma 8 (Implicit in [JNT11], Theorem 1) . Let ψ : Z ↦→ R be any non-negative function which is 1 -strongly convex w.r.t. ∥·∥ . Assume ∀ t ∈ [ T ] that E [ O ( z t )] = G ( z t ) and E [ ∥O ( z t ) -G ( z t ) ∥ 2 ∗ ] ≤ τ 2 .

Then for for any z ∈ Z Algorithm 2 satisfies

The above is slightly different than the statement in [JNT11], but can be easily extracted from their proof. Let V ψ denote the Bregman divergence w.r.t. ψ ; i.e. V ψ ( z, z ′ ) = ψ ( z ) -ψ ( z ′ ) -⟨∇ ψ ( z ′ ) , z -z ′ ⟩ . Under our assumptions [JNT11, Eqn. (80)] gives

Note that since z 0 is the minimizer of ψ , we have V ψ ( z, z 0 ) ≤ ψ ( z ) .

Before proving the relative accuracy guarantee of stochastic mirror prox, we also restate the following composition result for Gaussian mechanism known as the moments accountant.

Lemma 9 ([ACG + 16, KLL21]) . Let ϵ, δ ∈ (0 , 1] and c be a universal constant. Let D ∈ Y n be a dataset over some domain Y , and let h 1 , ..., h T : Y ↦→ R d be a series of (possibly adaptive) queries such that for any y ∈ Y , t ∈ [ T ] , ∥ h t ( y ) ∥ 2 ≤ L . Let σ ≥ cL √ T log(1 /δ ) nϵ and T ≥ n 2 ϵ b 2 . Then the algorithm which samples batches of size B 1 , .., B t of size b uniformly at random and outputs 1 b ∑ y ∈ B t h t ( y ) + g t for all t ∈ [ T ] where g t ∼ N (0 , I d σ 2 ) , is ( ϵ, δ ) -DP.

C.2 Proof of Lemma 3

Lemma 3 is easily established from the following theorem which holds more generally for any monotone operator (rather than just the saddle operator of a convex-concave function). This generalization will allow us to use this theorem again in our results on SVIs.

Recall we consider the norm ∥ [ w,θ ] ∥ = 1 B 2 w ∥ w ∥ 2 ¯ p + 1 B 2 θ ∥ θ ∥ 2 ¯ q and have defined κ = max { 1 ¯ p -1 , 1 ¯ q -1 } and ˜ κ = 1 + 1 { p < 2 ∨ q < 2 } · log( d ) . For any t ∈ { 0 , ..., T } define [ w t , θ t ] = z t , where z t is as given in Algorithm 2. We have the following.

Theorem 4. Let [ w 0 , θ 0 ] , [ w,θ ] satisfy E [ ∥ [ w 0 , θ 0 ] -[ w,θ ] ∥ ] ≤ ˆ D . Let g : W × Θ × X ↦→ B d w ∥·∥ ¯ p ( B w L w ) ×B d θ ∥·∥ ¯ q ( B θ L θ ) be a monotone operator and G S ( w,θ ) = 1 n ∑ x ∈ S g ([ w,θ ]; x ) . There exists an implementation of O such that Algorithm 2 is ( ϵ, δ ) -DP and the following holds

Further, the resulting algorithm makes O ( min { √ κn 2 ϵ 1 . 5 log 2 ( n ) √ d log(1 /δ )˜ κ , √ κn 3 / 2 log 3 / 2 ( n ) }) gradient evaluations.

Before proving this statement, we first quickly show how to obtain Lemma 3, the relative accuracy guarantee for SSPs, using this result.

Proof of Lemma 3. To obtain Lemma 3 from this statement, observe that in the special case where g is the saddle operator of the loss, f , convexity-concavity implies

and Lemma 3 is thus obtained from the bound in Theorem 4.

All that remains is to prove the above theorem. Note the following proof leverages the structure Z = W× Θ , but does not assume that g is the saddle operator of some convex-concave loss.

Proof of Theorem 4. Let L = B 2 2 L 2 w + B 2 θ L 2 θ and let p ∗ = ¯ p ¯ p -1 and q ∗ = ¯ q ¯ q -1 be the conjugate exponents of ¯ p and ¯ q , respectively. For t ∈ [ T ] denote the result of O ( z t ) as [ ∇ w,t , ∇ θ,t ] such that ∇ w,t ∈ R d w and ∇ θ,t ∈ R d θ .

We consider the following construction of the private operator oracle, O , for G S . Our implementation adds Gaussian noise to minibatch estimates of G S . That is, to evaluate O ( z t ) , we uniformly sample a minibatch of size m = max { n √ ϵ T , 1 } , call it M t , as well as Gaussian noise vectors ξ w,t ∼ N (0 , I d w σ 2 w ) and ξ θ,t ∼ N (0 , I d θ σ 2 θ ) , for some σ w , σ θ > 0 . We then have that

̸

Privacy Bound: We first bound the privacy of Algorithm 2. Since for any u , ∥ u ∥ 2 ≤ d 1 -2 /p ∗ ∥ u ∥ p ∗ , we can bound the privacy loss using the guarantees of the moments accountant, Lemma 9. Specifically, we set T = κ min { n, n 2 ϵ 2 d log(1 /δ )˜ κ } , η = ˆ D L √ T , use minibatches of size m = max { n √ ϵ T , 1 } , and set σ w = cB w L w √ Td 1 -2 /p ∗ w log(1 /δ ) nϵ and σ θ = cB θ L θ √ Td 1 -2 /p ∗ θ log(1 /δ ) nϵ for some universal constant c . It can be verified this scale of noise satisfies the conditions of Lemma 9 and thus ensures ( ϵ, δ ) -DP. Utility Bound: We now establish the convergence guarantee by applying the general convergence guarantee of stochastic mirror prox (Lemma 8, Appendix C.1) with ψ ([ w,θ ]) = κ 2 B 2 w ∥ w ∥ 2 ¯ p + κ 2 B 2 θ ∥ θ ∥ 2 ¯ q , which is 1 -strongly convex w.r.t. ∥ · ∥ . Clearly our saddle operator oracle yields unbiased estimates of G S ( z ) at each iteration. To bound the variance, τ , note when p = 2 the private estimate of ∇ w,t satisfies

When p = 2 , then p ∗ = 2 , and one can replace bound ( i ) with E [ ∥ ξ w,t ∥ 2 2 ] . Combining these cases yields a bound of ( c 2 B 2 w L 2 w Td w log(1 /δ )˜ κ n 2 ϵ 2 ) on the variance of ∇ w,t . A similar analysis holds for ∇ θ,t . Ultimately, with respect to the norm chosen above, we get E [ ∥O ( z t ) -G ( z t ) ∥ 2 ∗ ] ≤ τ 2 with

The last equality uses the fact that κ ≥ 1 . Recall we choose ψ ([ w,θ ]) = κ 2 B 2 w ∥ w ∥ 2 ¯ p + κ 2 B 2 θ ∥ θ ∥ 2 ¯ q , and thus ψ ([ w,θ ]) = κ ∥ [ w,θ ] ∥ 2 . Now the guarantees of Lemma 8 imply

The theorem statement now follows from plugging in the parameter settings of T and η and the bound on τ established above.

√

D Missing Results from Section 5

D.1 Proof of Lemma 4

Let h 1 ( z ; x ) = ⟨ g 1 ( z ; x ) , z -z ∗ ⟩ . First, we observe that h 1 is ( βB + L ) -Lipschitz. To see this, we have for any z, z ′ ∈ Z and x ∈ that

The rest of the proof is similar to existing stability implies generalization proofs, but with the additional accounting of the regularization term. In more detail, for any i ∈ [ n ] denote S ( i ) as the dataset which replaces the i 'th datapoint of S , x i , with a fresh sample from D , x ′ . We have the following:

The last step follows from the previously established Lipschitzness property of h 1 .

D.2 Convergence of Recursive Regularization for SVIs

In this section, we define ˜ L = βB + L . Define Gap VI ( z ) = max z ′ {⟨ z ′ , z -z ′ ⟩} . We have the following fact.

Fact 5. If g is L -bounded then ̂ Gap VI is L -Lipschitz.

Proof of 5. For any z 1 , z 2 ∈ Z we have

We recall the assumption made on ρ .

Assumption 2. For some κ > 0 let ρ : Z ↦→ R d be 1 κ -strongly monotone w.r.t. ∥ · ∥ and satisfy ∥ ρ ( z ) ∥ ∗ ≤ ∥ z ∥ for all z ∈ Z .

We will first prove the following more general version statement of Theorem 2, which will be useful later.

Theorem 5. Let λ ≥ 48 ˜ Lκ 2 B √ n ′ and A emp be such that for all t ∈ [ T ] it holds that E [ ∥ ∥ ¯ z t -z ∗ S,t ∥ ∥ 2 ] ≤ B 2 12 · 2 2 t κ 2 . Then Recursive Regularization satisfies

To prove this result, it will be helpful to first show several intermediate results. We start by defining several useful quantities. Define {F t } T t =0 as the filtration where F t is the sigma algebra induced by all randomness up to ¯ z t . For notational convenience we define g (0) ( z ; x ) = g ( z ; x ) . Then for every t ∈ { 0 , 1 , ..., T } we define

z ∗ t : equilibrium of G ( t ) D ( z ) := E x ∼D [ g ( t ) ( z ; x ) ] ;

z ∗ S,t : equilibrium of G ( t ) S ( z ) := 1 n ′ ∑ x ∈ S t g ( t ) ( z ; x ) ;

H ( t ) D (¯ z ) := 〈 G ( t ) D (¯ z ) , ¯ z -z ∗ t 〉 : the relative stationarity function w.r.t. G ( t ) D and z ∗ t ; and,

H ( t ) S (¯ z ) := 〈 G ( t ) S (¯ z ) , ¯ z -z ∗ t 〉 : the relative stationarity function w.r.t. G ( t ) S and z ∗ t .

As discussed in Section 5.2, the power of H ( t ) D is that it bounds the distance of a point to z ∗ t .

Fact 6. Let G : Z ↦→ R d be a µ -strongly monotone operator and let z ∗ be the equilibrium point. Then ∥ z -z ∗ ∥ 2 ≤ 2 ⟨ G ( z ) ,z -z ∗ ⟩ µ .

Proof. By strong monotonicity, for any z ∈ Z ,

The last step comes from the fact that ⟨-G ( z ∗ ) , z -z ∗ ⟩ = ⟨ G ( z ∗ ) , z ∗ -z ⟩ ≤ 0 since z ∗ is the equilibrium.

We now establish two distance inequalities which will be used when analyzing the final gap bound in Theorem 5. The first inequality below bounds the distance of the output of the t -th round to the equilibrium of G ( t ) D . The second inequality bounds how far the population equilibria moves after another regularization term is added.

Lemma 10. Assume the conditions of Theorem 5 hold. Then for every t ∈ [ T ] , the following holds

Proof. We will prove both properties via induction on B 1 , ..., B T . Specifically, for each t ∈ [ T ] we will introduce two terms E t and F t , and show that these terms are bounded if the bound on B t holds and that B t holds if E t -1 and F t -1 are bounded. Property P.1 is then established as a result of the fact that E [ ∥ ¯ z t -z ∗ t ∥ 2 ] ≤ 2( E t + F t ) . Note that B 1 holds as the base case because E [ ∥ z ∗ 1 -z ∗ 0 ∥ 2 ] ≤ B 2 .

Property P.1: We here prove that if B t is sufficiently bounded, then E t and F t are bounded where for t ∈ [ T ] we define

Additionally, this will establish property P.1 because for any t ∈ [ T ] it holds that,

The second inequality comes from the strong monotonicity of the operator (see Fact 6).

Since E t is bounded by the assumption made in the statement of Theorem 5, we focus on bounding F t . We have

The first inequality comes from the fact that stability implies generalization for H ( t ) , Lemma 4. Note the algorithm which outputs this exact equilibrium point is L 2 t λn ′ uniform argument stable (see Lemma 5/Assumption 1). The second inequality comes from the fact that z ∗ S,t is the exact empirical equilibrium point of the regularized objective, and so for any z ∈ Z , 〈 G ( t ) S ( z ∗ S,t ) , z ∗ S,t -z 〉 ≤ 0 . The final inequality uses the setting of λ .

We thus have a final bound 2( E t + F t ) ≤ B 2 2 2 t .

Property P.2: Now assume B t -1 holds. We have

Inequality ( i ) above comes from Young's inequality and the fact that z ∗ t -1 is the equilibrium point w.r.t. G ( t -1) D . Inequality ( ii ) comes from Fact 2/Assumption 1. After re-arranging we can continue as follows:

We now turn to analyzing the utility of the algorithm to complete the proof.

Note that because the above is a statement with respect to the unregularized gap function, we do not have to worry about whether or not the regularization term is smooth.

we have

Let z ′ = arg max z ′ ∈Z {⟨ G D ( z ′ ) , z ∗ T -z ′ ⟩} . We obtain the following bound on the ̂ Gap VI ( w ∗ T , θ ∗ T ) .

Above, ( i ) comes from the fact that z ∗ T is the equilibrium point of G ( T ) D . Inequality ( ii ) uses monotonicity of ρ , i.e. 0 ≤ ⟨ ρ ( z ∗ T -¯ z T ) -ρ ( z ′ -¯ z T ) , z ∗ T -z ′ ⟩ . Inequality ( iii ) comes from Holder's inequality and a triangle inequality. Finally, ( iv ) comes from Assumption 1.

Taking the expectation on both sides of the above we have the following derivation,

Above, ( i ) and the following inequality both come from the triangle inequality. Equality ( ii ) is obtained by rearranging the sums. Inequality ( iii ) comes from applying properties P.1 and P.2 proved above. The last equality comes from the setting of λ and T .

Now using this result in conjunction with Eqn. (18) we have

Finally, we prove Theorem 2 leveraging the relative stationarity assumption.

Proof of Theorem 2. First, observe that under the setting of λ = 48 B ( ˆ ακ 3 + ˜ Lκ 2 √ n ′ ) used in the theorem statement that log( n ) B 2 λ = O ( log( n ) B ˆ ακ 3 + log 3 / 2 ( n ) B ˜ Lκ 2 √ n ) . Thus what remains is to show that the distance condition required by Theorem 5 holds. That is, we now show that if A emp satisfies ˆ α -relative stationarity, then for all t ∈ [ T ] it holds that E [ ∥ ∥ ¯ z t -z ∗ S,t ∥ ∥ 2 ] ≤ B 2 12 · 2 2 t κ 2 .

To prove this property, we must leverage the induction argument made by Lemma 10. Specifically, to prove the condition holds for some t ∈ [ T ] , assume B 2 t = E [∥ ∥ z ∗ t -z ∗ t -1 ] ∥ ∥ ] 2 ≤ B 2 2 2( t -1) (recall the base case for t = 1 trivially holds). As shown in the proof of Lemma 10, this implies that the quantities E t , F t (as defined in 16) are bounded by B 2 2304 · 2 2 t . We thus have

where B t is as defined in property P.2 . Inequality ( i ) comes from the strong monotonicity of G ( t ) S , Fact 6. Inequality ( iii ) comes from the setting λ ≥ 48ˆ ακ/B . Inequality ( ii ) comes from the ˆ α -relative stationarity assumption on A emp , which holds so long as the expected distance is sufficiently bounded and the operator is bounded. In this regard, note that

Further, each g ( t ) is 5 L -bounded. That is, observe

D.3 Near Linear Time Algorithm for SVIs in the ℓ 2 Setting

Because we assume the operator is Lipschitz, in the ℓ 2 setting, we can leverage existing accelerated optimization techniques to achieve a near linear time version of A emp , in a similar fashion to [ZTOH22, BGM23]. Specifically, the work [PB16] gives the following result for strongly monotone variational inequalities when applying their accelerated SVRG algorithm 3 .

Lemma 11. (Implicit in [PB16, Theorem 3]) Let β, µ, K > 0 and c a universal constant. Let g : Z × X ↦→ R be monotone and β -Lipschitz and ρ : Z ↦→ R a µ -strongly monotone operator. Let z ∗ S be the equilibrium of G S ( z ) = 1 n ∑ x ∈ S g ( z ; x ) + ρ ( z ) . There exists an algorithm, which in O ( n + √ nK β µ ) gradient evaluations find a point ¯ z such that E [ ∥ z ∗ -¯ z ∥ 2 ] = cBe -K .

We now construct A emp in the following way. At each round t ∈ [ T ] , we use the accelerated algorithm mentioned above to find a point ˆ z such that E [ ∥ ˆ z -z ∗ S,t ∥ 2 ] ≤ ( δL 52 t λn ′ ) , where z ∗ S,t is the equilibrium point of G ( t ) S ( z ) = 1 n ′ ∑ x ∈ S g ( t ) ( z ; x ) . We then have A emp output the point ¯ z t = ˆ z t + ξ t , where ξ t ∼ N (0 , I d σ 2 t ) and σ t = 8 L √ 2 /δ 2 t λn ′ ϵ . Using this construction, we can obtain the following result.

Theorem 6. Let A be as described above. Then Algorithm 1 is ( ϵ, δ ) -DP and when run with

emp √

and runs in at most O ( n + βn log( n/δ )) gradient evaluations.

Proof. Privacy Guarantee We show that A emp is ( ϵ, δ ) at any iteration t ∈ [ T ] using the stability properties of the regularized operator. Specifically, the exact equilibrium to G t S . Now because A emp guarantees E [ ∥ ˆ z -z ∗ S,t ∥ 2 ] ≤ ( δL 52 t λn ′ ) , an application of Markov's inequality implies that with probability at least 1 -δ that ∥ ˆ z t -z ∗ S,t ∥ 2 = ( 1 δ ( cBe -K )) ≤ L 2 T λn . Thus by a triangle inequality ˆ z t is ( 2 L 2 t λn ′ ) -stable with probability at least 1 -δ . The guarantees of the Gaussian mechanism thus imply A emp is ( ϵ, δ ) -DP.

Utility Guarantee: We prove the utility guarantee by leveraging Theorem 5, which guarantees Gap VI ( R ) = O (log( n ) B 2 λ ) = O ( log 3 / 2 ( n ) BL √ n + log 2 ( n ) BL √ d log(1 /δ ) nϵ )

t ∈ [ T ] it holds that E [ ∥ ¯ z t -z ∗ S,t ] ∥ 2 2 ≤ B 12 · 2 2 t . Note here that under the choice of ρ in Eqn. (7), we satisfy Assuption 1 with κ = 1 . We thus finish the proof with the following analysis,

, so long as for every 2

Running Time: By the guarantees of Lemma 11, we can achieve the condition E [ ∥ ˆ z -z ∗ S,t ∥ 2 ] ≤ ( δL 52 t λn ′ ) made in the description of A emp by setting K = log ( cB δ · 2 T λn L ) . Recall T = log 2 ( L κBλ ) ≤ n . Thus the overall running time is O ( n + β µ K √ n ) = O ( n + βn log( n/δ )) .

E Lower Bound for SVIs

The lower bound for SVIs in the ℓ 2 setting was established in [BG23]. Their technique can easily be extended to other geometries. Specifically, the lower bound comes from two observations. First, for linear losses, the strong VI-gap is equal to the excess population risk when the operator in question is the gradient. Second, the nearly tight lower bound constructions for DP stochastic minimization problems use linear losses.

We establish the first fact more formally here.

Lemma 12. Let f ( z ; x ) = ⟨ z, x ⟩ and define the operator g ( z ; x ) = ∇ f ( z ; x ) = x . Then Gap VI ( z ) = F D ( z ) -min u ∈Z { F D ( u ) } . That is, the strong VI-gap w.r.t. g is equal to the excess population risk w.r.t. the f .

Proof. We have

We now restate the lower bound result of [BGN21] for non-Euclidean setups.

Theorem 7. ([BGN21, Theorem 7.1]) Let p ∈ (1 , 2) and p ∗ = p p -1 and Z = B ∥·∥ p (1) . Let ϵ > 0 and 0 < δ < 1 n 1+Ω(1) and let f ( z ; x ) = ⟨ z, x ⟩ . For any ( ϵ, δ ) -DP algorithm A , there exists a distribution D over X = B ∥·∥ p ∗ (1) such that

Note that a linear dependence on BL can be obtained using classic rescaling arguments. The two above results (and the result of [BG23]) then imply that the strong VI-gap rate we obtain, ˜ O ( BL max { 1 √ n , ( p -1) √ d nϵ }) , is near optimal for p ∈ (1 , 2] when p -1 = Ω(1) .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Claims made are given in Sections 3/4/5 and provide in the Appendix Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The specific assumptions made are detailed in the preliminaries and theorem statements.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Statements are proved or cite a relevant reference. Most of these proofs can be found in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: There are no experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: There is no associated code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: There are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: There are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: There are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The theoretical nature of the results means there are minimal ethical concerns.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The theoretical nature of the work means that any societal impact would be very indirect.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No such assets are used as a part of this research.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: No such assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No such assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No human subjects were involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subjects were involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper studies private SSP beyond Euclidean geometry. They prove a near optimal bound on SP-gap for geometry between $\ell_1 ,\ell_2$. This result is then extended to SVI.

STRENGTHS:
The results are solid improvement over previous work. The method on overcoming the generalization issue is novel and interesting.

WEAKNESSES:
I have a few concerns on the technical results.

(1) what’s $A_{emp}$ in your algorithm. It’s the key subroutine while never formally defined or specified. Can you give a concrete example as you claimed in line 182?

(2) is there any algorithmic novelty compared with previous work, or it’s just an improved analysis? The comparison with BGM23 can be made more clear.

(3) can you give some practical examples to prompt the need of considering noneuclidean geometry, otherwise it looks somewhat incremental.

There are numerous typos.

QUESTIONS:
See weaknesses.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-01

comments: SUMMARY:
This paper is quite far from my area, so please consider this review accordingly.

The paper addresses the problem of private Stochastic Saddle Points and Variational Inequalities. The primary contribution is extending previous work that focused solely on the L2/L2 setup to more general lp/lq settings, where the primal problem follows an lp-setup and the dual problem follows an lq-setup.

The main result is the development of an algorithm that achieves optimal excess error measured by the strong SP-gap.

STRENGTHS:
The paper, in general, feels quite dense. For instance, the second paragraph mentions monotone operators without providing a definition. Additionally, the contribution section is not clear to me.

The algorithmic aspect of the work is very similar to [BGM23]. However, the authors needed to make some changes to the analysis, and they did a good job describing these necessary modifications.

WEAKNESSES:
The main weakness of the work is its presentation. It is very difficult to parse many parts of the paper.

QUESTIONS:
n/a

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17

comments: SUMMARY:
This work studied stochastic saddle point and variational inequality problems in potentially non-Euclidean cases.  For stochastic saddle point problems, they proposed a recursive regularization framework, and provided the convergence guarantee and sample complexity for convex-concave problems. They further extended the framework to variational inequalities and incorporated differential privacy. Corresponding convergence guarantees and complexities results are also provided.

STRENGTHS:
1. First work on SSPs and SVIs in general non-Euclidean settings.
2. The proposed rate is nearly optimal

WEAKNESSES:
1. The boundedness assumption is a bit restricted, regarding many unconstrained problems in practice.
2. Some important assumptions are hidden in the statement of Theorems, for example, the strong convexity of $||\cdot||_\omega$  and $||\cdot||_o$, while it is not fully rationalized (beyond $\ell_p, p\in(1,2]$ case), and it may not be satisfied in some important special case like $\ell_1$, I think the motivation for non-Euclidean 
3. The paper flow and main results are a bit similar to [BGM23], which makes it a little incremental. Even though the authors claimed some differences, but from the appendix, many proofs are still very similar to those in [BGM23] with minor changes like $\kappa$. But I agree the changes in some parts like the proof of Property P.2 reveal certain novelty.

Typo:
1. In Algorithms, the parameters of $\mathcal{A}_{\text{emp}}(\cdot,\cdot,\cdot,\cdot)$ are not clearly defined

QUESTIONS:
/

LIMITATIONS:
/
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study differentially private algorithms for stochastic saddle point (SSP) problems and stochastic variational inequalities (SVI). The proposed method relies on recursive regularization approach and obtain near optimal rates for settings were the parameters of interest are constrained to be in a bounded \\ell_p ball and p \\in [1,2].\n\nSTRENGTHS:\nThe methods proposed by the authors recover many existing optimal results using different proof techniques. They extend the scope of the existing results with a unified analysis.\n\nWEAKNESSES:\nThe paper does not seem to be self-contained. Some important components of the algorithms are not explicitly described, which makes it difficult to verify some of the claims in this work. The authors should include more details in the supplementary materials.\n\nQUESTIONS:\nI have two main comments:\n\n1. The subroutine $\\mathcal{A}_{emp}$ is never explicitly introduced. Here are some of sources of confusion for the reader:\n-  Is this subroutine the same one in Algorithms 1 and 3? \n-  In line 6 of Algorithm 1 the subroutine takes 4 inputs, none of which seems to be related to privacy parameters ?. However,  in lines 258-258 the authors say that the privacy of Algorithm 1 follows from the privacy of $\\mathcal{A}_{emp}$.\n- The construction of $\\mathcal{A}_{emp}$ in lines 754-757 consists of taking the output of the SVRG algorithm of Palaniappan and Bach (2016) and add Gaussian noise to it? I think the precise SVRG algorithm the authors have in mind should also be presented. Some none trivial adaptations seems to be required.\n\n2. Algorithm 1 requires $T=O(\\log n)$ since $\\lambda\\geq \\frac{K\\kappa}{B\\sqrt{n}}$ and $T=\\log_2(\\frac{L}{B\\lambda})$. However, Lemma 3 requires a much larger number of gradient evaluations of roughly $\\tilde{\\Omega}(n^{3/2})$ to get the claimed accuracy that in turn is also needed in Corollary 1? Essentially the same comment applies to Theorem 2 and Corollary 2.\n\nLIMITATIONS:\nThe nature of this work is theoretical but it is natural to wonder if the methods are easy to implement. Have the authors tried to run any numerical experiments?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-30"}