Title: Online Performative Gradient Descent for Learning Nash Equilibria in Decision-Dependent Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: Performative Prediction

Nash Equilibrium

Reproducing Kernel Hilbert Space

Online Learning

Stochastic Gradient Methods
Abstract: We study the multi-agent game within the innovative framework of decision-dependent games, which establishes a feedback mechanism that population data reacts to agents’ actions and further characterizes the strategic interactions between agents. We focus on finding the Nash equilibrium of decision-dependent games in the bandit feedback setting. However, since agents are strategically coupled, traditional gradient-based methods are infeasible without the gradient oracle. To overcome this challenge, we model the strategic interactions by a general parametric model and propose a novel online algorithm, Online Performative Gradient Descent (OPGD), which leverages the ideas of online stochastic approximation and projected gradient descent to learn the Nash equilibrium in the context of function approximation for the unknown gradient. In particular, under mild assumptions on the function classes defined in the parametric model, we prove that OPGD can find the Nash equilibrium efficiently for strongly monotone decision-dependent games. Synthetic numerical experiments validate our theory.

Online Performative Gradient Descent for Learning Nash Equilibria in Decision-Dependent Games

Abstract

We study multi-agent games within the innovative framework of decisiondependent games, which establishes a feedback mechanism that population data reacts to agents' actions and further characterizes the strategic interactions among agents. We focus on finding the Nash equilibrium of decision-dependent games in the bandit feedback setting. However, since agents are strategically coupled, classical gradient-based methods are infeasible without the gradient oracle. To overcome this challenge, we model the strategic interactions by a general parametric model and propose a novel online algorithm, Online Performative Gradient Descent ( OPGD ), which leverages the ideas of online stochastic approximation and projected gradient descent to learn the Nash equilibrium in the context of function approximation for the unknown gradient. In particular, under mild assumptions on the function classes defined in the parametric model, we prove that the OPGD algorithm finds the Nash equilibrium efficiently for strongly monotone decision-dependent games. Synthetic numerical experiments validate our theory.

1 Introduction

The classical theory of learning and prediction fundamentally relies on the assumption that data follows a static distribution. This assumption, however, does not account for many dynamic realworld scenarios where decisions can influence the data involved. Recent literature on performative classification (Hardt et al., 2016; Dong et al., 2018; Miller et al., 2020) and performative prediction (Perdomo et al., 2020) offers a variety of examples where agents are strategic, and data is performative. For instance, in the ride-sharing market, both passengers and drivers engage with multiple platforms using various strategies such as 'price shopping". Consequently, these platforms observe performative demands, and the pricing policy becomes strategically coupled.

In this paper, we explore the multi-agent performative prediction problem, specifically, the multiagent decision-dependent games, as proposed by Narang et al. (2022). We aim to develop algorithms to find Nash equilibria with the first-order oracle. In this scenario, agents can only access their utility functions instead of gradients through the oracle. Finding Nash equilibria in decision-dependent games is a challenging task. Most existing works primarily focus on finding performative stable equilibria within the single-agent setting, an approach that approximates the Nash equilibrium and is relatively straightforward to compute (Mendler-Dünner et al., 2020; Wood et al., 2021; Drusvyatskiy and Xiao, 2022; Brown et al., 2022; Li and Wai, 2022).

There are two major challenges associated with this problem: (i) the distribution shift induced by performative data, and (ii) the lack of first-order information for the performative gradient. To address these two challenges, we propose a novel online gradient-based algorithm, Online Performative Gradient Descent ( OPGD ). In particular, our algorithm employs a general parametric framework to

model the decision-dependent distribution, which provides an unbiased estimator for the unknown gradient, and leverages online stochastic approximation methods to estimate the parametric functions.

1.1 Major Contributions

Our work provides new fundamental understandings of decision-dependent games. Expanding upon the linear parametric assumption in Narang et al. (2022), we propose a more comprehensive parametric framework that models decision-dependent distributions of the observed data. We also derive sufficient conditions under this parametric framework that guarantee a strongly monotone decision-dependent game, thereby ensuring a unique Nash equilibrium.

From the algorithmic perspective, we propose OPGD , the first online algorithm to find the Nash equilibrium under linear and kernel parametric models. The core problem in decision-dependent games is estimating the performative gradient. We remark that the existing algorithm only handles the linear case and cannot be extended to the non-linear parametric model (Section 3), and OPGD uses an essentially different method to learn the strategic interaction. To elaborate, under the proposed parametric framework, learning the Nash equilibrium in decision-dependent games can be formulated as a bilevel problem, where the lower level is learning the strategic model and the upper level is finding equilibriums. The OPGD algorithm leverages the ideas of online stochastic approximation for the lower problem and projected gradient descent to learn the Nash equilibrium. Moreover, we acknowledge this learning framework bridges online optimization and statistical learning with time-varying models.

We further prove that under mild assumptions, OPGD converges to the Nash equilibrium. For the linear function class, OPGD achieves a convergence rate of O ( t -1 ) , matching the optimal rate of SGD in the strongly-convex setting, where t represents the number of iterations. For the kernel function class H associated with a bounded kernel K , we posit that the parametric functions reside within the power space H β and evaluate the approximation error of OPGD under the α -power norm, where α represents the minimal value that ensures the power space H α possesses a bounded kernel. We present the first analysis for online stochastic approximation under the power norm (Lemma 4), in contrast to the classical RKHS norm (Tarres and Yao, 2014; Pillaud-Vivien et al., 2018; Lei et al., 2021). The difference between the RKHS H and the power space H β makes the standard techniques fail under the power norm, and we use novel proof steps to obtain the estimation error bound. We demonstrate that OPGD leverages the embedding property of the kernel K to accelerate convergence and achieves the rate of O ( t -β -α β -α +2 ) . Moreover, OPGD can handle the challenging scenario, where parametric functions are outside the RKHS. See Section 4.2 for more details.

1.2 Related Work

Performative prediction. The multi-agent decision-dependent game in this paper is inspired by the performative prediction framework (Perdomo et al., 2020). This framework builds upon the pioneering works of strategic classification (Hardt et al., 2016; Dong et al., 2018; Miller et al., 2020), and extends the classical statistical theory of risk minimization to incorporate the performativity of data. Perdomo et al. (2020); Mendler-Dünner et al. (2020); Miller et al. (2021) introduce the concepts of performative optimality and stability, demonstrating that repeated retraining and stochastic gradient methods converge to the performatively stable point. Miller et al. (2021), in pursuit of the performatively optimal point, model the decision-dependent distribution using location families and propose a two-stage algorithm. Similarly, Izzo et al. (2021) develop algorithms to estimate the unknown gradient using finite difference methods. More recently, Narang et al. (2022); Piliouras and Yu (2022) expand the performative prediction to the multi-agent setting, deriving algorithms to find the performatively optimal point.

Learning in continuous games. Our work aligns closely with optimization in continuous games. Rosen (1965) lays the groundwork, deriving sufficient conditions for a unique Nash equilibrium in convex games. For strongly monotone games, Bravo et al. (2018); Mertikopoulos and Zhou (2019); Lin et al. (2021) achieve the convergence rate and iteration complexity of stochastic and derivativefree gradient methods. For monotone games, the convergence of such methods is established by Tatarenko and Kamgarpour (2019, 2020). Additional with bandit feedback settings, zeroth-order methods (or derivative-free methods) achieve convergence (Bravo et al., 2018; Lin et al., 2021; Drusvyatskiy et al., 2022; Narang et al., 2022), albeit with slow convergence rates (Shamir, 2013; Lin et al., 2021; Narang et al., 2022). Relaxing the convex assumption, Ratliff et al. (2016); Agarwal et al. (2019); Cotter et al. (2019) study non-convex continuous games in various settings.

Learning with kernels. Our proposed algorithm closely relies on stochastic approximation, utilizing online kernel regression for the RKHS function class. Prior research investigates the generalization capability of least squares and ridge regression in RKHS De Vito et al. (2005); Caponnetto and De Vito (2007); Smale and Zhou (2007); Rosasco et al. (2010); Mendelson and Neeman (2010). Meanwhile, extensive works study algorithms for kernel regression. For instance, Yao et al. (2007); Dieuleveut and Bach (2016); Pillaud-Vivien et al. (2018); Lin and Rosasco (2017); Lei et al. (2021) propose offline algorithms with optimal convergence rates under the RKHS norm and L 2 norm using early stopping and stochastic gradient descent methods, while Ying and Pontil (2008); Tarres and Yao (2014); Dieuleveut and Bach (2016) design online algorithms with optimal convergence rates. The convergence of kernel regression in power norm (or Sobolev norm) is studied in Steinwart et al. (2009); Fischer and Steinwart (2020); Liu and Li (2020); Lu et al. (2022), with offline spectral filter algorithms achieving the statistical optimal rate under the power norm (Pillaud-Vivien et al., 2018; Blanchard and Mücke, 2018; Lin and Cevher, 2020; Lu et al., 2022).

Notation. Weintroduce some useful notation before proceeding. Throughout this paper, we denote the set 1 , 2 , · · · , n by [ n ] for any positive integer n . For two positive sequences { a n } n ∈ N and { b n } n ∈ N , we write a n = O ( b n ) or a n ≲ b n if there exists a positive constant C such that a n ≤ C · b n . For any integer d , we denote the d -dimensional Euclidean space by R d , with inner produce ⟨· , ·⟩ and the induced norm ∥·∥ = √ ⟨· , ·⟩ . For a Hilbert space H , let ∥·∥ H be the associated Hilbert norm. For a set X and a probability measure ρ X on X , let L 2 ρ X be the L 2 space on X induced by the measure ρ X , equipped with inner product ⟨· , ·⟩ ρ X and L 2 norm ∥ · ∥ ρ X = √ ⟨· , ·⟩ ρ X . For any matrix A = ( a ij ) , the Frobenius norm and the operator norm (or spectral norm) of A are ∥ A ∥ F = ( ∑ i,j a 2 ij ) 1 / 2 and ∥ A ∥ op = σ 1 ( A ) , where σ 1 ( A ) stands for the largest singular value of A . For any square matrix A = ( a ij ) , denote its trace by tr ( A ) = ∑ i a ii . For any y ∈ R d , we denote its projection onto a set X ⊂ R d by proj X ( y ) = arg min x ∈X ∥ x -y ∥ . The set denoted by N X ( x ) represents the normal cone to a convex set X at x ∈ X , namely, N X ( x ) = { v ∈ R d : ⟨ v, y -x ⟩ ≤ 0 , ∀ y ∈ X} . For any metric space Z with metric d ( · , · ) , the symbol P ( Z ) will denote the set of Radon probability measures µ on Z with a finite first moment E z ∼ µ [ d ( z, z 0 )] < ∞ for some z 0 ∈ Z .

2 Problem Formulation and Preliminaries

We briefly introduce the formulation of n -agent decision-dependent games based on Narang et al. (2022). In this setting, each agent i ∈ [ n ] takes the action x i ∈ X i from an action set X i ⊂ R d i . Define the joint action x := ( x 1 , x 2 , · · · , x n ) ∈ X and the joint action set X = X 1 ×···×X n ⊂ R d , where d := ∑ n i =1 d i . For all i ∈ [ n ] , we write x = ( x i , x -i ) , where x -i denotes the vector of all coordinates except x i . Let L i : X → R be the utility function of agent i . In the game, each agent i seeks to solve the problem

Here z i ∈ Z i represents the data observed by agent i , where the sample space Z i is assumed to be Z i = R p with p ∈ N throughout this paper. Moreover, D i : X → P ( Z i ) is the distribution map, and ℓ i : R d ×Z i → R denotes the loss function. During play, each agent i performs an action x i and observes performative data z i ∼ D i ( x ) , where the performativity is modeled by the decisiondependent distribution D i ( x ) . In the round t , the agent i only has access to z 1 i , · · · , z t -1 i as well as x 1 , · · · , x t -1 and seeks to solve the ERM version of (1). We assume the access to the first-order oracle, namely, loss functions ℓ i are known to agents but distribution maps D i are unknown.

Definition 1. (Nash equilibrium). In the game (1), a joint action x ∗ = ( x ∗ 1 , x ∗ 2 , · · · , x ∗ n ) is a Nash equilibrium (Nash Jr, 1996) if all agents play the best response against other agents, namely,

In general continuous games, Nash equilibria may not exist or there might be multiple Nash equilibria (Fudenberg and Tirole, 1991). The existence and uniqueness of a Nash equilibrium in a continuous game depend on the game's structure and property. In general, finding the unique Nash equilibrium is only possible for convex and strongly monotone games (Debreu, 1952).

Definition 2. (Convex game). Game (1) is a convex game if sets X i are non-empty, compact, convex and utility functions L i ( x i , x -i ) are convex in x i when x -i are fixed.

Suppose that utility functions L i are differentiable, we use ∇ i L i ( x ) to denote the gradient of L i ( x ) with respect to x i (the i -th individual gradient). We say the game (1) is C 1 -smooth if the gradient ∇ i L i ( x ) exists and is continuous for all i ∈ [ n ] . Using this notation, we define the gradient H ( x ) comprised of individual gradients

Definition 3. (Strongly monotone game). For a constant τ ≥ 0 , a C 1 -smooth convex game (1) is called τ -strongly monotone if it satisfies

Note that a τ -strongly monotone game ( τ > 0 ) over a compact and convex action set X admits a unique Nash equilibrium (Rosen, 1965). According to the optimal conditions in convex optimization (Boyd et al., 2004), this Nash equilibrium x ∗ is characterized by the variational inequality

We briefly talk about the challenges and our idea of designing the algorithm. In decision-dependent games, the classical theory of risk minimization does not work. The primary obstacles to finding the Nash equilibrium in the game (1) include: (i) the distribution shift induced by performative data, and (ii) the lack of first-order information (gradient). To make it clear, standard methods, such as gradient-based algorithms, necessitate the gradient H ( x ) . However, H ( x ) is unknown since distributions D i are unknown, and estimating H ( x ) is complex due to the dependency between D i ( x ) and x . Mathematically, assuming C 1 -smoothness, the chain rule directly yields the following expression for the gradient

where ∇ i ℓ i ( x, z i ) denotes the gradient of ℓ i ( x, z i ) with respect to x i . The main difficulty is estimating the second term in (4) due to the absence of closed-form expressions.

To estimate the unknown gradient H ( x ) , we impose a parametric assumption on the observed data z i and model the distribution maps D i using parametric functions. Note that the linear parametric assumption was first proposed in Narang et al. (2022). In this paper, we extend this assumption to a general framework and show that under the parametric assumption, the gradient H ( x ) has a closed-form expression, which yields an unbiased estimator for H ( x ) .

Assumption 1. (Parametric assumption). Suppose there exists a function class F and p -dimensional functions f i : X → R p over the joint action set X such that f i ∈ F p and

where ϵ i ∈ R p are zero-mean noise terms with finite variance σ 2 , namely, E ϵ i = 0 and E ∥ ϵ i ∥ 2 ≤ σ 2 .

Under Assumption 1, assuming that f i are differentiable and letting P i be the distribution of the noise term ϵ i , we derive the following expression for the utility functions L i ( x ) = E z i ∼D i ( x ) ℓ i ( x, z i ) = E ϵ i ∼P i ℓ i ( x, f i ( x ) + ϵ i ) . Then the individual gradient would be ∇ i L i ( x ) = ∇ i E z i ∼D i ( x ) ℓ i ( x, z i ) = ∇ i [ E ϵ i ∼P i ℓ i ( x, f i ( x ) + ϵ i )] . Consequently, the chain rule directly implies the following expression

where ∇ z i ℓ i ( x, z i ) denotes the gradient of ℓ i ( x, z i ) with respect to z i . Given a joint action x , each agent i observes data z i ∼ D i ( x ) . Equation (5) suggests the following unbiased estimator for H ( x ) :

However, direct computation of ̂ H ( x ) is infeasible because f i are unknown. To overcome this challenge, we approximate the unknown functions f i with the function class F p . In fact, the estimation of f i can be formed as a non-parametric regression problem, namely,

where ρ i is the joint distribution of ( x, z i ) induced by x ∼ ρ X and z i ∼ D i ( x ) . Here ρ X is a user-specified sampling distribution on X .

3 The OPGD Algorithm

In this section, we consider F to be the linear and kernel function classes and derive gradient-based online algorithms to find the Nash equilibrium in the game (1), namely, the Online Performative Gradient Descent ( OPGD ). In each iteration t , assuming that x t := ( x t 1 , · · · , x t n ) is the output of the previous iteration, OPGD performs the following update for all i ∈ [ n ] :

(i) (Estimation update). Update the estimation of f i by online stochastic approximation for (7).

(ii) (Individual gradient update). Compute the estimator (6) and perform projected gradient steps

Linear Function Class. Let F be the linear function class, namely, f i ( x ) = A i x for i ∈ [ n ] , where A i ∈ R p × d are unknown matrices. Then (7) becomes the least square problem A i = arg min A ∈ R p × d E ( u i ,y i ) ∼ ρ i ∥ y i -A i u i ∥ 2 with random variables u i ∼ ρ X , y i ∼ D i ( u i ) . We use the gradient of the least square objective ∥ y i -A i u i ∥ 2 to derive the online least square update: A new ← A -ν ( Au i -y i ) u ⊤ i (Dieuleveut et al., 2017; Narang et al., 2022). In each iteration t , we suppose that A t -1 i is the estimation of A i from the previous iteration, OPGD samples u t i ∼ ρ X and y t i ∼ D i ( u t i ) and performs the following estimation update:

Recalling (5), the individual gradient is ∇ i L i ( x ) = E z i ∼D i ( x ) [ ∇ i ℓ i ( x, z i ) + A ⊤ ii ∇ z i ℓ i ( x, z i ) ] , where A ii = ∂f i ( x ) /∂x i ∈ R p × d i denotes the submatrix of A i whose columns are indexed by the agent i . After step (i), OPGD draws a sample z t i ∼ D i ( x t ) and compute the estimator (6) to perform the projected gradient step:

Kernel Function Class. Now we consider F as the kernel function class, namely, we suppose f i ∈ ( H ) p , where H is an RKHS induced by a Mercer kernel K : X × X → R and a userspecified probability measure ρ X . By the reproducing property of H , f i can be represented as f i ( x ) = ⟨ f i , ϕ x ⟩ H , where ϕ : X → H is the feature map, i.e. ϕ x := K ( · , x ) ∈ H for any x ∈ X . Therefore, (7) becomes the kernel regression arg min f ∈ F p E ( u i ,y i ) ∼ ρ i ∥ y i -⟨ f, ϕ u i ⟩ H ∥ 2 . However, as H is generally an infinite-dimensional space, the aforementioned regression problem might lead to ill-posed solutions. Consequently, we consider the regularized kernel ridge regression arg min f ∈ F p E ( u i ,y i ) ∼ ρ i ∥ y i -⟨ f, ϕ u i ⟩ H ∥ 2 / 2 + λ t ∥ f ∥ 2 H . In each iteration t , we suppose that f t -1 i is the estimation of f i from the previous iteration, the OPGD algorithm samples u t i ∼ ρ X , y t i ∼ D i ( u t i ) and takes gradient steps on the kernel ridge objective ∥ y t i -⟨ f, ϕ u t i ⟩ H ∥ 2 / 2 + λ t ∥ f ∥ 2 H , i.e. it takes the online kernel ridge update (Tarres and Yao, 2014; Dieuleveut and Bach, 2016):

We suppose that the kernel K is 2-differentiable, i.e., K ∈ C 2 ( X , X ) . Define ∂ i ϕ : X → H as the partial derivative of the feature map ϕ with respect to x i , namely, ∂ i ϕ x = ∂ i K ( x, · ) = ∂K ( x, · ) /∂x i . Steinwart and Christmann (2008, Lemma 4.34) shows that ∂ i ϕ x exists, continuous and ∂ i ϕ x ∈ H . By the reproducing property ∂f i ( x ) /∂x i = ∂ ⟨ f i , ϕ x ⟩ H /∂x i = ⟨ f i , ∂ i ϕ x ⟩ H , the individual gradient ∇ i L i ( x ) has the form ∇ i L i ( x ) = E z i ∼D i ( x ) [ ∇ i ℓ i ( x, z i ) + ( ⟨ f i , ∂ i ϕ x ⟩ H ) ⊤ ∇ z i ℓ i ( x, z i )] . After step (i), OPGD draws a sample z t i ∼ D i ( x t ) and performs the projected gradient step:

We remark that the gradient steps η t , ν t and regularization terms λ t should be chosen carefully to ensure convergence (see Theorem 2). Specifically, the regularization terms λ t must shift to 0 gradually. If λ t is a constant, f t i in (10) converges to the solution of a regularized kernel ridge regression, which is a biased estimator of f i . Thus (11) fails to converge because the gradient estimation has a constant bias. We present the pseudocode of OPGD for the linear setting as Algorithm 1 and for the RKHS setting as Algorithm 2 in Appendix A.

Comparison with Narang et al. (2022). We clarify the difference between OPGD and the Adaptive Gradient Method (AGM) proposed in Narang et al. (2022). To elaborate, AGM samples z t i ∼ D i ( x t ) at current the action and let agents play again with an injected noise u t to obtain q t i ∼ D i ( x t + u t ) . The algorithm is based on the fact that E [ q t i -z t i | u t , x t ] = A i u t , which is not related to x t . Thus, A i can be estimated by online least squares. We remark that E [ q t i -z t i | u t , x t ] depends on agents' actions in the non-linear (RKHS) cases, because E [ q t i -z t i | u t , x t ] = f i ( x t + u t ) -f i ( x t ) = ⟨ f i , ϕ x t + u t -ϕ x t ⟩ H . Thus, the change of action will bring additional error that makes the estimation fail to converge. In contrast, OPGD lets agents play u t i ∼ ρ X to explore the action space and learn the strategic behavior of other agents. OPGD estimates the parametric function by solving the ERM version of (7) using online stochastic approximation (8) and (10). This learning framework can be applied to RKHS and potentially beyond that, such as overparameterized neural networks using the technique of neural tangent kernel (Allen-Zhu et al., 2019).

4 Theoretical Results

We provide theoretical guarantees for OPGD in both linear and RKHS settings. We first impose some mild assumptions. Similar assumptions are adopted in Mendler-Dünner et al. (2020); Izzo et al. (2021); Narang et al. (2022); Cutler et al. (2022).

Assumption 2. ( τ -strongly monotone). The game (1) is τ -strongly monotone.

Assumption 3. (Smoothness). H ( x ) is L -Lipschitz continuous:

Assumption 4. (Lipschitz continuity in z ). Define D = D 1 ×D 2 ×··· × D n : X → P ( Z ) , where Z is the sample space Z = Z 1 ×Z 2 ×· · · × Z n . For all i ∈ [ n ] , x ∈ X , there exists a constant δ > 0 ,

Assumption 5. (Finite variance). There exists a constant ζ > 0 ,

where ∇ i,z i ℓ i denotes the gradient of ℓ i ( x, z i ) with respect to x i and z i .

We remark that Assumption 3 is the standard smoothness assumption for the utility functions L i ( x ) (Boyd et al., 2004; Nesterov et al., 2018). Since X is a compact set within R d , Assumption 4 holds if ℓ i ( x, z i ) is Lipschitz continuous in z i and the gradient ∇ z i ℓ i ( x, z i ) is continuous in x , and Assumption 5 holds if ℓ i ( x, z i ) is Lipschitz in x and z i (thus ∇ i,z i ℓ i ( x, z i ) has a bounded norm). Assumption 5 implies that the variances of ∇ i ℓ i ( x, z i ) and ∇ z i ℓ i ( x, z i ) are both bounded by ζ 2 for any x ∈ X and z i ∼ D i ( x ) . We provide sufficient conditions for Assumption 2 in Appendix B.1.

4.1 Convergence Rate in the Linear Setting

We introduce two assumptions necessary to derive theoretical guarantees for the linear function class.

Assumption 6. (Linear assumption). Suppose that the parametric assumption holds (Assumption 1) and f i ( x ) = A i x for i ∈ [ n ] , where A i ∈ R p × d are unknown matrices.

Assumption 7. (Sufficiently isotropic). There exists constants l 1 , l 2 , R > 0 such that

Assumption 7 has been studied in the literature on online least squares regression (Dieuleveut et al., 2017; Narang et al., 2022). Essentially, this requires the distribution ρ X to be sufficiently isotropic and non-singular, and it ensures the random variable u t i ∼ ρ X in the online estimation update step (8) can explore all the "directions" of R p . A simple example that satisfies Assumption 7 is the uniform distribution ρ X = U [0 , 1] , in which case l 1 = l 2 = 1 / 3 , R = 3 / 5 .

The next theorem provides the convergence rate of OPGD under the linear setting.

Theorem 1. (Convergence in the linear setting). Suppose that Assumptions 2, 3, 4, 5, 6, and 7 hold. Set η t = 2 / ( τ ( t + t 0 )) , ν t = 2 / ( l 1 ( t + t 0 )) , where t 0 is a constant that satisfies t 0 ≥ 2 l 2 R/l 2 1 . For

all iterations t ≥ 1 , the x t generated by the OPGD algorithm in Section 3 for linear function class satisfies

where D 1 and D 2 are constants that

We refer the reader to Appendix C.1 for complete proof. Next, we illustrate the parameters involved in Theorem 1: τ is the strongly monotone parameter of the game (1), l 1 , l 2 , R are intrinsic parameters describing the isotropy of the distribution ρ X (Assumption 7), σ 2 is the variance of the noise term ϵ i defined in Assumption 1, ζ and δ describe the continuity of ℓ i (Assumption 4, 5), t 0 is a sufficiently large value, A 0 i is the initial estimation of A i , x 1 is the initial input. Theorem 1 is a combination of Lemma 2 and Lemma 3, where Lemma 2 is the statistical error of the online approximation step (8) and Lemma 3 is the one-step optimization error of the projected gradient step (9). Theorem 1 implies the convergence rate of OPGD in the linear setting is O ( t -1 ) , which matches the optimal rate of stochastic gradient descent in the strongly-convex setting.

4.2 Convergence Rate in the RKHS Setting

Suppose that K : X ×X → R is a continuous Mercer kernel, by Mercer's theorem, it has the spectral representation K = ∑ ∞ i =1 µ i e i ⊗ e i , where { µ i } ∞ i =1 are eigenvalues, { e i } ∞ i =1 are eigenfunctions, and ⊗ denotes the tensor product. Moreover, { e i } ∞ i =1 is an orthogonal basis of L 2 ρ X and { µ 1 / 2 i e i } ∞ i =1 is the orthogonal basis of H , which induces the representation H = { ∑ ∞ i =1 a i µ 1 / 2 i e i : { a i } ∞ i =1 ∈ ℓ 2 } . Definition 4. (Power space). For a constant α ≥ 0 , the α -power space of an RKHS H is defined by

equipped with the α -power norm ∥·∥ α and inner product ⟨· , ·⟩ α , where ∥ ∑ ∞ i =1 a i µ α/ 2 i e i ∥ α := (∑ ∞ i =1 a 2 i ) 1 / 2 and ⟨ ∑ ∞ i =1 a i µ α/ 2 i e i , ∑ ∞ i =1 b i µ α/ 2 i e i ⟩ α = ∑ ∞ i =1 a i b i .

We remark that: (i) H 1 = H and H α ⊂ H β for any α > β , (ii) ∥·∥ 1 = ∥·∥ H and ∥·∥ 0 = ∥·∥ ρ X , and (iii) H α is an RKHS on X with kernel K α := ∑ ∞ i =1 µ α i e i ⊗ e i and measure ρ X . We review more properties of RKHS and power spaces in Appendix Sections B.3 and B.4.

Wepresent assumptions on the kernel function class, similar assumptions can be found in the literature on kernel regression and stochastic approximation (Caponnetto and De Vito, 2007; Steinwart et al., 2009; Dicker et al., 2017; Pillaud-Vivien et al., 2018; Fischer and Steinwart, 2020).

Assumption 8. (Source condition). Suppose Assumption 1 holds and there exists an RKHS, H , with a bounded differentiable Mercer kernel, K , and constants β, κ > 0 such that sup x ∈X K ( x, x ) ≤ κ 2 and f i ∈ ( H β ) p for all i ∈ [ n ] .

Assumption 9. (Embedding property). There exist constants α ∈ (0 , 1] , A > 0 such that K α ( x, x ) = ∑ ∞ i =1 µ α i e 2 i ( x ) ≤ A 2 , for all x ∈ X .

Assumption 10. (Lipschitz kernel). Suppose Assumption 9 holds and there exists ξ > 0 such that ∥ ∂ i ϕ α x ∥ α ≤ ξ for any i ∈ [ n ] and x ∈ X , where ϕ α x : X → H α is the feature map of the kernel K α .

Assumption 8 holds when K is bounded, differentiable, and each coordinate of parametric functions f i lies in the power space H β . When β < 1 , Assumption 8 includes the challenging scenario, namely, f i / ∈ ( H ) p . Assumption 9 holds if there exists a power space H α such that the kernel K α is bounded. Thus, Assumption 9 holds with α = 1 for any bounded kernel K . We further propose Proposition 1 as sufficient conditions for the embedding property following Mendelson and Neeman (2010). Recalling the definition of partial derivative ∂ i ϕ α : X → H α (Section 3), Assumption 10 holds if ∂ i ∂ i + d K α ( x, x ) = ∥ ∂ i ϕ α x ∥ 2 α ≤ ξ 2 for any x ∈ X , i.e. it holds for any Lipschitz kernel K α .

Proposition 1. (Sufficient conditions for Assumption 9) Suppose there exist constants C, D, p > 0 and q ∈ (0 , 1) such that

where ∥·∥ ∞ denotes the L ∞ norm. Then Assumption 9 holds for any α > 2 p + q .

Proposition 1 follows from the inequality: sup x ∈X K α ( x, x ) = sup x ∈X ∑ ∞ i =1 ( µ p i e i ( x )) 2 µ α -2 p i ≤ C 2 D α -2 p ∑ ∞ i =1 i -( α -2 p ) /q < ∞ . We refer readers to Appendix B.4 for examples that satisfy these assumptions. Next, we provide the convergence of the proposed algorithm under the RKHS setting.

Theorem 2. (Convergence in the RKHS setting). Suppose that Assumptions 2, 3, 4, 5 hold, Assumption 8 holds for some β ∈ (0 , 2] , and Assumptions 9, 10 hold for some α ∈ (0 , 1] and α < β . For all iterations t ≥ 1 and positive constant a , define t = t + t 0 , where t 0 is a constant that satisfies t 0 ≥ ( aκ 2 +1) 2 . Set the gradient steps and regularization terms as

If a < √ ( β -α +2) / ( β -α )( t 0 +1) / ( t 0 +2) κ α -2 A -1 , the x t generated by the OPGD algorithm in Section 3 using kernel K for online estimation steps (10) and projected gradient steps (11) satisfies

See Appendix C.2 for complete proof. We remark that bounding the estimation error E ∥ f t i -f i ∥ 2 γ under the γ -power norm for some γ ∈ [ α, β ) and γ ≤ 1 (Lemma 4) plays a key role in the proof of Theorem 2, and we regard this lemma as the most challenging part of our theory. Lemma 4 extends the classical theory of online stochastic approximation under the RKHS norm ∥·∥ H into a continuous scale. For β > 1 and γ = 1 , this rate would be O ( t -( β -1) / ( β +1) ) and matches the optimal rate under the RKHS norm Ying and Pontil (2008); Tarres and Yao (2014). If the embedding property (Assumption 9) holds for some α < 1 , we choose γ = α to achieve a faster rate O ( t -( β -α ) / ( β -α +2) ) (which further leads to Theorem 2). To prove Lemma 4, we derive novel proof steps, where we decouple the power norm by considering semi-population iteration and recursive decomposition, we refer the reader to Appendix E.1 for more explanations.

We demonstrate the parameters involved in Theorem 2. Parameters α, β, κ, τ, A are intrinsic: β, κ, A are determined by source condition (Assumption 8), α is determined by embedding property (Assumption 9), and τ is the strongly monotone parameter. Parameters a, t 0 are user-specified: t 0 is a sufficiently large value, a is characterized by the inequality a < √ ( β -α +2) / ( β -α )( t 0 + 1) / ( t 0 + 2) κ α -2 A -1 when t 0 is determined, a smaller a leads to a larger constant term in (13). Theorem 2 implies that OPGD leverages the embedding property (Assumption 9) to obtain better convergence rates. For any bounded kernel, Assumption 9 holds for α = 1 , thus Theorem 2 guarantees the rate O ( t -β -1 β +1 ) . Moreover, suppose that the kernel satisfies some good embedding property, that is, α < 1 , since larger β -α leads to faster convergence rates. In that case, we obtain a better rate O ( t -β -α β -α +2 ) by setting the gradient steps and regularization terms ν t , λ t corresponding to α, β . Besides, OPGD can handle the challenging scenario ( f i / ∈ ( H ) p if β < 1 ) when the embedding property of kernel holds for α < β .

5 Numerical Experiments

In this section, we conduct experiments on decision-dependent games in both the linear and the RKHS settings to verify our theory. All experiments are conducted with Python on a laptop using 14 threads of a 12th Gen Intel(R) Core(TM) i7-12700H CPU.

Basic Setup. We consider two-agent decision-dependent games with 1 -dimensional actions. Namely, for all i ∈ [2] , define the game

where X = [0 , 1] × [0 , 1] , x ∈ X , z i ∈ R , and ℓ i ( x, z i ) is the loss function to be determined. Let the distribution map be D i ( x ) ∼ N ( f i ( x ) , 0 . 2) , where f i is the parametric function determined by the specific function class. Then the game (14) follows the parametric assumption (Assumption 1) with z i = f i ( x ) + ϵ i , where ϵ i ∼ N (0 , 0 . 2) the independent Gaussian noise term.

(a)

(b)

Comparisonofrate

Theoreticalrate

Simulation result

10-1

￥

10-2

10-3

101

102

103

104

log(Iteration)

Comparisonofrate

Theoreticalrate

2

￥

10-1

#

60l

10-2

101

102

103

104

log(Iteration)

Linear function class. Let the loss function be ℓ i ( x, z i ) = -z i + x 2 i and set the linear parametric function as f 1 ( x ) = x 1 and f 2 ( x ) = 2 x 2 , namely, the parametric model is z i = A i x + ϵ i where A 1 = [1 0] and A 2 = [0 2] . The the game (14) has the gradient H ( x ) = (2 x 1 -1 , 2 x 2 -2) , therefore, the game (14) is convex, C 1 -smooth, 1 -strongly monotone and the Nash equilibrium is x ∗ = (1 / 2 , 1) . We set the sampling distribution as ρ X = U [0 , 1] × U [0 , 1] , the initial point as x 0 ∼ ρ X , and the initial estimation as zero. Moreover, letting t 0 = 10 , we set the gradient step sizes as η t = 6 / ( t + t 0 ) , ν t = 6 / ( t + t 0 ) .

Kernel function class. Let X = [0 , 1] × [0 , 1] , ρ X = U [0 , 1] × U [0 , 1] , and define the kernel Q (( x 1 , x 2 ) , ( y 1 , y 2 )) = K ( x 1 , y 1 ) · K ( x 2 , y 2 ) as the product kernel of K ( x, y ) = 40 B 4 ( { x -y } ) . Suppose that H is the RKHS on X induced by the kernel Q and the distribution ρ X . Set the parametric function as the product of two 3 -order Bernoulli polynomials, namely, f ( x 1 , x 2 ) = B 3 ( x 1 ) · B 3 ( x 2 ) = ( x 3 1 -3 x 2 1 / 2+ x 1 / 2) · ( x 3 2 -3 x 2 2 / 2+ x 2 / 2) . Set ℓ i ( x, z i ) = -z i +cos(2 πx 1 ) cos(2 πx 2 ) -x i + x 2 i and let f i ( x ) = cos(2 πx 1 ) cos(2 πx 2 ) for i ∈ [2] . Then the gradient of this game is H ( x ) = (2 x 1 -1 , 2 x 2 -1) , thus, this game is convex, C 1 -smooth, 1 -strongly monotone and the Nash equilibrium is x ∗ = (0 . 5 , 0 . 5) . Following Example 1, Assumption 8, 9, 10 hold for any β > 1 and any α > 1 / 4 . Set t 0 = 10 , a = 7 , η t = 6 / ( t + t 0 ) , ν t = a/ ( t + t 0 ) 3 / 4 , and λ t = 1 / ( a ( t + t 0 ) 1 / 4 ) . Following Theorem 2, the convergence rate is O ( t -1 / 2 ) .

Results. We perform experiments for both parametric settings to verify the convergence rates and compare the theoretical and simulated rates, as shown in Figure 1, where both X and Y axes take the log scale. Figure 1(a) shows the converge rate of the linear setting within 10 , 000 iterations, the simulated rate matches our prediction, i.e. it is close to O ( t -1 ) . Figure 1(b) shows the convergence rate of the RKHS setting, it implies that the simulated rate is close to the theoretical rate O ( t -1 / 2 ) when the iteration t is larger than 1 , 000 . These results validate Theorems 1 and 2.

6 Conclusion and Discussion

In this paper, we study the problem of learning Nash equilibria in multi-agent decision-dependent games with access to the first-order oracle. We propose a parametric assumption to handle the distribution shift and develop a novel online algorithm OPGD in both the linear and RKHS settings. We derive sufficient conditions to ensure the decision-dependent game is strongly monotone under the parametric assumption. We show that OPGD converges to the Nash equilibrium at a rate of O ( t -1 )

in the linear setting and O ( t -β -α β -α +2 ) in the RKHS setting.

References

Abramowitz, M., Stegun, I. A. and Romer, R. H. (1988). Handbook of mathematical functions with formulas, graphs, and mathematical tables.

Adams, R. A. and Fournier, J. J. (2003). Sobolev spaces . Elsevier.

Agarwal, N., Gonen, A. and Hazan, E. (2019). Learning in non-convex games with an optimization oracle. In Conference on Learning Theory . PMLR.

Allen-Zhu, Z., Li, Y. and Liang, Y. (2019). Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in neural information processing systems , 32 .

Blanchard, G. and Mücke, N. (2018). Optimal rates for regularization of statistical inverse learning problems. Foundations of Computational Mathematics , 18 971-1013.

Boyd, S., Boyd, S. P. and Vandenberghe, L. (2004). Convex optimization . Cambridge university press.

Bravo, M., Leslie, D. and Mertikopoulos, P. (2018). Bandit learning in concave n-person games. Advances in Neural Information Processing Systems , 31 .

Brown, G., Hod, S. and Kalemaj, I. (2022). Performative prediction in a stateful world. In International Conference on Artificial Intelligence and Statistics . PMLR.

Caponnetto, A. and De Vito, E. (2007). Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics , 7 331-368.

Cotter, A., Jiang, H. and Sridharan, K. (2019). Two-player games for efficient non-convex constrained optimization. In Algorithmic Learning Theory . PMLR.

Cucker, F. and Smale, S. (2002). On the mathematical foundations of learning. Bulletin of the American mathematical society , 39 1-49.

Cutler, J., Díaz, M. and Drusvyatskiy, D. (2022). Stochastic approximation with decision-dependent distributions: asymptotic normality and optimality. arXiv preprint arXiv:2207.04173 .

De Vito, E., Rosasco, L., Caponnetto, A., De Giovannini, U., Odone, F. and Bartlett, P. (2005). Learning from examples as an inverse problem. Journal of Machine Learning Research , 6 .

Debreu, G. (1952). A social equilibrium existence theorem. Proceedings of the National Academy of Sciences , 38 886-893.

Dicker, L. H., Foster, D. P. and Hsu, D. (2017). Kernel ridge vs. principal component regression: Minimax bounds and the qualification of regularization operators. Electronic Journal of Statistics , 11 1022-1047.

Dieuleveut, A. and Bach, F. (2016). Nonparametric stochastic approximation with large step-sizes. The Annals of Statistics , 44 1363 - 1399.

Dieuleveut, A., Flammarion, N. and Bach, F. (2017). Harder, better, faster, stronger convergence rates for least-squares regression. The Journal of Machine Learning Research , 18 3520-3570.

Dong, J., Roth, A., Schutzman, Z., Waggoner, B. and Wu, Z. S. (2018). Strategic classification from revealed preferences. In Proceedings of the 2018 ACM Conference on Economics and Computation .

Drusvyatskiy, D., Fazel, M. and Ratliff, L. J. (2022). Improved rates for derivative free gradient play in strongly monotone games. In 2022 IEEE 61st Conference on Decision and Control (CDC) . IEEE.

Drusvyatskiy, D. and Xiao, L. (2022). Stochastic optimization with decision-dependent distributions. Mathematics of Operations Research .

Fischer, S. and Steinwart, I. (2020). Sobolev norm learning rates for regularized least-squares algorithms. The Journal of Machine Learning Research , 21 8464-8501.

Fudenberg, D. and Tirole, J. (1991). Game theory . MIT press.

Hardt, M., Megiddo, N., Papadimitriou, C. and Wootters, M. (2016). Strategic classification. In Proceedings of the 2016 ACM conference on innovations in theoretical computer science .

Izzo, Z., Ying, L. and Zou, J. (2021). How to learn when data reacts to your model: performative gradient descent. In International Conference on Machine Learning . PMLR.

Lei, Y., Hu, T. and Tang, K. (2021). Generalization performance of multi-pass stochastic gradient descent with convex loss functions. The Journal of Machine Learning Research , 22 1145-1185.

Li, Q. and Wai, H.-T. (2022). State dependent performative prediction with stochastic approximation. In International Conference on Artificial Intelligence and Statistics . PMLR.

Lin, J. and Cevher, V. (2020). Optimal convergence for distributed learning with stochastic gradient methods and spectral algorithms. The Journal of Machine Learning Research , 21 5852-5914.

Lin, J. and Rosasco, L. (2017). Optimal rates for multi-pass stochastic gradient methods. The Journal of Machine Learning Research , 18 3375-3421.

Lin, T., Zhou, Z., Ba, W. and Zhang, J. (2021). Optimal no-regret learning in strongly monotone games with bandit feedback. arXiv preprint arXiv:2112.02856 .

Liu, Z. and Li, M. (2020). On the estimation of derivatives using plug-in krr estimators. arXiv preprint arXiv:2006.01350 .

Lu, Y., Blanchet, J. and Ying, L. (2022). Sobolev acceleration and statistical optimality for learning elliptic equations via gradient descent. arXiv preprint arXiv:2205.07331 .

Mendelson, S. and Neeman, J. (2010). Regularization in kernel learning. Annals of statistics , 38 526-565.

Mendler-Dünner, C., Perdomo, J., Zrnic, T. and Hardt, M. (2020). Stochastic optimization for performative prediction. Advances in Neural Information Processing Systems , 33 4929-4939.

Mertikopoulos, P. and Zhou, Z. (2019). Learning in games with continuous action sets and unknown payoff functions. Mathematical Programming , 173 465-507.

Miller, J., Milli, S. and Hardt, M. (2020). Strategic classification is causal modeling in disguise. In International Conference on Machine Learning . PMLR.

Miller, J. P., Perdomo, J. C. and Zrnic, T. (2021). Outside the echo chamber: Optimizing the performative risk. In International Conference on Machine Learning . PMLR.

Narang, A., Faulkner, E., Drusvyatskiy, D., Fazel, M. and Ratliff, L. J. (2022). Multiplayer performative prediction: Learning in decision-dependent games. arXiv preprint arXiv:2201.03398 .

Nash Jr, J. (1996). Non-cooperative games. In Essays on Game Theory . Edward Elgar Publishing, 22-33.

Nesterov, Y. et al. (2018). Lectures on convex optimization , vol. 137. Springer.

Perdomo, J., Zrnic, T., Mendler-Dünner, C. and Hardt, M. (2020). Performative prediction. In International Conference on Machine Learning . PMLR.

Piliouras, G. and Yu, F.-Y. (2022). Multi-agent performative prediction: From global stability and optimality to chaos. arXiv preprint arXiv:2201.10483 .

Pillaud-Vivien, L., Rudi, A. and Bach, F. (2018). Statistical optimality of stochastic gradient descent on hard learning problems through multiple passes. Advances in Neural Information Processing Systems , 31 .

Ratliff, L. J., Burden, S. A. and Sastry, S. S. (2016). On the characterization of local nash equilibria in continuous games. IEEE transactions on automatic control , 61 2301-2307.

Rosasco, L., Belkin, M. and De Vito, E. (2010). On learning with integral operators. Journal of Machine Learning Research , 11 .

Rosen, J. B. (1965). Existence and uniqueness of equilibrium points for concave n-person games. Econometrica: Journal of the Econometric Society 520-534.

Shamir, O. (2013). On the complexity of bandit and derivative-free stochastic convex optimization. In Conference on Learning Theory . PMLR.

Smale, S. and Zhou, D.-X. (2007). Learning theory estimates via integral operators and their approximations. Constructive approximation , 26 153-172.

Steinwart, I. and Christmann, A. (2008). Support vector machines . Springer Science & Business Media.

Steinwart, I., Hush, D. R., Scovel, C. et al. (2009). Optimal rates for regularized least squares regression. In COLT .

Steinwart, I. and Scovel, C. (2012). Mercer's theorem on general domains: On the interaction between measures, kernels, and rkhss. Constructive Approximation , 35 363-417.

Tarres, P. and Yao, Y. (2014). Online learning as stochastic approximation of regularization paths: Optimality and almost-sure convergence. IEEE Transactions on Information Theory , 60 5716-5735.

Tatarenko, T. and Kamgarpour, M. (2019). Learning nash equilibria in monotone games. In 2019 IEEE 58th Conference on Decision and Control (CDC) . IEEE.

Tatarenko, T. and Kamgarpour, M. (2020). Bandit online learning of nash equilibria in monotone games. arXiv preprint arXiv:2009.04258 .

Triebel, H. (1995). Interpolation theory, function spaces. Differential Operators .

Triebel, H. (2010). Theory of Function Spaces . Modern Birkhäuser Classics, Springer Basel.

Wahba, G. (1990). Spline models for observational data . SIAM.

Wood, K., Bianchin, G. and Dall'Anese, E. (2021). Online projected gradient descent for stochastic optimization with decision-dependent distributions. IEEE Control Systems Letters , 6 1646-1651.

Yao, Y., Rosasco, L. and Caponnetto, A. (2007). On early stopping in gradient descent learning. Constructive Approximation , 26 289-315.

Ying, Y. and Pontil, M. (2008). Online gradient descent learning algorithms. Foundations of Computational Mathematics , 8 561-596.

Peer reviews:
comments: SUMMARY:
This paper studies the problem of multi-agent performative game. The authors generalize the existing assumption in (Narang et al., 2022) from linear cases to generalized linear cases, and proposed algorithms consisting online stochastic approximation of the link function and individual gradient updates. They prove convergence rates of linear function class and kernel function class, with numerical experiments validate their theory.

STRENGTHS:
The problem statement is clear, and the solution is intuitive. The authors were the first to discuss how to estimate the unknown structure of the distribution in multi-agent performative games, which is a definitely important topic in the machine learning literature.

WEAKNESSES:
The authors should elaborate more about their technical contribution. The current recipe of the proofs mostly follows (Narang et al., 2022), with the accurate parameter of distribution map replaced by an estimated version, which further generates similar error terms that can be controlled via similar techniques in bounding existing terms. It would be better if the authors could emphasize their technical contributions on the way to their theoretical results.

QUESTIONS:
Some notations are wrong or at least confusing: 

  Page 4, assumption 1: Is the second $\mathfrak{F}^{p}$ actually $\mathfrak{F}$?

  Page 4, Line 162: Would it be better to use another notation (i.e., $\hat{f}_i$) on the left-hand-side in (7), since $f_i$ already denotes the ground-truth function of player $i$?

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study multi-agent decision-dependent games. Due to strategic interactions among agents, classical gradient-based methods cannot be directly applied in computing Nash equilibrium. Instead, the authors propose using parametric models to model the agents' interactions and the resulting OPGD algorithm. The authors show that, For strongly monotone such games, OPGD converges to NE under (mostly) standard assumptions, including those on the function classes used in defining the parametric models. Numerical experiments on toy game instances complement the theoretical findings.\n\nSTRENGTHS:\n- Simple and implementable algorithms for a new problem setting.\n- The convergence rate results use mostly reasonable assumptions that are common in stochastic optimization and online regression.\n- Proof sketches are informative and help understand the proof ideas.\n\nWEAKNESSES:\n- Strong linearity assumption and simple, 1-agent experiments which seem to be the same as known online regression problems.\n- \nSee **Questions** for details.\n\nQUESTIONS:\n- The experiments are on very simple instances, on which the algorithms seem to degenerate to known online regression algorithms. Are there specific reasons (technical difficulty or otherwise) that the authors decide not to include any experiments with $\\geq 2$ agents? It would be helpful to have these experiments and ideally compare with existing algorithms (e.g., Section 4.3 Repeated Stochastic Gradient Method in https://arxiv.org/pdf/2201.03398.pdf). I believe for linear parametric models, OPGD would outperform those as it makes uses of this structure.\n\n- The linearity assumption (Assumption 6) is quite restrictive. However, as a first step toward understanding the convergence behavior of OPGD, this looks ok. It would be valuable to know, via experiments, that the algorithm (Eq. (8) and (9)) still runs and could give high-quality approximate NE even if linearity of the parametric model does not hold. Would you be able to empirically verify this too, by constructing multi-dimensional games such that $f_i(x)$'s are not linear in $x$? More generally, I would try to have new sets of experiments with less structured $f_i$'s, and compare the \"linear OPGD\", the \"kernel function OPGD\", and an existing (likely rudimentary) baseline algorithm to demonstrate the benefit of incorporating parametric models.\n\nRegarding the above two questions, for the rebuttal period, it would be sufficient to provide a plan for new experiment setups instead of results. Although it would be great to have preliminary results.\n\n- Line 118: It may help the reader understand the setting further if the authors can add a sentence after the last sentence of this paragraph, stating that (or similar): In round $t$, of the game, each agent $i$ only has access to realized $z_i^{1}, \\dots, z_i^{t-1}$, and solves the sampled (aka ERM) version of the optimization formulation (1).\n\n- For (5), if $\\mathcal{D}_i(x)$ is a continuous distribution, then exchanging differentiation and integration/expectation requires further (boundedness) assumptions. Please verify if existing assumptions are sufficient and point this out.\n- In Theorem 1, $D_1$ involves $t$ and should *not* be a constant? Maybe $t$ should be $1$ instead (since $t\\geq 1$)?\n\nLIMITATIONS:\nThe authors have stated the limitations in terms of technical assumptions. I do not see any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes stochastic approximation-based Online Performative Gradient Descent to compute the Nash equilibrium of multi-agent decision-dependent games with bandit feedback. To this end they extend the work of Narang et al. (2022) in two ways: 1. The work uses stochastic approximation method to estimate the best linear operator $A_i$ to approximate the distribution map whereas Narang et al. (2022) assumes $A_i$ to be known. 2. Extend the result to the RKHS setting.\n\nSTRENGTHS:\n1. The work studies a relevant and interesting problem. \n\n2. The experimental results are promising. \n\n3. Extension of results from Narang et al. (2022) to RKHS extend the applicability of the results.\n\nWEAKNESSES:\nThe theoretical novelty compared to Narang et al. (2022) seems inadequate and flawed to some extent. \n\n1.I am concerned about the use of $u_i^t$ to approximate $A_i$. The main challenge of problems with decision-dependent distribution is that the distribution depends on the *decisions* that is $x_i^t$ which makes estimation of $A_i$ difficult. OPGD assumes access to random samples from the distribution $\\rho_X$, and observes corresponding $y$ and use that to estimate $A_i$. If one assumes access to a separate sample $(u_i,y_i)\\sim \\rho_i$ then estimating $A_i$ is a simple stochastic approximation procedure and the corresponding theory is well-known in the online SGD and stochastic approximation (eqn (27), (28)). Combining these results with the ones (Theorem 6) in Narang et al. (2022) directly gives Theorem 1.  \n\n2. Say it was indeed possible to query such a point $u_i$. Then how to estimate $A_i$ is already explored in Section 6.3 of Narang et al. (2022) and it's almost the same method as OPGD with same theoretical rate (Theorem 7). \n\n3. The extension to RKHS increases the applicability but the proof techniques are almost the same, albeit involve more tedious calculations, as the linear case which is expected since it is after all a linear setting. The authors concur as well (line 310). \n\n4. Minor point: Assumption 7 is satisfied by Gaussian distn but here $\\rho_X$ is supposed to have a compact support since $X$ is compact. So $\\rho_X$ can't be Gaussian. But anyway, Assumption 7 can be satisfied by other distribution and it's not a major concern.\n\nQUESTIONS:\nIf you could clarify your novelties over Narang et al. (2022), I would love to reconsider my score.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This work studies the framework of decision dependent games, where the actions of the agents in the game do not only affect the utilities of the players but also the distribution of population data observed by them. The critical challenge of this type of games is that the data population sampling process is not in general differentiable with respect to the parameters of the agents. The authors circumvent this by proposing a noisy stochastic differentiable model of the sampling distribution. The model itself is parametric and the authors propose OPGD an algorithm that learns both a the model parameters and the players strategy. The authors analyze both the linear and the RKHS setting for strongly monotone games.

STRENGTHS:
The authors have clearly explained the proposed framework and the necessary background to understand the challenges involved. While elements of the linear model setting existed in Narang et al, at least the RKHS component is to the best of my knowledge novel. The experimental results may be helpful to practitioners and may lead to practical (in addition to the theoretical) impact.

WEAKNESSES:
I think the differentiation between Narang et al and this work should be made more clear throughout. More specifically, what is the unique insight that allowed this work to arrive at the novel results?

QUESTIONS:
Is it possible for the players to first approximately learn the parameters of $f$ through the bandit feedback and then armed with that knowledge play the monotone game? If this is the case, how do the proposed algorithms compare to this baseline?

LIMITATIONS:
Limitations are addressed by the authors.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
The authors focus on the analysis of learning the Nash equilibrium (which is unique) in strongly monotone games that are decision-dependent. They provide a definition of decision-dependent games generalizing a recent work of Narang et al. Intuitively speaking, if an agent chooses an action, he gets an observation (that depends on a distribution that is parametrized of everyone's actions) and his utility depends on the observed data (bandit setting) and everyone's actions. The authors provide an algorithm that converges to the Nash equilibrium in a rate of 1/t when the observed data depend linearly in the actions (linear regression type dependent-game), i.e., the focus is linear function classes. Further they focus on function kernel classes and provide similar type rates on convergence to the Nash equilibrium.

STRENGTHS:
The idea of decision-dependent games is interesting, though it was firstly defined in another paper.

WEAKNESSES:
The algorithm (projection type gradient descent with also updating the estimate on the A matrix) and the analysis seems not novel and not surprising. I think the paper would be more interesting if the authors could argue for monotone and not only strongly monotone games or even going beyond that (e.g., games that satisfy MVI for example).

QUESTIONS:
What is the main technical challenge of your work?

LIMITATIONS:
Not applicable
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-13