SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zihan Zhu*
Ethan X. Fang†
We study multi-agent games within the innovative framework of decision-dependent games, which establishes a feedback mechanism that population data reacts to agents' actions and further characterizes the strategic interactions among agents. We focus on finding the Nash equilibrium of decision-dependent games in the bandit feedback setting. However, since agents are strategically coupled, classical gradient-based methods are infeasible without the gradient oracle. To overcome this challenge, we model the strategic interactions by a general parametric model and propose a novel online algorithm, Online Performative Gradient Descent (OPGD), which leverages the ideas of online stochastic approximation and projected gradient descent to learn the Nash equilibrium in the context of function approximation for the unknown gradient. In particular, under mild assumptions on the function classes defined in the parametric model, we prove that the OPGD algorithm finds the Nash equilibrium efficiently for strongly monotone decision-dependent games. Synthetic numerical experiments validate our theory.
The classical theory of learning and prediction fundamentally relies on the assumption that data follows a static distribution. This assumption, however, does not account for many dynamic real-world scenarios where decisions can influence the data involved. Recent literature on performative classification (Hardt et al., 2016; Dong et al., 2018; Miller et al., 2020) and performative prediction (Perdomo et al., 2020) offers a variety of examples where agents are strategic, and data is performative. For instance, in the ride-sharing market, both passengers and drivers engage with multiple platforms using various strategies such as "price shopping". Consequently, these platforms observe performative demands, and the pricing policy becomes strategically coupled.
In this paper, we explore the multi-agent performative prediction problem, specifically, the multi-agent decision-dependent games, as proposed by Narang et al. (2022). We aim to develop algorithms to find Nash equilibria with the first-order oracle. In this scenario, agents can only access their utility functions instead of gradients through the oracle. Finding Nash equilibria in decision-dependent games is a challenging task. Most existing works primarily focus on finding performative stable equilibria within the single-agent setting, an approach that approximates the Nash equilibrium and is relatively straightforward to compute (Mendler-Dünner et al., 2020; Wood et al., 2021; Drusvyatskiy and Xiao, 2022; Brown et al., 2022; Li and Wai, 2022).
There are two major challenges associated with this problem: (i) the distribution shift induced by performative data, and (ii) the lack of first-order information for the performative gradient. To address these two challenges, we propose a novel online gradient-based algorithm, Online Performative Gradient Descent (OPGD). In particular, our algorithm employs a general parametric framework to
*Department of Statistical Science, Duke University. zihan.zhu@duke.edu
†Department of Biostatistics and Bioinformatics, Duke University. xingyuan.fang@duke.edu
$^{\ddagger}$ Department of Statistics and Data Science, Yale University. zhuoran.yang@yale.edu
model the decision-dependent distribution, which provides an unbiased estimator for the unknown gradient, and leverages online stochastic approximation methods to estimate the parametric functions.
1.1 Major Contributions
Our work provides new fundamental understandings of decision-dependent games. Expanding upon the linear parametric assumption in Narang et al. (2022), we propose a more comprehensive parametric framework that models decision-dependent distributions of the observed data. We also derive sufficient conditions under this parametric framework that guarantee a strongly monotone decision-dependent game, thereby ensuring a unique Nash equilibrium.
From the algorithmic perspective, we propose OPGD, the first online algorithm to find the Nash equilibrium under linear and kernel parametric models. The core problem in decision-dependent games is estimating the performative gradient. We remark that the existing algorithm only handles the linear case and cannot be extended to the non-linear parametric model (Section 3), and OPGD uses an essentially different method to learn the strategic interaction. To elaborate, under the proposed parametric framework, learning the Nash equilibrium in decision-dependent games can be formulated as a bilevel problem, where the lower level is learning the strategic model and the upper level is finding equilibriums. The OPGD algorithm leverages the ideas of online stochastic approximation for the lower problem and projected gradient descent to learn the Nash equilibrium. Moreover, we acknowledge this learning framework bridges online optimization and statistical learning with time-varying models.
We further prove that under mild assumptions,  $\mathsf{OPGD}$  converges to the Nash equilibrium. For the linear function class,  $\mathsf{OPGD}$  achieves a convergence rate of  $\mathcal{O}(t^{-1})$ , matching the optimal rate of SGD in the strongly-convex setting, where  $t$  represents the number of iterations. For the kernel function class  $\mathcal{H}$  associated with a bounded kernel  $K$ , we posit that the parametric functions reside within the power space  $\mathcal{H}^{\beta}$  and evaluate the approximation error of  $\mathsf{OPGD}$  under the  $\alpha$ -power norm, where  $\alpha$  represents the minimal value that ensures the power space  $\mathcal{H}^{\alpha}$  possesses a bounded kernel. We present the first analysis for online stochastic approximation under the power norm (Lemma 4), in contrast to the classical RKHS norm (Tarres and Yao, 2014; Pillaud-Vivien et al., 2018; Lei et al., 2021). The difference between the RKHS  $\mathcal{H}$  and the power space  $\mathcal{H}^{\beta}$  makes the standard techniques fail under the power norm, and we use novel proof steps to obtain the estimation error bound. We demonstrate that  $\mathsf{OPGD}$  leverages the embedding property of the kernel  $K$  to accelerate convergence and achieves the rate of  $\mathcal{O}(t^{-\frac{\beta - \alpha}{\beta - \alpha + 2}})$ . Moreover,  $\mathsf{OPGD}$  can handle the challenging scenario, where parametric functions are outside the RKHS. See Section 4.2 for more details.
Performatives prediction. The multi-agent decision-dependent game in this paper is inspired by the performative prediction framework (Perdomo et al., 2020). This framework builds upon the pioneering works of strategic classification (Hardt et al., 2016; Dong et al., 2018; Miller et al., 2020), and extends the classical statistical theory of risk minimization to incorporate the performativity of data. Perdomo et al. (2020); Mendler-Dünner et al. (2020); Miller et al. (2021) introduce the concepts of performative optimality and stability, demonstrating that repeated retraining and stochastic gradient methods converge to the performatively stable point. Miller et al. (2021), in pursuit of the performatively optimal point, model the decision-dependent distribution using location families and propose a two-stage algorithm. Similarly, Izzo et al. (2021) develop algorithms to estimate the unknown gradient using finite difference methods. More recently, Narang et al. (2022); Piliouras and Yu (2022) expand the performative prediction to the multi-agent setting, deriving algorithms to find the performatively optimal point.
Learning in continuous games. Our work aligns closely with optimization in continuous games. Rosen (1965) lays the groundwork, deriving sufficient conditions for a unique Nash equilibrium in convex games. For strongly monotone games, Bravo et al. (2018); Mertikopoulos and Zhou (2019); Lin et al. (2021) achieve the convergence rate and iteration complexity of stochastic and derivative-free gradient methods. For monotone games, the convergence of such methods is established by Tatarenko and Kamgarpour (2019, 2020). Additional with bandit feedback settings, zeroth-order methods (or derivative-free methods) achieve convergence (Bravo et al., 2018; Lin et al., 2021; Drusvyatskiy et al., 2022; Narang et al., 2022), albeit with slow convergence rates (Shamir, 2013; Lin et al., 2021; Narang et al., 2022). Relaxing the convex assumption, Ratliff et al. (2016); Agarwal et al. (2019); Cotter et al. (2019) study non-convex continuous games in various settings.
Learning with kernels. Our proposed algorithm closely relies on stochastic approximation, utilizing online kernel regression for the RKHS function class. Prior research investigates the generalization capability of least squares and ridge regression in RKHS De Vito et al. (2005); Caponnetto and De Vito (2007); Smale and Zhou (2007); Rosasco et al. (2010); Mendelson and Neeman (2010). Meanwhile, extensive works study algorithms for kernel regression. For instance, Yao et al. (2007); Dieuleveut and Bach (2016); Pillaud-Vivien et al. (2018); Lin and Rosasco (2017); Lei et al. (2021) propose offline algorithms with optimal convergence rates under the RKHS norm and  $L^2$  norm using early stopping and stochastic gradient descent methods, while Ying and Pontil (2008); Tarres and Yao (2014); Dieuleveut and Bach (2016) design online algorithms with optimal convergence rates. The convergence of kernel regression in power norm (or Sobolev norm) is studied in Steinwart et al. (2009); Fischer and Steinwart (2020); Liu and Li (2020); Lu et al. (2022), with offline spectral filter algorithms achieving the statistical optimal rate under the power norm (Pillaud-Vivien et al., 2018; Blanchard and Mücke, 2018; Lin and Cevher, 2020; Lu et al., 2022).
Notation. We introduce some useful notation before proceeding. Throughout this paper, we denote the set  $1,2,\dots ,n$  by  $[n]$  for any positive integer  $n$ . For two positive sequences  $\{a_{n}\}_{n\in \mathbb{N}}$  and  $\{b_n\}_{n\in \mathbb{N}}$ , we write  $a_{n} = \mathcal{O}(b_{n})$  or  $a_{n}\lesssim b_{n}$  if there exists a positive constant  $C$  such that  $a_{n}\leq C\cdot b_{n}$ . For any integer  $d$ , we denote the  $d$ -dimensional Euclidean space by  $\mathbb{R}^d$ , with inner produce  $\langle \cdot ,\cdot \rangle$  and the induced norm  $\| \cdot \| = \sqrt{\langle\cdot,\cdot\rangle}$ . For a Hilbert space  $\mathcal{H}$ , let  $\| \cdot \|_{\mathcal{H}}$  be the associated Hilbert norm. For a set  $\mathcal{X}$  and a probability measure  $\rho_{\mathcal{X}}$  on  $\mathcal{X}$ , let  $\mathcal{L}_{\rho_{\mathcal{X}}}^{2}$  be the  $L^2$  space on  $\mathcal{X}$  induced by the measure  $\rho_{\mathcal{X}}$ , equipped with inner product  $\langle \cdot ,\cdot \rangle_{\rho_{\mathcal{X}}}$  and  $L^2$  norm  $\| \cdot \|_{\rho_{\mathcal{X}}} = \sqrt{\langle\cdot,\cdot\rangle_{\rho_{\mathcal{X}}}}$ . For any matrix  $A = (a_{ij})$ , the Frobenius norm and the operator norm (or spectral norm) of  $A$  are  $\| A\| _F = (\sum_{i,j}a_{ij}^2)^{1 / 2}$  and  $\| A\|_{\mathrm{op}} = \sigma_1(A)$ , where  $\sigma_1(A)$  stands for the largest singular value of  $A$ . For any square matrix  $A = (a_{ij})$ , denote its trace by  $\operatorname {tr}(A) = \sum_i a_{ii}$ . For any  $y\in \mathbb{R}^d$ , we denote its projection onto a set  $\mathcal{X}\subset \mathbb{R}^d$  by  $\operatorname {proj}_X(y) = \arg \min_{x\in \mathcal{X}}\| x - y\|$ . The set denoted by  $N_{\mathcal{X}}(x)$  represents the normal cone to a convex set  $\mathcal{X}$  at  $x\in \mathcal{X}$ , namely,  $N_{\mathcal{X}}(x) = \{v\in \mathbb{R}^{d}: \langle v,y - x\rangle \leq 0, \forall y\in \mathcal{X}\}$ . For any metric space  $\mathcal{Z}$  with metric  $d(\cdot ,\cdot)$ , the symbol  $\mathbb{P}(\mathcal{Z})$  will denote the set of Radon probability measures  $\mu$  on  $\mathcal{Z}$  with a finite first moment  $\mathbb{E}_{z\sim \mu}[d(z,z_0)] < \infty$  for some  $z_0\in \mathcal{Z}$ .
We briefly introduce the formulation of  $n$ -agent decision-dependent games based on Narang et al. (2022). In this setting, each agent  $i \in [n]$  takes the action  $x_i \in \mathcal{X}_i$  from an action set  $\mathcal{X}_i \subset \mathbb{R}^{d_i}$ . Define the joint action  $x := (x_1, x_2, \dots, x_n) \in \mathcal{X}$  and the joint action set  $\mathcal{X} = \mathcal{X}_1 \times \dots \times \mathcal{X}_n \subset \mathbb{R}^d$ , where  $d := \sum_{i=1}^{n} d_i$ . For all  $i \in [n]$ , we write  $x = (x_i, x_{-i})$ , where  $x_{-i}$  denotes the vector of all coordinates except  $x_i$ . Let  $\mathcal{L}_i: \mathcal{X} \to \mathbb{R}$  be the utility function of agent  $i$ . In the game, each agent  $i$  seeks to solve the problem
$$
\min  _ {x _ {i} \in \mathcal {X} _ {i}} \mathcal {L} _ {i} \left(x _ {i}, x _ {- i}\right), \quad \text {w h e r e} \quad \mathcal {L} _ {i} (x) := \underset {z _ {i} \sim \mathcal {D} _ {i} (x)} {\mathbb {E}} \ell_ {i} \left(x, z _ {i}\right). \tag {1}
$$
Here  $z_{i} \in \mathcal{Z}_{i}$  represents the data observed by agent  $i$ , where the sample space  $\mathcal{Z}_i$  is assumed to be  $\mathcal{Z}_i = \mathbb{R}^p$  with  $p \in \mathbb{N}$  throughout this paper. Moreover,  $\mathcal{D}_i: \mathcal{X} \to \mathbb{P}(\mathcal{Z}_i)$  is the distribution map, and  $\ell_i: \mathbb{R}^d \times \mathcal{Z}_i \to \mathbb{R}$  denotes the loss function. During play, each agent  $i$  performs an action  $x_{i}$  and observes performative data  $z_{i} \sim \mathcal{D}_{i}(x)$ , where the performativity is modeled by the decision-dependent distribution  $D_{i}(x)$ . In the round  $t$ , the agent  $i$  only has access to  $z_{i}^{1}, \dots, z_{i}^{t - 1}$  as well as  $x^{1}, \dots, x^{t - 1}$  and seeks to solve the ERM version of (1). We assume the access to the first-order oracle, namely, loss functions  $\ell_{i}$  are known to agents but distribution maps  $D_{i}$  are unknown.
Definition 1. (Nash equilibrium). In the game (1), a joint action  $x^{*} = (x_{1}^{*}, x_{2}^{*}, \dots, x_{n}^{*})$  is a Nash equilibrium (Nash Jr, 1996) if all agents play the best response against other agents, namely,
$$
x _ {i} ^ {*} = \underset {x _ {i} \in \mathcal {X} _ {i}} {\arg \min } \mathcal {L} _ {i} \left(x _ {i}, x _ {- i} ^ {*}\right) = \underset {x _ {i} \in \mathcal {X} _ {i}} {\arg \min } \underset {z _ {i} \sim \mathcal {D} _ {i} \left(x _ {i}, x _ {- i} ^ {*}\right)} {\mathbb {E}} \ell_ {i} \left(x _ {i}, x _ {- i} ^ {*}, z _ {i}\right), \quad \forall i \in [ n ]. \tag {2}
$$
In general continuous games, Nash equilibria may not exist or there might be multiple Nash equilibria (Fudenberg and Tirole, 1991). The existence and uniqueness of a Nash equilibrium in a continuous game depend on the game's structure and property. In general, finding the unique Nash equilibrium is only possible for convex and strongly monotone games (Debreu, 1952).
Definition 2. (Convex game). Game (1) is a convex game if sets  $\mathcal{X}_i$  are non-empty, compact, convex and utility functions  $\mathcal{L}_i(x_i, x_{-i})$  are convex in  $x_i$  when  $x_{-i}$  are fixed.
Suppose that utility functions  $\mathcal{L}_i$  are differentiable, we use  $\nabla_i\mathcal{L}_i(x)$  to denote the gradient of  $\mathcal{L}_i(x)$  with respect to  $x_{i}$  (the  $i$ -th individual gradient). We say the game (1) is  $C^1$ -smooth if the gradient  $\nabla_i\mathcal{L}_i(x)$  exists and is continuous for all  $i\in [n]$ . Using this notation, we define the gradient  $H(x)$  comprised of individual gradients
$$
H (x) := \left(\nabla_ {1} \mathcal {L} _ {1} (x), \dots , \nabla_ {n} \mathcal {L} _ {n} (x)\right).
$$
Definition 3. (Strongly monotone game). For a constant  $\tau \geq 0$ , a  $C^1$ -smooth convex game (1) is called  $\tau$ -strongly monotone if it satisfies
$$
\langle H (x) - H \left(x ^ {\prime}\right), x - x ^ {\prime} \rangle \geq \tau \| x - x ^ {\prime} \| ^ {2}, \quad f o r a l l x, x ^ {\prime} \in \mathcal {X}.
$$
Note that a  $\tau$ -strongly monotone game  $(\tau > 0)$  over a compact and convex action set  $\mathcal{X}$  admits a unique Nash equilibrium (Rosen, 1965). According to the optimal conditions in convex optimization (Boyd et al., 2004), this Nash equilibrium  $x^{*}$  is characterized by the variational inequality
$$
0 \in H \left(x ^ {*}\right) + N _ {\mathcal {X}} \left(x ^ {*}\right). \tag {3}
$$
We briefly talk about the challenges and our idea of designing the algorithm. In decision-dependent games, the classical theory of risk minimization does not work. The primary obstacles to finding the Nash equilibrium in the game (1) include: (i) the distribution shift induced by performative data, and (ii) the lack of first-order information (gradient). To make it clear, standard methods, such as gradient-based algorithms, necessitate the gradient  $H(x)$ . However,  $H(x)$  is unknown since distributions  $\mathcal{D}_i$  are unknown, and estimating  $H(x)$  is complex due to the dependency between  $\mathcal{D}_i(x)$  and  $x$ . Mathematically, assuming  $C^1$ -smoothness, the chain rule directly yields the following expression for the gradient
$$
\nabla_ {i} \mathcal {L} _ {i} (x) = \underset {z _ {i} \sim \mathcal {D} _ {i} (x)} {\mathbb {E}} \nabla_ {i} \ell_ {i} \left(x _ {i}, x _ {- i}, z _ {i}\right) + \frac {d}{d u _ {i}} \underset {z _ {i} \sim \mathcal {D} _ {i} \left(u _ {i}, x _ {- i}\right)} {\mathbb {E}} \left. \ell_ {i} \left(x _ {i}, x _ {- i}, z _ {i}\right) \right| _ {u _ {i} = x _ {i}}, \tag {4}
$$
where  $\nabla_{i}\ell_{i}(x,z_{i})$  denotes the gradient of  $\ell_i(x,z_i)$  with respect to  $x_{i}$ . The main difficulty is estimating the second term in (4) due to the absence of closed-form expressions.
To estimate the unknown gradient  $H(x)$ , we impose a parametric assumption on the observed data  $z_{i}$  and model the distribution maps  $\mathcal{D}_i$  using parametric functions. Note that the linear parametric assumption was first proposed in Narang et al. (2022). In this paper, we extend this assumption to a general framework and show that under the parametric assumption, the gradient  $H(x)$  has a closed-form expression, which yields an unbiased estimator for  $H(x)$ .
Assumption 1. (Parametric assumption). Suppose there exists a function class  $\mathcal{F}$  and  $p$ -dimensional functions  $f_{i}:\mathcal{X}\to \mathbb{R}^{p}$  over the joint action set  $\mathcal{X}$  such that  $f_{i}\in \mathcal{F}^{p}$  and
$$
z _ {i} \sim \mathcal {D} _ {i} (x) \Longleftrightarrow z _ {i} = f _ {i} (x) + \epsilon_ {i}, \quad \forall i \in [ n ],
$$
where  $\epsilon_{i}\in \mathbb{R}^{p}$  are zero-mean noise terms with finite variance  $\sigma^2$ , namely,  $\mathbb{E}\epsilon_i = 0$  and  $\mathbb{E}\| \epsilon_i\| ^2\leq \sigma^2$
Under Assumption 1, assuming that  $f_{i}$  are differentiable and letting  $\mathcal{P}_i$  be the distribution of the noise term  $\epsilon_i$ , we derive the following expression for the utility functions  $\mathcal{L}_i(x) = \mathbb{E}_{z_i\sim \mathcal{D}_i(x)}\ell_i(x,z_i) = \mathbb{E}_{\epsilon_i\sim \mathcal{P}_i}\ell_i(x,f_i(x) + \epsilon_i)$ . Then the individual gradient would be  $\nabla_{i}\mathcal{L}_{i}(x) = \nabla_{i}\mathbb{E}_{z_{i}\sim \mathcal{D}_{i}(x)}\ell_{i}(x,z_{i}) = \nabla_{i}[\mathbb{E}_{\epsilon_{i}\sim \mathcal{P}_{i}}\ell_{i}(x,f_{i}(x) + \epsilon_{i})]$ . Consequently, the chain rule directly implies the following expression
$$
\nabla_ {i} \mathcal {L} _ {i} (x) = \underset {z _ {i} \sim \mathcal {D} _ {i} (x)} {\mathbb {E}} \nabla_ {i} \ell_ {i} (x, z _ {i}) + \left(\frac {\partial f _ {i} (x)}{\partial x _ {i}}\right) ^ {\top} \underset {z _ {i} \sim \mathcal {D} _ {i} (x)} {\mathbb {E}} \nabla_ {z _ {i}} \ell_ {i} (x, z _ {i}), \tag {5}
$$
where  $\nabla_{z_i}\ell_i(x,z_i)$  denotes the gradient of  $\ell_i(x,z_i)$  with respect to  $z_{i}$ . Given a joint action  $x$ , each agent  $i$  observes data  $z_{i}\sim \mathcal{D}_{i}(x)$ . Equation (5) suggests the following unbiased estimator for  $H(x)$ :
$$
\widehat {H} (x) := \left(\widehat {\nabla} _ {i} \mathcal {L} _ {i} (x)\right) _ {i \in [ n ]} = \left(\nabla_ {i} \ell_ {i} (x, z _ {i}) + \left(\frac {\partial f _ {i} (x)}{\partial x _ {i}}\right) ^ {\top} \nabla_ {z _ {i}} \ell_ {i} (x, z _ {i})\right) _ {i \in [ n ]}. \tag {6}
$$
However, direct computation of  $\widehat{H}(x)$  is infeasible because  $f_{i}$  are unknown. To overcome this challenge, we approximate the unknown functions  $f_{i}$  with the function class  $\mathcal{F}^p$ . In fact, the estimation of  $f_{i}$  can be formed as a non-parametric regression problem, namely,
$$
\hat {f} _ {i} = \underset {f \in \mathcal {F} ^ {p}} {\arg \min } \int_ {\mathcal {X} \times \mathcal {Z} _ {i}} \| z _ {i} - f (x) \| ^ {2} d \rho_ {i}, \quad \forall i \in [ n ], \tag {7}
$$
where  $\rho_{i}$  is the joint distribution of  $(x,z_{i})$  induced by  $x\sim \rho_{\mathcal{X}}$  and  $z_{i}\sim \mathcal{D}_{i}(x)$ . Here  $\rho_{\mathcal{X}}$  is a user-specified sampling distribution on  $\mathcal{X}$ .
3 The OPGD Algorithm
In this section, we consider  $\mathcal{F}$  to be the linear and kernel function classes and derive gradient-based online algorithms to find the Nash equilibrium in the game (1), namely, the Online Performative Gradient Descent (OPGD). In each iteration  $t$ , assuming that  $x^{t} \coloneqq (x_{1}^{t}, \dots, x_{n}^{t})$  is the output of the previous iteration, OPGD performs the following update for all  $i \in [n]$ :
(i) (Estimation update). Update the estimation of  $f_{i}$  by online stochastic approximation for (7).
(ii) (Individual gradient update). Compute the estimator (6) and perform projected gradient steps
$$
x _ {i} ^ {t + 1} = \operatorname {p r o j} _ {\mathcal {X} _ {i}} \left(x _ {i} ^ {t} - \eta_ {t} \widehat {\nabla} _ {i} \mathcal {L} _ {i} \left(x ^ {t}\right)\right), \quad \forall i \in [ n ].
$$
Linear Function Class. Let  $\mathcal{F}$  be the linear function class, namely,  $f_{i}(x) = A_{i}x$  for  $i\in [n]$ , where  $A_{i}\in \mathbb{R}^{p\times d}$  are unknown matrices. Then (7) becomes the least square problem  $A_{i} = \arg \min_{A\in \mathbb{R}^{p\times d}}\mathbb{E}_{(u_i,y_i)}\sim \rho_i\| y_i - A_i u_i\|^2$  with random variables  $u_{i}\sim \rho_{\mathcal{X}},y_{i}\sim \mathcal{D}_{i}(u_{i})$ . We use the gradient of the least square objective  $\| y_{i} - A_{i}u_{i}\|^{2}$  to derive the online least square update:  $A^{\mathrm{new}}\gets A - \nu (Au_i - y_i)u_i^\top$  (Dieuleveut et al., 2017; Narang et al., 2022). In each iteration  $t$ , we suppose that  $A_{i}^{t - 1}$  is the estimation of  $A_{i}$  from the previous iteration, OPGD samples  $u_{i}^{t}\sim \rho_{\mathcal{X}}$  and  $y_{i}^{t}\sim \mathcal{D}_{i}(u_{i}^{t})$  and performs the following estimation update:
$$
\left. \begin{array}{l} \text {(i)} \quad A _ {i} ^ {t} = A _ {i} ^ {t - 1} - \nu_ {t} \left(A _ {i} ^ {t - 1} u _ {i} ^ {t} - y _ {i} ^ {t}\right) \left(u _ {i} ^ {t}\right) ^ {\top}. \end{array} \right. \tag {8}
$$
Recalling (5), the individual gradient is  $\nabla_{i}\mathcal{L}_{i}(x) = \mathbb{E}_{z_{i}\sim \mathcal{D}_{i}(x)}\left[\nabla_{i}\ell_{i}(x,z_{i}) + A_{ii}^{\top}\nabla_{z_{i}}\ell_{i}(x,z_{i})\right]$ , where  $A_{ii} = \partial f_i(x) / \partial x_i\in \mathbb{R}^{p\times d_i}$  denotes the submatrix of  $A_{i}$  whose columns are indexed by the agent  $i$ . After step (i), OPGD draws a sample  $z_{i}^{t}\sim \mathcal{D}_{i}(x^{t})$  and compute the estimator (6) to perform the projected gradient step:
$$
\left. \left(x _ {i} ^ {t + 1} = \operatorname {p r o j} _ {\mathcal {X} _ {i}} \left(x _ {i} ^ {t} - \eta_ {t} \left(\nabla_ {i} \ell_ {i} \left(x ^ {t}, z _ {i} ^ {t}\right) + \left(A _ {i i} ^ {t}\right) ^ {\top} \nabla_ {z _ {i}} \ell_ {i} \left(x ^ {t}, z _ {i} ^ {t}\right)\right)\right)\right). \right. \tag {9}
$$