SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Online Parameter Estimation: We estimate the parameter  $\mathbf{A}$  via online least squares. In each round of the alternating gradient update, we first take the current decision  $\pmb{\theta}_t$  and its perturbed point  $\pmb{\theta}_t + \mathbf{u}_t$  to observe samples  $Z_{t} \sim \mathcal{D}(\pmb{\theta}_{t})$  and  $Z_{t}^{\prime} \sim \mathcal{D}(\pmb{\theta}_{t} + \mathbf{u}_{t})$ , respectively, where  $\mathbf{u}_t$  is an injected noise specified by the decision-maker. We have  $\mathbb{E}[Z_t - Z_t'|\mathbf{u}_t] = \mathbf{A}\mathbf{u}_t$ . Then, the least-square problem at the  $t$ th iteration is designed as
$$
\min _ {\mathbf {A}} \frac {1}{2} \left\| Z _ {t} ^ {\prime} - Z _ {t} - \mathbf {A u} _ {t} \right\| _ {2} ^ {2}.
$$
Let  $\widehat{\mathbf{A}}_{t-1}$  be the estimate of  $\mathbf{A}$  at the  $(t-1)$ th iteration. Based on it, we construct a new estimate  $\widehat{\mathbf{A}}_t$  for  $\mathbf{A}$  by using gradient descent on the above least-square objective. This gives us the update
$$
\widehat {\mathbf {A}} _ {t} = \widehat {\mathbf {A}} _ {t - 1} + \zeta_ {t} \left(Z _ {t} ^ {\prime} - Z _ {t} - \widehat {\mathbf {A}} _ {t - 1} \mathbf {u} _ {t}\right) \mathbf {u} _ {t} ^ {\top},
$$
Algorithm 1 Adaptive Primal-Dual Algorithm
1: Take decision  $\pmb{\theta} = \mathbf{0}$  and observe  $n$  samples  $Z_{0,i}\sim \mathcal{D}_0,\forall i\in [n]$
2: Initialize  $\theta_{1} \in \Theta$  arbitrarily. Set  $\lambda_{1} = 0$  and  $\widehat{\mathbf{A}}_{0} = \mathbf{0}$ .
3: for  $t = 1$  to  $T$  do
4: Take decision  $\pmb{\theta}_t$  and observe  $Z_{t}\sim \mathcal{D}(\pmb{\theta}_{t})$
5: Generate noise  $\mathbf{u}_t$
6: Take decision  $\pmb{\theta}_t + \mathbf{u}_t$  and observe  $Z_t^\prime \sim \mathcal{D}(\pmb {\theta}_t + \mathbf{u}_t)$
7: Update parameter estimate by  $\widehat{\mathbf{A}}_t = \widehat{\mathbf{A}}_{t - 1} + \zeta_t\left(Z_t' - Z_t - \widehat{\mathbf{A}}_{t - 1}\mathbf{u}_t\right)\mathbf{u}_t^\top$
8: Update gradient approximation  $\nabla_{\pmb{\theta}}\widehat{\mathrm{PR}}_t(\pmb{\theta}_t)$  by (8).
9: Compute  $\nabla_{\pmb{\theta}}\widehat{\mathcal{L}}_t(\pmb {\theta}_t,\pmb {\lambda}_t) = \nabla_{\pmb{\theta}}\widehat{\mathrm{PR}}_t(\pmb {\theta}_t) + \nabla_{\pmb{\theta}}\mathbf{g}(\pmb {\theta}_t)^\top \pmb {\lambda}_t$
0: Update the primal variable by  $\pmb{\theta}_{t + 1} = \Pi_{\Theta}\left(\pmb{\theta}_t - \eta \nabla_\theta \widehat{\mathcal{L}}_t(\pmb{\theta}_t,\pmb{\lambda}_t)\right)$
11: Compute  $\nabla_{\pmb{\lambda}}\mathcal{L}(\pmb{\theta}_t,\pmb{\lambda}_t) = \mathbf{g}(\pmb{\theta}_t) - \delta \eta \pmb{\lambda}_t$
12: Update the dual variable by  $\pmb{\lambda}_{t + 1} = [\pmb{\lambda}_t + \eta \nabla_{\pmb{\lambda}}\mathcal{L}(\pmb{\theta}_t,\pmb{\lambda}_t)]^+$ .
13: end for
where  $\zeta_t$  is the stepsize of the online least squares at the  $t$ th iteration.
Adaptive Primal-Dual Algorithm: With the above preparation, we obtain an approximation for the gradient  $\nabla_{\theta}\mathrm{PR}(\pmb{\theta}_t)$  at the  $t$ th iteration as
$$
\nabla_ {\boldsymbol {\theta}} \widehat {\operatorname {P R}} _ {t} (\boldsymbol {\theta} _ {t}) := \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \nabla_ {\boldsymbol {\theta}} \ell \left(\boldsymbol {\theta} _ {t}; Z _ {0, i} + \widehat {\mathbf {A}} _ {t} \boldsymbol {\theta} _ {t}\right) + \widehat {\mathbf {A}} _ {t} ^ {\top} \nabla_ {Z} \ell \left(\boldsymbol {\theta} _ {t}; Z _ {0, i} + \widehat {\mathbf {A}} _ {t} \boldsymbol {\theta} _ {t}\right) \right]. \tag {8}
$$
Given  $\nabla_{\theta}\widehat{\mathrm{PR}}_t(\theta_t)$ , we develop an adaptive primal-dual algorithm for the constrained performative prediction problem (1) based on the robust primal-dual framework in § 3.1, which is presented in Algorithm 1. In Algorithm 1, the initial decision is randomly chosen from the admissible set  $\Theta$ . Both the dual variable and the parameter estimate  $\widehat{\mathbf{A}}_0$  are initialized to be zero. The algorithm maintains two sequences. One is the estimate  $\widehat{\mathbf{A}}_t$ , which is updated based on the newly observed samples  $Z_t$  and  $Z_t'$ , as given in Step 7. The other is the alternating gradient update on the primal and dual variables, which are respectively given in Step 10 and Step 12.
Remark 1. While this paper considers the distribution maps within the location family, we emphasize that the proposed robust primal-dual framework does not restrict to any form of distribution. For instance, the exponential family considered in [Izzo et al., 2021] with their gradient approximation method can be directly applied to our robust primal-dual framework.
In this section, we analyze the convergence performance of the proposed adaptive primal-dual algorithm. We first provide the convergence result of the robust primal-dual framework. Then, we bound the error of gradient approximation in our adaptive algorithm for the location family. With these results, the convergence bounds of the adaptive primal-dual algorithm are derived. Our analysis is based on the following assumptions.
Assumption 1 (Properties of  $\ell(\theta; Z)$ ). The loss function  $\ell(\theta; Z)$  is  $\beta$ -smooth,  $L_{\theta}$ -Lipschitz continuous in  $\theta$ ,  $L_{Z}$ -Lipschitz continuous in  $Z$ ,  $\gamma_{\theta}$ -strongly convex in  $\theta$ , and  $\gamma_{Z}$ -strongly convex in  $Z$ . Moreover, we have  $\gamma_{\theta} - \beta^2 / \gamma_Z > 0$ .
Assumption 2 (Compactness and Boundedness of  $\Theta$ ). The set of admissible decisions  $\Theta$  is closed, convex, and bounded, i.e., there exists a constant  $R > 0$  such that  $\| \pmb{\theta} \|_2 \leq R$ ,  $\forall \pmb{\theta} \in \Theta$ .
Assumption 3 (Properties of  $\mathbf{g}(\pmb{\theta})$ ). The constraint function  $\mathbf{g}(\pmb{\theta})$  is convex,  $L_{\mathbf{g}}$ -Lipschitz continuous, and bounded, i.e., there exists a constant  $C$  such that  $\| \mathbf{g}(\pmb{\theta}) \|_2 \leq C, \forall \pmb{\theta} \in \Theta$ .
Assumption 4 (Bounded Stochastic Gradient Variance). For any  $i \in [n]$  and  $\theta \in \Theta$ , there exists  $\sigma \geq 0$  such that
$$
\mathbb {E} _ {Z _ {0, i} \sim \mathcal {D} _ {0}} \left\| \nabla_ {\boldsymbol {\theta}} \ell (\boldsymbol {\theta}; Z _ {0, i} + \mathbf {A} \boldsymbol {\theta}) + \mathbf {A} ^ {\top} \nabla_ {Z} \ell (\boldsymbol {\theta}; Z _ {0, i} + \mathbf {A} \boldsymbol {\theta}) - \nabla_ {\boldsymbol {\theta}} \operatorname * {P R} (\boldsymbol {\theta}) \right\| _ {2} ^ {2} \leq \sigma^ {2}.
$$
Assumption 1 is standard in the literature of performative prediction. Assumptions 2 and 3 are widely used in the analysis of constrained optimization problems [Tan et al., 2018; Yan et al., 2019; Cao and Basar, 2020], even with perfect knowledge of objectives. Assumption 4 bounds the variance of the stochastic gradient of  $\mathrm{PR}(\pmb{\theta})$ . Additionally, to ensure a sufficient exploration of the parameter space, we make the following assumption on the injected noises  $\{\mathbf{u}_t\}_{t=1}^T$ .
Assumption 5 (Injected Noise). The injected noises  $\{\mathbf{u}_t\}_{t=1}^T$  are independent and identically distributed. Moreover, there exist positive constants  $\kappa_1, \kappa_2, \kappa_3$  such that for any  $t \in [T]$ , the random noise  $\mathbf{u}_t$  satisfies
$$
\mathbf {0} \prec \kappa_ {1} \cdot \mathbf {I} \preceq \mathbb {E} \left[ \mathbf {u} _ {t} \mathbf {u} _ {t} ^ {\top} \right], \quad \mathbb {E} \left\| \mathbf {u} _ {t} \right\| _ {2} ^ {2} \leq \kappa_ {2}, \quad a n d \quad \mathbb {E} \left[ \left\| \mathbf {u} _ {t} \right\| _ {2} ^ {2} \mathbf {u} _ {t} \mathbf {u} _ {t} ^ {\top} \right] \preceq \kappa_ {3} \mathbb {E} \left[ \mathbf {u} _ {t} \mathbf {u} _ {t} ^ {\top} \right].
$$
Consider a Gaussian noise that  $\mathbf{u}_t\sim \mathcal{N}(0,\mathbf{I}),\forall t\in [T]$ , we have  $\kappa_{1} = 1$ ,  $\kappa_{2} = d$ , and  $\kappa_{3} = 3d$
With the above assumptions, we provide some supporting lemmas below. First, we show  $\varepsilon$ -sensitivity of the location family given in (7).
Lemma 1 (ε-Sensitivity of  $\mathcal{D}(\pmb{\theta})$ ). Define  $\sigma_{\max}(\mathbf{A}) \coloneqq \max_{\|\pmb{\theta}\|_2 = 1} \|\mathbf{A}\pmb{\theta}\|_2$ . The location family given in (7) is ε-sensitive with parameter  $\varepsilon \leq \sigma_{\max}(\mathbf{A})$ . That is, for any  $\pmb{\theta}, \pmb{\theta}' \in \Theta$ , we have  $\mathcal{W}_1(\mathcal{D}(\pmb{\theta}), \mathcal{D}(\pmb{\theta}')') \leq \varepsilon \|\pmb{\theta} - \pmb{\theta}'\|_2$ , where  $\mathcal{W}_1(\mathcal{D}, \mathcal{D}')$  denotes the Wasserstein-1 distance.
See § A of the supplementary file for the proof. Building upon Lemma 1, we have the following Lemma 2 about the performative risk  $\mathrm{PR}(\theta)$ .
Lemma 2 (Lipschitz Continuity and Convexity of PR(θ)). Consider the location family given in (7). With Assumption 1 and Lemma 1, we have that: 1) the performative risk PR(θ) is L-Lipschitz continuous for  $L \leq L_{\theta} + L_{Z}\sigma_{\max}(\mathbf{A})$ ; 2) the performative risk PR(θ) is  $\gamma$ -strongly convex for
$$
\gamma \geq \max \left\{\gamma_ {\pmb {\theta}} - \beta^ {2} / \gamma_ {Z}, \gamma_ {\pmb {\theta}} - 2 \varepsilon \beta + \gamma_ {Z} \sigma_ {\mathrm {m i n}} ^ {2} (\mathbf {A}) \right\},
$$
where  $\sigma_{\mathrm{min}}(\mathbf{A})\coloneqq \min_{\| \pmb {\theta}\| _2 = 1}\| \mathbf{A}\pmb {\theta}\| _2$
See § B of the supplementary file for the proof. Based on the Lipschitz continuity and convexity of  $\mathrm{PR}(\pmb{\theta})$ , we provide the convergence result of the robust primal-dual framework below.
Lemma 3 (Convergence Result of Robust Primal-Dual Framework). Set  $\eta = \frac{1}{\sqrt{T}}$ . Then, there exists a constant  $\delta \in \left[\frac{1 - \sqrt{1 - 32\eta^2L_{\mathbf{g}}^2}}{4\eta^2},\frac{1 + \sqrt{1 - 32\eta^2L_{\mathbf{g}}^2}}{4\eta^2}\right]$  such that under Assumptions 1-3, for  $T \geq 32L_{\mathbf{g}}^2$ , the regret satisfies:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(\mathbb {E} [ \mathrm {P R} (\pmb {\theta} _ {t}) ] - \mathrm {P R} (\pmb {\theta} _ {\mathrm {P O}})\right) \leq \frac {\gamma \sqrt {T}}{\gamma - a} \left(2 R ^ {2} + C ^ {2} + 2 L ^ {2}\right) \\ + \frac {\gamma}{\gamma - a} \left(\frac {1}{2 a} + \frac {1}{\sqrt {T}}\right) \sum_ {t = 1} ^ {T} \mathbb {E} \left\| \nabla_ {\pmb {\theta}} \widehat {\mathrm {P R}} _ {t} (\pmb {\theta} _ {t}) - \nabla_ {\pmb {\theta}} \mathrm {P R} (\pmb {\theta} _ {t}) \right\| _ {2} ^ {2}, \\ \end{array}
$$
where  $a \in (0, \gamma)$  is a constant. Further, for any  $i \in [m]$ , the constraint violation satisfies:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} g _ {i} (\boldsymbol {\theta} _ {t}) \right] \leq \sqrt {1 + \delta} (2 R + \sqrt {2} C + 2 L) \sqrt {T} \\ + \sqrt {1 + \delta} \left(\frac {T ^ {\frac {1}{4}}}{\sqrt {a}} + \sqrt {2}\right) \left(\sum_ {t = 1} ^ {T} \mathbb {E} \left\| \nabla_ {\pmb {\theta}} \widehat {\mathrm {P R}} _ {t} (\pmb {\theta} _ {t}) - \nabla_ {\pmb {\theta}} \mathrm {P R} (\pmb {\theta} _ {t}) \right\| _ {2} ^ {2}\right) ^ {\frac {1}{2}}. \\ \end{array}
$$
Remark 2. Lemma 3 reveals the impact of gradient approximation error on the convergence performance of the robust primal-dual framework. By Lemma 3, if the accumulated gradient approximation error is less than  $\mathcal{O}(\sqrt{T})$ , both the regret and the constraint violations are bounded by  $\mathcal{O}(\sqrt{T})$ . Although stochastic primal-dual methods for constrained problems without performativity also use approximated (stochastic) gradients, they generally require unbiased gradient approximation [Tan et al., 2018; Yan et al., 2019; Cao and Basar, 2022]. This requirement, however, is difficult to satisfy in performative prediction since the unknown performative effect of decisions changes the data distribution. In contrast, the robust primal-dual framework does not restrict the approximate gradients to be unbiased and hence offers more flexibility to the design of gradient approximation.
Proof of Lemma 3 is provided in § C of the supplementary file. In the next lemma, we bound the gradient approximation error of the adaptive primal-dual algorithm.
Lemma 4 (Gradient Approximation Error). Set  $\zeta_t = \frac{2}{\kappa_1(t - 1) + 2\kappa_3}$ ,  $\forall t \in [T]$ . Then, under Assumptions 4 and 5, the accumulated gradient approximation error is upper bounded by:
$$
\sum_ {t = 1} ^ {T} \mathbb {E} \left\| \nabla_ {\boldsymbol {\theta}} \widehat {\mathrm {P R}} _ {t} (\boldsymbol {\theta} _ {t}) - \nabla_ {\boldsymbol {\theta}} \mathrm {P R} (\boldsymbol {\theta} _ {t}) \right\| _ {2} ^ {2} \leq \frac {2 T \sigma^ {2}}{n} + \frac {4}{n} \left(2 L _ {Z} ^ {2} + \beta^ {2} R ^ {2} \left(1 + 2 \sigma_ {\max } (\mathbf {A})\right)\right) \overline {{\alpha}} \ln (T),
$$
where  $\overline{\alpha} := \max \left\{\frac{2\kappa_3}{\kappa_1}\|\widehat{\mathbf{A}}_0 - \mathbf{A}\|_{\mathrm{F}}^2, \frac{8\kappa_2\operatorname{tr}(\boldsymbol{\Sigma})}{\kappa_1^2}\right\}$ . In Algorithm 1, we set  $\widehat{\mathbf{A}}_0 = \mathbf{0}$ , and thus we have  $\overline{\alpha} = \max \left\{\frac{2\kappa_3}{\kappa_1}\|\mathbf{A}\|_{\mathrm{F}}^2, \frac{8\kappa_2\operatorname{tr}(\boldsymbol{\Sigma})}{\kappa_1^2}\right\}$ .
Remark 3. Lemma 4 demonstrates that the gradient approximation error of the adaptive primal-dual algorithm is upper bounded by  $\mathcal{O}(T / n + \ln (T))$ . If we set the number of initial samples  $n\geq \sqrt{T}$  we have  $\sum_{t = 1}^{T}\mathbb{E}\left\| \nabla_{\pmb{\theta}}\widehat{\mathrm{PR}} (\pmb{\theta}_t) - \nabla_{\pmb{\theta}}\widehat{\mathrm{PR}}_t(\pmb {\theta}_t)\right\| _2^2\leq \mathcal{O}(\sqrt{T})$ . According to Lemma 3, this suffices to make the regret and constraint violation bounds to be  $\mathcal{O}(\sqrt{T})$
Proof of Lemma 4 is presented in § D of the supplementary file. Combining Lemma 3 and Lemma 4 yields the regret and constraint violations of Algorithm 1, which is elaborated in Theorem 1 below.
Theorem 1. Set  $\eta = \frac{1}{\sqrt{T}}$  and  $\zeta_t = \frac{2}{\kappa_1(t - 1) + 2\kappa_3}$ ,  $\forall t \in [T]$ . Then, there exists a constant  $\delta \in \left[\frac{1 - \sqrt{1 - 32\eta^2L_{\mathbf{g}}^2}}{4\eta^2}, \frac{1 + \sqrt{1 - 32\eta^2L_{\mathbf{g}}^2}}{4\eta^2}\right]$  such that under Assumptions 1-5, for  $T \geq 32L_{\mathbf{g}}^2$ , the regret of Algorithm 1 is upper bounded by:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(\mathbb {E} [ \mathrm {P R} (\pmb {\theta} _ {t}) ] - \mathrm {P R} (\pmb {\theta} _ {\mathrm {P O}})\right) \leq \frac {\gamma \sqrt {T}}{\gamma - a} \left(2 R ^ {2} + C ^ {2} + 2 L ^ {2}\right) + \frac {\gamma \sigma^ {2}}{\gamma - a} \left(\frac {1}{a} + \frac {2}{\sqrt {T}}\right) \frac {T}{n} \\ + \frac {\gamma \bar {\alpha} \ln (T)}{n (\gamma - a)} \left(\frac {2}{a} + \frac {4}{\sqrt {T}}\right) \left(2 L _ {Z} ^ {2} + \beta^ {2} R ^ {2} \left(1 + 2 \sigma_ {\max } (\mathbf {A})\right)\right), \\ \end{array}
$$
Further, for any  $i \in [m]$ , the constraint violation is upper bounded by:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} g _ {i} (\pmb {\theta} _ {t}) \right] \leq \sqrt {1 + \delta} \left[ (2 R + \sqrt {2} C + 2 L) \sqrt {T} + \left(\frac {\sqrt {2}}{\sqrt {a}} + \frac {2}{T ^ {\frac {1}{4}}}\right) \frac {\sigma T ^ {\frac {3}{4}}}{\sqrt {n}} \right] \\ + \frac {2 \sqrt {\overline {{\alpha}} (1 + \delta) \ln (T)}}{\sqrt {n}} \left(\frac {T ^ {\frac {1}{4}}}{\sqrt {a}} + \sqrt {2}\right) \left(2 L _ {Z} ^ {2} + \beta^ {2} R ^ {2} \left(1 + 2 \sigma_ {\max} (\mathbf {A})\right)\right) ^ {\frac {1}{2}}. \\ \end{array}
$$
Remark 4. Theorem 1 demonstrates that Algorithm 1 achieves  $\mathcal{O}(\sqrt{T} + T / n)$  regret and  $\mathcal{O}(\sqrt{T} + T^{\frac{3}{4}} / \sqrt{n})$  constraint violations. By setting  $n = \sqrt{T}$ , we have  $T / n = \sqrt{T}$  and  $T^{\frac{3}{4}} / \sqrt{n} = \sqrt{T}$ , and hence both the regret and constraint violations are upper bounded by  $\mathcal{O}(\sqrt{T})$ . This indicates that Algorithm 1 attains the same order of performance as the stochastic primal-dual algorithm without performativity [Tan et al., 2018; Yan et al., 2019].
Remark 5. Throughout the time horizon  $T$ , Algorithm 1 requires a total of  $\sqrt{T} + 2T$  samples. Among them,  $\sqrt{T}$  samples are dedicated to approximate the expectation over the base component  $Z_0$ . Furthermore, each iteration requires an additional 2 samples to construct the online least-square objective, accumulating the remaining  $2T$  samples.
This section verifies the efficacy of our algorithm and theoretical results by conducting numerical experiments on two examples: multi-task linear regression and multi-asset portfolio.
We first consider a multi-task linear regression problem in an undirected graph  $\mathcal{G} \coloneqq (\mathcal{V}, \mathcal{E})$ , where  $\mathcal{V}$  represents the node set and  $\mathcal{E}$  represents the edge set. Each node  $i$  handles a linear regression task  $\mathrm{PR}_i(\pmb{\theta}_i) \coloneqq \mathbb{E}_{(\mathbf{x}_i, y_i) \sim \mathcal{D}_i(\pmb{\theta}_i)} \ell_i(\pmb{\theta}_i; (\mathbf{x}_i, y_i))$ , where  $\pmb{\theta}_i$  is the parameter vector and  $(\mathbf{x}_i, y_i)$  is a feature-label pair. The loss function of each task is  $\ell_i(\pmb{\theta}_i; (\mathbf{x}_i, y_i)) = \frac{1}{2} (y_i - \pmb{\theta}_i^\top \mathbf{x}_i)^2, \forall i \in \mathcal{V}$ . The parameters of each connected node pair are subject to a proximity constraint  $\| \pmb{\theta}_i - \pmb{\theta}_j \|_2^2 \leq b_{ij}^2$ ,  $\forall (i, j) \in \mathcal{E}$ . The entire network aims to solve the following problem:
$$
\min _ {\pmb {\theta} _ {i}, \forall i} \frac {1}{2} \sum_ {i \in \mathcal {V}} \mathbb {E} _ {(\mathbf {x} _ {i}, y _ {i}) \sim \mathcal {D} _ {i} (\pmb {\theta} _ {i})} (y _ {i} - \pmb {\theta} _ {i} ^ {\top} \mathbf {x} _ {i}) ^ {2} \quad \mathrm {s . t .} \quad \frac {1}{2} \| \pmb {\theta} _ {i} - \pmb {\theta} _ {j} \| _ {2} ^ {2} \leq b _ {i j} ^ {2}, \forall (i, j) \in \mathcal {E}.
$$
Figure 1: Multi-task linear regression. (a)  $\frac{\mathrm{Reg}(t)}{t\cdot\mathrm{Reg}(1)}$ ; (b)  $\frac{\mathrm{Vio}_i(t)}{t\cdot|\mathrm{Vio}_i(1)|}$ ; (c)  $\|\pmb{\theta}_t - \pmb{\theta}_{\mathrm{PO}}\|_2^2$ ; (d)  $\|\widehat{\mathbf{A}}_t - \mathbf{A}\|_{\mathrm{F}}^2$ .
Figure 2: Multi-asset portfolio. (a)  $\frac{\mathrm{Reg}(t)}{t\cdot\mathrm{Reg}(1)}$ ; (b)  $\frac{\mathrm{Vio}_i(t)}{t\cdot|\mathrm{Vio}_i(1)|}$ ; (c)  $\|\pmb{\theta}_t - \pmb{\theta}_{\mathrm{PO}}\|_2^2$ ; (d)  $\|\widehat{\mathbf{A}}_t - \mathbf{A}\|_{\mathrm{F}}^2$ .
The second example considers the multi-asset portfolio described in Example 1. The simulation details are provided in  $\S F$  of the supplementary file.
We compare the proposed adaptive primal-dual algorithm (abbreviated as APDA) with two approaches. The first approach is "PD-PS", which stands for the primal-dual (PD) algorithm used to find the performative stable (PS) points. The algorithm PD-PS is similar to APDA, but it uses only the first term in Eq. (8) as the approximate gradient. The second approach is "baseline", which runs the same procedures as APDA with perfect knowledge of  $\mathbf{A}$ , i.e., the performative effect is known. We consider four performance metrics: (a) relative time-average regret  $\frac{\mathrm{Reg}(t)}{t\cdot\mathrm{Reg}(1)}$ , (b) relative time-average constraint violation  $\frac{\mathrm{Vio}_i(t)}{t\cdot|\mathrm{Vio}_i(1)|}$ , (c) decision deviation  $\| \theta_t - \theta_{\mathrm{PO}}\|_2^2$ , and (d) parameter estimation error  $\| \widehat{\mathbf{A}}_t - \mathbf{A}\|_{\mathrm{F}}^2$ .
Fig. 1 and Fig. 2 show the numerical results of the multi-task linear regression and the multi-asset portfolio, respectively. In both figures, we consider two settings for the sensitivity parameter of  $\mathcal{D}(\theta)$ , namely  $\varepsilon = 1$  and  $\varepsilon = 10$ . The results of these two figures are qualitatively analogous. First, we observe that APDA outperforms PD-PS significantly that both the relative time-average regret and the decision derivation of the former achieve an accuracy around or up to  $10^{-3}$  for the setting of  $T = 10^{6}$ , while these of the latter have worse performance for  $\varepsilon = 1$  and converge to constants for  $\varepsilon = 10$ . The relative time-average constraint of all cases converges to zero or negative numbers. This corroborates the sublinearity of the regret and the constraint violations of APDA, as shown in Theorem 1. More importantly, this result implies that the larger the sensitivity parameter  $\varepsilon$ , the stronger the performative power is and, consequently, the worse PD-PS performs. In contrast, by tracking the performative gradient, APDA adapts to the unknown performative effect and performs well constantly. Moreover, both subfigures (d) show that the error of parameter estimates decreases sublinearly with iterations, validating the effectiveness of the online parameter estimation. Last but not least, the performance of APDA is close to the performance of the baseline, which manifests the effectiveness of our proposed APDA algorithm.
This paper has studied the performative prediction problem under inequality constraints, where the agnostic performative effect of decisions changes future data distributions. To find the performative optima for the problem, we have developed a robust primal-dual framework that admits inexact gradients up to an accuracy of  $\mathcal{O}(\sqrt{T})$ , yet delivers the same  $\mathcal{O}(\sqrt{T})$  regret and constraint violations as the stochastic primal-dual algorithm without performativity. Then, based on this framework, we have proposed an adaptive primal-dual algorithm for location family with effective gradient approximation method that meets the desired accuracy using only  $\sqrt{T} + 2T$  samples. Numerical experiments have validated the effectiveness of our algorithm and theoretical results.
The work was supported by the National Natural Science Foundation of China Grant 62203373.