Title: Two-timescale Derivative Free Optimization for Performative Prediction with Markovian Data
Venue: NeurIPS 2023
Accepted: no
Keywords: Performative Prediction

Derivative Free Optimization

Markovian data
Abstract: This paper studies the performative prediction problem where the learner aims to minimize the expected loss with a decision-dependent data distribution. Such setting is motivated when outcomes can be affected by the prediction model. We consider a state-dependent setting where the data distribution evolves according to an underlying controlled Markov chain. We focus on derivative free optimization (DFO) where the learner is given access to a loss function evaluation oracle with the above Markov chain data. We propose a two-timescale DFO($\lambda$) algorithm that features (i) a novel forgetting factor $\lambda$ to utilize every observed sample as opposed to the common sample burn-in approach, and  (ii) a two-timescale diminishing step size to balance the rates of DFO updates and bias reduction. Under a general non-convex optimization setting, we show that DFO($\lambda$) requires at most ${\cal O}(d^2/\epsilon^3)$ samples (up to a log factor)  to attain a near-stationary solution with expected squared gradient norm less than $\epsilon$. Numerical experiments verify our analysis.

1

Abstract

This paper studies the performative prediction problem where a learner aims to minimize the expected loss with a decision-dependent data distribution. Such setting is motivated when outcomes can be affected by the prediction model, e.g., in strategic classification. We consider a state-dependent setting where the data distribution evolves according to an underlying controlled Markov chain. We focus on stochastic derivative free optimization (DFO) where the learner is given access to a loss function evaluation oracle with the above Markovian data. We propose a two-timescale DFO( λ ) algorithm that features (i) a sample accumulation mechanism that utilizes every observed sample to estimate the overall gradient of performative risk, and (ii) a two-timescale diminishing step size that balances the rates of DFO updates and bias reduction. Under a general non-convex optimization setting, we show that DFO( λ ) requires O (1 /ϵ 3 ) samples (up to a log factor) to attain a near-stationary solution with expected squared gradient norm less than ϵ > 0 . Numerical experiments verify our analysis.

1 Introduction

Consider the following stochastic optimization problem with decision-dependent data:

Notice that the decision variable θ appears in both the loss function ℓ ( θ ; Z ) and the data distribution Π θ Z . The overall loss function L ( θ ) is known as the performative risk which captures the distributional shift due to changes in the deployed model. This setting is motivated by the recent studies on performative prediction (Perdomo et al., 2020), which considers outcomes that are supported by the deployed model θ under training. For example, this models strategic classification (Hardt et al., 2016; Dong et al., 2018) in economical and financial practices such as with the training of loan classifier for customers who may react to the deployed model θ to maximize their gains; or in price promotion mechanism (Zhang et al., 2018) where customers react to prices with the aim of gaining a lower price; or in ride sharing business (Narang et al., 2022) with customers who adjust their demand according to prices set by the platform.

The objective function L ( θ ) is non-convex in general due to the effects of θ on both the loss function and distribution. Numerous efforts have been focused on characterizing and finding the so-called performative stable solution which is a fixed point to the repeated risk minimization (RRM) process (Perdomo et al., 2020; Mendler-D¨ unner et al., 2020; Brown et al., 2022; Li & Wai, 2022; Roy et al., 2022; Drusvyatskiy & Xiao, 2022). While RRM might be a natural algorithm for scenarios when the learner is agnostic to the performative effects in the dynamic data distribution, the obtained solution maybe far from being optimal or stationary to (1).

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

On the other hand, recent works have studied performative optimal solutions that minimizes (1). This is challenging due to the non-convexity of L ( θ ) and more importantly, the absence of knowledge of Π θ . In fact, evaluating ∇L ( θ ) or its stochastic gradient estimate would require learning the distribution Π θ a-priori (Izzo et al., 2021). To design a tractable procedure, prior works have assumed structures for (1) such as approximating Π θ by Gaussian mixture (Izzo et al., 2021), Π θ depends linearly on θ (Narang et al., 2022), etc., combined with a two-phase algorithm that separately learns Π θ and optimizes θ . Other works have assumed a mixture dominance structure (Miller et al., 2021) on the combined effect of Π θ and ℓ ( · ) on L ( θ ) , which in turn implies that L ( θ ) is convex. Based on this assumption, a derivative free optimization (DFO) algorithm was analyzed in Ray et al. (2022).

This paper focuses on approximating the performative optimal solution without relying on additional condition on the distribution Π θ and/or using a twophase algorithm. We concentrate on stochastic DFO algorithms (Ghadimi & Lan, 2013) which do not involve first order information (i.e., gradient) about L ( θ ) . As an advantage, these algorithms avoid the need for estimating Π θ . Instead, the learner is given access to the loss function evaluation oracle ℓ ( θ ; Z ) and receive data samples from a controlled Markov chain. Note that the latter models the stateful and strategic agent setting considered in (Ray et al., 2022; Roy et al., 2022; Li & Wai, 2022; Brown et al., 2022).

Such setting is motivated when the actual data distribution adapts slowly to the decision model, which will be announced by the learner during the (stochastic) optimization process.

The proposed DFO ( λ ) algorithm features (i) a two-timescale step sizes design to control the biasvariance tradeoff in the derivative-free gradient estimates, and (ii) a sample accumulation mechanism with forgetting factor λ that aggregates every observed samples to control the amount of error in gradient estimates. In addition to the new algorithm design, our main findings are summarized below:

Under the Markovian data setting, we show in Theorem 3.1 that the DFO ( λ ) algorithm finds a nearstationary solution ¯ θ with E [ ∥∇L ( ¯ θ ) ∥ 2 ] ≤ ϵ using O ( d 2 ϵ 3 log 1 /ϵ ) samples/iterations. Compared to prior works, our analysis does not require structural assumption on the distribution Π θ or convexity condition on the performative risk (Izzo et al., 2021; Miller et al., 2021; Ray et al., 2022).

Our analysis demonstrates the trade-off induced by the forgetting factor λ in the DFO ( λ ) algorithm. We identify the desiderata for the optimal value(s) of λ . We show that increasing λ allows to reduce the number of samples requited by the algorithm if the performative risk gradient has a small Lipschitz constant.

For the rest of this paper, §2 describes the problem setup and the DFO ( λ ) algorithm, §3 presents the main results, §4 outlines the proofs. Finally, we provide numerical results to verify our findings in §5.

Finally, as displayed in Table 1, we remark that stochastic DFO under decision dependent (and Markovian) samples has a convergence rate of O (1 /ϵ 3 ) towards an ϵ -stationary point, which is worse than the decision independent setting that has O (1 /ϵ 2 ) in Ghadimi & Lan (2013). We believe that this is a fundamental limit for DFO-type algorithms when tackling problems with decision-dependent sample due to the challenges in designing a low variance gradient estimator; see §4.1.

Related Works . The idea of DFO dates back to Nemirovski˘ ı (1983), and has been extensively studied thereafter Flaxman et al. (2005); Agarwal et al. (2010); Nesterov & Spokoiny (2017); Ghadimi & Lan (2013). Results on matching lower bound were established in (Jamieson et al., 2012). While a similar DFO framework is adopted in the current paper for performative prediction, our algorithm is limited to using a special design in the gradient estimator to avoid introducing unwanted biases.

There are only a few works considering the Markovian data setting in performative prediction. Brown 82 et al. (2022) is the first paper to study the dynamic settings, where the response of agents to learner's 83 deployed classifier is modeled as a function of classifier and the current distribution of the population; 84 also see (Izzo et al., 2022). On the other hand, Li & Wai (2022); Roy et al. (2022) model the 85 unforgetful nature and the reliance on past experiences of single/batch agent(s) via controlled Markov 86 Chain. Lastly, Ray et al. (2022) investigated the state-dependent framework where agents' response 87 may be driven to best response at a geometric rate. 88

Algorithm 1 DFO ( λ ) Algorithm

Notations : Let R d be the d -dimensional Euclidean space equipped with inner product ⟨· , ·⟩ and 89 induced norm ∥ x ∥ = √ ⟨ x, x ⟩ . Let S be a (measurable) sample space, and µ , ν are two probability 90 measures defined on S . Then, we use δ TV ( µ, ν ) := sup A ⊂S µ ( A ) -ν ( A ) to denote the total variation 91 distance between µ and ν . Denote T θ ( · , · ) as the state-dependent Markov kernel and its stationary 92 distribution is Π θ ( · ) . Let B d and S d -1 be the unit ball and its boundary (i.e., a unit sphere) centered 93 around the origin in d -dimensional Euclidean space, respectively, and correspondingly, the ball and 94 sphere of radius r > 0 are r B d and r S d -1 . 95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

2 Problem Setup and Algorithm Design

In this section, we develop the DFO ( λ ) algorithm for tackling (1) and describe the problem setup. Assume that L ( θ ) is differentiable, we focus on finding an ϵ -stationary solution, θ , which satisfies

With the goal of reaching (2), there are two key challenges in our stochastic algorithm design: (i) to estimate the gradient ∇L ( θ ) , and (ii) to handle the stateful setting where one cannot draw samples directly from the distribution Π θ . We shall discuss how the proposed DFO ( λ ) algorithm, which is summarized in Algorithm 1, tackles the above issues through utilizing two ingredients: (a) two-timescales step sizes, and (b) sample accumulation with the forgetting factor λ ∈ [0 , 1) .

Estimating ∇L ( θ ) via Two-timescales DFO. First notice that the gradient of L ( · ) can be derived as

As a result, constructing the stochastic estimates of ∇L ( θ ) typically requires knowledge of Π θ ( · ) which may not be known a-priori unless a separate estimation procedure is applied; see e.g., (Izzo et al., 2021). To avoid the need for direct evaluations of ∇ θ log Π θ ( Z ) , we consider an alternative design via zero-th order optimization (Ghadimi & Lan, 2013). The intuition comes from observing that with δ → 0 + , L ( θ + δ u ) -L ( θ ) is an approximate of the directional derivative of L along u . This suggests that an estimate for ∇L ( θ ) can be constructed using the objective function values of ℓ ( θ ; Z ) only.

Inspired by the above, we aim to construct a gradient estimate by querying ℓ ( · ) at randomly perturbed points. Formally, given the current iterate θ ∈ R d and a query radius δ > 0 , we sample a vector u ∈ R d uniformly from S d -1 . The zero-th order gradient estimator for L ( θ ) is then defined as

In fact, as u is zero-mean, g δ ( θ ; u , Z ) is an unbiased estimator for ∇L δ ( θ ) . Here, L δ ( θ ) is a smooth approximation of L ( θ ) (Flaxman et al., 2005; Nesterov & Spokoiny, 2017) defined as

Furthermore, it is known that under mild condition [cf. Assumption 3.1 to be discussed later], ∥∇L δ ( θ ) -∇L ( θ ) ∥ = O ( δ ) and thus (4) is an O ( δ ) -biased estimate for ∇L ( θ ) .

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

We remark that the gradient estimator in (4) differs from the one used in classical works on DFO such as (Ghadimi & Lan, 2013). The latter takes the form of d δ ( ℓ ( ˇ θ ; Z ) -ℓ ( θ ; Z )) u . Under the setting of standard stochastic optimization where the sample Z is drawn independently of u and Lipschitz continuous ℓ ( · ; Z ) , the said estimator in (Ghadimi & Lan, 2013) is shown to have constant variance while it remains O ( δ ) -biased. Such properties cannot be transferred to (4) since Z is drawn from a distribution dependent on u via ˇ θ = θ + δ u . In this case, the two-point gradient estimator would become biased; see §4.1.

However, we note that the variance of (4) would increase as O (1 /δ 2 ) when δ → 0 , thus the parameter δ yields a bias-variance trade off in the estimator design. To remedy for the increase of variance, the DFO ( λ ) algorithm incorporates a two-timescale step size design for generating gradient estimates ( δ k ) and updating models ( η k ), respectively. Our design principle is such that the models are updated at a slower timescale to adapt to the gradient estimator with O (1 /δ 2 ) variance. Particularly, we will set η k +1 /δ k +1 → 0 to handle the bias-variance trade off, e.g., by setting α > β in line 4 of Algorithm 1.

Markovian Data and Sample Accumulation. We consider a setting where the sample/data distribution observed by the DFO ( λ ) algorithm evolves according to a controlled Markov chain (MC) . Notice that this describes a stateful agent(s) scenario such that the deployed models ( θ ) would require time to manifest their influence on the samples obtained; see (Li & Wai, 2022; Roy et al., 2022; Brown et al., 2022; Ray et al., 2022; Izzo et al., 2022).

To describe the setting formally, we denote T θ : Z ×Z → R + as a Markov kernel controlled by a deployed model θ . For a given θ , the kernel has a unique stationary distribution Π θ ( · ) . Under this setting, suppose that the previous state/sample is Z , the next sample follows the distribution Z ′ ∼ T θ ( Z, · ) which is not necessarily the same as Π θ ( · ) . As a consequence, the gradient estimator (4) is not an unbiased estimator of ∇L δ ( θ ) since Z ∼ Π ˇ θ ( · ) cannot be conveniently accessed.

A common strategy in settling the above issue is to allow a burn-in phase in the algorithm as in (Ray et al., 2022); also commonly found in MCMC methods (Robert et al., 1999). Using the fact that T θ admits the stationary distribution Π θ , if one can wait a sufficiently long time before applying the current sample, i.e., consider initializing with the previous sample Z (0) = Z , the procedure

would yield a sample Z + = Z ( τ ) that admits a distribution close to Π θ provided that τ ≫ 1 is sufficiently large compared to the mixing time of T θ .

Intuitively, the procedure (6) may be inefficient as a number of samples Z (1) , Z (2) , . . . , Z ( τ -1) will be completely ignored at the end of each iteration. As a remedy, the DFO ( λ ) algorithm incorporates a sample accumulation mechanism which gathers the gradient estimates generated from possibly non-stationary samples via a forgetting factor of λ ∈ [0 , 1) . Following (4), ∇L ( θ ) is estimated by

At a high level, the mechanism works by assigning large weights to samples that are close to the end of an epoch (which are less biased). Moreover, θ ( m ) is simultaneously updated within the epoch to obtain an online algorithm that gradually improves the objective value of (1). Note that with λ = 0 , the DFO (0) algorithm reduces into one that utilizes burn-in (6). We remark that from the implementation perspective for performative prediction, Algorithm 1 corresponds to a greedy deployment scheme (Perdomo et al., 2020) as the latest model θ ( m ) k + δ k u k is deployed at every sampling step. Line 6-10 of Algorithm 1 details the above procedure.

Lastly, we note that recent works have analyzed stochastic algorithms that rely on a single trajectory of samples taken from a Markov Chain, e.g., (Sun et al., 2018; Karimi et al., 2019; Doan, 2022), that are based on stochastic gradient. Sun & Li (2019) considered a DFO algorithm for general optimization problems but the MC studied is not controlled by θ .

3 Main Results

This section studies the convergence of the DFO ( λ ) algorithm and demonstrates that the latter finds an ϵ -stationary solution [cf. (2)] to (1). We first state the assumptions required for our analysis:

Assumption 3.1. (Smoothness) L ( θ ) is differentiable, and there exists a constant L > 0 such that

Assumption 3.2. (Bounded Loss) There exists a constant G > 0 such that 167

The conditions above state that the gradient of the performative risk is Lipschitz continuous and the state-dependent distribution vary smoothly w.r.t. θ . Note that Assumption 3.1 is found in recent works such as (Izzo et al., 2021; Ray et al., 2022), and Assumption 3.2 can be found in (Izzo et al., 2021). Assumption 3.3 is slightly strengthened from the Wasserstein-1 distance bound in (Perdomo et al., 2020), and it gives better control for distribution shift in our Markovian data setting.

Next, we consider the assumptions about the controlled Markov chain induced by T θ :

Assumption 3.4. (Geometric Mixing) Let { Z k } k ≥ 0 denote a Markov Chain on the state space Z with transition kernel T θ and stationary measure Π θ . There exist constants ρ ∈ [0 , 1) , M ≥ 0 , such that for any k ≥ 0 , z ∈ Z ,

Assumption 3.5. (Smoothness of Markov Kernel) There exists a constant L 2 ≥ 0 such that

Assumption 3.4 is a standard condition on the mixing time of the Markov chain induced by T θ ; Assumption 3.5 imposes a smoothness condition on the Markov transition kernel T θ with respect to θ . For instance, the geometric dynamically environment in Ray et al. (2022) constitutes a special case which satisfies the above conditions.

Unlike (Ray et al., 2022; Izzo et al., 2021; Miller et al., 2021), we do not impose any additional assumption (such as mixture dominance) other than Assumption 3.3 on Π θ . As a result, (1) remains an 'unstructured' non-convex optimization problem. Our main theoretical result on the convergence of the DFO ( λ ) algorithm towards a near-stationary solution of (1) is summarized as:

Theorem 3.1. Suppose Assumptions 3.1-3.5 hold, step size sequence { η k } k ≥ 1 , and query radius sequence { δ k } k ≥ 1 satisfy the following conditions,

Then, there exists constants t 0 , c 5 , c 6 , c 7 , such that for any T ≥ t 0 , the iterates { θ k } k ≥ 0 generated by DFO ( λ ) satisfy the following inequality,

We have defined the following quantities and constants: 188

with α = 2 3 , β = 1 6 . Observe the following corollary on the iteration complexity of DFO ( λ ) algorithm:

189

190

191

Corollary 3.1. ( ϵ -stationarity) Suppose that the Assumptions of Theorem 3.1 hold. Fix any ϵ > 0 , the condition min 0 ≤ k ≤ T -1 E ∥∇L ( θ k ) ∥ 2 ≤ ϵ holds whenever

In the corollary above, the lower bound on T is expressed in terms of the number of epochs that 192 Algorithm 1 needs to achieve the target accuracy. Consequently, the total number of samples required 193 (i.e., the number of inner iterations taken in Line 6-9 of Algorithm 1 across all epochs) is: 194

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

for any t ≥ 1 , where 214

We remark that due to the decision-dependent properties of the samples, the DFO ( λ ) algorithm exhibits a worse sampling complexity (12) than prior works in stochastic DFO algorithm, e.g., (Ghadimi & Lan, 2013) which shows a rate of O ( d/ϵ 2 ) on non-convex smooth objective functions. In particular, the adopted one-point gradient estimator in (4) admits a variance that can only be controlled by a time varying δ ; see the discussions in §4.1.

Achieving the desired convergence rate requires setting η k = Θ( k -2 / 3 ) , δ k = Θ( k -1 / 6 ) , i.e., yielding a two-timescale step sizes design with η k /δ k → 0 . Notice that the influence of forgetting factor λ are reflected in the constant factor of (9). Particularly, if c 5 > c 7 and c 5 ≥ c 6 , the optimal choice is λ = 1 -√ c 7 c 5 , otherwise the optimal choice is λ ∈ [0 , 1 -c 7 /c 6 ] . Informally, this indicates that when the performative risk is smoother (i.e. its gradient has a small Lipschitz constant), a large λ can speed up the convergence of the algorithm; otherwise a smaller λ is preferable.

4 Proof Outline of Main Results

This section outlines the key steps in proving Theorem 3.1. Notice that analyzing the DFO ( λ ) algorithm is challenging due to the two-timescales step sizes and Markov chain samples with time varying kernel. Our analysis departs significantly from prior works such as (Ray et al., 2022; Izzo et al., 2021; Brown et al., 2022; Li & Wai, 2022) to handle the challenges above.

Let F k = σ ( θ 0 , Z ( m ) s , u s , 0 ≤ s ≤ k, 0 ≤ m ≤ τ k ) be the filtration. Our first step is to exploit the smoothness of L ( θ ) to bound the squared norms of gradient. Observe that:

Lemma 4.1. (Decomposition) Under Assumption 3.1, it holds that

215

216

217

218

The lemma is achieved through the standard descent lemma implied by Assumption 3.1 and decomposing the upper bound on ||∇L ( θ k ) || 2 into respectful terms; see the proof in Appendix A. Among the terms on the right hand side of (13), we note that I 1 ( t ) , I 3 ( t ) and I 4 ( t ) arises directly from Assumption 3.1, while I 2 ( t ) comes from bounding the noise terms due to Markovian data.

219

220

221

222

223

224

225

226

227

We bound the four components in Lemma 4.1 as follows. For simplicity, we denote A ( t ) := 1 1+ t ∑ t k =0 E ∥∇L ( θ k ) ∥ 2 . Among the four terms, we highlight that the main challenge lies on obtaining a tight bound for I 2 ( t ) . Observe that

where ∆ k,m def = E F k -1 [ g ( m ) k -E Z ∼ Π ˇ θ k g k ( θ k ; u k , Z )] . There are two sources of bias in ∆ k,m : one is the noise induced by drifting of decision variable in every epoch, the other is the bias that depends on the mixing time of Markov kernel. To control these biases, we are inspired by the proof of (Wu et al., 2020, Theorem 4.7) to introduce a reference Markov chain ˜ Z ( ℓ ) k , ℓ = 0 , ..., τ k , whose decision variables remains fixed for a period of length τ k and is initialized with ˜ Z (0) k = Z (0) k :

and we recall that the actual chain in the algorithm evolves as

With the help of the reference chain, we decompose ∆ k,m into 228

229

230

231

232

233

234

235

236

237

238

We remark that A 1 reflects the drift of (16) from initial sample Z (0) k driven by varying ˇ θ ( m ) k , A 2 captures the statistical discrepancy between above two Markov chains (16) and (15) at same step m , and A 3 captures the drifting gap between ˇ θ k and ˇ θ ( m ) k . Applying Assumption 3.3, A 1 and A 2 can be upper bounded with the smoothness and geometric mixing property of Markov kernel. In addition, A 3 can be upper bounded using Lipschitz condition on (stationary) distribution map Π θ . Finally, the forgetting factor λ helps to control ∥ ˇ θ ( · ) k -ˇ θ k ∥ to be at the same order of a single update. Therefore, ∥ ∆ k,m ∥ can be controlled by an upper bound relying on λ, ρ, L .

The following lemma summarizes the above results as well as the bounds on the other terms:

Lemma 4.2. Under Assumption 3.2, 3.3, 3.4 and 3.5, with η t +1 = η 0 (1 + t ) -α , δ t +1 = δ 0 (1 + t ) -β and α ∈ (0 , 1) , β ∈ (0 , 1 2 ) . Suppose that 0 < 2 α -4 β < 1 and

Then, it holds that 239

where t 1 , t 2 are defined in (25) , (26) , and c 1 , c 2 , c 3 , c 4 are constants defined as follows: 240

̸

See Appendix B for the proof. We comment that the bound for I 4 ( t ) cannot be improved. As a 241 concrete example, consider the constant function ℓ ( θ ; z ) = c = 0 for all z ∈ Z , it can be shown that 242 ∥ g ( m ) k ∥ 2 = c 2 and consequently I 4 ( t ) = Ω( η k /δ 2 k ) = Ω( t 1 -( α -2 β ) ) , which matches (18). Finally, 243 plugging Lemma 4.2 into Lemma 4.1 gives: 244

245

Lemma 4.3. Under Assumption 3.1-3.5, with the step sizes η t +1 = η 0 (1 + t ) -α , δ t +1 = δ 0 (1 + 246 t ) -β , τ k ≥ 1 log 1 / max { ρ,λ } ( log(1 + k ) + max { log δ 0 d , 0 } ) , η 0 = d -2 / 3 , δ 0 = d 1 / 3 , α ∈ (0 , 1) , 247 β ∈ (0 , 1 2 ) . If 2 α -4 β < 1 , then there exists a constant t 0 such that the iterates { θ k } k ≥ 0 satisfies 248

Since A ( t ) ≥ 0 , the above is a quadratic inequality that implies the following bound:

Optimizing the step size exponents α, β in the above concludes the proof of Theorem 3.1. 249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

4.1 Discussions

We conclude by discussing two alternative zero-th order gradient estimators to (4), and argue that they do not improve over the sample complexity in the proposed DFO ( λ ) algorithm. We study:

where u ∼ Unif ( S d -1 ) . For ease of illustration, we assume that the samples Z, Z 1 , Z 2 are drawn directly from the stationary distributions Z ∼ Π θ + δu , Z 1 ∼ Π θ + δ u , Z 2 ∼ Π θ .

We recall from §2 that the estimator g 2pt -I is a finite difference approximation of the directional derivative of objective function along the randomized direction u 1 , as proposed in Nesterov & Spokoiny (2017); Ghadimi & Lan (2013). For non-convex stochastic optimization with decision independent sample distribution, i.e., Π θ ≡ ¯ Π for all θ , the DFO algorithm based on g 2pt -I is known to admit an optimal sample complexity of O (1 /ϵ 2 ) (Jamieson et al., 2012). Note that E u ∼ Unif ( S d -1 ) ,Z ∼ ¯ Π [ ℓ ( θ ; Z ) u ] = 0 . However, in the case of decision-dependent sample distribution as in (1), g 2pt -I would become a biased estimator since the sample Z is drawn from Π θ + δ u which depends on u . The DFO algorithm based on g 2pt -I may not converge to a stationary solution of (1).

A remedy to handle the above issues is to consider the estimator g 2pt -II which utilizes two samples Z 1 , Z 2 , each independently drawn at a different decision variable, to form the gradient estimate. In fact, it can be shown that E [ g 2pt -II ] = ∇L δ ( θ ) yields an unbiased gradient estimator. However, due to the decoupled random samples Z 1 , Z 2 , we have

where in (a) we use the fact that ( x + y ) 2 ≥ 3 4 x 2 -3 y 2 , in (b) we assume Var [ ℓ ( θ ; Z )] := E ( ℓ ( θ ; Z ) -L ( θ )) 2 ≥ σ 2 > 0 and ℓ ( θ ; z ) is µ -Lipschitz in θ . As such, this two-point gradient estimator does not reduce the variance when compared with the estimator in (4). Note that a two-sample estimator also incurs additional sampling overhead in the scenario of Markovian samples.

5 Numerical Experiments

We examine the efficacy of the DFO ( λ ) algorithm on a few toy examples by comparing DFO ( λ ) with a simple stochastic gradient descent scheme with greedy deployment. Unless otherwise specified, we use the step size choices in (8) for DFO ( λ ) . All experiments are conducted on a server with an Intel Xeon 6318 CPU using Python 3.7. To measure performance, we record the gradient norm ∥∇L ( θ ) ∥ and estimate its expected value using at least 8 trials.

1-Dimensional Case: Quadratic Loss. The first example considers a scalar quadratic loss function ℓ : R × R → R defined by ℓ ( θ ; z ) = 1 12 z θ (3 θ 2 -8 θ -48) . To simulate the controlled Markov chain scenario, the samples are generated dynamically according to an auto-regressive (AR) process Z t +1 = (1 -γ ) Z t + γ ¯ Z t +1 with ¯ Z t +1 ∼ N ( θ , (2 -γ ) γ σ 2 ) with parameter γ ∈ (0 , 1) . Note that the stationary distribution of the AR process is Π θ = N ( θ , σ 2 ) . As such, the performative risk function in this case is L ( θ ) = E Z ∼ Π θ [ ℓ ( θ ; Z )] = θ 2 12 ( θ 2 -8 θ -48) , which is quartic in θ . Note that L ( θ ) is not convex in θ and the set of stationary solution is { θ : ∇L ( θ ) = 0 } = { 4 , 0 , -2 } , among which the optimal solution is θ PO = arg min θ L ( θ ) = 4 .

In our experiments below, we initialize all the algorithms are initialized by θ 0 = 6 . In Figure 1 (left), we compare the norms of the gradient for performative risk with pure DFO (no burn-in), the DFO ( λ ) algorithm, and stochastic gradient descent with greedy deployment scheme (SGD-GD) against the number of samples observed by the algorithms. We first observe from Figure 1 (left) that pure DFO and SGD-GD methods do not converge to a stationary point to L ( θ ) even after more samples

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

7

10

0

10

2

10

4

10

6

10

8

number of samples

i

10

-

2

10

-

1

10

0

10

1

10

2

10

3

10

4

||∇L

(

θi

)

||

2

λ

=

0.0

λ

=

0.25

λ

=

0.5

λ

=

0.75

Pure DFO

SGD

fit:

k

-

0

.

36

·

exp(4

.

91)

10

0

10

2

10

4

10

6

10

8

number of samples

i

10

-

3

10

-

2

10

-

1

10

0

10

1

10

2

||∇L

(

θi

)

||

2

λ

=

0.0

λ

=

0.25

λ

=

0.5

λ

=

0.75

Pure DFO

SGD

fit:

k

-

0

.

44

·

exp(3

.

11)

10

0

10

1

10

2

10

3

10

4

10

5

10

6

10

number of samples

i

10

-

2

10

-

1

10

0

10

1

||∇L

(

θi

)

||

2

λ

=

0.0

λ

=

0.25

λ

=

0.5

λ

=

0.75

Pure DFO

SGD

are observed. On the other hand, DFO ( λ ) converges to a stationary point of L ( θ ) at the rate of ∥∇L ( θ ) ∥ 2 = O (1 /S 0 . 36 ) , matching Theorem 3.1 that predicts a rate of O (1 /S 1 / 3 ) , where S is the total number of samples observed.

Besides, we observe that with large λ = 0 . 75 , DFO ( λ ) converges at a faster rate at the beginning (i.e., transient phase), but the convergence rate slows down at the steady phase (e.g., when no. of samples observed is greater than 10 6 ) compared to running the same algorithm with smaller λ .

Higher Dimension Case: Markovian Pricing. The second example examines a multi-dimensional ( d = 5 ) pricing problem similar to (Izzo et al., 2021, Sec. 5.2). The decision variable θ ∈ R 5 denotes the prices of d = 5 goods and κ is a drifting parameter for the prices. Our goal is to maximize the average revenue E Z ∼ Π θ [ ℓ ( θ ; Z )] with ℓ ( θ ; z ) = -⟨ θ | z ⟩ , where Π θ ≡ N ( µ 0 -κ θ , σ 2 I ) is the unique stationary distribution of the Markov process (i.e., an AR process)

Note that in this case, the performative optimal solution is θ PO = arg min θ L ( θ ) = µ 0 / (2 κ ) . We set γ = 0 . 5 , σ = 5 , drifting parameter κ = 0 . 5 , initial mean of non-shifted distribution µ 0 = [ -2 , 2 , -2 , 2 , -2] ⊤ . All the algorithms are initialized by θ 0 = [2 , -2 , 2 , -2 , 2] ⊤ . We simulate the convergence behavior for different algorithms in Figure 1 (middle). Observe that the differences between the DFO ( λ ) algorithms with different λ becomes less significant than Figure 1 (left).

Markovian Performative Regression. The last example considers the linear regression problem in (Nagaraj et al., 2020) which is a prototype problem for studying stochastic optimization with Markovian data (e.g., reinforcement learning). Unlike the previous examples, this problem involves a pair of correlated r.v.s that follows a decision-dependent joint distribution. We adopt a setting similar to the regression example in (Izzo et al., 2021), where ( X,Y ) ∼ Π θ with X ∼ N (0 , σ 2 1 I ) , Y | X ∼

2

(

β

(

θ

)

X

, σ

2

)

,

β

(

θ

) =

a

0

+

a

In this case, the performative risk is:

N

⟨

|

⟩

⟨

|

⟩ -

µ

2

∥

∥

For simplicity, we assume σ 2 1 (1 -a 1 ) = σ 2 1 a 2 1 -2 σ 2 1 a 1 + σ 2 1 + µ/ 2 , from which we can deduce θ PO = a 0 . In this experiment, we consider Markovian samples ( ˜ X t , ˜ Y t ) T t =1 drawn from an AR process:

for any t ≥ 1 . We set d = 5 , a 0 = [ -1 , 1 , -1 , 1 , -1] ⊤ , a 1 = 0 . 5 , σ 2 1 = σ 2 2 = 1 , regularization parameter µ = 0 . 5 , mixing parameter γ = 0 . 1 . The algorithms are initialized with θ 0 = [1 , -1 , 1 , -1 , 1] ⊤ . Figure 1 (right) shows the result of the simulation. Similar to the previous examples, we observe that pure DFO and SGD fail to find a stationary solution to L ( θ ) . Meanwhile, DFO ( λ ) converges to a stationary solution after a reasonable number of samples are observed.

Conclusions. We have described a derivative-free optimization approach for finding a stationary point of the performative risk function. In particular, we consider a non-i.i.d. data setting with samples generated from a controlled Markov chain and propose a two-timescale step sizes approach in constructing the gradient estimator. The proposed DFO ( λ ) algorithm is shown to converge to a stationary point of the performative risk function at the rate of O (1 /T 1 / 3 ) .

1

θ

. The loss function is

ℓ

(

θ

;

x, y

) = (

x

θ

y

)

2

+

θ

2

.

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

References

Agarwal, A., Dekel, O., and Xiao, L. Optimal algorithms for online convex optimization with multi-point bandit feedback. In Annual Conference Computational Learning Theory , 2010.

Brown, G., Hod, S., and Kalemaj, I. Performative prediction in a stateful world. In International Conference on Artificial Intelligence and Statistics , pp. 6045-6061. PMLR, 2022.

Doan, T. T. Finite-time analysis of markov gradient descent. IEEE Transactions on Automatic Control , 2022.

Dong, J., Roth, A., Schutzman, Z., Waggoner, B., and Wu, Z. S. Strategic classification from revealed preferences. In Proceedings of the 2018 ACM Conference on Economics and Computation , pp. 55-70, 2018.

Drusvyatskiy, D. and Xiao, L. Stochastic optimization with decision-dependent distributions. Mathematics of Operations Research , 2022.

Flaxman, A. D., Kalai, A. T., and McMahan, H. B. Online convex optimization in the bandit setting: Gradient descent without a gradient. In Proceedings of the Sixteenth Annual ACM-SIAM Symposium on Discrete Algorithms , SODA '05, pp. 385-394, USA, 2005. Society for Industrial and Applied Mathematics. ISBN 0898715857.

Ghadimi, S. and Lan, G. Stochastic first- and zeroth-order methods for nonconvex stochastic programming, 2013. URL https://arxiv.org/abs/1309.5549 .

Hardt, M., Megiddo, N., Papadimitriou, C., and Wootters, M. Strategic classification. In Proceedings of the 2016 ACM conference on innovations in theoretical computer science , pp. 111-122, 2016.

Izzo, Z., Ying, L., and Zou, J. How to learn when data reacts to your model: Performative gradient descent. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pp. 4641-4650. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/izzo21a.html .

Izzo, Z., Zou, J., and Ying, L. How to learn when data gradually reacts to your model. In International Conference on Artificial Intelligence and Statistics , pp. 3998-4035. PMLR, 2022.

Jamieson, K. G., Nowak, R., and Recht, B. Query complexity of derivative-free optimization. In Pereira, F., Burges, C., Bottou, L., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc., 2012. URL https://proceedings. neurips.cc/paper/2012/file/e6d8545daa42d5ced125a4bf747b3688-Paper.pdf .

Karimi, B., Miasojedow, B., Moulines, E., and Wai, H.-T. Non-asymptotic analysis of biased stochastic approximation scheme. In Conference on Learning Theory , pp. 1944-1974. PMLR, 2019.

Li, Q. and Wai, H.-T. State dependent performative prediction with stochastic approximation. In International Conference on Artificial Intelligence and Statistics , pp. 3164-3186. PMLR, 2022.

Mendler-D¨ unner, C., Perdomo, J., Zrnic, T., and Hardt, M. Stochastic optimization for performative prediction. Advances in Neural Information Processing Systems , 33:4929-4939, 2020.

Miller, J., Perdomo, J. C., and Zrnic, T. Outside the echo chamber: Optimizing the performative risk. In International Conference on Machine Learning , 2021.

Nagaraj, D., Wu, X., Bresler, G., Jain, P., and Netrapalli, P. Least squares regression with markovian data: Fundamental limits and algorithms. Advances in neural information processing systems , 33: 16666-16676, 2020.

Narang, A., Faulkner, E., Drusvyatskiy, D., Fazel, M., and Ratliff, L. J. Multiplayer performative prediction: Learning in decision-dependent games. arXiv preprint arXiv:2201.03398 , 2022.

Nemirovski˘ ı, A. S. Problem complexity and method efficiency in optimization . Wiley series in discrete mathematics. Wiley, Chichester, 1983.

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

Nesterov, Y. and Spokoiny, V. G. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics , 17:527-566, 2017.

Perdomo, J., Zrnic, T., Mendler-D¨ unner, C., and Hardt, M. Performative prediction. In International Conference on Machine Learning , pp. 7599-7609. PMLR, 2020.

Ray, M., Ratliff, L. J., Drusvyatskiy, D., and Fazel, M. Decision-dependent risk minimization in geometrically decaying dynamic environments. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pp. 8081-8088, 2022.

Robert, C. P., Casella, G., and Casella, G. Monte Carlo statistical methods , volume 2. Springer, 1999.

Roy, A., Balasubramanian, K., and Ghadimi, S. Projection-free constrained stochastic nonconvex optimization with state-dependent markov data. In Advances in neural information processing systems , 2022.

Sun, T. and Li, D. Decentralized markov chain gradient descent. arXiv preprint arXiv:1909.10238 , 2019.

Sun, T., Sun, Y., and Yin, W. On markov chain gradient descent. Advances in neural information processing systems , 31, 2018.

Wu, Y. F., Zhang, W., Xu, P., and Gu, Q. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems , 33:17617-17628, 2020.

Zhang, D. J., Dai, H., Dong, L., Qi, F., Zhang, N., Liu, X., Liu, Z., and Yang, J. How do price promotions affect customer behavior on retailing platforms? evidence from a large randomized experiment on alibaba. Production and Operations Management , 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe performative prediction problem targets real-world application using deployed models, and this method  studies the optimiazation method for such a scenario. Since directly optimiazing the non-convex objective is challenging, this paper utilizes weighted Markovian samples of states to estimate the gradient and update with designed timescale step size.\n\nSTRENGTHS:\n1. The paper is in general well-written.\n2. The motivation is clear for tackling the targeted problem.\n3. The theoretical derivation and understanding are well-presented.\n4. The authors conduct various experiments settings to examine the the proposed method.\n\nWEAKNESSES:\n1. The experiments are conducted on toy examples and illustrate the efficacy of the method. The question would arise for applying the method for more challenging task setting. Also, how does the method perform in comparison with other strategies that also fit with such tasks?\n\n2. What the limitation of the proposed method? As it targets online deployed models and involves the Markov Chain sampling, what is the computational cost?\n\nQUESTIONS:\nIn general, the paper is well-written and would require background knowledge to understand the theoretical parts. My questions are listed:\n1. L90:  \"Let S be a (measurable) sample space, and µ, ν are two probability measures defined on S.\" \n    and L114: \"sample a vector µ uniformly from S^d-1\". Does that imply that µ is uniformly distributed \n    among S? \n\n2. Given that Πθ(·) is unknown (L105-106) while the Markov kernel Tθ(·, ·) is based on Πθ(·), how would the Markov kernel conduct toward sampling?\n\n3. L138-140: \"Under this setting, suppose that the previous state/sample is Z, the next sample follows the distribution Z′ ∼ Tθ(Z, ·) which is not necessarily the same as Πθ(·).\" It might needs some theoretical background to understand this, so my question is that since Πθ(·) serves a stational distribution given θ, why Z not converge to certain point and then Z′ should be sampled from the same distribution?\n\nLIMITATIONS:\nPlease see the weaknesses and questions part.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a derivative-free two-time scale  for performative prediction problem. The two-timescale facilitates a faster accumulation of samples to compute a gradient with smaller bias. For smooth nonconvex objective, the work proves a iteration complexity of $O(\\epsilon^{-3})$ for the $\\epsilon$-staitonary point. They illustrate results via numerical experiments.\n\nSTRENGTHS:\n1. This paper deals with an important problem: derivative-free optimization of nonconvex function under performative prediction setup. \n\n2. The convergence rate seems reasonable. \n\n3. The experiments are well-motivated.\n\nWEAKNESSES:\n1. **The applications are shown for squared loss. But it is not a bounded loss and does not satisfy Assumption 3.2. This leads me to believe that Assumption 3.2 is made for the convenience of theoretical analysis.**\n\nIn fact, **one of the major challenges in the decision-dependent noise or state-dependent Markov noise setup is that the algorithm is often not stable** for unconstrained optimization (see [1,2] below). Assumption 3.2 helps to avoid this issue.\n\n**So Assumption 3.2 is crucial for Lemma E.3 and Lemma B.i (i=1,2,3,4). Then the theoretical analysis of the algorithm does not apply to squared loss which is one of the most popular loss functions.**\n\n2. The proof techniques required are pretty similar to Wu et al., 2020 ([3] below). In fact, **the main result is almost same as Corollary 4.9 of Wu et al., 2020. It's just that Corollary 4.9 of Wu et al., 2020 is wrapped in the cover of reinforcement learning but the algorithm, underlying proof techniques, and result are same.**  Wu et al., 2020 does achieve a faster rate of $\\tilde{O}(\\epsilon^{-2.5})$ although I guess here the poor rate is due to derivative-free algorithm. \n\n\n[1] Liang, Faming. \"Trajectory averaging for stochastic approximation MCMC algorithms.\" (2010): 2823-2856.\n\n[2] Andrieu, Christophe, Éric Moulines, and Pierre Priouret. \"Stability of stochastic approximation under verifiable conditions.\" SIAM Journal on control and optimization 44, no. 1 (2005): 283-312.\n\n[3] Wu, Yue Frank, Weitong Zhang, Pan Xu, and Quanquan Gu. \"A finite-time analysis of two time-scale actor-critic methods.\" Advances in Neural Information Processing Systems 33 (2020): 17617-17628.\n\nQUESTIONS:\n1. It is not clear to me whether the derivative-free approach helps to deal with state-dependence of the Markov chain like mentioned in line 105-112? In other words, say there is a gradient oracle such that $g_{k}^{(m)}=\\nabla l(\\theta_k^{(m)},Z_{k}^{(m)})$. What will happen to the convergence rate of the algorithm then?\n\n2. Please highlight the technical novelties required compared to Wu et al., 2020 (apart from the zeroth-order approximation). See point 2 of Weakness for details. \n\nI may change my score depending on the reply.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies performative prediction when the data is stateful, in particular generated via a controlled Markov chain, and the learner's loss is possibly nonconvex. The paper develops a two-timescale derivative-free optimization algorithm for this setting and shows a O(1/eps^3) sample complexity for finding a point with squared gradient norm at most eps.\n\nSTRENGTHS:\nThere aren't many convergence results in performative prediction for nonconvex settings, so this is one strength. The stateful setting has been studied before in several papers, but this paper studies this setting under quite a bit of generality. The treatment of the stateful setting in this paper is probably my favorite treatment of the setting in the literature. The paper is very clearly written and easy to follow, also providing intuition for the ideas behind the analysis, which I appreciated.\n\nWEAKNESSES:\nThere are some limitations to the conceptual novelty, in the sense that a similar algorithm has been studied outside of performative prediction, and the controlled Markov chain model for the distribution map has been studied. It would be good to be more precise about the differences to the recent works in performative prediction studying the stateful setting (bottom of page 2).\n\nThe problem is perhaps arguably a bit niche since it goes away if the performativity is not stateful, which is the most commonly studied observation model in performative prediction. In that case, a simple application of the Flaxman et al. \"gradient descent without a gradient\" algorithm suffices. The issue in this paper is that we can't sample from the stationary distribution directly so a naive application of the Flaxman et al. algorithm doesn't work. All this being said, the stateful setting is very well-motivated and deserves its own analyses, and this paper gives a clever solution.\n\nQUESTIONS:\n1. Can you please clarify the differences to prior work studying stateful performative prediction?\n2. Do you have a sense of why SGD has these strange dips in Figure 1?\n3. In lines 181-182, you say that the geometric dynamically environment of Ray et al. (2022) constitutes a special case of your assumptions. Can you give more natural examples of stateful distribution maps that satisfy your assumptions but are different from Ray et al.?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is on performative prediction i.e. a setting where the data distribution changes in response to the predictions of a learned model. The prototypical example of such a setting is where a learned model is used to make loan decisions, and then people or companies adjust their behavior based on knowledge of the learned model in order to secure more favorable loan terms. This paper extends prior work on performative prediction to the setting where the data distribution follows a controlled Markov process, where the control is given by the predictive model. While this setting has been studied before, this paper removes previous structural assumptions on (1) the loss function used to evaluate the model and (2) the dependence of the data distribution on model. Notably the loss function is not assumed to be convex, and so convergence is established to a stationary point (i.e. a point where the magnitude of the gradient is small).\nThe algorithm designed is based on standard methods for derivative-free optimization, with a two-timescale step-size modification that updates model parameters more slowly than the generation of gradient estimates in order to deal with high variance in the gradient estimator.\n\nSTRENGTHS:\nThe extension of algorithms for performative prediction to the setting without convexity of the loss expands the range of techniques for such problems. The paper clearly discusses the differences with prior work and gives a concise and intuitive overview of the modifications required to make derivative-free optimization work for this setting.\n\nWEAKNESSES:\nWhile I do appreciate the generality of extending algorithms for performative prediction beyond convex losses (as well as beyond the mixture dominance assumption on the data distribution), there are a couple of limitations that arise from assuming so little about the loss and data distribution.\n\n1. The algorithm converges to a stationary point of the performative risk, rather than to a point that approximately minimizes the preformative risk. Of course this is necessary in the completely unstructured setting considered here, but it is clearly a weaker statement than e.g. the initial work of Perdomo et. al. that showed (for strongly convex losses) convergence of repeated risk minimization to a point that is close to an actual minimizer of the performative risk. This paper should state more clearly that such a strong result cannot be hoped for in the setting considered. In particular, the claim on line 43 seems somewhat misleading as written.\n\n2. Removing the convexity assumption seems to come at the cost of introducing an additional assumption. In particular, Assumption 3.3 requires that the distribution map $\\Pi_{\\theta}$ is Lipschitz with respect to the total variation distance, rather than the Wasserstein 1 distance used in prior work. This is a **much stronger** assumption in many natural settings e.g. the mean of $n$-independent random $\\pm 1$ valued variables has a distribution that converges in Wasserstein 1 distance to the Gaussian distribution at a rate of $\\frac{1}{\\sqrt{n}}$, but has the maximum possible total variation distance of 1 from the standard Gaussian distribution.\n\n3. The algorithm has a slower convergence rate than those in prior work, as the authors show is necessary in this highly general setting. The original paper introducing performative prediction shows that repeated risk minimization converges at a linear rate, and thus allows us to leverage whatever fast optimization method fits the particular problem for each risk minimization step. In contrast, the algorithm in this paper requires us to essentially sample many, many random perturbations of the model and then deploy/evaluate each one in order to obtain gradient estimates, making it highly impractical for real-world problems.\n\nTo summarize, the generality of the setting has several drawbacks in terms of the solution quality, performance of the algorithm, and most notably the additional assumption described in (2) above. While some of these drawbacks are provably necessary (again with the notable exception of the issue in (2)), taken together they suggest that perhaps this complete lack of structure is not a particularly good model of the types of problems for which performative prediction is interesting.\n\nQUESTIONS:\n1. Is the Lipschitz assumption with respect to the total variation distance necessary in this setting?\n2. Are there any intermediate models with assumptions weaker than strong convexity of the loss, but stronger than the assumptions made in this paper that could also be interesting?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper considers the performative prediction problem, where the goal of learner is to optimize the expectation of the known loss function over a decision-dependent unknown data distribution that evolves according to an underlying controlled Markov chain. Authors presents a stochastic derivative-free optimization algorithm $DFO(\lambda)$ that achieves $O(d^2/\varepsilon^3)$ sample complexity using gradient accumulation mechanism and two-timescale diminishing step-sizes.

STRENGTHS:
- The presented algorithm achieves a $O(1/\varepsilon^3)$ sample complexity under a challenging setting of Markovian data.
- The other possible types of gradient estimator were considered that creates a more complete picture in the presented setup.
- The presentation of the paper is clear.

WEAKNESSES:
- No presented lower bound for this problem, thus it is not clear is the presented rates improbable or not.

QUESTIONS:
- What could be challenges to generalize the proposed algorithm to the milder assumption on the Markov chain, i.e. V-ergodicity instead of the uniform ergodicity.
- What is the bias of the first version of two-point estimate $g_{2pt-I}$? On which variables it depends in the worst case?
- Is there some stronger assumptions on the Markov chain under which it is possible to recover $O(1/\varepsilon^2)$ rates as in i.i.d. case?
- Can be this algorithm applied to the reinforcement leaning problem?

LIMITATIONS:
This is a theoretical paper that does not need to address the potential societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07