Title: Zero-Regret Performative Prediction Under Inequality Constraints
Venue: NeurIPS 2023
Accepted: yes
Keywords: Performative prediction

decision-dependent distribution

inequality constraints

primal-dual algorithm.
Abstract: Performative prediction is a recently proposed framework where predictions guide decision-making and hence influence future data distributions. Such performative phenomena are ubiquitous in various areas, such as transportation, finance, public policy, and recommendation systems. To date, work on performative prediction has only focused on unconstrained problems, neglecting the fact that many real-world learning problems are subject to constraints. This paper bridges this gap by studying performative prediction under inequality constraints. Unlike most existing work that provides only performative stable points, we aim to find the optimal solutions. Anticipating performative gradient is a challenging task, due to the agnostic performative effect on data distributions. To address this issue, we first develop a robust primal-dual framework that requires only approximate gradients up to a certain accuracy, yet delivers the same order of performance as the stationary stochastic primal-dual algorithm without performativity. Based on this framework, we then propose an adaptive primal-dual algorithm for location families. Our analysis demonstrates that the proposed adaptive primal-dual algorithm attains $\mathcal{O}(\sqrt{T})$ regret and constraint violations, using only $\sqrt{T} + 2T$ samples, where $T$ is the time horizon. To our best knowledge, this is the first study and analysis on the optimality of the performative prediction problem under inequality constraints. Finally, we validate the effectiveness of our algorithm and theoretical results through numerical simulations.

Zero-Regret Performative Prediction Under Inequality Constraints

Abstract

Performative prediction is a recently proposed framework where predictions guide decision-making and hence influence future data distributions. Such performative phenomena are ubiquitous in various areas, such as transportation, finance, public policy, and recommendation systems. To date, work on performative prediction has only focused on unconstrained scenarios, neglecting the fact that many realworld learning problems are subject to constraints. This paper bridges this gap by studying performative prediction under inequality constraints. Unlike most existing work that provides only performative stable points, we aim to find the optimal solutions. Anticipating performative gradients is a challenging task, due to the agnostic performative effect on data distributions. To address this issue, we first develop a robust primal-dual framework that requires only approximate gradients up to a certain accuracy, yet delivers the same order of performance as the stochastic primal-dual algorithm without performativity. Based on this framework, we then propose an adaptive primal-dual algorithm for location family. Our analysis demonstrates that the proposed adaptive primal-dual algorithm attains O ( √ T ) regret and constraint violations, using only √ T +2 T samples, where T is the time horizon. To our best knowledge, this is the first study and analysis on the optimality of the performative prediction problem under inequality constraints. Finally, we validate the effectiveness of our algorithm and theoretical results through numerical simulations.

1 Introduction

Stochastic optimization plays a critical role in statistical sciences and data-driven computing, where the goal is to learn decision rules (e.g., classifiers) based on limited samples that generalize well to the entire population. Most prior studies on stochastic optimization [Heyman and Sobel, 2004; Karimi et al., 2019; Powell, 2019] rely on the assumption that the data of the entire population follows a static distribution. This assumption, however, does not hold in applications where the data distributions change dynamically in response to decision-makers' actions [Hardt et al., 2016; Dong et al., 2018]. For instance, in transportation, travel time estimates [Mori et al., 2015] influence routing decisions, resulting in realized travel times; in banking, credit evaluation criteria [Abdou and Pointon, 2011] guide borrowers' behaviors and subsequently their credit scores; and in advertising, recommendations [García-Sánchez et al., 2020] shape customer preferences, leading to consumption patterns. Such interplay between decision-making and data distribution arises widely in various areas, such as transportation, finance, public policy, and recommendation systems.

The seminal work [Perdomo et al., 2020] formalized the phenomenon as performative prediction , which represents the strategic responses of data distributions to the taken decisions via decision-

dependent distribution maps [Quinonero-Candela et al., 2008]. Since then, an increasing body of research has been dedicated to performative prediction problems. Most existing studies are focused on identifying performative stable points [Li and Wai, 2022; Li et al., 2022; Drusvyatskiy and Xiao, 2022; Brown et al., 2022; Mendler-Dünner et al., 2020; Wood et al., 2021; Ray et al., 2022], given the complexities of the decision-induced distribution shifts and the unknown decision-dependent distributions. The proposed algorithms typically iteratively retrain the deployed models until convergence. However, performative stability generally does not imply performative optimality. Aiming to achieve optimal performance, a few recent works designed effective algorithms by leveraging rich performative feedback [Jagadeesan et al., 2022], or by making some parametric assumptions on the underlying distribution maps. For instance, the distribution maps belong to the location family with linear structure [Miller et al., 2021] or are from the exponential family [Izzo et al., 2021].

All the aforementioned work on performative prediction is focused on unconstrained learning problems. However, in the real world, many performative prediction applications are subject to constraints [Detassis et al., 2021; Wood and Dall'Anese, 2022a]. Constraints can be used to ensure the satisfaction of desired properties, such as fairness, safety, and diversity. Examples include safety and efficiency constraints in transportation [Metz, 2021], relevance and diversity constraints in advertising [Khamis, 2020], and risk tolerance and portfolio constraints in financial trading [Föllmer and Schied, 2002], etc. In addition, constraints can serve as side information to enhance the learning outcomes, e.g., by narrowing the scope of exploration or by incorporating prior knowledge Serafini and Garcez [2016]; Wu et al. [2018]. As performative shifts can rarely be analyzed offline, incorporating constraints on what constitutes safe exploration [Turchetta et al., 2019] or facilitates optimization [Wood and Dall'Anese, 2022b] is of crucial importance.

Despite its importance, research on performative prediction under constraints has been neglected so far. Although some work [Izzo et al., 2021; Piliouras and Yu, 2022] restricted decision variables to certain regions, this feasible set restriction was simply handled by projections. This paper bridges this gap by studying the performative prediction problem under inequality constraints, for which simple projection is inadequate to handle. Unlike most existing work that provides only performative stable points, we aim to find the optimal solutions. As aforementioned, finding performative optima is a challenging task because we now need to anticipate performative effect actively rather than simply retrain models in a myopic manner.

However, the performative effect on distribution maps is unknown, which hinders the computation of exact performative gradient. To solve this problem, we develop a robust primal-dual framework that admits inexact gradients. We ask the following questions: How does the gradient approximation error affect the performance of the primal-dual framework? Under what accuracy can the approximate gradients maintain the performance order of the stochastic primal-dual algorithm without performativity? How to construct effective gradient approximations that attain the desired accuracy? We answer the above thoroughly. Our idea hinges on enhancing gradient approximation with the parametric knowledge of distribution maps. In particular, we follow existing studies [Miller et al., 2021; Jagadeesan et al., 2022] and focus on the family of location maps. Location family exhibit a favorable linear structure for algorithm development while maintaining broad generality to model many real-world applications. Distribution maps of this type are ubiquitous throughout the performative prediction literature, such as strategic classification [Hardt et al., 2016; Perdomo et al., 2020], linear regression [Miller et al., 2021], email spam classification [Li et al., 2022], ride-share [Narang et al., 2022], among others. Nevertheless, we emphasize that our robust primal-dual framework is applicable to other forms of distributions with effective gradient approximation methods.

To our best knowledge, this paper provides the first study and analysis on the optimality of performative prediction problems under inequality constraints. We highlight the following key contributions:

We develop a robust primal-dual framework that requires only approximate gradients up to an accuracy of O ( √ T ) , yet delivers the same order of performance as the stochastic primal-dual algorithm without performativity, where T is the time horizon. Notably, the robust primal-dual framework does not restrict the approximate gradients to be unbiased and hence offers more flexibility to the design of gradient approximation.

Based on this framework, we propose an adaptive primal-dual algorithm for location family, which consists of an online stochastic approximation and an offline parameter estimation for the performative gradient approximation. Our analysis demonstrates that the proposed algorithm achieves O ( √ T ) regret and constraint violations, using only √ T +2 T samples.

Finally, we conduct experiments on two examples: multi-task linear regression and multi-asset portfolio. The numerical results validate the effectiveness of our algorithm and theoretical analysis.

1.1 Related Work

The study on performative prediction was initiated in [Perdomo et al., 2020], where the authors defined the notion of performative stability and demonstrated how performative stable points can be found through repeated risk minimization and stochastic gradient methods. Since then, substantial efforts have been dedicated to identifying performative stable points in various settings, such as single-agent [Mendler-Dünner et al., 2020; Drusvyatskiy and Xiao, 2022; Brown et al., 2022], multiagent [Li et al., 2022; Piliouras and Yu, 2022], games [Narang et al., 2022], reinforcement learning [Mandal et al., 2023], and online learning [Wood et al., 2021; Wood and Dall'Anese, 2022a].

A few recent works aimed to achieve performative optimality, a more stringent solution concept than performative stability. In [Miller et al., 2021], the authors evaluated the conditions under which the performative problem is convex and proposed a two-stage algorithm to find the performative optima for distribution maps in location family. Another paper on performative optimality is [Izzo et al., 2021], which proposed a PerfGD algorithm by exploiting the exponential structure of the underlying distribution maps. Both works took advantage of parametric assumptions on the distribution maps. Alternatively, [Jagadeesan et al., 2022] proposed a performative confidence bounds algorithm by leveraging rich performative feedback, where the key idea is to exhaustively explore the feasible region with an efficient discarding mechanism.

A closely related work is [Wood and Dall'Anese, 2022b], which studied stochastic saddle-point problems with decision-dependent distributions. The paper focused on performative stable points (equilibrium points), whereas we aim at the performative optima, which is more challenging. Another difference is that [Wood and Dall'Anese, 2022b] only demonstrated the convergence of the proposed primal-dual algorithm in the limit, without providing an explicit finite-time convergence rate. In contrast, we provide O ( √ T ) regret and O ( √ T ) constraint violation bounds for the proposed algorithm in this paper.

2 Problem Setup

We study a performative prediction problem with loss function glyph[lscript] ( θ ; Z ) , where θ ∈ Θ is the decision variable, Z ∈ R k is an instance, and Θ ∈ R d is the set of available decisions. Different from in stationary stochastic optimization where distributions of instances are fixed, in performative prediction, the distribution of Z varies with the decision variable θ , represented by Z ∼ D ( θ ) . In this paper, we consider that the decision variable θ is subject to a constraint g ( θ ) glyph[precedesequal] 0 , where g ( · ) : Θ → R m . The constraint g ( · ) can be imposed on θ to ensure certain properties, such as fairness, safety, and diversity, or to incorporate prior knowledge. We assume that g ( · ) is available at the decision-maker in advance of the optimization. Ideally, the goal of the decision-maker is to solve the following stochastic problem:

where E Z ∼D ( θ ) glyph[lscript] ( θ ; Z ) is referred to as performative risk , denoted by PR( θ ) .

Problem (1) is, however, impossible to be solved offline because the distribution map D ( θ ) is unknown. Instead, the decision-maker needs to interact with the environment by making decisions to explore the underlying distributions. Given the online nature of this task, we measure the loss of a sequence of chosen decisions θ 1 , · · · , θ T by performative regret , defined as

where the expectation is taken over the possible randomness in the choice of { θ t } T t =1 , and θ PO is the performative optimum, defined as

Performative regret measures the suboptimality of the chosen decisions relative to the performative optima. Another performance metric for problem (1) on evaluating the decision sequence { θ t } T t =1 is

constraint violation , given by

where we use the symbol [ m ] to represent the integer set { 1 , · · · , m } throughout this paper.

Applications pertaining to problem (1) are ubiquitous. Next is an example.

Example 1 ( Multi-Asset Portfolio ) . Consider a scenario where an investor wants to allocate his/her investment across a set of l assets, such as stocks, bonds, and commodities. The objective is to maximize the expected return subject to certain constraints, including liquidity, diversity, and risk tolerance. Let z i denote the rate of return of the i th asset and θ i denote its weight of allocation, ∀ i ∈ [ l ] . The investment can affect the future rates of return of the assets and, consequently, the overall expected return of the portfolio. For example, excessive investment in a particular asset may lead to a decline in the rate of return of other assets. Let z = [ z 1 , · · · , z l ] glyph[latticetop] and θ = [ θ 1 , · · · , θ l ] glyph[latticetop] . Then, the expected return of the portfolio is E [ r p ] := E z ∼D ( θ ) z glyph[latticetop] θ . Typically, the risk of the portfolio is measured by the variance of its returns, given by θ T Ψ θ , where Ψ is the covariance matrix of z . One common approach to model liquidity is using the bid-ask spread, which measures the gap between the highest price a buyer is willing to pay (the bid) and the lowest price a seller is willing to accept (the ask) for a particular asset. Denote the vector of the bid-ask spread of the l assets by s = [ s 1 , · · · , s l ] glyph[latticetop] . Then, a liquidity constraint on the portfolio can be defined as s glyph[latticetop] θ ≤ S , where S is the maximum allowable bid-ask spread. The multi-asset portfolio problem can be formulated as:

where glyph[epsilon1] restricts the maximum amount of investment to one asset, and ρ is the risk tolerance threshold.

In this paper, our goal is to design an online algorithm that achieves both sublinear regret and sublinear constraint violations with respect to the time horizon T , i.e., Reg( T ) ≤ o ( T ) and Vio i ( T ) ≤ o ( T ) , for all i ∈ [ m ] . Then, the time-average regret satisfies Reg( T ) /T ≤ o (1) , and the time-average constraint violations satisfy Vio i ( T ) /T ≤ o (1) , for all i ∈ [ m ] . Both asymptotically go to zero as T goes to infinity. Therefore, the performance of the decision sequence { θ t } T t =1 generated by the algorithm approaches that of the performative optimum θ PO as T goes to infinity.

3 Adaptive Primal-Dual Algorithm

3.1 Robust Primal-Dual Framework

In this subsection, we develop a robust primal-dual framework for the performative prediction problem under inequality constraints. Our approach involves finding a saddle point for the regularized Lagrangian of problem (1). The Lagrangian, denoted by L ( θ , λ ) , is defined as

where θ is the primal variable (decision), λ is the dual variable (multiplier), η > 0 is the stepsize of the algorithm, and δ > 0 is a control parameter. In (2), we add the regularizer -δη 2 ‖ λ ‖ 2 2 to suppress the growth of the multiplier λ , so as to improve the stability of the algorithm.

To find the saddle point of the Lagrangian L ( θ , λ ) , we utilize alternating gradient update on the primal variable θ and the dual variable λ . The gradients of L ( θ , λ ) with respect to θ and λ are respectively given by

where ∇ θ g ( θ ) is the Jacobian matrix of g ( · ) . In (3), ∇ θ PR( θ ) is the gradient of the performative risk PR( θ ) , given by

where p θ ( Z ) is the density of D ( θ ) .

Since the data distribution D ( θ ) is unknown, the exact gradient of the performative risk PR( θ ) is unavailable, posing a significant challenge to the algorithm design. In this paper, we tackle this

issue using a robust primal-dual framework. The main idea is to construct gradient approximations from data and then perform alternating gradient updates based on the inexact gradients. Denote by ∇ θ ̂ PR t ( θ ) the approximation of the gradient ∇ θ PR( θ ) at the t th iteration. Correspondingly, an approximation for the Lagrangian gradient ∇ θ L ( θ , λ ) at the t th iteration is given by

The robust alternating gradient update is then performed as

Then, the next question is how to construct effective gradient approximations that achieve satisfactory performance.

By (4), the expectation over D ( θ ) in the gradient ∇ θ PR( θ ) can be approximated by samples, while the unknown probability density p θ ( Z ) presents the main challenge. Most existing research circumvented this problem by omitting the second term in ∇ θ PR( θ ) . This essentially gives a performative stable point. However, as pointed out in [Miller et al., 2021], performative stable points can be arbitrarily sub-optimal, leading to vacuous solutions. Instead, if we have further knowledge about the parametric structure of p θ ( Z ) , the complexity of gradient approximation can be greatly reduced. In this regard, [Miller et al., 2021] and [Jagadeesan et al., 2022] exploited the linear structure of location family, and [Izzo et al., 2021] considered distribution maps within exponential family. Following [Miller et al., 2021; Jagadeesan et al., 2022], we focus on the family of location maps in this paper because it exhibits a favorable linear structure for algorithm development while maintaining broad generality to various applications. Next, we develop an adaptive algorithm for problem (1) with location family distribution maps based on the above robust primal-dual framework.

3.2 Algorithm Design for Location family

In the setting of location family, the distribution map depends on θ via a linear shift, i.e.

where Z 0 ∼ D 0 is a base component representing the data without performativity, A ∈ R k × d captures the performative effect of decisions, and d = means equal in distribution. Denote by Σ the covariance matrix of the base distribution D 0 . Note that D 0 is still unknown. Plugging the distribution definition (7) into (4), we obtain a more explicit expression for ∇ θ PR( θ ) as

To compute ∇ θ PR( θ ) , we still need to address two problems: the unknown base distribution D 0 and the unknown performative parameter A . We tackle them as follows.

Offline Stochastic Approximation: We approximate the base distribution D 0 offline by sample average approximation [Kleywegt et al., 2002]. Specifically, before the start of the alternating gradient update, we first draw n samples { Z 0 ,i } n i =1 from D ( 0 ) . These samples are used to approximate the expectation over Z 0 throughout the algorithm iteration. Hence, the sample complexity from this expectation approximation is fixed at n .

Online Parameter Estimation: We estimate the parameter A via online least squares. In each round of the alternating gradient update, we first take the current decision θ t and its perturbed point θ t + u t to observe samples Z t ∼ D ( θ t ) and Z ′ t ∼ D ( θ t + u t ) , respectively, where u t is an injected noise specified by the decision-maker. We have E [ Z t -Z ′ t | u t ] = Au t . Then, the least-square problem at the t th iteration is designed as

Let ̂ A t -1 be the estimate of A at the ( t -1) th iteration. Based on it, we construct a new estimate ̂ A t for A by using gradient descent on the above least-square objective. This gives us the update

Algorithm 1 Adaptive Primal-Dual Algorithm

1: Take decision θ = 0 and observe n samples Z 0 ,i ∼ D 0 , ∀ i ∈ [ n ] .

2: Initialize θ 1 ∈ Θ arbitrarily. Set λ 1 = 0 and ̂ A 0 = 0 .

3: for t = 1 to T do

4: Take decision θ t and observe Z t ∼ D ( θ t ) .

5: Generate noise u t .

7: Update parameter estimate by ̂ A t = ̂ A t -1 + ζ t ( Z ′ t -Z t -̂ A t -1 u t ) u glyph[latticetop] t .

6: Take decision θ t + u t and observe Z ′ t ∼ D ( θ t + u t ) .

8: Update gradient approximation ∇ θ ̂ PR t ( θ t ) by (8).

9: Compute ∇ θ ̂ L t ( θ t , λ t ) = ∇ θ ̂ PR t ( θ t ) + ∇ θ g ( θ t ) glyph[latticetop] λ t .

10: Update the primal variable by θ t +1 = Π Θ ( θ t -η ∇ θ ̂ L t ( θ t , λ t ) ) .

11: Compute ∇ λ L ( θ t , λ t ) = g ( θ t ) -δη λ t .

12: Update the dual variable by λ t +1 = [ λ t + η ∇ λ L ( θ t , λ t )] + .

13: end for

where ζ t is the stepsize of the online least squares at the t th iteration.

Adaptive Primal-Dual Algorithm: With the above preparation, we obtain an approximation for the gradient ∇ θ PR( θ t ) at the t th iteration as

Given ∇ θ ̂ PR t ( θ t ) , we develop an adaptive primal-dual algorithm for the constrained performative prediction problem (1) based on the robust primal-dual framework in § 3.1, which is presented in Algorithm 1. In Algorithm 1, the initial decision is randomly chosen from the admissible set Θ . Both the dual variable and the parameter estimate ̂ A 0 are initialized to be zero. The algorithm maintains two sequences. One is the estimate ̂ A t , which is updated based on the newly observed samples Z t and Z ′ t , as given in Step 7. The other is the alternating gradient update on the primal and dual variables, which are respectively given in Step 10 and Step 12.

Remark 1. While this paper considers the distribution maps within the location family, we emphasize that the proposed robust primal-dual framework does not restrict to any form of distribution. For instance, the exponential family considered in [Izzo et al., 2021] with their gradient approximation method can be directly applied to our robust primal-dual framework.

4 Convergence Analysis

In this section, we analyze the convergence performance of the proposed adaptive primal-dual algorithm. We first provide the convergence result of the robust primal-dual framework. Then, we bound the error of gradient approximation in our adaptive algorithm for the location family. With these results, the convergence bounds of the adaptive primal-dual algorithm are derived. Our analysis is based on the following assumptions.

Assumption 1 ( Properties of glyph[lscript] ( θ ; Z ) ) . The loss function glyph[lscript] ( θ ; Z ) is β -smooth, L θ -Lipschitz continuous in θ , L Z -Lipschitz continuous in Z , γ θ -strongly convex in θ , and γ Z -strongly convex in Z . Moreover, we have γ θ -β 2 /γ Z > 0 .

Assumption 2 ( Compactness and Boundedness of Θ ) . The set of admissible decisions Θ is closed, convex, and bounded, i.e., there exists a constant R > 0 such that ‖ θ ‖ 2 ≤ R , ∀ θ ∈ Θ .

Assumption 3 ( Properties of g ( θ ) ) . The constraint function g ( θ ) is convex, L g -Lipschitz continuous, and bounded, i.e., there exists a constant C such that ‖ g ( θ ) ‖ 2 ≤ C , ∀ θ ∈ Θ .

Assumption 4 ( Bounded Stochastic Gradient Variance ) . For any i ∈ [ n ] and θ ∈ Θ , there exists σ ≥ 0 such that

Assumption 1 is standard in the literature of performative prediction. Assumptions 2 and 3 are widely used in the analysis of constrained optimization problems [Tan et al., 2018; Yan et al., 2019; Cao and Ba¸ sar, 2020], even with perfect knowledge of objectives. Assumption 4 bounds the variance of the stochastic gradient of PR( θ ) . Additionally, to ensure a sufficient exploration of the parameter space, we make the following assumption on the injected noises { u t } T t =1 .

Assumption 5 ( Injected Noise ) . The injected noises { u t } T t =1 are independent and identically distributed. Moreover, there exist positive constants κ 1 , κ 2 , and κ 3 such that for any t ∈ [ T ] , the random noise u t satisfies

Consider a Gaussian noise that u t ∼ N (0 , I ) , ∀ t ∈ [ T ] , we have κ 1 = 1 , κ 2 = d , and κ 3 = 3 d .

With the above assumptions, we provide some supporting lemmas below. First, we show ε -sensitivity of the location family given in (7).

Lemma 1 ( ε -Sensitivity of D ( θ ) ) . Define σ max ( A ) := max ‖ θ ‖ 2 =1 ‖ A θ ‖ 2 . The location family given in (7) is ε -sensitive with parameter ε ≤ σ max ( A ) . That is, for any θ , θ ′ ∈ Θ , we have W 1 ( D ( θ ) , D ( θ ′ )) ≤ ε ∥ ∥ θ -θ ′ ∥ ∥ 2 , where W 1 ( D , D ′ ) denotes the Wasserstein-1 distance.

See § A of the supplementary file for the proof. Building upon Lemma 1, we have the following Lemma 2 about the performative risk PR( θ ) .

Lemma 2 ( Lipschitz Continuity and Convexity of PR( θ ) ) . Consider the location family given in (7) . With Assumption 1 and Lemma 1, we have that: 1) the performative risk PR( θ ) is L -Lipschitz continuous for L ≤ L θ + L Z σ max ( A ) ; 2) the performative risk PR( θ ) is γ -strongly convex for

where σ min ( A ) := min ‖ θ ‖ 2 =1 ‖ A θ ‖ 2 .

See § B of the supplementary file for the proof. Based on the Lipschitz continuity and convexity of PR( θ ) , we provide the convergence result of the robust primal-dual framework below.

Lemma 3 ( Convergence Result of Robust Primal-Dual Framework ) . Set η = 1 √ T . Then, there exists a constant δ ∈ [ 1 -√ 1 -32 η 2 L 2 g 4 η 2 , 1+ √ 1 -32 η 2 L 2 g 4 η 2 ] such that under Assumptions 1-3, for T ≥ 32 L 2 g , the regret satisfies:

where a ∈ (0 , γ ) is a constant. Further, for any i ∈ [ m ] , the constraint violation satisfies:

Remark 2. Lemma 3 reveals the impact of gradient approximation error on the convergence performance of the robust primal-dual framework. By Lemma 3, if the accumulated gradient approximation error is less than O ( √ T ) , both the regret and the constraint violations are bounded by O ( √ T ) . Although stochastic primal-dual methods for constrained problems without performativity also use approximated (stochastic) gradients, they generally require unbiased gradient approximation [Tan et al., 2018; Yan et al., 2019; Cao and Ba¸ sar, 2022]. This requirement, however, is difficult to satisfy in performative prediction since the unknown performative effect of decisions changes the data distribution. In contrast, the robust primal-dual framework does not restrict the approximate gradients to be unbiased and hence offers more flexibility to the design of gradient approximation.

Proof of Lemma 3 is provided in § C of the supplementary file. In the next lemma, we bound the gradient approximation error of the adaptive primal-dual algorithm.

Lemma 4 ( Gradient Approximation Error ) . Set ζ t = 2 κ 1 ( t -1)+2 κ 3 , ∀ t ∈ [ T ] . Then, under Assumptions 4 and 5, the accumulated gradient approximation error is upper bounded by:

where α := max { 2 κ 3 κ 1 ‖ ̂ A 0 -A ‖ 2 F , 8 κ 2 tr( Σ ) κ 2 1 } . In Algorithm 1, we set ̂ A 0 = 0 , and thus we have α = max { 2 κ 3 κ 1 ‖ A ‖ 2 F , 8 κ 2 tr( Σ ) κ 2 } .

Remark 3. Lemma 4 demonstrates that the gradient approximation error of the adaptive primal-dual algorithm is upper bounded by O ( T/n +ln( T )) . If we set the number of initial samples n ≥ √ T , we have ∑ T t =1 E ∥ ∥ ∥ ∇ θ ̂ PR( θ t ) -∇ θ ̂ PR t ( θ t ) ∥ ∥ ∥ 2 2 ≤ O ( √ T ) . According to Lemma 3, this suffices to make the regret and constraint violation bounds to be O ( √ T ) .

Proof of Lemma 4 is presented in § D of the supplementary file. Combining Lemma 3 and Lemma 4 yields the regret and constraint violations of Algorithm 1, which is elaborated in Theorem 1 below.

Remark 4. Theorem 1 demonstrates that Algorithm 1 achieves O ( √ T + T/n ) regret and O ( √ T + T 3 4 / √ n ) constraint violations. By setting n = √ T , we have T/n = √ T and T 3 4 / √ n = √ T , and hence both the regret and constraint violations are upper bounded by O ( √ T ) . This indicates that Algorithm 1 attains the same order of performance as the stochastic primal-dual algorithm without performativity [Tan et al., 2018; Yan et al., 2019]. √

Remark 5. Throughout the time horizon T , Algorithm 1 requires a total of T + 2 T samples. Among them, √ T samples are dedicated to approximate the expectation over the base component Z 0 . Furthermore, each iteration requires an additional 2 samples to construct the online least-square objective, accumulating the remaining 2 T samples.

5 Numerical Experiments

This section verifies the efficacy of our algorithm and theoretical results by conducting numerical experiments on two examples: multi-task linear regression and multi-asset portfolio.

We first consider a multi-task linear regression problem in an undirected graph G := ( V , E ) , where V represents the node set and E represents the edge set. Each node i handles a linear regression task PR i ( θ i ) := E ( x i ,y i ) ∼D i ( θ i ) glyph[lscript] i ( θ i ; ( x i , y i )) , where θ i is the parameter vector and ( x i , y i ) is a feature-label pair. The loss function of each task is glyph[lscript] i ( θ i ; ( x i , y i )) = 1 2 ( y i -θ glyph[latticetop] i x i ) 2 , ∀ i ∈ V . The parameters of each connected node pair are subject to a proximity constraint ‖ θ i -θ j ‖ 2 2 ≤ b 2 ij , ∀ ( i, j ) ∈ E . The entire network aims to solve the following problem:

The second example considers the multi-asset portfolio described in Example 1. The simulation details are provided in § F of the supplementary file.

Wecompare the proposed adaptive primal-dual algorithm (abbreviated as APDA) with two approaches. The first approach is 'PD-PS', which stands for the primal-dual (PD) algorithm used to find the performative stable (PS) points. The algorithm PD-PS is similar to APDA, but it uses only the first term in Eq. (8) as the approximate gradient. The second approach is 'baseline', which runs the same procedures as APDA with perfect knowledge of A , i.e., the performative effect is known. We consider four performance metrics: (a) relative time-average regret Reg( t ) t · Reg(1) , (b) relative time-average constraint violation Vio i ( t ) t ·| Vio i (1) | , (c) decision deviation ‖ θ t -θ PO ‖ 2 2 , and (d) parameter estimation error ‖ ̂ A t -A ‖ 2 F .

Fig. 1 and Fig. 2 show the numerical results of the multi-task linear regression and the multi-asset portfolio, respectively. In both figures, we consider two settings for the sensitivity parameter of D ( θ ) , namely ε = 1 and ε = 10 . The results of these two figures are qualitatively analogous. First, we observe that APDA outperforms PD-PS significantly that both the relative time-average regret and the decision derivation of the former achieve an accuracy around or up to 10 -3 for the setting of T = 10 6 , while these of the latter have worse performance for ε = 1 and converge to constants for ε = 10 . The relative time-average constraint of all cases converges to zero or negative numbers. This corroborates the sublinearity of the regret and the constraint violations of APDA, as shown in Theorem 1. More importantly, this result implies that the larger the sensitivity parameter ε , the stronger the performative power is and, consequently, the worse PD-PS performs. In contrast, by tracking the performative gradient, APDA adapts to the unknown performative effect and performs well constantly. Moreover, both subfigures (d) show that the error of parameter estimates decreases sublinearly with iterations, validating the effectiveness of the online parameter estimation. Last but not least, the performance of APDA is close to the performance of the baseline, which manifests the effectiveness of our proposed APDA algorithm.

6 Conclusions

This paper has studied the performative prediction problem under inequality constraints, where the agnostic performative effect of decisions changes future data distributions. To find the performative optima for the problem, we have developed a robust primal-dual framework that admits inexact gradients up to an accuracy of O ( √ T ) , yet delivers the same O ( √ T ) regret and constraint violations as the stochastic primal-dual algorithm without performativity. Then, based on this framework, we have proposed an adaptive primal-dual algorithm for location family with effective gradient approximation method that meets the desired accuracy using only √ T +2 T samples. Numerical experiments have validated the effectiveness of our algorithm and theoretical results.

Acknowledgments and Disclosure of Funding

The work was supported by the National Natural Science Foundation of China Grant 62203373.

References

Hussein A Abdou and John Pointon. 2011. Credit scoring, statistical techniques and evaluation criteria: a review of the literature. Intelligent systems in accounting, finance and management 18, 2-3 (2011), 59-88.

Gavin Brown, Shlomi Hod, and Iden Kalemaj. 2022. Performative prediction in a stateful world. In International Conference on Artificial Intelligence and Statistics . PMLR, 6045-6061.

Xuanyu Cao and Tamer Ba¸ sar. 2020. Decentralized multi-agent stochastic optimization with pairwise constraints and quantized communications. IEEE Transactions on Signal Processing 68 (2020), 3296-3311.

Xuanyu Cao and Tamer Ba¸ sar. 2022. Distributed constrained online convex optimization over multiple access fading channels. IEEE Transactions on Signal Processing 70 (2022), 3468-3483.

Fabrizio Detassis, Michele Lombardi, and Michela Milano. 2021. Teaching the old dog new tricks: Supervised learning with constraints. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 35. 37423749.

Jinshuo Dong, Aaron Roth, Zachary Schutzman, Bo Waggoner, and Zhiwei Steven Wu. 2018. Strategic classification from revealed preferences. In Proceedings of the 2018 ACM Conference on Economics and Computation . 55-70.

Dmitriy Drusvyatskiy and Lin Xiao. 2022. Stochastic optimization with decision-dependent distributions. Mathematics of Operations Research (2022).

Hans Föllmer and Alexander Schied. 2002. Convex measures of risk and trading constraints. Finance and stochastics 6 (2002), 429-447.

Francisco García-Sánchez, Ricardo Colomo-Palacios, and Rafael Valencia-García. 2020. A social-semantic recommender system for advertisements. Information Processing & Management 57, 2 (2020), 102153.

Michael Grant and Stephen Boyd. 2014. CVX: Matlab software for disciplined convex programming, version 2.1.

Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. 2016. Strategic classification. In Proceedings of the 2016 ACM conference on innovations in theoretical computer science . 111-122.

Daniel P Heyman and Matthew J Sobel. 2004. Stochastic models in operations research: stochastic optimization . Vol. 2. Courier Corporation.

Zachary Izzo, Lexing Ying, and James Zou. 2021. How to learn when data reacts to your model: performative gradient descent. In International Conference on Machine Learning . PMLR, 4641-4650.

Meena Jagadeesan, Tijana Zrnic, and Celestine Mendler-Dünner. 2022. Regret minimization with performative feedback. In International Conference on Machine Learning . PMLR, 9760-9785.

Belhal Karimi, Blazej Miasojedow, Eric Moulines, and Hoi-To Wai. 2019. Non-asymptotic analysis of biased stochastic approximation scheme. In Conference on Learning Theory . PMLR, 1944-1974.

Susie Khamis. 2020. Branding diversity: New advertising and cultural strategies . Routledge.

Anton J Kleywegt, Alexander Shapiro, and Tito Homem-de Mello. 2002. The sample average approximation method for stochastic discrete optimization. SIAM Journal on optimization 12, 2 (2002), 479-502.

Qiang Li and Hoi-To Wai. 2022. State dependent performative prediction with stochastic approximation. In International Conference on Artificial Intelligence and Statistics . PMLR, 3164-3186.

Qiang Li, Chung-Yiu Yau, and Hoi To Wai. 2022. Multi-agent Performative Prediction with Greedy Deployment and Consensus Seeking Agents. In Advances in Neural Information Processing Systems .

Debmalya Mandal, Stelios Triantafyllou, and Goran Radanovic. 2023. Performative reinforcement learning. In International Conference on Machine Learning . PMLR, 23642-23680.

Celestine Mendler-Dünner, Juan Perdomo, Tijana Zrnic, and Moritz Hardt. 2020. Stochastic optimization for performative prediction. Advances in Neural Information Processing Systems 33 (2020), 4929-4939.

David Metz. 2021. Time constraints and travel behaviour. Transportation planning and technology 44, 1 (2021), 16-29.

John P Miller, Juan C Perdomo, and Tijana Zrnic. 2021. Outside the echo chamber: Optimizing the performative risk. In International Conference on Machine Learning . PMLR, 7710-7720.

Usue Mori, Alexander Mendiburu, Maite Álvarez, and Jose A Lozano. 2015. A review of travel time estimation and forecasting for advanced traveller information systems. Transportmetrica A: Transport Science 11, 2 (2015), 119-157.

Adhyyan Narang, Evan Faulkner, Dmitriy Drusvyatskiy, Maryam Fazel, and Lillian Ratliff. 2022. Learning in Stochastic Monotone Games with Decision-Dependent Data. In International Conference on Artificial Intelligence and Statistics . PMLR, 5891-5912.

Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. 2020. Performative prediction. In International Conference on Machine Learning . PMLR, 7599-7609.

Georgios Piliouras and Fang-Yi Yu. 2022. Multi-agent performative prediction: From global stability and optimality to chaos. arXiv preprint arXiv:2201.10483 (2022).

Warren B Powell. 2019. A unified framework for stochastic optimization. European Journal of Operational Research 275, 3 (2019), 795-821.

Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. 2008. Dataset shift in machine learning . Mit Press.

Mitas Ray, Lillian J Ratliff, Dmitriy Drusvyatskiy, and Maryam Fazel. 2022. Decision-dependent risk minimization in geometrically decaying dynamic environments. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 36. 8081-8088.

Luciano Serafini and Artur d'Avila Garcez. 2016. Logic tensor networks: Deep learning and logical reasoning from data and knowledge. arXiv preprint arXiv:1606.04422 (2016).

Conghui Tan, Tong Zhang, Shiqian Ma, and Ji Liu. 2018. Stochastic primal-dual method for empirical risk minimization with O (1) per-iteration complexity. Advances in Neural Information Processing Systems 31 (2018).

Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. 2019. Safe exploration for interactive machine learning. Advances in Neural Information Processing Systems 32 (2019).

Killian Wood, Gianluca Bianchin, and Emiliano Dall'Anese. 2021. Online projected gradient descent for stochastic optimization with decision-dependent distributions. IEEE Control Systems Letters 6 (2021), 1646-1651.

Killian Wood and Emiliano Dall'Anese. 2022a. Online Saddle Point Tracking with Decision-Dependent Data. arXiv preprint arXiv:2212.02693 (2022).

Killian Wood and Emiliano Dall'Anese. 2022b. Stochastic saddle point problems with decision-dependent distributions. arXiv preprint arXiv:2201.02313 (2022).

Yu Wu, Wei Wu, Can Xu, and Zhoujun Li. 2018. Knowledge enhanced hybrid neural network for text matching. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 32.

Yan Yan, Yi Xu, Qihang Lin, Lijun Zhang, and Tianbao Yang. 2019. Stochastic Primal-Dual Algorithms with Faster Convergence than O (1 / √ T ) for Problems without Bilinear Structure. arXiv preprint arXiv:1904.10112 (2019).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a new algorithm for performative prediction under general inequality constraints. The proposed algorithm has sublinear regret with respect to the performative optimum if the distribution map follows the location family model. The theory is supported by simulations.\n\nSTRENGTHS:\nThe motivation behind introducing constraints in performative prediction is strong and justified. The explicit regret guarantees are compelling. I thought Example 1 was well-motivated, clear, and a great example to keep in mind throughout. Section 3 is well-written and gives a clear step-by-step outline of the main method. The connection to existing work is thorough.\n\nWEAKNESSES:\nThe novelty has limitations. The main method combines existing ideas in the literature quite directly. The method for constructing a zeroth-order estimate of the gradient follows the same recipe as the two-stage method of Miller et al., and in a way this is the heart of the proposed method. This I see as the most relevant weakness. Below I'm including additional points.\n\nIt is not clear what the meaning of the time horizon T is in your analyses. Normally this should be the number of collected samples. But in your paper this is not the case, and I think that for this reason your T has a somewhat arbitrary meaning. I would rewrite the result statements to make T the total number of collected samples (I don't think this should change your rates). This is important especially because when you tune n you set it to grow with T as sqrt(T).\n\nAssumption 1 is just there to ensure convexity of the objective, right? If so, maybe it's cleaner to state that assumption directly, and give the conditions in Assumption 1 as one way to satisfy the convexity condition. Otherwise, the way things are stated right now, your Example 1 wouldn't be handled by the assumptions?\n\nIn Lemma 3 (and other results that rely on this lemma) I would probably write something like \"there exists a delta(\\eta, L_g) such that...\" Right now the interval for delta is too cumbersome.\n\nYou mention that derivative-free methods such as Flaxman et al. could be used for your problem. I think it would be a good idea to spell out exactly what they give you, because although you would lose dimension-dependent factors the results would be applicable much more broadly (all convex problems, not just location families). Location families are interesting but they are too specific.\n\nQUESTIONS:\nIn line 45, when you say \"exponential structure\" you mean exponential families?\n\nLine 148, denoted -> denote.\n\nI don't understand why you have both eta and delta. In problem (2) they appear together. I don't understand why they are tuned separately in the end? Why are they not treated as a single parameter?\n\nThe notation in Algorithm 1 is strange. What do you mean by Z_t \\sim \\D(Z_0 + A-hat_t theta_t)? Also, shouldn't it be the true A and not A_t-hat?\n\nPlease define PD-PS used in line 306.\n\nLine 320, sensitive -> sensitivity.\n\nLine 323 says accuracy decreases. You mean error decreases?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper attempts to study the problem of learning where the problem instances arise from a distribution. What is special is that this distribution is dependent on the decision-maker. The specific problem is when the distribution is given by a linear shift over a fixed distribution where the degree of shift is given by the decision parameters itself, and the decision parameters itself need to satisfy an inequality constraint. The authors propose a PD algorithm that  (under certain assumptions on the loss function) queries $T+2\\cdot\\sqrt{T}$ samples and violates $\\sqrt{T}$ constraints to yield a $O(\\sqrt{T})$ regret across $T$ rounds.\n\nSTRENGTHS:\nThe paper makes a good attempt at trying to look at the optimization problem with the inclusion of inequality constraints and it able to consider two simultaneous performance measures, namely, regret and (extent of) constraint violations.\n\nWEAKNESSES:\nThe paper promises a study in the direction of general optimization under (decision dependent) uncertainty but only tackles the problem in the limited case where the distribution is linear in the decision space. There is no discussion (or any insight) on how these techniques may generalize for more complicated distributions. The algorithm design itself does not appear to be very novel and uses fairly standard statistical estimation techniques.\n\nQUESTIONS:\nHow is Eq (4) (gradient of $PR(\\theta)$ wrt $\\theta$ ) on Pg 4 derived from Eq(1) (Definition of $PR(\\theta)$)?\n\nIs there any hope (or negative results) in the case where the distribution does not follow a linear behavior wrt to decision parameter ? I would expect the sample complexity to rise wrt to the complexity of the dependence b/w $\\theta$ and the distribution function $D(\\theta)$. \n\nThis body of work sounds similar to areas where the prediction problem is different from the optimization problem i.e. areas where the performance measure depends on both the unknown problem parameter as well as algorithm design (decision parameter), and the algorithm itself makes decision based on this caveat. \nExamples: \n1. Customizing ML predictions for online algorithms (Anand et al)\n2. Learning Predictions for Algorithms with Predictions (Khodak et al) \nCan you draw any parallels or is there no connection?\n\nLIMITATIONS:\nAuthors have addressed limitation adequately.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of performative prediction under inequality constraints and gives an no-regret learning algorithm that obtains $O(\\sqrt{T})$ regret and uses $\\sqrt{T} + 2T$ samples.\n\nIn the problem of performative prediction, the data distribution $Z\\sim D(\\theta)$ depends on the choice $\\theta$ of decision maker, and one formulates it as an optimization problem $\\min_{\\theta}E_{Z\\sim D(\\theta)}\\ell(\\theta, Z)$ and one only has sample access to the distribution $D(\\theta)$. The major difference from previous work is that the paper considers a constrained optimization problem, where $\\theta$ needs to satisfies certain constraints.\n\nThe paper proposes to use primal-dual gradient descent-ascent to solve the problem. Especially, the paper considers location families (where $Z \\sim Z_0 + A\\theta$ for some unknown matrix $A$) and provides convergence guarantee when the loss function is well-behaved (e.g. strongly convex and smooth).\n\nNumerical experiments are also conducted to verify the practicality of the proposed method.\n\nSTRENGTHS:\nThe performative prediction task is a well-studied problem and the constrained one (studied by this paper) is well-motivated. The primal-dual method proposed in this paper is practical and the author also provides regret guarantee under assumptions. The writing is clear and easy to read.\n\nWEAKNESSES:\nThe technical novelty is limited, the primal dual method (and its robustness to gradient error) is fairly common in the literature.\n\nQUESTIONS:\nThe theoretical result mainly focus on the cumulative regret, but as an optimization problem, it is also important to obtain a good decision at the end of $T$ rounds. Do you think taking the average of $\\theta_1,\\dots,\\theta_T$ would be result in a feasible decision with optimality guarantee?\n\nI have a few questions regarding the writing of the paper:\n\n(1)  Algorithm Line 4,6, it should be $A$ instead of $\\hat{A}_t$?\n\n(2) Assumption 3, what do you mean by saying the vector valued function $g(\\theta)$ is convex? \n\n(3) Line 266, the LHS should be $g_i$ instead of $g_m$, right?\n\nLIMITATIONS:\n.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper studies performative prediction under inequality constraints. In particular, the paper develops a robust primal-dual framework that requires only approximate gradients up to a certain accuracy but achieves the same order of performance as the stationary stochastic primal-dual algorithm even without performativity. Based on this framework, the authors propose an adaptive primal-dual algorithm for location families. The paper also presents the regret analysis and the perform numerical simulations to validate their findings.

STRENGTHS:
This paper studies and analyze the optimality of the performative prediction problem under inequality constraints, which is an important yet missing piece in the literature. As far as I know, this is the first paper attempts to provide a solution to this problem. The paper is well-written, easy to follow, and the proposed robust primal-dual method is intuitive. The theoretical analysis is throughout and it also provides empirical study to justify the findings.

WEAKNESSES:
The major weakness of the paper is the assumptions are very strong, and the setting is somewhat restricted. For example, the proposed algorithm only works for a particular family of distribution, namely the location family, and it requires an accurate estimation of all the parameters involved in the computation, which might not be realistic. However, it is also almost unavoidable for the performative prediction setting.

QUESTIONS:
1. Can the author be more explicit about the dependency on the sample required per iteration? In particular, for line 282, why does the initial sample $n\geq \sqrt{T}$ implies the sum of the expected gradient difference is bounded by $O(\sqrt{T})$?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of performative prediction under inequality constraints. The authors propose an adaptive, robust primal-dual algorithm that achieves sublinear regret and constraint violations, where the regret is with respect to the performative optima (instead of just a performative stable point). The algorithm is robust in the sense that it only requires approximate gradients up to a certain accuracy level and doesn't require the estimated gradients to be unbiased.\n\nSTRENGTHS:\n* The main paper is very clearly written. The remarks around lemmas and theorems are helpful in interpreting the formal results.\n\n* The proposed primal-dual algorithm is very nice in the sense that (i) the gradient estimators don't need to be unbiased; and (ii) the authors prove a regret bound in terms of the accumulated gradient approximation error. This can be a valuable contribution to the research community.\n\n* The theoretical results are supplemented by good experimental results on multi-task linear regression and multi-asset portfolio.\n\nWEAKNESSES:\n* The model assumes the learner can query the distribution for samples (lines 204 - 213). In particular, the learner can query the performative risk at $\\theta_t$ and $\\theta_t + u_t$. This is a strong assumption and should be stated more explicitly. What if the learner cannot query the distribution and only observes a single sample (or, say, $k$ samples) in each round depending on $\\theta_t$? This is more similar to the setting of Jagdeesan et al.\n\n* I acknowledge that Assumption 1 is standard in the literature on performative prediction, but I find the strong-convexity assumption quite strong.\n\n* There is no discussion on lower bounds for the sample complexity.\n\n* The bound seems to have a factor of $d$ (the dimension). It is unclear to me whether this can be improved and how this compares to existing results in the performative prediction literature and results on performative regret in the unconstrained setting. (See one my questions below.)\n\nQUESTIONS:\n* I really like example 1 (multi-asset portfolio). However, I have one question - can you elaborate on how \"excessive investment in a particular asset can lead to a declination of its rate of return\"?\n\n* Do you have thoughts on whether the sample complexity of $\\sqrt{T} + 2 T$ is tight?\n\n* Can you provide some intuition for why the third part of Assumption 5 is needed?\n\n* What is the dependence of the bounds on the dimension $d$? For example, for the case of Gaussian noise, $\\kappa_2 = d$ and $\\kappa_3 = 3d$ (line 250) and in Lemma 4 this can result in a factor as large as $d$ (through the definition of $\\bar{\\alpha}$. Can this dependence be improved? Do existing results in the performative prediction literature (and results on performative regret in the unconstrained setting) have a similar dependence?\n\n* (This is a comment, not a question.) It would help to include some proof sketches in the main paper. (I have not checked the proofs in the appendix.)\n\nLIMITATIONS:\nLimitations: Assumptions are clearly stated in Section 4. (The assumption that one can query the distribution could be stated more explicitly.)\nNegative societal impact: N/A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-20"}