Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes and formalizes a novel framing to understand a decision maker’s ability to affect the behavior of the population it is making decisions on. The paper relates this notion of “performative power” to classical notions in the economics literature: they draw parallels with the notions of “market power”, of being a price-taker vs a price-maker or a monopoly, and with Bertrand competitions in competitive settings.
strengths_and_weaknesses: - The main strength of the paper comes in the framing. Using a statistical measure like performative power abstracts away from market specifics. It is useful in the sense that modelling the specifics of a given problem often gives rise to complex problems with intractable equilibria, versus performative power can be measured from simply looking at the outcomes of said markets. 
- The notion of performative power is a measure of/proxy for a firm’s ability to affect its users. By extension, it provides some useful understanding of when it is vs is not possible to benefit from affecting the target population’s behavior.
- The parallels with economics are nice and useful to understand the value of the framework. I think the monopoly and the Bertrand competition are especially nice settings where the notion of performative power is “tight”: competition implies 0 performative power, meaning the firms have no influence on the populations and might as well solve the ex-ante problem, now equivalent to the ex-post one; for monopolies, the performative power is closely bounded by the surplus utility.


Weaknesses:
- Overall, the main theorem remains fairly weak. Theorem 1 seems almost immediate given the Lipschitzness assumption, and it only gives an upper bound on the difference between the performances of \theta_PO and \theta_SL. In turn, it says that with low performative power, the performance of \theta_PO is unlikely to me much better than \theta_SL, given that we have little power to steer the population hence the ex-ante and ex-post optimization do not differ by much anyway. It does not however say that high performative power guarantees significant differences in performance between \theta_PO and \theta_SL. 
- Related to the point above: bcause the notion of performative power looks at the supremum over possible actions/decision rules f deployed by the firm, higher performative power seems to only guarantee that the population can be steered in some, not all cases. In turn, it is not clear that performative power means that you can steer the population *in a way that is beneficial to the firm*. It would be nice to have more insights on situations in which perfomative power is “tight” in that sense. 

Overall, I think the approach and setting of the paper are valuable and practically useful. As such, I think this paper would be a good addition to the program and spark discussion. However, I think the results could be pushed further and it may be worth illustrating the power (pun intended) of the framework a bit more. 


questions: - Can the authors provide more examples of standard economic/learning settings with strategic agents and relate the performative power there to the actual ability to steer the population in a beneficial way? The performative power is defined to be best-case and it would be useful to understand practically when this best case is realistic.  
limitations: The limitations of the paper seem discussed. In particular, the second paragraph of Section 5 takes potential harms into account, and echoes some of the weaknesses discussed in this review (performative power measures *potential* harm or benefits from affecting population behavior, not necessarily *realized* harms/benefits.)
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors study the degree to which the deployment of predictors can influence, or cause changes in, a population interacting with said predictor.  This paradigm is framed through the lens of recommender systems and strategic classification in which a firm deploys a model $f$ which makes decisions on a population $U$, where each $u\in U$ has original data $z(u)$ and modified data $z_f(u)$ in the presence of model $f$. A firm's ability to shift a population is formalized in the proposed definition of Performative Power which measures the largest average deviation to agents' data that the firms choice of $f$ can have.  Several theoretical results are shown such as demonstrating how performative power changes with and without competition, as well as linking  performative power to the well established notion of performative prediction. 
strengths_and_weaknesses: ## Strengths

- This paper is well placed in the literature and lays a solid foundation for an interesting problem. The idea of performative power is an insightful extension of performative prediction and opens many directions for future work.

- The paper is extremely well written.

- The authors provide several strong theoretical results which serve to outline the types of questions/observations one may expect to see in the setting of performative power.

- Intuition and interpretation for each result is provided.

- Example 2.1 provides both a concrete way to understand the problem, as well as a highly topical and realistic setting which the authors work is directly applicable.

- Section 4 demonstrates that performative power is relevant  in several well studied fields/settings.

- Performative power is examined in both isolation (single firm) and competition (multiple firms).

## Weaknesses

- Proofs being deferred to the supplement can cause parts of the paper to feel incomplete. I would have liked to see at least a proof sketch or some intuitive reason as to why the results hold. Some results, such as Theorem 1 have an intuitive explanation preceding their formal statement, but I would still prefer to see proof sketches. 

- Similar to the comment of proofs being deferred to the supplement, I think the section on display discontinuity design could benefit from more details. My first thought when looking at Definition 1 (as pointed out on line 131) is that performative power is likely to be intractable, or impossible, to measure in any type of realistic setting. The idea of display discontinuity design lends credence to performative power not only as a worthwhile metric and analytical tool, but as something that may be feasible to examine in practice. With that said I understand that at a certain point things must be deferred to the supplement, and this paper covers a lot of material so I'm sure the choice of what to defer is not straightforward.

- There is a slight disconnect between the definition of Performative Power and some of the motivation/examples given by the authors. Specifically, in the context of a firm steering a population in a desired direction. The notion of performative power measures the worst case average shift that a firm can induce in a population. This shift is measured in terms of magnitude only. I understand that results such as Theorem 1 are trying to tie performative power to a firms ability to steer the data in a desired direction, but there should perhaps be some clarification that performative power does not necessarily, or directly, measure a firm's ability to steer a population in a desired direction. In the language of Adversarial Machine Learning, the current definition of performative power takes an "untargeted" perspective. Perhaps examining performative power from a "targeted" perspective is also worth investigating.
questions: - **Framing of performative power**: Throughout the paper performative power is framed as a potentially negative phenonium, meaning we may want to be wary of firms with large performative power. While this certainly seems true, there is also be a positive side to performative power which appears to be absent from the paper. For example, in the context of lending, agents can manipulate their features to appear more creditworthy to the firm, and through the lens of Strategic Classification, it is assumed that these are non-productive manipulations such as lying on applications (i.e. an agents true creditworthiness is not effected). However, some "manipulations" may be productive in the sense that not only do they cause the agent to appear more credit worthy, but they also cause the agent to actually be more creditworthy. For example, decreasing one's debt and closing excessively opened credit cards will have both effects. In a setting like this, where the alterations to an agents features have a causal relationship on their true label in a "positive" way, the performative power of a firm is desirable in the sense that the firm could help induce stronger financial literacy and true creditworthiness throughout the population.


- **Definition 1**: Performative power is defined via supremum over the firm's action space $\mathcal{F}$. While I understand the motivation for the supremum, I am curious how the results or interpretations change when performative power is defined in terms of an average over $\mathcal{F}$. Are we missing anything of value by using a supremum instead of an average?

- **Proposition 2**: How reliant is Proposition 2 on the fact that the classifiers from each firm are thresholds? It seems that when we move away from thresholds to more complex models, the idea of  threshold predictors can still somewhat apply in the sense that  $x$ can be thought of as the outcome of the classifier's score function $h$, and we are now thresholding on that score. However, in this setting the cost function should now depend on the underlying features, not on the model's scores, and each firm now has the option to choose both the score function and the associated threshold . It is unclear to me how to extend this result past the 1-D case. Can we say anything for more complex models?


- **Minor comments/issues**:

    - Line 102: It may help readability to specify the domain of $z(u)$. For example, does $z(u)$ contain only predictive features? Is there a label associated with $z(u)$. This is cleared up later in the paper, but referring to $z(u)$ simply as a data point may be too vague for some readers.

    - Line 102: Using $d$ instead of $\text{dist}$ may be more in line the notation of prior literature.



limitations: The authors clearly outline their limitations and are overtly transparent about the applicableness of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new measure of the power of a firm in a market, namely the performative power.  The idea is the firm's actions can affect the behavior of the participants of the market, causing them to deviate from the "natural" state without the presence of the firm.  And the performative power of the firm is the maximum average deviation it can cause by performing some action.  The authors show how this notion connects to a number of issues, including (1) whether a firm should focus on learning customers' preferences stastistically or changing their preferences to align better with the firm's goal, and (2) how much benefit the firm can achieve by optimizing the objective post the response, vs optimizing statistically without considering the effect of the response.  They also show how to bound the performative power in some concrete settings.
strengths_and_weaknesses: Strengths

The contributions of the paper are predominantly conceptual, so my evaluation is inevitably subjective.  I like the idea of measuring the power of a firm in a clean and well-defined way.  In particular, I think it's very nice that the definition doesn't rely too much on the specific form of the market (we still need some prior knowledge in order to properly choose the set of actions, the dist function, etc.).  The authors are able to confirm some natural phenomenon within the new framework and draw connections to other quantities of interest, which makes the new measure more credible.  They also show how to bound the measure (albeit in somewhat simple settings), which makes it more useful.


Weaknesses

Although the new measure definitely makes sense in some ways, it's not immediately clear to me it's the best way to measure the potential harm a firm can do to the market.  One potential criticism is that the measure is too pessimistic, and a rational and profit-driven firm wouldn't actually cause nearly as much harm.  Also, the authors present a number of applications of the new measure, but I'd be more excited if there's a "killer" application where the new measure helps solve a real problem that was previously beyond reach (it doesn't even have to be a technical one).  Also see detailed comments for some minor points.
questions: (Also putting some detailed comments here.)

Line 40, "a second way": any evidence that this actually happens?  And to what extent?  I know this might be hard to measure -- just curious.

Paragraphs between line 114 and line 130: I find part of the argument a bit shaky.  First the authors say that the actions that we should consider are those which "increase profit for the firm".  It seems this should really be interpreted as actions that "align with the objective of the firm".  But later, the authors argue that it's "not necessary to know the firm's objective function ..."  I wonder how the two claims should be reconciled.

Line 216: "let \phi by any ..."

Sec 4.2, \gamma vs \Delta \gamma: while this is conceptually interesting, I wonder how much technical difference there is when \gamma is replaced by \Delta \gamma.  If I understand it correctly, one can simply replace every appearance of \Delta \gamma in sec 4.2 with \gamma, and then we are back in the world where conceptually there isn't an outside option?  In that case I wouldn't overemphasize the effect of outside options.  Conceptually, the outside option formulation also implicitly assumes that a participant can always get utility \beta if they choose so, no matter what their features are.  This in a sense creates asymmetry between the "inside" option and the outside option (one has to "qualify" to get the inside utility, but can get the outside utility no matter what).  So perhaps the outside option is more like unemployment benefits...

Presentation in general: the paper is generally quite well written and polished, but a small thing that I think can be improved is the (somewhat unnecessary) use of jargon.  Some examples: I'd replace "performativity" with "best response" or something like that; in line 363, it's not clear what "standard microfoundations" are and why we need them if the reader is not familiar with "alternative microfoundations", and it seems totally fine to just say "Assumption 1".  Overall I think the presentation of the paper is heavily influenced by a particular line of work (which is of course fine), but maybe the paper would be more enjoyable for the general audience if presented in a more self-contained way.
limitations: Overall I'm satisfied (see detailed comments above for some minor suggestions).
review_title: null

Meta-review: metareview: The paper proposes the notion of "performative power" to measure a firm's ability to affect its users. This notion is insightful in performative prediction, and the authors have demonstrated it in multiple concrete settings. Overall, all reviewers are very positive about the paper. We recommend including the paper in the program and  believe it could open up potential future research directions. 
confidence: Certain
Decision: Accept