An engine not a camera: Measuring performative power of online search
Authors: Celestine Mendler-Dünner
Gabriele Carovano
Moritz Hardt
Venue: NeurIPS 2024 poster (2024)
Keywords: Performativity
Power
Digital Markets
Search Engine
Ranking
Online Experiment
Primary area: machine_learning_for_social_sciences
TL;DR: We design and conduct an online experiment to get quantitative insights into the ability of search engines to steer web traffic

Abstract: The power of digital platforms is at the center of major ongoing policy and regulatory efforts. To advance existing debates, we designed and executed an experiment to measure the performative power of online search providers. Instantiated in our setting, performative power quantifies the ability of a search engine to steer web traffic by rearranging results. To operationalize this definition we developed a browser extension that performs unassuming randomized experiments in the background. These randomized experiments emulate updates to the search algorithm and identify the causal effect of different content arrangements on clicks. Analyzing tens of thousands of clicks, we discuss what our robust quantitative findings say about the power of online search engines, using the Google Shopping antitrust investigation as a case study. More broadly, we envision our work to serve as a blueprint for how the recent definition of performative power can help integrate quantitative insights from online experiments with future investigations into the economic power of digital platforms.

An engine not a camera: Measuring performative power of online search
Celestine Wendler-Dünner""
Gabriele Carovano
Moritz Hardt
"ELLIS Institute Tübingen
Max-Planck Institute for Intelligent Systems, Tübingen and Tübingen Al Center Italian Competition Authority
Abstract
The power of digital platforms is at the center of major ongoing policy and regula- tory efforts. To advance existing debates, we designed and executed an experiment measure the performative power of online search providers. Instantiated in our setting, performative power quantifies the ability of a search engine to steer web traffic by rearranging results. To operationalize this definition developed a browser extension that performs unassuming randomized experiments the background. These randomized experiments emulate updates the search algo- rithm and identify the causal effect of different content arrangements on clicks. Analyzing tens of thousands of clicks, we discuss what our robust quantitative findings say about the power of online search engines, using the Google Shopping antitrust investigation as a case study. More broadly, we envision our work to serve as a blueprint for how the recent definition of performative power can help integrate quantitative insights from online experiments with future investigations into the economic power of digital platforms.
1 Introduction
At the heart of one of Europe's most prominent antitrust case seemingly mundane question: How much can search engine redirect traffic through content positioning? In 2017, the European Commission alleged that Google favored its own comparison shopping service by steering clicks away from search results towards Google's own product comparison service. The technical centerpiece of the case was an ad-hoc data analysis about the position and display biases of Google search results. Google appealed the European Commission's 's charges, pointing to, among other arguments, methodological errors
The case is emblematic of broader problem. Although urgently needed, there is currently no accepted technical framework for answering basic questions about the economic power of digital platforms. Lawyers, economists, and policy makers agree that traditional antitrust tools struggle with multi-sided platforms [ Against this backdrop, a recently developed concept from the machine learning literature, called performative power [3), suggests way augment existing antitrust enforcement tools and mitigate some of their limitations. Performative power measures how much platform causally influence platform power to a causal effect, sidesteps the complexities underlying conventional market definitions and offers a promising framework integrate data and experimental methods with digital market investigations. Although the definition of performative power enjoys appealing theoretical properties, a proof of its practical applicability was still missing.
through its algorithmic actions. By directly relating
Cass C-48/22 P. Google and Alphabet Commission (Google Shopping). 1121:1
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

If Effect of Swap 1-2
41 Effect Swap 1-3
32%-
33%-
30%
d
20%
10%
10%
10%
Figure The ability to influence web traffic through content arrangement. Blue bars show average click probability observed for generic search results in position to on Google search under different counterfactual arrangements; default arrangement (left), swapping results and 2 (middle), swapping results 1 and 3 (right). We provide detailed discussion in Scction 5 where we also explore arrangement changes beyond reranking.
Our contributions. We present first proof of concept showing how to use performative power as an investigative tool in practice. The instantiation of performative power we consider is motivated by the recent Google Shopping antitrust investigation ran by the European Commission against Alphabet Inc. It concerns the ability of a search engine to impact web traffic through decisions of how to arrange content.
Our core contribution is to design and implement an online experiment to establish lower bound on performative power for the two most widely used search engines, Google Search and Bing, by providing quantitative insights into the causal effect of algorithmic updates on clicks. Our experiment based on a browser extension, called Powermeter, that emulates updates to the platform's 's algorithm by modifying how search results are displayed to users. The arrangement to which a user is exposed chosen at random every time they perform search. We implement different counterfactual arrangements to inspect the effect of re-ranking and favored positioning (c.g., Ads or Shopping boxes) on clicks, both in isolation and jointly. We discuss several technical steps we implemented to take care of the internal validity of our experimental design.
Using Powermeter we collected data of about 57,000 search queries from more than 80 different subjects, over the period of months. Our experiment is designed to measure the causal effects of arrangement under natural interactions of users with the platform and the queries for any given user follow the distribution of queries under their every-day use of online search. Figure i first glimpse into the observed effects. In summary, element by one position causes an average reduction in clicks of 42% for the respective element on Google search. Down-ranking the same element by two positions yields a reduction of more than 50%. For Bing we find an even larger effect of ranking, although with less tight confidence intervals due to the small number of Bing queries performed by our participants. When combining down-ranking with the addition of Ads or Shopping boxes, the effect of arrangement is even more pronounced, showing a distortion clicks for the first result of 66% averaged across queries where such elements are naturally present on Google search. Inspecting different subsets of queries we find that the effect of position is larger for queries with a high number of candidate search results. To the best of our knowledge, we are the first to offer independent quantitative experimental insights into display effects on Google search and Bing.
provides a find that consistently down-ranking the first
Finally, we outline how to formally relate our quantitative piece of evidence to questions about self- preferencing relevant in the context of the Google Shopping case. Together, we hope our empirical and theoretical results can serve as a first blueprint for what future antitrust investigations of digital platforms' market power based on performative power might look like.
2 Preliminaries and related work
The market power of digital platforms is the subject of robust debate in policy, legal, and technical circles. See, for example, Newman 44. Crémer et al. [2] Stigler Committee [1. Furman [5). Cabral et al. [6]. Conventional antitrust enforcement tools have been put into question [7] and new concepts of market power have been proposed to deal with the complexities of digital markets [8) These account for the multi-sided nature of the markets of consumers albeit with limited success. We refer to a comprehensive literature survey about behavioral aspects in online market by the UK Competition and Market Authority [9]
well as the role of behavioral weaknesses
2

---

Performative power. The concept of performative power is inspired by recent developments in performative prediction [T] from the computer science literature. We refer the reader Hardt and Mendler-Dünnel [1]] for a recent survey on the topic. A robust insight from performative prediction that beyond learning patterns in data, the ability to steer the data-generating distribution similarly factors into a predictive system's s performance. Performative power 3 recognizes that the ability to steer depends on power-in in terms of reach and scalc-of the platform making the predictions. Thus, the core idea behind the new notion of power is to measure the degree to which predictions are performative to obtain an estimate of the power of a platform. Formally, performative power relates the ability of platform to steer the population of participants, to the causal effect of algorithmic actions.
Definition 1 (Performative power [3. Given the algorithmic action ag and a set of alternative conducts A, population Q and an outcome variable Performative power is defined
PP sup aEA
E 1100 (4) za)1)l1
qE@
where (g)(g) (g) denotes the outcome for unit Q under do and =a(4) denotes the counterfactual outcome, would the platform implement a E A instead. Expectations are taken with respect to the randomness in the potential outcome.
Performative power a measure of influence that predictive systems can have over their participants. It offers family of definitions that can be instantiated flexibly in a given context. The specific meaning determined by each instantiation. Performative power can be applied boroaid-looking to understand whether platform has the ability to plausibly cause a specific change, as well as in retrospect to measure the effect of an observed conduct. In this work we use performative power to quantify the effect of an algorithmic update a" central to a recent antitrust investigation against Alphabet Inc. ran by the European Commission implementable counterfactuals such as to provide plausible lower bound on the effect of a'.
? In practical terms, we instantiate with a set of conservative and
The Google Shopping case. In 2017 the European Commission imposed fine of 2.42 billion EUR
on Alphabet Inc. for "abusing its dominance as a search engine by favouring its comparison shopping service." The General Court dismissed Google's s action against the decision in 2021 and the Court of Justice of the European Union upheld the Court's ruling in 2024. It represents a landmark in EU competition law. The conduct under investigation concerned a specific update to the Google search algorithm. The update a) demoted rival comparison shopping services among the general search results, often by multiple positions, and, at the same time, b) systematically gave prominent placement to Google's 's own comparison shopping service by triggering visually appealing boxes for shopping queries, reserved for Google's 's own service. The goal of this work is to provide quantitative insights into the effect of this conduct web traffic by means of online experiments.
Display effects.
Consumer choices on digital platforms are critically mediated by how plat- forms present content to users. Choice architecture designs [12), presentation bias [13], position bias 14 15 and trust bias [16] are known to play an important role. There is rich literature in machine learning aiming to mechanistically understand such biases for debiasing click data 17:24, building better ranking models and auctions 225 26). and interpreting user feedback in recommender systems [1], to name few. Unfortunately behavioral aspects often resist clean mathematical speci- fication. By focusing on measuring a dircctly observable statistic, performative power circumvents the challenges of modeling behavioral biases for monitoring, auditing and measuring digital economies.
Measuring the effect of algorithmic updates. Several works have been interested in measuring the effects of potential arrangement changes of online platforms. For example, Ursu (27 rely on public data collected under randomized result ordering to investigate the role of positioning on Expedia. Narayanan and Kalyanam 28 investigated position bias in search advertising using regression
discontinuity design. Also focusing on online advertising, Agarwal et al. [29] investigate position bias by experimentally randomizing bids to indirectly influence the ranking. Similarly in information retrieval researchers have studied active interventions in the form of order randomization (30), or relied harvesting click data collected under multiple historical rankings [II). In our work we collect experimental data ourselves. We use a browser extension to emulate the algorithmic updates of interest without requiring control over the platform's 's algorithm.
European Commission, AT.39740, Google Search (Shopping). 27. 77.06.2.17
3

---

Browser extensions have previously been used as a tool for automatically collecting data to audit systems. Robertson al. [32] audit Google search for polarization on politically-related searches. Gleason et al. [3 collect data via an extension to directly investigate the effect of search result components on clicks. Also the ongoing National Internet Observatory (34) relies on a browser extension to collect web traffic data. While prior works focus on collecting observational data for monitoring systems, we use the extension to conduct online experiments.
3 Performativity in online search
We start by formalizing the causal question under investigation. We model online search platform as a distribution over events. An event is a triplet of user query Q, content arrangement and click outcome C. A user query corresponds to person visiting the search page and entering search query in the search bar. The query is processed by the platform and results in an arrangement of content the website. The mapping typically defined by a proprietary pipeline involving a ranking algorithm that determines the order in which search results are ranked and displayed, including the positioning of components such as Ads or featured elements. Then, mediated by the arrangement, the user query leads to click outcome C. The categorical random variable C' indexes the element clicked over by the user. is a function of the user query and the arrangement.
3.1 From the causal effect of arrangement to performative power
Assume the platforms were to change their algorithm that determines the content arrangement. We seek to answer the following causal questions: How much does a change the arrangement impact clicks of a content element on the search page?
If clicks were solely determined by stable preferences. then we would no effect. Performativity is the reason why we see an effect. Display baises, the limited ability to process large amounts data, and trust in the platform can be a source for performativity. The more performative the arrangement is, the stronger the effect. We use dg to refer to the reference arrangement of results on Google search. For given user query q we define the potential outcome of click event under the arrangement a as Ca(a). The variable C takes on categorical values, indexing the elements on the page. Let (cc. } denote the top K general search results indexed in the order they appear under ag- Definition 2 (Performativity gap). Given counterfactual arrangement a', we define the performativ- ity gup at position i with respect to a population of queries Q as
8 (aa)) E [1(C,(a') = ct] E (1C)( = =c]]
where expectations are taken over queries Q and the randomness in the potential outcome.
The performativity gap quantifies how much the click through rate of search item e; changed, in expectation across queries Q, had the platform deployed arrangement a' instead of arrangement ag- The following result generalizes Theorem 8 in Hardt et al. [3:
Theorem (Lowerbound on performative power). Let PP be the performative power of a search platform defined with respect to set of arrangements A, population of search queries Q performed on the platform, and the outcome variable za (g) 1(C,(a) c1 Then, performative power is lower bounded by the performativity gap as PP 2 suPaE.4 8 (a).
Note that the instantiation of performative power in Theorem to which we relate the performativity gap measures a platform's 's ability to steer outgoing traffic from its online search website. We will discuss how to relate this notion to broader discussion of the power of online search in vertically integrated markets in Section
Algorithmic distortion. Often it can be useful to express the performativity gap relative to the base click rate. Thus, we define the algorithmic distortion factor as the smallest factor 3 > such that
8' (a') BE (o)) = =.]]
This quantity serves as way to denote the fraction of clicks taken away from content item as result of the update ag - a'. Also, as we will see, helps to express performative power relative to a base click through rate which offers a more interpretable quantity for investigators.

---

3.2 Estimating the performativity gap using an RCT
To estimate the performativity gap for the different arrangements, we rely on randomized controlled trial (RCT), the gold standard methodology to estimate causal effects [3]] As we can not observe a search query simultaneously exposed to different arrangements, the idea of an RCT is to randomly select, for each query q E 2, the arrangement they are exposed to. We write Qa C 2 for the subset of queries that are exposed to treatment A a. We also refer to these subsets as treatment groups. Comparing the click events across groups allows us to obtain an estimate of the performatiiity gap as
8' (a') = (a') - (a)) with CTR'(a) = Qu
Σ M(C((a) = =...
qEQu
For (a') to provide an unbiased estimate of (a we rely on an application of the stable unit treatment value assumption (SUTVA) (38), referred to as isolation assumption by Bottou et al. [3 Assumption (Independence across queries). User behavior in response to query q is not affected by the treatment status of other queries, i.e., for all q Q we have (A,) IL Ag Vq' + q where A, denotes the random variable assigning query & to a treatment group.
This assumption justifies why we can interleave the measurement of different interventions. It requires that the intervention performed on one query does not change individuals' browsing behavior in response to subsequent queries, Crucially, this can only be satisfied, if individual interventions under investigation do not impede user experience in lasting manner. In the following section we discuss steps we take in our experimental design towards justifying Assumption T
More broadly, the key advantage of using an experimental approach to measure the performativity gap is that, while the mechanism mapping user queries to clicks can be arbitrarily complex, this complexity does not affect the experiment. Aspects such users' preference for clicking links the left side of the screen [40]. the effect of visually appealing elements [41), users' trust in the platform [16,, the relevance gap between search results will naturally enter our measurement.
4 Powermeter: Experiment in the wild
We designed an online experiment to measure the performative power of two popular online search platforms operated by Google and Bing. The experiment is built around a Chrome browser extension that modifies the arrangement of search result pages and records user click statistics in a privacy- preserving fashion. The extension allows to observe organic set of user queries and click outcomes under different arrangements without having control over the platforms' algorithm.
4.1 Browser extension
Browser extensions can add functionalities to the web-browser and change how a website is displayed to the user. Powermeter makes use of these functionalities emulate algorithmic updates by implementing different counterfactual arrangements on Google search and Bing search. We emphasize that Powermeter only hides or reorders, but never modifies or adds any content on the search page.
Technical details. Once activated, the extension triggers the experiment whenever the user enters a eeepeee search query on either Google search Bing search. This can be identified by monitoring the url string of the current tab. Before search results are loaded the extension immediately hides the content of the website, inspects the html document, randomly assigns the user to one of the experimental groups and then implements the respective changes to the website before making the page visible. The implementation of the counterfactuals done by identifying the relevant items hide or swap by their html class names or ids. We also add custom tags and event listeners to the identified elements that we can fall back on at a later stage. The entire setup of the experiment usually takes around 40 Hiding the html body of the website with the first possible Chrome event is crucial to avoid glitches in case of bad internet connection and make sure the control arrangement is not revealed to the participant. To ensure internal validity of our experiment, we also have to ensure a participant is never reassigned to a new experimental group when reloading page, navigating between tabs or repeatedly entering the same search query. This is done by storing a hash of user ID and search query together with the assigned experimental group in the browser cache.
milliseconds. This delay is far below what was found to be noticeable to users 42 43
5

---

Google
Google
Generic search result
Tedm??
(Shopping) Box
Generic search result
Do Retarll pudtion
Peop eto
Specialized search result
Deepingg
20 Ooeeran
Ad
Mennast Google Shepping derton
Nouss FC Earce Ticc Dae Rabiia Wotsle
Soogle (Chepping
Figure 2: Illustration of different elements on the Google search website.
Table 1: Counterfactual content arrangements implemented by Powermeter as part of the RCT.
Arrangement
Description
44g
control
Search results are displayed without any modification.
swap 1-2
The position of the first and the second generic search result are swapped.
02
swap 1-3
The position of the first and the third generic search result swapped.
03
swap 2-3
The position of second and the third generic search result are swapped.
04
hide Ads/Box
Top Ads and Shopping boxes are hidden.
05
hide + swap
Combines the latter modification (a4) with swap 1-2 (a1).
46
hide Box
The shopping boxes are hidden.
Backend and data collection. Every participant assigned a unique random number that serves as anonymous user ID) upon installation of the extension. This user II) persists throughout the experiment. Every time a click event on an element on the search page registered, the click data aggregated into json object and sent to a database server hosted locally at out institution via post request using the encrypted https protocol. This concerns information about the index of the clicked search result, the click element type, the page index, and the experimental group. In addition, statistics about the website such as the number of search results, the presence of ads and boxes, the number of candidate results, and the position of specialized search results extracted from the website are recorded. The database server is built using the Microsoft .Net core framework and deployed within docker container. The database access is rate limited and the Get endpoint of the database key protected. We use a SQlite database that is mapped to persistent memory.
Privacy considerations. The information that is stored with every click does not contain any personally identifiable information. While we record the position of the clicked element on the search page, store search queries any information about the websites visited by the user. This an intentional choice to preserve user privacy, and to demonstrate that valuable insights can be gathered without privacy invasive data collection. The experiment went through an internal approval procedure and the privacy policy can be found on our website
4.2 Experimental groups
We implement six different counterfactual arrangements, summarized in Table T each defining treatment group. We refer to Figure[]oor the terminology used to refer to individual elements on the general search page. It equivalently applies both, Google search and Bing. Arrangements a as are designed to emulate conservative variants of the conduct a" of interest, to inform a plausible lower bound on performative power. The first three arrangements a1-a3 concern the reordering of organic search results, leaving the other elements on the website untouched. The arrangements
Thee project website can be found at: httgs:/powwemmerrriissue.mmggg.

---

ag and ag perform modifications not directly concerning organic search results: Arrangement ds hides a specific element, called the Shopping box, appearing either in the right side panel or on top of the search results page. Arrangement 04 hides the box together with all the Ads. Finally, Arrangement as combines the latter change with a change in search result order. For Bing we only implemented the counterfactual not to implement larger modifications is also users' sensitivity to the resulting deterioration of quality towards ensuring Assumption [] The Bing experiment of the European Commission's 's investigation had to be discontinued after one week for that exact reason 4 We made sure to avoid a similar failure point. Based on user feedback collected during initial test round there was indication that the modifications were even noticeable to users.
to ensure statistical power despite data scarcity. A practical reason
4.3 Onboarding
Participants were provided the link to the project website as entry point. The website contains information about the experiment, the purpose of the study, an onboarding video, as well as the privacy policy of the extension. The extension itself is distributed through the official Chrome webstore and there is a button directly navigating the user to the item in the store. We did not list the extension publicly ensure participants informed about the purpose of the study, and protect the integrity of our data. The installation follows the standard procedure of adding a browser extension to Chrome. The user has to give consent access Google and Bing websites, the storage API. The extension remains active until participants remove it from their web-browser, or until the experiment is stopped. The study participants are trusted individuals of different age groups and backgrounds, recruited by reaching out personally or via email. We provide demographic statistics over our pool of participants in Figure
well as to use
I in the appendix.
Data preprocessing. For each participant we ignored the clicks collected during the first four days after onboarding, as suggested by Keusch et al. 44 in order to avoid potential confounding due to participation awareness. We also removed clicks where the search elements could not be identified reliably for implementing the RCT to avoid selection bias towards the control group.
Empirical results
Using our Powermeter browser extension we collected click data from 85 participants over the course of 5 months, from September 2023 until January 2024. This resulted in 56, 971 click events, and total of 45, 625 clicks after preprocessing. Out of the clicks 98.9% were registered on Google, and 1.1% on Bing. Figure g in the appendix visualizes some aggregate statistics over the clicks collected. We will consider several subsets of these events for which we measure the performativity gap and algorithmic distortion. In the following we discuss the main insights from the collected data. For all plots, we provide bootstrap confidence bounds over 200 resamples.
5.1 Reordering search results
We first inspect the three counterfactual arrangements a1, a22 ag concerning reranking. Recall that ca indexes search results in the order in which they appears under ap. In Figure we visualize the event probabilities C' c for each search result 1, 2, .... 6 under the control group (blue bars) and compare it to the respective probabilities under the three counterfactuals (orange hars). The figures on the left show the results across Google search queries. Here the counterfactuals correspond to swapping the position of the first two results (left), the first and the third (middle) and the second Bing search queries when the
and the third (right). The right figure shows the results evaluated first two results are swapped. The lower figure visualizes the corresponding performativity gap (a)), corresponding to the change in clicks item caused by the respective arrangement change.
We observe a consistently large effect of arrangement on clicks. Being down ranked by one position on Google search decreases average click through rate of from 43% to 24%, resulting in 8 (a1) 19.19 and an algorithmic distortion of 3 = 0.44. Being down-ranked by 2 positions results in 5' (az) 0.27 and average loss of more than 50% of traffic. Note that similar effect size has been reported in the case decision for the UK market for two position shift [5] para 460], indicating
"See Case T-612/17, supra 2, para 399
7

---

queries
63-
64-
62-
Starch
Figure 3: Click through rate and performativity gap for general search results c2 to Cs under the counterfactual arrangements a1, d2, as for Google and the counterfactual arrangement a1 for Bing, compared to the control arrangement ag (in blue).
that the estimate is robust across different user populations. Finally, by down- the second content element by one position we still observe significant traffic reduction, corresponding to an algorithmic distortion of B 0.39.
An interesting observation is also that for every counterfactual arrangement, the element shown first ends up getting most clicks on average. Implying that all the rankings are close to performatively stable [10] with respect to the non-personalized reranking strategies considered. However, there are several indications of Google's ranking ag reflecting relevance of search results better then the other arrangements. Namely, e₁ gets more clicks when displayed first, compared to other results displayed in the same position (c2 corresponds first result under 02 and C3 the first result under ag). A second indicator is that 0o benefits from arrangement ag: under the hypothesis that users consider search results in order this indicates that content item C3 absorbs less clicks than C1.
For Bing the position effect seems be even more pronounced, although confidence intervals are significantly larger in this case. We conjecture that this could be due to the average number of specialized search results and Ads present on the search page being larger on Bing. This results in a larger spacial separation of search results and potentially larger display effects. Statistics about the type of elements present on the the search pages are reported in Figure 8 in the Appendix.
5.2 Indirect effect of visually appealing elements
Next, we consider the counterfactuals a4 and ag that leave generic search results untouched and hide certain elements on the website. We first inspect the number of clicks these elements absorb if present on the page. We focus on Google search. Figure we compare the fraction of clicks going to generic search results, Ads, and Boxes for Q4. ag. We plot the statistics across the aggregate queries (left panel), the subset of queries where Ads are present on the page under ag (middle panel) and the subset of queries where the box is present on the page under (right panel). We find the addition of boxes absorbs 22.4% of the clicks on average across queries where present under do and these clicks are mostly taken away form the generic search results. Similarly, Ads absorb close to 30% of the clicks on average for queries where they are present. However, considering the overall number of clicks the effect is smaller since a large fraction of queries contains neither Ads nor Boxes.
Combined conduct. We now consider the combined conduct of adding the box and down-ranking an element. We constrain our focus on queries where the Shopping box is present under dg. either in the center column or in the right sidebar. These are 3.2% of all the events. We show the corresponding click probabilities for the three first search results Figure [5] In both figures the blue bars correspond to the control group ag, and the red bars correspond to a1 For these groups boxes are present on the page. the left figure we investigate the effect of hiding boxes only and the orange bars correspond to arrangement a, In the right figure we investigate the effect of also hiding Ads, here the orange bars correspond to ag. We find that when adding Boxes, all three content items loose a significant fraction of clicks, whereas Ads mostly take away from C₁- In the right figure we additionally show as using the hatched bars (i.e., down-ranking the first element by one position box is hidden). For ei the combined effect of adding Ads and Boxes the click through rate is almost as large the effect
8

---

Google queries
Google queries with Ads
Google queries with Box
0.8
control
0.8
control
0.8
control
0.6
clekk grobatsiiit
grobabalte
0.6
0.4
0.2
0.0
0.0
0.0
Ads type
scarch
type
type
Figure 4: Effect of arrangement on the click distribution across different element types (generic search results,
Ads, boxes), visualized for three different subsets of Google queries.
Google cueries
0.5
Conduct
Distortion of c1
0.3
0.3
add Box
8 0.23
add Ads/Box
0.44
swap+add Box
3 = 0.53
0.1
swap+add Ads/Box
8 = 0.66
CRarch TIIAET
SHarch TINAFt
Figure 5: Effect of hiding box and swapping elements on the click probability of generic search results. Statistics are evaluated for the subset of queries for which box is naturally present. The hashed bar shows the click prohability under as when top content is hidden and the first two elements swapped. The right table reports the empirical measure algorithmic distortion for different conducts, extracted form the results the left figure.
of down ranking the same item by one position. What we can see consistently is that combining the conduct of adding visually appealing elements on top of the page, and down-ranking content item, has combined effect that is larger than the effect of the two individual modifications alone. For element the measured distortion is reported in the table. The combined effect leads to a reduction of 25% in clicks and an algorithmic distortion of 66% when considering the effect of Ads/Boxes, and 53% when only considering Boxes (comparing orange and red bar). We believe this to be the first time that quantitative insights into this combined conduct are made public.
5.3 Factors that impact performative power
We perform additional investigations into what factors have reinforcing effect on the performativity of ranking. To this end, we inspect different subsets of Google queries and measure the performativity gap as well as algorithmic distortion for e1 under the counterfactual 01. First, we split the data across two different axes depending on whether Ads or boxes are present, and whether Specialized Search results (SSR) are present in between the first two generic search results. The respective comparisons are visualized in the left and middle panel of Figure 6 We observe that the performativity gap in the presence of Ads and boxes is smaller, and about the same whether special search results are present in between search results. However, if we plot distortion we get different picture, since the base click probability content item ej across different splits is different for the three cases. We find that while Ads and Boxes have little effect on distortion, the presence of specialized search result in between the swapped results tends to increase the effect of down-ranking C1.
For the second investigation we group the queries by the number of candidate search results available on Google. This number was extracted from the website where it appeared as string on top of the page in the form 'About 323 000 000 results (0.65 second) >. The right panel of Figure 6 shows algorithmic distortion for each percentile of the data. We see clear trend that the performativity gap increases with the number of candidate results. We suspect this to be connected to the smaller relevance gap across results for queries with more potential results, leading to higher influence of the arrangement clicks. However, note that findings in this subsection no causal claims.

---

Google queries
Google queries
Google querles no Ads/Box
1.0
1.0
0.0
0.8
0.0
T
0.4
0.4
11
yeu
0.0
0.0
05-100
Advllie 2re
SGA present?
AdhiBix present?
present)
numberol candicate mexutt (percent
Figure 6: Performativity gap and algorithmic distortion for content item C1 under the counterfactual arrangement di measure across different subsets of Google search queries.
Discussion
We presented a flexible experimental design, based on a browser extension, to investigate the effect of search algorithms on user clicks. The browser extension performs interventions at the level of display to emulate algorithmic updates, without access to the platform algorithm. We implement different counterfactuals relevant for the Google Shopping antitrust investigation, and provide quantitative in- sights into the causal effect clicks. Theorem formally relates quantitative findings to an instan- tiation of performative power, measuring the platform's 's ability to steer outgoing traffic from search.
In final step, we describe how our findings could fit with broader anti-trust investigation potentially concerning effects spanning different markets. Take the Google Shopping case as an example. is concerned with the ability of Google search to distort incoming traffic to a business operating in the market of comparison shopping services by changing where it appears on Google search relative to its competitors. Establishing the causal link between arrangements on search and their effect on incoming traffic to a third party website composes into two steps: a) establish Google's s ability to steer outgoing traffic, b) quantify how much of the incoming traffic is mediated by Google search. The first step is a notion of power that experiments like ours operationalize, the second is a number that can readily be obtained from web traffic data. The overall performative power will be the product of the two factors. For putting this together, let's 's work though the following thought experiment:
Suppose, 80% of the referrals to the competitor's website come from Google Search Further, assume that 70% of the referrals from Google happen while the service ranked among the top two generic search results. Our estimates suggest that distortion of traffic at the first position can be as large 66% for small urrangement changes. Assuming for the second position the effect is 20% smaller, giving conservative average effect size of 3 = 0.8 0.66. Multiplying the effect size by the fraction of incoming clicks it concerns, we get 0.8 0.7 B = 30% This is the fraction of traffic to the site Google redirect.
Turning this number into conservative lower bound on performative power, it offers an interpretable measure of power for an investigator to judge whether the algorithmic lever of self- eeff-preferencing through arrangement changes should be a concern for competition in the down-stream market or not. We can use the same logic to compare search engines, and assess the effectiveness of remedies.
More broadly, we hope our work can serve as blue print for how performative power can be used to integrate experiments with future digital market investigations, and how tools from computer science, causality, and performative prediction can inform ongoing legal debates related to the power of digital platforms. Our work situated within growing scholarship [c.f., 48) that takes advantage of the accessibility of digital markets for monitoring and regulating digital platforms. Beyond data, we demonstrate how experimental methods can offer an additional tool for power assessments.
From the perspective of computer science our work offers measurements of performativity in the context of online search, contributing quantitative and empirical support to the study of performativity on digital platforms. As its name suggests a search engine is performative, acts as an engine steering consumption through its ranking algorithm, rather than camera merely picturing candidate results results-we we borrow this analogy from MacKenzie 44
In 2022 up to 82% of incoming traffic to comparison shopping services in the European Economic Area mediated by Google search, reported in the Google Shopping case decision (Section 7.2.4.1, Table 24).
10

---

Acknowledgements
We are particularly grateful everyone who installed the extension and participated in our study; without you such project would not have been feasible. Further, the authors would like to thank Jonathan Williams for the design of the project website, Alejandro Posada for producing the on- boarding video and helping with the logo, Telintor Ntounis for assistance in setting up the server infrastructure, and Ana-Andreea Stoica, André Cruz and Jiduan Wu for feedback on the manuseript. We would also like to thank two anonymous reviewers at NeurIPS who provided valuable feedback related the presentation and framing of our work. Celestine Mendler-Dünne, acknowledges financial support of the Hector Foundation. Opinions expressed in the paper do not necessarily reflect the opinions the AGCM.
References
[1] Stigler Committee. Final report: Stigler committee on digital platforms, September 2019.
[2] Jacques Crémer, the digital era: Final report. Publications Office of the European Union, 2019.
de Montjoye, and Heike Schweitzer. Competition Policy for
[3] Moritz Hardt, Meena Jagadeesan, and Celestine Mendler-Dünner. Performative Power. In Advances in Neural Information Processing Systems, pages 22969-22981, 2022.
[4] Nathan Newman. Search, antitrust, and the economics of the control of user data. Yale Journal on Regulation, 31:401, 2014.
[5] Jason Furman. Unlocking digital competition, report of the digital competition expert panel, 2019. https://doi.oon org. 100 17639 waops//wjssjccll
[6] Cabral, J Haucap, G Parker, G Petropoulos, T Valletti, and M Van Alstyne. The EU digital markets act. Technical Report KJ-02-21-116-EMN (online), 2021.
[7] Chad Syverson. Macroeconomics and market power: Context, implications, and open questions. Journal of Economic Perspectives, 33(3):23-43 August 2019.
[8] OECD. The evolving concept market power in the digital economy, 2022. OECD Competition Policy Roundtable Background Note.
[9] CMA. Online search: Consumer and firm behavior review of the existing literature, 2017.
[10] Juan C. Perdomo, Tijana Zrnie, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In International Conference on Machine Learning, pages 7599-7609, 2020.
[11] Moritz Hardt and Celestine Mendler-Dünneer Performative prediction: Past and future. ArXiv preprint arXiv:23 16608, 2023.
[12] R.H. Thaler and C.R. Sunstein. Nudge: Improving Decisions about Health, Wealth, and Happiness. Yale University Press, 2008
[13] Yisong Yue, Rajan Patel, and Hein Roehrig. Beyond position bias: examining result attractive- ness as source of presentation bias in clickthrough data. In International Conference on World Wide Web, page 101 10118, 2010.
[14] Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. Accurately interpreting clickthrough data as implicit feedback. In ACM SIGIR Conference on Research and Development in Information Retrieval, page 154-161, 2005.
[15) Zhiwei Guan and Edward Cutrell. An eye tracking study of the effect of target rank on web search. In SIGCHI Conference on Human Factors in Computing Systems, page 417-420. Association for Computing Machinery, 2007.
[16] Bing Pan, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, and Laura Granka. In Google We Trust: Users' Decisions on Rank, Position, and Relevance. Journal of Computer- Mediated Communication, 12(3):801-823, 2007.

---

[17] Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. An experimental comparison of click position-bias biation---iassaae models. In International Conference on Web Search and Data Mining, page 87-94, 2008.
[18] Aleksandr Chuklin and Ilya Markov and. Click Models for Web Search. Springer Cham, 2015.
[19] Guipeng Xv, Si Chen, Chen Lin, Wanxian Guan, Xingyuan Bu, Xubin Li, Hongbo Deng, Jian Xu, and Bo Zheng. Visual encoding and debiasing for ctr prediction. In ACM International Conference on Information & Knowledge Management, page
2022.
[20] Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Trans. Inf. Syst., 41(3), 2023.
[21] Aman Agarwal, Kenta Takatsu, Ivan Zaitsev, and Thorsten Joachims. A general framework for counterfactual Searning-to-rank, In ACM SIGIR Conference on Research and Development in Information Retrieval, page 5-14, 2019.
[22] Zhen Qin, Suming Chen, Donald Metzler, Yongwoo Noh, Jingzheng Qin, and Xuanhui Wang. Attribute- based propensity for unbiased learning in recommender systems: Algorithm and case studies. In ACM SIGKDD International Conference on Knowledge Discovery Data Mining. page 2359-2361 2020.
[23] Mouxiang Chen, Chenghao Liu, Zemin Liu, and Jianling Sun. Scalar is not enough: Vectorization-basel unbiased learning to rank. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.
[24] Yunan Zhang, Le Yan, Zhen Qin, Honglei Zhuang. Jiaming Shen, Xuanhui Wang, Michael Bendersky, and Mare Najork. Towards disentangling relevance and bias in unbiased learning to rank. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 6618-5627) 2023.
[25] Gagan Aggarwal, Jon Feldman, and S. Muthukrishhan. Bidding to the top: Veg and equilibria of position-based auctions. In International Conference on Approximation and Online Algorithmss page 15-28, 2006.
[26] Susan Athey and Glenn Ellison. Position Auctions with Consumer Search. The Quarterly Journal of Economics, 126(3):1213-1270, 2011.
[27] Raluca M Ursu. The power of rankings: Quantifying the effect of rankings on online consumer search and purchase decisions. Marketing (itence,3(44):553)) 552, 2018.
[28] Sridhar Narayanan and Kirthi Kalyanam. Position effects in search advertising and their moderators: A regression discontinuity approach. Marketing Science, 34(3):388-477 2015.
[29] Ashish Agarwal, Kartik Hosanagar, and Michael D. Smith. Location, location, location: An analysis of profitability of position in online advertising markets. Journal of Marketing Research, 1056 2011.
[30] Filip Radlinski and Thorsten Joachims. Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. In National Conference Artificial Intelligence, page 1406-1412, 2006.
[31] Zhichong Fang, Aman Agarwal, and Thorsten Joachims. Intervention harvesting for context- dependent examination-bias estimation. In International ACM SIGIR Conference on Research and Development in Information Retrieval, page 825-834, 2019.
[32] Ronald E. Robertson, David Lazer, and Christo Wilson. Auditing the Personalization and Com- position of Politically-Related Search Engine Results Pages. In International Web Conference, 2018.
[33] Jeffrey Gleason, Desheng Hu, Ronald E. Robertson, and Christo Wilson. Google the gatekeeper: How search components affect clicks and attention. AAAI Conference on Web and Social Media, (71):245-256, 2023.
12

---

[34]
NSF. The National Internet Observatory, 2021.
org
[35] R.A. Fisher. The design of experiments. 1935.
[36]
Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. Controlled experi- ments the web: survey and practical guide. Data Mining and Knowledge Discovery, 18(1): 180-181,220 2009.
[37] G.W. Imbens and D.B. Rubin. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press, 2015.
[38] Donald B. Rubin. Randomization analysis of experimental data: The fisher randomization test comment. Journal of the American Statistical Association, 75(371):591-592 1980.
[39] Léon Bottou, Jonas Peters, Joaquin Quinonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14(11), 2013.
[40] Dimitar Dimitrov, Philipp Singer, Florian Lemmerich, and Markus Strohmaier. Visual positions of links and clicks on wikipedia. In International Conference Companion on World Wide Web, page 27-28, 2016.
[41] Erik Fubel, Niclas Michael Groll, Patrick Gundlach, Qiwei Han, and Maximilian Kaiser. Beyond rankings: Exploring the impact of serp features on organic click-through rates. Arxiv preprint wwx112300600785,, 2023.
[42] Jake D Brutlag, Hilary Hutchinson, and Maria Stone. User preference and search engine latency. JSM Proceedings, Qualtiy and Productivity Research Section, 2008.
[43] Ioannis Arapakis, Xiao Bai, and B. Barla Cambazoglu. Impact of response latency on user behavior in web search. In International ACM SIGIR Conference Research & Development in Information Retrieval, page 103-112, 2014.
[44] Florian Keusch, Ruben Bach, and Alexandru Cernat. Reactivity in measuring sensitive online behavior. Internet Research, 2022. ISSN 1066-2243.
[45] European Commission. Google shopping commission decision, 2017.
[46] Danaë Metaxa, Joon Sung Park, Ronald E. Robertson, Karrie Karahalios, Christo Wilson, Jeff Hancock, and Christian Sandvig. Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends Human Computer Interaction, 14(4):272-344
2021.
[47] Jack Bandy. Problematic machine behavior: A systematic literature review of algorithm audits. ACM Human-Computer Interaction, 5, 2021.
[48] Abhisek Dash, Abhijnan Chakraborty, Saptarshi Ghosh, Animesh Mukherjee, Jens Frankenreiter, Stefan Bechtold, and Krishna P. Gummadi. Antitrust, amazon, and algorithmic auditing. Arxiv preprint arxiv:2403. 1102::2403.1863,,, 2024.
[49] Donald MacKenzie. An engine, not camera: How financial models shape markets. MIT Press. 2008.
[50] Elizabeth A. Stuart, Stephen R. Cole, Catherine P. Bradshaw, and Philip J. Leaf. The use of propensity scores assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174(2):369-388 2011.
13

---

A Limitations and Broader Impact
We develop a flexible methodology, to provide insights into the power of digital platforms. We hope our framework can support future digital market investigations, complement and address some of the limitations of current antitrust tools. This could help make firms accountable for steering behavior on digital platforms, and support cases of anti-trust, consumer protection and abuse of dominance in digital markets. Compared traditional tools, our approach offers more straights forward way to integrate experimental insights with regulatory questions and requires fewer assumptions on the market dynamics. Furthermore, by providing quantitative measurement, the methodology also allows to compare platforms and assess the effectiveness of potential remedies. That being said, instantiating the definition in the right way is still at the discretion of investigator and requires substantial domain knowledge. Further, fitting the approach within existing legal frameworks is an open question, we hope to further make this concrete in future work.
On technical side, our design ensures that for any given users, the observed clicks follow a natural distribution. However, our participants form convenience sample of online search users. Depending on the target of the investigation this might not be sufficient to argue for external validity of the quantitative insights. We provide statistics about the users to support such a judgement in Figure 9 Further, we propose to link this data with collected clicks which can potentially help to adjust estimates using propensity score reweighting Stuart et [50]. More rigorously arguing about the external validity of our results in specific contexts is left for future work. However, we expect the qualitative take-aways of our work to generalize beyond our study, and hope they can inform future modeling and problem statements around performativity in online search.
Lastly, we want to reiterate that our results for Bing should be taken with caution. While we designed our experiment to take most out of the available data, it is still small sample of ~ 600 search queries. Nevertheless, we decided to share the results with the reader.
B
Additional technical results
B.1
Causal model
To support the arguments about composability of performative power in the discussion, consider the simplified diagram of how user navigates to a website illustrated in Figure 7 The random variable U denotes user request and T indicates which website is being visited in response. The user either navigates the website via Google search (gray box), or they navigate the website on some alternative way. This can be by entering the url directly, or using different search service. Naturally, the arrangement A only impacts the outcome have any influence the outcome. For a search query, the user query leads to an arrangement of content shown on the website, and the arrangement mediates the click.
the user uses Google search, otherwise A does not
Arrangement
Google Search
Query
Click
C
U
User Request
Traffic
Figure 7: Causal graph of online search users. web request leads to the visit of a website, partially mediated by Google search.
14

---

B.2 Proof of Theorem 1
We instantiate performative power with respect to set of action A, population search queries Q. and the outcome variable za (g) ([C = ci). Since the outcome is scalar the L1 norm reduces an absolute value and we get
PP = sup aEA
Σ
E |z0 (4)
za (1)
(3)
O
Q
Using the definition of the performativity gap and the definition of the L₁ norm the proof is direct consequence of
PP sup acA
(a) = all
(4)
|Q]
sup acA
(P)[C Pr(C,((a) cil
(5)
qE@
2 sup aEA
Σ
(r(C)((a) cil [ TE
Pr(C c.]
(6)
RE
sup 8 (a)
acA
where the performativity gap is defined with respect to the set of queries Q. In words, Theorem formalizes the idea that the average effect of an arrangement change on an individual query q E can be bounded by the average aggregate statistics across queries.
C Additional details the study
C.1 Aggregate click statistics
In Figure 8 we show aggregate statistics the clicks collected. In Figure 9 we provide aggregate statistics over the user base. The latter information was collected through the onboarding form.
number rezul
numbar of
number top
special
per page
queeies quertes
0.8
0.5
0.6
Google
sang
111
44
30
Figure 8: Aggregate statistics clicks and search result pages collected during our experiment. The hlue bars show the statistics Google and orange show the statistics Bing. Numbers are aggregated based on original search pages, before any modifications are performed.
age groups
30-
disclssed
Demman
20-30 30-40
40.50
10-50
60-10
>70
Figure 9: Aggregate user statistics collected from the 85 participants with the onboarding form. Age distirbution (left) and language in which they consume online search (right). That's all the data we have about the demo- graphics of our participants.
15

---

C.2 Project website
The project website provides the entry point the experiment. Users where provided with details about the experiment, instructions for how to participate, and information about data usage. See Figure for a screenshot.
In
Join today! Fovorri Chome
Centts
PowerMeter
Proquantsy
Surmrrry
ghad cousie ANDON 1p rehaingt scheeiik EOCONI
perural endgrdd
Nota 201 port-ipate?
HOW DOES IT WORK?
Figure 10: Project website. URL and institution names are removed for the sake of anonymity.
C.3
Onboarding form
Upon installation of the extension the is navigated the onboarding form, illustrated in Figure 11
Providing the information is not mandatory and answers are binned to only provide coarse grained information and personally identifiable information. The main purpose of the information serves debugging different languages and website versions.
PowerMeter
Welcomee on boad
Orbcarsiing Torm
Figure 11: Onboarding form.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
Assumptions are stated in Section 3 and the proofs can be fund in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The experimental protocol described in detail, including all the technical steps we took ensure validity of the experimental design. As such it can be reproduced as a general framework.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-souree dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
18

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: Our data was collected under the "need know" principle, and not intended for publication. The data is also very specific to the question under investigation, providing limited additional insights. Regarding code. The extension is publicly available in the Chrome store, and the code can fully be inspected. The link can be found on the project website.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
Sublic/guides Cooesuubiissionoooicoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips /nips.cc/publl wacc/public/pae hwwpc//ggiessseeee http://///eoeeuuubaiiiiooooooooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification:
We do not report results model training
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide error bars all figures, using bootstrapped sampling, as described.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: Experimental results concern data evaluation, compute-intensive model training of inference is performed.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp/s///errrpp..ppppbbb.cllle itw.e
Answer: [Yes]
Justification: To clarify the question privacy. The project is purposefully designed not to collect personal data. The privacy policy of our extension went through internal legal review. It is linked on our website.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section A for broader impact statement.
Guidelines:
20

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release and model dataset.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We provide a screenshot of the website that provides the entry point to the experiment. It contains information about the experiment and participation instructions. In addition, the extension policy describes the experiment in full detail, see project website.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
IRB approval was waved for this type of study.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "2PnPn7sWss", "reviewer": "Reviewer_9NWC", "summary": "The authors conduct a user study on the performative power of search engines, i.e., how much search engine providers can affect the information seen by the end user by tweaking the algorithmic ordering of results. In the specific context and assumptions formulated in this paper, this essentially amounts to measuring click-through rate differences across arrangements of search results.\nThe authors ran a live RCT experiment by injecting different arrangements directly on the result page with the help of a browser extension of rheir design. Results show that there are notable differences in CTR and therefore commercial search engines have a large performative power.", "questions": "- Do your results contradict or enrich previous estimations of position bias ? If so, what explains this ?\n- Do you think your method base on the extension is more reliable, or complementary to other methods used in past studies, e.g., eye tracking, result randomization, intervention harvesting,... ?\n- How do your results relate to other types of biases that have been identified, especially trust bias ?", "limitations": "Limitations are correctly adressed and very clearly laid out.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper is very well written, with all methodology and assumptions being laid out clearly. Moreover the experiment design and scale seem sufficient for the task at hand according to my understanding. \nThe results are made more solid and generalizable thanks to the use of different providers. Also this work is more complete than some previous user studies of this kind and the performative power angle is original and relevant.", "weaknesses": "I have two major concerns :\n- I am not sure this paper should appear in NeurIPS. It does contin any neural system, nor does it indicate how to create one from the results. While these results could in general be useful to an ML practitioner working with search engines, this could be said of many other research outcomes from different fields, that ypicaltly don't appear in NeurIPS proceedings.\n- The paper does not relate enough to previous work on position bias and other types of biases in searh results. While I believe new, updated user studies are always valuable, I think the authors should compare their results with those obtained by past studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback. We are glad you enjoyed reading the paper. We hope to clarify your questions by providing additional context related to the comparison points you mentioned. We will incorporate these discussions in the manuscript for a future version. \n\n**Other types of biases.** As you pointed out, position bias [6], trust bias [4], presentation bias [3], and other behavioral aspects have been shown to impact the effect of ranking on user clicks. In our work we are not interested in pinpointing any mechanism specifically, rather we are interested in directly quantifying the effect of ranking updates on clicks within the context of a given platform (we call this the performativity gap). The different types of biases will naturally enter such a measurement. We argue that it is not necessary to understand all these complex behavioral mechanisms to measure performative power and monitor algorithmic systems. \n\nIn that regard a browser extension is very powerful. It serves as a tool to conduct a controlled experiment and measure the effect of specific algorithmic changes *under natural interactions of users with the platform*. As a result it implicitly captures trust, display, and other behavioral aspects specific to the platform under investigation. \n\n**Methodological differences.** *Information harvesting*, and *randomization* are methods to explicitly or implicitly exploit interventions to the ranking algorithm to measure their effect on clicks. These methods all get at the fundamental problem of causal inference; the effect of ranking can not be measured from click data alone, due to unobserved feedback and biases. Our work is also a way to gather and exploit interventional data. In contrast to prior work, e.g., the work on data harvesting in [2] where the authors consider interventions to the Arxiv ranking algorithm, the Powermeter design performs interventions at the level of the platform-user interface. We emulate updates to the algorithm without actually touching the algorithm underneath. While the observed effect is equivalent to implementing the corresponding change to the algorithm, it allows us to study the Google search algorithm, without controlling it, which is an important methodological novelty. We are not aware of any such experimental design in the literature. \n\nFinally, you mentioned *eye-tracking* studies. They are designed to measure the allocation of visual attention, e.g., to support the design of click models, such as cascade models [5]. However, they can not replace click statistics, and are complementary to performing the interventions to study. \n\n**Discussion of prior quantitative insights.** While our study pursues a different goal than estimating parameters of a click model, some of our intermediate experimental insights can be compared to prior work. While we are not aware of comparable studies related to interventions a4-a6, intervention a1-a3 constitute pairwise swaps routinely used to infer propensities in click models. The most relevant comparison point is work by researchers out of Google from 2010 [3, Figure 1] who published click statistics on FairPairs (adjacent swaps [7]) on Google search. They report an average gain in click counts of 75% when ranked first instead of second under swap 1-2, we observe a gain of 82%. For swap 2-3 they report 48% gain, whereas we see a gain of 66%. Thus, while qualitatively similar the effect size in our results is larger. This might be attributed to changes to the website design since 2009, for example, including the introduction of featured snippets that lead to a larger spatial separation between results, and may increase performativity. While this is only a hypothesis, it demonstrates how effects can change over time even for a fixed platform. With Powermeter we offer a method to reassess such effects within the context relevant for a specific investigation. Beyond swaps, we can provide insights into different counterfactuals that have not been studied before, such as the combination of downranking results and adding visually appealing elements, which is relevant in a regulatory context. \n\n**Why NeurIPS?** We build on a concept of performative power that has its origin in the NeurIPS community [1], and we offer a first account of its practical applicability which we see as an important contribution to the area of “Social and economic aspects of machine learning\" referred to in the CFP. Focusing on neural systems in a broader context is not uncommon for NeurIPS, see work related to auditing and regulation. Similarly, citing again from the CFP: \"Machine learning is a rapidly evolving field, and so we welcome interdisciplinary submissions that do not fit neatly into existing categories.\" While we understand that this is subjective, there is an easy case to be made for why our work is in scope, accessible and relevant for the broader NeurIPS audience. In any case, we believe that the AC/SAC can ultimately make an executive decision about this and we would appreciate if this could be treated independent from the assessment of the quality of our work.\n\nReferences\n\n[1] Hardt, Jagadeesan, Mendler-Dünner. \"Performative power.\" NeurIPS. 2022.\n\n[2] Fang, Agarwal, Joachims. “Intervention Harvesting for Context-Dependent Examination-Bias Estimation”. SIGIR. 2019. \n\n[3] Yue, Patel, Roehrig. “Beyond Position Bias: Examining Result Attractiveness As a Source of Presentation Bias in Clickthrough Data.” WWW. 2010.\n\n[4] O’Brien, Keane. “Modeling result–list searching in the World Wide Web: The role of relevance topologies and trust bias.” CogSci. 2008.\n\n[5] Craswell, Zoeter, Taylor, Ramsey. “An Experimental Comparison of Click Position-bias Models”. WSDM. 2008.\n\n[6] Joachims et al. “Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search”. TOIS. 2007.\n\n[7] Radlinski Joachims. Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. AAAI. 2006.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Adc4HjPueH", "reviewer": "Reviewer_7fp1", "summary": "The paper titled \"An engine not a camera: Measuring performative power of online search\" presents a study on the performative power of online search engines, specifically focusing on how they can influence web traffic through the arrangement of search results. The authors designed and executed an experiment to quantify the ability of search engines to steer user clicks by rearranging search results, a concept known as performative power.\n\nThe main contributions of the paper are as follows:\n\n**Conceptual Framework**: The authors build upon the recent definition of performative power to develop a framework for understanding the influence of digital platforms, particularly search engines, on user behavior.\n\n**Experiment Design and Quantitative Findings:**  \n\nThe study involved tens of thousands of clicks collected over a five-month period from more than 80 participants, providing robust quantitative insights into the causal effects of search result arrangements. The paper reports significant quantitative findings, such as the observation that consistently downranking the first search result by one position causes an average reduction in clicks of 42% on Google search, and even more for Bing.", "questions": "See Weaknesses", "limitations": "The author has already discussed the limitations of the paper.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality:**\n\nThe paper introduces a novel concept, the performative power of online search engines, which is a significant contribution to the field of digital platform policy and regulation.\n\n**Quality:**\n\nThe analysis is thorough, with robust quantitative findings supported by extensive data collection over a five-month period from a diverse participant base.\n\n**Clarity:**\n\nThe paper is well-structured, with clear definitions and explanations of key concepts such as performative power and the methodology used for the study.\n\n**Significance**\n\nThe researchers presents that search engines are active \"engines\" that can significantly shape and influence the information landscape, with important sociopolitical implications.", "weaknesses": "1. **Limited Sample Size and Diversity.** The paper could benefit from a larger and more diverse sample to ensure the results are representative of different user behaviors across various demographics.\n2. **Lack of case studies**. While the paper provides clear quantitative results, additional qualitative analysis or case studies could help interpret why certain patterns emerge, offering deeper insights into user decision-making processes.\n3. **Lack of discussion on interventions**. The researchers provide a compelling framework for measuring this performative power, but the paper does not extensively address potential countermeasures or interventions that could help mitigate the negative aspects of search engines' performative power.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback. Let us explain why we see the focus on measurement (rather than qualitative modeling) as an important opportunity of our approach, rather than a weakness.\n\n**Qualitative insights.** We agree that qualitative insights are highly valuable. However, it is possible to measure and monitor the influence of AI systems on user behavior without the need to understand the complex mechanism behind it. This is an important takeaway of our work in the context of market regulation specifically, as regulators have been struggling with the complexity of digital markets, see e.g., [1]. Our work shows that the complexity to model digital markets should not prevent us from developing effective measurement and monitoring tools.\n\n**Countermeasures.**  Related to the above comment, it is useful to treat measurement separately from the design of countermeasures. Remedies are often context specific and require the balancing of many competing objectives. This goes far beyond measurement. And there are many use cases where measurement is what you primarily care about. This separation is also common in antitrust investigations for example, where there is no need to suggest an effective remedy to run a case. But as a first step towards designing effective remedies, what we can offer is an investigative tool for researchers, regulators, and potentially also platform owners to measure the effectiveness of remedies, as well as setting transparent evaluation criteria. \n\n**Sample size.** We agree that a larger pool of participants would always strengthen the results, and a larger sample size could be useful specifically to reduce uncertainty on some of the query subset evaluations, where the bootstrap confidence sets are not as tight as for the aggregate results. We can not add additional data at this point, but we have included additional robustness checks in the supplementary PDF, where we evaluate Jackknife error bars with respect to the impact of excluding a random agent from the study. They can not speak to ecological validity, but they can provide an indication for how stable the effect measures are to the observed collection of agents.\n\nReferences:\n\n[1] Final report: Stigler committee on digital platforms. 2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yPzBcI5LzL", "reviewer": "Reviewer_6Zv1", "summary": "This paper describes an online experiment seeking to measure how much power online search provides have in terms of impacting what content people consume. In short, the study attempts to measure the causal effect of small ranking rearrangements on click-rate for a population of web users. The authors present experimental data that can be used to estimate the expected impact that operators of search engines, recommender systems, and other ranking technologies might have on viewership of items when they make small changes to their rankings.", "questions": "A few questions that might be worth addressing in the next version of the paper\n- Are there major constraints you would expect to face if trying to apply these methods to arbitrary types of other ranking platforms (feeds, matching/marketplaces platforms, etc.).\n- To what extent, if any, do you expect the specific algorithmic / modelling choices made by platforms to matter here?\n- Are there domains that might be hard to study, because e.g. the number of items is too high?", "limitations": "Limitations of the methods are reasonably discussed throughout.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "In terms of originality, quality, and clarity:\n- Originality is high. The authors review (in reasonably terse fashion) a number of studies that have sought to understand the impact of ranking items in search on attention (i.e., clicks and  visual attention that items receive). The study is grounded in this past work and notes its major contribution is to begin studying this effect experimentally.\n- Quality of the study is high overall. See some concerns / questions below, but overall I would consider this to be a very convincing set of results.\n- Clarity is very high throughout. Experimental details are crisply described.\n\nIn terms of significance\n- Reasonable dataset (for this kind of study) with 57,000 \"real\" queries from 80 participants\n- Known caveat that getting this data without direct access to search operator datasets is prohibitively expensive, hence why this is novel.", "weaknesses": "Two weaknesses (with respect to venue fit and potential of the current draft) stood out: \n\nFirst, a minor note: this kind of experimental work might be somewhat unusual at NeurIPS as it doesn't engage with the modelling side of ranking. Personally, I do not think this should be a blocking reason -- I think many in the community would like to see future works that incorporate this kind of experimental approach -- but it felt like an important piece of context to note.\n\nSecond, some readers may concerned with perceived issues with ecological validity. To some extent these are unavoidable in any experimental study like this -- there will always be some set of ecological validity concerns, they just trade off with each other.\n\n- While this choice is reasonable, it may impact the kinds of queries used: \"The study participants are trusted individuals of different age groups and backgrounds, recruited by reaching out personally or via email.\"\n- It might be helpful (esp. given the work may impact policy discussions) to know more about the domain / type of queries, but very reasonable privacy choice to avoid sharing any information about that.\n- Very minor: It's (by choice of technology companies) unclear if the kinds of perturbations studied here map to the kinds of a/b tests or experiments that are frequently rolled out by those companies. I don't think this is something the paper needs to address explicitly, but is also worth noting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback on our work. In the following we first discuss your questions and then provide some thoughts on the additional comments below.\n\n**Applying method to other use cases.** This is an interesting point we have not discussed in our work. An important feature of our design is that the complexity of the underlying algorithm does not impact the experimental design. The intervention is implemented at the level of the user interface, rather than the algorithmic system. This means the system does not have to be changed. From a technical viewpoint, changing how results are displayed to users can in principle be applied to any website using a browser extension, independent of how these results are selected by an underlying algorithm. However, one important constraint is that the updates that can be emulated at the display level are limited to the information available on a website. This means, we can swap and hide elements, but not replace them with an alternative that is hidden. Similarly, we can not use any proprietary data the firm might use to determine content allocation. This limits the updates we can emulate. However, for the purpose of lower-bounding performative power, it is sufficient to argue for one feasible update that can be implemented. Evaluating the effect of this update provides a lower bound on the corresponding definition of performative power. Here, the swap of adjacent results usually offers a good proxy for potentially larger updates a firm could implement. \n\n**Domains that are harder to study.** In light of the constraint mentioned above the design of counterfactuals is harder for settings where the firm offers the user fewer choices, like in the case of the Amazon Buy Box where a single element is selected to be displayed behind a visually appealing button. In such cases natural updates are harder to argue without additional insights into the algorithm, or the alternative options. Large number of items offer the opposite situation which does not seem to be a problem, but we are not entirely sure we understand the question. One interesting aspect of a large number of items is usually that more of them can be relevant, which leads to a stronger effect of ranking and a larger performativity gap. But this would just impact the effect size, not the implementation. \n\n**Ecological validity.** As the reviewer mentioned, concerns related to ecological validity are unavoidable to some extent. What we can offer is an additional ablation and robustness checks where we remove individual participants from the evaluation to check the sensitivity of the statistics. Results are shown in the supplementary PDF, but error bars are still very small.\n\n**Recording query.** Thank you for supporting this design choice. Search queries are one of the most sensitive personal data. For our proof of concept it was not necessary to collect this information which is why we refrained from doing it. However, we agree that when inspecting more nuanced policy related questions additional information might be necessary to record, and the benefits may outweigh the privacy costs in a specific case. However, the exact question should come first and guide the corresponding decision. From a technical perspective there is nothing preventing us from recording user queries, we have the information readily available. For a future study we might consider storing an embedding of the search query, but for the current data we simply do not have this information at our disposal.\n\n**Why we chose NeurIPS.** We agree that this type of experimental work is less typical from the perspective of developing better ranking models. However, from a perspective of causal inference, and algorithmic auditing, directly tackling the measurement problem is quite natural. We also believe that decoupling modeling from empirical measurement is very important for developing investigative tools in the context of AI systems. Modeling behavioral aspects in the interaction of participants with platforms is very complex and this complexity should not prevent us from developing effective monitoring and measurement tools. For additional arguments for why we think our work is interesting for the NeurIPS community and falls under the call for paper can be found in the response to Reviewer 9NWC, but it does not seem that we need to convince you of this.\n\nWe hope our rebuttal addresses your questions satisfactorily and we will incorporate additional discussion of the points you raised in a future version. \n\nReferences\n\n[1] Hardt, Jagadeesan, and Mendler-Dünner. \"Performative power.\" NeurIPS. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0h9x7sXPVN", "reviewer": "Reviewer_iJky", "summary": "The authors describe performative power, a pre-existing proposed measure of platform market power, and give an approach to measuring performative power using a browser extension. The browser extension perform random modifications to the search results page of results from target search engines, and measure click behavior. The modifications and user clicks are logged, providing sufficient statistics to compute a variant of the performative power metric. They deploy the extension to 85 users, resulting in about 57k clicks, and produce performative power calculations based on these interactions. The authors then provide some discussion of the results.", "questions": "I'd love to hear the authors' thoughts on the three issues I raised in the weaknesses section above. Here are a few smaller questions also (I'll start numbering at 4, assuming the three points above are questions 1-3)\n\n4. Could you discuss the fact that you consider only a single click, given that the intervention is likely to produce a higher number of clicks than the control (as satisfaction with the top result will be lower)? Likewise, the browser extension has the capability to consider more details of the interaction, such as immediate click-back from an unsatisfactory result page, followed by a click on a following page. I know it's biting off a lot to consider these types of interactions, but perhaps you could discuss a little.\n\n5. Nit: line 112 has a typo in the cite of Chuklin et al\n\n6. Could you clarify the triggering for arrangements a4-a6? Do they trigger only on results that have ads / shopping content? How is the accounted for in the analysis? Sorry, this may already be specified in the paper and I may have missed it.\n\n7. Latency sounds pretty good, but probably worth referencing literature about the impact on search user behavior from additional latency of this magnitude\n\n8. Study participants are trusted individuals solicited by personal outreach -- this suggests they will likely trust the researchers, tend to give the browser plugin the \"benefit of the doubt,\" and so forth. Could you discuss the issues of conditioning that result from this choice of users?", "limitations": "I feel that the limitations I discuss above should be covered in more detail by the authors. For ethics review, the authors indicate that this type of study does not require IRB approval at their institution, so I take this at face value.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I feel there are two main strengths to the paper\n1. The authors make the case for measuring performative power using a browser extension\n2. The authors created a browser extension to measure performative power. They appear to have been somewhat careful, discussing issues with hiding the DOM until rewriting is complete, discussing some latency numbers that are not bad (around 40ms, certainly enough to impact user behavior, but not enough to be wildly visible). They discuss privacy implications of their storage and logging of user search events, and have taken a reasonable approach with this, logging only intervention ids and click positions.", "weaknesses": "W1: supporting ongoing work. I didn't see a reference to the source code for the browser extension, or any discussion of it being available for further work -- this seems like a surprising omission, though perhaps I didn't read the right section?\nW2: on novelty, I have a hard time characterizing exactly where the main contribution of the paper lies. The development of the performative power measure, and the argument for its relevance in regulation, comes from prior work. The understanding of the impact of position on click likelihood is also quite heavily studied, so the headline numbers the authors show have reasonable support in the literature already. The authors make a small modification to the definition to incorporate changes to the page beyond the organic results, but this does not seem to be the key point. The observation that PP can be computed from a browser extension seems fairly straightforward, not a centerpiece for the argument of novelty -- such approaches have been taken for modifying search results and measuring interactions in the past. The paper could arguably make the case for taking a straightforward idea (browser extension for performative power) and exploring the many thorny problems in designing and deploying this measurement, but my second primary concern below is that these issues remain largely outside the scope of the paper. Overall, I feel that there isn't a clear case to be made for the dimension of novelty.\nW3: platform power. The authors paint a picture of developing a causal understanding of the \"power\" present in the platform to differentially route user attention across resources. My concern is that some interventions are sustainable (ie, the platform could actually implement such a modification) while others are not. As an extreme and somewhat laughable example, consider the intervention that replaced every search result with a link to the CEO's gofundme account. This would change clicks dramatically, resulting in high performative power, but nobody would argue that the platforms could sustainably deploy this intervention. Instead, the goal is to consider power the platform has to alter the *ongoing* distribution of user attention to online resources, so making this measure robust requires significant attention to the issue of reasonable interventions. The related literature the authors cite (cascade, eye-tracking, MLIR) is quite careful in these areas. It is not clear that placing a pure navigational result at position 3 instead of position 1 is sustainable, and the issue is not so simple: users habituate to platform behavior, and will respond differently when the platform stops behaving as expected, both in determining which results to consider and hence where to click, and in bigger ways (ie, changing providers). Correspondingly, platforms themselves rely on user feedback, which would be adulterated by these types of interventions, with unclear implications. For a proposal intended to be a \"blue print for how performative power can be used to guide digital market investigations,\" I think it's a significant omission not to consider these issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback and comments. We hope to provide additional clarification and address your questions in the following.\n\n**W1: Source code.** The extension has been published in the Chrome store. The code can be inspected using the developer console in the Chrome browser. We did not include the link for anonymity reasons, but we will add it to the final version. If you wish to inspect the code, we uploaded it to an anonymized repository shared with the AC.\n\n**W2: Novelty.** The core contribution of our work is to take a theoretical concept and provide a first demonstration for how it can be instrumentalized in a practical scenario. A priori it is unclear how to relate performative power to micro experiments that are feasible to perform on digital platforms. Our experimental design together with the required theoretical assumptions outline a possible avenue for doing this. By relating tools from computer science with a policy relevant question at the heart of a major antitrust investigation our work provides a promising interface between regulatory questions and the expertise of the NeurIPS community. \n\n**W3: Choice of interventions.** Note that the instantiation of the actions set $\\mathcal A$ is part of the performative power definition. Different choices are valid, also larger changes, but they lead to different conclusions about power. So it needs to be instantiated carefully in any given context. In our case we care about the effect of actions the platform has implemented in the past (downranking search results and adding visually appealing elements). Our counterfactuals are designed carefully to give us insights into performative power related to these interventions.\n\nAs you pointed out it is important not to interleave experiments with treatments that have a disruptive impact on user experience and behavior, as we want to measure the effects of our interventions under natural interactions of users with the platform. Being exposed to one intervention should not impact the behavior for a subsequent query. We make this requirement explicit in Assumption 1. It is also why we refrained from including larger swaps as an additional treatment group, after feedback from an initial testing phase. All our counterfactuals constitute minimal interventions. As you point out, the largest modification we perform is to move the first search results down by two positions. This is comparable to having two Ads on top of the page and it typically does not move the result out of sight. Based on our own experience and feedback from participants there is no evidence that the extension was noticeable by any of the participants.\n\nIt is also important to note that modifications are triggered at random. This offers no structure a user could implicitly or explicitly adapt to. The swap 1-3 only happens with probability ⅛ for any user query (including navigational queries), and for 45% of the queries the extension does not swap results. We checked the behavior in the control group by splitting the data collected before and after November 10. The difference in the average click rate for the first result differs in 0.8% across these two groups (the base rate is 43%) which is significantly smaller than the sampling uncertainty. We will think of a more rigorous experiment to add to the appendix to demonstrate that behavior of participants is stable across the duration of the study.\n\n**Triggering a4-a6.** Arrangements a4-a6 are triggered each with a fixed probability (p=1/6). If there is no content to hide for this query, a5 solely performs a swap and the other interventions leave the website as is. This ensures that treatment assignment is independent of the potential click outcome which is important to obtain unbiased estimates. On aggregate the effect of hiding Ads will naturally be larger if Ads are present more often.\n\n**Latency.** The latency refers to the time for loading the search page after entering the search query. A technical report on Google [1] could not find any effect for delays below 100ms on search behavior. Similarly, [2] find that  “up to a point (500ms) added response time delays are not noticeable by the users.”  Our delay of 40ms is below any of these thresholds and thus not expected to impact our results. We will include these references.\n\n**Considering a single click only.** As you pointed out we are not making a distinction between follow-up searches and new searches. What we measure is the ability of the platform to steer clicks treating all clicks equal. To analyze a different outcome variable, such as a specific type of click, we would have to adjust the experiment accordingly. However, the scenario you mentioned will still meaningfully affect our measurements. If a user returns to click the second result after clicking the first under intervention a1, this will surface as a reduction in the performativity gap. This leads to the interesting observation that performative power is larger in situations where equally relevant results are being ranked.  \n\n**Participants tend to give the browser plugin the \"benefit of the doubt.\"** There is a bias in the selection of participants, as people who trust us in performing a legitimate scientific experiment and take privacy seriously, are more likely to install the extension. But once they have the extension installed, the experiment is randomized at a query level and we can safely assume that they consume Google search in a natural way. There is no structured change they could respond to. We even deleted the data from the first 4 days after onboarding to be on the safe side and avoid potential biases due to participation awareness.\n\nPlease clarify if we misunderstood your last questions. But we hope the additional context is helpful. \n\nReferences:\n\n[1] J. Brutlag. Speed matters for google web search. Online report by Google. 2009.\n[2] Arapakis et al. Impact of response latency on user behavior in web search. SIGIR. 2014.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for the feedback. The attached PDF contains an additional robustness evaluation to support the author-reviewer discussion. We address questions below, responding individually to all reviewers.

Meta-review: This submission presents a Chrome browser extension that modifies the arrangement of search result pages and records user click statistics, for the purpose of studying how the change of result ranking affects user clicks. At the first glance, the settings and approach conducted in this work seem identical to what studied in the prior information retrieval research, and the conclusions also follow the previous observations. Hence, most reviewers concerned the novelty of this work.

Throughout lengthy and engaged discussions with the authors, the main motivation became clearer to the reviewers that this work actually focuses on quantifying the effect of one specific intervention in a **retrospective** manner, instead of the general phenomenon itself or in a controlled test to study its impact. This helps the reviewers to reach an agreement about the evaluation of this work, though it also raises some concerns regarding the effective sample size for such analysis to be practically valid.   

During the discussion with the reviewers, all of them expressed their support of this work, but also urged the authors to revise the manuscript to reflect what they discussed and clarified in the rebuttal, which would help others to most precisely position and understand this work. 

To summarize, we recommend acceptance of this work, but ask the authors to carefully revise the content to make the motivation clearer to its readers.
Decision: Accept (poster)