Ranking-Incentivized Document Manipulations for Multiple Queries

Venue: ACM_org_SIGIR_ICTIR_2024_Conference
Published: 2024-04-25 16:56:41

Abstract: In competitive retrieval settings, document publishers (authors)
modify their documents in response to induced rankings so as to
potentially improve their future rankings. Previous work has focused
on analyzing ranking-incentivized document modifications for a {\em single
  query}. We present a novel theoretical and empirical study of
document modification strategies applied for potentially improved ranking for {\em multiple queries}; e.g., those representing the same information need. Using game theoretic analysis, we
show that in contrast to the single-query setting, an equilibrium does
not necessarily exist; we provide full characterization of when it
does for a basic family of ranking functions. We empirically study document modification strategies in the
multiple-queries setting by organizing ranking competitions. In
contrast to previous ranking competitions devised for the single-query
setting, we also used a neural ranker and allowed in some competitions
the use of generative AI tools to modify documents. We found that
publishers tend to mimic content from documents highly ranked in the
past, as in the single-query setting, although this was a somewhat less emphasized trend when generative AI tools were allowed. We also found that it was much more
difficult with neural rankers to promote a document to the highest
rank simultaneously for multiple queries than it was with a feature-based learning-to-rank method. In addition, we demonstrate the merits of using
information induced from multiple queries to predict which document might
be the highest ranked in the next ranking for a given query.

Main paper: 
 introduction: 
 In the Web search setting, publishers (authors) of documents are sometimes incentivized to have their documents highly ranked by search engines for some queries. The basic motivation is the fact that documents at top ranks often attract most user engagement [19] . As a result of their ranking incentives, publishers might modify their documents in response to rankings induced by a search engine so as to improve their future rankings. The search setting then becomes competitive [21] with corpus dynamics driven in part by ranking-incentivized document modifications. These modifications are often referred to as search engine optimization (SEO) [14] .Lately, there has been a growing body of work on studying ranking-incentivized document modifications, and more generally, on competitive retrieval [3, 7, 8, 11-13, 23, 25, 32, 34, 36, 37, 39, 40] . This renewed interest in the often dubbed adversarial retrieval setting [6] can be attributed in part to a large volume of work on adversarial machine learning in the era of neural network models [16-18, 20, 30, 35, 42, 44] .The focus of some recent work on competitive retrieval is on algorithmic attacks on ranking functions; e.g., substituting terms with their synonyms [7, 8, 13, 23, 25, 34, 37, 39] . There is also an emerging thread of work on analyzing, and improving, the robustness of ranking functions to ranking-incentivized document modifications [3, 7, 11, 36, 40] . Another type of work, which is our focus in this paper, is analyzing human strategies of ranking-incentivized document manipulations [32] and their resultant effect on the corpus [3, 13, 32] . For example, it was shown theoretically and empirically that a prominent (worthwhile) strategy of publishers is to mimic content in documents that were highly ranked in the past for queries of interest [32] , which results in an herding effect [13] .Almost all recent work on competitive retrieval has focused on a single-query setting; i.e., assuming a publisher modifies her document to improve its ranking for a single query 1 . However, publishers often opt to have their document highly ranked for multiple queries; e.g., those representing a topic in the document. Accordingly, in this paper we present the first -to the best of our knowledge -theoretical and empirical study of ranking-incentivized document-modification strategies of (human) publishers opting to promote their documents for multiple queries.Using game theoretic analysis we show that when publishers modify their documents to improve their rankings for multiple queries, there is not necessarily an equilibrium. This result implies instability: a potentially endless document modification race which can have negative effects on users; e.g., documents at top ranks consistently change not due to pure editorial considerations but rather due to ranking incentives. This theoretical result is in contrast to recent findings about the existence of an equilibrium when documents are modified for a single query [32] .We then fully characterize the conditions for an equilibrium in the multiple-queries setting for a basic family of ranking functions. Furthermore, we show that best response dynamics [27] does not necessarily converge to an equilibrium in case it exists. Best response dynamics is a process where each publisher modifies her document to attain the best possible ranking given the documents written by all other publishers. The implication is that reaching a steady state in case it is achievable might call for an external intervention; e.g., of the search engine.We next present an empirical study of document manipulation strategies applied by publishers that opt for improved ranking for multiple queries representing the same information need (topic). The study is based on ranking competitions we organized between students. Our competitions were inspired by those organized for the single-query setting [11, 13, 32] . The competitions we report differ from previous competitions [11, 13, 32] not only by the virtue of having publishers modify their documents for multiple queries rather than a single one, but also by two additional aspects. First, our competitions are the first to employ not only a feature-based learning-to-rank (LTR) method [24] as the ranking function, but also a neural ranker [22] . Second, in some of our competitions, we encouraged the participants to use generative AI tools (e.g., GPT [29] ) to help modify their documents. Indeed, the SEO industry is using generative AI tools (https://www.seoclarity.net/research/ impact-generative-ai), and our (controlled) study sheds light on usage strategies and implications. The competitions were approved by ethics committees. The data of the competitions and our code will be made public upon publication of this paper.Analysis of the ranking competitions revealed that, as is the case for the single-query setting [32] , publishers tend to mimic content from documents that were previously highly ranked. This trend was more prominent when publishers were not allowed to use generative AI tools. We also found that the neural ranker's rankings for different queries representing the same information need were more diverse than those of the LTR method. Accordingly, it was much harder for publishers to have their documents highly ranked for multiple queries with the neural ranker.Finally, as in work on the single-query setting [32] , we pursue the task of predicting which document among those not ranked first in the last ranking will become the most highly ranked assuming that the top ranked document is going to change. As it turns out, utilizing information induced from rankings for other queries representing the same information need can help improve prediction effectiveness for the query at hand. This is yet another difference between the multiple-queries setting and the single-query setting.Our contributions can be summarized as follows:• A theoretical and empirical study of document manipulations intended to improve ranking for multiple queries. • A full game theoretic characterization of when equilibrium of document manipulation strategies exists. • Showing that best response dynamics might not converge even when an equilibrium exists. • Organizing novel ranking competitions where publishers compete for multiple queries. Additional novel aspects are using a neural ranker and allowing publishers to use generative AI tools. • A method for predicting which documents among those not ranked first might be ranked first in the next ranking; the method uses information induced from other queries. 
related work: 
 The work most related to ours is based on a game theoretic and empirical analysis of ranking incentivized document manipulations applied for a single query [32] . In contrast to our multiple queries settings, an equilibrium in the single-query setting always exists. Raifer et al. [32] and Goren et al. [13] found using ranking competitions that publishers tend to mimic content from documents highly ranked in the past for a query: a strategy justified by Raifer et al. [32] using a game theoretic analysis. We found a similar pattern in ranking competitions organized for the multiple-queries setting.We use both a neural ranker and a feature-based ranker while Raifer et al. [32] used only the latter. In contrast to our work, Raifer et al. [32] did not allow the use of generative AI tools for document manipulation in their ranking competitions. Several algorithmic attacks on neural retrieval methods have been recently reported [7, 8, 13, 23, 25, 34, 37, 39] . The attacks are almost always for a single query, except for an attack for topically related queries [25] . In contrast, in our game theoretic analysis, the document manipulations are changes of the relative emphasis of a document on different queries. This type of strategic manipulation has not been studied in previous work to the best of our knowledge. Furthermore, we are not aware of any other studies, as ours, of human document-modification strategies for multiple queries with or without generative AI tools.The suboptimality of the probability ranking principle [33] in competitive retrieval settings was demonstrated using a game theoretic approach [3] . Publishers can write either a single topic document or a two-topics document with equal emphasis for the topics. Since the resulting ranking game has a finite number of players (publishers) and a finite number of actions a player can play (i.e., documents they can write), by Nash's theorem [26] , a mixed-strategies equilibrium exists. That is, documents are stochastically created by applying a distribution over a document set. The setting is less realistic than the one we address here where publishers can write a document with different emphasis (represented as a real number) on different queries. Our resulting games have an infinite number of actions a publisher can take, and we show that there is not necessarily an equilibrium. 
game theoretic analysis: 
 To analyze the search setting when publishers modify their documents to have them highly ranked for multiple queries, we use game theory. We consider publishers as players in a ranking game: the documents they publish are their strategies and the search engine's ranking function is the mediator [21] .As noted above, some previous work used game theory to analyze the single-query search setting where publishers opt to promote their documents for a single query [32] ; single-peaked ranking functions were used [32] . Several sparse retrieval methods are single peaked; e.g., cosine between tf.idf vectors and negative KL divergence over multinomial distributions [43] . Furthermore, considering the common technique of stuffing query terms in the document to improve its ranking [14] , the retrieval function could also be viewed as single peaked with respect to query term occurrence. That is, moderately increasing query term occurrence usually results in monotonic increase of retrieval scores. However, as from a certain point, further increasing query term occurrence can result in increased penalty to retrieval scores when document quality measures are used [5] . There is also some empirical evidence in prior work [32] and ours (Section 5) that publishers might view the undisclosed ranking function as single peaked. When considering incomplete information about the ranker (unknown peak), the ranking game has a minimax regret equilibrium where publishers mimic content of documents that were previously the highest ranked [32] .Turning to our multiple-queries setting, we assume that a document has split emphasis on various queries; e.g., different passages match different queries to varying degrees. We assume as Raifer et al. [32] a single peaked ranking function, and focus as Basat et al. [2] on ranking games where the retrieval function is disclosed (i.e., its peak is known). We show that the resulting situation in the multiple-queries setting with a disclosed ranking function is considerably more subtle than that in the single query setting where it is trivial to show that an equilibrium exists. Specifically, we prove that there is not necessarily an equilibrium in the multiple-query setting and fully characterize the cases when it exists.Let 𝐺 = ⟨𝑛, 𝑚, 𝑝⟩ be a ranking game defined as follows: 𝑛 publishers (players) write and modify documents in a corpus 𝐷; a query set 𝑄 contains 𝑚 queries: {𝑞 1 , . . . , 𝑞 𝑚 }. Each player writes a singledocument 𝑑 ∈ 𝐷 𝑑𝑒 𝑓 = {(𝑑 1 , . . . , 𝑑 𝑚 ) ∈ [0, 1] 𝑚 : 𝑚 𝑗=1 𝑑 𝑗 ≤ 1}, where 𝑑 𝑗 (∈ [0, 1]) is the relative emphasis of the document 𝑑 on information relevant to query 𝑞 𝑗 . 2Let 𝑓 : [0, 1] → [0, 1] be a single peak function: there exists a unique 𝑝 ∈ [0, 1] such that 𝑓 is monotonically increasing in [0, 𝑝] and monotonically decreasing in [𝑝, 1]. The retrieval function 𝑟 : 𝐷 ×𝑄 → [0, 1] assigns document 𝑑 (∈ 𝐷) the retrieval score𝑟 (𝑑, 𝑞 𝑗 ) 𝑑𝑒 𝑓= 𝑓 (𝑑 𝑗 ) with respect to query 𝑞 𝑗 (𝑗 ∈ {1, . . . , 𝑚}); documents are ranked in descending order of scores with ties broken arbitrarily.We will refer to document 𝑑 𝑖 written by player 𝑖 as the (pure) strategy of player 𝑖 in the game 3 . That is, the strategy 𝑑 𝑖 is defined by the choice of relative emphasis on each query in the document 𝑑 𝑖 .A strategy profile 𝑠 𝑑𝑒 𝑓 = (𝑑 1 , . . . , 𝑑 𝑛 ) is the tuple of pure strategies of the players in the game; i.e., the set of documents in the corpus.The utility of a player 𝑖 who wrote document 𝑑 𝑖 , denoted 𝑈 𝑖 (𝑠), is the number of queries for which 𝑑 𝑖 is ranked first; 𝑑 𝑖 's rank for a query depends on the strategies of all other players (i.e., their documents). To summarize, 𝐺 is a strategic game where 𝑛 players (publishers) opt to maximize their documents' ranks for 𝑚 queries.A strategy profile 𝑠 is a (pure) Nash equilibrium if there is no incentive for any player 𝑖 to change her strategy given the strategies {𝑑 𝑗 } 𝑗≠𝑖 of all other players; i.e., a change will result in reduced utility.Our goal is to find a full characterization of the existence (or lack thereof) of pure Nash equilibria for 𝐺, given 𝑛, 𝑚 and 𝑝.It is easy to see that for a small enough 𝑝, there exists a pure Nash equilibrium where the documents published by all players have the same emphasis on all queries:4 Lemma 3.1. If 𝑝 ≤ 1 𝑚 , then the profile where all players write 𝑑 = (𝑝, . . . , 𝑝) is a pure Nash equilibrium.To extend our analysis for any 𝑝 (∈ [0, 1]), we start by analyzing the case of two players (𝑛 = 2). This case has an interesting property, because in any pure Nash equilibrium, the two players have the same utility. They can attain this utility, for example, by using the same strategy (i.e., writing the same document): Lemma 3.2. Let 𝑛 = 2 and 𝑚 > 𝑛. If 𝑠 = (𝑑 1 , 𝑑 2 ) is a pure Nash equilibrium, then 𝑈 1 (𝑠) = 𝑈 2 (𝑠) = 𝑚 2 . Proof. Observe that for any profile 𝑠 = (𝑑 1 , 𝑑 2 ), 𝑈 1 (𝑠) +𝑈 2 (𝑠) = 𝑚. Assume by contradiction that 𝑈 1 (𝑠) ≠ 𝑈 2 (𝑠), w.l.o.g. 𝑈 1 (𝑠) < 𝑈 2 (𝑠); i.e., 𝑈 1 (𝑠) < 𝑚 2 < 𝑈 2 (𝑠). If player 1 adopts player 2's strategy, i.e., writes < 𝑝. This allows to show that if there is an equilibrium, then 𝑝 ≤ 1 𝑚-1 . If 𝑝 ≤ 1 𝑚 , then by Lemma 3.1 the profile where both players publish 𝑑 = (𝑝, . . . , 𝑝) is a pure Nash equilibrium. We show that for 1 𝑚 < 𝑝 ≤ 1 𝑚-1 the profile where𝑑 ′ 1 = 𝑑 2 then 𝑈 1 (𝑑 ′ 1 , 𝑑 2 ) = 𝑈 2 (𝑑 ′ 1 , 𝑑 2 ) = 𝑚 2 > 𝑈 1 (𝑑𝑑 1 = 𝑑 2 = (𝑝, . . . , 𝑝, 1 -𝑝 • (𝑚 -1)) is a pure Nash equilibrium. □To extend the analysis to a game with more than two players, we start by considering the case of 𝑚 > 𝑛: Proof. The full proof is mostly technical and fully detailed in Appendix B. The central idea is to show that in any pure Nash equilibrium the following properties must hold: (i) the highest ranked document 𝑑 for each query 𝑞 𝑗 has 𝑑 𝑗 ≥ 𝑝; and, (ii) for each player there is at most one query for which her document is ranked first solely. For any 𝑝 >1 ⌈ 2•𝑚 𝑛 -1⌉, these properties cannot hold, and therefore there is no pure Nash equilibrium. Conversely, we showthat if 𝑝 ≤ 1 ⌈ 2•𝑚 𝑛 -1⌉then we can construct a pure Nash equilibrium as follows.Given some strategy profile 𝑠 = (𝑑 1 , . . . , 𝑑 𝑛 ), let ℎ 𝑗 (𝑠) denote the number of documents assigned the same highest retrieval score for query 𝑞 𝑗 when 𝑠 is played. The strategy profile 𝑠 alg obtained by the following algorithm is a pure Nash equilibrium : In the case 𝑛 ≥ 𝑚, it is easy to see that a pure Nash equilibrium exists for every 𝑝. Specifically, w.l.o.g. each (disjoint) group of 𝑛 𝑚 players is assigned to an arbitrarily different query 𝑞 𝑗 among 𝑞 1 , . . . , 𝑞 𝑚 ; each player in this group writes a document 𝑑 with 𝑑 𝑗 = 𝑝 and 𝑑 𝑘 = 0 for 𝑘 ≠ 𝑗. Together with Theorem 3.4, we arrive to the final result: Corollary 3.5. G=⟨𝑛, 𝑚, 𝑝⟩ has a pure Nash equilibrium iff 𝑝 ≤1 max { ⌈ 2•𝑚 𝑛 -1⌉,1}.The best response of a player to the strategies played by all other players (i.e., their documents) is the strategy (document) that maximizes her utility. Iteratively finding best responses until convergence (if exists) -a process known as best response dynamicsis a standard mechanism for reaching Nash equilibria, specifically in potential games [27] . In these games, the best response dynamics is guaranteed to converge to an equilibrium. We now show that in our ranking games the best response dynamics does not necessarily converge. Hence, these are not potential games. An important implication is that in order to achieve stability (specifically, equilibrium), the game might need external intervention.Consider the game 𝐺 = ⟨𝑛, 𝑚, 𝑝⟩ for 𝑛 = 2, 𝑚 = 3 and 𝑝 = 1 2 . We show that the best response dynamics does not converge in this game, although by Theorem 3.3 an equilibrium exists. Let 𝑠 𝑡 denote the strategy profile played at round 𝑡 by two players. The following sequence of strategies is a best response dynamics given an initial strategy profile 𝑠 0 = ((0.3, 0.4, 0), (0.2, 0.3, 0.5)): 𝑠 1 = ((0.3, 0.4, 0), (0.4, 0.5, 0.1)), 𝑠 2 = ((0.5, 0.3, 0.2), (0.4, 0.5, 0.1)), 𝑠 3 = ((0.5, 0.3, 0.2), (0, 0.4, 0.3)). For example, since player 1's document at round 0 is (0.3, 0.4, 0) then player 2's document at round 1, (0.4, 0.5, 0.1), attains the maximal possible utility (3) as it outranks player 1's document for all three queries. Player 1's best possible utility given player 2's document from round 1, (0.4, 0.5, 0.1), is 2, which is obtained by writing, for example, a document (0.5, 0.3, 0.2) at round 2, and so forth. Since 𝑠 0 and 𝑠 3 are symmetric, the dynamics does not converge.The proof of Theorem 3.4 relies on the following Lemma, which asserts that the winning value in every query across the entire game must meet or exceed the peak value 𝑝. Lemma B.9. Let 𝐺 = ⟨𝑛, 𝑚, 𝑝⟩ such that 𝑛 < 𝑚 and 𝑠 be a pure equilibrium profile. Then for every query 𝑞 𝑗 ∈ 𝑄, 𝑤 𝑗 (𝑠) ≥ 𝑝 We begin by proving the Lemma for the following special scenario. Consider the case where there exist a query where the winning value is lower than the peak, and there is only one player winning in that query. We show that the winner in that query must be the unique winner in all the queries where he wins. Then, we use the upper bounds we derived on 𝑥 𝑖 (𝑠) to show there must exist a player with profitable deviation. The proof hinges on the interplay between the number of queries a player wins and their values in queries they do not win. Lemma B.10. Let 𝐺 = ⟨𝑛, 𝑚, 𝑝⟩ such that 𝑛 < 𝑚, 𝑝 ≤ 1 2 , and let 𝑠 be a pure equilibrium profile. Then for every query 𝑞 ′ 𝑗 where ℎ ′ 𝑗 (𝑠) = 1:𝑤 𝑗 ′ (𝑠) ≥ 𝑝Proof. Assume for contradiction there exists a query 𝑞 𝑗 1 where ℎ 𝑗 1 (𝑠) = 1 and 𝑤 𝑗 1 (𝑠) < 𝑝. Let 𝑖 1 be the sole winner of 𝑞 𝑗 1 .By the definition of 𝑦 𝑖 (𝑠), since 𝑖 1 is the only winner in query 𝑞 𝑗 1 :𝑦 𝑖 1 (𝑠) = 1 - ∑︁ 𝑞 𝑗 ∈ 𝐽 𝑖 1 (𝑠 ) 𝑤 𝑗 (𝑠 -𝑖 1 ) > 1 - ∑︁ 𝑞 𝑗 ∈ 𝐽 𝑖 1 (𝑠 ) 𝑤 𝑗 (𝑠) ≥ 1 - ∑︁ 𝑞 𝑗 ∈𝑄 𝑑 𝑗 𝑖 1 ≥ 0 (5)Next, assume by contradiction that there exists another query 𝑞 𝑗 2 ∈ 𝐽 𝑖 1 (𝑠) where ℎ 𝑗 2 (𝑠) > 1 -meaning 𝑖 1 is not the only winner in 𝑞 𝑗 2 . By Lemma B.5, if 𝑤 𝑗 2 (𝑠) < 𝑝 then the residual of player 𝑖 1 is zero, in contradiction to equation 5. Therefore, we get that 𝑤 𝑗 2 (𝑠) ≥ 𝑝 > 𝑤 𝑗 1 (𝑠).Consider 𝑖 𝑡 , another winner in 𝑞 𝑗 2 . Suppose 𝑖 𝑡 deviates by reallocating his resources from 𝑞 𝑗 2 to 𝑞 𝑗 1 , aiming to win solely in the latter. Such a deviation would make 𝑖 𝑡 the only winner in 𝑞 𝑗 1 , without affecting his rewards from other queries 𝑞 𝑗 ∈ 𝐽 𝑖 𝑡 (𝑠) \ {𝑞 𝑗 2 }. Thus, this deviation would be profitable for 𝑖 𝑡 , in contradiction to the assumption that 𝑠 is an equilibrium. Therefore, we conclude that 𝑖 1 must be the unique winner in all his winning queries.We define 𝐼 𝑢 as the set of players who are unique winners in all the queries in which they are ranked highest. Under the Theorem's conditions, 𝐼 𝑢 is non-empty. Let 𝛼 = |𝐼 𝑢 | represent the number of such winners, where 1 ≤ 𝛼 ≤ 𝑛.Let 𝑘 = 𝑤 𝑗 (𝑠 -𝑖 2 ) + 𝑝 < (𝛼 -1)• 𝑝 + 𝑝 = 𝛼 • 𝑝 ≤ 𝑘 • 𝑝 ≤ 1Rearranging, we obtain that the residual of player 𝑖 2 is larger that the peak value:𝑝 < 1 - ∑︁ 𝑞 𝑗 ∈ 𝐽 𝑖 2 (𝑠 )𝑤 𝑗 (𝑠 -𝑖 2 ) = 𝑦 𝑖 2 (𝑠)By Lemma B.4, player 𝑖 2 has a profitable deviation, in contradiction.For the case where 𝛼 ≥ 𝑘 + 1, we show a similar result. However, there's a more nuanced point to consider. The residuals not allocated to winning are distributed across multiple queries involving multiple players. If these residuals are focused on a single player, he might not have a profitable deviation. However, this distribution implies that another player will be minimally affected by others in the queries they lose, and he will have an incentive to deviate.Let 𝑖 𝑚 be a winner in query 𝑞 𝑚 . We bound 𝑥 𝑖 𝑚 (𝑠) using the upper bound from Lemma B.8,𝑥 𝑖 𝑚 (𝑠) ≤ 1 -𝑘 • 𝑤 𝑚 (𝑠)Recall that if ℎ 𝑚 (𝑠) ≥ 2, 𝑥 𝑖 𝑚 (𝑠) = 0. Now that we've bounded 𝑥 𝑖 (𝑠) for all players, we get the following: We now show that if 𝑝 > 1 2 there is no pure equilibrium in the game when 𝑛 < 𝑚.Lemma B.11. Let 𝐺 = ⟨𝑛, 𝑚, 𝑝⟩ such that 𝑛 < 𝑚 and 𝑝 > 1 2 . Then 𝐺 doesn't have a pure equilibrium.Proof. Following similar steps as in the proof of B.10, we can obtain that 𝑤 𝑚 (𝑠) ≤ 1 2 . If 𝑤 𝑚 (𝑠) > 1 2 then the number of queries in which each of the players win is exactly 1. However, this leads to 𝑛 𝑖=1 |𝐽 𝑖 (𝑠)| = 𝑛 < 𝑚, which means there exists a query where all players publish 𝑑 𝑗 𝑖 = 0. Every player would then have a profitable deviation, by contradiction.Thus, every player can win in no more than two queries, i.e. |𝐽 𝑖 (𝑠)| ≤ 2. Therefore 𝑛 < 𝑚 ≤ 𝑛 𝑖=1 |𝐽 𝑖 (𝑠)| ≤ 2 • 𝑛, and the minimal winning value is at least 1 2 : 𝑤 𝑚 (𝑠) ≥ 1 2 . There exists a player 𝑖 1 such that 𝑈 𝑖 1 (𝑠) > 1, therefore 𝑤 𝑚 (𝑠) = 1 2 . Overall, we get that 𝑥 𝑖 (𝑠) < 1 2 for every player 𝑖. So if 𝑤 1 (𝑠) > 1 2 , then the residual of the player 𝑖 2 that is the winner in query 𝑞 1 is:𝑦 𝑖 2 (𝑠) = 1 -𝑤 1 (𝑠 -𝑖 2 ) > 1 2= 𝑤 𝑚 (𝑠) By Lemma B.4, 𝑖 2 has a profitable deviation, by contradiction. In conclusion, 𝑤 1 (𝑠) = . . . = 𝑤 𝑚 (𝑠) = 1 2 . Thus, for all players: 𝑥 𝑖 (𝑠) = 0.Going back to player 𝑖 1 , recall that 𝑈 𝑖 1 (𝑠) > 1. Hence, 𝑖 1 is the unique winner in at least one of the two queries in which he wins. Therefore we get that the residual of 𝑖 1 is:𝑦 𝑖 1 (𝑠) > 1 2By Lemma B.4, 𝑠 is not an equilibrium -by contradiction. □ Following the specific case where a query had a sole winner with a winning value less than the peak value 𝑝, we now extend our analysis to cover all possible configurations of the strategy profile 𝑠. Specifically, we complete the proof for Lemma B.9: Lemma B.9. Let 𝐺 = ⟨𝑛, 𝑚, 𝑝⟩ such that 𝑛 < 𝑚 and 𝑠 be a pure equilibrium profile. Then for every query 𝑞 𝑗 ∈ 𝑄, 𝑤 𝑗 (𝑠) ≥ 𝑝 Proof. Let 𝑘 = 1 𝑝 , and assume by contradiction that 𝑤 𝑚 (𝑠) < 𝑝.According to Lemma B.11, if 𝑝 > 1 2 then the game doesn't possess a pure equilibrium. Given that 𝑠 is a pure equilibrium profile, it necessarily follows that 𝑝 ≤ Next, we show that if 𝑘 ≥ 2•𝑚 𝑛 -1, the game has an equilibrium, which can be achieved by Algorithm 1. Let 𝑠 alg denote the strategy profile obtained by Algorithm 1.At the conclusion of the algorithm, each player wins in exactly 𝑘 queries. Since 𝑘 ≥ 2•𝑚 𝑛 -1 , it ensures that at least one player selects each query 𝑞 𝑗 , resulting in 𝑤 𝑗 (𝑠) = 𝑝 for every query. Throughout the algorithm, players choose the query with the fewest current winners, maintaining the condition: max 𝑞 𝑗 ∈𝑄 ℎ 𝑗 (𝑠)min𝑞 𝑗 ∈𝑄 ℎ 𝑗 (𝑠) ≤ 1By the end of the algorithm, the number of winners in each query 𝑞 𝑗 is bounded by: 𝑛 • 𝑘 𝑚 ≤ ℎ 𝑗 (𝑠 alg ) ≤ 𝑛 • 𝑘 𝑚 Assuming a lexical tie-breaking rule, no player ends up being the sole winner in more than one query. If any player 𝑖 𝑡 were to deviate with a strategy 𝑑 𝑡 , they could not potentially win in more than 𝑘 queries due to 𝑤 𝑗 (𝑠) ≥ 𝑝 > 1 𝑘+1 . Introducing this strategy would only increase the number of winners in any query in which 𝑖 𝑡 didn't previously won by one. Consequently, such a change would not improve the player's utility, as it does not offer a better payoff than the current strategy. Thus, no deviation from 𝑠 alg yields a higher utility, and 𝑠 alg is a pure Nash equilibrium. □ 
ranking competitions: 
 Our next goal is to empirically analyze ranking competitions in which publishers (document authors) modify documents with respect to multiple queries (𝑚 > 1). To the best of our knowledge, there are no datasets that support this type of analysis. There are two publicly available datasets [12, 32] that resulted from ranking competitions. However, in these competitions, documents were modified with respect to a single query. Thus, we created a new dataset by organizing our own ranking competitions inspired by those organized in [12, 32] . International and institutional ethics committees approved the competitions. The participants signed a consent form and could withdraw at any time.In addition to having publishers compete for multiple queries rather than a single one, we introduced two novel aspects in our competitions. The first is the ranking function. While in previous competitions only a feature-based learning-to-rank (LTR) method was used [12, 32] , we also used a neural (NEU) ranker (BERT-based [28] ). The ranking functions are described below. The second aspect is the use of AI tools to modify documents in some of the competitions. In total, we organized four ranking competitions which differ by the ranking function applied and/or whether generative AI tools were allowed: LTR∧AI, LTR∧⌝AI, NEU∧AI, and NEU∧⌝AI. Table 1 provides an overview of the competitions. Each competition lasted for ten rounds; in each round, 30 games were held, each with respect to a different backstory (topic) from the UQV dataset [1] . 5 Each topic was represented using 𝑚 = 3 queries: the query selected to be the focus of the backstory and two additional query variants selected from all those provided for the backstory. The same 30 topics were used in all four competitions. Refer to Appendix A.1 for further details about the topic and query selection procedures.Competition structure. Before the start of the game, the participants were provided with three queries representing the topic of the game, the topic description (backstory) and an example of a relevant document. This document was generated using GPT-3.5, as described in Appendix A.1. The participants were then asked to write and submit their own document. In each round of the game, the participants were presented with the documents submitted in the previous round and the rankings induced over these documents for each of the three queries. They were encouraged to further modify their document to improve its ranking in the next round. All the documents were in plain text format, limited to 150 words. Refer to Appendix A.2 for detailed competition guidelines.A group of 84 undergraduate and graduate students enrolled in an information retrieval course participated in our competitions. Each student participated in three (repeated) games and submitted one document per game in a round. We made sure that the three games were selected from at least two different competitions and that there was no overlap between the topics of these games. Each game included exactly five participants: a combination of students and bots. Two or three students participated in every game together with two bots that used GPT-3.5 to create documents, as explained in Appendix A.3. If a game had only four participants, i.e., two students and two bots, we added a third static bot that published the initial example document in every round. To prevent any potential bias, the students' identities were anonymized, and they were not informed about the use of bots. Unless otherwise mentioned, our analysis is performed for the students' documents only.To encourage students to participate and modify their documents so as to attain high rankings in the competitions, we offered bonus points for their course grades. We note that the students could have earned a perfect grade in the course without participating in the competitions. The bonus per game was awarded based on the median reciprocal rank of a student's document across the three queries in the game. For example, if a student's document was ranked first, second, and third across the queries, they would earn half a point for that game. The points were accumulated over the student's three games per round and over the ten rounds.Our dataset includes a total of 2520 documents submitted by students (847 of which are unique), 2400 documents generated by bots (1833 of which are unique), and 30 documents that served as the initial document per topic. We will make the dataset publicly available upon the publication of this work.Ranking functions. As noted above, we experimented with two ranking functions in our competitions: LTR and NEU. For our learning-to-rank (LTR) approach, as in previous work [12, 32, 36] , we use LambdaMART [41] via the RankLib toolkit 6 . Unless otherwise specified, the implementation details of this model, including the training procedure and setting of hyper-parameter values, are the same as those of Vasilisky et al. [36] . The model was trained on the Combined dataset [36] , which includes data from the two previous ranking competitions [12, 32] . Each document was represented using a nine-dimensional feature vector. Some of the features are estimates of the similarity between the query and document, and the rest are query-independent document quality measures shown to be effective for Web retrieval [4] . The features are: (i) TF: sum of query term frequencies in the document, (ii) NormTF: TF normalized by the document's length, (iii) BM25: BM25 similarity between the query and document, (iv) LMIR: language-model-based similarity between the query and document7 , (v) BERT: BERT-based similarity between the query and document8 , (vi) LEN: document's length, (vii) FracStop: percentage of terms in the document that are stopwords9 , (viii) StopCover: percentage of stopwords on a stopword list that appear in the document, and (ix) ENT: entropy of the document's term distribution. For our neural ranking function (NEU), we used BERT as in feature (v) above. Thus, BERT served both as a feature in LTR and as a standalone ranking function. We note that the ranking functions were not disclosed to the students, who also did not know that four ranking competitions were held.Relevance and quality judgements. Each student document was judged by three crowd workers in CloudResearch's connect platform [15] for binary relevance to the topic and by five workers for content quality, with 0.75 and 0.49 inter-annotator agreement rates (free-marginal multi-rater Kappa), respectively. All workers were native English speakers. More than 97% of all student documents were judged relevant by at least two out of three annotators. This high percentage is in accordance with previous reports on competitions for a single query [13, 32] , albeit a bit elevated. Although students were not instructed to write relevant documents, the documents were short and students modified them for multiple queries representing the same topic. We note that our focus is on rankingincentivized manipulations and not on ranking effectiveness.Five annotators judged each student document for content quality using the categories: valid, keyword stuffed and spam (useless). When using AI tools, 89% and 71% of the documents were judged to be valid by at least three and at least four out of the five annotators, respectively. When not using AI tools, the percentages were 85% and 61%, respectively. This attests to the merit of using AI tools for rank promotion while maintaining content quality. Only 9% and 6% of the documents created with no AI tools and with AI tools, respectively, were judged as keyword stuffed by at least three annotators. Almost none of the documents were judged as spam. The distribution of quality judgments was not different between the LTR and NEU competitions. 
empirical analysis 5.1 analysis of strategies: 
 Feature Values. Inspired by work on the single-query setting [32] , we examine the document modification strategies in our multiplequeries setting: we analyze the changes in the documents' feature values along competition rounds. Recall that three document rankings were induced per topic, each for one of the three queries representing the topic. In this analysis, we examine each query (document ranking) separately. We focus on changes in documents that lost (i.e., were not ranked first) for at least four consecutive rounds before reaching the first rank (L). We compare the average feature values of the winners (i.e., the highest ranked documents) per query (W q ) with the average feature values of two types of losing documents (𝐿): those whose feature value was lower than or equal to that of the winning document four rounds prior to their win (L ≤ W q ), and those whose feature value was higher (L > W q ). We also present for reference the average feature values of the (at most three) winning documents per topic (W t ). Figure 1 presents the results for representative query-independent and query-dependent features of LTR. Figures for other features exhibit the same patterns and are omitted as they convey no additional insight.We see in Figure 1 that regardless of the ranking function (LTR or NEU), the use of generative AI tools (AI or ⌝AI) and the initial feature values (𝐿 ≤ 𝑊 𝑞 or 𝐿 > 𝑊 𝑞 ), with a few exceptions, the features of the losing documents tend to gradually converge to those of the winners. This finding is in line with results for the setting where publishers modified their documents for a single query [32] . Contrary to Raifer et al. [32] , we see that the feature values of the winning documents changed throughout the competition rounds. This might be attributed to the fact that documents in our competitions were modified for three different queries. For example, say a document was ranked first for some query. The student who wrote the document may have continued to modify it to improve its ranking for the other two queries, and thus affected its feature values. We delve deeper into this point in Section 5.2.We also note that changes of winners' feature values conceptually echo our game theoretic findings in Section 3: in the singlequery setting an equilibrium is guaranteed in contrast to the multiplequeries setting. Hence, the single-query setting is more likely to reach a stable state than the multiple-queries setting 10We observe in Figure 1a a general upward trend for the queryindependent features: LEN, StopCover, and ENT; the trends are more pronounced for LTR∧AI and LTR∧⌝AI. Hence, the documents became longer (LEN) with increased content diversity (ENT) and stopwords occurrence (StopCover). Interestingly, Raifer et al. [32] 1 : Averaged feature values of documents that lost in at least four consecutive rounds before winning. The documents are grouped based on whether their feature values four rounds before the win were lower than or equal to (𝐿 ≤ 𝑊 𝑞 ) or higher than (𝐿 > 𝑊 𝑞 ) the values of the winners for the given query. 𝑊 𝑞 and 𝑊 𝑡 : the averaged feature values of the corresponding winner per query and all the winners per the same topic, respectively. x-axis: the (negative) number of rounds leading up to a win. Note: in NEU∧AI and NEU∧⌝AI, when BERT is used as a standalone ranking function, the feature values of the losing documents are always lower than those of the winning document; therefore, 𝐿 > 𝑊 𝑞 is not shown. found a downward trend for ENT when students modified documents for a single query. We therefore hypothesize that when competing for a single query there is an attempt to focus the content for this query, while competing for multiple queries results in increased diversity so as to better cover the query set.We see in Figure 1b that the query-dependent feature values for 𝐿 ≤ 𝑊 𝑞 (where the initial feature value of the losing document was lower than or equal to that of the winner) increased along the rounds, whereas the values for 𝐿 > 𝑊 𝑞 (where the initial feature value of the losing document was higher than that of the winner) somewhat fluctuated and even decreased at times. This implies that in the former case, the students gradually increased the number of query term occurrences in their documents, but in the latter case, the students were careful not to add too many query terms, possibly to avoid keyword stuffing which they were warned about. Furthermore, this finding potentially suggests that from the point of view of students, the ranking function might be single peaked with respect to query term occurrence and the features based on it.Finally, we can see that the differences between 𝑊 𝑞 and 𝑊 𝑡 are smaller for LTR than for NEU. This finding might be attributed to the fact that for LTR, a winning document was more likely to win for multiple queries, resulting in fewer unique winning documents per topic. We present further support to this claim in Section 5.2.Prompts. To gain additional insights about the document modification strategies, we asked the students to share the prompts they used for the generative AI tools. We collected 44 prompts from rounds 8 to 10 of the competitions. The prompts typically included instructions to create a document that would be ranked high for the specified queries. All prompts, except for three, had no mention of the reward mechanism used in the competitions. As an example, one of the three prompts included the text: "Since the score is given by the median ranking among 3 queries, we can focus on maximizing the rankings on two queries. "Out of the 44 prompts, 29 included example documents: the initial example document (2 prompts), the student's document from the previous round (15 prompts), at least one of the winning documents from the previous round (24 prompts), or even all the documents in the previous round (5 prompts) . This further indicates that the students considered winning documents a valuable source of information about the undisclosed ranking functions.Thus far, we studied the document modification strategies in our competitions. We now shift our focus to examining the effect of the ranking functions used. We first analyze the differences between rankings induced by a function for the three queries representing a topic. We computed for each ranking function the RBO similarity (with 𝑝 = 0.9) [38] between the document rankings induced for each pair of queries per topic. We then averaged these values over the pairs per topic and over topics. Figure 2 presents the results for Previous research demonstrated BERT's sensitivity to document modifications in competitions held for a single query [36] .We further study the effect of the ranking functions on the competitions in Table 2 by examining documents that won for at least one of the queries per topic. We present the percentage of these documents that won for exactly one, two or three queries. Additionally, we present the average ranks of these documents in rankings induced for queries for which these documents did not win. We can see that for LTR, at least 65% of the winners won all three queries in a game, while for NEU, less than 9% did so. Furthermore, the average rank of documents for queries they did not win was, on average, higher when NEU was used than when LTR was used. 11These findings demonstrate that the ranking function used affects the ability of a document to win for multiple queries per topic. That is, a document that won for one of the queries is more likely to win for additional queries for the same topic in competitions that employed LTR compared to those that employed NEU.We allowed the use of generative AI tools in some of our competitions. We next compare the documents created by students who were allowed to use such tools (AI) with those who were not allowed (⌝AI). We use the cosine of tf.idf vectors to compute per topic the similarity between each document submitted by a student in a specific round and (i) documents in the previous round that won for at least one of the queries and (ii) all the other documents from the previous round. We averaged the similarities over the documents and grouped them by whether or not the students were allowed to use AI tools. We see in Figure 3 that there were noticeable differences in the similarity values of AI and ⌝AI. Specifically, the documents created without the help of AI tools in LTR∧⌝AI and NEU∧⌝AI were much more similar to documents submitted in the previous round, whether they were winners or not, than those created with the help of such tools in LTR∧AI and NEU∧AI. This finding indicates that using AI tools may lead to the creation of more diverse content. We also observe higher similarities with the winning documents than with the losing ones for both AI and ⌝AI, further supporting our earlier findings that the documents, whether they were directly modified by students or generated using AI tools, tended to converge to the winners. 
predicting winners: 
 In this section, we address the challenge of predicting which document among all those that did not rank first in round 𝑙 -1 will be ranked the highest (win) in round 𝑙 if the winner indeed changes. The documents in our competitions were modified with respect to 𝑚 = 3 queries in the query set 𝑄 for a topic; yet, the prediction is performed separately for each query 𝑞 ∈ 𝑄. A similar challenge was previously tackled for competitions held for a single query [32] .For prediction, we represent each document as a 27-dimensional feature vector. The features we use can be divided into seven sets. The first two sets, Micro (12 features) and Macro (3 features), were proposed by Raifer et al. [32] . These features model properties of the document in round 𝑙 (𝐷 𝑙 ), the document published by the same author in round 𝑙 -1 (𝐷 𝑙 -1 ), and the winning document in round 𝑙 -1 for the query 𝑞 (𝑊 𝑞 𝑙 -1 ). The features in Micro quantify the changes in 𝐷 𝑙 with respect to 𝑊 𝑞 𝑙 -1 by counting term additions and deletions. These features are defined for three groups of unique terms: query terms, stopwords, and terms that are neither query terms nor stopwords. For each group of terms, four features were defined: the number of terms in 𝑊 𝑞 𝑙 -1 that were either (i) added to or (ii) deleted from 𝐷 𝑙 -1 , and the number of terms that were not in 𝑊 𝑞 𝑙 -1 and were either (iii) deleted from or (iv) added to 𝐷 𝑙 -1 . The features in Macro are the similarities between 𝐷 𝑙 , 𝐷 𝑙 -1 , and 𝑊 𝑞 𝑙 -1 ; the similarity between a pair of documents was computed using the cosine of their tf.idf vector representations.The features just discussed only consider the winner for the given query (𝑊 𝑞 𝑙 -1 ), disregarding the rankings induced for the other two queries and their corresponding winners. Accordingly, we introduce an additional set of features, TopicMacro, that consider the group of all winners for the three queries in 𝑄 (𝑊 𝑄 𝑙 -1 ). We compute the similarities between 𝐷 𝑙 and 𝑊 𝑄 𝑙 -1 and between 𝐷 𝑙 -1 and 𝑊 𝑄 𝑙 -1 . We use cosine to compute the similarities and represent the documents in 𝑊 𝑄 𝑙 -1 via the centroid of their tf.idf vectors. In the following two feature sets, we quantify properties of 𝐷 𝑙 -1 's ranks in the three rankings induced for all three queries per topic. The first set, TopicRank, includes three features: the minimum, median, and maximum rank of 𝐷 𝑙 -1 across the three queries in 𝑄. The second set, QueryRank, includes three binary features that are set to 1 if 𝐷 𝑙 -1 's minimum, median, and maximum rank for the three queries is equal to its rank for the given query 𝑞; otherwise, they are set to 0.The next feature set, PrevChange, includes three features that quantify past changes (improvements) in the document's ranks. The underlying assumption is that if a publisher improved her document's rank in round 𝑙 -1, she might do so again in round 𝑙. To this end, we also examine the publisher's document in round 𝑙 -2 (𝐷 𝑙 -2 ). For each of the three queries, we compute the difference between the ranks of 𝐷 𝑙 -1 and 𝐷 𝑙 -2 , and scale it by the maximal possible rank change given 𝐷 𝑙 -2 's rank in round 𝑙 -2. We use the minimum, median, and maximum values across the three queries.Finally, we found in Section 5.1 that the documents tended to become longer throughout the four competition rounds leading to a win; accordingly, the feature in the final set, Len, is 𝐷 𝑙 's length.Our dataset includes documents submitted by both students and bots. We discarded queries for which a bot generated the winning document. In addition, we discarded queries for which the winners in round 𝑙 -1 (𝑊 𝑞 𝑙 -1 ) and round 𝑙 (𝑊 𝑞 𝑙 ) were generated by the same publisher. This was done because we aimed to predict which of the losing documents would win in the following round. As a result, the number of queries per competition round ranged from 3 to 39.We experimented with four different classifiers with the features defined above via the scikit-learn library [31] : logistic regression (LReg), linear SVM (LSVM), polynomial SVM (PSVM) and random forests (RForest). We applied min-max normalization to feature values per query. The document assigned the highest prediction score per query by a classifier was deemed the winner; the remaining documents were considered losers.The features in PrevChange can only be computed when data for at least two previous rounds is available; therefore, our experiments were conducted for rounds 3 to 10. We used leave-one-out crossvalidation over rounds to train our models and select parameter values. We used the queries from one of the rounds for testing and those from the remaining seven for training and validation. We repeatedly trained a model on data from six of the seven rounds with different parameter configurations and validated the effectiveness over the seventh round. We selected the parameter values that yielded the highest prediction effectiveness on average across the seven validation rounds. Then, we trained a final model using data from all seven rounds and applied it to the held-out test round (fold). We repeated this entire procedure for each test fold. The models were trained separately for each of the four competitions. We use Acc (percentage of correctly classified winners and losers) and F1 (harmonic mean of precision and recall), averaged over queries and test folds, to measure prediction effectiveness; the former served to select parameter values. Statistically significant prediction differences were determined across queries using the two-tailed paired t-tests with 𝑝 ≤ 0.05. LReg, LSVM and PSVM were trained with L1 regularization; the regularization parameter was selected from {1, 10, 50, 100}. The degree of the polynomial kernel in PSVM was selected from {2, 3, 4, 5}. For RForest, the number of trees and leaves were set to values in {10, 50, 100, 500} and {10, 20, 30}, respectively. All other parameters were set to default values [31] .We compare in Table 3 the prediction effectiveness of our four classifiers (LReg, LSVM, PSVM, and RForest) with that of five baselines12 : (i) Random (Rand) : the winner is randomly selected, (ii) QueryMajority (QMaj) : the winner is the document whose publisher had the highest number of past wins for the given query; ties were broken randomly, (iii) TopicMajority (TMaj): the winner is the document whose publisher had the highest number of past wins for all three queries of the given topic; ties were broken randomly, (iv) AllWinners (AllW): all the documents are predicted to be winners (only one document is correctly classified per query), (v) AllLosers (AllL): all the documents are predicted to be losers (all but one of the documents are correctly classified per query). We also report the effectiveness of the classifiers when using only the two feature sets proposed by Raifer et al. [32] (Micro and Macro). For this baseline, we show the results only for the classifier (LReg, LSVM, PSVM, or RForest) with the highest Acc per competition.The documents in our competitions were modified with respect to three queries. Table 3 shows that when using only the two feature sets proposed by Raifer et al. [32] , which use information only about the query at hand, the performance surpasses that of all baselines. This attests to the effectiveness of these feature sets, whether the documents are modified with respect to a single query as in [32] or with respect to multiple queries as in our setting. Furthermore, we see that when adding features that quantify various properties of all three queries and the corresponding rankings, all our classifiers consistently and statistically significantly outperform all the baselines, including that of Raifer et al. [32] . This attests to the merits of using information about other queries for the same topic to predict for a given query.To further examine the effectiveness of the different feature sets, we performed ablation tests, removing one of the sets each time. We anzlye the results for PSVM, for which the best overall performance was attained for two of the four competitions in Table 3 . Actual numbers are omitted as they convey no additional insight and due to space considerations. Except for Macro, removing each feature set results in at least one statistically significant drop in performance, attesting to the complementary nature of our features. Comparing Macro and TopicMacro, the drop for the latter is more substantial. These findings suggest that these two feature sets might be somewhat correlated and that TopicMacro, which considers the winners for all queries of the topic in the previous round, is somewhat more informative in our setting. 
conclusions: 
 We presented a novel theoretical and empirical study of the competitive retrieval setting where document authors modify documents to improve their ranking for multiple queries. We showed using game theory that in contrast to past work on document modifications for a single query, in the multiple-queries setting there is not necessarily an equilibrium and characterized the cases when it exists. We organized novel ranking competitions where publishers modified documents for multiple queries representing the same topic. We used both neural and non-neural rankers, and allowed the use of generative AI tools in some competitions. We found that publishers use content from documents highly ranked in the past, but to a somewhat reduced extent when they apply AI tools. Finally, we addressed the task of predicting which document will be promoted to the highest rank in the next ranking round. We demonstrated the merits of using information induced from other queries representing the same topic to this end. 
 
 Appendix: We selected from the UQV dataset [1] topics with commercial intent that were more likely to incite competition 13 . For each topic, three queries were used. The first query was the focus of the backstory, while the other two were selected from all the available query variants as follows. To ensure that the variants were of high quality, we retrieved 1000 documents per variant from the ClueWeb12 category B corpus using LMIR via the Indri toolkit 14 . We filtered out variants with average precision (AP) lower than 0.05. In addition, to ensure that the queries were diverse enough, we computed the Jacquard similarity between the terms of each pair of queries and the RBO similarity (with 𝑝 = 0.9) [38] between the corresponding retrieved document lists. These two scores were combined using Reciprocal Rank Fusion (RRF) with 𝑘 = 60 [9] . Finally, from the remaining query variants, we iteratively selected the following query with the lowest average similarity to the already selected queries.We provided the students with an example of a relevant document for each topic. We generated these documents using GPT-3.5 with a prompt per topic that included the backstory and the three selected queries. We continued generating the document until it was deemed relevant by four annotators. We used the same document per topic across all four competitions, resulting in 30 documents.The students were instructed to create high quality plain text documents without using any formatting tags, hyperlinks, or special characters. They were advised to avoid using slang or informal language and were warned against keyword stuffing. We mentioned that using such techniques would result in a penalty, although no actual penalty was imposed. Copying parts of other students' documents or Web pages was allowed as long as the document also included original content. Documents that were copied completely from other documents received no reward. We noticed after the third round that some students were submitting the initial example document without modifications. Thus, we prohibited submitting the initial documents starting from the fourth round. Students who still used these documents did not receive points. We emphasized that the goal was to promote documents in rankings regardless of their relevance to the information needs expressed by the queries. 13 The selected topics are 201, 203, 204, 209, 210, 211, 212, 218, 226, 228, 233, 235, 244, 245, 246, 249, 250, 252, 255, 258, 261, 262, 268, 272, 274, 281, 283, 289 , 291, and 296. 14 www.lemurproject.org/indriWe used bots in our competitions to increase the number of players and to make the games more dynamic. The bots generated documents using GPT-3.5. The prompts asked to create documents no longer than 150 terms, but some of the generated documents were longer. In these cases, we removed sentences from the end of a document until its length was between 140 and 150 terms. If no sentence could be removed without making the document shorter than 140 terms, we truncated the last sentence to 150 terms.We provided our bots with the backstory, the three queries representing the backstory, and the content and ranks of the bot's document in the previous round. In addition, we provided information about other submitted documents that varied across bots based on two parameters. The first parameter determined which of the documents submitted in a round were accessible to the bot. We tested bots that had access to all submitted documents (including their ranks), the document with the highest median rank across the three queries (excluding the actual ranks), and the bot's document only. The second parameter determined the number of rounds for which the above information was made available: last round or last three rounds. We used six bot configurations in our competitions, each corresponding to combinations of the two parameters. Each bot was assigned to ten different topics and competed in all ten rounds of the game.The documents generated by bots were generally ranked lower than those written by students. The average rank for the bots' documents was 3.2, while the students' documents had an average rank of 2.6. Moreover, the bots' documents won less: 67.9% of the winning documents were written by students, compared to only 23.5% generated by bots. 15Theorem 3.3. If 𝑛 = 2 and 𝑚 > 𝑛, then the game 𝐺 has a pure Nash equilibrium iff 𝑝 ≤ 1 𝑚-1 . Proof. Assume there exists some strategy profile (𝑑 1 , 𝑑 2 ) that is a pure Nash equilibrium. Let 𝑑 2 = (𝑑 < 𝑝. Define 𝑑 𝑡 ∈ 𝐷 such that 𝑑 𝑖 𝑡 = min{𝑑 𝑖 2 + 𝑑 1 2 2 , 𝑝}, 𝑑 𝑗 𝑡 = min{𝑑 𝑗 2 + 𝑑 1 2 2 , 𝑝}, 𝑑 1 𝑡 = 0 and 𝑑 𝑘 𝑡 =≤ 𝑚 𝑙=1 𝑑 𝑙 2 ≤ 1.If player 1 writes 𝑑 𝑡 , then 𝑑 𝑡 will be ranked (i) first (solely) for 𝑞 𝑖 and 𝑞 𝑗 , (ii) below the first rank for 𝑞 1 , (iii) the same as 𝑑 2 for the rest of the queries. Thus, if player 1 writes 𝑑 𝑡 her utility is 𝑚-3 2 +2 = 𝑚+1 2 . By Lemma 3.2, if (𝑑 1 , 𝑑 2 ) is a pure equilibrium then the utility of player 1 is 𝑚 2 . Therefore, the move from writing 𝑑 1 to writing 𝑑 𝑡 is beneficial for player 1, which is a contradiction to the fact that (𝑑 1 , 𝑑 2 ) is a Nash equilibrium. Thus, we conclude that if (𝑑 1 , 𝑑 2 ) is a pure Nash equilibrium then there is at most one query 𝑞 𝑗 such that 𝑑 𝑗 2 < 𝑝. Now, if 1 𝑚-1 < 𝑝, then there exists no such document 𝑑 2 ∈ 𝐷. Therefore, 𝑝 ≤ 1 𝑚-1 . Hence, we proved that if there is a Nash equilibrium, 𝑝 ≤ 1 𝑚-1 . We now turn to show that for any 𝑝 ≤ 1 𝑚-1 there is a pure Nash equilibrium.If 𝑝 ≤ 1 𝑚 , then by Lemma 3.1 the profile where both players publish 𝑑 = (𝑝, . . . , 𝑝) is a pure Nash equilibrium. If 1 𝑚 < 𝑝 ≤ 1 𝑚-1 , the profile where 𝑑 1 = 𝑑 2 = (𝑝, . . . , 𝑝, 1 -𝑝 • (𝑚 -1)) is a pure Nash equilibrium as we show next. (Note that (𝑚 -1) • 𝑝 ≤ 1 and therefore the strategies (𝑑 1 and 𝑑 2 ) are well defined.) W.l.o.g. assume that player 1 has an incentive (in terms of increased utility) to move from writing 𝑑 1 to writing some 𝑑 𝑡 . For every query 𝑞 𝑗 s.t. 𝑗 < 𝑚, if 𝑑 𝑗 𝑡 ≠ 𝑝, then 𝑑 𝑡 cannot be ranked first because the retrieval function reaches its single peak at 𝑝. Thus, the only way to increase utility would be to have 𝑑 𝑡 ranked first for query 𝑞 𝑚 . This can only be achieved by increasing the emphasis on 𝑞 𝑚 but as a result decreasing the emphasis on some other query 𝑞 𝑗 , where 𝑗 < 𝑚. Consequently, 𝑑 𝑡 will be ranked (i) first (solely) for query 𝑞 𝑚 , (ii) ranked second for query 𝑞 𝑗 (iii) the same as 𝑑 2 for the rest of the queries. Therefore, player's 1 utility will be 𝑚-22 + 1 = 𝑚 2 , which is the same as her utility when writing 𝑑 1 ; i.e., there was no incentive to move to 𝑑 𝑡 . □Notations. Let 𝑤 𝑗 (𝑠) represent the value of the document ranked highest in query 𝑞 𝑗 ∈ 𝑄 under the strategy profile 𝑠. We use 𝑠 -𝑖 to denote the strategies of all players except player 𝑖, and 𝑤 𝑗 (𝑠 -𝑖 ) to denote the highest value of the documents in query 𝑞 𝑗 when player 𝑖 doesn't participate. Hence, if player 𝑖 is not ranked highest in query 𝑞 𝑗 , or if the highest rank is shared among multiple players, then 𝑤 𝑗 (𝑠 -𝑖 ) = 𝑤 𝑗 (𝑠). However, if player 𝑖 uniquely tops the ranking in query 𝑞 𝑗 , then 𝑤 𝑗 (𝑠 -𝑖 ) < 𝑤 𝑗 (𝑠).Player 𝑖 is called a winner in query 𝑞 𝑗 if 𝑑 𝑗 𝑖 = 𝑤 𝑗 (𝑠). Define 𝐽 𝑖 (𝑠) as the set of queries in which player 𝑖 is a winner under strategy profile 𝑠:𝐽 𝑖 (𝑠) ≔ {𝑞 𝑗 ∈ 𝑄 : 𝑑 𝑗 𝑖 = 𝑤 𝑗 (𝑠)} We begin by identifying a set of strategies in the game that are weakly dominated. This initial step enables us to focus solely on strategies in subsequent proofs where 𝑑 𝑗 𝑖 ≤ 𝑝 for every player 𝑖 and query 𝑞 𝑗 ∈ 𝑄.Lemma B.1. For every player 𝑖, publishing a document 𝑑 with 𝑑 𝑗 > 𝑝 for some query 𝑞 𝑗 ∈ 𝑄 is weakly dominated.Proof. Let 𝑑 𝑖 be a strategy of player 𝑖, where max 𝑞 𝑗 ∈𝑄 𝑑 𝑗 𝑖 > 𝑝. Consider the strategy 𝑑 𝑡 which is defined by𝑑 𝑗 𝑡 = min{𝑑 𝑗 𝑖 , 𝑝}.In queries where 𝑑 𝑗 𝑖 > 𝑝, the document 𝑑 𝑡 will be ranked first by 𝑟 . There will be no change in the other queries. Therefore, 𝑈 𝑖 (𝑠 -𝑖 , 𝑑 𝑖 ) ≤ 𝑈 𝑖 (𝑠 -𝑖 , 𝑑 𝑡 ) and 𝑑 𝑖 is weakly-dominated by 𝑑 𝑡 . □To facilitate the analysis of the game dynamics, we introduce some key definitions. The residual of a player represents their maximum potential unused resource when deviating to another strategy profile that allows them to maintain victory in the same set of queries. Formally, Definition B.2 (Residual). The residual of player 𝑖 in the game 𝐺 = ⟨𝑛, 𝑚, 𝑝⟩ under the strategy profile 𝑠 is given by:𝑦 𝑖 (𝑠) ≔ 1 - ∑︁ 𝑞 𝑗 ∈ 𝐽 𝑖 (𝑠 )𝑤 𝑗 (𝑠 -𝑖 )The residual can be equivalently calculated by: Lemma B.3. Let 𝐺 = ⟨𝑛, 𝑚, 𝑝⟩ and 𝑠 be some pure strategy profile.1 -𝑦 𝑖 (𝑠) = min 𝑑 ′ ∈𝐷 𝑖        ∑︁ 𝑞 𝑗 ∈𝑄 𝑑 ′ 𝑗 𝐽 𝑖 (𝑠) = 𝐽 𝑖 (𝑠 -𝑖 , 𝑑 ′ )       This expression computes the minimal total sum of document values ( 𝑞 𝑗 ∈𝑄 𝑑 𝑗 ) needed for player 𝑖 to continue winning the same set of queries when deviating from their current strategy. Although the set of winning queries remains unchanged, the utility for player 𝑖 might vary because the number of winners in each query could increase.The following lemma proposes a necessary condition for the existence of a pure equilibrium by considering the residual of a player relative to the minimal value of the winner among all queries where they are currently not winning. A larger residual indicates an opportunity for a profitable deviation, as it reflects the presence of unused resources that could be strategically allocated to win at least one additional query. Lemma B.4. Let 𝐺 = ⟨𝑛, 𝑚, 𝑝⟩ and 𝑠 be some pure strategy profile. For every player 𝑖, if𝑦 𝑖 (𝑠) > min{ min 𝑞 𝑗 ∉𝐽 𝑖 (𝑠 ){𝑤 𝑗 (𝑠)}, 𝑝} then 𝑖 has a profitable deviation, indicating that 𝑠 is not an equilibrium.Proof. Consider a player 𝑖 1 for whichEQUATIONWe utilize the equivalent definition of the residual from Lemma B.3. Define 𝑑 as:𝑑 = arg min 𝑑 ′ ∈𝐷 𝑖 1        ∑︁ 𝑞 𝑗 ∈𝑄 𝑑 ′ 𝑗 𝐽 𝑖 1 (𝑠) = 𝐽 𝑖 (𝑠 -𝑖 1 , 𝑑 ′ )        Let 𝑞 𝑗 𝑡 = arg min 𝑞 𝑗 ∉𝐽 𝑖 1 (𝑠 ) {𝑤 𝑗 (𝑠 -𝑖 1 )}.Consider a deviation to 𝑑 𝑡 defined by:EQUATIONwhere𝜖 0 = 𝑦 𝑖 1 (𝑠 ) -𝑤 𝑗 𝑡 (𝑠 -𝑖 1 ) | 𝐽 𝑖 1 (𝑠 ) |+1.To validate the deviation, we demonstrate that 𝑞 𝑗 ∈𝑄 𝑑 𝑗 𝑡 ≤ 1:EQUATIONAccording to the definition of 𝑑 𝑡 , for every query 𝑞 𝑗 ∈ 𝐽 𝑖 1 (𝑠), it follows that 𝑞 𝑗 also belongs to 𝐽 𝑖 1 (𝑠 -𝑖 1 , 𝑑 𝑡 ). After the deviation, the number of winners (ℎ 𝑗 (𝑠)) in queries in 𝐽 𝑖 1 (𝑠) can only decrease, leading to an increase in player 𝑖 1 's reward for them. Furthermore, player 𝑖 1 is now ranked first in query 𝑞 𝑗 𝑡 . As a result, player 𝑖 1 experiences a minimum increase in overall utility by at least 1 𝑛 , making the deviation profitable. □ Consider a scenario in a game where a player is one of multiple winners (i.e., ℎ 𝑗 (𝑠) ≥ 2) for a particular query 𝑞 𝑗 . If the winning score 𝑤 𝑗 (𝑠) is not at the peak value 𝑝, a minor strategic adjustment could enable him to outscore other players, thus becoming the sole winner. Such a shift would increase his utility since he would no longer share the reward for the query. Consequently, at equilibrium, this player must have strategically allocated all available resources to maximize his outcome across contested queries. This implies that his residual resources should be zero. Proof. Assume by contradiction that there exists a player 𝑖 1 ∈ 𝐼 𝑡 with a residual 𝑦 𝑖 (𝑠) > 0, indicating that they have unutilized resources. Consider the following strategic deviation:EQUATIONBy definition of 𝐼 𝑡 , there exists a query 𝑞 𝑗 1 ∈ 𝐽 𝑖 1 (𝑠) for which 𝑤 𝑗 1 (𝑠) < 𝑝 and ℎ 𝑗 1 (𝑠) ≥ 2. Player 𝑖 1 's deviation would increase his score sufficiently to potentially become the unique winner in query 𝑞 𝑗 1 , thereby increasing their utility by at least 1 2 . This gain in utility contradicts the assumption of equilibrium, thus proving that 𝑦 𝑖 (𝑠) = 0 for all 𝑖 ∈ 𝐼 𝑡 . □The definition of the residual allows us to measure the quantity of available resources that a player has when playing each strategy. We introduce another strategic metric, 𝑥 𝑖 (𝑠), which quantifies the resources that a player allocates to queries they do not currently win. Formally, 𝑥 𝑖 (𝑠) ≔ ∑︁𝑞 𝑗 ∉𝐽 𝑖 (𝑠 ) 𝑑 𝑗 𝑖This metric measures the extent to which a player invests in preventing other winners from significantly reducing the winning value, thus establishing a floor on the resources required for others to secure a win in those queries. Although 𝑥 𝑖 (𝑠) does not directly impact player 𝑖's own utility, it plays a critical role in limiting the potential for other players to deviate. By strategically allocating his residual resources, a player can influence the residual values of other players, thereby affecting their ability to deviate profitably from their current strategies.We now derive an upper bound for 𝑥 𝑖 (𝑠), stating that the amount of resources a player uses in queries where he is the loser is no more than his overall available resources. 𝑤 𝑗 (𝑠 -𝑖 ) + 𝑥 𝑖 (𝑠)Rearranging, we get that the residual of player 𝑖 satisfies:𝑦 𝑖 (𝑠) = 1 - ∑︁ 𝑞 𝑗 ∈ 𝐽 𝑖 (𝑠 )𝑤 𝑗 (𝑠 -𝑖 ) ≥ 𝑥 𝑖 (𝑠)The proof theorem 3.4 relies on two key lemmas. The first states that in an equilibrium, the winner's value must be at least 𝑝. The second lemma states that every player is the unique winner in no more than one of his winning queries. In order to prove the first lemma, we begin by lower bounding the number of queries in which a player wins in equilibrium. Lemma B.7. Let 𝐺 = ⟨𝑛, 𝑚, 𝑝⟩ and 𝑠 be a pure equilibrium profile. Let 𝑘 ∈ N s.t. 𝑑 𝑗 𝑡 =          min{𝑑 𝑗 𝑖 , 𝑝}, 𝑞 𝑗 ∈ 𝐽 𝑖 1 (𝑠) 𝑝, 𝑞 𝑗 = 𝑞 𝑗 1 0, otherwiseGiven that 𝑝 ≤ 1 𝑘 , the deviation is feasible:EQUATIONPlayer 𝑖 1 maintains their winning status in all queries 𝑞 𝑗 ∈ 𝐽 𝑖 1 (𝑠) under 𝑑 𝑗 𝑡 , and now potentially wins 𝑞 𝑗 1 as well. Moreover, ℎ 𝑗 (𝑠) ≥ ℎ 𝑗 (𝑠 -𝑖 1 , 𝑑 𝑡 ) for every 𝑞 𝑗 ∈ 𝐽 𝑖 1 (𝑠). Therefore, 𝑖 1 's utility after deviation is: 𝑈 𝑖 1 (𝑠 -𝑖 , 𝑑 𝑡 ) = ∑︁ 𝑞 𝑗 ∈ 𝐽 𝑖 (𝑠 -𝑖 ,𝑑 𝑡 ) 1 ℎ 𝑗 (𝑠 -𝑖 1 , 𝑑 𝑡 ) = ∑︁ 𝑞 𝑗 ∈ 𝐽 𝑖 (𝑠 ) 1 ℎ 𝑗 (𝑠 -𝑖 1 , 𝑑 𝑡 ) + 1 ℎ 𝑗 1 (𝑠 -𝑖 1 , 𝑑 𝑡 ) ≥ ∑︁ 𝑞 𝑗 ∈ 𝐽 𝑖 (𝑠 ) 1 ℎ 𝑗 (𝑠) + 1 ℎ 𝑗 1 (𝑠 -𝑖 1 , 𝑑 𝑡 ) = 𝑈 𝑖 1 (𝑠) + 1 ℎ 𝑗 1 (𝑠 -𝑖 1 , 𝑑 𝑡 ) > 𝑈 𝑖 1 (𝑠)EQUATIONApplying Lemma B.7, we know |𝐽 𝑖 (𝑠)| ≥ 1 𝑝 . Thus, plugging into Equation 4,𝑥 𝑖 (𝑠) ≤ 1 -|𝐽 𝑖 (𝑠)| • 𝑤 𝑚 (𝑠) ≤ 1 - 1 𝑝 • 𝑤 𝑚 (𝑠) □ B.In reference to work on focused retrieval[10], 𝑑 𝑗 can be defined as the portion of the text in 𝑑 that is marked as relevant to query 𝑞 𝑗 .There is also a notion of mixed strategies where players apply a distribution over pure strategies. Herein, we focus on pure strategies.Recall that the retrieval function has a single peak at 𝑝.The dataset includes query variants for TREC's 2013-2014 Web track topics (201-300).https://sourceforge.net/p/lemur/wiki/RankLib/Unigram language models with Dirichlet smoothing (𝜇 = 1000) are used.We used BERT-Large fine-tuned for passage ranking on MS MARCO[28].The NLTK stopword list was used in our experiments: www.nltk.org/nltk_data/.We hasten to point out that the game theoretic analysis was performed for single peak functions. The rankers used in the competition are not single peaked.The rank of the highest-ranked document is 1. Increased rank means lower ranking.Except for TopicMajority, all the baselines were also used by Raifer et al.[32].The percentages do not add up to 100 as we excluded from this analysis the static bots that submitted the initial document in all competition rounds.Conference'17, July 2017, Washington, DC, USA Anon. 


Decision: This paper proposes an extension of a previous approach proposed by the authors, which has the objective of incentivizing the modification of documents by their authors by means of improved rankings. The formal ground is game theory. The proposed extesion applies to considering multiple queries instead of single query settings.
The paper reports interesting researcg, which can raise discussions in tha audience. The  presentation needs some clarifications to improve the final version of the manuscript: 1) better  explain the link between the model related formulation and the experimental part; 2) some aspects related to the experimental evaluations need clarification (e.g. crowdsourcing); 3) in general the authors are recommended to account for the reviewers’ comments in preparing the final version of the paper.

Reviews: review: main_review: main_review: What I like to see in the paper is an explicit list of research questions that clearly sets the expectations straight from the experiments and their results. What is it that we want to verify empirically and how much of that relates back to the theory should be clearly described.

Moreover, while I understand that the paper is different from [12, 32] because it models multiple queries, there should be a more clear comparison on the analysis proposed in [12, 32] vs. the ones proposed in this paper. For example, does the current paper, in some sense, generalize the theoretical analysis framework of [12]? If so how? What are the new variables and the new interactions between them? What are the new assumptions?


About the experiments, with human subjects - I see a potential ethics red flag because the following, IMO, doesn’t appear to be fair to the students.

“To encourage students to participate and modify their documents so as to attain high rankings in the competitions, we offered bonus points for their course grades."  --- this, IMO, is putting stress on the students and where is the guarantee that such a stressful environment will actually result in a productive outcome?
main_review: main_review: Overall, the paper is well-structured, and the discussed topic is relevant to the ICTIR community. The method is original and the corresponding experimental results are sufficient and bring some interesting insights.
main_review: main_review: I think generally the strength of this paper is its extensive comparisons and experiments. Many different metrics, settings and configuration were tried and reported on. With the rationale that parties doing document optimization (e.g. SEO companies) are also using AI tools nowadays, these were also included in the study. The difference in document modifications with vs without AI tools was an interesting extra finding. While the number of abbreviations in the paper is a bit staggering, everything is clearly defined and referred to.
main_review: main_review: I'd like to suggest a few points for the authors to consider:

- Incorporating examples and figures could enhance comprehension for readers outside this field, although the current presentation is adequate.
- There seems to be a significant disconnect between the theories presented in Section 3 and the subsequent experiments. While each component is valuable on its own, their integration does not clearly support the paper’s conclusions. It would be beneficial to illustrate how these elements interact more effectively.
- The paper mentions using crowd workers to evaluate the relevance of documents written by students, yet I did not observe these evaluations being utilized in the experiments. Could you clarify if I've overlooked something?
- Most claims and conclusions in the paper lack explicit implications for the field, which is surprising given the relevance of this work to modern search engines. It might be fruitful to discuss potential implications for both publishers and search engine providers, possibly suggesting ways they could intervene.