SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The TvN game models a scenario where the user population comprises multiple interest groups, each with orthogonal preference representations. In this game, the largest group consists of nearly half the population. Each content creator has the option to cater to one—and only one—user group. While this game is simple and stylized, it captures the essence of real-world user populations and the dilemmas faced by creators. Creators often find themselves at a crossroad: they must decide whether to pursue popular trends for a broader audience population, leading to intense competition, or focus on niche topics with a smaller audience and reduced competition. Our subsequent result shows that if the platform adopts  $\mathcal{M}^3$  in the TvN game, this tension of content creation turns out to be a curse in the sense that a unique PNE is achieved when all players opt for the same strategy — catering to the largest user group — and we quantify the social welfare loss at this PNE in the following.
Theorem 1. For any rewarding mechanism  $M \in \mathcal{M}^3$  applied to any TvN instance, we have
1. the resultant game admits a unique NE  $s^*$ ;
2. the welfare of this NE is at most  $\frac{K}{K + 1}$  fraction of the optimal welfare for large  $n$ . Formally,
$$
\frac {W \left(\boldsymbol {s} ^ {*}\right)}{\max  _ {\boldsymbol {s} \in \mathcal {S}} W (\boldsymbol {s})} \leq \frac {K}{K + 1} + O \left(\frac {1}{n}\right). \tag {4}
$$
The proof is in Appendix 8.4, where we explicitly characterize both  $s^*$  and the welfare maximizing strategy profile and calculate their difference in terms of welfare. It is worthwhile to point out that the reciprocal of left-hand side of (4) is commonly known as the Price of Anarchy (PoA). This metric gauges the welfare loss at equilibrium compared to optimal welfare. (4) suggests that the PoA of  $\mathcal{G}$  under  $\mathcal{M}^3$  could be as significant as  $1/2$  for users who primarily care about the top relevant content, which is shown to be realistic given the diminishing attention spans of Internet users [6]. This theorem shows that no mechanisms in  $\mathcal{M}^3$  can achieve the optimal welfare at the (unique) equilibrium of any TvN game. This naturally motivates our next question about how to design welfare-maximizing rewarding mechanisms in recommender systems.
5 Welfare Maximization via Backward Rewarding Mechanisms
The aforementioned failure of the generic  $\mathcal{M}^3$  class for welfare maximization urges us to re-think the rewarding mechanism design in recommender systems, especially for platforms where user attention is concentrated on the top few positions. Theorem 1 demonstrates certain inherent incompatibility between merit-based conditions and group monotonicity when it comes to optimizing welfare. Thus, a compromise must be made between the two, and our choice is the latter one. On one hand, any violation to the merit-based properties is challenging to justify as it undermines creators' perceptions about the value of the matching score metric. If creators discover that highly relevant content can receive lower payoffs or no rewards despite being the most relevant, it can be detrimental to the platform's reputation. On the other hand, while an increase in a creator's matching score  $\sigma_{i}$  would naturally lead to an expected increase in his/her reward, it is generally unnecessary for the total rewards to increase as required by the monotonicity property. In fact, such non-monotonicity is widely observed in free markets, e.g., monopoly vs duopoly markets. For instance, consider a monopoly market with a high-quality producer and a low-quality producer, each catering to their distinct consumer bases. Now suppose the low-quality producer dramatically elevates his/her production quality to transition the market into a duopoly. While such an action would naturally augment the producer's profits, it would concurrently establish intensified competition with the high-quality producer, typically resulting in a marked decline in the latter's profitability. This would subsequently result in a decrease in the two producers' total profit [7, 25]. As will be clear later, our designed rewarding mechanism will lead to similar situations among content creators.
To enhance welfare, it is crucial to incentivize content creators who predominantly target larger user groups to also produce content for smaller groups. However, the monotone property encourages creators to continuously increase their matching scores to a user group, even when those users already have abundant options, resulting in diminishing welfare contributions. To address this, we introduce the class of Backward Rewarding Mechanisms (BRMs). The name of BRM suggests its essential characteristic: the reward for a specific creator-  $i$  depends solely on their ranking and the matching scores of creators ranked lower than  $i$ . The formal definition of BRM is provided below:
Definition 3 (BRM and BRCM). A Backward Rewarding Mechanism (BRM)  $M$  is determined by a sequence of Riemann integrable functions  $\{f_i(t):[0,1]\to \mathbb{R}_{\geq 0}\}_{i = 1}^n$ , satisfying  $f_{1}(t)\geq \dots \geq$
(a) Reward for  $\sigma_{1}$
(b) Reward for  $\sigma_{2}$
Figure 1: An illustration of the BRM parameterized by ordered functions  $\{f_1, f_2, f_3\}$ . There are 3 creators with matching scores  $1 \geq \sigma_1 \geq \sigma_2 \geq \sigma_3 \geq 0$ . The area of the blue region in each figure precisely gives the corresponding creator's reward in the BRM.
(c) Reward for  $\sigma_3$
$f_{n}(t) \forall t \in [0,1]$ , such that for any matching score sequence  $1 \geq \sigma_{1} \geq \dots \geq \sigma_{n} \geq 0$ , the reward to any creator  $i$  is given by
$$
M \left(\sigma_ {i}; \sigma_ {- i}\right) = \sum_ {k = i} ^ {n} \int_ {\sigma_ {k + 1}} ^ {\sigma_ {k}} f _ {k} (t) d t, \tag {5}
$$
where  $\sigma_{n+1} = 0$  and  $f_1(t) > 0, \forall t \in [0,1]$ . We use  $M[f_1(t), \dots, f_n(t)]$  to denote the BRM determined by ordered function sequence  $\{f_i(t)\}_{i=1}^n$ .
In addition, we identify a sub-class of mechanisms  $BRCM \subset BRM$  which includes those  $M$  such that  $\{f_i(t) \equiv f_i\}$  are a set of constant functions. Any  $M \in BRCM$  can be parameterized by an  $n$ -dimensional vector in the polytope  $\mathcal{F} = \{(f_1, \dots, f_n) | f_1 \geq \dots \geq f_n \geq 0\}$ .
Figure 1 illustrates an example of BRM with function  $\{f_1, f_2, f_3\}$ . According to its definition, the blue areas represent the rewards assigned by BRM and we can easily see why BRM preserves the merit-based property. Generally, the function  $f_i(\cdot)$  encapsulates the significance of the matching score difference between the  $i$ -th and  $(i+1)$ -th ranked content in contributing to the  $i$ -th ranked creator's reward. The constraint  $f_1(t) \geq \dots \geq f_n(t)$  is necessary to satisfy merit-based properties, as shown in the proof of Proposition 1. The broad class of BRM offers granular control over creator incentives. Meanwhile, the subclass BRCM provides opportunities for parameterized optimization over welfare, which we will discuss in Section 5.2. Additional concrete examples of BRM are illustrated in Appendix 8.3.
5.1 Properties of BRM
While the class of BRM might appear abstract at the first glance, one can confirm that it preserves all merit-based properties, making it a natural class of rewarding mechanisms. Nevertheless, in order to secure a better welfare guarantee, the monotonicity is dropped, as characterized in the following:
Proposition 1. Any  $M \in BRM$  is merit-based but not necessarily monotone.
The detailed proof is provided in the Appendix 8.5. Next we establish formal characterizations about the welfare guarantee of BRM. First, we show that any  $C^3$  game under BRM possesses a PNE because it is a potential game [16]. A strategic game is called a potential game if there exists a function  $P: \prod_{i} S_i \to \mathbb{R}$  such that for any strategy profile  $s = (s_1, \dots, s_n)$ , any player-  $i$  and strategy  $s_i' \in S_i$ , whenever player-  $i$  deviates from  $s_i$  to  $s_i'$ , the change of his/her utility function is equal to the change of  $P$ , i.e.,
$$
P (\boldsymbol {s} _ {i} ^ {\prime}, \boldsymbol {s} _ {- i}) - P (\boldsymbol {s} _ {i}, \boldsymbol {s} _ {- i}) = u _ {i} (\boldsymbol {s} _ {i} ^ {\prime}, \boldsymbol {s} _ {- i}) - u _ {i} (\boldsymbol {s} _ {i}, \boldsymbol {s} _ {- i}).
$$
This leads us to the main result of this section:
Theorem 2. Consider any  $C^3$  game  $\mathcal{G}(\{\mathcal{S}_i\}, \{c_i\}, \mathcal{F}, \sigma, M, \{r_i\})$ .
1. The  $C^3$  game is a potential game under any mechanism  $M \in BRM$ , and thus admits a pure Nash equilibrium (PNE);
2. Moreover, if the mechanism  $M = M[r_1, \dots, r_n] \in BRCM (\subset BRM)$ , the potential function is precisely the welfare function, i.e.,  $W(\pmb{s}) = P(\pmb{s}; M)$ . Consequently, the always exists a PNE that obtains the optimal welfare.
The proof is in Appendix 8.6, where we construct its potential function explicitly. According to [16], we also conclude: 1. the maximizers of  $P$  are the PNEs of  $\mathcal{G}$ , and 2. if the evolution of creators' strategic behavior follows a better response dynamics (i.e., in each iteration, an arbitrary creator deviates to a strategy that increases his/her utility), their joint strategy profile converges to a PNE.
Theorem 2 suggests another appealing property of BRM: one can always select an  $M$  within BRM to align the potential function with the welfare metric, which can be simply achieved by setting each  $f_{i}$  identical to  $r_{i}$ . Consequently, any best response dynamic among creators not only converges to a PNE but also generates a strictly increasing sequence of  $W$ , thus ensuring at least a local maximizer of  $W$ . Denote the set of PNEs of  $\mathcal{G}$  as  $PNE(\mathcal{G})$ . When  $PNE(\mathcal{G})$  coincides with the global maximizers of its potential function, i.e.,  $PNE(\mathcal{G}) = \operatorname{argmax}_{s} P(s; M)$ , we conclude that any PNE of  $\mathcal{G}$  also maximizes the welfare  $W$ . The following corollary indicates that such an optimistic situation occurs in TvN games, providing a stark contrast to the findings in Theorem 1.
Corollary 1. For any TvN instance  $\mathcal{G}$ , there exists  $M \in BRCM$  such that any PNE  $s^* \in PNE(\mathcal{G})$  attains the optimal  $W$ , i.e.,
$$
\max  _ {\boldsymbol {s} \in \mathcal {S}} W (\boldsymbol {s}) = W \left(\boldsymbol {s} ^ {*}\right). \tag {6}
$$
The proof is in Appendix 8.7. Despite the promising results presented in Corollary 1, it remains uncertain whether the strong welfare guarantee for TvN can be extended to the entire class of  $C^3$ . This uncertainty arises because, in general, we only know that  $\operatorname{argmax}_s P(s; M) \subseteq PNE(\mathcal{G})$ . However, [21] noted that the subset of PNEs corresponding to  $\operatorname{argmax}_s P(s; M)$  in any potential game is robust in the following sense: in an incomplete information relaxation of  $\mathcal{G}$ , where each creator possesses a private type and must take actions based on their beliefs about other creators' types, they will play the strategies in  $\operatorname{argmax}_s P(s; M)$  at the Bayesian Nash equilibrium with a probability close to 1. This insight suggests that BRM has the potential to achieve optimal social welfare in real-world scenarios. While we lack a conclusive theoretical determination of whether BRM can attain globally optimal welfare, our empirical study in Section 6 consistently reveals that BRM outperforms baseline mechanisms in  $\mathcal{M}^3$  in terms of improving welfare.
5.2 Welfare Optimization within BRCM
Theorem 2 suggests that, provided the parameters  $\{r_i\}$  are known, the platform can select a mechanism within BRCM with a better welfare guarantee. However, in many practical scenarios, the platform may not have access to the exact values of  $\{r_i\}$  but can only evaluate the resulting welfare metric using certain aggregated statistics. This presents a challenge as it may not be analytically feasible to pinpoint the optimal  $M$  as suggested by Theorem 2. In these cases, although perfect alignment between the potential function  $P$  and social welfare  $W$  may not be feasible, we can still find a mechanism that approximates the maximizer of  $W$  in creator competition. This leads us to formulate the following bi-level optimization problem:
$$
\max  _ {M \in B R C M} W \left(\boldsymbol {s} ^ {*} (M)\right) \tag {7}
$$
$$
s. t., \quad \boldsymbol {s} ^ {*} (M) = \underset {\boldsymbol {s}} {\operatorname {a r g m a x}} P (\boldsymbol {s}; M) \tag {8}
$$
In problem (7), the inner optimization (8) is executed by creators: for any given  $M$ , we have justified that the creators' strategies are very likely to settle at a PNE  $s^{*}(M)$  that corresponds to a maximizer of  $P(s;M)$ . However, the exact solution to the inner problem is neither analytically solvable by the platform (owing to the combinatorial nature of  $P$ ) nor observable from real-world feedback (due to creators' potentially long feedback cycles). Therefore, we propose to approximate its solution by simulating creators' strategic response sequences (See Appendix 8.8, Algorithm 1), on top of which we solve (7). Algorithm 1 is a variant of better response dynamics, incorporating randomness and practical considerations to more accurately emulate creator behavior, and will be employed as a subroutine in Algorithm 2. We should note that the specifics of the creator response simulator are not critical to our proposed solution: the optimizer can select any equilibrium-finding dynamic to replace our simulator 1, as long as it is believed to better represent creators' responses in reality.
Another challenge of solving (7) lies in the presence of ranking operations in  $W$ , which makes it non-differentiable in  $s$  and renders first-order optimization techniques ineffective. Consequently, we resort to coordinate update and apply finite differences to estimate the ascending direction of  $W$  with respect to each  $M$  parameterized by  $\pmb{f} = (f_{1},\dots ,f_{n})\in \mathcal{F}$ . Our proposed optimization algorithm for solving (7) is presented in Algorithm 2 in Appendix 8.9, and is structured into  $L_{1}$  epochs. At the
beginning of each epoch, the optimizer randomly perturbs the current  $M$  along a direction within the feasible polytope and simulates creators' responses for  $L_{2}$  steps using Algorithm 1. Welfare is re-evaluated at the end of this epoch, and the perturbation on  $M$  is adopted if it results in a welfare increase.
To validate our theoretical findings and demonstrate the efficacy of Algorithm 2, we simulate the strategic behavior of content creators and compare the evolution of social welfare under various mechanisms. These include Algorithm 2 and several baselines from both the  $\mathcal{M}^3$  and BRCM classes.
6.1 Specification of Environments
We conduct simulations on game instances  $\mathcal{G}(\{\mathcal{S}_i\}, \{c_i\}, \mathcal{F}, \sigma, M, \{r_i\})$  constructed from synthetic data. Results on MovieLens-1m dataset [11] are shown in Appendix 8.10. For the synthetic data, we consider a uniform distribution  $\mathcal{F}$  on  $\mathcal{X}$  as follows: we fix the embedding dimension  $d$  and randomly sample  $Y$  cluster centers, denoted as  $\mathbf{c}_1, \dots, \mathbf{c}_Y$ , on the unit sphere  $\mathbb{S}^{d-1}$ . For each center  $\mathbf{c}_i$ , we generate users belonging to cluster- $i$  by first independently sampling from a Gaussian distribution  $\tilde{\boldsymbol{x}} \sim \mathcal{N}(\mathbf{c}_i, v^2 I_d)$ , and then normalize it to  $\mathbb{S}^{d-1}$ , i.e.,  $\boldsymbol{x} = \tilde{\boldsymbol{x}} / \| \tilde{\boldsymbol{x}} \|_2$ . The sizes of the  $Y$  user clusters are denoted by a vector  $\boldsymbol{z} = (z_1, \dots, z_Y)$ . In this manner, we generate a population  $\mathcal{X} = \cup_{i=1}^{Y} \mathcal{X}_i$  with size  $m = \sum_{i=1}^{Y} z_i$ . The number of creators is set to  $n = 10$ , with action sets  $\mathcal{S}_i = \mathbb{S}^{d-1}$ . The relevance function  $\sigma(\boldsymbol{x}, \boldsymbol{s}) = \frac{1}{2} (\boldsymbol{s}^\top \boldsymbol{x} + 1)$  is the shifted inner product such that its range is exactly  $[0,1]$ .  $\{r_i\}_{i=1}^n$  is set to  $\left\{\frac{1}{\log_2(2)}, \dots, \frac{1}{\log_2(5)}, \frac{1}{\log_2(6)}, 0, \dots, 0\right\}$ . These synthetic datasets simulate situations where content creators compete over a clustered user preference distribution. We consider two types of game instances, denoted  $\mathcal{G}_1$  and  $\mathcal{G}_2$ , distinguished by their cost functions:
1. In  $\mathcal{G}_1$ , creators have zero cost and their initial strategies are set to the center of the largest user group. This environment models the situation where the social welfare is already trapped at suboptimal due to its unbalanced content distribution. We aim to evaluate which mechanism is most effective in assisting the platform to escape from such a suboptimal state.
2. In  $\mathcal{G}_2$ , creators have non-trivial cost functions  $c_{i} = 0.5\| s_{i} - \bar{s}_{i}\|_{2}^{2}$ , where the cost center  $\bar{s}_i$  is randomly sampled on  $\mathbb{S}^{d - 1}$ . Their initial strategies are set to the corresponding cost centers, i.e., all creators start with strategies that minimize their costs. This environment models a "cold start" situation for creators: they do not have any preference nor knowledge about the user population and gradually learn about the environment under the platform's incentivizing mechanism.
In our experiment, we set  $(d,v,Y,m) = (10,0.3,8,52)$  and the cluster sizes  $z = (20,10,8,5,3,3,2,1)$ . The 8 clusters are divided into 3 groups ((20), (10,8), (5,3,3,2,1)), namely group-1,2,3, corresponding to the majority, minority, and niche groups.
6.2 Algorithm and Baseline Mechanisms
We simulate the welfare curve produced by Algorithm 2 alongside five baseline mechanisms below.
1.  $\mathrm{BRCM}_{opt}$ : This refers to the dynamic mechanism realized by optimization Algorithm 2. The parameters are set to  $T = 1000$ ,  $L_{1} = 200$ ,  $L_{2} = 5$ ,  $\eta_{1} = \eta_{2} = 0.1$ ,  $\pmb{f}^{(0)} = (1,1,1,1,1,0,\dots ,0)$ .
2. BRCM*: This denotes the theoretically optimal mechanism within BRCM, as indicated by Theorem 2. The corresponding parameters of  $M$  are derived based on the knowledge of  $\{r_i\}_{i=1}^n$ .
3.  $\mathrm{BRCM}_1$ :  $\mathrm{BRCM}_1 = M[1, \frac{1}{2}, \frac{1}{3}, \frac{1}{4}, \frac{1}{5}, 0, \dots, 0] \in \mathrm{BRCM}$ . This baseline aims to assess the impact of deviation from the theoretically optimal mechanism on the result.
4.  $M^3(0)$ : This mechanism assigns each content creator a reward equal to the relevance score, i.e.,  $M(\sigma_i; \sigma_{-i}) = \sigma_i$ . It is obvious that this mechanism belongs to the  $\mathcal{M}^3$  class and is therefore denoted as  $M^3(0)$ . Under  $M^3(0)$ , each creator's strategy does not affect other creators' rewards at all, and thus every creator will be inclined to match the largest user group as much as their cost allows. This mechanism acts as a reference to indicate the worst possible scenario.
5.  $M^3 (exo.)$ : The mechanism based on exposure, defined in Section 4 with  $K = 5, \beta = 0.05$ .
6.  $M^3(enga)$ : The mechanism based on engagement, defined in Section 4 with  $K = 5$ ,  $\beta = 0.05$ .
(a) Welfare on  $\mathcal{G}_1$
(b) Avg. Group Util. on  $\mathcal{G}_1$
(c) Welfare on  $\mathcal{G}_2$
Figure 2: Social welfare curve and average user utilities per group. Error bars represent half standard deviation range  $(0.5\sigma)$ , and are generated from simulations on 10 randomly sampled game instances.
(d) Avg. Group Util. on  $\mathcal{G}_2$
We let creators play  $\mathcal{G}_1$  and  $\mathcal{G}_2$  repeatedly under mechanisms specified in Section 6.2 and record the social welfare over  $T = 1000$  steps with Algorithm 1 in in Figure 2.
As illustrated in Figure 2a, BRCM family consistently outperformed  $\mathcal{M}^3$ . As anticipated,  $M^3(0)$  does little to enhance social welfare when creators have already primarily focused on the most populous user group. The  $M^3(expo.)$  and  $M^3(enga.)$  mechanisms demonstrate a notable improvement over  $M^3(0)$  as they instigate a competitive environment for creators striving to reach the top- $K$  positions. Nevertheless, they still do not perform as effectively as  $\mathrm{BRCM}_1$ , even though  $\mathrm{BRCM}_1$  's parameter deviates from the theoretically optimal one. Within the BRCMs,  $\mathrm{BRCM}_{opt}$  exhibits remarkable performance and even surpasses the theoretically optimal instance  $\mathrm{BRCM}^*$ . One possible explanation for the empirical sub-optimality of  $\mathrm{BRCM}^*$  is the stochastic nature of creators' response dynamics, which might prevent the convergence to PNE associated with the maximum welfare without sufficient optimization. This observation underscores the importance of Algorithm 2, as it empowers the platform to pinpoint an empirically optimal mechanism in more practical scenarios. As depicted in Figure 2b, the primary source of advantage stems from the increased utility among minority and niche user groups: compared to  $M^3(expo.)$  and  $M^3(enga.)$ , BRCM class results in higher average utility for groups 2 and 3 while preserving overall satisfaction for group-1.
Similar observations can be made for  $\mathcal{G}_2$ . However, it is worth noting that  $\mathrm{BRCM}_{opt}$  underperformed slightly in comparison to  $\mathrm{BRCM}^*$  as shown in Figure 2c. Despite this, the BRCM class of mechanisms continued to significantly surpass those in  $\mathcal{M}^3$ . Figure 2d further highlights that BRCM mechanisms lead to a more equitable distribution of average user utility across different user groups. Nevertheless, the gap in comparison becomes less pronounced, which is probably due to the existence of costs. Creators burdened with such costs are inherently inclined towards serving specific user groups, making them less susceptible to the influence of platform's incentives.
Our work reveals an intrinsic limitation of the monotone reward principle, widely used by contemporary online content recommendation platforms to incentivize content creators, in optimizing social welfare. As a rescue, we introduce BRM, a novel class of reward mechanisms with several key advantages. First, BRM ensures a stable equilibrium in content creator competition, thereby fostering a consistent and sustainable content creation environment. Second, BRM can guide content creators' strategic responses towards optimizing social welfare, providing at least a local optimum for any given welfare metric. Finally, BRM offers a parameterized subspace that allows the platform to empirically optimize social welfare, enhancing platform performance dynamically.
For future work, we identify two potential directions. From a theoretical standpoint, it would be intriguing to ascertain whether a stronger welfare guarantee for BRM could be established when the scoring function is equipped with certain simple structures, e.g., dot product. On the empirical side, we look for developments of our suggested mechanism by addressing some practical considerations. For instance, how can we enhance the robustness of BRM to account for the estimation noise in relevance scores? And how can a platform optimize welfare subject to budget constraints? Deeper insights into these questions could significantly enhance our understanding of the rapidly evolving online content ecosystems.
Acknowledgment This work is supported in part by an NSF Award CCF-2303372, IIS-2128019, and IIS-2007492, an Army Research Office Award W911NF-23-1-0030, and an Office of Naval Research Award N00014-23-1-2802.