Title: Learning to Mitigate Externalities: the Coase Theorem with Hindsight Rationality
Venue: NeurIPS 2024
Accepted: yes
Keywords: Online Learning

Bandits

Algorithmic Game Theory

Externality

Policy Design

Two-Players Game
Abstract: In Economics, the concept of externality refers to any indirect effect resulting from an interaction between players and affecting a third party without compensation. Most of the models within which externality has been studied assume that agents have perfect knowledge of their environment and preferences. This is a major hindrance to the practical implementation of many proposed solutions. To adress this issue, we consider a two-players bandit game setting where the actions of one of the player affect the other one. Building upon this setup, we extend the Coase theorem [Coase, 2013], which suggests that the optimal approach for maximizing the social welfare in the presence of externality is to establish property rights, i.e., enabling transfers and bargaining between the players. Nonetheless, this fundamental result relies on the assumption that bargainers possess perfect knowledge of the underlying game. We first demonstrate that in the absence of property rights in the considered online scenario, the social welfare breaks down. We then provide a policy for the players, which allows them to learn a bargaining strategy which maximizes the total welfare, recovering the Coase theorem under uncertainty.

Learning to Mitigate Externalities: the Coase Theorem with Hindsight Rationality

Abstract

In economic theory, the concept of externality refers to any indirect effect resulting from an interaction between players that affects the social welfare. Most of the models within which externality has been studied assume that agents have perfect knowledge of their environment and preferences. This is a major hindrance to the practical implementation of many proposed solutions. To address this issue, we consider a two-player bandit setting where the actions of one of the players affect the other player and we extend the Coase theorem [Coase, 2013]. This result shows that the optimal approach for maximizing the social welfare in the presence of externality is to establish property rights, i.e., enable transfers and bargaining between the players. Our work removes the classical assumption that bargainers possess perfect knowledge of the underlying game. We first demonstrate that in the absence of property rights, the social welfare breaks down. We then design a policy for the players which allows them to learn a bargaining strategy which maximizes the total welfare, recovering the Coase theorem under uncertainty.

1 Introduction

The concept of externality is used in economics to capture phenomena that impact the global welfare stemming from economic interactions without any compensation [Buchanan and Stubblebine, 2006, Shah et al., 2018]. Externality is generally considered as market failure since they result in a loss of collective welfare. Given its practical importance [Dahlman, 1979, Greenfield et al., 2009], mechanisms that characterize and mitigate externalities are central to modern economic thinking.

A first approach to tackle the adverse effects of externalities in modern economics was based on quotas and taxation [see, e.g., Pigou, 2017, and the references therein]. However, Coase's theorem [Coase, 2013] shows that in the presence of well-defined property rights and low transaction costs, parties affected by externalities can privately negotiate efficient solutions, and recover a welfare efficient allocation through transfers and bargaining.

Throughout the paper, we use the following simple example to illustrate our results.

Example 1. Consider two firms 1 (upstream) and 2 (downstream) respectively producing quantities q 1 ⩾ 0 and q 2 ⩾ 0 of a good sold at a fixed price p > 0 . They incur strictly increasing and convex costs, captured by the differentiable cost functions c 1 : q 1 ↦→ c 1 ( q 1 ) and c 2 : q 2 ↦→ c 2 ( q 2 ) , satisfying

c 1 (0) = 0 and c 2 (0) = 0 . We assume that firm 1 exerts on firm 2 a constant externality α > 0 per unit produced. In other words, their profit functions (or utilities) are given by

One simple concrete illustration consists in an upstream firm emitting pollutants that reduce the downstream firm's production. If the upstream firm owns the property rights, it may receive a payment from the downstream one to reduce its output and thereby pollution. On the other hand, if the downstream firm owns the property right, it may require compensation from the upstream firm to allow its operation.

The Coase theorem demonstrates that in both cases with appropriate property rights, the resulting levels of production would be welfare efficient. The theorem is typically explained in textbooks under the assumption that the players have perfect knowledge of their own utility or profit function, as well as that of others. However, this assumption is unlikely to hold in real-world scenarios, where players have to learn about their own preferences and those of their competitors.

This example is, of course, a simplification of real world scenarios. For example, Abildtrup et al. [2012] consider a more complex setting to model the interaction between farmers and waterworks in Denmark where the farmers have the property rights whereas the waterworks can pay to reduce pollution. In particular, they demonstrate the failure of the theorem, attributing it to the breakdown of the main assumptions: no transaction costs, maximizing behaviors and perfect information, with an important focus on strategic behaviors and the asymmetry of information. We restore the latter in our work and provide foundations for the theorem to hold in more realistic scenarios.

A key question is whether the Coase theorem holds when players learn their preferences over time.

We investigate this question within the framework of a multi-armed bandit learning. We build upon recent works that extend the classical bandit setting to economics in which there are two players interacting via principal-agent protocols [Dogan et al., 2023b,a, Scheid et al., 2024]. This allows us to capture, for example, a version of the two-firm problem where firms are uncertain regarding both their profit functions and the degree of externality on other firms. We represent production decisions in this problem as arms which can be played by the firms at any round over time, with the goal of finding decisions that maximize their rewards. More precisely, we assume that the reward of the upstream firm only depends on its own action, while the reward of the downstream firm depends on both its action and the upstream firm's action. This dependency on both actions allows us to capture externalities.

The property rights in Example 1, or more generally over a bandit instance, amount to giving a firm the possibility of engaging in monetary transfers that influence the arms that are played over time. The owner of the bandit instance will face a problem of bandit learning with transfers with an upstream player who is also learning his preferences.

To account for the efficiency of a policy in this setup, we extend the classical static notion of welfare efficiency to the online setting. We say that a policy is Welfare efficient if the social welfare regret is sub-linear. Proving the Coase theorem within our setup therefore boils down to show that if the bandit owner (who is without loss of generality the upstream player in this study 1 ) runs a no-regret bandit algorithm to learn and exploit his preferences, the downstream player can then choose an optimal transfer scheme leading to a sub-linear total regret.

Our contributions are as follows:

We show that when an upstream agent exerts externality on a downstream agent, in the absence of property right, the social welfare breaks down. Put differently, no joint policy of the agents can be welfare efficient.

We then introduce property rights and show how it affects the game. In this case, bargaining and transfers are available to the players. We propose a policy for the downstream player that leads to welfare efficiency when the upstream player follows any black-box no-regret policy under mild assumption. This solution addresses the breakdown issue at equilibrium. Put together, we show an online version of the Coase theorem .

2 Setup and Inefficiency of Externality

2.1 Bandit game

We consider a sequential bandit game in which two players (downstream and upstream) simultaneously play actions in a bandit instance for a horizon T ∈ N ⋆ . The action set for both players is A = { 1 , . . . , K } , K ∈ N ⋆ .

The reward distributions of the agents differ. Given a family of distributions { γ a : a ∈ A} indexed by A , the upstream player's rewards are provided by an i.i.d. family of random variables

To model the externality exerted by the upstream on the downstream player, we assume that the latter has a reward that depends both on her action and on that of the upstream player. Formally, this is modeled through a family of distributions { ν a,b : a, b ∈ A} double-indexed by A and an i.i.d. family of random variables

is the reward received by the downstream player at time t if she pulls the arm b and the upstream player pulls the arm a .

Players. We assume that players are risk-neutral expected-utility maximizers, and we define their expected utilities for any ( a, b ) ∈ A × A as

The distributions ( γ a ) a ∈A and ( ν a,b ) ( a,b ) ∈A 2 are unknown to both the downstream and the upstream players and they aim to learn the distributions with best mean rewards by sequentially observing samples from { ( Z a ( t )) t ∈ [ T ] : a ∈ A} and { ( X a,b ( t )) t ∈ [ T ] : a, b ∈ A} .

Moreover, we suppose that players are rational in hindsight; that is, they minimize their regret. Formally, the upstream player aims to minimize his regret defined as

while the downstream player seeks to minimize her external regret defined as

where the players' actions ( A t ) t ∈ [ T ] , ( B t ) t ∈ [ T ] as well as their policies Π up n , Π down n are defined below. Note that the utility of the downstream player also depends on the actions taken by the upstream player, which represents the externality exerted by the upstream player on the downstream player, hence the strategic dimension of our setting. We first consider a game where no property right is defined, so each player is free to pick his preferred arm irrespectively of the other player's choice. This will result in a breakdown of the total utility.

Policies without property rights. Consider first the upstream player. Based on a policy Π up n (for example a no-regret bandit algorithm such as the Upper Confidence Bounds algorithm ( UCB ) [Auer, 2002] or the ε -greedy algorithm [Robbins, 1952, Langford and Zhang, 2007]), we define his history ( H up , n t ) t ∈ [ T ] by induction. We set H up , n 0 = ∅ and supposing that H up , n t is defined for t ∈ [ T ] , then

where ( V s ) s ∈ N ⋆ is a family of independent uniform random variables in [0 , 1] , allowing for randomization in the policy, and A t +1 is provided by Π up n , following Π up n : ( V t +1 , H up , n t ) ↦→ A t +1 .

Second, consider the downstream player and an algorithm Π down n (specifically a no-regret bandit algorithm). We define her history ( H down , n t ) t ∈ [ T ] by induction. We set H down , n 0 = ∅ and supposing that H down , n t is defined for t ∈ [ T ] , then

where ( U s ) s ∈ N ⋆ is a family of independent uniform random variables in [0 , 1] allowing for randomization in the policy and B t +1 is provided by Π down n , following Π down n : ( U t +1 , H down , n t ) ↦→ B t +1 .

Welfare efficiency. We now introduce the notion of Welfare efficiency for our setup. The global utility, or social welfare , of the players at round t is defined as v up ( A t ) + v down ( A t , B t ) . We define the socially optimal action ( a sw , b sw ) ∈ A × A of the game as

as well as the global regret (or social welfare regret ) associated with policies Π up n and Π down n as

Then, the joint policies Π up n and Π down n for the players are said to be Welfare efficient if

Intuitively, this condition implies that the frequency of the socially optimal action ( a sw , b sw ) tends to 1 as T goes to infinity. In this sense, it mimics the usual, static Welfare efficiency criterion. As we will see, (Π up n , Π down n ) is typically not Welfare efficient when there is a disalignment in the game between the players' individual interests based on their rationality and the social welfare.

2.2 Inefficiency without property rights

We first present a result that captures the adverse consequence of externality on social welfare. The upstream player does not take into account the indirect cost incurred by the downstream player when he chooses his action. This drives the social welfare away from its optimal level. We illustrate this fact within our simple bilateral externality example.

Example 1 (continuing from p. 1) . We show that the competitive outcome, where each firm maximizes its profit independently, is not welfare efficient in the presence of externality. Define the social welfare as the function

By definition, the welfare efficient outcome ( q ⋆ 1 , q ⋆ 2 ) ∈ R 2 + satisfies W ( q ⋆ 1 , q ⋆ 2 ) ⩾ W ( q 1 , q 2 ) for any ( q 1 , q 2 ) ∈ R 2 + . Since W is differentiable and strictly concave, ( q ⋆ 1 , q ⋆ 2 ) is uniquely defined by the condition ∇ W ( q ⋆ 1 , q ⋆ 2 ) = 0 , that is

Note that at the welfare efficient optimum, firm 1 does not equalize marginal cost with marginal profit, but produces less to account for the negative effect of externality on firm 2. We now characterize the competitive outcome ( q ′ 1 , q ′ 2 ) ∈ R 2 + . Since π 1 and π 2 are differentiable and strictly concave, ( q ′ 1 , q ′ 2 ) satisfies

For the competitive outcome to be welfare efficient, we require, by Equation (6) and Equation (7) ,

This proves that whenever there are externalities, no competitive outcome is efficient.

We now show that in our model, when there is no property right and under mild assumptions, no achievable policy is welfare efficient whenever there is a misalignment between the players' interests and the social welfare. The upstream player's policy Π up n is said to be no-regret if lim T → + ∞ R up n ( T, Π up n ) /T = 0 , where R up n is defined in (1).

Theorem 2. Suppose that argmax a ∈A v up ( a ) is the singleton { a u ⋆ } and that

for any b ∈ A . In the absence of property rights and when the upstream player runs any no-regret policy Π up n , we have R sw ( T, Π up n , Π down n ) = Ω( T ) . Therefore, R sw ( T, Π up n , Π down n ) = Ω( T ) and (Π up n , Π down n ) is not welfare efficient.

Condition (8) in Theorem 2 represents the unalignment between the upstream player's preference and the optimal choice from a social welfare point of view. Note that the upstream and downstream players can both have an o ( T ) external regret, while the social welfare regret still grows linearly with T because of the unfavorable interactions between their policies.

3 Online Property Game with Bargaining Players

3.1 Online Property Game

We now consider the same repeated game in the form of a property game where one of the players possesses the bandit instance ( upstream player ). As in the original setup of Coase [2013], the other player ( downstream player ) will provide the bandit owner with transfers to incentivize him to choose some specific action and influence the outcome of the game in her favor.

We show in Appendix B that our method applies similarly when property rights are given to the upstream player rather than the downstream player. Hence, there is no loss of generality in considering the aforementioned framework. In this sense, we recover the invariance property of the Coasean bargaining [Mas-Colell et al., 1995].

Example 1 (continuing from p. 1) . We now illustrate how Coasean bargaining re-instaures efficiency. Suppose without loss of generality that property rights are such that firm 2 can pay τ ∈ R + to firm 1 for it to operate at a level ˜ q 1 . Profits become

¯ π 2 : ( q 1 , q 2 , τ, ˜ q 1 ) ↦→ π 2 ( q 1 , q 2 ) -✶ { q 1 =˜ q 1 } τ and ¯ π 1 : ( q 1 , τ, ˜ q 1 ) ↦→ π ( q 1 ) + ✶ { q 1 =˜ q 1 } τ. Consider the competitive outcome ( q 1 , q 2 , τ, ˜ q 1 ) ∈ R 4 + which satisfies

The condition on q 1 accounts for the rationality of the firm 1 and the fact that its choice depends on the payment ( τ, ˜ q 1 ) . Obviously, the optimal solution is reached for ˜ q 1 = q 1 and τ = max q ′ π 1 ( q ′ ) -π 1 (˜ q 1 ) . Plugging this back in the expression of ¯ π 2 then yields

so the competitive outcome ( q 1 , q 2 ) is welfare efficient.

The transfers at each step can be interpreted as a contract between two players [see, e.g., Bolton and Dewatripont, 2004, Salanié, 2005, for general contract theory] and providing the right amount of incentives relates to adjusting a contract in an online setting [see Dütting et al., 2019, Guruganesh et al., 2021, Zhu et al., 2022, Fallah and Jordan, 2023, Guruganesh et al., 2024, Ananthakrishnan et al., 2024, for learning-based perspectives about contracts].

Similarly to Example 1, we modify the players' policies to now account for the transfer τ ( t ) that the downstream player offers at round t to the upstream player if he picks action ˜ a t . The downstream player's policy at round t does not only output an arm B t but now a triple (˜ a t , τ ( t ) , B t ) , where B t is the arm that she should play and ˜ a t is the arm on which a transfer τ ( t ) is offered to the upstream player. On the upstream player's side, the policy still outputs an arm A t to play but also takes as an input the incentive (˜ a t , τ ( t )) . In addition, the instantaneous utility of the upstream player becomes Z A t ( t ) + 1 ˜ a t ( A t ) τ ( t ) , whereas the downstream player receives X A t ,B t ( t ) -1 ˜ a t ( A t ) τ ( t ) .

Policies with property rights. Based on policies Π up p for the upstream player and Π down p for the downstream player, we define their histories ( H up , p t ) t ∈ [ T ] and ( H down , p t ) t ∈ [ T ] by induction. We set H up , p 0 = ∅ , H down , p 0 = ∅ and supposing that H up , p t , H down , p t are defined for t ∈ [ T ] , then

and

where ( V s ) s ∈ N ⋆ , ( U s ) s ∈ N ⋆ are two families of independent uniform random variables in [0 , 1] allowing for randomization in the policies, and the remaining quantities are given by Π down p : ( U t +1 , H down , p t ) ↦→ (˜ a t +1 , τ ( t +1) , B t +1 ) and Π up p : (˜ a t +1 , τ ( t +1) , V t +1 , H up , p t ) ↦→ A t +1 .

Players' goal. Given a transfer τ from the downstream to the upstream player on arm ˜ a , actions a and b respectively are chosen by the upstream and the downstream player, the upstream player's expected utility reads v up ( a ) + 1 ˜ a ( a ) τ while the downstream player's expected utility is v down ( a, b ) -1 ˜ a ( a ) τ . This defines the upstream player's expected regret for a horizon T as

Based on the upstream player's utility, the downstream player aims on a single round at proposing an optimal transfer τ opt on an arm a opt ∈ A as well as picking an arm b opt ∈ A which solves

Her regret for any horizon T is defined as

where we define µ ⋆, down = v down ( a opt , b opt ) -τ opt as the optimal utility she can aim for. We can see that the downstream player's influence is exerted through her action choice B t as well as through transfers which enable her to influence the upstream player's actions. Hence, the notion of external regret is obsolete here. The game has now the form of a repeated Stackelberg game [Von Stackelberg, 2010].

Lemma 1. Recall that µ ⋆, down is the downstream player's optimal reward as defined as a solution of (10) . We have µ ⋆, down = max a,b ∈A { v down ( a, b ) + v up ( a ) } -max a ′ ∈A { v up ( a ′ ) } , as well as ( a opt , b opt ) = ( a sw , b sw ) and µ ⋆, up + µ ⋆, down = v up ( a sw )+ v down ( a sw , b sw ) = max a,b ∈A { v up ( a )+ v down ( a, b ) } , where µ ⋆, up is defined in Equation (1) . Moreover, for any integer T ∈ N ⋆ , and policies Π up p , Π down p , we have that

This lemma has an interesting economic interpretation: if both players individually seek for their own interest within this online property game, they will together converge towards the optimal global utility. Individual rationality moves the outcome of the game towards the optimal social welfare. The transfers allow the players to align their goals and share the global reward, in line with the Coase theorem. Consequently, if both players run no-regret policies Π up p and Π down p , the social welfare regret will also be in o ( T ) . The rest of the paper shows that such no-regret policies exist. To this end, we introduce the following assumptions.

Without loss of generality, we assume that the upstream player's utility is rescaled and shifted, which corresponds to the following assumption on the reward distribution ( γ a ) a ∈A in R .

We now make a high probability bound assumption on the upstream player's regret. 2 .

H2. There exist C , ζ > 0 , κ ∈ [0 , 1) such that for any s, t ∈ [ T ] with s + t ⩽ T , any { τ a } a ∈ [ K ] ∈ R K + and any policy Π down p that offers almost surely a transfer (˜ a l , τ ( l )) = (˜ a l , τ ˜ a l ) for any l ∈ { s + 1 , . . . , s + t } , the batched regret of the upstream player following Π up p satisfies, with probability at least 1 -t -ζ ,

The constraint on the downstream player's algorithm Π down p enforces constant incentives associated with any arm a ∈ A withtin the batch, while the incentivized actions (˜ a l ) l ∈{ s +1 ,...,s + t } may change. Proposition 2 in Appendix C shows that an adaptation of UCB taking account the incentives satisfies H 2 with C = 8 √ K log( KT 3 ) , κ = 1 / 2 and ζ = 2 . Note that usual bandit algorithms such as AAE , ETC or EXP-IX also satisfy the assumption [see, e.g., Donahue et al., 2024, Lattimore and Szepesvári, 2020].

3.2 Downstream player's procedure

We fix the policy Π up p which can be any algorithm satisfying H 2 for the upstream player and introduce the algorithm BELGIC (Bandits and Externalities for a Learning Game with Incentivized Coase) which provides a policy achieving sub-linear regret for the downstream player. It can be seen as an online bargaining strategy to mitigate externalities. Simply put, BELGIC unfolds in two steps. First note that for any action a ∈ A , the optimal (lowest) transfer to offer to the upstream player to make him choose a is

̸

as detailed in Appendix A. Therefore, a batched binary search procedure (Algorithm 2) first allows the downstream player to estimate the optimal transfers τ ⋆ 1 , . . . , τ ⋆ K with a good precision level of 1 /T β , where β > 0 . More precisely, the downstream player offers a constant incentive (˜ a, τ ˜ a ) for a batch of time steps of length ˜ T = ⌈ T α ⌉ . The observation of T = ˜ a , the number of steps from the batch for which the upstream player does not pick ˜ a allows her to estimate whether τ ˜ a is above or below τ ⋆ ˜ a and adjust it, following Lemma 2 in Appendix A under the condition that α, β satisfy

The procedure needs to be run for K ⌈ T α ⌉⌈ log 2 T β ⌉ rounds since we have to make ⌈ log 2 T β ⌉ batches of binary search of length ⌈ T α ⌉ on each of the K arms [see Scheid et al., 2024]. This corresponds to the first phase of BELGIC as described in Algorithm 2. At the end of this stage, the estimated transfers (ˆ τ a ) a ∈A satisfy the bound in Proposition 1. These are then used to feed the subroutine Bandit-Alg .

Proposition 1. Under H 1 and H 2, after the first phase of BELGIC which consists in K ⌈ T α ⌉⌈ log T β ⌉ steps of binary search grouped in ⌈ log 2 T β ⌉ batches per arm a ∈ A , we have that

̸

The additional term 1 /T β +C T κ -1 in ˆ τ a ensures that if H 2 holds, the upstream player necessarily plays the incentivized action ˜ a t at round t with high probability. Lemmas 2 and 6 from Appendix C show how the binary search batches in BELGIC allow us to estimate τ ⋆ a depending on ˜ T -T = a , the number of times that arm a has been pulled by the upstream player during the batch.

Then, any bandit subroutine Bandit-Alg , such as UCB or ε -greedy , for instance, can be run in a black-box fashion on the shifted bandit instance, where the rewards are shifted by the upper estimated transfers (ˆ τ a ) a ∈A . The downstream player computes a shifted history ˜ H down , p t such that for any t ⩽ K ⌈ T α ⌉⌈ log 2 T β ⌉ , ˜ H down , p t = ∅ and for any t > K ⌈ T α ⌉⌈ log 2 T β ⌉

which serves to feed Bandit-Alg , following

For any family of constant incentives { τ a } a ∈A ∈ R K + , we define R Bandit-Alg ( T, ν, { τ a } a ∈A ) as the regret for the downstream player's subroutine Bandit-Alg on the bandit instance with shifted means over T rounds, following

Note that here, Bandit-Alg aims to maximize the shifted reward ( v down ( a, b ) -τ a ) ( a,b ) ∈A 2 .

Algorithm 1 BELGIC

Algorithm 2 Binary Search Subroutine

̸

̸

̸

Theorem 3. Assume that H 1 and H 2 hold. Then BELGIC , run with α, β satisfying (13) and any bandit subroutine Bandit-Alg , has an overall regret R down p such that

where, for ease of notation

Knowledge of C and κ . An upper bound on C and κ is sufficient to compute the hyperparameters in BELGIC . Theorem 3 shows that the bigger C and κ are, the worse is the downstream player's regret, hence the interest of knowing them more precisely.

Corollary 1. Assume that the upstream player's distribution ( γ a ) a ∈A is such that H 1 holds. In addition, suppose that the distributions ( γ a ) a ∈A and ( ν a,b ) a,b ∈A×A are 1 -sub-Gaussian and that the upstream player plays Π up p = Algorithm 3 (a slight modification of UCB to take into account the incentives). Then the downstream player's regret when she runs BELGIC with parameters α = 3 / 4 and β = 1 / 4 (which satisfy (13) ) and subroutine Bandit-Alg = UCB satisfies the following upper

̸

̸

The upper bound on the social welfare regret in Lemma 1 together with Corollary 1 shows that when the upstream player runs Π up p = UCB and the downstream player runs BELGIC , the social welfare regret then satisfies R sw ( T, Π up p , BELGIC ) = O ( K log( T ) T ( κ +1) / 2 ) .

In other words, if the downstream player runs BELGIC which produces a policy Π n down , for any upstream policy Π up p , (Π n down , Π up p ) is welfare efficient.

Influence of the upstream performance. It is interesting to note that in the downstream player's regret bound, the upstream player's regret bound in O ( T κ ) plays a significant role: the downstream player never learns faster than the upstream player. The latter's performance determines the social welfare convergence rate towards the social optimum. We can observe that the players' bounded rationality [Selten, 1990, Jones, 1999] and personal interest make the game converge towards the optimal social welfare equilibrium-even though they are both learning here.

Upstreamprofit

1.2

Downstreamprofit

Globalwelfare

Banditinstance

1.0

Empiricalfrequencies

0.981134

0.8

0.6

0.4

0.2

0.0

0.0001Q00028B002980055906746

0.009148011800034B00182

0

1

2

3

4

5

6

7

8

6

Upstreamprofit

Downstreamprofit

Globalwelfare

Banditinstance

Empiricalfrequencies

0.999611

0

1

2

3

4

5

6

7

8

6

Experiments. We conclude this section with experiments showing the empirical convergence of our algorithm to a social optimum. In the simulation, we consider two firms, with firm 1 being upstream and firm 2 being downstream. Their profit functions are respectively given by

and

Thus, firm 1's and firm 2's profit functions depends quadratically on q 1 with an firm 1 optimum at q ′ 1 = 5 and a social optimum at q ⋆ 1 = 8 . Note that in the expression of π 2 , q 2 has very little influence as compared to q 1 - which allows to plot profits for only one value of q 2 .

We discretize the setup, consider a bandit instance (horizon T = 5 . 10 6 , 10 arms, average over 10 rounds) and we assume that UCB is used as a subroutine. In the first setting, there are no property rights and each firm runs UCB on their side. Second, property rights are defined and firm 2 runs BELGIC as its policy. The plots in Figure 1 display the empirical frequencies and show empirically the effectiveness of BELGIC to mitigate externalities.

4 Related work

Our work addresses the impact of externalities and is therefore related to taxation theory [see, e.g., Mirrlees et al., 2011, Salanie, 2011, and the references therein], a prominent solution for this issue, as

1.2

1.0

0.8

0.6

0.4

0.2

0.0

exemplified by the Pigouvian tax [see Pigou, 2017]. Taxation is a fundamental aspect of all developed economies, with 30% to 50% of national income derived from taxes. The topic has been fruitful for various scenarios, including the carbon tax [Carattini et al., 2018, Metcalf and Weisbach, 2009], alcohol markets [Griffith et al., 2019], or business taxation [Boadway and Bruce, 1984]. Taxation can also be studied through an operations research lens, where it is used to enhance system efficiency or manage specific games [Roughgarden, 2010, Caragiannis et al., 2010, Bilò and Vinci, 2019]. Recent work by Cui et al. [2024] explores online mechanisms to maximize efficiency in congestion games.

Mechanism designs [Myerson, 1989, Nisan and Ronen, 1999, Laffont and Martimort, 2009] allow to design games that have specific desired outcomes. Deploying these mechanisms in their classical ecnomical form assume that players' utility functions are known a priori, which is often unrealistic. There is a major need to blend mechanism design with machine learning.

However, our approach differs, since, drawing inspiration from Coase's theory, we implement an online version of his theorem [Coase, 2013, Cooter, 1982], incorporating uncertainty to tackle the breakdown of social welfare in an online setting. We use the bandit setup [see Lattimore and Szepesvári, 2020, Slivkins et al., 2019] as a general and convenient way to model the game introduced by Coase. However, our work differs from considering a single agent playing a bandit game. Instead, we focus on the more general problem of multi-players bandits, a field receiving a growing attention from the community [see e.g., Boursier and Perchet, 2019, 2022, Sankararaman et al., 2019].

Our approach is inspired by the principal-agent model introduced by Dogan et al. [2023b], which was further extended by Dogan et al. [2023a], Scheid et al. [2024]. However, unlike the models proposed in the work of Dogan et al. [2023b,a], we do not specify a particular bandit algorithm for the upstream player and instead, we allow him to use any no-regret algorithm satisfying H 2 in a black-box fashion. Conversely, the model of Scheid et al. [2024] assumes that the upstream player is always fully informed and best-responding, whereas we assume that he is also learning. Chen et al. [2023b] leverages a similar model to study information acquisition by a principal through an agent's actions. However, in their model, the agent is also almighty and knows exactly the costs associated with each action. Designing incentives in an unknown environment is related to auction theory incorporating uncertainty, as it is explored in the work of Feng et al. [see 2018], Li et al. [see 2023]. Similar issues have been explored in the Reinforcement Learning framework within a leader-follower game [see Chen et al., 2023a, with quantal responses by the follower] or in a principal-agent game with incentive design as done by Ben-Porat et al. [2023]. Donahue et al. [2024] also study a two-players repeated Stackelberg game on a bandit instance but instead of allowing for transfers, their main focus concerns the achievability of a Stackelberg equilibrium through iterations of bandit policies: the same kind of goal also appears in Collina et al. [2023]. Such principal-agent setups are of some interest to model various real-world situations such as the design of fundings for hospitals [Wang et al., 2024] or have been studied with multiple agents through the lens of auction design in dynamic setups [Bergemann and Said, 2010, Chen et al., 2023c], or to account for fairness [Fallah et al., 2024].

In our game, the downstream player needs to learn the optimal transfers/incentives to offer to the upstream player. This is related to the Incentivized Exploration literature [Mansour et al., 2016, Simchowitz and Slivkins, 2023, Esmaeili et al., 2023], which is often cast in terms of a benevolent planner who aims to optimize the global welfare of agents via plausible recommendations. A related model is Bayesian Persuasion [Kamenica and Gentzkow, 2011], where a sender influences a receiver's action through sending a signal. This model has begun to be studied in learning settings [see, e.g., Castiglioni et al., 2020, Bernasconi et al., 2022, Wu et al., 2022b,a].

5 Conclusion

This paper studies a model of externalities in a two-players sequential game where both players learn their optimal actions. We first show that when the players act independently, then a misalignment between the players' interests and the social welfare leads to a breakdown of the global utility. We then introduce interactions through transfers, which restores a social welfare optimum, representing the online version of the Coase theorem . To that purpose, we propose a policy for the downstream player which allows her to estimate the optimal transfers as well as choosing the best actions. The mathematical difficulty comes from the learning aspect on both sides. Since our work is coined in a learning framework for mechanism design, several directions for research are open, as for instance extensions to the multi-agent setting, which raises many questions.

References

Jens Abildtrup, Frank Jensen, and Alex Dubgaard. Does the coase theorem hold in real markets? an application to the negotiations between waterworks and farmers in denmark. Journal of environmental management , 93(1):169-176, 2012.

Nivasini Ananthakrishnan, Stephen Bates, Michael Jordan, and Nika Haghtalab. Delegating data collection in decentralized machine learning. In International Conference on Artificial Intelligence and Statistics , pages 478-486. PMLR, 2024.

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

Omer Ben-Porat, Yishay Mansour, Michal Moshkovitz, and Boaz Taitler. Principal-agent reward shaping in mdps. arXiv preprint arXiv:2401.00298 , 2023.

Dirk Bergemann and Maher Said. Dynamic auctions: A survey. Wiley Encyclopedia of Operations Research and Management Science , 2010.

Martino Bernasconi, Matteo Castiglioni, Alberto Marchesi, Nicola Gatti, and Francesco Trovò. Sequential information design: Learning to persuade in the dark. Advances in Neural Information Processing Systems , 35:15917-15928, 2022.

Vittorio Bilò and Cosimo Vinci. Dynamic taxes for polynomial congestion games. ACM Transactions on Economics and Computation (TEAC) , 7(3):1-36, 2019.

Robin Boadway and Neil Bruce. A general proposition on the design of a neutral business tax. Journal of Public Economics , 24(2):231-239, 1984.

Patrick Bolton and Mathias Dewatripont. Contract theory . MIT press, 2004.

Etienne Boursier and Vianney Perchet. Sic-mmab: Synchronisation involves communication in multiplayer multi-armed bandits. Advances in Neural Information Processing Systems , 32, 2019.

Etienne Boursier and Vianney Perchet. A survey on multi-player bandits. arXiv preprint arXiv:2211.16275 , 2022.

Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning , 5(1):1-122, 2012.

James M Buchanan and Wm Craig Stubblebine. Externality. In Inframarginal Contributions to Development Economics , pages 55-73. World Scientific, 2006.

Ioannis Caragiannis, Christos Kaklamanis, and Panagiotis Kanellopoulos. Taxes for linear atomic congestion games. ACM Transactions on Algorithms (TALG) , 7(1):1-31, 2010.

Stefano Carattini, Maria Carvalho, and Sam Fankhauser. Overcoming public resistance to carbon taxes. Wiley Interdisciplinary Reviews: Climate Change , 9(5):e531, 2018.

Matteo Castiglioni, Andrea Celli, Alberto Marchesi, and Nicola Gatti. Online bayesian persuasion. Advances in Neural Information Processing Systems , 33:16188-16198, 2020.

Siyu Chen, Mengdi Wang, and Zhuoran Yang. Actions speak what you want: Provably sampleefficient reinforcement learning of the quantal stackelberg equilibrium from strategic feedbacks. arXiv preprint arXiv:2307.14085 , 2023a.

Siyu Chen, Jibang Wu, Yifan Wu, and Zhuoran Yang. Learning to incentivize information acquisition: Proper scoring rules meet principal-agent model. In International Conference on Machine Learning , pages 5194-5218. PMLR, 2023b.

Yurong Chen, Qian Wang, Zhijian Duan, Haoran Sun, Zhaohua Chen, Xiang Yan, and Xiaotie Deng. Coordinated dynamic bidding in repeated second-price auctions with budgets. In International Conference on Machine Learning , pages 5052-5086. PMLR, 2023c.

Ronald Coase. The problem of social cost. Journal of Law and Economics , 56(4):837 - 877, 2013. URL https://EconPapers.repec.org/RePEc:ucp:jlawec:doi:10.1086/674872 .

Natalie Collina, Eshwar Ram Arunachaleswaran, and Michael Kearns. Efficient stackelberg strategies for finitely repeated games. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , pages 643-651, 2023.

Robert Cooter. The cost of coase. The Journal of Legal Studies , 11(1):1-33, 1982.

Qiwen Cui, Maryam Fazel, and Simon S Du. Learning optimal tax design in nonatomic congestion games. arXiv preprint arXiv:2402.07437 , 2024.

Carl J Dahlman. The problem of externality. The journal of law and economics , 22(1):141-162, 1979.

Ilgin Dogan, Zuo-Jun Max Shen, and Anil Aswani. Estimating and incentivizing imperfect-knowledge agents with hidden rewards. arXiv preprint arXiv:2308.06717 , 2023a.

Ilgin Dogan, Zuo-Jun Max Shen, and Anil Aswani. Repeated principal-agent games with unobserved agent rewards and perfect-knowledge agents. arXiv preprint arXiv:2304.07407 , 2023b.

Kate Donahue, Nicole Immorlica, Meena Jagadeesan, Brendan Lucier, and Aleksandrs Slivkins. Impact of decentralized learning on player utilities in stackelberg games. arXiv preprint arXiv:2403.00188 , 2024.

Paul Dütting, Tim Roughgarden, and Inbal Talgam-Cohen. Simple versus optimal contracts. In Proceedings of the 2019 ACM Conference on Economics and Computation , pages 369-387, 2019.

Seyed A Esmaeili, Suho Shin, and Aleksandrs Slivkins. Robust and performance incentivizing algorithms for multi-armed bandits with strategic agents. arXiv preprint arXiv:2312.07929 , 2023.

Alireza Fallah and Michael I Jordan. Contract design with safety inspections. arXiv preprint arXiv:2311.02537 , 2023.

Alireza Fallah, Michael I Jordan, and Annie Ulichney. Fair allocation in dynamic mechanism design. arXiv preprint arXiv:2406.00147 , 2024.

Zhe Feng, Chara Podimata, and Vasilis Syrgkanis. Learning to bid without knowing your value. In Proceedings of the 2018 ACM Conference on Economics and Computation , pages 505-522, 2018.

Thomas K Greenfield, Yu Ye, William Kerr, Jason Bond, Jürgen Rehm, and Norman Giesbrecht. Externalities from alcohol consumption in the 2005 us national alcohol survey: implications for policy. International journal of environmental research and public health , 6(12):3205-3224, 2009.

Rachel Griffith, Martin O'Connell, and Kate Smith. Tax design in the alcohol market. Journal of public economics , 172:20-35, 2019.

Guru Guruganesh, Jon Schneider, and Joshua R Wang. Contracts under moral hazard and adverse selection. In Proceedings of the 22nd ACM Conference on Economics and Computation , pages 563-582, 2021.

Guru Guruganesh, Yoav Kolumbus, Jon Schneider, Inbal Talgam-Cohen, Emmanouil-Vasileios Vlatakis-Gkaragkounis, Joshua R Wang, and S Matthew Weinberg. Contracting with a learning agent. arXiv preprint arXiv:2401.16198 , 2024.

Bryan D Jones. Bounded rationality. Annual review of political science , 2(1):297-321, 1999.

Emir Kamenica and Matthew Gentzkow. Bayesian persuasion. American Economic Review , 101(6): 2590-2615, 2011.

Jean-Jacques Laffont and David Martimort. The theory of incentives: the principal-agent model. In The theory of incentives . Princeton university press, 2009.

John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. Advances in neural information processing systems , 20, 2007.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Ningyuan Li, Yunxuan Ma, Yang Zhao, Zhijian Duan, Yurong Chen, Zhilin Zhang, Jian Xu, Bo Zheng, and Xiaotie Deng. Learning-based ad auction design with externalities: the framework and a matching-based approach. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1291-1302, 2023.

Yishay Mansour, Aleksandrs Slivkins, Vasilis Syrgkanis, and Zhiwei Steven Wu. Bayesian exploration: Incentivizing exploration in bayesian games. arXiv preprint arXiv:1602.07570 , 2016.

Andreu Mas-Colell, Michael D. Whinston, and Jerry R. Green. Microeconomic Theory . Oxford University Press, New York, 1995.

Gillbert E Metcalf and David Weisbach. The design of a carbon tax. Harv. Envtl. L. Rev. , 33:499, 2009.

James Mirrlees et al. Tax by design: The Mirrlees review . OUP Oxford, 2011.

Roger B Myerson. Mechanism design . Springer, 1989.

Noam Nisan and Amir Ronen. Algorithmic mechanism design. In Proceedings of the thirty-first annual ACM symposium on Theory of computing , pages 129-140, 1999.

Vianney Perchet, Philippe Rigollet, Sylvain Chassang, and Erik Snowberg. Batched bandit problems. The Annals of Statistics , 44:660-681, 2016.

Arthur Pigou. The economics of welfare . Routledge, 2017.

Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society , page 527-535, 1952.

Tim Roughgarden. Algorithmic game theory. Communications of the ACM , 53(7):78-86, 2010.

Bernard Salanié. The economics of contracts: a primer . MIT press, 2005.

Bernard Salanie. The economics of taxation . MIT press, 2011.

Abishek Sankararaman, Ayalvadi Ganesh, and Sanjay Shakkottai. Social learning in multi agent multi armed bandits. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 3 (3):1-35, 2019.

Antoine Scheid, Daniil Tiapkin, Etienne Boursier, Aymeric Capitaine, El Mahdi El Mhamdi, Éric Moulines, Michael I Jordan, and Alain Durmus. Incentivized learning in principal-agent bandit games. ICML , 2024.

Reinhard Selten. Bounded rationality. Journal of Institutional and Theoretical Economics (JITE)/Zeitschrift für die gesamte Staatswissenschaft , 146(4):649-658, 1990.

Virag Shah, Jose Blanchet, and Ramesh Johari. Bandit learning with positive externalities. Advances in Neural Information Processing Systems , 31, 2018.

Max Simchowitz and Aleksandrs Slivkins. Exploration and incentives in reinforcement learning. Operations Research , 2023.

Aleksandrs Slivkins et al. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning , 12(1-2):1-286, 2019.

Heinrich Von Stackelberg. Market structure and equilibrium . Springer Science & Business Media, 2010.

Serena Wang, Stephen Bates, P Aronow, and Michael Jordan. On counterfactual metrics for social welfare: Incentives, ranking, and information asymmetry. In International Conference on Artificial Intelligence and Statistics , pages 1522-1530. PMLR, 2024.

Jibang Wu, Zixuan Zhang, Zhe Feng, Zhaoran Wang, Zhuoran Yang, Michael I Jordan, and Haifeng Xu. Markov persuasion processes and reinforcement learning. In ACM Conference on Economics and Computation , 2022a.

Jibang Wu, Zixuan Zhang, Zhe Feng, Zhaoran Wang, Zhuoran Yang, Michael I Jordan, and Haifeng Xu. Sequential information design: Markov persuasion process and its efficient reinforcement learning. arXiv preprint arXiv:2202.10678 , 2022b.

Banghua Zhu, Stephen Bates, Zhuoran Yang, Yixin Wang, Jiantao Jiao, and Michael I Jordan. The sample complexity of online contract design. arXiv preprint arXiv:2211.05732 , 2022.

A Algorithmic Subroutine for the Binary Search

Optimal Transfer. For any given round t ⩾ 1 , action a ∈ A and ε > 0 , the downstream player can incentivize any best-responding upstream player to choose a by offering a transfer, τ ⋆,ε a ∈ R + , defined as:

̸

With this transfer, it holds that for any a ′ ∈ A , a ′ = a , we have v up ( a ′ ) < v up ( a ) + τ ⋆,ε a , ensuring the upstream player's action A t = a , since action a yields a superior reward. Consequently,

represents the infimal transfer necessary to make arm a the best upstream player's choice.

First step of BELGIC : estimation of the optimal transfers. Suppose that we consider an arm a ∈ A and that the downstream player offers an incentive τ a to the upstream player if he picks this arm. We consider this procedure with a constant incentive τ a for a batch of time steps of length ˜ T = ⌈ T α ⌉ due to the fact that the upstream player is learning [see Perchet et al., 2016, for batched bandits in the usual multi-armed setting]. Lemma 2 shows that for BELGIC to accurately estimate τ ⋆ a with high probability, it must hold

This is why we impose the condition (13) on α and β , namely β/α < 1 -κ , which ensures that κ ( α -1) + β < 0 and therefore lim T → + ∞ T κ ( α -1)+ β = 0 . More precisely, we define for a ∈ [ K ]

which is the step after which starts the binary search procedure on arm a . For any d ∈ { 1 , . . . , ⌈ log 2 T β ⌉} on arm a , we define

as the step after which starts the d -th batch iteration on arm a , and we consider

̸

̸

where ( A t ) t ∈{ 1 ,...,K ⌈ log 2 T β ⌉ ˜ T } is given by Algorithm 2. Lemma 2 shows that for any a ∈ A , d ∈ { 1 , . . . , ⌈ log 2 T β ⌉} , with high probability

̸

̸

where ˆ τ a,d is the current estimate of τ ⋆ a offered for iteration k a,d . In case (18) does not hold, it means that the upstream player has misplayed and has chosen most of the steps a suboptimal action, leading to an instantaneous regret for him larger than the bound given in H 2. This is why (19) holds with high probability. To sum up, the first phase of BELGIC consists in ⌈ log 2 T β ⌉ batches of binary search on each arm a ∈ A to obtain a precision level 1 /T β on the optimal transfer τ ⋆ a .

During this phase in Algorithm 2, we define τ a ( d ) ∈ R + as the upper estimate and τ a ( d ) ∈ R + as the lower estimate of τ ⋆ a after d ∈ { 1 , . . . , ⌈ log 2 T β ⌉} rounds of binary search on arm a . For any t ∈ [ T ] and a ∈ A , we define τ mid a ( t ) = ( τ a ( t ) + τ a ( t )) / 2 . τ a ( d ) , τ a ( d ) , τ mid a ( d ) are updated at the end of the d -th binary search batch of length ˜ T on arm a . We define N T = ⌈ log 2 T β ⌉ as the number of binary search steps per arm.

After this first binary search phase, the downstream player computes estimates of the optimal transfers τ ⋆ a

and offers these transfers ( τ ⋆ a ) a ∈A to make the upstream player play any action ˜ a ∈ A she wants.

Second Step. After the first phase during which the optimal incentives are estimated by the downstream player through (ˆ τ a ) a ∈A , she runs in the second phase the subroutine Bandit-Alg on the A×A bandit instance driven by her action B t and the upstream player's one A t . More precisely, any bandit subroutine Bandit-Alg , such as UCB or ε -greedy , for instance, can be run in a black-box fashion

on the shifted bandit instance, where rewards are shifted by the upper estimated transfers (ˆ τ a ) a ∈A . The downstream player computes a shifted history ˜ H down , p t = ∅ for any t ⩽ K ⌈ T α ⌉⌈ log 2 T β ⌉ and for any t > K ⌈ T α ⌉⌈ log 2 T β ⌉

which serves to feed Bandit-Alg , following Bandit-Alg : ( U t , ˜ H down , p t -1 ) ↦→ (˜ a t , B t ) ∈ A × A . Note that here, Bandit-Alg aims to maximize the shifted reward ( ν down ( a, b ) -ˆ τ a ) ( a,b ) ∈A 2 .

Based on this decision by Bandit-Alg , Π down p offers the incentive ˆ τ ˜ a t associated with action ˜ a t and plays action B t . Lemma 5 ensures that ˜ a t is the upstream player's best choice. Therefore, H 2 ensures that the upstream player will not deviate from the downstream player's recommendation with high probability.

B Invariance when the property rights are given to the downstream player

Our focus in the paper was the case where the upstream player possesses the bandit instance and receives monetary payments. We argue here that the symmetric situation, i.e. when the property rights are given to the downstream player, can be analysed in the exact same way.

Assume that the the downstream player owns the bandit instance. This implies that (i) they can prescribe what arm the upstream player has to play at each round, and (ii) the upstream player may perform a monetary transfer to influence the arm they are allowed to pull.

Consider the same bandit setup as before. Formally, the downstream player's action is ( A t , B t ) ∈ A×A where A t is the arm that the upstream player is prescribed to pull (he cannot deviate since the downstream player has the property rights), while B t is the arm played by the downstream player. On the other hand, the upstream player's policy outputs at each round the action (˜ a t , τ ( t )) ∈ A × R + , where ˜ a t is the arm they choose to incentivize and τ ( t ) is the amount of transfer. It means that the downstream player receives a transfer τ ( t ) if she prescribes action A t = ˜ a t . As a consequence, the instantaneous utility of the upstream player is Z A t -1 ˜ a t ( A t ) τ ( t ) , while the downstream player receives X A t ,B t + 1 ˜ a t ( A t ) τ ( t ) . In that case, the upstream player may perform a binary search on each arm ¯ a ∈ A to identify the optimal incentive τ ⋆ ¯ a , by considering Card { t ∈ [ T ] such that ˜ a t = ¯ a } during batches designed for the binary search and then play on the shifted bandit instance as we explained. This situation and the upstream player's strategy are now equivalent to the one presented before.

C Proofs and Technical Results

Recall that we defined the shifted history ˜ H down , p t that will serve to feed Π down p at time t as ˜ H down , p t = (˜ a s , τ ( s ) , A s , V s , Z A s ( s )) s ⩽ t .

Theorem 4. Suppose that argmax a ∈A v up ( a ) is the singleton { a u ⋆ } and that

for any b ∈ A . In the absence of property rights and when the upstream player runs any noregret policy Π up n , we have R sw ( T, Π up n , Π down n ) ⩾ T ∆ sw -R up n ( T, Π up n )∆ sw / ∆ up , where ∆ up = min a ′ ∈A\{ a u ⋆ } v up ( a u ⋆ ) -v up ( a ′ ) and ∆ sw = v up ( a sw ) + v down ( a sw , b sw ) -max b ∈A ( v up ( a u ⋆ ) + v down ( a u ⋆ , b )) . Therefore, R sw ( T, Π up n , Π down n ) = Ω( T ) and (Π up n , Π down n ) is not welfare efficient.

Proof of Theorem 4. Since argmax a ∈A v up ( a ) is the singleton { a up ⋆ } , we define ∆ up = min a ′ ∈A\{ a u ⋆ } v up ( a u ⋆ ) -v up ( a ′ ) as the upstream player reward gap and ∆ sw = v up ( a sw ) + v down ( a sw , b sw ) -max b ∈A ( v up ( a u ⋆ ) + v down ( a u ⋆ , b )) as the social welfare reward gap if the upstream player plays his most preferred action.

̸

Denote N up ⋆ ( T ) the number of pulls of the upstream player up to time T on the arm a u ⋆ . By definition of ∆ up , we have that for any step t ∈ [ T ] such that A t = a up ⋆ , max a ∈A { v up ( a ) } -v up ( A t ) =

v up ( a up ⋆ ) -v up ( A t ) ⩾ min a ′ ∈A\{ a u ⋆ } v up ( a u ⋆ ) -v up ( a ′ ) = ∆ up . There are T -N up ⋆ ( T ) such steps, which leads to

and we obtain

Moreover, for any t ∈ [ T ] such that A t = a up ⋆ and any B t ∈ A , v up ( a sw ) + v down ( a sw , b sw ) -( v up ( A t ) + v down ( A t , B t )) = v up ( a sw ) + v down ( a sw , b sw ) -( v up ( a up ⋆ ) + v down ( a up , B t )) ⩾ ∆ sw by definition, which leads to

and we obtain

Since lim T → + ∞ R up n ( T, Π up n ) /T = 0 , we have lim T → + ∞ R sw ( T, Π up n , Π down n ) /T = ∆ sw > 0 , hence the result.

The proof of Theorem 2 is an immediate consequence of Theorem 4.

Lemma 1. Recall that µ ⋆, down is the downstream player's optimal reward as defined as a solution of (10) . We have µ ⋆, down = max a,b ∈A { v down ( a, b ) + v up ( a ) } -max a ′ ∈A { v up ( a ′ ) } , as well as ( a opt , b opt ) = ( a sw , b sw ) and µ ⋆, up + µ ⋆, down = v up ( a sw )+ v down ( a sw , b sw ) = max a,b ∈A { v up ( a )+ v down ( a, b ) } , where µ ⋆, up is defined in Equation (1) . Moreover, for any integer T ∈ N ⋆ , and policies Π up p , Π down p , we have that

Proof of Lemma 1. Recall that µ ⋆, down is defined as µ ⋆, down = sup a,b ∈A 2 ,τ ∈ R + { v down ( a, b ) -τ } , such that a ∈ argmax a ′ ∈A { v up ( a ′ ) + 1 a ( a ′ ) τ } and a up ⋆ = argmax a ′ ∈A v up ( a ) . Note that we can write

where ˜ A = { ( a, τ ): v up ( a ) + τ ⩾ max a ′ v up ( a ′ ) + 1 a ( a ′ ) τ } which is the set of pairs ( a, τ ) ∈ A× R + such that the constraint binds. However, we also have by definition that v up ( a up ⋆ ) + 0 ⩾ max a ′ ∈A v up ( a ′ ) + 1 a up ⋆ ( a ′ ) · 0 and hence, ( a up ⋆ , 0) ∈ ˜ A . Therefore, since v down ( a up ⋆ , b ) ⩾ 0 for any b ∈ A , we can write

First note that if ( a, τ ) ∈ ˜ A , then for any a ′ ∈ A , τ ∈ R + , v up ( a ) + τ ⩾ v up ( a ′ ) + 1 a ( a ′ ) τ , which gives

̸

However, either a = a ′ and hence v up ( a ) -v up ( a ′ ) = 0 , either a = a ′ and hence 1 a ( a ′ ) = 0 . Therefore, we have that

which implies by definition of the optimal incentives that τ ⩾ v up ( a ′ ) -v up ( a ) for any a ′ ∈ A , and hence τ ⋆ a ⩽ τ for any ( a, τ ) ∈ ˜ A .

In addition, ( a, τ ⋆ a ) ∈ ˜ A by definition. Consequently,

hence the first part of the result. Since µ ⋆, up is defined as µ ⋆, up = max a ∈A v up ( a ) , we have that

Now summing R down p and R up p as defined in (9) and (11), we obtain

hence the result.

For a downstream player's policy Π down p , we define ˜ R up p as the upstream player's regret without expectation, following

where (˜ a l , τ ( l )) l ∈ [ T ] are the incentives output by Π down p and ( A l ) l ∈ [ T ] are the output of Π up p . Recall the assumption that we use on the upstream player's regret for a policy Π up p . We show here that it is satisfied by typical no-regret bandit algorithms.

H2. There exist C , ζ > 0 , κ ∈ [0 , 1) such that for any s, t ∈ [ T ] with s + t ⩽ T , any { τ a } a ∈ [ K ] ∈ R K + and any policy Π down p that offers almost surely a transfer (˜ a l , τ ( l )) = (˜ a l , τ ˜ a l ) for any l ∈ { s + 1 , . . . , s + t } , the batched regret of the upstream player following Π up p satisfies, with probability at least 1 -t -ζ ,

We present the upstream player's UCB subroutine.

Proposition 2. Let s, t ∈ [ T ] such that s + t ⩽ T . Suppose that there exists a family ( τ a ) a ∈A of constant incentives associated with each arm a ∈ A such that we have in Algorithm 3: (˜ a l , τ ( l )) s + l ∈{ s +1 ,...,s + t } = (˜ a l , τ ˜ a l ) l ∈{ s +1 ,...,s + t } as an output of Π down p . Suppose that the distributions γ a are 1 -sub-Gaussian. Then with probability at least 1 -T -2 , the regret of the version of UCB given in Algorithm 3 run by the upstream player satisfies

Note that the major difference between this assumption and the regret bounds that we generally consider in multi-armed bandit problems is that we consider the regret without expectation here.

Algorithm 3 Upstream player's UCB

1: Input: Set of arms K , horizon T .

2: Initialize: For any arm a ∈ [ K ] , set ˆ µ a = 0 , T a = 0 .

3: for 1 ⩽ t ⩽ K : do

4: Pull arm A t = t

5: Update ˆ µ A t = X A t ( t ) , T A t ( t ) = 1

6: end for

7: for t ⩾ K +1 do

8: Observe the incentive (˜ a t , τ ( t )) .

9: Pull arm A t ∈ argmax a ∈ [ K ] { ˆ µ a ( t -1) + 2 √ log( KT 3 ) T a ( t -1) + 1 ˜ a t ( a ) τ ( t ) }

10: Update T A t ( t ) = T A t ( t -1) + 1 , ˆ µ A t ( t ) = 1 T At ( t ) ( T A t ( t -1)ˆ µ A t ( t -1) + X A t ( t ))

11: end for

Proof of Proposition 2. The proof is adapted from the proof of Bubeck et al. [2012, Theorem 2.1]. Let s, t ∈ [ T ] such that s + t ⩽ T . For any integer l ∈ { s + 1 , . . . , s + t } , we write n l ( a ) = Card { l ′ ∈ [ l ] such that A l ′ = a } for the number of pulls of arm a and ˆ µ a ( l ) for the empirical mean utility of the arm a ∈ A estimated on the batch { 1 , . . . , l } : ˆ µ a ( l ) = n l ( a ) -1 ∑ l k =1 1 a ( A k ) X a ( k ) . Since the rewards on the incentivized bandit instance are 1 -sub-Gaussian, a Hoeffding bound gives that for any δ ∈ (0 , 1) , a ∈ A , l ∈ { s +1 , . . . , s + t } , and any family of arms ( a l ′ ) l ′ ∈{ 1 ,...,l } such that Card { j : a j = a } = k , we have that

Therefore, for any δ ∈ (0 , 1) , a ∈ A , l ∈ { s +1 , . . . , s + t } , we have the following bound

)

and with an union bound, we obtain that

Considering the probability of the opposite event, we have that

where we rescaled δ as δ/T 2 K .

For the remaining of the proof, we take δ = T -2 and define a ⋆ l = argmax a ∈A { v up ( a ) + 1 ˜ a l ( a ) τ } . We now assume that the event { for any l ∈ { s + 1 , . . . , s + t } , a ∈ A , | ˆ µ a ( l ) -v up ( a ) | ⩽ 2 √ log(2 T 2 K/δ ) /n l ( a ) } holds. If at some step l ∈ { s + 1 , . . . , s + t } , action A l is chosen in Algorithm 3, it means that

with regards to the choice of actions in UCB based on the upper confidence bound. We now decompose the whole regret on the batch { s +1 , . . . , s + t } defined as ˜ R up p ( { s +1 , . . . , s + t } , UCB , Π down p ) . (20) ensures that with probability at least 1 -1 /T 2

and we have

where the last step holds because for any integers s, t ∈ N ⋆ , we have that

Using Cauchy-Schwarz inequality we obtain from (21)

which gives ∑ s + t l = s +1 √ 1 /n l ( A l ) ⩽ 2 tK .

Finally plugging all the terms together, since δ = 1 /T 2 , we obtain that with probability at least 1 -1 /T 2 √

̸

̸

Lemma 2. Assume H 2 holds and consider some arm a ∈ A such that we run the d -th batch of binary search on a with d ∈ { 1 , . . . , ⌈ log 2 T β ⌉} : for any t ∈ { k a,d + 1 , . . . , k a,d + ˜ T } , we have (˜ a t , τ ( t )) = ( a, τ a ) with τ a = τ mid a ( d ) and ˜ T = ⌈ T α ⌉ . Recall that we defined in (18) : T = a,d = Card { t ∈ { k a,d + 1 , . . . , k a,d + ˜ T } such that A t = a } . Let β ∈ (0 , 1) be such that β < α (1 -κ ) . Given that the event { ˜ R up p ( { k a,d +1 , . . . , k a,d + ˜ T } , Π up p , Π down p ) ⩽ C ˜ T κ } holds, we have that

̸

̸

Consequently, with probability at least 1 -2 T -αζ , if C ˜ T κ + β/α < T = a,d < ˜ T -C ˜ T κ + β/α , then | τ ⋆ a -τ a | ⩽ 1 /T β .

̸

Proof of Lemma 2. The whole proof is done conditionally on the event

Note that it holds with probability at least 1 -˜ T -ζ ⩾ 1 -T -αζ since we suppose that Π up p satisfies H 2.

Suppose that we have τ a ⩾ τ ⋆ a +1 /T β . By definition of the optimal incentives, we obtain, using by assumption τ a ⩾ τ ⋆ a +1 /T β

which ensures that a is the optimal arm for the upstream player during the batch { k a,d +1 , . . . , k a,d + ˜ T } that we consider. In that case, since a is the best arm, by definition of the upstream player's utility, the reward gap E [ v up ( a ) + τ a -( v up ( A t ) + 1 a ( A t ) τ a )] for the upstream player at any step t ∈ { k a,d +1 , . . . , k a,d + ˜ T } is at least v up ( a ) + τ a -max a ′ ∈A v up ( a ′ ) , and we obtain

̸

̸

by H 2, T = a,d being the number of steps for which a suboptimal arm has been chosen. Therefore, by definition of the optimal incentives, we obtain that

̸

̸

̸

̸

which gives: T = a,d ⩽ C ˜ T κ + β/α . Therefore, if we take the contrapositive, we obtain that during the sequence { k a,d +1 , . . . , k a,d + ˜ T } , if T = a,d > C ˜ T κ + β/α , then with probability at least 1 -T -αζ , τ a < τ ⋆ a +1 /T β , or equivalently τ ⋆ a > τ a -1 /T β .

Now suppose that τ a ⩽ τ ⋆ a -1 /T β . By definition of the optimal incentives, we obtain

̸

which ensures that a is a suboptimal arm for the upstream player during this batch of time steps. Therefore, arm a which has a reward gap bigger than 1 /T β since max a ′ ∈A { v up ( a ) + 1 a ( a ′ ) } -( v up ( a ) + τ a ) ⩾ 1 /T β and arm a has been picked ˜ T -T = a,d times. Consequently, we have that

̸

̸

which gives T = a,d ⩾ ˜ T -C ˜ T κ + /β/α . Therefore, if we take the contrapositive, we obtain that if T = a,d < ˜ T -C ˜ T κ + /β/α , then with probability at least 1 -T -αζ , τ a > τ ⋆ a -1 /T β , or equivalently τ ⋆ a < τ a +1 /T β .

̸

̸

For the second part of the proof, suppose that: C ˜ T κ + β/α < T = a,d < ˜ T κ + β/α -C ˜ T κ + β/α .

From the above result, we have that τ ⋆ a < τ a +1 /T β and τ ⋆ a > τ a -1 /T β with probability at least 1 -2 T -αζ . Plugging these inequalities in the absolute value | τ ⋆ a -τ a | concludes the proof.

Lemma 3. Assume that we run Algorithm 2 and consider some binary search batch iteration d ∈ ⌈ log T β ⌉ run on arm a ∈ A . Then 0 ⩽ τ a ( d ) ⩽ τ mid a ( d ) ⩽ τ a ( d ) ⩽ 1 .

̸

̸

Proof of Lemma 3. The proof proceeds by induction. Considering some action a ∈ A , we have for the initialisation before any binary search is run: τ a (0) = 0 , τ a (0) = 1 and therefore τ mid a (0) ∈ [ τ a (0) , τ a (0)] . We now consider that a number d of binary search batches has been run on a . Suppose that we run an additional binary search batch on action a . We have

After this iteration of binary search, we either have τ a ( d +1) = τ mid a ( d +1)+1 /T β and τ a ( d +1) = τ a ( d ) or τ a ( d + 1) = τ mid a ( d ) and τ a ( d + 1) = τ mid a ( d + 1) -1 /T β . Therefore, we still have 0 ⩽ τ a ( d + 1) ⩽ τ mid a ( d + 1) ⩽ τ a ( d + 1) ⩽ 1 , hence the result for any d ∈ ⌈ log 2 T β ⌉ by induction.

Lemma 4. Consider some arm a ∈ A and suppose that we have run D ∈ N ⋆ batches of binary search of length ˜ T on a . Then, we have that

Proof of Lemma 4. First observe that for any batch d ∈ { 1 , . . . , D } of binary search run on arm a during steps { k a,d +1 , . . . , k a,d + ˜ T } , we have by H 2 that

and applying a union bound over the D batches, we have that

which gives that

hence the result.

Lemma5. Suppose that the upstream player runs a subroutine Π up p satisfying H 2. Consider some action a ∈ A and the D -th binary search batch of length ˜ T run on arm a with D ∈ { 1 , . . . , ⌈ log 2 T β ⌉} . Then

and the probability of these events is at least 1 -⌈ log 2 T β ⌉ /T αζ . We also have that

Proof of Lemma 5. Suppose that the conditions of the lemma hold and consider some arm a .

We show by induction on the number of binary search batches D that have been run on a that ⋂ d ∈ [ D ] { ˜ R up p ( { k a,d +1 , . . . , k a,d + ˜ T } , Π up p , Π down p ) ⩽ C ˜ T κ } ⊆ { τ ⋆ a ∈ [ τ a ( D ) , τ a ( D )] } . If it is true, Lemma 4 completes this first part of the proof.

The initialisation holds since τ a (0) = 0 , τ a (0) = 1 and τ ⋆ a = max a ′ ∈A v up ( a ′ ) -v up ( a ) ∈ [0 , 1] with probability 1 - since max a ′ ∈A v up ( a ′ ) ∈ [0 , 1] and v up ( a ) ∈ [0 , 1] .

We suppose that the property is true for some integer D < ⌈ log 2 T β ⌉ and that we have run one more binary search on arm a . We have

τ mid a ( D + 1) being the incentive offered to the upstream player if he chooses action a during the D -th batch { k a,D + 1 , . . . , k a,D + ˜ T } . After this batch, if T = a,D < ˜ T -C ˜ T κ + β/α , BELGIC updates τ a ( D + 1) = τ mid a ( D + 1) + 1 /T β , τ a ( D + 1) = τ a ( D ) and Lemma 2 ensures that τ a ( D +1) < τ ⋆ a < τ a ( D +1) given { ˜ R up p ( { k a,D +1 +1 , . . . , k a,D +1 + ˜ T } , Π up p , Π down p ) ⩽ C ˜ T κ } . Thus the induction holds.

̸

̸

Otherwise, if T = a,D > C ˜ T κ + β/α , BELGIC updates τ a ( D + 1) = τ mid a ( D + 1) -1 /T β , τ a ( D + 1) = τ a ( D ) and Lemma 2 ensures that τ a ( D + 1) < τ ⋆ a < τ a ( D + 1) given { ˜ R up p ( { k a,D +1 + 1 , . . . , k a,D +1 + ˜ T } , Π up p , Π down p ) ⩽ C ˜ T κ } . The induction still holds.

Consequently, we have that for any number D of binary search batches run on arm a , τ ⋆ a ∈ [ τ a ( D ) , τ a ( D )] with probability 1 -D/T αζ

̸

For the second part of the proof, we define u ( D ) = τ a ( D ) -τ a ( D ) ⩾ 0 as the length of the interval containing τ ⋆ a with probability at least 1 -D/T αζ . We have u (0) = 1 . Suppose that after D iterations of binary search batches, the next batch of binary search { k a,D +1 +1 , . . . , k a,D +1 + ˜ T } outputs T = a,D +1 < ˜ T -C ˜ T κ + β/α . Then, the update of Algorithm 2 gives

On the other hand, if T = a,D +1 > C ˜ T κ + β/α , the update gives

We can see that ( u D ) D ⩾ 0 is an arithmetico-geometric sequence defined by u D +1 = u D / 2 + 1 /T β with an initial term u 0 = 1 . Writing r = 1 /T β / (1 -1 / 2) = 2 /T β , we obtain that

Lemma 6. Suppose that the upstream player runs a policy Π up p satisfying H 2. Considering some action a ∈ A , we have that after the binary search batch D = ⌈ log 2 T β ⌉ : P ( τ a ( D ) ⩽ τ ⋆ a ⩽ τ a ( D ) ⩽ τ a +3 /T β ) ⩾ 1 -D/T αζ .

Proof of Lemma 6. We suppose that the event ⋂ d ∈ [ ⌈ log 2 T β ⌉ ] { ˜ R up p ( { k a,d + 1 , . . . , k a,d + ˜ T } , Π up p , Π down p ) ⩽ C ˜ T κ } holds. Lemma 4 ensures that this event holds with probability at least 1 -⌈ log 2 T β ⌉ /T αζ .

̸

After D = ⌈ β log 2 T ⌉ batches of binary search on arm a , we have by Lemma 5 that

Lemma 5 guarantees that τ ⋆ a ∈ [ τ a ( D ); τ a ( D )] with probability at least 1 -D/T αζ , and we obtain τ a ( D ) ⩽ τ ⋆ a ⩽ τ a ( D ) ⩽ τ a ( D ) + 3 /T β with the same probability.

Proposition 1. Under H 1 and H 2, after the first phase of BELGIC which consists in K ⌈ T α ⌉⌈ log T β ⌉ steps of binary search grouped in ⌈ log 2 T β ⌉ batches per arm a ∈ A , we have that

Proof of Proposition 1. We consider a number of binary search batches D = ⌈ log T β ⌉ and we define the event G as G = { for any a ∈ A , τ a ( D ) ⩽ τ ⋆ a ⩽ τ a ( D ) ⩽ τ a ( D ) + 3 /T β } . We have that

where the last inequality holds with an union bound. Lemma 6 with D = ⌈ log 2 T β ⌉ ensures that we have

and since Card {A} = K , we obtain

Since the estimated incentives are defined as ˆ τ a = τ a ( ⌈ log 2 T β ⌉ ) + 1 /T β + C T ( κ -1) / 2 , we can conclude

since whenever τ a ( ⌈ log 2 T β ⌉ ) ⩽ τ ⋆ a ⩽ τ a ( ⌈ log 2 T β ⌉ ) ⩽ τ a ( ⌈ log 2 T β ⌉ ) + 1 /T β , we also have by definition: ˆ τ a ( ⌈ log 2 T β ⌉ ) -4 /T β -C T ( κ -1) / 2 ⩽ τ a ( ⌈ log 2 T β ⌉ ) ⩽ τ ⋆ a ⩽ ˆ τ a ( ⌈ log 2 T β ⌉ ) .

Theorem 3. Assume that H 1 and H 2 hold. Then BELGIC , run with α, β satisfying (13) and any bandit subroutine Bandit-Alg , has an overall regret R down p such that

where, for ease of notation

Proof of Theorem 3. Suppose that the conditions of Theorem 3 are satisfied. By definition, Λ K +1 + 1 ∈ [ T ] is the step at which starts the run of the subroutine Bandit-Alg , since Λ K +1 = K ⌈ T α ⌉⌈ β log T ⌉ . All the binary searche batches have length ˜ T = ⌈ T α ⌉ . For any a ∈ A , d ∈ { 1 , . . . , ⌈ log 2 T β ⌉} , we define the event

as well as

and by H 2 and Lemma 4, with an union bound, we have that

and we now decompose

where ˜ R down p is defined as

By definition, µ ⋆, down ⩾ min a,b ∈A×A v down ( a, b ) ⩾ v and since Lemma 1 allows to write µ ⋆, down = max a,b ∈A×A { v down ( a, b ) + v up ( a ) } -max a ′ ∈A { v up ( a ′ ) } , we have that

and note that since κ < 1 , for any a ∈ A , ˆ τ a ⩽ 2 + C T ( κ -1) / 2 ⩽ 2 + C . Consequently, Tv ⩽ ˜ R down p ( T, Π up p , Π down p ) ⩽ (3 + C + ¯ v -v ) T almost surely. Therefore

We consider the second term in (23). We decompose it between the steps of binary search { 1 , . . . , K ⌈ T α ⌉⌈ log 2 T β ⌉} during which we run the Binary Search Subroutine and the following ones when we run Bandit-Alg , which gives

Similarly to (24), we use the bound on µ ⋆, down to bound ( A ) , which gives

After the binary search, at each step t ∈ { Λ K +1 +1 , . . . , T } , Bandit-Alg recommends (˜ a t , B t ) ∈ A×A following (15) and BELGIC offers an incentive (˜ a t , ˆ τ ˜ a t ) with ˆ τ a = τ a ( ⌈ log 2 T β ⌉ ) + 1 /T β + C T ( κ -1) / 2 .

Lemma 5 ensures that E ⊆ { τ ⋆ a ∈ [ τ a ( ⌈ log 2 T β ⌉ ) , τ a ( ⌈ log 2 T β ⌉ )] for any a ∈ A} ⋂ { ˜ R up p ( { Λ K +1 + 1 , . . . , T } , Π up p ) ⩽ C( T -Λ K +1 ) κ } . Therefore, if E holds, for any a ∈ A , we have that

̸

and therefore, for any a ′ ∈ A such that a ′ = a , we have that

This shows that at any steps t ∈ { Λ K +1 +1 , . . . , T } after the binary search, we have on the event E that

̸

and the reward gap at step t for any a = ˜ a t is defined as

Following (27), the reward gap from (29) satisfies

We now define two sets

̸

which satisfy I T ∪ J T = { K ⌈ T α ⌉⌈ log 2 T ⌉ + 1 , . . . , T } almost surely. As shown in (28), I T corresponds to all the steps during which the upstream player picked the best arm and for any t ∈ I T

while by (29), for any t ∈ J T , we have that

H 2 ensures that if E holds, then R up p ( { Λ K +1 +1 , . . . , T } , Π up p , BELGIC ) ⩽ C T κ , and this condition together with (30) gives that

and consequently Card { J T } ⩽ T ( κ +1) / 2 . We now bound ( B ) as follows

where the first step holds by Lemma 1. Using Lemma 5, as well as the definition of E , we obtain

which finally gives

E [ 1 ( E )( B )] ⩽ R Bandit-Alg ( T, ν, { ˆ τ a } a ∈A ) + 4 T 1 -β +(3 + 2C + ¯ v -v ) T (1+ κ ) / 2 , (31) and plugging together (26) and (31) in the decomposition (25) gives the following bound

and summing the bounds on the events E and E c finally gives

Corollary 1. Assume that the upstream player's distribution ( γ a ) a ∈A is such that H 1 holds. In addition, suppose that the distributions ( γ a ) a ∈A and ( ν a,b ) a,b ∈A×A are 1 -sub-Gaussian and that the upstream player plays Π up p = Algorithm 3 (a slight modification of UCB to take into account the incentives). Then the downstream player's regret when she runs BELGIC with parameters α = 3 / 4 and β = 1 / 4 (which satisfy (13) ) and subroutine Bandit-Alg = UCB satisfies the following upper bound 4

Proof of Corollary 1. First note that Π up p = Algorithm 3 satisfies H 2 with constants κ = 1 / 2 , ζ = 2 , C = 8 √ K log( KT 3 ) , following Proposition 2. Note that β/α = 1 / 3 < 1 / 2 = 1 -κ , therefore Equation (13) is satisfied. Plugging these terms in the bound from Theorem 3 with α = 3 / 4 , β = 1 / 4 gives

where we use the bound for R Bandit-Alg ( T, ν, { ˆ τ a } a ∈A ) with Bandit-Alg = UCB run on any bandit instance with K 2 arms, 1 -subgaussian rewards and reward gaps of at most max a,b ∈A×A v down ( a, b ) -min a,b ∈A×A v down ( a, b ) = ¯ v -v , following [Lattimore and Szepesvári, 2020, Theorem 7.2]. Therefore, we have that

which finally gives

hence the result.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract mostly presents the issue that we tackle in the paper, namely presenting an online version of the Coase theorem . This contribution is the most important part of our paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Our work presents a model to explore an online version of a theory from economics. Thereby, it implicitly has limitations due to the fact that a choice was made in the model.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our work presents a lot of theorems supported by assumptions (see H 2, Theorems 2 and 3...). All of Appendix C is here to provide a theoretical support to these results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: Since our work is mostly a theoretic contribution, we do not present experiments. Therefore, our paper is not concerned by this issue.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: Since our work is mostly a theoretic contribution, we do not present experiments. Therefore, our paper is not concerned by this issue (same answer as for item 5).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: Since our work is mostly a theoretic contribution, we do not present experiments. Therefore, our paper is not concerned by this issue (same answer as for item 5).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Since our work is mostly a theoretic contribution, we do not present experiments. Therefore, our paper is not concerned by this issue (same answer as for item 5).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Since our work is mostly a theoretic contribution, we do not present experiments. Therefore, our paper is not concerned by this issue (same answer as for item 5).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: As explained in the Related Works, an issue with externalities is the harm caused to all the parties involved in the considered setting. Here, we try to provide a setup so the players can negociate and achieve a social welfare optimal equilibrium. Therefore, we are confident in the fact that we follow the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We consider an issue arising in several real-world situations and provide an algorithmic solution to it. We realize that this theoretical contribution still needs some work to be implemented but we hope that it will contribute to positive social impacts in the future.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: Our work is a theoretical contribution. Thereby, it does not propose a release of data or any kind of trained model. We mix learning and theories from economics, which does not involve yet risks for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We do not use any assets requiring such conditions here.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not present such kind of new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not incorporate any experiment involving human subjects. Therefore, we are not concerned by this item.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not incorporate any experiment involving human subjects, hence our answer.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers a two player game with externalities imposed from one agent (“upstream”) to the other (“downstream”) agent. Equilibrium in the absence of taxation or payments between players would result in heavy efficiency loss due to the externalities.\n\nThe Coase theorem tells us that in these situations we should expect to see payments from the downstream agent to the upstream agent in order to restore the socially optimal outcome.\n\nThis work formulates this scenario as a multi-armed bandit problem, in which the payment term forms a part of the players payoffs and options. Namely, the downstream agent can put a conditional payment on the arms played by the upstream bandit. In this situation, the paper asks whether or not normal bandit learning algorithms can learn the optimal outcome.\n\nThe paper puts forwards an assumption that must be satisfied on the upstream agents learning behavior, which is satisfied for many common bandit learning algorithms. A translation is given for the downstream agent to go from the raw bandit problem to a transformed bandit problem. With these, sublinear regret is realized relative to the socially optimal outcome.\n\nSTRENGTHS:\nThe work takes a well-known economic theory and asks whether a given type of ML can recover the expected phenomenon without knowing the specific externality a prior. Rather than showing an exact algorithm for recovering it, mappings are done so that any suitable bandit learning algorithm can successfully learn the results. This is a nice addition over prior work, and helps in suggesting that this is something with broad learnability. In the process, it is interesting to see the manner in which the externality needs to make it into the payoffs and how that space needs to be structured.\n\nOverall the paper is well written and has good fit at NeurIPS. Relative to prior work, it is able to show that as long as the upstream agent is using a suitable approach, any bandit learning algorithm will work. This is a much stronger guarantee for the downstream agent as they are less dependent on exactly what the upstream agent is doing.\n\nWEAKNESSES:\nThe paper and situation is quite interesting, but is one specific (classic) example scenario so applicability is good because it is a classic example, but still limited. A natural followin question is how broad do we expect these situations to go - can we recover general externalities in multi agent situations?\n\nQUESTIONS:\nHow broad do you expect the BELGIC routine to be applicable? In a general $M$ agent situation with less obvious structure on the externalities would you expect optimal transfers to still be learned efficiently?\n\nLIMITATIONS:\nYes the authors have adequately addressed limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the field of mitigating externalities in economic interactions by applying the Coase Theorem with hindsight rationality. The Coase Theorem, a fundamental concept in economics, suggests that in the presence of externalities, property rights, and bargaining strategies can be utilized to achieve an optimal outcome for all parties involved. However, traditional applications of the Coase Theorem assume that players have perfect knowledge of the game, which may not hold true in real-world scenarios.\n\nIn this paper, the authors introduce the concept of hindsight rationality, where players learn from past interactions to improve their decision-making processes. By incorporating hindsight rationality into the Coase Theorem framework, the paper aims to provide a mechanism through which players can adapt their strategies over time to maximize social welfare, even in the presence of uncertainty. The theoretical foundation of the paper is built upon a series of theorems supported by assumptions and proofs. \n\nOverall, the paper offers a novel perspective on addressing externalities in economic settings by integrating hindsight rationality into the Coase Theorem framework. Through a rigorous theoretical analysis, ethical considerations, and a discussion of limitations, the paper contributes to the ongoing discourse on optimizing social welfare in the presence of externalities.\n\nSTRENGTHS:\nS1. One of the main strengths of the paper is its solid theoretical foundation. The authors create a clear framework based on well-known economic principles, like the Coase Theorem, and expand it by adding the idea of hindsight rationality. The thorough theoretical analysis, backed by detailed proofs and assumptions, strengthens the credibility and robustness of the proposed approach.\n\nS2. By introducing the concept of hindsight rationality within the context of mitigating externalities, the paper offers a novel perspective on addressing economic inefficiencies.\n\nWEAKNESSES:\nW1. One significant weakness of the paper is the lack of empirical validation or experimental results to back up the proposed theoretical framework. Although the theoretical analysis is good, empirical evidence would improve the practical applicability and real-world relevance of the research findings.\n\nW2. It would have been insightful to see a comprehensive discussion on the potential challenges and complexities associated with implementing the proposed approach in practical economic settings. Addressing implementation hurdles could provide valuable insights for policymakers and practitioners.\n\nW3. The setting (extension of Coase theorem in a two-player bandit setting where the actions of one of the players affect the other player) considered in the paper is narrow and limited. The results are also exemplified for a simple setting of two firms, an upward and a downward. Do the observations apply to potentially broader settings? \n\nW4. The mathematical notations provided in the paper sometimes become too difficult to follow and correlate with the text.\n\nQUESTIONS:\nPlease clarify the points in the Weaknesses section.\n\nLIMITATIONS:\nYes, the limitations of the proposed method are discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a two-player sequential game with bandit feedback where one player's actions create externalities that affect the other player's outcomes. In this model, the utility of only one player (the downstream player) is impacted by the collective actions of both the upstream and downstream players. They consider two scenarios: one where players act independently to maximize individual utility, and another where players can interact via transfers.\n\nIn the first scenario (without transfers), the authors show that individual utility maximization can lead to suboptimal social welfare in some instances. In the second scenario, where the downstream player can choose a transfer, they demonstrate that social welfare can reach its optimum. They propose an algorithm for the downstream player that achieves this outcome, provided the upstream player follows any no-regret policy. This effectively presents an online version of the Coase theorem, establishing that even with imperfect information, the two-player sequential game converges to the global optimum outcome when transfers are allowed.\n\nSTRENGTHS:\n1. The algorithm does not assume a specific policy for the upstream player and achieves sublinear regret for any no-regret upstream policy (with regret scaling as $T^\\kappa$).\n2. Lemma 1 provides a clean problem formulation, showing that bounds on regret in individual utility (with transfers) lead to a bound on the social welfare regret.\n3. For the specific instance where the upstream player follows the Upper Confidence Bound (UCB) policy (or any other algorithm with  $\\sqrt{T}$ regret) the proposed algorithm achieves regret scaling as  $T^\\frac{3}{4}$.\n\nWEAKNESSES:\nPlease see the questions section.\n\nQUESTIONS:\n1. It is not clear whether the regret incurred by the downstream player is optimal. Even if the upstream player is assumed to follow UCB, it's not evident whether a regret scaling as $\\sqrt{T}$ policy is achievable for the downstream player as well. \n2. It is stated (on lines 111-112) that the assumption that the rewards are bounded is \"without loss of generality\" (WLOG). Can you please explain why? Would this work even in the case of other standard reward distributions in bandit literature, such as sub-Gaussian rewards?\n3. In the current setup, it is assumed that transfers occur only on a single action, with the transfer function taking a value of 0 for all but one action. Is there a particular reason for assuming such a transfer function? Can we work with more general transfer functions where the downstream player can choose to make transfers on multiple actions? How would this affect the regret of the downstream player?\n\nLIMITATIONS:\nThis paper does not have any negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}