Learning to Mitigate AI Collusion on Economic Platforms
Authors: Gianluca Brero
Eric Mibuari
Nicolas Lepore
David C. Parkes
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-agent Systems
Reinforcement Learning
Platform Design

Abstract: Algorithmic pricing on online e-commerce platforms raises the concern of tacit collusion, where reinforcement learning algorithms learn to set collusive prices in a decentralized manner and through nothing more than profit feedback. This raises the question as to whether collusive pricing can be prevented through the design of suitable "buy boxes," i.e., through the design of the rules that govern the elements of e-commerce sites that promote particular products and prices to consumers. In this paper, we demonstrate that reinforcement learning (RL) can also be used by platforms to learn buy box rules that are effective in preventing collusion by RL sellers. For this, we adopt the methodology of Stackelberg POMDPs, and demonstrate success in learning robust rules that continue to provide high consumer welfare together with sellers employing different behavior models or having out-of-distribution costs for goods.

Learning to Mitigate AI Collusion on Economic Platforms
Gianluea Brero" Data Science Initiative
Eric Mibuari
School of Engineering and Applied Sciences Harvard University
Brown University gianluca_brent har.iiaabbbrrrrroo.eeeee
mibuar og. harvard.edd edu
Nicolas Lepore
David C. Parkes School of Engineering and Applied Sciences Harvard University parkes0g harvard. edu
School of Engineering and Applied Sciences Harvard University
nlepore33@gmai..cmm com
Abstract
Algorithmic pricing online e-commerce platforms raises the concern of tacit collusion, where reinforcement learning algorithms learn to set collusive prices in decentralized manner and through nothing more than profit feedback. This raises the question as to whether collusive pricing can be prevented through the design of suitable "buy boxes," i.e., through the design of the rules that govern the elements of e-commerce sites that promote particular products and prices to consumers. In this paper, demonstrate that reinforcement learning (RL) can also be used by platforms to learn buy box rules that are effective in preventing collusion by RI sellers. For this, adopt the methodology of Stackelbery POMDPs, and demonstrate success learning robust rules that continue to provide high consumer welfare together with sellers employing different behavior models or having out-of- distribution costs for goods.
Introduction
The last decade has witnessed dramatic shift of trading from retailers online e-commerce platforms such as Amazon and Alibaba. In these platforms, sellers are increasingly using algorithms to set prices. Algorithmic pricing can be beneficial for market efficiency, enabling sellers quickly react market changes and also in enabling price competition. At the same time. the U.S. Federal Trade Commission (FTC) U.S. Federal Trade Commission (2018) and European Commission (The Organisation for Economic Co-operation and Development) (217)) ) have raised concerns that algorithmic pricing may (2020a) support these concerns through a study of pricing agents in simulated platform economy, and show that commonly used leinforcement-leaniii (RL) algorithms learn to initiate and sustain collusive behaviors Assad et al. (2020) also provide empirical support for algorithmic collusion in study of Germany's 's retail gas stations, showing an association between algorithmic pricing and an increase in price markups. As highlighted by Callano
facilitate collusive behaviors. Calv
et al. 22220b). these kinds of collusive behaviors are unlikely to be violation of antitrust law, as they are learned responses to profit signals and not the result of explicit agreements.
One can try to prevent algorithmic collusion by introducing suitable rules by which platforms can choose which sellers promote to buyers, thus promoting competition. Could Amazon's "buy box
These authors contributed equally this work.
MMor precisely, Calvano et al. 2020a) showed that RL algorithms learn to quote supra-competitive prices and punish deviations from collusive agreements via lower prices.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

algorithm," for example, play this role in the future, in determining for given consumer search which products and prices to highlight a consumer? Responding to this, JJhnnson al. (20TT) design hand-crafted rules that succeed in hindering collusion between RI. algorithms. At the same time, their rules introduce the undesirable effect of limiting consumers to single seller, and there remains potential for more effective interventions.
In this paper, we demonstrate for the first time how RL can also be used defensively by a platform to automatically design rules that promote consumer welfare and prevent collusive pricing. This is a problem of mulli-agent learning, with the interaction between the platform and sellers modeled as a Stackelberg game (Fudenberg and Tirole 1991). The leader is the platform designer and sets the platform rules and the sellers respond, using RL to set prices given these rules. We introduce the class of threshold platform rules, and formally show that this class contains rules that approximately maximize consumer surplus a unique subgame perfect equilibrium MMs-Colell et al. 1995 chapter 9). At the same time, this class of threshold rules is fragile to unexpected deviations by sellers, for example caused by cost perturbations. The role of RL on the part of the platform is learn rules with similar performance that are also more robust.
To solve the Stackelberg problem, we make use of the Stackelberg partially observable Markov decision process (POMDP) framework (Brero el al., 2022), which defines an episode structure of a POMDP such that the RL algorithm representing the leader will learn to optimize its reward (here, consumer surplus) given that its rules cause re- re-equilibration on the part of the followers (here, the sellers who use Q-learning algorithms to set prices). The Stackelberg POMDP framework is well formed as long as the re-equilibration behavior of the sellers can be modeled through Markovian dynamics, as is the case with Q-learning.
We show successful results in learning effective platform policies that outperform handcrafted rules (Johnson et al. 202T). This demonstrates how the Stackelberg POMDP framework can be successfully applied in settings where followers play repeated games, and their strategies are also policies trained via reinforcement learning algorithms. We then show how our threshold platform rules allow us to obtain a similar learning performance when training the platform policy "in the wild," i... without accessing the sellers' private information. With this, we demonstrate how the Stackelberg POMDP framework can be applied in more general learning scenarios than the offline learning ones for which was originally designed. Finally, we show how the platform rules learned via our Stackelberg POMDP framework continue to be effective when market conditions change, for example as the result of change to the cost structure of sellers.
Further related work. Zheng et al. (022); Tang 2017); Shhn et al. (2020); Brero et al. 2021 make use of RL to optimize different economic systems (including matching markets, internet advertising, tax policies, and auctions) under strategic agents' responses. Unlike our work, these methods do not leverage the designer's commitment power or the Stackelberg structure of the induced game. [Brero et al. (002)) introduce and study the Stackelberg POMDP framework for matrix games and simple auctions using regret minimization for followers!? P Abada and Lambin (022) study collusion by RL pricing in markets for electric power, and use machine learning by regulator agent for the mitigation of collusion, albeit without a Stackelberg framing. The broader research program on disferentiable economics uses representation learning for optimal economic design (Duetting
let al.
2019:
Shenn et al. 2019| Kuo et all, 2020 T'cchettii et al. 2019 Rahme et al. 202 ag Curry >) this work avoids the need for mechanisms. Also related
al.
2022 Rahme al. 2021b, Curry al. 2020 Peri al. 2021):
Stackelberg design by emphasizing the use of direct, incentive-coppatin
empirical mechanism design (Areyan Viqueira al. 2019 Vorobeychik et al.) 2006 Brinkman and Wellman 2017). which applies empirical game theory to search for the equilibria of mechanisms with a set of candidate strategies (Wellman 2006; Kiekintveld and Wellman 2008 Jordan et al. 2010 see also Bünz et al. (2018) for the design of iterative auctions.
"The only other method we know for Stackelberg learning in stochastic games with provably guarantees solves for single follower (Mishra et al. (2020;; see also Mguni et al. (2019): [Cheng et al.) (2017); (Shi et al. (020) and Shu and Tian 2019), and [ T'harakunnel and Bhattacharyya (200)) for partial convergence result for a static game with two followers. For other convergence results for single- folloerr static, and often zero-sur sum games see et al. (2019): : Sengupta and Kambhampati 2020); Xu et al 2021) Fiez et al. (2020); Jin et al. et
2019):/Cheng
2020). For multi follower static games, Wang et al. (022)) make use of differentiable relaxation of follower best-response behavior together with subroutine to solve an optimization problem for follower behavior.
2

---

2 Preliminaries
Seller Competition Model. There is a set of sellers == (1...,, economic platform. Each seller has the same marginal cost c >> 0 for producing one unit of its product. Sellers interact with each other repeatedly over time in setting prices and selling goods. At each time period, L = 0, 1,.... each seller observes all past prices, and sets a price Pi,t 2 0. We let pt. = (P1.1...... Pn.t)) denote a generic price profile quoted at time I. The platform sets the rules of buy box that governs, in each period I, which set N₁ C N of sellers available. Consumers can only buy from these sellers and others forfeit sales. There is also an outside option, indexed by O, which provides each consumer with a fallback choice with zero utility.
"T,...,,],, n), each of whom sells a
differentiated product
Following Johnson et al. (001)) competition between sellers for consumer demand is modeled through the standard logit model of consumer choice. For this, seller has quality index a, > 0, this providing horizontal differentiation across products, and the outside good has quality index ag > O. In the logit model, each consumer samples Co- SI,... independently from type I extreme value distribution with scale parameter μu > O, for each product and the outside option, with utility o; + 155--;,, Pte for product and + Co for the outside option. Considering unit mass of consumers in period I, seller i E EN, receives fractional demand D. (g) = exp((a, 1004--PP (ete,) pP..)/ h1) (Ppr; where A(pr; = Σ jeN, exp((ay - P),,)/H) /)) exp( (x0/ 1t), and any seller i & .Ng has zero demand. Scale parameter μ > Ô serves control the extent of horizontal differentiation, with differentiation and perfect substitutes obtained as + O. The total consumer surplus is U NV) = log (X Wi)], and is maximized with minimum prices and all sellers displayed (so consumers have a full choice of products). Seller i's profit pr in period / is p; (pi, - c) (;(p; i.e., its per-unit profit multiplied by demand.
Reinforcement learning by sellers. In a single-agent Markov decision process (MDP), an agent faces a sequential decision problem under uncertainty. At each step I, the agent observes a state variable 8, S and chooses an action a E A. Upon action a, in state 8,, the agent obtains reward r(st, at), and the environment moves to state st41 according to ((s dt). We let (so, ag ..., ST, aT) denote state-action trajectory determined by executing policy policy : - A. and (T) denote the probability of trajectory т. The optimal policy x solves r" E argmax, a Er-g. (r) E
5 5'r(st, T can be finite or infinite. In partially-obserrable MDP (POMDP), the policy π cannot access state 8y but only observation of sampled from (0))(s)). A mulli-agent MDP (Boutilier) 1996) for n agents has states S common to all agents and a set of actions A, for each agent i. When each agent i picks action ai,r in state 8t, the environment moves to state 8t+-1 according to a distribution ((st+1/5e. 01... ... (2 and agent i obtains a reward ri (52 az) that depends on the joint action. We follow Calvano et al. (2020a) and Johnson et al. (22TT) and adopt decentralized Q-learning by sellers as positive theory for sellers in regard to their behavior in setting prices on an e-commerce platform (see Appendix A). Although Q-learners may not converge, we also confirm these earlier studies in showing convergence in our simulations (defined over particular time horizon as detailed by Johnson
where 5 E [0. 1] is the discount factor and time-horizon
al. (202I).
3
The Platform Stackelberg Problem
To formalize the problem facing the platform designer in mitigating collusive behavior by sellers, we model the interaction between the platform, which sets the rules of the buy box, and the sellers as Stackelberg game. The platform designer is the leader, and fixes the platform rules. The sellers are the followers, and play an infinitely repeated game according to these rules. As discussed above, and following Calvano et al. (2200)) and Johnson et al. (2021), we model the sellers' behavior through decentralized Q-learning. As a result, the problem facing the platform is a behavioral Stackelberg problem, in that the followers are modeled as Q-learners (and need not, necessarily, be playing an equilibrium of the induced game).
The sellers. this model, we fix the states that comprise the MDP of seller to include the prices set by all sellers in the last period, i.e., 8, Pt-1- We initialize so to be randomly selected price profile. The action of a seller is modeled as one of m equally-spaced points in the interval ranging from just below the sellers' cost to just above the monopoly price pm At each step t > 0, each
3

---

seller selects a price Ps.t and is rewarded by its per-period profit ps (p(::N)), which depends on Pt (P.,.....,PP.. Pn,t)) and the choice of which sellers N₁ are displayed by the platform.
The platform. To formalize the platform's 's problem, let a = =oi,,., oa)) denote strategy profile selected by Q-learning on the part of sellers, in response to the platform rule, and in the long run, after a suitably large number of steps. We leave implicit here the dependence of seller strategy profile on the platform's 's policy. The platform must decide in each period which sellers to display consumers. For this, we denote the platform rule as policy x, and we adopt for the state of the platform policy the prices quoted by sellers in step t. Pt, so that the platform' 's policy uses these prices to select a set of agents to display, with N₁ selected according to T (pt). Let P a (s,) denote a price profile chosen under seller strategies a², i.e., in response to the platform rules, and at some large enough time step t", and let -" ( +1... denote a trajectory of prices forward from 1" (the dependence on the platform's 's policy left implicit this notation). We denote the distribution of these trajectories as qa (r*). The Stackelberg problem facing the platform is to find platform policy that maximizes consumer surplus given the effect of this policy on the induced strategy profile of sellers
Definition 1 (Behavioral Stackelberg Problem) The optimal platform policy solves T' E argmax, (T),), where (T) is the expected sum consumer surplus when sellers follow strategy o² forward from period t", i.e.,
CS(T)= = Eg-ngg(T")
Σ (ultim(tit)
where T is suitably chosen horizon and 9x (T*) denotes the distribution over Q-learning induced seller pricing trajectories in response to platform policy n.
Learning Optimal Platform Rules
In this section, we solve the platform's problem, in responding to Q-learning sellers, through the Stackelberg POMDP framework (Brero et al. 2022). This creates a suitably defined POMDP in which the optimal policy solves the behavioral Stackelberg problem (Definition I
Definition (Stackelberg POMDP for platform rules) The Stackelberg POMDP for platform rules is finite-horizon POMDP, where each episode has the following two phases:
An equilibrium phase, consisting n. 2 steps. In this phase, each state st includes the step counter i, the sellers' current Q-matrices, and the prices quoted by the agents. Observations consists of the prices quoted by the sellers (or = pt) and policy actions determine the set of agents displayed (in their more general version, a₁ = Ny). State transitions are determined by Q-learning, where each seller updates its Q-matrix after being rewarded by NC). The policy has zero reward in every time step (r(St, 0.) = 0, for S ne).
A reward phase, consisting nf, 2 1 steps, each with the same actions, states, and observations as the equilibrium steps. The reward phase differs in two ways. First, the Q-matrices of the sellers are not updated, and second, the platform policy now receives non-zero reward, and this is in each step to be equal to the consumer surplus in that step (r(*t, at) = (pi; N.),, for >> ne).
This Stackelberg POMDP formulation is an adaptation of that provided by Brero et al. (202)), who used it to learn leader strategies in matrix games and allocation mechanisms. Following Brero et al. (0022), show the Stackelberg POMDP formulation is well-found rell-founded by showing that an optimal policy will also solve the Behavioral Stackelberg design problem of Definition [ Specifically, when the number of reward steps n, is large enough and when n. 2 t", the optimal policy, denoted o Th.,,, for the Stackelberg POMDP with ne equilibrium and n, reward steps maximizes the objective in Equation 11
Proposition 1 The optimal Stackelberg POMDP policy nh for an equilibrium phase with n. 2 steps and a reward phase with n, 2 steps, maximizes CS(T)), ), for seller behavior induced after n. steps when n, = T*
NNte that we can use finite time horizon as trajectories 7" consists of price cycles due to our sellers' behavior model.

---

The proposition follows from the construction of the Stackelberg POMDP, especially the fact the our policy is only rewarded under the response behavior reached after n. steps, in line with the definition of (T) (see Appendix B for the full proof argument).
Brero al.
2022 use the Stackelberg POMDP framework in "offline" environment, i.e., in a simulation that assumes access, at design time, to followers' internal information. This allows them to solve their POMDP using the paradigm of centralized training and decentralized execution Lowe et al. 2017). The leader policy is trained via an actor-critic deep RL algorithm, and the critic network (which estimates the sum of rewards until the end of the episode) accesses the full state during training. Only the actor network, which represents the policy, is restricted to the partial-state information.
Here, we also study the use of the Stackelberg POMDP framework to train useful leader policies "in the wild," " where the learning algorithm of the platform can only access the kind of information that an economic platform would have based on observations of sellers. As we will empirically demonstrate, we can successfully operate without access sellers' private information in regard to Q-matrices and exploration rate without affecting learning performance
Threshold platform rules. In our experiments, we consider the class of threshold platform rules. These rules use the current prices set by sellers to set a price threshold above which a seller will not be displayed, with the same threshold set for all sellers.
Definition 3 (Threshold Platform Rule) A threshold platform rule sets threshold T(p:) 2 for each price profile pt, such that N₁ = E (1,..,n} n) Pi,t S (p:)), iee, any seller whose price is no greater than the threshold is displayed to consumers.
This class of threshold rules has corresponding optimality result: there is threshold rule that makes the market competitive, with all sellers displayed and consumer surplus maximized in subgame perfect Nash equilibrium (SPE) of the induced continuous pricing game. Even though the pricing behaviors that arise from Q-learning need not converge to SPEs, we use these equilibria as a theoretical support for our platform intervention choice, as previously done by Dohnson et al. (021 We have the following result:
Proposition 2
For any >> O, there exists a threshold platform rule such that CS(T) 2 CS Ee 8'e under a subgame perfect Nash equilibrium (SPE) of the feinitely-repaater continuous pricing game induced by platform rule π.
This proposition follows from a platform rule with limiting threshold that is arbitrarily close to the sellers' cost e (see Appendix for the proof). Under this rule, sellers are displayed only if their price is minimal. At the same time, this particular threshold rule is fragile, and would lead to market failure seller costs vary in unexpected ways. By letting the threshold also vary with the price profile pt. as is allowed by the family of threshold platform rules, we seek to learn milder interventions that still mitigate collusion but remain robust to variations in the costs faced by sellers in the marketplace.
Experimental Results
In this section, we evaluate our learning approach via three main experiments. We first consider performance in terms of consumer surplus, benchmarking
RL interventions against the ones introduced by Dohnson et al. 202TT. We demonstrate the ability to learn optimal leader strategies in the Stackelberg game with the followers across all the seeds we tested, significantly outperforming existing interventions. We then train platform rules without access to the sellers' private information ("in the wild"), and show that this is not crucial for our learning performances. We conclude by lesting the robustness of our interventions, adding price perturbations during training and evaluating the effect on the robustness of our learned platform rules in environments where sellers have different costs from those assumed during training.
*We notice this also in line with the recent findings in Fujimoto et al. 2022) highlighting how the Bellman error minimization (for which we require environments to be Markovian) may not be a good proxy of the accuracy of the value function.
5

---

Experimental set-up. As in Calvano et al. (2020a) and Johnson et al. 2021 we consider settings with two pricing agents with cost 1, quality indexes ai Q2 2. and ag 0, and we set parameter μ = 0.25 to control horizontal differentiation. The seller Q-learning algorithms are also trained using discount factor 5 = 0.95, exploration rate E = e- with / 1e 5, and learning rate = 0.15. We also include results for variations of this default setting in Appendix D A particular concern is that if the exploration rate is still high during the reward phase of the Stackelberg POMDP episode, our policy may be rewarded based on random prices. To address this problem, one could extend the Stackelberg POMDP equilibrium phase to reach a minimal E, or isolate stages when price profiles more stable to audit rewards. To satisfy our computational constraints, we pause exploration during the reward phase of the Stackelberg POMDP. As in Johnson et al. (02T), these prices range from 0.95 (just below the sellers' cost) to 2.1 (which is above the monopoly price under no intervention). Farlier work provided sellers with a choice of fifteen different prices (over a similar range). We need a smaller grid in order satisfy our computational constraints; earlier work studied the effect of different, hand-designed platform rules, and did not also use RL for the automated design of suitable platform rules. We also follow the choices of earlier work, and study economy with two sellers (again, for reasons of computational resources). This coarsened price grid allows us to train a platform policy through Stackelberg POMDP for 50 million steps in 18 hours using a single core on Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz machine.
Learning algorithm. To train the platform policy, we start from the A2C algorithm provided by Stable Baselines3 (Rathin et al., 202T MIT License). Given that our policy is only rewarded at the end of Stackelberg POMDP episode, we configure A2C so that neural network parameters are only updated after this reward phase. In this way, we guarantee that policies inducing desired followers' equilibria are properly rewarded. Furthermore, reduce variance in sellers' responses due to non-deterministic policy behavior, we maintain an observation-action map throughout each episode. When a new observation is encountered during the episode, the policy chooses an action following the default training behavior and stores this new observation-action pair in the map. We will show the importance of this variation via an ablation study that is presented in Appendix E In this section, we assume that sellers restart the Q-learning process by re-initializing exploration rates every time the platform rules change (i.e., at the beginning of every Stackelberg POMDP episode). In Appendix F. we also show how the training approach is robust to different sellers' behaviors, where the sellers restart the learning rate asynchronously. and not necessarily at the beginning of episodes.
5.1 Platform Learning Performance
In this section, we evaluate the performance of our learned platform policies. For this, we train policies for 50 million steps in total. We set up the Stackelberg POMDP environment using 50k equilibrium steps and 30 reward steps In these initial experiments, we train our policies using the centralized
execution paradigm as used for this Stackelberg learning problem by Brero al. (2022), , giving the critic network access the sellers' learning information (i.e., Q-tables and exploration rates). We relax this below in studying the robustness of the computational framework to online training ("in the wild"). We consider the following interventions on behalf of the platform designer:
No intervention: Sellers are always displayed, no matter the price they quote. To derive this baseline, we run our Q learning dynamics until convergence (as described in Johnson et al. 202 for each seed and then average the surplus at final strategies.
PDP: We test price-directed prominence, a platform intervention introduced by Johnson et al. Here, the platform only displays the seller who quotes the lower price (breaking ties at random), thus enhancing competition. As for no intervention, we compute the performance of PDP by averaging consumer surplus after Q-learning dynamics converge.
2021
DPDP: Dynamic directed prominence is another intervention introduced by Dohnson et al. (02T), which also conditions the choice of the (unique) displayed seller on past prices. Under this intervention, quoting prices equal to cost is a subgame perfect equilibrium of the induced game (under suitable discount factors). As for the previous baselines, we compute the performance of DPDP by averaging consumer surplus after Q-learning dynamics converge.
SSee Appendix for discussion around parameter selection.

---

1.0
0.8
Consumer Surplus
0.6
0.6
Intervention
No State RL
State based RL
PDP
DPDP
No intervention
0.2
Training Steps
Training Steps
Figure 1: Learning performance of No State RL and State-based RL compared with different baselines. The results are averaged over 50 runs and shaded regions show 95% conf. intervals. The No-Stackelberg interventions are displayed on the left, the Stackelberg ones are on the right.
No State RL: Here we use the Stackelberg POMDP methodology to train platform policy that does not use prices pt determine the threshold which admit each seller the buy box (thus, "no state") Here, Q-learning is restarted whenever Stackelberg POMDP episode begins.
No Stackelberg No State RI.: A variation "No State RL"" that does not the Stackelberg POMDP methodology. Rather, the platform and sellers each follow decentralized learning, and the platform receives consumer surplus reward every step. Q-learning is restarted after the same number of steps that are used in a Stackelberg POMDP episode.
State-based RL: Here sets a threshold at which to admit each seller as function of the price profile quoted by the sellers (thus, state-basedd") This is the full class of threshold platform rules. Here, Q-learning is restarted whenever a Stackelberg POMDP episodes begins.
the Stackelberg POMDP methodology to train platform policy that
No Stackelberg State-based RL: A variation on "State-based RI." that does not use the Stackelberg POMDP methodology. Rather, the platform and sellers each follow decentralized learning, and the platform receives a consumer surplus reward at every step. Q-learning is restarted after the same number steps that are used in a Stackelberg POMDP episode.
Figure shows the consumer surplus that is realized under these different interventions. First, we confirm the results of Johnson et al. (202T), and see consumer surplus improvements from both PDP and DPDP compared to No intervention, with DPDP outperforming PDP. At the same time, the no Stackelberg baselines are not able to outperform DPDP, confirming the benefits of using learning methodologies that exploit the leader-follower structure of our game. Indeed, our RL. interventions based on the Stackelberg framework dramatically improve consumer surplus, driving to (approximately. in the state- state-based scenario) its maximal level. In our setting, this optimal level for surplus approximately 0.94. This is confirmed by the fact that, for both No State and State-based RL, all sellers are displayed and they invariably quote minimum prices the end of training. This is the optimal (i.e., surplus maximizing) seller behavior, confirming the effectiveness the Stackelberg- based learning methodology in finding optimal leader strategy given the Q-learning behavior of sellers. It is easier for No State RL to reach the optimal performance since its class of policies is much smaller than the class considered by -tased RL. However, the mrteebbsed policy is more flexible and is robust to the case that the cost basis changes for sellers while No State RL is not.
will see in Section 5.3
This class of policies already includes the optimal policy described in the proof of Proposition
7

---

1.0
1.0
0.8
Consumer Surphus
Consumer Surplus
No State RL
State state-based RL.
0.
0.
DPOP
No Intervention
02
Training Steps
Training Steps
Figure 2: Offline learning (left) vs. online, "learning in the wild" (right) performance. The results are averaged over 50 runs, and the shaded regions show 95% conf. intervals.
5.2 Learning in the Wild
We now test the performance of the Stackelberg POMDP learning methodology when it has no access sellers' private information during training. This can potentially create learning instabilities given that actor-critic training such as A2C generally require that the environment accessed by their critic networks Markovian (Grondman et al. 2012). Despite this, we find success in this test of "in the wild" learning. The results are displayed in Figure and show, despite relaxing this Markov assumption, that the A2C algorithm able to learn optimal policies for both policy classes (No State and State-based), We conjecture that the reason behind this good performance is related to the class of threshold platform policies. Given a threshold policy, it possible to predict the overall episode reward based only on the action taken by the policy (the threshold) and ignoring the part of the state that is internal to the sellers (i.e., the Q-matrix and exploration rate).
In Appendix G we also demonstrate successful experimental results when we replace the use of consumer surplus (I) for reward with reward that corresponds the number of agents displayed and the sum of the negated prices offered by sellers. This shows robustness to possible knowledge gap in knowing the specific functional form of consumer surplus.
5.3 Robustness of Learned Platform Rules
As observed in our previous experiments, the Stackelberg- based RL algorithm is effective in learning interventions that maximize consumer surplus for given economic setting. However, they tailored to the economic setting at hand, these interventions can perform poorly when facing settings that different from those during training. To learn more robust platform rules, also train with a modified version the Stackelberg POMDP: at each reward step, with some random-pres -prcc probability, sellers quote prices sampled uniformly random from the price grid. In this way, the platform rewarded during training for performance that remains robust to prices that are not produced by the Q-learning equilibrium dynamics (given seller costs training).
We evaluate the effect of adding this perturbation-basee settings with different seller costs: in addition to the default = 1.0, also test with cost 1.38 (between the second and the third price in the grid of prices between 0.95 and 2.1) and cost c 1.67 (between the third and the fourth price in the price grid).
robustness to the training procedure in
As we see in Figure 3 (right), this training approach (and in particular with probability 0.4 of random- price perturbation) succeeds in making the state-based policy much more robust in the face of sellers who experience a different cost environment at test time. The robust, state-based policy displays sellers with higher prices (due to their higher costs), while continuing to significantly mitigate collusion when seller costs are as they were during training. This is also confirmed by the policy visualizations in Figure (left), which show how the buy box learned for State- based RL tends to be

---

State-based
2008
State-based RL trained w/ rpp=0.4
2.0
A
B
No state RL trained w/ rpp-0.4
1.81
1-11
1.53
1.33
100
0.6
4.
Price
Price
C
1.81
0.2
1.53
1.53
1.0
1.39
1.67
Marginal Cost
Price
Figure 3: Left: Policy visualization, with number of displayed agents given price selection, averaged over 50 seeds (white-ave avg. num. sellers displayed 2, black-avg. num. sellers displayed 0). A: No State RL with price perturbation during training. B: No State RL with 40% random price perturbation (rpp) during training. C: State-based RL with no price perturbation during training. D: State- based RL with 40% random price perturbation during training. Right: Robustness test, with buy box policy trained without price perturbation and with price perturbation with prob. 0.4, averaged over 50 runs.
much more open under this modified training regime. In contrast, the policy learned by No State RL performs very poorly when tested at costs that differ from those assumed during training, and even under this modified training regime. There no single threshold that provides good compromise between performance at cost 1 and handling price perturbations.
6 Conclusion
This work has demonstrated that rules that are effective in preventing collusion by sellers can be learned through a framework that correctly solves the two-level, Stackelberg problem (making use of the platform's commitment power). Specifically, we have introduced the class of threshold policies that contain policies that optimize consumer surplus and a learning methodology that is effective in learning optimal leader policies in this class. The interventions outperform the hand-designed interventions introduced in prior work when the cost environment at test time anticipated during training. We also showed how our learned platform interventions can be made more robust when settings are dynamic, with varying seller cost structures, by adopting suitably-modifte training methodology. This also highlights the importance of the state-based platform rule relative to no-state rule.
learned shown substantially
Interesting future directions include testing approach in complex settings, e.g., when sellers' costs vary between training episodes. this case, optimal policy actions are based on the prices quoted during the sellers' equilibration phase, these prices may provide useful information about the current underlying costs (intuitively, the quoted prices will be higher under higher costs). In this scenario, may be necessary represent platform policies via recurrent neural networks, keeping a memory of past prices. Finally, we believe that this approach can also be effective in other applications, e.g., to design and understand effective interventions for the electricity markets studied by AAada and Lambin (2022). setting where the successful use of RL as defensive response by platform not yet established.
Acknowledgements
This research funded part by Defense Advanced Research Projects Agency under Cooperative Agreement HR001 100111990000 11920029. The content of the information does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred. This approved for public release; distribution is unlimited. The work of G. Brero was also supported by the SNSF

---

(Swiss National Science Foundation) under Fellowship P2ZHP1 P2ZHPI_19125; and Justin Johnson for their availability answer questions about their work and for guidance in replicating some of their results. We also thank Alon Eden, Matthias Gerstgrasser, and Alexander MacKay for for helpful discussions and feedback, Many computations in this paper were run on the FASRC Cannon cluster supported by the FAS Division of Science Research Computing Group at Harvard University.
191253. We thank Emilio Calvano
References
Ibrahim Abada and Xavier Lambin. 2022. Artificial Intelligence: Can Seemingly Collusive Outcomes Be Avoided? Available at SSRN 3559308 (2022).
Simon P Anderson and Andre De Palma. 1992. The logit as model of product differentiation. Oxford Economic Papers 44, (1992), 51-67.
Enrique Areyan Viqueira, Cyrus Cousins, Yasser Mohammad, and Amy Greenwald. 2019. Empir- ical Mechanism Design: Designing Mechanisms from Data. In Proceedings of the Thirty- Conference Uncertainty in Artificial Intelligence (Proceedings of Machine Learning Research, Vol. 115). 1094-1104.
Stephanie Assad, Robert Clark, Daniel Ershov, and Lei Xu. 2020. Algorithmic Pricing and Competi- tion: Empirical Evidence from the German Retail Gasoline Market. CESifo Working Paper Series 8521 (2020).
Craig Boutilier. 1996. Planning, Learning and Coordination in Multiagent Decision Processes. In Proceedings of the 6th Conference on Theoretical Aspects of Rationality and Knowledge. 195-210.
Gianluca Brero, Alon Eden, Darshan Chakrabarti, Matthias Gerstgrasser, Vincent Li, and David C. Parkes. 2022. Learning Stackelberg Equilibria and Applications to Economic Design Games. arXiv preprint arXiv:2210.0388 (2022).
Gianluca Brero, Alon Eden, Matthias Gerstgrasser, David C. Parkes, and Duncan Rheingans- Yoo. 2021. Reinforcement Learning of Sequential Price Mechanisms. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI. 5219-5227.
Erik Brinkman and Michael P. Wellman. 2017. Empirical Mechanism Design for Optimizing Clearing Interval Frequent Call Markets. In Proceedings of the 2017 ACM Conference on Economics and Computation. 205-221.
Benedikt Bünz, Benjamin Lubin, and Sven Seuken. 2018. Designing Core-selecting Payment Rules: A Computational Search Approach. In Proceedings of the 2018 ACM Conference on Economics and Computation. 109.
Emilio Calvano, Giacomo Calzolari, Vincenzo Denicolò, Joseph E Harrington, and Sergio Pastorello. 2020b. Protecting consumers from collusive prices due to AL. Science 370, 6520 (2020), 1040- 1042.
Emilio Calvano, Giacomo Calzolari, Vincenzo Denicolo, and Sergio Pastorello. 2020a. Artificial intelligence, algorithmic pricing, and collusion. American Economic Review 110, 10 (2020), 3267-97.
C. Cheng, Z. Zhu, B. Xin, and C Chen. 2017. A multi-agent reinforcement learning algorithm based on Stackelberg game. In 6th Data Driven Control and Learning Systems (DDCLS). 727-732.
Michael J. Curry, Ping- Yeh Chiang, Tom Goldstein, and John Dickerson. 2020. Certifying Strate- gyproof Auction Networks. In Proc. 33rd Annual Conference on Neural Information Processing Systems.
Michael J. Curry, Uro Lyi, Tom Goldstein, and John Dickerson. 2022. Learning Revenue- Maximizinxnng Auctions With Differentiable Matching. Proc. 25th International Conference on Artificial Intelligence and Statistics. Forthcoming.
10

---

Paul Ductting, Zhe Feng, Harikrishna Narasimhan, David C. Parkes, and Sai Srivatsa Ravindranath. 2019. Optimal Auctions through Deep Learning. In Proceedings of the 36th International Confer- ence on Machine Learning. 1706-1715.
Tanner Fiez, Benjamin Chasnov, and Lillian J. Ratifff. 2020. Implicit Learning Dynamies in Stackel- berg Games: Equilibria Characterization, Convergence Analysis, and Empirical Study, In Proceed- ings of the 37th International Conference on Machine Learning. 3133-3144.
Drew Fudenberg and Jean Tirole. 1991. Game theory. MIT press.
Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, and Shixiang Shane Gu. 2022. Why Should Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error. arXiv preprint arXiv:2201.12411 (2022).
Ivo Grondman, Lucian Busoniu, Gabriel AD Lopes, and Robert Babuska. 2012. survey of actor- critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 42, 6 (2012), 1291-1307.
Chi Jin, Praneeth Netrapalli, and Michael I. Jordan. 2020. What is Local Optimality in Nonconvex- Nonconcave Minimax Optimization?. In Proceedings of the 37th International Conference on Machine Learning, Vol. 119. 4880-4889.
Justin Johnson, Andrew Rhodes, and Matthij R Wildenbeest. 2021. Platform design when sellers use pricing algorithms. Available at SSRN 3753903 (2021).
Patrick R. Jordan, L. Julian Schvartzman, and Michael P. Wellman. 2010. Strategy exploration in empirical games. In 9th International Conference on Autonomous Agents and Multiagent Systems. 1131-1138.
Christopher Kiekintveld and Michael P. Wellman. 2008 Selecting strategies using empirical game models: an experimental analysis of meta-strategies. In 7th International Joint Conference on Autonomous Agents and Multiagent Systems. 1095-1101.
Kevin Kuo, Anthony Ostuni, Elizabeth Horishny, Michael J. Curry, Samuel Dooley, Ping- Yeh Chiang, Tom Goldstein, and John P. Dickerson. 2020. ProportionNet: Balancing Fairness and Revenue for Auction Design with Deep Learning. CoRR abs/2010.06398 (2020).
Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. 2019. Robust Multi- Agent Reinforcement Learning via Minimax Deep Deterministie Policy Gradient. Proceedings of the AAAI Conference on Artificial Intelligence 33 (July 2019), 4220-
Ryan Lowe, Yi Wu, Aviy Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed ccoperative-competiivee environments. Advances in Neural Information Processing Systems (NeurIPS) 30 (2017).
Andreu Mas-Colell, Michael Dennis Whinston, Jerry R Green, et al. 1995. Microeconomic theory. Vol. 1. Oxford university press New York.
David Mguni, Joel Jennings, Emilio Sison, Sergio Valcarcel Macua, Sofia Ceppi, and Enrique Munoz de Cote. 2019. Coordinating the Crowd: Inducing Desirable Equilibria in Non-Cooperative Sys- tems. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems. 386-394.
Rajesh K. Mishra, Deepanshu Vasal, and Sriram Vishwanath. 2020. Model-free Reinforcement Learning for Stochastic Stackelberg Security Games. In 59th IEEE Conference on Decision and Control. 348-353.
Nechar Peri. Michael J. Curry, Samuel Dooley, and John P. Dickerson. 2021. PreferenceNet: Encoding Human Preferences in Auction Design with Deep Learning. In Proceedings of the 35th Conference on Neural Information Processing Systems.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Reliable Reinforcement Learning Implementations. Journal
Dormann. 2021. of Machine Learning Research 22, 268 (2021), 1-8.

---

Jad Rahme, Samy Jelassi, Joan Bruna, and S. Matthew Weinberg, 2021b. A Permutation Equivarrant Neural Network Architecture For Auction Design. In Proc. Thirty-Fifhe AAAI Conference on Artificial Intelligence.
Jad Rahme, Samy Jelassi, and S. Matthew Weinberg. 2021a. Auction Learning Two-Player Game. In Proc. 9th International Conference on Learning Representations, ICLR.
Sailik Sengupta and Subbarao Kambhampati. 2020. Multi-agent Reinforcement Learning in Bayesian Stackelberg Markov Games for Adaptive Moving Target Defense. arXiv:2007.10457 [es] (July 2020). arxiv: 2007.10457.
Weiran Shen, Binghui Peng, Hanpeng Liu, Michael Zhang. Ruohan Qian, Yan Hong, Zhi Guo, Zongyao Ding, Pengjun Lu, and Pingzhong Tang. 2020. Reinforcement mechanism design, with applications to dynamic pricing in sponsored search auctions. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 2236-223 2243.
W. Shen, P. Tang, and S. Zuo. 2019. Automated Mechanism Design via Neural Networks. In Proceedings of the 18th International Conference on Automomous Agents and Multiagent Systems.
Zhenyu Shi, Runsheng Yu, Xinrun Wang, Rundong Wang, Youzhi Zhang. Hanjiang Lai, and Bo An. 2020. Learning Expensive Coordination: An Event-Based Deep RL Approach. In 8th International Conference on Learning Representations.
Tianmin Shu and Yuandong Tian. 2019. M Mind-aware Multi-agent Management Reinforce- ment Learning. In 7th International Conference Learning Representations.
Andrea Tacchetti, DJ Strouse, Marta Garnelo, Thore Graepel, and Yoram Bachrach. 2019. A Neural Architecture for Designing Truthful and Efficient Auctions. CoRR 1907.05181 (2019).
Pingzhong Tang. 2017. Reinforcement mechanism design.. In Proc. 26th Int Joint Conf. Art. Intell. (JCAI).. 5146 5150.
K. Tharakunnel and S. Bhattacharyya. 2007.
Leader-follower semi-markov decision problems: theoretical framework and approximate solution. In IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning. 111-118.
The Organisation for Economic Co-operation and Development. 2017. Algorithms and Collusion- Note from the European Union. www. oecd. wttpg//ommeiitio/aggeeeeeeeeoeee wtmms-ttttt-e aneiims--nn---e and-collusion
htm
U.S. Federal Trade Commission. 2018 The Competition and Consumer Protection Issues of Algo- rithms, Artificial Intelligence, and Predictive Analytics, Hearing on Competition and Consumer Protection in the 21st Century, U.S. Federal Trade Commission. https://www. ftc. gov/ cowpeetiiiiiiiiccooooooeeeeeeeep protection
Yevgeniy Vorobeychik, Christopher Kiekintveld, and Michael P. Wellman. 2006. Empirical mech- anism design: methods, with application to a supply-chain scenario. In Proceedings 7th ACM Conference Electronic Commerce (EC-2006). 306-315.
Kai Wang, Lily Xu, Andrew Perrault, Michael K. Reiter, and Milind Tambe. 2022. Coordinating Followers to Reach Better Equilibria: End-to-End Gradient Descent for Stackelberg Games. In AAAI Conference on Artificial Intelligence.
Michael P. Wellman. 2006. Methods for Empirical Game-Theoretic Analysis. Proceedings, The Twenty-First National Conference on Artificial Intelligence. 1552-1556.
Lily Xu, Andrew Perrault, Fei Fang, Haipeng Chen, and Milind Tambe. 2021. Robust Reinforcement Learning Under Minimax Regret for Green Security. In Proc. 37th Conference on Uncertainty in Artifical Intelligence (UAI-21).
Stephan Zheng, Alexander Trott, Sunil Srinivasa. David Parkes, and Richard Socher. 2022. The AI Economist: Optimal Economic Policy Design via Two-level Deep Reinforcement Learning. Science Advances 8 (2022).
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A]. You are strongly encouraged include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section
S
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See conclusion.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] The paper itself is focused on tackling a potential negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Proofs are provided in the appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material as URL)? [Yes] We will include
it in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] But not in Figure 3 to reduce clutter.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the beginning of Section
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We used Stable Baselines3, which we cite at the beginning of Section
(b) Did you mention the license of the assets? [Yes] We mentioned the Stable Baselines3 at the beginning of Section 5
(c) Did you include any new assets either the supplemental material or as URL.? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you' 're
using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "YyIaGcHMSOd", "reviewer": "Reviewer_7ViN", "summary": "This is an emergency review of the paper. \n\nA recent topic on the border of ML and economics is that of reinforcement learning approaches that learn \"tacit collusion\" in the sense that decentralized independent self-play based on these algorithms in various auction settings reaches non-competitive (collusive) prices. \n\nThe authors take a stab at tackling this problem by designing a mechanisms for ecommerce sites that promotes produces and prices to consumers so as to prevent such tacit RL collusion (so-called applying a suitable \"buy-box\"). The authors show that certain such buy-box rules indeed manage to avoid such tacit collution, at least for various types of RL methods. The methodology is based on a leader-follower model (Stackelberg) on a reasonable POMDP environment. Hence, the authors show that such rules manage to keep consumer welfare high. ", "questions": "See above - can you consider additional RL algorithms (there are lots of libraries you can just launch on this environment, I think). Also, any thoughts on how the welfare is distributed among the sellers? ", "limitations": "The authors acknowledge the limitations here, but I think a more detailed discussion of various RL algorithms is warranted. ", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: I really love the topic of the paper. Tacit collusion is indeed an important problem in the age of algorithmic trading, and due to the immense progress in RL algorithms (especially multi-agent deep RL), it makes sense to see whether bad behaviours can be prevented. This topic is very recent, and should, in my opinion get more attention. \n\nI find the model reasonable, and the suggested mechanism simple yet effective. \n\nThe weakeness of the paper lies in its presentation. First, I think the results may be different depending on the RL approach used. You consider both Q-based approaches and policy gradient / actor critic approaches. However, I would have liked a more detailed experimental analysis, considering multiple algorithms, from vanilla reinforce, SARSA (tabular and using a neural function approximator) as well as DQN. You also examine A2C, but I think it makes sense to also examine other algorithms (V)MPO and PPO, for example. Are the results sensitive to choices of hyperparameters (e.g. learning rates, neural architecture, optimizer)? Please be clearer about where your theoretical results hold (e.g. in the form of a table, early in the paper), and where you have empirical analysis supporting the claims. If the results are sensitive to the type of RL applied, any inisght as to the properties of the RL algorithm driving this would be welcome. \n\nFurther, you discuss results from the field of automated mechanism design, but much of this work deals with mechanism families of provable guarantees (e.g. VCG and variants), and the setting here is different. Could you make the comparison a bit more detailed? \nThere are lot of resutls on collusion for vcg mechanisms (including based on cooperative game theory - e.g. the core and Shapley value). Can you say (or conjecture) anything along these lines here? In the case of this tacit collusion, the consumer loses welfare, but which of the sellers makes most of the gains? \n\nAll in all, this is a very interesting topic, and I like the approach presented. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "V_dLz8yGJ_l", "reviewer": "Reviewer_DPgu", "summary": "This paper uses reinforcement learning (RL) by a platform to learn buy box rules that are effective in preventing collusion by RL sellers. The authors make use of the approach of Stackelberg POMDPs.", "questions": "Have the authors performed additional simulations? Do different simulations provide further insights or further demonstrate competitiveness of your proposed approach?", "limitations": "Not applicable", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the paper are that the topic is an interesting and important one, and the proposed approach is competitive in reducing collusion on platforms under the studied setting.\n\nThe main weakness of the paper is that the numerical simulations, including the ones in the Appendix, are not very comprehensive. It would have been interesting to see numerical simulations in more different settings than the manuscript had. An additional weakness is that the paper does not provide strong or interesting theoretical characterizations in the applicable settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LvdQm-gbH3d", "reviewer": "Reviewer_VP8z", "summary": "This paper aims to prevent collusive pricing by sellers through a learned framework that solves the two-level, Stackelberg problem. Specifically, the paper introduces the class of threshold policies that optimize consumer surplus and a method to learn optimal leader policies in this class. Then it is experimentally shown that the learned interventions outperform the hand-designed interventions proposed in earlier work. ", "questions": "- I think maybe I have missed something, but it isn't super clear to me what do you mean by \"collusion\" or \"collusive pricing\" formally. Empirically, it seems the goal for the proposed methods is to maximize consumer surplus. Are these two objectives interchangeable? If not, please elaborate; if yes, then maybe the writing can be revised a bit. \n\n- The title \"Learning to Mitigate AI Collusion on Economic Platforms\" sounds more general than the text actually addresses, which is limited to price collusion on e-commerce sites. \n\n- The key contribution of this paper is using RL, for the first time, to prevent collusive pricing. Therefore, there is no prior RL work to compare to, but hand-crafted rules proposed in earlier work. Looking at Figure 2, the RL approaches start to outperform only after 1e7 training steps. This corresponds to a huge number of transactions, during which the seller's cost is assumed constant. How realistic is it? Shouldn't the cost be modeled rather than a constant? (For example to reflect the economy of scale. This point is also raised in the conclusion.)", "limitations": "N.A.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- As algorithmic pricing may indeed facilitate collusive behaviors (though sometimes unintentionally), this is a super interesting research direction, especially considering the rising popularity of algorithmic pricing in the e-commence space. \n\n- To cast this problem as multi-agent learning and model it as Stackelberg game is novel and interesting. \n\nWeaknesses:\n\n- This work is, to a great extent, a thought experiment with limited support in a real-world setting. Learning in an e-commence environment often involves noisy training data, unexpected user behaviors. Without real-world datasets/applications, it is hard to tell if the setups outlined in Section 5 are reasonable and useful. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JGLNjfVdegW", "reviewer": "Reviewer_aSeB", "summary": "This paper proposes a reinforcement learning based method to prevent the collusive pricing, \nwhich is important to the development of AI based economic platforms.\n\nSpecifically, they model the economic platforms as a Stackelberg where the leader is the platform and the followers are the sellers.\nThe leader uses the AC to train while the follower is Q learning based.\n\nThe authors provide some interesting theoretical results (Although some lacks important steps).\n\nThe experiments show that the proposed method is well-performing comparing with some baselines.", "questions": "Please answer the questions mentioned above.", "limitations": "It would be a plus if the authors could provide more details on 1. mentioned in 'Strengths And Weaknesses' part. \n\n\nSince this paper focuses on economic system, the potential negative societal impact lies in how the learned platforms are stable enough. The authors propose a robust analysis, which address my concerns.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe proposed method is robust and the ablations are detailed.\n\n\nWeaknesses:\n1. This paper is limited in solving a special economic platform. \nThat is, the paper (including both the theoretical and experimental studies) is mainly focusing on \nthose systems that can be represented in lines 88-100, and it is doubtful that whether this type of economic platform is general enough.\n\nCould the authors discuss more details about whether this economic platform is widely adopted and used?\n\n2. Novelty is somewhat minor, the used RL methods have already been introduced in Mguni et al. (2019); Shi et al.\n(2020) and Shu and Tian (2019). \nThe proposed threshold platform rule is new to me, but it only fits a special economic platform.\n\n3. Regarding the empirical studies, the baselines seem not strong enough. The authors discuss many related RL methods to solve Stackelberg game but does not compare them in the experiments. Is it possible to compare some of them?\n\nMinors:\n\n1). The notations are somewhat ambiguous. The authors firstly define $p$ as generic price profile, but then reload it as probability, which incurs some ambiguity. It would be better to change the notation of the generic price profile to another.\n\n2). some important definitions are missing, e.g., Nash equilibrium. It would be better to define them.\n\n3). Some of the proofs are lacking important steps, e.g., in lines 592-596 (Prop. 2), why $\\rho(p;\\mathcal{N})$ is strictly increasing implies Nash equilibrium. To my best knowledge, Nash equilibrium is a set of mixed strategies from which no player has a unilateral\nincentive to deviate [1]. It would be nice to add more details.\n\n[1] Daskalakis, Constantinos, Paul W. Goldberg, and Christos H. Papadimitriou. \"The complexity of computing a Nash equilibrium.\" SIAM Journal on Computing 39.1 (2009): 195-259.\n\n\n\n####post-rebuttal####\n\nThank the authors for the feedback. It helps the reviewer better understand the proposed method. \n\ni) However, the reviewer is still somewhat concerned about the limited scope of this paper, as the proposed algorithm is only designed and testified in a specific economic model. Reviewer DPgu and Reviewer VP8z also ask similar questions.\n\nAlthough the authors provide some related works [Calvano et al. (2020a), Johnson et al. (2020)],  they all belong to the economic area. For the submission to an AI-related conference, it would be nice to be more general.\n\nii) Also, regarding the Nash Eq analysis, it is recommended to add additional definition and proof directly to the revision, rather than just roughly saying the proof sketches, as this hand-waving claim is hard for the reviewer to check whether it is correct.\n\nThe reviewer is also confused about why the authors promised to add proofs and formal definitions but ignored them when they submitted the revision.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept