Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper aims to prevent collusive pricing by sellers through a learned framework that solves the two-level, Stackelberg problem. Specifically, the paper introduces the class of threshold policies that optimize consumer surplus and a method to learn optimal leader policies in this class. Then it is experimentally shown that the learned interventions outperform the hand-designed interventions proposed in earlier work. 
strengths_and_weaknesses: Strengths:

- As algorithmic pricing may indeed facilitate collusive behaviors (though sometimes unintentionally), this is a super interesting research direction, especially considering the rising popularity of algorithmic pricing in the e-commence space. 

- To cast this problem as multi-agent learning and model it as Stackelberg game is novel and interesting. 

Weaknesses:

- This work is, to a great extent, a thought experiment with limited support in a real-world setting. Learning in an e-commence environment often involves noisy training data, unexpected user behaviors. Without real-world datasets/applications, it is hard to tell if the setups outlined in Section 5 are reasonable and useful. 


questions: - I think maybe I have missed something, but it isn't super clear to me what do you mean by "collusion" or "collusive pricing" formally. Empirically, it seems the goal for the proposed methods is to maximize consumer surplus. Are these two objectives interchangeable? If not, please elaborate; if yes, then maybe the writing can be revised a bit. 

- The title "Learning to Mitigate AI Collusion on Economic Platforms" sounds more general than the text actually addresses, which is limited to price collusion on e-commerce sites. 

- The key contribution of this paper is using RL, for the first time, to prevent collusive pricing. Therefore, there is no prior RL work to compare to, but hand-crafted rules proposed in earlier work. Looking at Figure 2, the RL approaches start to outperform only after 1e7 training steps. This corresponds to a huge number of transactions, during which the seller's cost is assumed constant. How realistic is it? Shouldn't the cost be modeled rather than a constant? (For example to reflect the economy of scale. This point is also raised in the conclusion.)
limitations: N.A.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This is an emergency review of the paper. 

A recent topic on the border of ML and economics is that of reinforcement learning approaches that learn "tacit collusion" in the sense that decentralized independent self-play based on these algorithms in various auction settings reaches non-competitive (collusive) prices. 

The authors take a stab at tackling this problem by designing a mechanisms for ecommerce sites that promotes produces and prices to consumers so as to prevent such tacit RL collusion (so-called applying a suitable "buy-box"). The authors show that certain such buy-box rules indeed manage to avoid such tacit collution, at least for various types of RL methods. The methodology is based on a leader-follower model (Stackelberg) on a reasonable POMDP environment. Hence, the authors show that such rules manage to keep consumer welfare high. 
strengths_and_weaknesses: Strength: I really love the topic of the paper. Tacit collusion is indeed an important problem in the age of algorithmic trading, and due to the immense progress in RL algorithms (especially multi-agent deep RL), it makes sense to see whether bad behaviours can be prevented. This topic is very recent, and should, in my opinion get more attention. 

I find the model reasonable, and the suggested mechanism simple yet effective. 

The weakeness of the paper lies in its presentation. First, I think the results may be different depending on the RL approach used. You consider both Q-based approaches and policy gradient / actor critic approaches. However, I would have liked a more detailed experimental analysis, considering multiple algorithms, from vanilla reinforce, SARSA (tabular and using a neural function approximator) as well as DQN. You also examine A2C, but I think it makes sense to also examine other algorithms (V)MPO and PPO, for example. Are the results sensitive to choices of hyperparameters (e.g. learning rates, neural architecture, optimizer)? Please be clearer about where your theoretical results hold (e.g. in the form of a table, early in the paper), and where you have empirical analysis supporting the claims. If the results are sensitive to the type of RL applied, any inisght as to the properties of the RL algorithm driving this would be welcome. 

Further, you discuss results from the field of automated mechanism design, but much of this work deals with mechanism families of provable guarantees (e.g. VCG and variants), and the setting here is different. Could you make the comparison a bit more detailed? 
There are lot of resutls on collusion for vcg mechanisms (including based on cooperative game theory - e.g. the core and Shapley value). Can you say (or conjecture) anything along these lines here? In the case of this tacit collusion, the consumer loses welfare, but which of the sellers makes most of the gains? 

All in all, this is a very interesting topic, and I like the approach presented. 


questions: See above - can you consider additional RL algorithms (there are lots of libraries you can just launch on this environment, I think). Also, any thoughts on how the welfare is distributed among the sellers? 
limitations: The authors acknowledge the limitations here, but I think a more detailed discussion of various RL algorithms is warranted.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a reinforcement learning based method to prevent the collusive pricing, 
which is important to the development of AI based economic platforms.

Specifically, they model the economic platforms as a Stackelberg where the leader is the platform and the followers are the sellers.
The leader uses the AC to train while the follower is Q learning based.

The authors provide some interesting theoretical results (Although some lacks important steps).

The experiments show that the proposed method is well-performing comparing with some baselines.
strengths_and_weaknesses: Strengths:
The proposed method is robust and the ablations are detailed.


Weaknesses:
1. This paper is limited in solving a special economic platform. 
That is, the paper (including both the theoretical and experimental studies) is mainly focusing on 
those systems that can be represented in lines 88-100, and it is doubtful that whether this type of economic platform is general enough.

Could the authors discuss more details about whether this economic platform is widely adopted and used?

2. Novelty is somewhat minor, the used RL methods have already been introduced in Mguni et al. (2019); Shi et al.
(2020) and Shu and Tian (2019). 
The proposed threshold platform rule is new to me, but it only fits a special economic platform.

3. Regarding the empirical studies, the baselines seem not strong enough. The authors discuss many related RL methods to solve Stackelberg game but does not compare them in the experiments. Is it possible to compare some of them?

Minors:

1). The notations are somewhat ambiguous. The authors firstly define $p$ as generic price profile, but then reload it as probability, which incurs some ambiguity. It would be better to change the notation of the generic price profile to another.

2). some important definitions are missing, e.g., Nash equilibrium. It would be better to define them.

3). Some of the proofs are lacking important steps, e.g., in lines 592-596 (Prop. 2), why $\rho(p;\mathcal{N})$ is strictly increasing implies Nash equilibrium. To my best knowledge, Nash equilibrium is a set of mixed strategies from which no player has a unilateral
incentive to deviate [1]. It would be nice to add more details.

[1] Daskalakis, Constantinos, Paul W. Goldberg, and Christos H. Papadimitriou. "The complexity of computing a Nash equilibrium." SIAM Journal on Computing 39.1 (2009): 195-259.



####post-rebuttal####

Thank the authors for the feedback. It helps the reviewer better understand the proposed method. 

i) However, the reviewer is still somewhat concerned about the limited scope of this paper, as the proposed algorithm is only designed and testified in a specific economic model. Reviewer DPgu and Reviewer VP8z also ask similar questions.

Although the authors provide some related works [Calvano et al. (2020a), Johnson et al. (2020)],  they all belong to the economic area. For the submission to an AI-related conference, it would be nice to be more general.

ii) Also, regarding the Nash Eq analysis, it is recommended to add additional definition and proof directly to the revision, rather than just roughly saying the proof sketches, as this hand-waving claim is hard for the reviewer to check whether it is correct.

The reviewer is also confused about why the authors promised to add proofs and formal definitions but ignored them when they submitted the revision.
questions: Please answer the questions mentioned above.
limitations: It would be a plus if the authors could provide more details on 1. mentioned in 'Strengths And Weaknesses' part. 


Since this paper focuses on economic system, the potential negative societal impact lies in how the learned platforms are stable enough. The authors propose a robust analysis, which address my concerns.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper uses reinforcement learning (RL) by a platform to learn buy box rules that are effective in preventing collusion by RL sellers. The authors make use of the approach of Stackelberg POMDPs.
strengths_and_weaknesses: The strengths of the paper are that the topic is an interesting and important one, and the proposed approach is competitive in reducing collusion on platforms under the studied setting.

The main weakness of the paper is that the numerical simulations, including the ones in the Appendix, are not very comprehensive. It would have been interesting to see numerical simulations in more different settings than the manuscript had. An additional weakness is that the paper does not provide strong or interesting theoretical characterizations in the applicable settings.
questions: Have the authors performed additional simulations? Do different simulations provide further insights or further demonstrate competitiveness of your proposed approach?
limitations: Not applicable
review_title: null

Meta-review: metareview: This paper proposes an RL based method to prevent collusive pricing by sellers through a framework that solves the two-level, Stackelberg problem.

The paper received a mixed evaluation from the reviewers, ranging from accept (7) to weak reject (4).

The strengths of the paper mentioned by the reviewers were: 
- The considered problem was appreciated and acknowledged to be an important research direction
- Novelty in casting the problem as a Stackelberg game
- The proposed approach is competitive and robust 

On the other hand, the identified weaknesses were:
- Potentially limited support in a real-world setting and a limited scope (only designed and testified in a specific economic model)
- Problems with the presentation

Despite the weaknesses mentioned above, I lean toward acceptance with my recommendation.
confidence: Less certain
Decision: Accept