Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: After reading the rebuttal, I increased my score to weak accept, since it addressed my concern.
----------------------------------------
Summary
This paper presents a general machine learning method for contest / auction problems. The underlying idea is to collect data pairs (i.e., [design, utility]), fit a model to the data, and then optimize over all the designs to figure out the best one. The authors mainly applied their method on an auction design problem, and finished a few experiments. However, due to lack of novelty, I lean to vote for rejecting this paper.
Weaknesses
- My major concern of this paper is the lack of novelty. As the authors stated in the introduction, the contribution of this paper is a machine learning method for designing crowdsourcing contest. However, as the authors demonstrated in Figure 1, the main idea of this approach is: collect the data, fit a model, and finally optimize the objective, which is a pretty common approach. I do not see something special or interesting in this approach.
- The authors spend a lot of space discussing how to deal with the auction, but I do not see their relationship with the machine learning algorithm, or how can these tricks be generalizable to other scenarios. It seems all these discussions are specific to this auction scenario, and there is almost no relationship between these tricks with the machine learning algorithm. However, if these tricks can be applied to other scenarios, these discussions will make sense.
Possible Improvements
I am very happy to increase my score if the authors could demonstrate why their approach in Figure 1 is novel, and how their discussion about the auction can be generalized to other scenarios.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: 1. Summary

The authors employ a multi-agent learning approach for learning how to set payoffs optimally for crowdsourcing contests
and auctions. Optimality means e.g. incentive alignment (the principal problem) between the principal (e.g. the organizer) and participants (e.g. bidders), assuming e.g. that participants can be strategic about their behavior. In this work the principal uses ReLU-log utility.

First, the authors use fictitious play and multi-agent RL to train agents on a distribution of payoffs. Then, a neural net is fitted to the samples (payyoffs, expected principal utility), and finally iteratively attempts to improve the payoffs using mirror ascent within the convex set of admissable payoffs.

The authors compare the payoffs with theoretically known solutions and in situations where the optimal solution is not known.

3, 4-agent all-pay auction (Nash eq known).
Same as above, but with noise added to bids (Nash eq not known).
The authors analyze in some detail how the principal's utility and bidder ranking behave as the participants' bids change.

1. Decision (accept or reject) with one or two key reasons for this choice.

Reject. Although the high-level approach is interesting (use learning to design auctions for cases where no theoretical solution is known), the actual experimental results and methodological improvement over e.g. Dutting 2017 are weak. The authors only consider 3, 4-agent auctions. There are no other learned baselines (e.g., constrained optimization without neural nets) that the authors could consider.

3. Supporting arguments

See above.

4. Additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.

M-EMA and M-EMD: ascent and descent? in Algo 1, 2?


--- 
I've read the rebuttal, but still lean towards reject. The scope/analysis of the experiments (e.g. auction type), still seems limited, even though both agents and mechanism are adaptive.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper considers a scenario of bidding contest where the goal os to find optimal allocation w = (w_1, w_2, ..., w_n) of the total prize that maximizes the principal’s expected utility function. The problem is formulated into an optimization task within the simplex where the total allocation is fixed at w. Then the authors proposed simulation methods to solve this problem and use experiments to demonstrate the method's advantages. The paper is sound an clear, but it's not clear to me which part is novel and which part is from existing work, hence I doubt the contribution level of this paper. Furthermore, I'm not quite sure whether the topic fits ICLR as it's more related to game theoretic society and not related to representation learning.

Thanks for the response from the authors. I have read it carefully, especially regarding the novelty part. My review remains unchanged based on the author feedback.
review_title: Official Blind Review #1

Meta-review: This paper demonstrates a framework for optimizing designs in auction/contest problems. The approach relies on considering a multi-agent learning process and then simulating it. 

To a large degree there is agreement among reviewers that this approach is sensible and sound, however lacks substantial novelty. The authors provided a rebuttal which clarified the aspects that they consider novel, however the reviewers remained mostly unconvinced. Furthermore, it would help if the improvement over past approaches is demonstrated in a more convincing way, for example with increased scope experiments that also involve richer analysis.

Decision: Reject