Neural Design of Contests and All-Pay Auctions using Multi-Agent Simulation

Abstract: We propose a multi-agent learning approach for designing crowdsourcing contests and all-pay auctions. Prizes in contests incentivise contestants to expend effort on their entries, with different prize allocations resulting in different incentives and bidding behaviors. In contrast to auctions designed manually by economists, our method searches the possible design space using a simulation of the multi-agent learning process, and can thus handle settings where a game-theoretic equilibrium analysis is not tractable. Our method simulates agent learning in contests and evaluates the utility of the resulting outcome for the auctioneer. Given a large contest design space, we assess through simulation many possible contest designs within the space, and fit a neural network to predict outcomes for previously untested contest designs. Finally, we apply mirror descent to optimize the design so as to achieve more desirable outcomes. Our empirical analysis shows our approach closely matches the optimal outcomes in settings where the equilibrium is known, and can produce high quality designs in settings where the equilibrium strategies are not solvable analytically. 

Reviews and author responses:

After reading the rebuttal, I increased my score to weak accept, since it addressed my concern.
----------------------------------------
Summary
This paper presents a general machine learning method for contest / auction problems. The underlying idea is to collect data pairs (i.e., [design, utility]), fit a model to the data, and then optimize over all the designs to figure out the best one. The authors mainly applied their method on an auction design problem, and finished a few experiments. However, due to lack of novelty, I lean to vote for rejecting this paper.
Weaknesses
- My major concern of this paper is the lack of novelty. As the authors stated in the introduction, the contribution of this paper is a machine learning method for designing crowdsourcing contest. However, as the authors demonstrated in Figure 1, the main idea of this approach is: collect the data, fit a model, and finally optimize the objective, which is a pretty common approach. I do not see something special or interesting in this approach.
- The authors spend a lot of space discussing how to deal with the auction, but I do not see their relationship with the machine learning algorithm, or how can these tricks be generalizable to other scenarios. It seems all these discussions are specific to this auction scenario, and there is almost no relationship between these tricks with the machine learning algorithm. However, if these tricks can be applied to other scenarios, these discussions will make sense.
Possible Improvements
I am very happy to increase my score if the authors could demonstrate why their approach in Figure 1 is novel, and how their discussion about the auction can be generalized to other scenarios.

1. Summary

The authors employ a multi-agent learning approach for learning how to set payoffs optimally for crowdsourcing contests
and auctions. Optimality means e.g. incentive alignment (the principal problem) between the principal (e.g. the organizer) and participants (e.g. bidders), assuming e.g. that participants can be strategic about their behavior. In this work the principal uses ReLU-log utility.

First, the authors use fictitious play and multi-agent RL to train agents on a distribution of payoffs. Then, a neural net is fitted to the samples (payyoffs, expected principal utility), and finally iteratively attempts to improve the payoffs using mirror ascent within the convex set of admissable payoffs.

The authors compare the payoffs with theoretically known solutions and in situations where the optimal solution is not known.

3, 4-agent all-pay auction (Nash eq known).
Same as above, but with noise added to bids (Nash eq not known).
The authors analyze in some detail how the principal's utility and bidder ranking behave as the participants' bids change.

1. Decision (accept or reject) with one or two key reasons for this choice.

Reject. Although the high-level approach is interesting (use learning to design auctions for cases where no theoretical solution is known), the actual experimental results and methodological improvement over e.g. Dutting 2017 are weak. The authors only consider 3, 4-agent auctions. There are no other learned baselines (e.g., constrained optimization without neural nets) that the authors could consider.

3. Supporting arguments

See above.

4. Additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.

M-EMA and M-EMD: ascent and descent? in Algo 1, 2?


--- 
I've read the rebuttal, but still lean towards reject. The scope/analysis of the experiments (e.g. auction type), still seems limited, even though both agents and mechanism are adaptive.

This paper considers a scenario of bidding contest where the goal os to find optimal allocation w = (w_1, w_2, ..., w_n) of the total prize that maximizes the principal’s expected utility function. The problem is formulated into an optimization task within the simplex where the total allocation is fixed at w. Then the authors proposed simulation methods to solve this problem and use experiments to demonstrate the method's advantages. The paper is sound an clear, but it's not clear to me which part is novel and which part is from existing work, hence I doubt the contribution level of this paper. Furthermore, I'm not quite sure whether the topic fits ICLR as it's more related to game theoretic society and not related to representation learning.

Thanks for the response from the authors. I have read it carefully, especially regarding the novelty part. My review remains unchanged based on the author feedback.

Thank you very much for taking the time to review our paper. Your review calls out two main areas of improvement: 1) the need to better highlight what in our approach is novel, and 2) better explaining how our approach and the topic of mechanism design fits into the conference.

Thank you for identifying these. We feel like we can address them and that our paper will be better for it.

The main concern you have identified is the “lack of novelty”, a criticism shared by R3 as well. We now realize we should have done a better job calling out which aspects of our work are novel, and which are just novel applications of existing methods. We will update our manuscript shortly to make sure the distinction is clear; please see our detailed response to R1 to find a list of contributions which we hope you will find helpful.

A key component of this paper is learning a representation of different mechanism designs. To do this, we need to simulate agent learning. This is novel for the automated mechanism design literature, where analytic solutions have previously been used, and is also an interesting perspective for the wider representation learning community: in our setting data cannot be taken for granted, and instead depends on modelling assumptions.

We believe Game Theory is highly relevant to ICLR, indeed several works on and related to GT have been presented at ICLR. For example, Stable Opponent Shaping in Differentiable Games by Letcher et al ‘19) discussed RL from a Game Theoretic perspective.

Game Theory is of increasing importance to the ICLR community: constructing multi-agent games for agents to play in training has proven an important tool (Large-Scale Study of Curiosity-Driven Learning, Burda et al. ICLR ‘19, Large Scale GAN Training for High Fidelity Natural Image Synthesis, Brock et al. ICLR ‘19). Designing these systems involves choosing the incentive structure of the agents, and so is itself a Mechanism Design problem.

We feel this rebuttal addresses your two concerns in detail: 1) lack of novelty and 2) relevance of the game theoretic topic to ICLR. We would appreciate it if you would please consider raising your score.

Thank you very much for reviewing our paper and for your feedback. Also, thank you for raising your concerns regarding a comparison of our work to Dutting et. al. We mention this work (among others) at the end of the paper, however, maybe it deserves a more pointed comparison.

Your review calls out the marginal improvement relative to previous work in two main ways: 1) the methodology and 2) the experimental results.

Thank you for identifying these concerns, we will comment on the methodology and experimental results both relative to Dutting et. al. specifically and in general.

Regarding methodology, our main contribution is to see both sides of a mechanism (both the auctioneer and the bidders) as adaptive. Dutting et. al. on the other hand uses a dominant strategy incentive compatible argument and models the bidders as truth-tellers -- the bidders are static, i.e., not adaptive learning agents. To our knowledge, designing mechanisms based on the behavior of adaptive learning agents is novel and more general than previous approaches in the literature. We did not emphasize this distinction as much as we should in the original submission. We appreciate you bringing this to our attention.
As you pointed out, we consider auctions without any known solution. In contrast, previous approaches including Dutting et. al. focus on specific classes of mechanisms where optimal behavior is already known. Therefore, our approach enables the design of mechanisms for a much wider class of games for which computing equilibrium strategies is intractable. For example, note that our approach places no restriction on the number of bidders in the auction (as opposed to theoretical analyses). Again, this is made possible by bringing adaptive learning agents into the pipeline.

Regarding experimental results, your review states that we only considered 3 and 4 bidder auctions, however, we point out that we also presented results for a noisy 10 bidder auction in Figure 6 and Table 1. We will restructure the paper to avoid confusion and make the noisy 10 bidder auction stand out more.
We also note that Dutting et al. considers a different class of auctions (truthful auctions vs allpay auctions in our paper) so our results cannot be readily compared in a quantitative way. We will highlight this difference in the paper so it is more clear and add a more in depth comparison to that work.
Regarding learned experimental baselines, we only assume differentiable models. For example, we could have fit a Gaussian process instead. We chose neural networks as they are known to be strong function approximators (and generalizers), and we felt they would be most familiar to the ICLR community. However, our choice of neural networks is not critical to the general applicability of our approach or our novel perspective of both sides of the mechanism as adaptive. We will include experiments with other differentiable models in the appendix.
We point out though that Figure 3 and 7 both show the neural network’s ability to fit the data. Figure 3 displays a near perfect fit suggesting the MLP model class was sufficient for the noiseless 2 bidder setting. Figure 7 shows that there is potentially room for improvement in the noisy 10 bidder setting. We are considering alternative architectures for this setting, but per your suggestion, we will consider other differentiable models as well.

Lastly, thank you for pointing out the typo in Algorithm 2. The correct reference should be M-EMA.

We feel this rebuttal addresses your two concerns in detail: 1) strength of methodology and 2) experimental results. We would appreciate it if you would please consider raising your score.

Your second concern regards making a better distinction between which parts of our pipeline are generally applicable, and which are specific to the all-pay auction we consider for our experimental evaluation. We will update our manuscript shortly to include this information in the discussion. Here is a detailed answer which we hope helps.

Stage 1: The purpose of the first stage is simply to generate data samples, specifically tuples of the form (design, auctioneer utility). The rest of our pipeline is agnostic to how this data is obtained, but we took the novel approach of training learning agents in a simulated auction again because we are the first to view both sides of the mechanism (bidders/players and auctioneer/game) as adaptive. We explored both agents trained via fictitious self-play and reinforce. One specific choice we made was our desire to find a symmetric Nash equilibrium. This led to our training of agents with tied weights to maintain symmetry. This restriction can be released without repercussions to the general pipeline.

Stage 2: Any differentiable model can be used to approximate the mapping from designs to utilities here. We chose to use a multi-layer perceptron (neural network).

Stage 3: M-EMA was designed with the allpay auction in mind, however, we mentioned above other settings where it may be useful. A sophisticated optimization algorithm is not required by the pipeline though. We only make the assumption that the model that is fit in the second stage of the pipeline be differentiable with respect to its inputs (designs). This condition enables the use of gradient descent in the third stage. If the design space adheres to complex constraints, a flexible approach would be to use Lagrange multipliers to penalize deviation of iterates from the feasible set. We leave the choice of how to incorporate the constraints into the optimization process up to the user though. We simply provide this example of Lagrange multipliers to show that our pipeline is not limited to using only M-EMA.

We feel this rebuttal addresses your two concerns in detail: 1) lack of novelty and 2) generality / specificity of approach. We would appreciate it if you would please consider raising your score.

Thank you very much for taking the time to review our paper and for your feedback, which we think will help us write a better manuscript.
Your review calls out two main areas of improvement: 1) the need to better highlight what in our approach is novel, and 2) the discussion of which aspects of our approach are generally applicable to any mechanism design problems, and which are specific to all-pay auctions.

First off, thank you for identifying these, we feel like we can address them relatively easily and that our paper will be better for it.

The main concern you have identified is the “lack of novelty”, a criticism shared by R1 as well. We now realize we should have done a better job calling out which aspects of our work are novel, and which are just novel applications of existing methods. We will update our manuscript shortly to make sure the distinction is clear; here is a list of contributions which we hope you will find helpful.

1) Given this view, the first contribution of our paper, which we feel is a substantial one, is that we are the first to view both sides of a mechanism as adaptive.
In classic mechanism design work, researchers use either equilibrium behavior analysis, or appeal to dominant strategy arguments to model the behavior of the bidders (e.g. Nash equilibrium or envy-free equilibria, and dominant strategy incentive compatible mechanisms). This is also true of modern machine learning approaches to mechanism design such as Dutting et al. 2017, Feng et al. 2018, Manish et al. 2018 and Tacchetti et al. 2019. 
Restricting to mechanisms for which calculating the equilibrium behavior is tractable is a substantial limitation which shrinks the space of mechanisms and designs one can consider to a relatively small subset: as we highlight in the paper, mechanisms that have sufficient complexity, or disruptive enough levels of noise are excluded.
Classic mechanism design would throw in the towel here, and conclude that one cannot properly evaluate designs in these settings, or use models chosen for their tractability rather than their accuracy. This is important: in this work we show that a mechanism designed for the tractable noiseless setting is far from optimal in the noisy setting.
In this paper we present a novel way around this limitation. We view both the bidders and the auctioneer as adaptive, and use the behavior at convergence of learning agents as a stand-in for equilibrium behavior. We validate this idea when the equilibrium can be computed, and show that our learning agents converge to very similar strategies (Sec. 3.1), and then show that our method extends to situations where the equilibrium behavior is unknown (Sec. 3.2).
Because we are the first to view both sides of the mechanism as adaptive, we can apply this general pipeline, which as you pointed out uses standard machine learning techniques, to the challenging domain of mechanism design.

The second contribution of our paper concerns the specific optimization method we employ. Entropic Mirror Descent [Beck and Teboulle ‘03] is a non-Euclidean first-order optimization method tailored for optimization problems where the feasible set is a simplex. In our setting, we are maximizing the auctioneer’s utility over designs (feasible designs lie on a subset of the simplex), which is why we refer to it as Entropic Mirror Ascent (EMA) (simply flip the sign of the learning rate). However, not all designs on the simplex are feasible. We only want to consider designs with monotonically decreasing prizes (1st prize > 2nd prize > …). Our trick of introducing new variables that represent the marginals between the prizes (e.g., 1st prize - 2nd prize) transforms the feasible set into a new one in which a linear transformation of the marginals must lie on a simplex. By introducing this transformation and applying EMA in this new space, we effectively constrain gradient ascent search to the desired feasible design set (monotonically decreasing, positive prizes with constant sum). We name the novel application of EMA on this transformed space Monotonic EMA. An algorithm for optimizing over the subset of the simplex with monotonically decreasing values may be of general interest, not just to the allpay auction setting. For instance,
Classical Economics often studies monotonically increasing production functions (i.e., positive marginal returns). A company way want optimize over the space of possible production functions to see which maximizes its profit given its role in the market.
Some tree search algorithms such as A* require monotonic heuristic functions. Searching over the space of possible heuristic functions for a single path from root to a leaf node of a given value implies the feasible set is an (n-1)-simplex where n is the number of nodes on the path (excluding the leaf node).

Meta-review: This paper demonstrates a framework for optimizing designs in auction/contest problems. The approach relies on considering a multi-agent learning process and then simulating it. 

To a large degree there is agreement among reviewers that this approach is sensible and sound, however lacks substantial novelty. The authors provided a rebuttal which clarified the aspects that they consider novel, however the reviewers remained mostly unconvinced. Furthermore, it would help if the improvement over past approaches is demonstrated in a more convincing way, for example with increased scope experiments that also involve richer analysis.


Decision: reject