Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a transformer-based network architecture called RegretFormer to address the problem of optimal auction design. RegretFormer differs from prior optimization techniques that frame the problem as a mini-max between regret and revenue. Instead, RegretFormer defines a fixed regret budget and maximizes revenue. Maximum regret is used as a single interpretable hyperparameter to tune the auction design. The transformer architecture allows for variable sized inputs, and attention is used to enforce equivariance. RegretFormer only studies symmetric auctions, leaving asymmetric auctions (with the use of positional encoding) as future work. Different from prior works (c.f. RegretNet and EquivarientNet), RegretFormer uses a single network to predict both allocations and payments. Lastly, this paper proposes two supplemental evaluation protocols, namely cross-misreport regret estimates and network distillation to verify that the performance improvement over prior works is genuine.

strengths_and_weaknesses: This paper is generally well written and clear. The main takeaways are clear and the authors appropriately juxtapose their work in the context of prior work (c.f. RegretNet and EquivariantNet) and make it clear how their work differs. The authors convincingly show that RegretFormer performs better than both RegretNet and EquivariantNet. Moreover, the authors ask the right question - is the performance gap genuine? Authors devise protocols for addressing this question. 

The cross-misreport regret estimate, although well motivated, is not convincing because there is no empirical proof to suggest how much an adversarially constructed example would underestimate regret. Without this point of reference, it's difficult to interpret the results presented in Table 3 (given variance between training runs). However, the network distillation seems to have the opposite problem, the result is convincing, but since RegretFormer is able to outperform EquivariantNet and RegretNet, the motivation that network architecture of RegretFormer impairs its ability to approximate optimal misreports isn't a major concern. To be clear, these attempts at deeply understanding if the performance improvement by RegretFormer is genuine are appreciated.

One concern about evaluation is w.r.t setting a maximum regret for comparison (Table 1). From the main paper, it's not clear how maximum regret is enforced for RegretNet and  EquivariantNet since these are optimized as a mini-max of regret and revenue. According to Appendix 1.2, it seems like exponential annealing is used to achieve this constraint. However, since this is not how either RegretNet or EquivariantNet are normally trained and (L182) both models are sensitive to the hyperparameters and training protocol, is the comparison with RegretFormer at a fixed maximum regret fair? Here's a protocol that may address this concern: First, train RegretNet using the standard unconstrained (w.r.t maximum regret) optimization. Given the maximum regret of the trained RegretNet, set this is the maximum regret of RegretFormer and train the model. Now the comparison is fair. This protocol can be naturally extended to EquivariantNet. Ideally, it might be the case that RegretFormer still outperforms prior methods.

questions: 1. How long does it take to train RegretFormer when compared to RegretNet and EquivariantNet. How does this scale according to the size of the auction? 
2. Since the distribution of bids are different depending on the size of the auction, it isn't clear why we would expect "zero-shot" out-of-setting generalization by default. Is it the case that the RegretFormer architecture allows us to fine tune for new settings with fewer examples?
3. Does evaluating with the proposed "fair" protocol defined above change any conclusions?

limitations: Authors highlight that although modulating the maximum regret parameter should proportionally change revenue, this sometimes does not hold. Since the interpretable maximum regret parameter is a key selling point of this work, it would be useful to implement the proposed solution (L247) to show that this tighter revenue/regret ratio holds. Authors also present general limitations of a regret-based approach to auction design, highlighting that under the smallest regret budget, regret is still non-zero. 
It would be helpful to make limitations of this method an explicit subsection. The current writeup (L250) reads as "all methods have limitations, but RegretFormer does better", which isn't particularly helpful for understanding where RegretFormer fails.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper modifies the RegreNet architecture and training in the following ways -  
(1) it replaces the neural network architecture with a transformer  
(2) it replaces the augmented lagrangian optimization with a new loss formulation (rather than enforcing IC violation to be strictly 0, it only requires them to be at some threshold $R_{max}$
strengths_and_weaknesses: **Strengths**: 
The paper aims to tackle problems that are significant and relevant and are in the right direction - it would be nice to use optimization methods that were less sensitive to the hyperparameters than the augmented Lagrangian optimization method.

**Weaknesses**: 
(1) The contribution is not significantly novel. The use of transformers was already proposed by [54]
(2) The transformer architecture has limitations - it can't handle asymmetric distributions.
(3) Eqn (6) doesn't seem to be a proper relaxation of the optimization problem in (5). 
(4) The evaluation is limited. The authors only consider uniform distribution on the interval
(5) Missing details on regret computation at test time
(6) I'm not convinced that the cross-misreport testing rules out the possibility of regret being accurate. The loss landscapes are very different for different neural network architectures. While low regret on cross misreports is a necessary condition, it isn't a sufficient one. 
questions: (1) Is Eqn (6) missing a term perhaps something like $\max (\tilde{R} - R_{max}, 0)$ ?   
(2) There are other symmetric distributions where this architecture can be tested - single bidder but from the uniform interval [c, c+1], uniform triangle settings/correlated valuations, and settings with unit-demand valuations where we know the optimal solution analytically.  
(3) How is the regret computed at test time? How many samples of misreports are initialized and how many gradient descent steps to compute them? What is the size of the test set?  
(4) Can you visualize the solution for the 1x2 setting? The optimal auction is given by Maneli and Vincent which has 4 deterministic menu options. This achieves a revenue of $0.55$. Selling it as a grand bundle has revenue that's slightly less than $0.55$ My intuition is that the setting when $R_{max} = 1e-5$, the latter is recovered based on the revenue values reported. If this is the case, perhaps the proposed optimization algorithm here is sensitive to $\gamma$ initialization as well.
(5) Would be possible to use tools from [Certifying Strategyproof Auction Networks](https://arxiv.org/abs/2006.08742) to compute regret more accurately?
limitations: Discussed above!
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a new architecture for auction design, RegretFormer, which is based on the Transformer architecture. Unlike the original RegretNet, RegretFormer is invariant to the number of items and bidders and is permutation equivariant due to attention mechanisms. In addition, the authors relax the loss objective by introducing a regret budget $R_{max}$ to allow for smoother training and more robustness to hyperparameters. RegretFormer is shown to outperform baselines in both revenue and regret overall. Moreover, the authors confirm the validity of the regret estimates via a novel cross-misreports approach where the regret of one network is estimated on the optimal misreports approximated by another network. The results show that there is little evidence of over-estimate/bias in the regret estimates.
strengths_and_weaknesses: ### Strengths:
- The RegretFormer architecture offers desirable properties of invariance and permutation equivariance due to attention.
- The authors propose a smoother loss function that allows for pre-specification of a regret budget.
- The cross-misreport experiments offer a nice approach to validate the regret estimates which may be heavily biased by the trained network.

### Weakness:
- The authors claim that there modified loss function is more robust to hyperparameters. However, there are no experiments to prove this. It would be nice to see the performance under different hyperparameters compared to baselines.
- While RegretFormer is well justified for this problem, the architecture itself is a straightforward application of the transformer architecture.
questions: - What is the size of your networks in terms of number of parameters compared to baselines? It would nice to see these number to ensure that the comparison between baselines is fair enough.
- How robust is RegretFormer to different hyperparameters compared to RegretNet and EquivariantNet?
- The generalization results in Table 3 show that RegretFormer outperforms EquivariantNet when trained on 3x10 setting but the same is not true for smaller settings like 1x2 and 2x2? Do you have intuition on why this is the case?
- How much does RegretFormer benefit from the modified loss objective? An small ablation study would be helpful here.
limitations: While the results are positive, I found the paper to be lacking in terms of analysis of the results to better understand the advantages/disadvantages of RegretFormer. For example, how sample efficient is RegretFormer compared to baselines?
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The use of deep learning for mechanism design is an expanding area. One important paper in the field, RegretNet, directly uses deep neural networks to approximate strategyproof and revenue-maximizing auctions. The current work builds on RegretNet in several ways. First, it uses an attention-based architecture -- this seems to work well, and has the key advantage of being permutation-equivariant (a desirable property, as it is known that permutation-equivariant mechanisms are optimal for symmetric auction design problems and often good in general). It also allows networks trained on one input size to generalize successfully to other input sizes. Second, the authors prevent a modified training algorithm that makes it easier to specify an acceptable small violation of the strategyproofness constraint, while reducing the complexity of tuning RegretNet's strange hyperparameters. The authors test their method and show performance that usually beats previous methods across a wide range of settings.
strengths_and_weaknesses: - Originality
    - It's a fairly natural idea to use transformers for auctions, because apparently people can use them for everything else too. There is concurrent work (CITransNet of Duan et al.) that also uses transformers and has some of the same advantages (generalization, equivariance).
    - In addition to the transformer idea, this paper presents some useful techniques, either explicitly claimed as contributions (e.g. the tweaked training objective) or implicitly (the methodology for making sure the performance gap is real).
- Significance:
    - I think this is a significant and valuable piece of work. It presents new state-of-the-art results for automated mechanism design, a new training procedure which is easier to work with, and is very methodologically careful in the way it evaluates performance.
- Quality
    - The quality is high, especially in the extremely thorough experiments. It's particularly nice that the authors take the time (section 4.2) to do experiments to make sure apparent outperformance is not just due to "gradient obfuscation" interfering with the misreport procedure, but is more likely due to actually finding a better mechanism.
    - The actual results in terms of performance don't seem like a complete slam dunk. It is true that in basically all cases, the RegretFormer *architecture* does better than the RegretNet/EquivariantNet architectures.
And for the medium 10^{-4} regret budget, things look great. But for the high regret budget, revenues seem oddly high across all networks. And for the low regret budget, revenues are depressingly low. So I am convinced that the RegretFormer architecture is better than previous alternatives, but have some reservations about the new training method (although it still seems to have advantages).
- Clarity
    - Everything is explained quite clearly.

questions: - How might your architecture be modified to support unit-demand constraints (i.e. both rows and columns of the allocation matrix sum to <=1)? Is it as straightforward as just tacking the RegretNet approach on the end?

- How might your method be modified when equivariance is undesirable (e.g. for non-symmetric valuation distributions where you really do want to discriminate against one bidder)?
limitations: - The claim, mentioned as an aside, that the Myerson auction relies on having the full parametric valuation distribution seems not entirely correct to me, and I think this should be clarified. For one thing, Duetting et al. gives the "MyersonNet" architecture in their appendix; for another, there is a lot of work (cited with admirable thoroughness in the related work section) on sample complexity of revenue maximizing auctions where you can learn to approximate the right reserve price. I think methods like these would count as a "Myerson baseline" but work fine from samples.

- The authors are honest about where their work has limitations (e.g. low regret budget underperforming Myerson). I think a little bit more discussion and speculation about reasons the performance of the high/low regret budgets is warranted, although the NeurIPS page limit may prevent this.
review_title: null

Meta-review: metareview: This paper builds on the nascent optimal auction design via deep learning literature.  It modifies the at-this-point standard approach, RegretNet, to this setting from Duetting et al (ICML-19, but on arXiv earlier) by incorporating a transformer-based architecture into, roughly, the same training scenario, albeit with a different loss function that is more appropriate for this setting.  That loss function is a nice add-on in that RegretNet and kin are notoriously hard to train, and this seems to help.  From an empirical point of view alone, I believe this paper is worth accepting because it pushes the boundary on a tricky empirical problem, both in terms of ease of training but also in terms of (in many cases) the quality of the final network itself, measured in revenue / IC violation.  I agree with some of the negative sentiment from Reviewer u7uQ, though, w.r.t. over-claims of "optimal-er" and poor comparison against analytic baselines; this should be changed in a final/future version of the work.
confidence: Certain
Decision: Accept