PreferenceNet: Encoding Human Preferences in Auction Design with Deep Learning

1 Introduction . Auctions are an essential tool in many marketplaces , including those in electricity [ 8 ] , advertising [ 14 ] , and telecommunications [ 9 , 26 ] . The design of auctions with desirable properties is thus an important theoretical and practical problem . A typical assumption is that bidders may choose to bid strategically and will successfully anticipate the behavior of other bidders . This results in a potentially complicated Bayes-Nash equilibrium which may be difficult to predict . To evade this problem , a common requirement is that an auction should be strategyproof ( i.e . bidders should be incentivized to truthfully share their valuations regardless of other bidders ’ actions ) . If the goal of an auction is to maximize the total welfare of all participants , the Vickrey-ClarkeGroves ( VCG ) mechanism is both strategyproof and welfare-maximizing [ 40 , 7 , 20 ] . Intuitively , in many cases the VCG mechanism corresponds to a second-price auction . If the goal is to maximize the revenue gained , the problem is more challenging . Myerson ’ s work completely characterizes strategyproof , revenue-maximizing auctions for a single item [ 29 ] . Beyond this case , results are more limited . Some results are known for the “ multiple-good monopolist ” problem , in which the auctioneer sells multiple items to a single bidder [ 11 , 12 , 24 , 31 , 28 , 21 , 18 ] . In addition , designing auctions for the related but weaker notion of Bayes-Nash incentive compatibility is reasonably well understood [ 2 , 3 , 4 ] . There has been significant difficulty in designing strategyproof auctions involving multiple items and multiple bidders . [ 41 ] shares significant , but limited results which solve the case in which items * The first two authors contributed equally to this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . may have at most 2 values . Due to the apparent difficulty of analytically designing strategyproof , revenue maximizing auctions , recent methods instead approximate optimal auctions using machine learning approaches [ 13 , 10 , 19 , 25 , 15 , 33 , 32 ] . [ 37 ] proposes an method which guarantees exact strategyproofness in the single-agent setting . [ 39 , 1 ] also use neural networks in the design of auctions . These methods primarily focus on standard mechanism design goals of revenue or welfare maximization , but these may not be the only goals of an auction . An auction might be conducted for public goods ( i.e . electromagnetic spectrum distribution [ 9 ] ) , where the auctioneer must consider the effect not only on the auctioneer and bidders but on third-parties as well [ 30 ] . In other cases , such as auctions involving job or credit advertisements , it might be necessary to additionally constrain the auction mechanism to ensure fairness with respect to protected characteristics [ 23 , 6 ] . Recent work has considered the problem of determining revenue-maximizing , strategyproof , fair auctions , either from a specific class [ 5 ] or with a general neural network approach [ 25 ] . The underlying notions of “ fairness ” used in these papers ( e.g . total-variation fairness [ 23 ] ) are defensible but somewhat arbitrary – they are attempts to formally capture human intuition . It might instead be better to attempt to capture , from data , human intuitions about fairness or other ethical requirements that are not explicitly defined [ 38 , 16 ] . In this paper , we introduce PreferenceNet , an extension to RegretNet that directly encodes socially desirable constraints from data , and captures noisy human preferences . We are motivated by advertising auctions where the allocations of the auction mechanism must satisfy human preference constraints alongside the typical goals of strategyproofness and revenue maximization . We conduct a number of experiments using synthetic data ( as is typical for neural network based auction mechanisms [ 13 ] ) and evaluate our method on different auction settings and fairness constraints . We show that PreferenceNet is able to effectively capture each fairness constraint and matches the performance of standard approaches . We also conduct two surveys to further study human preferences . In the first survey ( n=140 ) , when given a specific definition of fairness , we ask participants to determine if a given auction setting is fair in order to test the noise in human judgements . In the second survey ( n=345 ) , we elicit judgements of pairwise comparisons of two auction settings to determine preferences without priming the participants with any particular definition of fairness . We train PreferenceNet on these data and show that our approach can capture nuanced human preferences in auction design . 2 Background and Related Work . RegretNet . [ 13 ] presents RegretNet , a neural-network architecture for learning approximately strategyproof auctions that maximize revenue . RegretNet treats the auction mechanism as a function from bids to allocations and payments , parameterized as a neural network . Revenue is optimized via gradient descent on sampled truthful bid profiles ; strategyproofness is enforced by computing strategic bids in an adversarial manner to minimize violations . RegretNet has been modified and extended in a variety of ways , particularly to enforce additional desirable constraints . [ 10 , 39 , 19 , 25 , 15 , 33 ] . Special Case : Single-Bidder Auctions . We highlight a special case in which deep learning for auctions has been particularly successful – auctions with a single bidder . In the single-bidder setting , the set of strategyproof mechanisms can be easily characterized [ 34 ] , and it is possible to design neural network architectures which will always lie in this set . [ 13 , 37 ] present two different learningbased solutions . Both methods are guaranteed to be strategyproof , and revenue can be maximized by unconstrained optimization . There are some known optimal single-agent mechanisms ( we highlight those of Manelli-Vincent [ 28 ] and Pavlov [ 31 ] , as they are relevant to auction settings we study below ) . Moreover , the theory of single-bidder mechanisms is well understood [ 11 , 12 , 18 ] , so it is also possible to learn empirically strong mechanisms using these approaches and then prove their optimality [ 37 , 13 ] . Unfortunately , when moving beyond the single-agent setting , it is necessary to use more general neural network architectures for which these guarantees do not apply . Because we are interested in such settings , we focus on these general architectures in this work . Fairness and Human Preferences . As mentioned , while revenue , strategyproofness , and individual rationality are the classic goals of auction design , it might also be necessary for allocations made by an auction to satisfy certain other requirements such as fairness [ 5 , 25 ] . However , it is not always clear if these mathematical definitions of these concepts actually capture human intuitions . [ 36 , 38 ] considers human reactions to different definitions of fairness in a classification setting . [ 35 ] tests the extent to which human participants are able to understand and apply fairness metrics . [ 16 ] considers the problem of learning to perform fair clustering from human-provided demonstrations . As discussed below , we also crowdsource human opinions on fairness in auctions and analyze the results in Section 6 . 3 Problem Setting . Auction Model . An auction is defined as a set of agents N = { 1 , . . . , n } bidding for items M = { 1 , . . . , m } . Each agent i ∈ N has a corresponding private valuation vi , randomly drawn from a set of n valuations as v = ( v1 , ... vn ) ∈ Vi . In general vi may be functions over the power set of items 2M . However , we only consider simpler cases with additive valuations and unit-demand valuations , where the valuation is simply a vector vi ∈ Rm of values per item . Each agent reports a bid vector bi to the auctioneer , which may differ from the private valuation vi . Given the profile of bids b = ( b1 , . . . , bn ) of all agents , the auction has allocation and payment rules g ( b ) : Rmn → [ 0 , 1 ] nm and p ( b ) : Rmn → Rn . We will refer to the matrix of allocation probabilities , whose rows must sum to 1 , as g ( b ) = z . Likewise agent i ’ s value of the jth item is vi , j , and bidder i has payment function pi . Moreover , for unit-demand auctions , we restrict the allocation to allow each bidder to win , in expectation , at most 1 item . Given the allocation , each agent receives a utility which can in either case be represented in linear form as ui ( v ) = ∑ j vi , jzi , j − pi . Desirable Auction Properties . A mechanism is individually rational ( IR ) when an agent is guaranteed non-negative utility : ui ( vi ; v ) ≥ 0 ∀i ∈ N , v ∈ V . A mechanism is dominant-strategy incentivecompatible ( DSIC ) or strategyproof if every agent maximizes their own utility by bidding truthfully , regardless of the other agents ’ bids . We can define regret , the difference in utility between the bid player i actually made and the best possible strategic bid : rgti ( v ) = maxbi ui ( bi , v−i ) − ui ( vi , v−i ) . In addition to satisfying the IR and DSIC constraints , the auctioneer seeks to maximize their expected revenue . If the auction is truly DSIC , players will bid truthfully , and as a result revenue is simply Ev∼V [ ∑ i∈N pi ( v ) ] . 4 PreferenceNet : Encoding Human Preferences . We first explore a new metric to evaluate the adherence to socially desirable constraints in item allocations . Next , we describe the implementation details of PreferenceNet and important considerations when training the model . Evaluation Metrics . There are limited evaluation criteria that quantitatively measures an auction mechanism ’ s ability to enforce constraints on item allocations . If the constraints are not known explicitly , one can qualitatively examine the allocation graphs to evaluate the underlying allocation function g ( b ) . However , visual inspection does not scale to larger auction settings . As shown in Figure 1 , our proposed metric is not only able to capture the same insights as qualitative analysis , but also scales to arbitrarily large auction mechanisms . Given the limitations of existing analysis techniques , we propose Preference Classification Accuracy ( PCA ) , a new metric to evaluate how well a learned auction model satisfies an arbitrary constraint . For a set of test bids b and allocations g ( b ) : Rmn → [ 0 , 1 ] nm generated by our learned auction model , we assign a label s ( b ) ∈ { 1 , 0 } to each allocation according to a ground truth labeling function based on the underlying preference . For each test bid b , s ( b ) is 1 if the learned auction network satisfies the ground truth constraint . PCA is calculated by averaging the value of s ( b ) over n test bids . Note that this metric remains valid in cases when we know the underlying preference function ( e.g . total variation fairness , entropy ) as well as when we are sampling from an unknown distribution ( e.g . human preference elicitation ) . For cases where the underlying preference function is known , we can directly compute the preference score for a given allocation and apply a threshold to obtain a label . For cases where the preference function is unknown , as is the case in human preference elicitation , we can use the ground truth allocation-label pair to assign preference labels s ( b ) to new allocations based on the nearest neighbor in the ground truth set . This metric gives us a formal way to measure the degree to which preference constraints are violated , which crucially can be used whether or not the constraints follow an explicitly-known function . Preference Elicitation . In order to effectively elicit preferences , we rely on pairwise comparisons between allocations to identify both positive and negative exemplars . We compare each input set of of allocations against n other allocations to determine if a particular sample is preferred over these alternatives ( see Figure 2 ) . This method of group preference elicitation reduces noise and ensures that the learned preference is satisfactory to a majority of the participants . We use this ranking approach to generate training labels in all of our experiments as described below . Training Algorithm . We present the training algorithm of PreferenceNet below . We use the same additive and unit-demand network architectures as RegretNet for arbitrary numbers of agents and items . Our training algorithm follows closely from RegretNet . PreferenceNet consists of two subnetworks : RegretNet and a 3-layer MLP with a sigmoid activation at the output . We first train the MLP using a uniformly drawn sample of allocations as inputs , and optimize the binary cross entropy loss function using ground truth labels ( generated as in Figure 2 ) identifying positive and negative exemplars . Next , we train RegretNet using the standard training procedure with the modified loss function described in Eq . 1 . Lastly , we sample allocations and payments from the partially trained RegretNet model every c epoch and augment the MLP training set to adapt to the distributional shifts in allocations over the course of training . Our modified loss function Cρ ( w ; λ ) is defined as : Lrgt = ∑ i∈N λ ( r , i ) rgti ( w ) + ρr 2 ∑ i∈N rgti ( w ) 2 , Ls = ∑ j∈M sj Cρ ( w ; λ ) = − 1 L L∑ l=1 ∑ i∈N pwi ( v ( l ) ) + Lrgt − Ls ( 1 ) where Ls is the output of the trained MLP . RegretNet is optimized such that it is strategyproof , revenue-maximizing , satisfies the preference learned by the MLP ( i.e . maximizes the output scores of the MLP ) . For each configuration of n agents and m items , we train RegretNet for a maximum of 200 epoch using 160,000 training samples . We also train the MLP with 80,000 initial training samples and iteratively retrain the MLP with 5,000 additional samples from the partially trained RegretNet every 5 epoch . For all networks , we use the Adam optimizer and 100 hidden nodes per layer . We apply warm restarts to the MLP optimizer each time we add new training data to to prevent the model from settling in a local minima . We incremented ρr every 2500 iterations and λr every 25 iterations . Finally , we report the preference classification accuracy , mean regret , and mean payments by simulating the allocations of 20,000 testing samples . We run all our experiments on an NVIDIA Titan X ( Pascal ) GPU . We refer readers to GitHub for our implementation . MLP Architecture . We learn implicit preference functions using a simple 3-layer multi-layer perceptron that takes as input a mini-batch of allocations and outputs a vector that scores the input ∈ [ 0 , 1 ] as a measure of how closely the allocation satisfies the ground truth preference . Given that neural networks are universal function approximators , with enough parameters the MLP can represent any arbitrary preference function . In practice , we find that ReLU non-linear activations and batch normalization are essential to effectively train this network . Class Balanced Sampling . The distribution of positive and negative training examples for arbitrary preferences are often imbalanced – preferred examples may be concentrated in a small region of possible allocations . Often , this imbalance can make it difficult to train a robust model . Commonly , neural networks with improper class balance fail to learn a good decision boundary . In order to compensate for such imbalanced datasets , we can explicitly over-sample sparse classes until there are an equal number of positive and negative training examples . In practice this allows the network to quickly learn the decision boundary and allows us to train with fewer data points . MLP Co-Training . The distribution of payments and allocations generated by RegretNet shift significantly during the course of training . As a result , the initially trained MLP may not effectively enforce the preference loss as RegretNet continues to train . In order to adapt to the distribution shift , we sample allocations from RegretNet at fixed intervals while it is training , add them back to the training set , and retrain the MLP . We generate noisy labels [ 27 ] using the existing MLP to reduce the cost of collecting expensive labels . In practice , we find that this approach effectively reinforces the decision boundary between positive and negative exemplars . Model Validation and Selection . The optimal model minimizes regret , while maximizing payments and preference classification accuracy . In practice , satisfying all three conditions may be difficult . Often , we find that choosing a fixed epoch to evaluate does not provide consistent results . Rather , we evaluate each checkpoint on a validation set and maximize the following criteria in Eq . 2 : α ∗ PCA + β ∗ ¯p ( b ) max ( p ( b ) ) + γ ∗ ( 1− ¯rgt ( b ) max ( rgt ( b ) ) ) s.t . α+ β + γ = 1 ( 2 ) In our experiments we set α = 0.45 , β = 0.1 , γ = 0.45 . It is important to note that the maximum regret and payments are calculated over the entire training process , while the mean regret , payment , and preference classification accuracy are calculated at each epoch .

Summary: This paper covers a somewhat interesting and novel problem: learning constraints of an auction setting, and then learning to run an auction that respects those. Fairness in allocations is cited as a justification for this in practice, where it may be beneficial to learn from a human’s choices to determine how to decide who should win in the auction, for example to determine similar allocations across demographics.  The approach builds on the RegretNet line of work that learns to run revenue maximizing auctions. The approach broadly is to learn a feasibility constraint that is scored by human subjects, and then use that in the RegretNet architecture to learn to generate auctions that respect that constraint. In this work, the motivating constraint is a binary classification of fairness. The implementation is a composition of learning then constraint, followed by leveraging RegretNet to learn according to that function.   To support the approach, the authors first compare PreferenceNet to RegretNet trained with the exact constraints, and see similar performance suggesting some amount of usefulness. The authors then train PreferenceNet on responses from human subjects in two surveys regarding the binary qualitative fairness of allocations.