PreferenceNet: Encoding Human Preferences in Auction Design with Deep Learning

1 Introduction . Auctions are an essential tool in many marketplaces , including those in electricity [ 8 ] , advertising [ 14 ] , and telecommunications [ 9 , 26 ] . The design of auctions with desirable properties is thus an important theoretical and practical problem . A typical assumption is that bidders may choose to bid strategically and will successfully anticipate the behavior of other bidders . This results in a potentially complicated Bayes-Nash equilibrium which may be difficult to predict . To evade this problem , a common requirement is that an auction should be strategyproof ( i.e . bidders should be incentivized to truthfully share their valuations regardless of other bidders ’ actions ) . If the goal of an auction is to maximize the total welfare of all participants , the Vickrey-ClarkeGroves ( VCG ) mechanism is both strategyproof and welfare-maximizing [ 40 , 7 , 20 ] . Intuitively , in many cases the VCG mechanism corresponds to a second-price auction . If the goal is to maximize the revenue gained , the problem is more challenging . Myerson ’ s work completely characterizes strategyproof , revenue-maximizing auctions for a single item [ 29 ] . Beyond this case , results are more limited . Some results are known for the “ multiple-good monopolist ” problem , in which the auctioneer sells multiple items to a single bidder [ 11 , 12 , 24 , 31 , 28 , 21 , 18 ] . In addition , designing auctions for the related but weaker notion of Bayes-Nash incentive compatibility is reasonably well understood [ 2 , 3 , 4 ] . There has been significant difficulty in designing strategyproof auctions involving multiple items and multiple bidders . [ 41 ] shares significant , but limited results which solve the case in which items * The first two authors contributed equally to this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . may have at most 2 values . Due to the apparent difficulty of analytically designing strategyproof , revenue maximizing auctions , recent methods instead approximate optimal auctions using machine learning approaches [ 13 , 10 , 19 , 25 , 15 , 33 , 32 ] . [ 37 ] proposes an method which guarantees exact strategyproofness in the single-agent setting . [ 39 , 1 ] also use neural networks in the design of auctions . These methods primarily focus on standard mechanism design goals of revenue or welfare maximization , but these may not be the only goals of an auction . An auction might be conducted for public goods ( i.e . electromagnetic spectrum distribution [ 9 ] ) , where the auctioneer must consider the effect not only on the auctioneer and bidders but on third-parties as well [ 30 ] . In other cases , such as auctions involving job or credit advertisements , it might be necessary to additionally constrain the auction mechanism to ensure fairness with respect to protected characteristics [ 23 , 6 ] . Recent work has considered the problem of determining revenue-maximizing , strategyproof , fair auctions , either from a specific class [ 5 ] or with a general neural network approach [ 25 ] . The underlying notions of “ fairness ” used in these papers ( e.g . total-variation fairness [ 23 ] ) are defensible but somewhat arbitrary – they are attempts to formally capture human intuition . It might instead be better to attempt to capture , from data , human intuitions about fairness or other ethical requirements that are not explicitly defined [ 38 , 16 ] . In this paper , we introduce PreferenceNet , an extension to RegretNet that directly encodes socially desirable constraints from data , and captures noisy human preferences . We are motivated by advertising auctions where the allocations of the auction mechanism must satisfy human preference constraints alongside the typical goals of strategyproofness and revenue maximization . We conduct a number of experiments using synthetic data ( as is typical for neural network based auction mechanisms [ 13 ] ) and evaluate our method on different auction settings and fairness constraints . We show that PreferenceNet is able to effectively capture each fairness constraint and matches the performance of standard approaches . We also conduct two surveys to further study human preferences . In the first survey ( n=140 ) , when given a specific definition of fairness , we ask participants to determine if a given auction setting is fair in order to test the noise in human judgements . In the second survey ( n=345 ) , we elicit judgements of pairwise comparisons of two auction settings to determine preferences without priming the participants with any particular definition of fairness . We train PreferenceNet on these data and show that our approach can capture nuanced human preferences in auction design . 2 Background and Related Work . RegretNet . [ 13 ] presents RegretNet , a neural-network architecture for learning approximately strategyproof auctions that maximize revenue . RegretNet treats the auction mechanism as a function from bids to allocations and payments , parameterized as a neural network . Revenue is optimized via gradient descent on sampled truthful bid profiles ; strategyproofness is enforced by computing strategic bids in an adversarial manner to minimize violations . RegretNet has been modified and extended in a variety of ways , particularly to enforce additional desirable constraints . [ 10 , 39 , 19 , 25 , 15 , 33 ] . Special Case : Single-Bidder Auctions . We highlight a special case in which deep learning for auctions has been particularly successful – auctions with a single bidder . In the single-bidder setting , the set of strategyproof mechanisms can be easily characterized [ 34 ] , and it is possible to design neural network architectures which will always lie in this set . [ 13 , 37 ] present two different learningbased solutions . Both methods are guaranteed to be strategyproof , and revenue can be maximized by unconstrained optimization . There are some known optimal single-agent mechanisms ( we highlight those of Manelli-Vincent [ 28 ] and Pavlov [ 31 ] , as they are relevant to auction settings we study below ) . Moreover , the theory of single-bidder mechanisms is well understood [ 11 , 12 , 18 ] , so it is also possible to learn empirically strong mechanisms using these approaches and then prove their optimality [ 37 , 13 ] . Unfortunately , when moving beyond the single-agent setting , it is necessary to use more general neural network architectures for which these guarantees do not apply . Because we are interested in such settings , we focus on these general architectures in this work . Fairness and Human Preferences . As mentioned , while revenue , strategyproofness , and individual rationality are the classic goals of auction design , it might also be necessary for allocations made by an auction to satisfy certain other requirements such as fairness [ 5 , 25 ] . However , it is not always clear if these mathematical definitions of these concepts actually capture human intuitions . [ 36 , 38 ] considers human reactions to different definitions of fairness in a classification setting . [ 35 ] tests the extent to which human participants are able to understand and apply fairness metrics . [ 16 ] considers the problem of learning to perform fair clustering from human-provided demonstrations . As discussed below , we also crowdsource human opinions on fairness in auctions and analyze the results in Section 6 . 3 Problem Setting . Auction Model . An auction is defined as a set of agents N = { 1 , . . . , n } bidding for items M = { 1 , . . . , m } . Each agent i ∈ N has a corresponding private valuation vi , randomly drawn from a set of n valuations as v = ( v1 , ... vn ) ∈ Vi . In general vi may be functions over the power set of items 2M . However , we only consider simpler cases with additive valuations and unit-demand valuations , where the valuation is simply a vector vi ∈ Rm of values per item . Each agent reports a bid vector bi to the auctioneer , which may differ from the private valuation vi . Given the profile of bids b = ( b1 , . . . , bn ) of all agents , the auction has allocation and payment rules g ( b ) : Rmn → [ 0 , 1 ] nm and p ( b ) : Rmn → Rn . We will refer to the matrix of allocation probabilities , whose rows must sum to 1 , as g ( b ) = z . Likewise agent i ’ s value of the jth item is vi , j , and bidder i has payment function pi . Moreover , for unit-demand auctions , we restrict the allocation to allow each bidder to win , in expectation , at most 1 item . Given the allocation , each agent receives a utility which can in either case be represented in linear form as ui ( v ) = ∑ j vi , jzi , j − pi . Desirable Auction Properties . A mechanism is individually rational ( IR ) when an agent is guaranteed non-negative utility : ui ( vi ; v ) ≥ 0 ∀i ∈ N , v ∈ V . A mechanism is dominant-strategy incentivecompatible ( DSIC ) or strategyproof if every agent maximizes their own utility by bidding truthfully , regardless of the other agents ’ bids . We can define regret , the difference in utility between the bid player i actually made and the best possible strategic bid : rgti ( v ) = maxbi ui ( bi , v−i ) − ui ( vi , v−i ) . In addition to satisfying the IR and DSIC constraints , the auctioneer seeks to maximize their expected revenue . If the auction is truly DSIC , players will bid truthfully , and as a result revenue is simply Ev∼V [ ∑ i∈N pi ( v ) ] . 4 PreferenceNet : Encoding Human Preferences . We first explore a new metric to evaluate the adherence to socially desirable constraints in item allocations . Next , we describe the implementation details of PreferenceNet and important considerations when training the model . Evaluation Metrics . There are limited evaluation criteria that quantitatively measures an auction mechanism ’ s ability to enforce constraints on item allocations . If the constraints are not known explicitly , one can qualitatively examine the allocation graphs to evaluate the underlying allocation function g ( b ) . However , visual inspection does not scale to larger auction settings . As shown in Figure 1 , our proposed metric is not only able to capture the same insights as qualitative analysis , but also scales to arbitrarily large auction mechanisms . Given the limitations of existing analysis techniques , we propose Preference Classification Accuracy ( PCA ) , a new metric to evaluate how well a learned auction model satisfies an arbitrary constraint . For a set of test bids b and allocations g ( b ) : Rmn → [ 0 , 1 ] nm generated by our learned auction model , we assign a label s ( b ) ∈ { 1 , 0 } to each allocation according to a ground truth labeling function based on the underlying preference . For each test bid b , s ( b ) is 1 if the learned auction network satisfies the ground truth constraint . PCA is calculated by averaging the value of s ( b ) over n test bids . Note that this metric remains valid in cases when we know the underlying preference function ( e.g . total variation fairness , entropy ) as well as when we are sampling from an unknown distribution ( e.g . human preference elicitation ) . For cases where the underlying preference function is known , we can directly compute the preference score for a given allocation and apply a threshold to obtain a label . For cases where the preference function is unknown , as is the case in human preference elicitation , we can use the ground truth allocation-label pair to assign preference labels s ( b ) to new allocations based on the nearest neighbor in the ground truth set . This metric gives us a formal way to measure the degree to which preference constraints are violated , which crucially can be used whether or not the constraints follow an explicitly-known function . Preference Elicitation . In order to effectively elicit preferences , we rely on pairwise comparisons between allocations to identify both positive and negative exemplars . We compare each input set of of allocations against n other allocations to determine if a particular sample is preferred over these alternatives ( see Figure 2 ) . This method of group preference elicitation reduces noise and ensures that the learned preference is satisfactory to a majority of the participants . We use this ranking approach to generate training labels in all of our experiments as described below . Training Algorithm . We present the training algorithm of PreferenceNet below . We use the same additive and unit-demand network architectures as RegretNet for arbitrary numbers of agents and items . Our training algorithm follows closely from RegretNet . PreferenceNet consists of two subnetworks : RegretNet and a 3-layer MLP with a sigmoid activation at the output . We first train the MLP using a uniformly drawn sample of allocations as inputs , and optimize the binary cross entropy loss function using ground truth labels ( generated as in Figure 2 ) identifying positive and negative exemplars . Next , we train RegretNet using the standard training procedure with the modified loss function described in Eq . 1 . Lastly , we sample allocations and payments from the partially trained RegretNet model every c epoch and augment the MLP training set to adapt to the distributional shifts in allocations over the course of training . Our modified loss function Cρ ( w ; λ ) is defined as : Lrgt = ∑ i∈N λ ( r , i ) rgti ( w ) + ρr 2 ∑ i∈N rgti ( w ) 2 , Ls = ∑ j∈M sj Cρ ( w ; λ ) = − 1 L L∑ l=1 ∑ i∈N pwi ( v ( l ) ) + Lrgt − Ls ( 1 ) where Ls is the output of the trained MLP . RegretNet is optimized such that it is strategyproof , revenue-maximizing , satisfies the preference learned by the MLP ( i.e . maximizes the output scores of the MLP ) . For each configuration of n agents and m items , we train RegretNet for a maximum of 200 epoch using 160,000 training samples . We also train the MLP with 80,000 initial training samples and iteratively retrain the MLP with 5,000 additional samples from the partially trained RegretNet every 5 epoch . For all networks , we use the Adam optimizer and 100 hidden nodes per layer . We apply warm restarts to the MLP optimizer each time we add new training data to to prevent the model from settling in a local minima . We incremented ρr every 2500 iterations and λr every 25 iterations . Finally , we report the preference classification accuracy , mean regret , and mean payments by simulating the allocations of 20,000 testing samples . We run all our experiments on an NVIDIA Titan X ( Pascal ) GPU . We refer readers to GitHub for our implementation . MLP Architecture . We learn implicit preference functions using a simple 3-layer multi-layer perceptron that takes as input a mini-batch of allocations and outputs a vector that scores the input ∈ [ 0 , 1 ] as a measure of how closely the allocation satisfies the ground truth preference . Given that neural networks are universal function approximators , with enough parameters the MLP can represent any arbitrary preference function . In practice , we find that ReLU non-linear activations and batch normalization are essential to effectively train this network . Class Balanced Sampling . The distribution of positive and negative training examples for arbitrary preferences are often imbalanced – preferred examples may be concentrated in a small region of possible allocations . Often , this imbalance can make it difficult to train a robust model . Commonly , neural networks with improper class balance fail to learn a good decision boundary . In order to compensate for such imbalanced datasets , we can explicitly over-sample sparse classes until there are an equal number of positive and negative training examples . In practice this allows the network to quickly learn the decision boundary and allows us to train with fewer data points . MLP Co-Training . The distribution of payments and allocations generated by RegretNet shift significantly during the course of training . As a result , the initially trained MLP may not effectively enforce the preference loss as RegretNet continues to train . In order to adapt to the distribution shift , we sample allocations from RegretNet at fixed intervals while it is training , add them back to the training set , and retrain the MLP . We generate noisy labels [ 27 ] using the existing MLP to reduce the cost of collecting expensive labels . In practice , we find that this approach effectively reinforces the decision boundary between positive and negative exemplars . Model Validation and Selection . The optimal model minimizes regret , while maximizing payments and preference classification accuracy . In practice , satisfying all three conditions may be difficult . Often , we find that choosing a fixed epoch to evaluate does not provide consistent results . Rather , we evaluate each checkpoint on a validation set and maximize the following criteria in Eq . 2 : α ∗ PCA + β ∗ ¯p ( b ) max ( p ( b ) ) + γ ∗ ( 1− ¯rgt ( b ) max ( rgt ( b ) ) ) s.t . α+ β + γ = 1 ( 2 ) In our experiments we set α = 0.45 , β = 0.1 , γ = 0.45 . It is important to note that the maximum regret and payments are calculated over the entire training process , while the mean regret , payment , and preference classification accuracy are calculated at each epoch .

Summary: This paper proposes PreferenceNet, which is a new deep learning approach for designing revenue-maximizing auctions that encodes human preferences. PreferenceNet extends RegretNet [Duetting et al. 2019] with modified loss functions and additional constraints.