Credible Sample Elicitation by Deep Learning, for Deep Learning

1 INTRODUCTION . The availability of a large quantity of credible samples is crucial for building high-fidelity machine learning models . This is particularly true for deep learning systems that are data-hungry . Arguably , the most scalable way to collect a large amount of training samples is to crowdsource from a decentralized population of agents who hold relevant sample information . The most popular example is the build of ImageNet ( Deng et al. , 2009 ) . The main challenge in eliciting private information is to properly score reported information such that the self-interested agent who holds a private information will be incentivized to report truthfully . At a first look , this problem of eliciting quality data is readily solvable with the seminal solution for eliciting distributional information , called the strictly proper scoring rule ( Brier , 1950 ; Winkler , 1969 ; Savage , 1971 ; Matheson & Winkler , 1976 ; Jose et al. , 2006 ; Gneiting & Raftery , 2007 ) : suppose we are interested in eliciting information about a random vector X = ( X1 , ... , Xd−1 , Y ) ∈ Ω ⊆ Rd , whose probability density function is denoted by p with distribution P. As the mechanism designer , if we have a sample x drawn from the true distribution P , we can apply strictly proper scoring rules to elicit p : the agent who holds p will be scored using S ( p , x ) . S is called strictly proper if it holds for any p and q that Ex∼P [ S ( p , x ) ] > Ex∼P [ S ( q , x ) ] . The above elicitation approach has two main caveats that limited its application : • When the outcome space |Ω| is large and is even possibly infinite , it is practically impossible for any human agents to report such a distribution with reasonable efforts . This partially inspired a line of follow-up works on eliciting property of the distributions , which we will discuss later . • The mechanism designer may not possess any ground truth samples . In this work we aim to collect credible samples from self-interested agents via studying the problem of sample elicitation . Instead of asking each agent to report the entire distribution p , we hope to elicit samples drawn from the distribution P truthfully . We consider the samples xp ∼ P and xq ∼ Q . In analogy to strictly proper scoring rules1 , we aim to design a score function S s.t . Ex∼P [ S ( xp , x′ ) ] > Ex∼P [ S ( xq , x′ ) ] for any q ̸= p , where x′ is a reference answer that can be defined using elicited reports . Often , this scoring procedure requires reports from multiple peer agents , and x′ is chosen as a function of the reported samples from all other agents ( e.g. , the average across all the reported xs , or a randomly selected x ) . This setting will relax the requirements of high reporting complexity , and has wide applications in collecting training samples for machine learning tasks . Indeed our goal resembles similarity to property elicitation ( Lambert et al. , 2008 ; Steinwart et al. , 2014 ; Frongillo & Kash , 2015b ) , but we emphasize that our aims are different - property elicitation aims to elicit statistical properties of a distribution , while ours focus on eliciting samples drawn from the distributions . In certain scenarios , when agents do not have the complete knowledge or power to compute these properties , our setting enables elicitation of individual sample points . Our challenge lies in accurately evaluating reported samples . We first observe that the f -divergence function between two properly defined distributions of the samples can serve the purpose of incentivizing truthful report of samples . We proceed with using deep learning techniques to solve the score function design problem via a data-driven approach . We then propose a variational approach that enables us to estimate the divergence function efficiently using reported samples , via a variational form of the f -divergence function , through a deep neutral network . These estimation results help us establish an approximate incentive compatibility in eliciting truthful samples . It is worth to note that our framework also generalizes to the setting where there is no access to ground truth samples , where we can only rely on reported samples . There we show that our estimation results admit an approximate Bayesian Nash Equilibrium for agents to report truthfully . Furthermore , in our estimation framework , we use a generative adversarial approach to reconstruct the distribution from the elicited samples . We want to emphasize that the deep learning based estimators considered above are able to handle complex data . And with our deep learning solution , we are further able to provide estimates for the divergence functions used for our scoring mechanisms with provable finite sample complexity . In this paper , we focus on developing theoretical guarantees - other parametric families either can not handle complex data , e.g. , it is hard to handle images using kernel methods , or do not have provable guarantees on the sample complexity . Our contributions are three-folds . ( 1 ) We tackle the problem of eliciting complex distribution via proposing a sample elicitation framework . Our deep learning aided solution concept makes it practical to solicit complex sample information from human agents . ( 2 ) Our framework covers the case when the mechanism designer has no access to ground truth information , which adds contribution to the peer prediction literature . ( 3 ) On the technical side , we develop estimators via deep learning techniques with strong theoretical guarantees . This not only helps us establish approximate incentive-compatibility , but also enables the designer to recover the targeted distribution from elicited samples . Our contribution can therefore be summarized as “ eliciting credible training samples by deep learning , for deep learning '' . 1.1 RELATED WORKS . The most relevant literature to our paper is strictly proper scoring rules and property elicitation . Scoring rules were developed for eliciting truthful prediction ( probability ) ( Brier , 1950 ; Winkler , 1969 ; Savage , 1971 ; Matheson & Winkler , 1976 ; Jose et al. , 2006 ; Gneiting & Raftery , 2007 ) . Characterization results for strictly proper scoring rules are given in McCarthy ( 1956 ) ; Savage ( 1971 ) ; Gneiting & Raftery ( 2007 ) . Property elicitation notices the challenge of eliciting complex distributions ( Lambert et al. , 2008 ; Steinwart et al. , 2014 ; Frongillo & Kash , 2015b ) . For instance , Abernethy & Frongillo ( 2012 ) characterize the score functions for eliciting linear properties , and Frongillo & Kash ( 2015a ) study the complexity of eliciting properties . Another line of relevant research is peer prediction , where solutions can help elicit private information when the ground truth verification might be missing ( De Alfaro et al. , 2016 ; Gao et al. , 2016 ; Kong et al. , 2016 ; 1Our specific formulation and goal will be different in details . Kong & Schoenebeck , 2018 ; 2019 ) . Our work complements the information elicitation literature via proposing and studying the question of sample elicitation via a variational approach to estimate f -divergence functions . Our work also extends the line of work on divergence estimation . The simplest way to estimate divergence starts with the estimation of density function ( Wang et al. , 2005 ; Lee & Park , 2006 ; Wang et al. , 2009 ; Zhang & Grabchak , 2014 ; Han et al. , 2016 ) . Another method based on the variational form ( Donsker & Varadhan , 1975 ) of the divergence function comes into play ( Broniatowski & Keziou , 2004 ; 2009 ; Nguyen et al. , 2010 ; Kanamori et al. , 2011 ; Ruderman et al. , 2012 ; Sugiyama et al. , 2012 ) , where the estimation of divergence is modeled as the estimation of density ratio between two distributions . The variational form of the divergence function also motivates the well-know Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ) , which learns the distribution by minimizing the Kullback-Leibler divergence . Follow-up works include Nowozin et al . ( 2016 ) ; Arjovsky et al . ( 2017 ) ; Gulrajani et al . ( 2017 ) ; Bellemare et al . ( 2017 ) , with theoretical analysis in Liu et al . ( 2017 ) ; Arora et al . ( 2017 ) ; Liang ( 2018 ) ; Gao et al . ( 2019 ) . See also Gao et al . ( 2017 ) ; Bu et al . ( 2018 ) for this line of work . 1.2 NOTATIONS . For the distribution P , we denote by Pn the empirical distribution given a set of samples { xi } ni=1 following P , i.e. , Pn = 1/n · ∑n i=1 δxi , where δxi is the Dirac measure at xi . We denote by ∥v∥s = ( ∑d i=1 |v ( i ) |s ) 1/s the ℓs norm of the vector v ∈ Rd where 1 ≤ s < ∞ and v ( i ) is the i-th entry of v. We also denote by ∥v∥∞ = max1≤i≤d |v ( i ) | the ℓ∞ norm of v. For any real-valued continuous function f : X → R , we denote by ∥f∥Ls ( P ) : = [ ∫ X |f ( x ) | s dP ] 1/s the Ls ( P ) norm of f and ∥f∥s : = [ ∫ X |f ( x ) | s dµ ] 1/s the Ls ( µ ) norm of f ( · ) , where µ is the Lebesgue measure . Also , we denote by ∥f∥∞ = supx∈X |f ( x ) | the L∞ norm of f ( · ) . For any real-valued functions g ( · ) and h ( · ) defined on some unbounded subset of the real positive numbers , such that h ( α ) is strictly positive for all large enough values of α , we write g ( α ) ≲ h ( α ) and g ( α ) = O ( h ( α ) ) if |g ( α ) | ≤ c · h ( α ) for some positive absolute constant c and any α > α0 , where α0 is a real number . We denote by [ n ] the set { 1 , 2 , . . . , n } . 2 PRELIMINARY . We formulate the question of sample elicitation . 2.1 SAMPLE ELICITATION . We consider two scenarios . We start with an easier case where we , as the mechanism designer , have access to a certain number of group truth samples . This is a setting that resembles similarity to the proper scoring rule setting . Then we move to the harder case where the inputs to our mechanism can only be elicited samples from agents . Multi-sample elicitation with ground truth samples . Suppose that the agent holds n samples , with each of them independently drawn from P , i.e. , xi ∼ P 2 for i ∈ [ n ] . The agent can report each sample arbitrarily , which is denoted as ri ( xi ) : Ω → Ω . There are n data { x∗i } i∈ [ n ] independently drawn from the ground truth distribution Q3 . We are interested in designing a score function S ( · ) that takes inputs of each ri ( · ) and { rj ( xj ) , x∗j } j∈ [ n ] : S ( ri ( xi ) , { rj ( xj ) , x∗j } j∈ [ n ] ) such that if the agent believes that x∗ is drawn from the same distribution x∗ ∼ P , then for any { rj ( · ) } j∈ [ n ] , it holds with probability at least 1− δ that n∑ i=1 Ex , x∗∼P [ S ( xi , { xj , x∗j } j∈ [ n ] ) ] ≥ n∑ i=1 Ex , x∗∼P [ S ( ri ( xi ) , { rj ( xj ) , x∗j } j∈ [ n ] ) ] − n · ϵ . 2Though we use x to denote the samples we are interested in , x potentially includes both the feature and labels ( x , y ) as in the context of supervised learning . 3The number of ground truth samples can be different from n , but we keep them the same for simplicity of presentation . It will mainly affect the terms δ and ϵ in our estimations . We name the above as ( δ , ϵ ) -properness ( per sample ) for sample elicitation . When δ = ϵ = 0 , it is reduced to the one that is similar to the properness definition in scoring rule literature ( Gneiting & Raftery , 2007 ) . We also shorthand ri = ri ( xi ) when there is no confusion . Agent believes that her samples are generated from the same distribution as of the ground truth samples , i.e. , P and Q are same distributions . Sample elicitation with peer samples . Suppose there are n agents each holding a sample xi ∼ Pi , where the distributions { Pi } i∈ [ n ] are not necessarily the same - this models the fact that agents can have subjective biases or local observation biases . This is a more standard peer prediction setting . We denote by their joint distribution as P = P1 × P2 × .... × Pn . Similar to the previous setting , each agent can report her sample arbitrarily , which is denoted as ri ( xi ) : Ω → Ω for any i ∈ [ n ] . We are interested in designing and characterizing a score function S ( · ) that takes inputs of each ri ( · ) and { rj ( xj ) } j ̸=i : S ( ri ( xi ) , { rj ( xj ) } j ̸=i ) such that for any { rj ( · ) } j∈ [ n ] , it holds with probability at least 1− δ that Ex∼P [ S ( xi , { rj ( xj ) = xj } j ̸=i ) ] ≥ Ex∼P [ S ( r ( xi ) , { rj ( xj ) = xj } j ̸=i ) ] −ϵ . We name the above as ( δ , ϵ ) -Bayesian Nash Equilibrium ( BNE ) in truthful elicitation . We only require that agents are all aware of above information structure as common knowledge , but they do not need to form beliefs about details of other agents ’ sample distributions . Each agent ’ s sample is private to herself . 2.2 f -DIVERGENCE It is well known that maximizing the expected proper scores is equivalent to minimizing a corresponding Bregman divergence ( Gneiting & Raftery , 2007 ) . More generically , we take the perspective that divergence functions have great potentials to serve as score functions for eliciting samples . We define the f -divergence between two distributions P and Q with probability density function p and q , respectively , as Df ( q∥p ) = ∫ p ( x ) f ( q ( x ) p ( x ) ) dµ . ( 2.1 ) Here f ( · ) is a function satisfying certain regularity conditions , which will be specified later . Solving our elicitation problem involves evaluating the Df ( q∥p ) successively based on the distributions P and Q , without knowing the probability density functions p and q . Therefore , we have to resolve to a form of Df ( q∥p ) which does not involve the analytic forms of p and q , but instead sample forms . Following from Fenchel ’ s convex duality , it holds that Df ( q∥p ) = max t ( · ) Ex∼Q [ t ( x ) ] − Ex∼P [ f† ( t ( x ) ) ] , ( 2.2 ) where f† ( · ) is the Fenchel duality of the function f ( · ) , which is defined as f† ( u ) = supv∈R { uv − f ( v ) } , and the max is taken over all functions t ( · ) : Ω ⊂ Rd → R .

Summary: This paper studies the sample elicitation problem where agents are asked to report samples. The goal is then to evaluate the quality of these reported samples by means of a scoring function S. Following previous related works, the authors use the equivalence between maximizing the expected proper score and minimizing some f-divergence. Their approach relies on the dual expression of the f-divergence which writes as a maximum over a set of functions t. Theoretical guarantees are given for f-scorings obtained (with or without ground truth samples) by first computing the empirical optimal function t, then plugged to estimate the f-divergence. Finally, a deep learning approach is proposed by considering functions f parameterized as sparse deep neural networks.