Credible Sample Elicitation by Deep Learning, for Deep Learning

1 INTRODUCTION . The availability of a large quantity of credible samples is crucial for building high-fidelity machine learning models . This is particularly true for deep learning systems that are data-hungry . Arguably , the most scalable way to collect a large amount of training samples is to crowdsource from a decentralized population of agents who hold relevant sample information . The most popular example is the build of ImageNet ( Deng et al. , 2009 ) . The main challenge in eliciting private information is to properly score reported information such that the self-interested agent who holds a private information will be incentivized to report truthfully . At a first look , this problem of eliciting quality data is readily solvable with the seminal solution for eliciting distributional information , called the strictly proper scoring rule ( Brier , 1950 ; Winkler , 1969 ; Savage , 1971 ; Matheson & Winkler , 1976 ; Jose et al. , 2006 ; Gneiting & Raftery , 2007 ) : suppose we are interested in eliciting information about a random vector X = ( X1 , ... , Xd−1 , Y ) ∈ Ω ⊆ Rd , whose probability density function is denoted by p with distribution P. As the mechanism designer , if we have a sample x drawn from the true distribution P , we can apply strictly proper scoring rules to elicit p : the agent who holds p will be scored using S ( p , x ) . S is called strictly proper if it holds for any p and q that Ex∼P [ S ( p , x ) ] > Ex∼P [ S ( q , x ) ] . The above elicitation approach has two main caveats that limited its application : • When the outcome space |Ω| is large and is even possibly infinite , it is practically impossible for any human agents to report such a distribution with reasonable efforts . This partially inspired a line of follow-up works on eliciting property of the distributions , which we will discuss later . • The mechanism designer may not possess any ground truth samples . In this work we aim to collect credible samples from self-interested agents via studying the problem of sample elicitation . Instead of asking each agent to report the entire distribution p , we hope to elicit samples drawn from the distribution P truthfully . We consider the samples xp ∼ P and xq ∼ Q . In analogy to strictly proper scoring rules1 , we aim to design a score function S s.t . Ex∼P [ S ( xp , x′ ) ] > Ex∼P [ S ( xq , x′ ) ] for any q ̸= p , where x′ is a reference answer that can be defined using elicited reports . Often , this scoring procedure requires reports from multiple peer agents , and x′ is chosen as a function of the reported samples from all other agents ( e.g. , the average across all the reported xs , or a randomly selected x ) . This setting will relax the requirements of high reporting complexity , and has wide applications in collecting training samples for machine learning tasks . Indeed our goal resembles similarity to property elicitation ( Lambert et al. , 2008 ; Steinwart et al. , 2014 ; Frongillo & Kash , 2015b ) , but we emphasize that our aims are different - property elicitation aims to elicit statistical properties of a distribution , while ours focus on eliciting samples drawn from the distributions . In certain scenarios , when agents do not have the complete knowledge or power to compute these properties , our setting enables elicitation of individual sample points . Our challenge lies in accurately evaluating reported samples . We first observe that the f -divergence function between two properly defined distributions of the samples can serve the purpose of incentivizing truthful report of samples . We proceed with using deep learning techniques to solve the score function design problem via a data-driven approach . We then propose a variational approach that enables us to estimate the divergence function efficiently using reported samples , via a variational form of the f -divergence function , through a deep neutral network . These estimation results help us establish an approximate incentive compatibility in eliciting truthful samples . It is worth to note that our framework also generalizes to the setting where there is no access to ground truth samples , where we can only rely on reported samples . There we show that our estimation results admit an approximate Bayesian Nash Equilibrium for agents to report truthfully . Furthermore , in our estimation framework , we use a generative adversarial approach to reconstruct the distribution from the elicited samples . We want to emphasize that the deep learning based estimators considered above are able to handle complex data . And with our deep learning solution , we are further able to provide estimates for the divergence functions used for our scoring mechanisms with provable finite sample complexity . In this paper , we focus on developing theoretical guarantees - other parametric families either can not handle complex data , e.g. , it is hard to handle images using kernel methods , or do not have provable guarantees on the sample complexity . Our contributions are three-folds . ( 1 ) We tackle the problem of eliciting complex distribution via proposing a sample elicitation framework . Our deep learning aided solution concept makes it practical to solicit complex sample information from human agents . ( 2 ) Our framework covers the case when the mechanism designer has no access to ground truth information , which adds contribution to the peer prediction literature . ( 3 ) On the technical side , we develop estimators via deep learning techniques with strong theoretical guarantees . This not only helps us establish approximate incentive-compatibility , but also enables the designer to recover the targeted distribution from elicited samples . Our contribution can therefore be summarized as “ eliciting credible training samples by deep learning , for deep learning '' . 1.1 RELATED WORKS . The most relevant literature to our paper is strictly proper scoring rules and property elicitation . Scoring rules were developed for eliciting truthful prediction ( probability ) ( Brier , 1950 ; Winkler , 1969 ; Savage , 1971 ; Matheson & Winkler , 1976 ; Jose et al. , 2006 ; Gneiting & Raftery , 2007 ) . Characterization results for strictly proper scoring rules are given in McCarthy ( 1956 ) ; Savage ( 1971 ) ; Gneiting & Raftery ( 2007 ) . Property elicitation notices the challenge of eliciting complex distributions ( Lambert et al. , 2008 ; Steinwart et al. , 2014 ; Frongillo & Kash , 2015b ) . For instance , Abernethy & Frongillo ( 2012 ) characterize the score functions for eliciting linear properties , and Frongillo & Kash ( 2015a ) study the complexity of eliciting properties . Another line of relevant research is peer prediction , where solutions can help elicit private information when the ground truth verification might be missing ( De Alfaro et al. , 2016 ; Gao et al. , 2016 ; Kong et al. , 2016 ; 1Our specific formulation and goal will be different in details . Kong & Schoenebeck , 2018 ; 2019 ) . Our work complements the information elicitation literature via proposing and studying the question of sample elicitation via a variational approach to estimate f -divergence functions . Our work also extends the line of work on divergence estimation . The simplest way to estimate divergence starts with the estimation of density function ( Wang et al. , 2005 ; Lee & Park , 2006 ; Wang et al. , 2009 ; Zhang & Grabchak , 2014 ; Han et al. , 2016 ) . Another method based on the variational form ( Donsker & Varadhan , 1975 ) of the divergence function comes into play ( Broniatowski & Keziou , 2004 ; 2009 ; Nguyen et al. , 2010 ; Kanamori et al. , 2011 ; Ruderman et al. , 2012 ; Sugiyama et al. , 2012 ) , where the estimation of divergence is modeled as the estimation of density ratio between two distributions . The variational form of the divergence function also motivates the well-know Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ) , which learns the distribution by minimizing the Kullback-Leibler divergence . Follow-up works include Nowozin et al . ( 2016 ) ; Arjovsky et al . ( 2017 ) ; Gulrajani et al . ( 2017 ) ; Bellemare et al . ( 2017 ) , with theoretical analysis in Liu et al . ( 2017 ) ; Arora et al . ( 2017 ) ; Liang ( 2018 ) ; Gao et al . ( 2019 ) . See also Gao et al . ( 2017 ) ; Bu et al . ( 2018 ) for this line of work . 1.2 NOTATIONS . For the distribution P , we denote by Pn the empirical distribution given a set of samples { xi } ni=1 following P , i.e. , Pn = 1/n · ∑n i=1 δxi , where δxi is the Dirac measure at xi . We denote by ∥v∥s = ( ∑d i=1 |v ( i ) |s ) 1/s the ℓs norm of the vector v ∈ Rd where 1 ≤ s < ∞ and v ( i ) is the i-th entry of v. We also denote by ∥v∥∞ = max1≤i≤d |v ( i ) | the ℓ∞ norm of v. For any real-valued continuous function f : X → R , we denote by ∥f∥Ls ( P ) : = [ ∫ X |f ( x ) | s dP ] 1/s the Ls ( P ) norm of f and ∥f∥s : = [ ∫ X |f ( x ) | s dµ ] 1/s the Ls ( µ ) norm of f ( · ) , where µ is the Lebesgue measure . Also , we denote by ∥f∥∞ = supx∈X |f ( x ) | the L∞ norm of f ( · ) . For any real-valued functions g ( · ) and h ( · ) defined on some unbounded subset of the real positive numbers , such that h ( α ) is strictly positive for all large enough values of α , we write g ( α ) ≲ h ( α ) and g ( α ) = O ( h ( α ) ) if |g ( α ) | ≤ c · h ( α ) for some positive absolute constant c and any α > α0 , where α0 is a real number . We denote by [ n ] the set { 1 , 2 , . . . , n } . 2 PRELIMINARY . We formulate the question of sample elicitation . 2.1 SAMPLE ELICITATION . We consider two scenarios . We start with an easier case where we , as the mechanism designer , have access to a certain number of group truth samples . This is a setting that resembles similarity to the proper scoring rule setting . Then we move to the harder case where the inputs to our mechanism can only be elicited samples from agents . Multi-sample elicitation with ground truth samples . Suppose that the agent holds n samples , with each of them independently drawn from P , i.e. , xi ∼ P 2 for i ∈ [ n ] . The agent can report each sample arbitrarily , which is denoted as ri ( xi ) : Ω → Ω . There are n data { x∗i } i∈ [ n ] independently drawn from the ground truth distribution Q3 . We are interested in designing a score function S ( · ) that takes inputs of each ri ( · ) and { rj ( xj ) , x∗j } j∈ [ n ] : S ( ri ( xi ) , { rj ( xj ) , x∗j } j∈ [ n ] ) such that if the agent believes that x∗ is drawn from the same distribution x∗ ∼ P , then for any { rj ( · ) } j∈ [ n ] , it holds with probability at least 1− δ that n∑ i=1 Ex , x∗∼P [ S ( xi , { xj , x∗j } j∈ [ n ] ) ] ≥ n∑ i=1 Ex , x∗∼P [ S ( ri ( xi ) , { rj ( xj ) , x∗j } j∈ [ n ] ) ] − n · ϵ . 2Though we use x to denote the samples we are interested in , x potentially includes both the feature and labels ( x , y ) as in the context of supervised learning . 3The number of ground truth samples can be different from n , but we keep them the same for simplicity of presentation . It will mainly affect the terms δ and ϵ in our estimations . We name the above as ( δ , ϵ ) -properness ( per sample ) for sample elicitation . When δ = ϵ = 0 , it is reduced to the one that is similar to the properness definition in scoring rule literature ( Gneiting & Raftery , 2007 ) . We also shorthand ri = ri ( xi ) when there is no confusion . Agent believes that her samples are generated from the same distribution as of the ground truth samples , i.e. , P and Q are same distributions . Sample elicitation with peer samples . Suppose there are n agents each holding a sample xi ∼ Pi , where the distributions { Pi } i∈ [ n ] are not necessarily the same - this models the fact that agents can have subjective biases or local observation biases . This is a more standard peer prediction setting . We denote by their joint distribution as P = P1 × P2 × .... × Pn . Similar to the previous setting , each agent can report her sample arbitrarily , which is denoted as ri ( xi ) : Ω → Ω for any i ∈ [ n ] . We are interested in designing and characterizing a score function S ( · ) that takes inputs of each ri ( · ) and { rj ( xj ) } j ̸=i : S ( ri ( xi ) , { rj ( xj ) } j ̸=i ) such that for any { rj ( · ) } j∈ [ n ] , it holds with probability at least 1− δ that Ex∼P [ S ( xi , { rj ( xj ) = xj } j ̸=i ) ] ≥ Ex∼P [ S ( r ( xi ) , { rj ( xj ) = xj } j ̸=i ) ] −ϵ . We name the above as ( δ , ϵ ) -Bayesian Nash Equilibrium ( BNE ) in truthful elicitation . We only require that agents are all aware of above information structure as common knowledge , but they do not need to form beliefs about details of other agents ’ sample distributions . Each agent ’ s sample is private to herself . 2.2 f -DIVERGENCE It is well known that maximizing the expected proper scores is equivalent to minimizing a corresponding Bregman divergence ( Gneiting & Raftery , 2007 ) . More generically , we take the perspective that divergence functions have great potentials to serve as score functions for eliciting samples . We define the f -divergence between two distributions P and Q with probability density function p and q , respectively , as Df ( q∥p ) = ∫ p ( x ) f ( q ( x ) p ( x ) ) dµ . ( 2.1 ) Here f ( · ) is a function satisfying certain regularity conditions , which will be specified later . Solving our elicitation problem involves evaluating the Df ( q∥p ) successively based on the distributions P and Q , without knowing the probability density functions p and q . Therefore , we have to resolve to a form of Df ( q∥p ) which does not involve the analytic forms of p and q , but instead sample forms . Following from Fenchel ’ s convex duality , it holds that Df ( q∥p ) = max t ( · ) Ex∼Q [ t ( x ) ] − Ex∼P [ f† ( t ( x ) ) ] , ( 2.2 ) where f† ( · ) is the Fenchel duality of the function f ( · ) , which is defined as f† ( u ) = supv∈R { uv − f ( v ) } , and the max is taken over all functions t ( · ) : Ω ⊂ Rd → R .

Summary: This paper proposes a sample elicitation framework to tackle the problem of eliciting credible samples from agents for complex distributions. The authors suggest that deep neural frameworks can be applied in this framework for sample elicitation through the derivations. The authors also show the connection between the problem of sample elicitation and f-GAN. However, some problems in the proof on sample elicitation should be clarified or carefully explained.