Validation Free and Replication Robust Volume-based Data Valuation

1 Introduction . Data is increasingly recognized as a valuable resource [ 19 ] , so we need a principled measure of its worth . A suitable data valuation has wide-ranging applications such as fairly compensating clinical trial researchers for their collected data [ 12 , 16 , 25 ] , fostering collaborative machine learning and federated learning among industrial organizations [ 35 , 36 , 39 ] , encouraging trusted data sharing and building data marketplaces [ 7 , 30 , 32 , 37 ] , among others . A popular viewpoint is that the value of data should correlate with the learning performance of a model trained on the data [ 14 , 18 ] , which enforces a close coupling between data valuation and validation . However , a validation set may not always be available in practice [ 35 ] . Also , as different choices of the validation set can lead to different data valuations , it is challenging for the data providers to agree on the choice of such a validation set [ 35 ] . Since valuation is coupled with validation , if the ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . validation set is not sufficiently representative of the distribution of test queries in a learning task , the resulting valuation may not be as accurate/useful [ 40 ] . We adopt a different perspective : The value of data should be related to its intrinsic properties and valuation can be decoupled from validation by considering the inherent diversity of the data . Intuitively , a more diverse collection of data points corresponds to a higher-quality dataset and thus yields a larger value . This perspective circumvents the above practical limitations and allows our valuation method to be model- and task-agnostic . We formalize diversity via the volume of the data matrix ( i.e. , determinant of its left Gram ) . Data replication is another practical issue in data valuation due to the digital nature and anonymous setting of data marketplaces [ 15 ] . Supposing a dataset has some value and a data provider instead offers one containing two copies of every data point in this dataset , is this “ new ” dataset twice as valuable as the original one ? Intuitively , the answer should be no as replication adds no new data and so does not increase diversity . We formalize this intuition by constructing a compressed version of the original data to assign little value to replicated data and still preserve its inherent diversity , hence guaranteeing replication robustness . We provide theoretical justifications for formalizing diversity via volume : Firstly , diversity should be non-negative and monotonic [ 14 , 18 , 35 , 38 ] and volume satisfies both properties . Secondly , a greater diversity should lead to a better learning performance [ 23 ] : We formally show that a larger volume generally leads to a better performance using the ordinary least squares ( OLS ) framework and our method can be flexibly adapted to handle more complex machine learning models ( i.e. , various neural networks ) in our experiments . Specifically , data with a larger volume can lead to a more accurate pseudo-inverse ( i.e. , a key component of the least squares solution ) and a smaller mean squared error . To ensure replication robustness , we find that the marginal increase in value from replication must diminish to zero . Otherwise , a data provider can exploit this valuation by making infinite copies of the data to achieve infinite value . We thus formalize the notion of replication robustness via the asymptotic value attainable through replication . Unfortunately , the conventional definition of volume does not have this property . So , we propose a robust volume ( RV ) measure by constructing a compressed version of the original data that groups similar data via discretized cubes of the input feature space and represents those in each cube via a statistic . The RV measure offers practitioners the flexibility to trade off between diversity representation and replication robustness via the cube ’ s width . We perform extensive experiments on synthetic and real-world datasets to demonstrate that our method produces consistent valuations with existing methods while making fewer assumptions . The specific contributions of our work here include : • Formalizing a measure of data diversity via the volume of data ( Sec . 2 ) and justifying the suitability of volume for data valuation both theoretically ( Sec . 3 ) and empirically ( Sec . 5 ) ; • Formalizing the notion of replication robustness and designing a data valuation method based on the robust volume ( RV ) measure with a theoretical guarantee on replication robustness ( Sec . 4 ) ; • Performing extensive empirical comparisons with baselines to demonstrate that our method is consistent in valuation without validation , replication robust , and can be flexibly adapted to handle complex machine learning models such as various neural networks ( Sec . 5 ) . 2 Problem Setting and Notations . Consider two data submatrices XS and XS′ to be valued that contain s and s′ rows of d-dimensional input feature vectors , respectively . Let PS : = [ X > S 0 ] > ∈ Rn×d be the zero-padded version of XS ∈ Rs×d . We concatenate the data submatrices along the rows to form the full data matrix X ∈ Rn×d , i.e. , X : = [ X > S X > S′ ] > and n = s + s′ . We denote the corresponding observed labels/responses as y : = [ y > S y > S′ ] > ∈ Rn×1 . The least squares solution from OLS is w : = X+y = argminβ ‖y −Xβ‖ 2 where X+ : = ( X > X ) −1X > is the pseudo-inverse of X . Similarly , we denote X+S as the pseudo-inverse of XS and wS : = X + SyS . To ease notations , let V : = Vol ( X ) and VS : = Vol ( XS ) where Vol ( ) is defined below . Let |A| denote the determinant of a square matrix A . The left Gram matrix of X is G : = X > X ∈ Rd×d , so for data submatrix XS , GS : = X > SXS ∈ Rd×d . Definition 1 ( Volume ) . For a full-rank X ∈ Rn×d with n ≥ d , Vol ( X ) : = √ | ( X > X ) | = √ |G| . We adopt the above definition of volume for several reasons : ( a ) Often , the input feature space of the data is pre-determined and fixed due to the data collection process . But , new data can stream in and so , n can grow indefinitely while d remains fixed [ 9 , 10 ] . ( b ) By leveraging the formal connection between volume and learning performance ( Sec . 3 ) , we can design a validation free volume-based data valuation to assign a larger value to data leading to a better learning performance . ( c ) This affords an intuitive interpretation between volume and diversity : Adding a data point to a dataset can increase the diversity/volume depending on the data points already in the dataset ( Lemma 1 ) . We restrict our discussion to full-rank matrices X , XS , and XS′ since otherwise we can adopt the Gram-Schmidt process to remove the linearly dependent columns [ 9 , 10 ] . In practice , we perform pre-processing such as principal component analysis to reduce the dimension of the input feature space to ensure that this assumption is satisfied . This assumption is to ensure that there are no redundant features , namely , features that can be exactly reconstructed using other features . For instance , if a dataset already contains monthly salaries , then an annual salary would be redundant . 3 Larger Volume Entails Better Learning Performance . The value of a data ( sub ) matrix depends on the learning performance trained on it [ 14 , 18 ] which , we will show , depends on its volume . Simply put , the larger the volume , the better the learning performance . In this section , we will formalize this claim through the ordinary least squares ( OLS ) framework . In particular , we will investigate two metrics for learning performance : ( a ) the quality of the pseudo-inverse represented by biasS : = ∥∥P+S −X+∥∥ because estimating X+ accurately is important to achieving small mean squared error ( MSE ) [ 9 ] and where P+S : = ( X > SXS ) −1P > S , and ( b ) the MSE denoted as L ( wS ) : = ‖y −XwS‖2 . 3.1 Larger Volume Entails Smaller Bias . In regression problems , the closed-form optimal solution is constructed via X+ computed using X . So , the bias of P+S from X + indirectly determines the value of XS [ 9 ] , i.e. , a smaller bias means a larger value . We show in Proposition 1 below that ‘ a larger volume means a smaller bias ’ always holds for d = 1 . For d > 1 , it requires additional assumptions which are mostly satisfied via empirical verification ( Fig . 1 ) . Proposition 1 ( Volume vs . Bias for d = 1 ) . For non-zero XS , XS′ of X ∈ Rn×1 , VS ≥ VS′ ⇐⇒ biasS − biasS′ ≤ 0 . The above result can be generalized to M > 2 non-zero data submatrices : Let X : = [ X > S1 X > S2 · · · X > SM ] > and w.l.o.g. , suppose that VS1 ≥ VS2 ≥ . . . ≥ VSM . Then , biasS1 ≤ biasS2 ≤ . . . ≤ biasSM . For d > 1 , counterexamples exist ( see Fig . 1 ) , so we instead compare biasS and biasS′ in the next result : Proposition 2 ( Volume vs . Bias in General ) . For full-rank XS , XS′ of X ∈ Rn×d , bias2S − bias2S′ = 1 V 4S ∥∥QSX > S ∥∥2 − 1V 4S′ ∥∥QS′X > S′∥∥2 + 2 〈 1 V 2 QX > , 1 V 2S′ QS′P > S′ − 1 V 2S QSP > S 〉 where Q : = ∑k l=1 ( λlσl ) −1∏k j=1 , j 6=l ( G − λjI ) , { λl } kl=1 denotes the k unique eigenvalues of the left Gram matrix G of X , QS , QS′ are similarly defined w.r.t . GS , GS′ , PS and PS′ are , respectively , zero-padded versions of XS and XS′ , and σl : =∑k g=1 ( −1 ) g+1λ k−g l [ ∑ H⊆ { 1 , ... , k } \ { l } , |H|=g−1 ( ∏ h∈ { 1 , ... , k } \H λ −1 h ) ] . The proof of Proposition 1 ( Appendix A.1 ) relies on a key observation that for d = 1 , the left Gram matrix is a number and the rest of the proof follows . However , it can not be generalized to that for d > 1 , so we resort to a different proof technique . The proof of Proposition 2 requires Lemma 2 in Appendix A.1 which establishes a formal connection between volume and G−1 using the Sylvester ’ s formula . To obtain VS ≥ VS′ =⇒ biasS ≤ biasS′ , there are two cases requiring different additional assumptions : ( A ) VS VS′ , and ( B ) ∥∥QSX > S ∥∥ ≈ ∥∥QS′X > S′∥∥ and V max ( VS , VS′ ) . Case A is intuitive : VS VS′ means XS is much “ larger ” in volume than XS′ , so biasS is smaller . Case B is when XS and XS′ are similar ( e.g. , when they are sampled from the same data distribution ) . The intuition is that the first difference term will be relatively large in magnitude ( so , its sign will dominate the overall expression ) , while the second inner product term will be relatively small in magnitude . This is because the first difference term involves 1/V 4S and 1/V 4 S′ but the second inner product term involves 1/ ( V 2×V 2S ) and 1/ ( V 2×V 2S′ ) , and we show V max ( VS , VS′ ) ( Lemma 3 in Appendix A.1 ) . Subsequently , ∥∥QSX > S ∥∥ ≈ ∥∥QS′X > S′∥∥ and VS ≥ VS′ suggest that the first difference term ( and thus the overall expression ) is likely negative . We empirically verify in Fig . 1 that VS ≥ VS′ =⇒ biasS ≤ biasS′ holds for more than 80 % of times .

Summary: This paper intends to design a method to evaluate the value of a dataset. Such methods have been previously developed by calculating the Shapley value of the dataset based on its effect on model accuracy. The authors argued that since the model accuracy depends on a validation data set, selection of the validation dataset highly affects the Shapley value of each party's dataset, and hence will be a source of disagreement. To overcome this issue the authors set the value of a dataset based on the determinant (volume) of the left Gram matrix of its input features. The value of duplicated data is removed by constructing a compressed version of the dataset that preserves its diversity. The authors refer to this as robustness, since the values are robust to duplications. The authors provide some propositions connecting volume with bias and MSE. This has been validated experimentally as well.