Title: Data Reliability Scoring
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Reliability Scoring, Gram Determinant Score, Information Elicitation, Mechanism Design

Abstract: How can we assess the reliability of a dataset without access to ground truth? We introduce the problem of reliability scoring for datasets collected from potentially strategic sources. The true data are unobserved, but we see outcomes of an unknown statistical experiment that depends on them. To benchmark reliability, we define ground-truth–based orderings that capture how much reported data deviate from the truth. We then propose the Gram determinant score, which measures the volume spanned by vectors describing the empirical distribution of the observed data and experiment outcomes. We show that this score preserves several ground-truth-based reliability orderings and, uniquely up to scaling, yields the same reliability ranking of datasets regardless of the experiment -- a property we call experiment agnostic. Experiments on synthetic noise models, CIFAR-10 embeddings, and real employment data demonstrate that the Gram determinant score effectively captures data quality across diverse observation processes.

TL;DR: We propose the Gram determinant score, an experiment-agnostic method for reliability scoring that assesses dataset quality without access to ground truth.