Faster Algorithms and Constant Lower Bounds for the Worst-Case Expected Error

The study of statistical estimation without distributional assumptions on data values , but with knowledge of data collection methods was recently introduced by Chen , Valiant and Valiant ( NeurIPS 2020 ) . In this framework , the goal is to design estimators that minimize the worst-case expected error . Here the expectation is over a known , randomized data collection process from some population , and the data values corresponding to each element of the population are assumed to be worstcase . Chen , Valiant and Valiant show that , when data values are ` ∞-normalized , there is a polynomial time algorithm to compute an estimator for the mean with worst-case expected error that is within a factor π2 of the optimum within the natural class of semilinear estimators . However , their algorithm is based on optimizing a somewhat complex concave objective function over a constrained set of positive semidefinite matrices , and thus does not come with explicit runtime guarantees beyond being polynomial time in the input . In this paper we design provably efficient algorithms for approximating the optimal semilinear estimator based on online convex optimization . In the setting where data values are ` ∞-normalized , our algorithm achieves a π2 -approximation by iteratively solving a sequence of standard SDPs . When data values are ` 2-normalized , our algorithm iteratively computes the top eigenvector of a sequence of matrices , and does not lose any multiplicative approximation factor . Further , using experiments in settings where sample membership is correlated with data values ( e.g . `` importance sampling '' and `` snowball sampling '' ) , we show that our ` 2-normalized algorithm gives a similar advantage over standard estimators as the original ` ∞-normalized algorithm of Chen , Valiant and Valiant , but with much lower computational complexity . We complement these positive results by stating a simple combinatorial condition which , if satisfied by a data collection process , implies that any ( not necessarily semilinear ) estimator for the mean has constant worst-case expected error . 1 Introduction . Standard methods in statistical analysis are often based on the assumption that data values are drawn independently from some underlying distribution , and may perform poorly when this assumption is violated . Methods from robust statistics often modify this assumption , for example by allowing a small fraction of data values to be arbitrary outliers while the bulk of the data values remain independent samples . However , there are natural settings in which data values may be strongly correlated to sample membership , so that distributional assumptions ( e.g . independence of data values ) can lead to inaccurate results . Furthermore , inaccuracy of predictions in such settings can have serious societal consequences–for example predictions of election outcomes or the spread of a contagious disease . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Recently , Chen , Valiant , and Valiant [ 4 ] introduced a framework to capture statistical estimation in such difficult settings , by assuming that the data values are worst-case , but that the estimation algorithm is able to leverage knowledge of the randomized data collection process . The framework is simple to describe . There is a set { 1 , . . . n } of n indices with corresponding data values x = { x1 , . . . xn } along with a probability distribution P over pairs of subsets A , B ⊆ { 1 , . . . n } . Here A is called the sample set and B is called the target set . A sample ( A , B ) is drawn from P and the values xA = { xi | i ∈ A } are revealed . The goal is then to estimate the value of some function g ( xB ) given only A , B and xA . The quality of an estimator in this framework is measured by its worst-case expected error . Here the data values x are worst-case and the expectation is with respect to the distribution P . The worst-case expected error framework captures several natural settings where data values may be correlated to sample membership . Importance sampling is the setting where the target set B is always equal to the whole population { 1 , . . . n } , and the sample set A is chosen by independently sampling each element i ∈ { 1 , . . . n } with probability pi . In the worst-case expected error framework , the values xi may be arbitrarily correlated to the probability pi of being sampled . In snowball sampling [ 12 ] , the elements of { 1 , . . . n } correspond to the vertices of a graph ( e.g . a social network ) . A sample set A is chosen by first picking a few initial vertices . Next , each chosen vertex recruits a randomly chosen subset of its neighbors into the sample , and this process is repeated until a desired sample size is reached . The target set B can be either the whole population or a subset given by a few additional iterations of the recruitment process . Here it is natural to assume that the data values xi at neighboring vertices will be highly correlated . Finally , selective prediction [ 10 , 18 ] is a temporal sampling method where the population { 1 , . . . n } corresponds to time steps , and the goal is to predict the average of future data values given the past ( e.g . the average change in the stock market ) . The target set B is some time window { t , . . . t+ w } and the sample A is { 1 , . . . t } . If the random process for choosing the starting point t and length w of the time window is chosen appropriately , sub-constant error is attainable even when the xi are chosen to be worst-case values bounded by a constant [ 10 , 18 ] . In [ 4 ] , the authors design an algorithm for computing estimators for the mean of the target set B . In particular , they restrict their attention to the class of semilinear estimators i.e . estimators which are restricted to be a linear combination of the sample values xA , but where the weights of the linear combination may depend arbitrarily on P , A , and B . In the setting where the data values are bounded , the authors design an algorithm that outputs a semilinear estimator with worst-case expected error at most a π2 factor larger than that of the optimal semilinear estimator . They then demonstrate that the estimator output by their algorithm has significantly lower expected error than many standard estimators in the importance sampling , snowball sampling , and selective prediction settings where data values are correlated to sample membership . However , several open questions remain , even for the case of computing the optimal semilinear estimator for the mean . First , the algorithm in [ 4 ] is a concave maximization over a constrained set of positive semidefinite matrices , which is solved using a general purpose convex programming solver . In particular this means that no explicit bounds on the runtime of the algorithm are given beyond being polynomial in n. This leads us to ask : Question 1 : Is there a simpler algorithm to compute a semilinear estimator with approximately optimal worst-case expected error that comes with explicit runtime guarantees ? Second , the choice of ` ∞-normalization ( i.e . assuming the data values are bounded ) is natural for some settings such as polling or testing for infectious disease . However , ` 2-normalization ( i.e . 1 n ∑ i x 2 i ≤ 1 ) is also a natural choice for settings where some data values may be much larger than others such as predicting stock prices . This motivates : Question 2 : Is there an efficient algorithm to compute a semilinear estimator with approximately optimal worst-case expected error in the ` 2-bounded case ? Finally , the setting of the worst-case expected error is quite challenging , and so one would expect that for many randomized data collection processes it is not possible to achieve non-trivial worst-case expected error . Thus , one might ask : Question 3 : Are there simple properties of a data collection process that ensure that no estimator can achieve sub-constant worst-case expected error ? 1.1 Our Results . We answer the first two questions by designing simple algorithms based on online gradient descent for approximating the optimal semilinear estimator in both the ` ∞-bounded and the ` 2-bounded case . To be able to describe our algorithms we first formally define the input . Definition 1.1 ( Definition 2 in [ 4 ] ) . A joint sample-target distribution P over { 1 , . . . n } is given by the uniform distribution over m pairs ( Ai , Bi ) where Ai , Bi ⊆ { 1 , . . . n } . Because any distribution P on pairs ( A , B ) may be approximated to arbitrary accuracy by a uniform distribution this is a reasonable choice of parametrization . In many settings the m uniform pairs are obtained by sampling sufficiently many times from the known randomized data-collection process ( see supplementary material for details ) . Thus , one should think of m as being much larger than n , say at least Ω ( n2 ) . Our algorithms ’ runtimes also depend on a parameter ρ which we define to be n times the worst-case expected error of the optimal semilinear estimator . Theorem 1.2 ( Informal–see Theorem 2.3 ) . There is an algorithm based on online gradient descent for approximating the optimal semilinear estimator to within a multiplicative π2 and additive error for ` ∞-bounded values . The algorithm runs in O ( ρ2 log ρ ) iterations each taking Õ ( mnω−1 + n7/2 ) time , where ω is the current matrix multiplication exponent . Note that when there exists a semilinear estimator achieving worst-case expected error O ( 1n ) , the algorithm requires O ( 1 ) iterations . Each iteration of the algorithm requires solving a standard SDP in n dimensions with n constraints ( in fact the only constraints are that the diagonal entries of the psd matrix are constrained to all be equal to one ) . Theorem 1.3 ( Informal–see Theorem 2.4 ) . There is an algorithm based on online gradient descent for approximating the optimal semilinear estimator to within an additive error for ` 2-bounded values . The algorithm runs in O ( ρ2 log ρ ) iterations each taking O ( mn ) time . In addition to the improved runtime , each iteration of the algorithm for ` 2-bounded values only requires computing the eigenvector corresponding to the largest eigenvalue of an n × n matrix , making the algorithm practical and simple to implement . We provide the details of both algorithms in Section 2 . Recall that the SDP algorithm from [ 4 ] obtained lower expected error than several standard estimators on various synthetic datasets where data values are correlated to sample membership . To compare to these empirical results obtained for the ` ∞-bounded case , we implement our algorithm for ` 2-bounded values and perform experiments on the same synthetic datasets from [ 4 ] . We find that the expected error of our algorithm on these synthetic datasets matches that of the SDP algorithm from [ 4 ] , while being simpler to implement and having lower computational complexity . See Section 5 for details . Finally , in Section 4 we answer the third question by providing a simple combinatorial condition for a sample-target distribution P , which ensures that any ( not necessarily semilinear ) estimator has constant worst-case expected error .

Summary: This paper studies the _worst-case expected error framework_ of statistical estimation proposed by Chen, Valiant, and Valiant, where data values are worst-case, but the estimation algorithm exploits randomness in the data collection process. In this worst-case expected error framework, there are $n$ data values $x_1$ to $x_n$ and a probability distribution $P$ over pairs of subsets $A, B \subseteq {1, \ldots, n}$, where the variables $x_A$ over the sample set $A$ are revealed, and we want to estimate the value of some fixed function $g(X_B)$ of the variables $x_B$ over the target set $B$. Restricting attention to _semilinear_ estimators, which are linear combinations of the sample values whose weights could depend on the sample-target distribution $P$ or the indices $x_A$ and $x_B$ of the sample and target variables, Chen, Valiant, and Valiant gave a complicated semilinear estimator for the mean of the target set $B$ which is at most $\pi/2$ factor larger than the optimal semilinear estimator, when the data values are $\ell_\infty$-bounded.  This paper presents the following three results: 1. a simpler algorithm to compute a semilinear estimator within a multiplicative $\pi/2$ factor and an additive $\epsilon$ factor for $\ell_\infty$-bounded values, with explicit efficient runtime guarantees. 2. an efficient algorithm to compute a semilinear estimator within an additive $\epsilon$ error for $\ell_2$-bounded values. 3. a family of lower bound instances where all estimators (not necessarily semilinear) must have at least constant error, which generalizes the trivial lower bounds of disjoint $A$ and $B$.