Faster Algorithms and Constant Lower Bounds for the Worst-Case Expected Error

The study of statistical estimation without distributional assumptions on data values , but with knowledge of data collection methods was recently introduced by Chen , Valiant and Valiant ( NeurIPS 2020 ) . In this framework , the goal is to design estimators that minimize the worst-case expected error . Here the expectation is over a known , randomized data collection process from some population , and the data values corresponding to each element of the population are assumed to be worstcase . Chen , Valiant and Valiant show that , when data values are ` ∞-normalized , there is a polynomial time algorithm to compute an estimator for the mean with worst-case expected error that is within a factor π2 of the optimum within the natural class of semilinear estimators . However , their algorithm is based on optimizing a somewhat complex concave objective function over a constrained set of positive semidefinite matrices , and thus does not come with explicit runtime guarantees beyond being polynomial time in the input . In this paper we design provably efficient algorithms for approximating the optimal semilinear estimator based on online convex optimization . In the setting where data values are ` ∞-normalized , our algorithm achieves a π2 -approximation by iteratively solving a sequence of standard SDPs . When data values are ` 2-normalized , our algorithm iteratively computes the top eigenvector of a sequence of matrices , and does not lose any multiplicative approximation factor . Further , using experiments in settings where sample membership is correlated with data values ( e.g . `` importance sampling '' and `` snowball sampling '' ) , we show that our ` 2-normalized algorithm gives a similar advantage over standard estimators as the original ` ∞-normalized algorithm of Chen , Valiant and Valiant , but with much lower computational complexity . We complement these positive results by stating a simple combinatorial condition which , if satisfied by a data collection process , implies that any ( not necessarily semilinear ) estimator for the mean has constant worst-case expected error . 1 Introduction . Standard methods in statistical analysis are often based on the assumption that data values are drawn independently from some underlying distribution , and may perform poorly when this assumption is violated . Methods from robust statistics often modify this assumption , for example by allowing a small fraction of data values to be arbitrary outliers while the bulk of the data values remain independent samples . However , there are natural settings in which data values may be strongly correlated to sample membership , so that distributional assumptions ( e.g . independence of data values ) can lead to inaccurate results . Furthermore , inaccuracy of predictions in such settings can have serious societal consequences–for example predictions of election outcomes or the spread of a contagious disease . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Recently , Chen , Valiant , and Valiant [ 4 ] introduced a framework to capture statistical estimation in such difficult settings , by assuming that the data values are worst-case , but that the estimation algorithm is able to leverage knowledge of the randomized data collection process . The framework is simple to describe . There is a set { 1 , . . . n } of n indices with corresponding data values x = { x1 , . . . xn } along with a probability distribution P over pairs of subsets A , B ⊆ { 1 , . . . n } . Here A is called the sample set and B is called the target set . A sample ( A , B ) is drawn from P and the values xA = { xi | i ∈ A } are revealed . The goal is then to estimate the value of some function g ( xB ) given only A , B and xA . The quality of an estimator in this framework is measured by its worst-case expected error . Here the data values x are worst-case and the expectation is with respect to the distribution P . The worst-case expected error framework captures several natural settings where data values may be correlated to sample membership . Importance sampling is the setting where the target set B is always equal to the whole population { 1 , . . . n } , and the sample set A is chosen by independently sampling each element i ∈ { 1 , . . . n } with probability pi . In the worst-case expected error framework , the values xi may be arbitrarily correlated to the probability pi of being sampled . In snowball sampling [ 12 ] , the elements of { 1 , . . . n } correspond to the vertices of a graph ( e.g . a social network ) . A sample set A is chosen by first picking a few initial vertices . Next , each chosen vertex recruits a randomly chosen subset of its neighbors into the sample , and this process is repeated until a desired sample size is reached . The target set B can be either the whole population or a subset given by a few additional iterations of the recruitment process . Here it is natural to assume that the data values xi at neighboring vertices will be highly correlated . Finally , selective prediction [ 10 , 18 ] is a temporal sampling method where the population { 1 , . . . n } corresponds to time steps , and the goal is to predict the average of future data values given the past ( e.g . the average change in the stock market ) . The target set B is some time window { t , . . . t+ w } and the sample A is { 1 , . . . t } . If the random process for choosing the starting point t and length w of the time window is chosen appropriately , sub-constant error is attainable even when the xi are chosen to be worst-case values bounded by a constant [ 10 , 18 ] . In [ 4 ] , the authors design an algorithm for computing estimators for the mean of the target set B . In particular , they restrict their attention to the class of semilinear estimators i.e . estimators which are restricted to be a linear combination of the sample values xA , but where the weights of the linear combination may depend arbitrarily on P , A , and B . In the setting where the data values are bounded , the authors design an algorithm that outputs a semilinear estimator with worst-case expected error at most a π2 factor larger than that of the optimal semilinear estimator . They then demonstrate that the estimator output by their algorithm has significantly lower expected error than many standard estimators in the importance sampling , snowball sampling , and selective prediction settings where data values are correlated to sample membership . However , several open questions remain , even for the case of computing the optimal semilinear estimator for the mean . First , the algorithm in [ 4 ] is a concave maximization over a constrained set of positive semidefinite matrices , which is solved using a general purpose convex programming solver . In particular this means that no explicit bounds on the runtime of the algorithm are given beyond being polynomial in n. This leads us to ask : Question 1 : Is there a simpler algorithm to compute a semilinear estimator with approximately optimal worst-case expected error that comes with explicit runtime guarantees ? Second , the choice of ` ∞-normalization ( i.e . assuming the data values are bounded ) is natural for some settings such as polling or testing for infectious disease . However , ` 2-normalization ( i.e . 1 n ∑ i x 2 i ≤ 1 ) is also a natural choice for settings where some data values may be much larger than others such as predicting stock prices . This motivates : Question 2 : Is there an efficient algorithm to compute a semilinear estimator with approximately optimal worst-case expected error in the ` 2-bounded case ? Finally , the setting of the worst-case expected error is quite challenging , and so one would expect that for many randomized data collection processes it is not possible to achieve non-trivial worst-case expected error . Thus , one might ask : Question 3 : Are there simple properties of a data collection process that ensure that no estimator can achieve sub-constant worst-case expected error ? 1.1 Our Results . We answer the first two questions by designing simple algorithms based on online gradient descent for approximating the optimal semilinear estimator in both the ` ∞-bounded and the ` 2-bounded case . To be able to describe our algorithms we first formally define the input . Definition 1.1 ( Definition 2 in [ 4 ] ) . A joint sample-target distribution P over { 1 , . . . n } is given by the uniform distribution over m pairs ( Ai , Bi ) where Ai , Bi ⊆ { 1 , . . . n } . Because any distribution P on pairs ( A , B ) may be approximated to arbitrary accuracy by a uniform distribution this is a reasonable choice of parametrization . In many settings the m uniform pairs are obtained by sampling sufficiently many times from the known randomized data-collection process ( see supplementary material for details ) . Thus , one should think of m as being much larger than n , say at least Ω ( n2 ) . Our algorithms ’ runtimes also depend on a parameter ρ which we define to be n times the worst-case expected error of the optimal semilinear estimator . Theorem 1.2 ( Informal–see Theorem 2.3 ) . There is an algorithm based on online gradient descent for approximating the optimal semilinear estimator to within a multiplicative π2 and additive error for ` ∞-bounded values . The algorithm runs in O ( ρ2 log ρ ) iterations each taking Õ ( mnω−1 + n7/2 ) time , where ω is the current matrix multiplication exponent . Note that when there exists a semilinear estimator achieving worst-case expected error O ( 1n ) , the algorithm requires O ( 1 ) iterations . Each iteration of the algorithm requires solving a standard SDP in n dimensions with n constraints ( in fact the only constraints are that the diagonal entries of the psd matrix are constrained to all be equal to one ) . Theorem 1.3 ( Informal–see Theorem 2.4 ) . There is an algorithm based on online gradient descent for approximating the optimal semilinear estimator to within an additive error for ` 2-bounded values . The algorithm runs in O ( ρ2 log ρ ) iterations each taking O ( mn ) time . In addition to the improved runtime , each iteration of the algorithm for ` 2-bounded values only requires computing the eigenvector corresponding to the largest eigenvalue of an n × n matrix , making the algorithm practical and simple to implement . We provide the details of both algorithms in Section 2 . Recall that the SDP algorithm from [ 4 ] obtained lower expected error than several standard estimators on various synthetic datasets where data values are correlated to sample membership . To compare to these empirical results obtained for the ` ∞-bounded case , we implement our algorithm for ` 2-bounded values and perform experiments on the same synthetic datasets from [ 4 ] . We find that the expected error of our algorithm on these synthetic datasets matches that of the SDP algorithm from [ 4 ] , while being simpler to implement and having lower computational complexity . See Section 5 for details . Finally , in Section 4 we answer the third question by providing a simple combinatorial condition for a sample-target distribution P , which ensures that any ( not necessarily semilinear ) estimator has constant worst-case expected error .

Summary: The authors build on the work of Chen, Valiant and Valiant from NeurIPS 2020 on estimators minimizing the worst-case expected error by providing two new algorithms for the same problem formulation, but with explicit runtime guarantees. The first algorithm uses a standard semidefinite program within a gradient descent loop to arrive at a runtime that is (with some abusive simplification) $\tilde O(mn^5 \log n/\epsilon^6)$. The second relaxes the setting so that data values are bounded in $\ell_2$ norm (rather than $\ell_\infty$), and uses this new setting to reduce the runtime to $O(mn^3 \log n/\epsilon^3)$. As far as I can tell, these algorithms don't offer any guarantees on sample complexity, but the newly explicit runtime guarantee is (subject to questions below) an improvement. As an impossibility result, the authors provide a combinatorial condition on the sample-target distribution under which the worst-case expected error for all estimators is at least a constant. Finally, the authors verify their new algorithm experimentally, showing comparable performance to [4] but faster runtime.