Robust and Fully-Dynamic Coreset for Continuous-and-Bounded Learning (With Outliers) Problems

1 Introduction . As the rapid increasing of data volume in this big data era , we often need to develop low-complexity ( e.g. , linear or even sublinear ) algorithms for machine learning tasks . Moreover , our dataset is often maintained in a dynamic environment so that we have to consider the issues like data insertion and deletion . For example , as mentioned in the recent article [ 20 ] , Ginart et al . discussed the scenario that some sensitive training data have to be deleted due to the reason of privacy preserving . Obviously , it is prohibitive to re-train our model when the training data is changed dynamically , if the data size is extremely large . To remedy these issues , a natural way is to construct a small-sized summary of the training data so that we can run existing algorithms on the summary rather than the whole data . Coreset [ 18 ] , which was originally studied in the community of computational geometry [ 1 ] , has become a widely used data summary for many large-scale machine learning problems [ 7 , 27 , 34 , 38 , 36 , 25 ] . As a succinct data compression technique , coreset also enjoys several nice properties . For instance , coreset is usually composable and thus can be applied in the environment like distributed computing [ 28 ] . Also , it is usually able to obtain small coresets for streaming algorithms [ 23 , 15 ] and fully-dynamic algorithms with data insertion and deletion [ 10 , 24 ] . ∗Part of this work was done when Zixiu Wang was an intern under the supervision of Yiwen Guo at ByteDance . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , the existing coreset construction methods are still far from being satisfactory in practice . A major bottleneck is that most of them are sensitive to outliers . We are aware that real-world dataset is usually noisy and may contain outliers ; note that the outliers can be located arbitrarily in the space and even a single outlier can significantly destroy the final machine learning result . A typical example is poisoning attack , where an adversarial attacker may inject several specially crafted samples into the training data which can make the decision boundary severely deviate and cause unexpected misclassification [ 5 ] . In the past decades , a number of algorithms have been proposed for solving optimization with outliers problems , like clustering [ 11 , 14 , 12 , 21 , 42 ] , regression [ 39 , 37 , 17 ] , and PCA [ 8 ] . To see why the existing coreset methods are sensitive to outliers , we can take the popular sampling based coreset framework [ 19 ] as an example . The framework needs to compute a “ sensitivity ” for each data item , which measures the importance degree of the data item to the whole data set ; however , it tends to assign high sensitivities to the points who are far from the majority of the data , that is , an outlier is likely to have a high sensitivity and thus has a high chance to be selected to the coreset . Obviously , the coreset obtained by this way is not pleasant since we expect to contain more inliers rather than outliers in the coreset . It is also more challenging to further construct a fully-dynamic robust coreset . The existing robust coreset construction methods [ 19 , 26 ] often rely on simple uniform sampling and are efficient only when the number of outliers is a constant factor of the input size ( we will discuss this issue in Section 3.1 ) . Note that other outlier-resistant data summary methods like [ 21 , 13 ] usually yield large approximation factors and are not easy to be maintained in a fully dynamic scenario , to our knowledge . 1.1 Our Contributions . In this paper , we propose a unified fully-dynamic robust coreset framework for a class of optimization problems which is termed continuous-and-bounded ( CnB ) learning . This type of learning problems covers a broad range of optimization objectives in machine learning [ 41 , Chapter 12.2.2 ] . Roughly speaking , “ CnB learning ” requires that the optimization objective is a continuous function ( e.g. , smooth or Lipschitz ) , and meanwhile the solution is restricted within a bounded region . We emphasize that this “ bounded ” assumption is quite natural in real machine learning scenarios . To shed some light , we can consider running an iterative algorithm ( e.g. , the popular gradient descent or expectation maximization ) for optimizing some objective ; the solution is always restricted within a local region except for the first few rounds . Moreover , it is also reasonable to bound the solution range in a dynamic environment because one single update ( insertion or deletion ) is not likely to dramatically change the solution . Our coreset construction is a novel hybrid framework . First , we suppose that there exists an ordinary coreset construction method A for the given CnB optimization objective ( without considering outliers ) . Our key idea is to classify the input data into two parts : the “ suspected ” inliers and the “ suspected ” outliers , where the ratio of the sizes of these two parts is a carefully designed parameter λ . For the “ suspected ” inliers , we run the method A ( as a black box ) ; for the “ suspected ” outliers , we directly take a small sample uniformly at random ; finally , we prove that these two parts together yield a robust coreset . Our framework can be also efficiently implemented under the mergeand-reduce framework for dynamic setting ( though the original merge-and-reduce framework is not designed for the case with outliers ) [ 4 , 23 ] . A cute feature of our framework is that we can easily tune the parameter λ for updating our coreset dynamically , if the fraction of outliers is changed in the dynamic environment . The other contribution of this paper is that we propose two different coreset construction methods for CnB optimization objectives ( i.e. , the aforementioned black box A ) . The first method is based on the importance sampling framework [ 19 ] , and the second one is based on a space partition idea . Our coreset sizes depend on the doubling dimension of the solution space rather than the VC ( shattering ) dimension . This property is particularly useful if the VC dimension is too high or not easy to compute , or considering the scenarios like sparse optimization ( the domain of the solution vector has a low doubling dimension ) . To our knowledge , the only existing coreset construction methods that depend on doubling dimension are from Huang et al . [ 26 ] and Cohen-Addad et al . [ 16 ] , but their results are only for clustering problems . Our methods can be applied for a broad range of widely studied optimization objectives , such as logistic regression [ 38 ] , Bregman clustering [ 2 ] , and truth discovery [ 31 ] . It is worth noting that although some coreset construction methods for them have been proposed before ( e.g. , [ 33 , 38 , 43 , 17 , 25 ] ) , they are all problem-dependent and we are the first , to the best of our knowledge , to study them from a unified “ CnB ” perspective . 2 Preliminaries . We introduce several important notations used throughout this paper . Suppose P is the parameter space . Let X be the input data set that contains n items in a metric space X , and each x ∈ X has a weight w ( x ) ≥ 0 . Further , we use ( X , z ) to denote a given instance X with z outliers . We always use | · | and J·K respectively to denote the number of data items and the total weight of a given data set . We consider the learning problem whose objective function is the weighted sum of the cost over X , i.e. , f ( θ , X ) : = ∑ x∈X w ( x ) f ( θ , x ) , ( 1 ) where f ( θ , x ) is the non-negative cost contributed by x with the parameter vector θ ∈ P . The goal is to find an appropriate θ so that the objective function f ( θ , X ) is minimized . Usually we assume each x ∈ X has unit weight ( i.e. , w ( x ) = 1 ) , and it is straightforward to extend our method to weighted case . Given the pre-specified number z ∈ Z+ of outliers in X ( for weighted case , “ z ” refers to the total weight of outliers ) , we then define the “ robust ” objective function : fz ( θ , X ) : = min O⊂X , JOK=z f ( θ , X\O ) . ( 2 ) Actually , the above definition ( 2 ) comes from the popular “ trimming ” idea [ 39 ] that has been widely used for robust optimization problems . Below we present the formal definition of continuous-and-bound learning problem . A function g : P → R is α-Lipschitz continuous if for any θ1 , θ2 ∈ P , |g ( θ1 ) − g ( θ2 ) | ≤ α∥∆θ∥ , where ∆θ = θ1 − θ2 and ∥ · ∥ is some specified norm in P . Definition 1 ( Continuous-and-Bounded ( CnB ) Learning [ 41 ] ) . Let α , ℓ > 0 , and θ̃ ∈ P . Denote by B ( θ̃ , ℓ ) the ball centered at θ̃ with radius ℓ in the parameter space P . An objective ( 1 ) is called a CnB learning problem with the parameters ( α , ℓ , θ̃ ) if ( i ) the loss function f ( · , x ) is α-Lipschitz continuous for any x ∈ X , and ( ii ) θ is always restricted within B ( θ̃ , ℓ ) . Remark 1 . We can also consider other variants for CnB learning with replacing the “ α-Lipschitz continuous ” assumption . For example , a differentiable function g is “ α-Lipschitz continuous gradient ” if its gradient ∇g is α-Lipschitz continuous ( it is also called “ α-smooth ” ) . Similarly , a twice-differentiable function g is “ α-Lipschitz continuous Hessian ” if its Hessian matrix ∇2g is α-Lipschitz continuous . In this paper , we mainly focus the problems under the “ α-Lipschitz continuous ” assumption , and our analysis can be also applied to these two variants via slight modifications . We define the coreset for CnB learning problems below . Definition 2 ( ε-coreset ) . Let ε > 0 . Given a dataset X ⊂ X and the objective function f ( θ , X ) , we say that a weighted set C ⊂ X is an ε-coreset of X if for any θ ∈ B ( θ̃ , ℓ ) , we have |f ( θ , C ) − f ( θ , X ) | ≤ εf ( θ , X ) . ( 3 ) If C is an ε-coreset of X , we can run an existing optimization algorithm on C so as to obtain an approximate solution . Obviously , we expect that the size of C to be as small as possible . Following Definition 2 , we also define the corresponding robust coreset ( the similar definition was also introduced in [ 19 , 26 ] before ) . Definition 3 ( robust coreset ) . Let ε > 0 , and β ∈ [ 0 , 1 ) . Given the dataset X ⊂ X and the objective function f ( θ , x ) , we say that a weighted dataset C ⊂ X is a ( β , ε ) -robust coreset of X if for any θ ∈ B ( θ̃ , ℓ ) , we have ( 1− ε ) f ( 1+β ) z ( θ , X ) ≤ fz ( θ , C ) ≤ ( 1 + ε ) f ( 1−β ) z ( θ , X ) . ( 4 ) Roughly speaking , if we obtain an approximate solution θ′ ∈ P on C , its quality can be preserved on the original input data X . The parameter β indicates the error on the number of outliers if using θ′ as our solution on X . If we set β = 0 , that means we allow no error on the number of outliers . In our full paper [ 44 ] , we present our detailed discussion on the quality loss ( in terms of the objective value and the number of outliers ) of this transformation from C to X . The rest of this paper is organized as follows . In Section 3 , we introduce our robust coreset framework and show how to implement it in a fully-dynamic environment . In Section 4 , we propose two different ordinary coreset ( without outliers ) construction methods for CnB learning problems , which can be used as the black box in our robust coreset framework of Section 3 . Finally , in Section 5 we illustrate the application of our coreset method in practice . Due to the space limit , some proofs and the more detailed experimental results are placed in our full paper [ 44 ] .

Summary: This paper studies coresets for continuous-and-bounded learning problems. In the optimization version of the problem, given a weighted input data set X with weight w : X \to \mathbb{R}_+, the goal is to find a parameter \theta in some parameter space P such that \sum_{x \in X} w(x) \cdot f(\theta, x) is minimized where f : P \times X \to \mathbb{R}_+ is a cost function. Furthermore, the continuous-and-boundedness condition requires that the parameter space P form a metric space (so there is an implicit associated distance function on the parameters), and that a) there exist l, \alpha > 0, such that all parameters in P are contained in a ball of radius l, and b) f(\cdot, x) is \alpha-Lipschitz.  An \epsilon-coreset for this problem is defined as a weighted subset S \subseteq X of the data set with weight u : S \to \mathbb{R}_+, such that for every parameter \theta \in P, the total cost on S is close to that on X, i.e., \sum_{x \in S} u(x) \cdot f(\theta, x) \in (1 \pm \epsilon) \sum_{x \in X} w(x) \cdot f(\theta, x).   The paper also studies coresets for the outlier version of the problem, where we are additionally given a parameter z (the number of outliers) and the cost becomes f_z(\theta, X) = \min_{O \subset X, |O| = z} f(\theta, X \setminus O). Coresets are defined similarly for the outlier version, except that this paper allows bi-criteria guarantee: a (\beta, \epsilon)-coreset preserves the cost for every \theta, such that the cost is within [f_{(1+\beta)z}(\theta, X), f_{(1-\beta)z}(\theta,X)] (the exact definition is subject to proper normalization).  Several results are presented. For the outlier version, the first result shows that uniform sampling yields a (\beta, 0)-coreset of size \tilde{O}(n / (\beta z))^2, provided that the VC-dimension for the (metric ball) range space induced by {f(\cdot, x) : x} is bounded. This result is weak and is inadequate for small z (even for moderately small z = O(\sqrt{n})). The second result is more interesting and it gives a reduction that reduces the outlier version to the non-outlier version. Intuitively, it “glues” a black-box coreset for the non-outlier version and a uniform sampling based coreset. Specifically, it yields a (\beta, \epsilon) coreset of size T + min{ O((\beta\epsilon)^{-2} vcdim), O(z / epsilon)}, where T is the size of some \epsilon-coreset in the non-outlier setting, and vcdim is the VC-dimension for the range space induced by {f(\cdot, x) : x}. The paper also gives a fully-dynamic algorithm that maintains the “glued” coreset.  Besides, two coreset results for the non-outlier version are presented. The first one is based on the well-known importance sampling framework [11], but instead of analyzing the VC-dimension of the range space, this paper shows that the VC-dimension term could be replaced by the doubling dimension of the parameter space P. However, the authors were not able to give an upper bound for the total sensitivity, and therefore this first result does not yield an explicit coreset size bound. Then the other result is based on a spatial partition method proposed in [9], which helps to gets rid of the total sensitivity bound, but it introduces a log n factor.