Robust and Fully-Dynamic Coreset for Continuous-and-Bounded Learning (With Outliers) Problems

1 Introduction . As the rapid increasing of data volume in this big data era , we often need to develop low-complexity ( e.g. , linear or even sublinear ) algorithms for machine learning tasks . Moreover , our dataset is often maintained in a dynamic environment so that we have to consider the issues like data insertion and deletion . For example , as mentioned in the recent article [ 20 ] , Ginart et al . discussed the scenario that some sensitive training data have to be deleted due to the reason of privacy preserving . Obviously , it is prohibitive to re-train our model when the training data is changed dynamically , if the data size is extremely large . To remedy these issues , a natural way is to construct a small-sized summary of the training data so that we can run existing algorithms on the summary rather than the whole data . Coreset [ 18 ] , which was originally studied in the community of computational geometry [ 1 ] , has become a widely used data summary for many large-scale machine learning problems [ 7 , 27 , 34 , 38 , 36 , 25 ] . As a succinct data compression technique , coreset also enjoys several nice properties . For instance , coreset is usually composable and thus can be applied in the environment like distributed computing [ 28 ] . Also , it is usually able to obtain small coresets for streaming algorithms [ 23 , 15 ] and fully-dynamic algorithms with data insertion and deletion [ 10 , 24 ] . ∗Part of this work was done when Zixiu Wang was an intern under the supervision of Yiwen Guo at ByteDance . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , the existing coreset construction methods are still far from being satisfactory in practice . A major bottleneck is that most of them are sensitive to outliers . We are aware that real-world dataset is usually noisy and may contain outliers ; note that the outliers can be located arbitrarily in the space and even a single outlier can significantly destroy the final machine learning result . A typical example is poisoning attack , where an adversarial attacker may inject several specially crafted samples into the training data which can make the decision boundary severely deviate and cause unexpected misclassification [ 5 ] . In the past decades , a number of algorithms have been proposed for solving optimization with outliers problems , like clustering [ 11 , 14 , 12 , 21 , 42 ] , regression [ 39 , 37 , 17 ] , and PCA [ 8 ] . To see why the existing coreset methods are sensitive to outliers , we can take the popular sampling based coreset framework [ 19 ] as an example . The framework needs to compute a “ sensitivity ” for each data item , which measures the importance degree of the data item to the whole data set ; however , it tends to assign high sensitivities to the points who are far from the majority of the data , that is , an outlier is likely to have a high sensitivity and thus has a high chance to be selected to the coreset . Obviously , the coreset obtained by this way is not pleasant since we expect to contain more inliers rather than outliers in the coreset . It is also more challenging to further construct a fully-dynamic robust coreset . The existing robust coreset construction methods [ 19 , 26 ] often rely on simple uniform sampling and are efficient only when the number of outliers is a constant factor of the input size ( we will discuss this issue in Section 3.1 ) . Note that other outlier-resistant data summary methods like [ 21 , 13 ] usually yield large approximation factors and are not easy to be maintained in a fully dynamic scenario , to our knowledge . 1.1 Our Contributions . In this paper , we propose a unified fully-dynamic robust coreset framework for a class of optimization problems which is termed continuous-and-bounded ( CnB ) learning . This type of learning problems covers a broad range of optimization objectives in machine learning [ 41 , Chapter 12.2.2 ] . Roughly speaking , “ CnB learning ” requires that the optimization objective is a continuous function ( e.g. , smooth or Lipschitz ) , and meanwhile the solution is restricted within a bounded region . We emphasize that this “ bounded ” assumption is quite natural in real machine learning scenarios . To shed some light , we can consider running an iterative algorithm ( e.g. , the popular gradient descent or expectation maximization ) for optimizing some objective ; the solution is always restricted within a local region except for the first few rounds . Moreover , it is also reasonable to bound the solution range in a dynamic environment because one single update ( insertion or deletion ) is not likely to dramatically change the solution . Our coreset construction is a novel hybrid framework . First , we suppose that there exists an ordinary coreset construction method A for the given CnB optimization objective ( without considering outliers ) . Our key idea is to classify the input data into two parts : the “ suspected ” inliers and the “ suspected ” outliers , where the ratio of the sizes of these two parts is a carefully designed parameter λ . For the “ suspected ” inliers , we run the method A ( as a black box ) ; for the “ suspected ” outliers , we directly take a small sample uniformly at random ; finally , we prove that these two parts together yield a robust coreset . Our framework can be also efficiently implemented under the mergeand-reduce framework for dynamic setting ( though the original merge-and-reduce framework is not designed for the case with outliers ) [ 4 , 23 ] . A cute feature of our framework is that we can easily tune the parameter λ for updating our coreset dynamically , if the fraction of outliers is changed in the dynamic environment . The other contribution of this paper is that we propose two different coreset construction methods for CnB optimization objectives ( i.e. , the aforementioned black box A ) . The first method is based on the importance sampling framework [ 19 ] , and the second one is based on a space partition idea . Our coreset sizes depend on the doubling dimension of the solution space rather than the VC ( shattering ) dimension . This property is particularly useful if the VC dimension is too high or not easy to compute , or considering the scenarios like sparse optimization ( the domain of the solution vector has a low doubling dimension ) . To our knowledge , the only existing coreset construction methods that depend on doubling dimension are from Huang et al . [ 26 ] and Cohen-Addad et al . [ 16 ] , but their results are only for clustering problems . Our methods can be applied for a broad range of widely studied optimization objectives , such as logistic regression [ 38 ] , Bregman clustering [ 2 ] , and truth discovery [ 31 ] . It is worth noting that although some coreset construction methods for them have been proposed before ( e.g. , [ 33 , 38 , 43 , 17 , 25 ] ) , they are all problem-dependent and we are the first , to the best of our knowledge , to study them from a unified “ CnB ” perspective . 2 Preliminaries . We introduce several important notations used throughout this paper . Suppose P is the parameter space . Let X be the input data set that contains n items in a metric space X , and each x ∈ X has a weight w ( x ) ≥ 0 . Further , we use ( X , z ) to denote a given instance X with z outliers . We always use | · | and J·K respectively to denote the number of data items and the total weight of a given data set . We consider the learning problem whose objective function is the weighted sum of the cost over X , i.e. , f ( θ , X ) : = ∑ x∈X w ( x ) f ( θ , x ) , ( 1 ) where f ( θ , x ) is the non-negative cost contributed by x with the parameter vector θ ∈ P . The goal is to find an appropriate θ so that the objective function f ( θ , X ) is minimized . Usually we assume each x ∈ X has unit weight ( i.e. , w ( x ) = 1 ) , and it is straightforward to extend our method to weighted case . Given the pre-specified number z ∈ Z+ of outliers in X ( for weighted case , “ z ” refers to the total weight of outliers ) , we then define the “ robust ” objective function : fz ( θ , X ) : = min O⊂X , JOK=z f ( θ , X\O ) . ( 2 ) Actually , the above definition ( 2 ) comes from the popular “ trimming ” idea [ 39 ] that has been widely used for robust optimization problems . Below we present the formal definition of continuous-and-bound learning problem . A function g : P → R is α-Lipschitz continuous if for any θ1 , θ2 ∈ P , |g ( θ1 ) − g ( θ2 ) | ≤ α∥∆θ∥ , where ∆θ = θ1 − θ2 and ∥ · ∥ is some specified norm in P . Definition 1 ( Continuous-and-Bounded ( CnB ) Learning [ 41 ] ) . Let α , ℓ > 0 , and θ̃ ∈ P . Denote by B ( θ̃ , ℓ ) the ball centered at θ̃ with radius ℓ in the parameter space P . An objective ( 1 ) is called a CnB learning problem with the parameters ( α , ℓ , θ̃ ) if ( i ) the loss function f ( · , x ) is α-Lipschitz continuous for any x ∈ X , and ( ii ) θ is always restricted within B ( θ̃ , ℓ ) . Remark 1 . We can also consider other variants for CnB learning with replacing the “ α-Lipschitz continuous ” assumption . For example , a differentiable function g is “ α-Lipschitz continuous gradient ” if its gradient ∇g is α-Lipschitz continuous ( it is also called “ α-smooth ” ) . Similarly , a twice-differentiable function g is “ α-Lipschitz continuous Hessian ” if its Hessian matrix ∇2g is α-Lipschitz continuous . In this paper , we mainly focus the problems under the “ α-Lipschitz continuous ” assumption , and our analysis can be also applied to these two variants via slight modifications . We define the coreset for CnB learning problems below . Definition 2 ( ε-coreset ) . Let ε > 0 . Given a dataset X ⊂ X and the objective function f ( θ , X ) , we say that a weighted set C ⊂ X is an ε-coreset of X if for any θ ∈ B ( θ̃ , ℓ ) , we have |f ( θ , C ) − f ( θ , X ) | ≤ εf ( θ , X ) . ( 3 ) If C is an ε-coreset of X , we can run an existing optimization algorithm on C so as to obtain an approximate solution . Obviously , we expect that the size of C to be as small as possible . Following Definition 2 , we also define the corresponding robust coreset ( the similar definition was also introduced in [ 19 , 26 ] before ) . Definition 3 ( robust coreset ) . Let ε > 0 , and β ∈ [ 0 , 1 ) . Given the dataset X ⊂ X and the objective function f ( θ , x ) , we say that a weighted dataset C ⊂ X is a ( β , ε ) -robust coreset of X if for any θ ∈ B ( θ̃ , ℓ ) , we have ( 1− ε ) f ( 1+β ) z ( θ , X ) ≤ fz ( θ , C ) ≤ ( 1 + ε ) f ( 1−β ) z ( θ , X ) . ( 4 ) Roughly speaking , if we obtain an approximate solution θ′ ∈ P on C , its quality can be preserved on the original input data X . The parameter β indicates the error on the number of outliers if using θ′ as our solution on X . If we set β = 0 , that means we allow no error on the number of outliers . In our full paper [ 44 ] , we present our detailed discussion on the quality loss ( in terms of the objective value and the number of outliers ) of this transformation from C to X . The rest of this paper is organized as follows . In Section 3 , we introduce our robust coreset framework and show how to implement it in a fully-dynamic environment . In Section 4 , we propose two different ordinary coreset ( without outliers ) construction methods for CnB learning problems , which can be used as the black box in our robust coreset framework of Section 3 . Finally , in Section 5 we illustrate the application of our coreset method in practice . Due to the space limit , some proofs and the more detailed experimental results are placed in our full paper [ 44 ] .

Summary: The goal of this paper is to investigate a coreset approach for robust supervised learning specifically with outliers. The authors study this for logistic regression and k-means clustering. This paper is mainly theoretical and the empirical results are mainly in the appendix.