document: Scalable robust covariance and correlation estimates for data mining Covariance and correlation estimates have important applications in data mining . In the presence of outliers , classical estimates of covariance and correlation matrices are not reliable . A small fraction of outliers , in some cases even a single outlier , can distort the classical covariance and correlation estimates making them virtually useless . That is , correlations for the vast majority of the data can be very erroneously reported ; principal components transformations can be misleading ; and multidimensional outlier detection via Mahalanobis distances can fail to detect outliers . There is plenty of statistical literature on robust covariance and correlation matrix estimates with an emphasis on affine-equivariant estimators that possess high breakdown points and small worst case biases . All such estimators have unacceptable exponential complexity in the number of variables and quadratic complexity in the number of observations . In this paper we focus on several variants of robust covariance and correlation matrix estimates with quadratic complexity in the number of variables and linear complexity in the number of observations . These estimators are based on several forms of pairwise robust covariance and correlation estimates . The estimators studied include two fast estimators based on coordinate-wise robust transformations embedded in an overall procedure recently proposed by ( 14 ) . We show that the estimators have attractive robustness properties , and give an example that uses one of the estimators in the new Insightful Miner data mining product .
doc_bio_tags: O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O