Repairing Systematic Outliers by Learning Clean Subspaces in VAEs

1 INTRODUCTION . Often practitioners have to deal with dirty datasets before they can start applying machine learning ( ML ) models . We focus on datasets where some instances have been corrupted by noise , producing outliers . The corresponding clean instances prior to corruption are called inliers , as well as any other instances that have not been corrupted . Because the presence of outliers can degrade the performance of ML methods ( Krishnan et al. , 2016 ; Liu et al. , 2020 ) , a standard option is to resort to a data cleaning pipeline before applying any model . This pipeline includes two key tasks : i ) outlier detection ( Ruff et al. , 2021 ) , detecting all outliers ; ii ) repair those outlier instances ( Neutatz et al. , 2021 ; Wan et al. , 2020 ) , recovering the underlying inlier instance . Ultimately , the goal is to propose a method that performs these steps automatically , i.e . automatic detection and repair . Generally , we can consider two types of errors present in an outlier : random or systematic ( Taylor , 1997 ; Liu et al. , 2020 ) . Random errors corrupt each instance independently , and feature value changes are sampled from an unknown distribution . This noising can not be replicated in a repeatable manner . For continuous features , a common example of this type of error are those well-modelled by additive noise with zero-mean . Systematic errors result from deterministic , or nearly deterministic , transformations that occur repeatedly in the data . Examples of systematic errors include watermarks or deterministic pixel corruption ( e.g . artifacts ) in images ; additive offsets or replacement by default values ( e.g . NaN ) in sensor data ; deterministic change of categories ( mislabelling ) or of name formats in categorical features in tabular data . Usually , the same features are affected , but not always . In most cases , this noising can be replicated . These two types of errors have a different impact when it comes to models performing detection or repair . Random errors do not show a distinct pattern across outliers and thus are not predictable . As a result , unsupervised models using regularization or data reweighting ( Zhou & Paffenroth , 2017 ; Akrami et al. , 2019 ; Eduardo et al. , 2020 ) can avoid overfitting to random errors . On the contrary , a systematic error shows a pattern across outliers , as a result of the ( nearly ) deterministic transformation , making them predictable ( Liu et al. , 2020 ) . This property makes higher capacity models ( e.g . deep learning ) prone to overfitting to these errors , even in the presence of regularization . As a re- sult , models for outlier detection and repair in the presence of systematic errors more easily conflate outliers with inliers . In this work we study how to develop a model for data cleaning robust to the effect of systematic errors . One solution is to provide some supervision , so the model can distinguish between inliers and outliers with systematic errors . This supervision can be provided in different forms , such as logic rules ( Rekatsinas et al. , 2017 ) or programs ( Lew et al. , 2021 ) describing the underlying clean data . However , these may require expert knowledge or substantial effort to formalize . Hence , it is easier and less time consuming for the practitioner to simply provide a trusted set as form of supervision . A trusted set is a small labelled subset of the data , which can be used to train a method for detection and repair . The user labels the instances either inlier or outlier , providing a few examples ( e.g . 10 instances ) per type of systematic error to repair . Overall , this might correspond to less than 2 % of the entire dataset ( sparse semi-supervision ) . Equally important , labelling does not require the user to manually repair the instances in the trusted set . Deep generative models ( DGM ) have high capacity ( flexible ) and thus can easily overfit to systematic errors . This motivates us to propose a method for detection and repair of such errors based on a semi-supervised DGM , which to the best of our knowledge has not been explored . We postulate that inlier data needs a smaller representation relative to outlier data when being represented by DGMs which we call the compression hypothesis . In addition , we claim that outliers are a combination of a representation describing the inlier portion of an instance , and a representation describing the type of systematic error . Given these insights , we propose Clean Subspace Variational Autoencoder ( CLSVAE ) , a novel semisupervised model for detection and automated repair of systematic errors . This model deviates from a standard VAE ( Kingma & Welling , 2014 ) , in two ways . First , the latent representation is partitioned into two subspaces : one that describes the data if it were an inlier ( clean subspace ) , and the other that describes what systematic error ( if any ) has been applied ( dirty subspace ) . The model is encouraged to learn a disentangled representation using a simple yet effective approach : for outliers the decoder will use a clean subspace concatenated with the dirty subspace ; in turn , for inliers the decoder will reuse the same clean subspace but concatenated with random noise . Then , at repair time the decoder will only need the clean subspace to reconstruct the underlying inlier . Secondly , we introduce semi-supervision through a trusted set , in so helping the model distinguish between inliers and outliers with systematic errors . Additionally , to encourage the clean subspace to represent inlier data , and the dirty subspace to represent systematic errors , we introduce a novel penalty term minimizing their mutual information ( MI ) . This penalty is based on the distance correlation ( DC ) ( Székely et al. , 2007 ) , and it improves model performance ( stability ) and repair quality . Compared to baselines we provide superior repairs , particularly , we show significant advantage in smaller trusted sets or when more of the dataset is corrupted . 2 RELATED WORK . For random errors , several works have explored detection ( Akrami et al. , 2019 ; Ruff et al. , 2019 ; Liu et al. , 2020 ; Lai et al. , 2019 ) . Some have proposed methods for detection and automated repair of random errors ( Eduardo et al. , 2020 ; Zhou & Paffenroth , 2017 ; Krishnan et al. , 2016 ) . A few works have explored outlier detection for systematic outliers , both semi-supervised ( Ruff et al. , 2019 ) and unsupervised ( Liu et al. , 2020 ; Lai et al. , 2020 ) , but these works do not consider automated repair . For tabular data , methods have been proposed that can do repair after detection has been performed ; for an overview , see ( Ilyas & Chu , 2019 ; Chu et al. , 2016 ) . These use probabilistic models melded with logic rules , i.e . probabilistic relational models ( Rekatsinas et al. , 2017 ) , or user-written programmatic descriptions of the data , e.g . in a probabilistic programming language ( Lew et al. , 2021 ) . They have the potential to capture systematic errors . However , these require the user to provide rules or programs that characterize clean data , which requires manual effort and expertise . In contrast , labelling a few inlier and outlier instances to build a trusted set may be more user-friendly . The idea of using unsupervised models for outlier detection is not new ( Schölkopf et al. , 1999 ; Liu et al. , 2008 ) . Unsupervised removal from a dataset of a small fraction of instances suspected of being outliers has been explored in Koh et al . ( 2018 ) ; Diakonikolas et al . ( 2018 ) ; Liu et al . ( 2020 ) , when used against adversarial errors it is called data sanitization ( Koh et al. , 2018 ) . However , if one wants to repair existing dirty data , a common unsupervised approach for moderate corruption is to apply enough regularization or data reweighting to an autoencoder ( Eduardo et al. , 2020 ; Zhou & Paffenroth , 2017 ; Akrami et al. , 2019 ) , hopefully repairing the errors at reconstruction . This has proven successful for random errors , though as we show , this is less effective for systematic errors . Often regularization is too strong leading to bad repair quality , since reconstruction is collapsing to mean behaviour – e.g . blurry image samples , missing details . Attribute manipulation models ( Klys et al. , 2018 ; Choi et al. , 2020 ) usually rely on extensive labelled data , usually fully supervised . Data cleaning can be seen as an attribute manipulation problem with one attribute : either the instance is an inlier or outlier . Some models like CVAE ( Sohn et al. , 2015 ) may use discrete latent variables , instead of continuous , to model attributes . These may lack the capacity to capture diversity in the same attribute ( Joy et al. , 2020 ) , e.g . distinct types of systematic errors in the data , and thus offer a poor fit to the data . Later , this may result in poor repair . Disentanglement models , unsupervised ( Locatello et al. , 2019 ; Tonolini et al. , 2019 ) and semisupervised ( Ilse et al. , 2020 ; Locatello et al. , 2019 ; Joy et al. , 2020 ) , encourage individual ( continuous ) latent variables to capture different attributes of the data instances . In theory , a disentanglement model could isolate the inlier / outlier attribute into a latent variable , then after training one could find a value for this variable that repairs the outlier . In practice , this is much simpler with semisupervised models , since we know which variable corresponds the attribute , but these tend to use more labelled data than we consider . In either case , these models usually need additional processing , often requiring human in the loop to explore the latent variable and find the best repair . Conversely , once our model is trained , further human intervention is not needed , so that repair is automated . 3 PROBLEM DEFINITION . We assume the user knows that systematic errors have corrupted dataset X , and thus outliers exist therein , but the inliers are still the majority . The user also has an idea of what patterns constitute systematic errors , and is able to recognize them . Intuitively , we think of inlier data being characterized by a set of patterns , which we call clean patterns , and those that constitute ( systematic ) errors , called dirty patterns . An outlier is an instance that has been corrupted by systematic errors , where prior to that would be constituted by clean patterns only . Formally , for x̃ an underlying inlier instance , an outlier is defined by x = fcr ( x̃ ) . We define fcr as a general transformation that corrupts the inlier instance with systematic errors , and not necessarily invertible . Each type of systematic error usually affects specific features of x̃ in the same predictable way , repeatedly in the dataset . This is unlike random errors , which both the feature and changed values are at random throughout instances . The user builds a small trusted set by labelling a few of the inliers and outliers in the train set X , forming the labelled subset Xl = { xn } Nln=1 . So we have the dirty dataset X = Xl ∪ Xu , where Xu = { xn } Nun=1 is the unlabelled part . The overall size of the train set is N = Nl + Nu . Each xn ∈ Xl is associated with a label yn ∈ { 0 , 1 } , which indicates whether xn is an inlier ( yn = 1 ) or an outlier ( yn = 0 ) . We write Yl = { yn } Nln=1 and thus the trusted set is formally defined by ( Xl , Yl ) . The trusted set should be representative of the inliers and outliers in the data . Note that there is a set of different corrupting transformations , i.e . systematic error types , each applied to several instances . Hence , the trusted set should provide at least a few labelled samples per type of systematic error . This is important so as to help the model distinguish between inliers and outliers . In our problem , the labelled portion of the dataset Xl ( trusted set ) is significantly smaller than the unlabelled portion Xu , e.g . 0.5 % of Nu . Given how small the trusted set is , we refer to this as sparse semi-supervision . In this work , the main task is to first perform outlier detection on X , in order to discover y for each instance x . The second task is repair those instances that are considered to be outliers ( y = 0 ) . Specifically , the model needs to provide a repair transformation gr such that x̂ = gr ( x ) where x̂ ≈ x̃ ( i.e . repair is close enough to the underlying inlier instance ) . Thus , we want a model that performs these two steps automatically , i.e . without a human in the loop .

Summary: This paper develops a deep learning based method for detecting and repairing systematic outliers in data (before performing model learning on the data).  The method is called Clean Subspace Variational Autoencoder (CLSVAE). CLSVAE takes as input a trusted set provided by the user (the trusted set contains a set of inliers and outliers, but not requiring information about how the outliers are corrupted), and seeks to learn a variational autoencoder where the latent code is a concatenation of two subspaces, a clean subspace and a dirty subspace. CLSVAE is semi-supervised because the labelled trusted set is much smaller than the unlabelled set.  The paper performs outlier detection and repair benchmarks on three image datasets with systematic outliers, and shows that CLSVAE achieves favorable results.