Repairing Systematic Outliers by Learning Clean Subspaces in VAEs

1 INTRODUCTION . Often practitioners have to deal with dirty datasets before they can start applying machine learning ( ML ) models . We focus on datasets where some instances have been corrupted by noise , producing outliers . The corresponding clean instances prior to corruption are called inliers , as well as any other instances that have not been corrupted . Because the presence of outliers can degrade the performance of ML methods ( Krishnan et al. , 2016 ; Liu et al. , 2020 ) , a standard option is to resort to a data cleaning pipeline before applying any model . This pipeline includes two key tasks : i ) outlier detection ( Ruff et al. , 2021 ) , detecting all outliers ; ii ) repair those outlier instances ( Neutatz et al. , 2021 ; Wan et al. , 2020 ) , recovering the underlying inlier instance . Ultimately , the goal is to propose a method that performs these steps automatically , i.e . automatic detection and repair . Generally , we can consider two types of errors present in an outlier : random or systematic ( Taylor , 1997 ; Liu et al. , 2020 ) . Random errors corrupt each instance independently , and feature value changes are sampled from an unknown distribution . This noising can not be replicated in a repeatable manner . For continuous features , a common example of this type of error are those well-modelled by additive noise with zero-mean . Systematic errors result from deterministic , or nearly deterministic , transformations that occur repeatedly in the data . Examples of systematic errors include watermarks or deterministic pixel corruption ( e.g . artifacts ) in images ; additive offsets or replacement by default values ( e.g . NaN ) in sensor data ; deterministic change of categories ( mislabelling ) or of name formats in categorical features in tabular data . Usually , the same features are affected , but not always . In most cases , this noising can be replicated . These two types of errors have a different impact when it comes to models performing detection or repair . Random errors do not show a distinct pattern across outliers and thus are not predictable . As a result , unsupervised models using regularization or data reweighting ( Zhou & Paffenroth , 2017 ; Akrami et al. , 2019 ; Eduardo et al. , 2020 ) can avoid overfitting to random errors . On the contrary , a systematic error shows a pattern across outliers , as a result of the ( nearly ) deterministic transformation , making them predictable ( Liu et al. , 2020 ) . This property makes higher capacity models ( e.g . deep learning ) prone to overfitting to these errors , even in the presence of regularization . As a re- sult , models for outlier detection and repair in the presence of systematic errors more easily conflate outliers with inliers . In this work we study how to develop a model for data cleaning robust to the effect of systematic errors . One solution is to provide some supervision , so the model can distinguish between inliers and outliers with systematic errors . This supervision can be provided in different forms , such as logic rules ( Rekatsinas et al. , 2017 ) or programs ( Lew et al. , 2021 ) describing the underlying clean data . However , these may require expert knowledge or substantial effort to formalize . Hence , it is easier and less time consuming for the practitioner to simply provide a trusted set as form of supervision . A trusted set is a small labelled subset of the data , which can be used to train a method for detection and repair . The user labels the instances either inlier or outlier , providing a few examples ( e.g . 10 instances ) per type of systematic error to repair . Overall , this might correspond to less than 2 % of the entire dataset ( sparse semi-supervision ) . Equally important , labelling does not require the user to manually repair the instances in the trusted set . Deep generative models ( DGM ) have high capacity ( flexible ) and thus can easily overfit to systematic errors . This motivates us to propose a method for detection and repair of such errors based on a semi-supervised DGM , which to the best of our knowledge has not been explored . We postulate that inlier data needs a smaller representation relative to outlier data when being represented by DGMs which we call the compression hypothesis . In addition , we claim that outliers are a combination of a representation describing the inlier portion of an instance , and a representation describing the type of systematic error . Given these insights , we propose Clean Subspace Variational Autoencoder ( CLSVAE ) , a novel semisupervised model for detection and automated repair of systematic errors . This model deviates from a standard VAE ( Kingma & Welling , 2014 ) , in two ways . First , the latent representation is partitioned into two subspaces : one that describes the data if it were an inlier ( clean subspace ) , and the other that describes what systematic error ( if any ) has been applied ( dirty subspace ) . The model is encouraged to learn a disentangled representation using a simple yet effective approach : for outliers the decoder will use a clean subspace concatenated with the dirty subspace ; in turn , for inliers the decoder will reuse the same clean subspace but concatenated with random noise . Then , at repair time the decoder will only need the clean subspace to reconstruct the underlying inlier . Secondly , we introduce semi-supervision through a trusted set , in so helping the model distinguish between inliers and outliers with systematic errors . Additionally , to encourage the clean subspace to represent inlier data , and the dirty subspace to represent systematic errors , we introduce a novel penalty term minimizing their mutual information ( MI ) . This penalty is based on the distance correlation ( DC ) ( Székely et al. , 2007 ) , and it improves model performance ( stability ) and repair quality . Compared to baselines we provide superior repairs , particularly , we show significant advantage in smaller trusted sets or when more of the dataset is corrupted . 2 RELATED WORK . For random errors , several works have explored detection ( Akrami et al. , 2019 ; Ruff et al. , 2019 ; Liu et al. , 2020 ; Lai et al. , 2019 ) . Some have proposed methods for detection and automated repair of random errors ( Eduardo et al. , 2020 ; Zhou & Paffenroth , 2017 ; Krishnan et al. , 2016 ) . A few works have explored outlier detection for systematic outliers , both semi-supervised ( Ruff et al. , 2019 ) and unsupervised ( Liu et al. , 2020 ; Lai et al. , 2020 ) , but these works do not consider automated repair . For tabular data , methods have been proposed that can do repair after detection has been performed ; for an overview , see ( Ilyas & Chu , 2019 ; Chu et al. , 2016 ) . These use probabilistic models melded with logic rules , i.e . probabilistic relational models ( Rekatsinas et al. , 2017 ) , or user-written programmatic descriptions of the data , e.g . in a probabilistic programming language ( Lew et al. , 2021 ) . They have the potential to capture systematic errors . However , these require the user to provide rules or programs that characterize clean data , which requires manual effort and expertise . In contrast , labelling a few inlier and outlier instances to build a trusted set may be more user-friendly . The idea of using unsupervised models for outlier detection is not new ( Schölkopf et al. , 1999 ; Liu et al. , 2008 ) . Unsupervised removal from a dataset of a small fraction of instances suspected of being outliers has been explored in Koh et al . ( 2018 ) ; Diakonikolas et al . ( 2018 ) ; Liu et al . ( 2020 ) , when used against adversarial errors it is called data sanitization ( Koh et al. , 2018 ) . However , if one wants to repair existing dirty data , a common unsupervised approach for moderate corruption is to apply enough regularization or data reweighting to an autoencoder ( Eduardo et al. , 2020 ; Zhou & Paffenroth , 2017 ; Akrami et al. , 2019 ) , hopefully repairing the errors at reconstruction . This has proven successful for random errors , though as we show , this is less effective for systematic errors . Often regularization is too strong leading to bad repair quality , since reconstruction is collapsing to mean behaviour – e.g . blurry image samples , missing details . Attribute manipulation models ( Klys et al. , 2018 ; Choi et al. , 2020 ) usually rely on extensive labelled data , usually fully supervised . Data cleaning can be seen as an attribute manipulation problem with one attribute : either the instance is an inlier or outlier . Some models like CVAE ( Sohn et al. , 2015 ) may use discrete latent variables , instead of continuous , to model attributes . These may lack the capacity to capture diversity in the same attribute ( Joy et al. , 2020 ) , e.g . distinct types of systematic errors in the data , and thus offer a poor fit to the data . Later , this may result in poor repair . Disentanglement models , unsupervised ( Locatello et al. , 2019 ; Tonolini et al. , 2019 ) and semisupervised ( Ilse et al. , 2020 ; Locatello et al. , 2019 ; Joy et al. , 2020 ) , encourage individual ( continuous ) latent variables to capture different attributes of the data instances . In theory , a disentanglement model could isolate the inlier / outlier attribute into a latent variable , then after training one could find a value for this variable that repairs the outlier . In practice , this is much simpler with semisupervised models , since we know which variable corresponds the attribute , but these tend to use more labelled data than we consider . In either case , these models usually need additional processing , often requiring human in the loop to explore the latent variable and find the best repair . Conversely , once our model is trained , further human intervention is not needed , so that repair is automated . 3 PROBLEM DEFINITION . We assume the user knows that systematic errors have corrupted dataset X , and thus outliers exist therein , but the inliers are still the majority . The user also has an idea of what patterns constitute systematic errors , and is able to recognize them . Intuitively , we think of inlier data being characterized by a set of patterns , which we call clean patterns , and those that constitute ( systematic ) errors , called dirty patterns . An outlier is an instance that has been corrupted by systematic errors , where prior to that would be constituted by clean patterns only . Formally , for x̃ an underlying inlier instance , an outlier is defined by x = fcr ( x̃ ) . We define fcr as a general transformation that corrupts the inlier instance with systematic errors , and not necessarily invertible . Each type of systematic error usually affects specific features of x̃ in the same predictable way , repeatedly in the dataset . This is unlike random errors , which both the feature and changed values are at random throughout instances . The user builds a small trusted set by labelling a few of the inliers and outliers in the train set X , forming the labelled subset Xl = { xn } Nln=1 . So we have the dirty dataset X = Xl ∪ Xu , where Xu = { xn } Nun=1 is the unlabelled part . The overall size of the train set is N = Nl + Nu . Each xn ∈ Xl is associated with a label yn ∈ { 0 , 1 } , which indicates whether xn is an inlier ( yn = 1 ) or an outlier ( yn = 0 ) . We write Yl = { yn } Nln=1 and thus the trusted set is formally defined by ( Xl , Yl ) . The trusted set should be representative of the inliers and outliers in the data . Note that there is a set of different corrupting transformations , i.e . systematic error types , each applied to several instances . Hence , the trusted set should provide at least a few labelled samples per type of systematic error . This is important so as to help the model distinguish between inliers and outliers . In our problem , the labelled portion of the dataset Xl ( trusted set ) is significantly smaller than the unlabelled portion Xu , e.g . 0.5 % of Nu . Given how small the trusted set is , we refer to this as sparse semi-supervision . In this work , the main task is to first perform outlier detection on X , in order to discover y for each instance x . The second task is repair those instances that are considered to be outliers ( y = 0 ) . Specifically , the model needs to provide a repair transformation gr such that x̂ = gr ( x ) where x̂ ≈ x̃ ( i.e . repair is close enough to the underlying inlier instance ) . Thus , we want a model that performs these two steps automatically , i.e . without a human in the loop .

Summary: The paper presents a Clean Subspace Variational Autoencoder (CLSVAE) model for systematic outlier detection. Building on VAE for latent variable modelling, the paper proposes a semi-supervised learning to infer the potential outlier.   The author made the following assumptions: 1. outliers exist therein, the inliers are still the majority; 2. Outliers are systematic with predictable recurring patterns; 3.  Compression hypothesis. Inlier data can be compressed further than outlier data. 4. Outliers are a combination of clean and dirty patterns.  On the model side, a mixture model is proposed parametrized by Bernoulli mixing weight and decoder p(x|z), while the encoder q(z|x) is applied for sampling latent variable z. A standard ELBO is formulated to maximise the log likelihood for both y known and unknown scenarios. For outlier repair, the author proposed to minimize the correlation between noise and clean signal.  Experimentally, the author evaluate two tasks: outlier detection, and automated repair on Frey-Faces3 , Fashion-MNIST (Xiao et al., 2017), Synthetic-Shapes.