Year: 2021

Abstract:
Data cleaning often comprises outlier detection and data repair. Systematic errors result from nearly deterministic transformations that occur repeatedly in the data, e.g. specific image pixels being set to default values or watermarks. Consequently, models with enough capacity easily overfit to these errors, making detection and repair difficult. Seeing as a systematic outlier is a combination of patterns of a clean instance and systematic error patterns, our main insight is that inliers can be modelled by a smaller representation (subspace) in a model than outliers. By exploiting this, we propose Clean Subspace Variational Autoencoder (CLSVAE), a novel semi-supervised model for detection and automated repair of systematic errors. The main idea is to partition the latent space and model inlier and outlier patterns separately. CLSVAE is effective with much less labelled data compared to previous related models, often with less than 2% of the data. We provide experiments using three image datasets in scenarios with different levels of corruption and labelled set sizes, comparing to relevant baselines. CLSVAE provides superior repairs without human intervention, e.g. with just 0.25% of labelled data we see a relative error decrease of 58% compared to the closest baseline.

heading: 1 INTRODUCTION
text: Often practitioners have to deal with dirty datasets before they can start applying machine learning (ML) models. We focus on datasets where some instances have been corrupted by noise, producing outliers. The corresponding clean instances prior to corruption are called inliers, as well as any other instances that have not been corrupted. Because the presence of outliers can degrade the performance of ML methods (Krishnan et al., 2016; Liu et al., 2020), a standard option is to resort to a data cleaning pipeline before applying any model. This pipeline includes two key tasks: i) outlier detection (Ruff et al., 2021), detecting all outliers; ii) repair those outlier instances (Neutatz et al., 2021; Wan et al., 2020), recovering the underlying inlier instance. Ultimately, the goal is to propose a method that performs these steps automatically, i.e. automatic detection and repair.
Generally, we can consider two types of errors present in an outlier: random or systematic (Taylor, 1997; Liu et al., 2020). Random errors corrupt each instance independently, and feature value changes are sampled from an unknown distribution. This noising cannot be replicated in a repeatable manner. For continuous features, a common example of this type of error are those well-modelled by additive noise with zero-mean. Systematic errors result from deterministic, or nearly deterministic, transformations that occur repeatedly in the data. Examples of systematic errors include watermarks or deterministic pixel corruption (e.g. artifacts) in images; additive offsets or replacement by default values (e.g. NaN) in sensor data; deterministic change of categories (mislabelling) or of name formats in categorical features in tabular data. Usually, the same features are affected, but not always. In most cases, this noising can be replicated.
These two types of errors have a different impact when it comes to models performing detection or repair. Random errors do not show a distinct pattern across outliers and thus are not predictable. As a result, unsupervised models using regularization or data reweighting (Zhou & Paffenroth, 2017; Akrami et al., 2019; Eduardo et al., 2020) can avoid overfitting to random errors. On the contrary, a systematic error shows a pattern across outliers, as a result of the (nearly) deterministic transformation, making them predictable (Liu et al., 2020). This property makes higher capacity models (e.g. deep learning) prone to overfitting to these errors, even in the presence of regularization. As a re-
sult, models for outlier detection and repair in the presence of systematic errors more easily conflate outliers with inliers. In this work we study how to develop a model for data cleaning robust to the effect of systematic errors.
One solution is to provide some supervision, so the model can distinguish between inliers and outliers with systematic errors. This supervision can be provided in different forms, such as logic rules (Rekatsinas et al., 2017) or programs (Lew et al., 2021) describing the underlying clean data. However, these may require expert knowledge or substantial effort to formalize. Hence, it is easier and less time consuming for the practitioner to simply provide a trusted set as form of supervision. A trusted set is a small labelled subset of the data, which can be used to train a method for detection and repair. The user labels the instances either inlier or outlier, providing a few examples (e.g. 10 instances) per type of systematic error to repair. Overall, this might correspond to less than 2% of the entire dataset (sparse semi-supervision). Equally important, labelling does not require the user to manually repair the instances in the trusted set.
Deep generative models (DGM) have high capacity (flexible) and thus can easily overfit to systematic errors. This motivates us to propose a method for detection and repair of such errors based on a semi-supervised DGM, which to the best of our knowledge has not been explored. We postulate that inlier data needs a smaller representation relative to outlier data when being represented by DGMs which we call the compression hypothesis. In addition, we claim that outliers are a combination of a representation describing the inlier portion of an instance, and a representation describing the type of systematic error.
Given these insights, we propose Clean Subspace Variational Autoencoder (CLSVAE), a novel semisupervised model for detection and automated repair of systematic errors. This model deviates from a standard VAE (Kingma & Welling, 2014), in two ways. First, the latent representation is partitioned into two subspaces: one that describes the data if it were an inlier (clean subspace), and the other that describes what systematic error (if any) has been applied (dirty subspace). The model is encouraged to learn a disentangled representation using a simple yet effective approach: for outliers the decoder will use a clean subspace concatenated with the dirty subspace; in turn, for inliers the decoder will reuse the same clean subspace but concatenated with random noise. Then, at repair time the decoder will only need the clean subspace to reconstruct the underlying inlier. Secondly, we introduce semi-supervision through a trusted set, in so helping the model distinguish between inliers and outliers with systematic errors. Additionally, to encourage the clean subspace to represent inlier data, and the dirty subspace to represent systematic errors, we introduce a novel penalty term minimizing their mutual information (MI). This penalty is based on the distance correlation (DC) (Székely et al., 2007), and it improves model performance (stability) and repair quality. Compared to baselines we provide superior repairs, particularly, we show significant advantage in smaller trusted sets or when more of the dataset is corrupted.

heading: 2 RELATED WORK
text: For random errors, several works have explored detection (Akrami et al., 2019; Ruff et al., 2019; Liu et al., 2020; Lai et al., 2019). Some have proposed methods for detection and automated repair of random errors (Eduardo et al., 2020; Zhou & Paffenroth, 2017; Krishnan et al., 2016). A few works have explored outlier detection for systematic outliers, both semi-supervised (Ruff et al., 2019) and unsupervised (Liu et al., 2020; Lai et al., 2020), but these works do not consider automated repair.
For tabular data, methods have been proposed that can do repair after detection has been performed; for an overview, see (Ilyas & Chu, 2019; Chu et al., 2016). These use probabilistic models melded with logic rules, i.e. probabilistic relational models (Rekatsinas et al., 2017), or user-written programmatic descriptions of the data, e.g. in a probabilistic programming language (Lew et al., 2021). They have the potential to capture systematic errors. However, these require the user to provide rules or programs that characterize clean data, which requires manual effort and expertise. In contrast, labelling a few inlier and outlier instances to build a trusted set may be more user-friendly.
The idea of using unsupervised models for outlier detection is not new (Schölkopf et al., 1999; Liu et al., 2008). Unsupervised removal from a dataset of a small fraction of instances suspected of being outliers has been explored in Koh et al. (2018); Diakonikolas et al. (2018); Liu et al. (2020), when used against adversarial errors it is called data sanitization (Koh et al., 2018). However, if
one wants to repair existing dirty data, a common unsupervised approach for moderate corruption is to apply enough regularization or data reweighting to an autoencoder (Eduardo et al., 2020; Zhou & Paffenroth, 2017; Akrami et al., 2019), hopefully repairing the errors at reconstruction. This has proven successful for random errors, though as we show, this is less effective for systematic errors. Often regularization is too strong leading to bad repair quality, since reconstruction is collapsing to mean behaviour – e.g. blurry image samples, missing details.
Attribute manipulation models (Klys et al., 2018; Choi et al., 2020) usually rely on extensive labelled data, usually fully supervised. Data cleaning can be seen as an attribute manipulation problem with one attribute: either the instance is an inlier or outlier. Some models like CVAE (Sohn et al., 2015) may use discrete latent variables, instead of continuous, to model attributes. These may lack the capacity to capture diversity in the same attribute (Joy et al., 2020), e.g. distinct types of systematic errors in the data, and thus offer a poor fit to the data. Later, this may result in poor repair.
Disentanglement models, unsupervised (Locatello et al., 2019; Tonolini et al., 2019) and semisupervised (Ilse et al., 2020; Locatello et al., 2019; Joy et al., 2020), encourage individual (continuous) latent variables to capture different attributes of the data instances. In theory, a disentanglement model could isolate the inlier / outlier attribute into a latent variable, then after training one could find a value for this variable that repairs the outlier. In practice, this is much simpler with semisupervised models, since we know which variable corresponds the attribute, but these tend to use more labelled data than we consider. In either case, these models usually need additional processing, often requiring human in the loop to explore the latent variable and find the best repair. Conversely, once our model is trained, further human intervention is not needed, so that repair is automated.

heading: 3 PROBLEM DEFINITION
text: We assume the user knows that systematic errors have corrupted dataset X , and thus outliers exist therein, but the inliers are still the majority. The user also has an idea of what patterns constitute systematic errors, and is able to recognize them. Intuitively, we think of inlier data being characterized by a set of patterns, which we call clean patterns, and those that constitute (systematic) errors, called dirty patterns. An outlier is an instance that has been corrupted by systematic errors, where prior to that would be constituted by clean patterns only. Formally, for x̃ an underlying inlier instance, an outlier is defined by x = fcr(x̃). We define fcr as a general transformation that corrupts the inlier instance with systematic errors, and not necessarily invertible. Each type of systematic error usually affects specific features of x̃ in the same predictable way, repeatedly in the dataset. This is unlike random errors, which both the feature and changed values are at random throughout instances.
The user builds a small trusted set by labelling a few of the inliers and outliers in the train set X , forming the labelled subset Xl = {xn}Nln=1. So we have the dirty dataset X = Xl ∪ Xu , where Xu = {xn}Nun=1 is the unlabelled part. The overall size of the train set is N = Nl + Nu . Each xn ∈ Xl is associated with a label yn ∈ {0, 1}, which indicates whether xn is an inlier (yn = 1) or an outlier (yn = 0). We write Yl = {yn}Nln=1 and thus the trusted set is formally defined by (Xl ,Yl). The trusted set should be representative of the inliers and outliers in the data. Note that there is a set of different corrupting transformations, i.e. systematic error types, each applied to several instances. Hence, the trusted set should provide at least a few labelled samples per type of systematic error. This is important so as to help the model distinguish between inliers and outliers. In our problem, the labelled portion of the dataset Xl (trusted set) is significantly smaller than the unlabelled portion Xu , e.g. 0.5% of Nu . Given how small the trusted set is, we refer to this as sparse semi-supervision. In this work, the main task is to first perform outlier detection on X , in order to discover y for each instance x. The second task is repair those instances that are considered to be outliers (y = 0). Specifically, the model needs to provide a repair transformation gr such that x̂ = gr(x) where x̂ ≈ x̃ (i.e. repair is close enough to the underlying inlier instance). Thus, we want a model that performs these two steps automatically, i.e. without a human in the loop.

heading: 4 OUR PROPOSAL: CLEAN SUBSPACE VAE (CLSVAE)
text: In this section, we introduce the generative and variational models for our proposal. Our Variational Autoencoder (VAE) model is motivated by three observations:
• Outliers are systematic. Outlier instances can be described by a number of predictable recurring patterns, some of which are considered dirty (e.g. black patch on an image). Our assumption is that the latter patterns can be well represented by a DGM.
• Compression hypothesis. Inlier data can be compressed further than outlier data. We assume that less capacity (parameters or variables) is needed by DGMs to represent inliers.
• Outliers are a combination of clean and dirty patterns. Because clean and dirty patterns produce different visible effects in the instance, the effects of clean and dirty patterns can be modelled separately. Hence, this DGM only requires the parameters (or variables) of the clean patterns to generate a repair.
We call our model the sparse semi-supervised Clean Subspace VAE (CLSVAE). This reflects the fact that our model has a partitioned latent space: a subspace for clean patterns, and a subspace for dirty patterns. Thus at test time only the subspace for clean patterns is used to generate a repair. A standard VAE (Kingma & Welling, 2014), with weight decay, is provided in Annex A for readers unfamiliar with this type of model.

heading: 4.1 GENERATIVE MODEL
text: The main idea is that if the compression hypothesis holds inlier samples (y = 1) will have a smaller representation relative to outliers (y = 0), whilst outliers need an extra representation to model errors. In our model, zc will represent inliers, zd will represent the error pattern for outliers, and z is random noise. The overall latent code is z = [zc; y z + (1− y) zd], where [ ; ] defines vector concatenation, leading to a latent space that is partitioned into two subspaces. If generating an inlier (y = 1) then zc ∈ Rq is by itself responsible for modelling all the clean patterns present in an instance, encouraging a lower dimensional manifold for inlier data. This is due to z being a random noise vector of the same dimension as zd ∈ Rp, devoid of additional information. If generating an outlier (y = 0) both zc and zd are used, where zd models the different types of dirty patterns that can be detected in an instance. In fact, using z during inlier generation encourages zd to model dirty patterns only, encouraging a higher dimensional manifold for outlier data.
Our generative model is therefore defined by the joint distribution between y and latent subspaces zc and zd. The main idea expressed above can be written as a two component mixture model for the decoder, see eq. (4), where y is the gating variable. Hence we have
pθ(x, zc, zd, z , y) = pθ(x|zc, zd, z , y)pσc(zc)pσd(zd)pσ (z )pα(y), (1) where
pα(y) = Bernoulli(y|α), (2) pσc(zc) = N (zc|0, σ2cI) pσd(zd) = N (zd|0, σ2dI) pσ (z ) = N (z |0, σ2 I), (3)
pθ(x|zc, zd, z , y) = pθ(x| [zc; z ])ypθ(x| [zc; zd])(1−y), (4) where the density pθ(x | [·; ·]) is parameterized by a neural network. We assume inlier data has a smaller variance as whole than outlier data, so we use a σc that is much smaller than σd. Note that z is just Gaussian random noise1 with σ < σd. Hence, after training, the region around 0 (zero mean) for this subspace encourages pθ(x | [·; ·]) to only use zc for reconstruction, obtaining a repaired instance. The parameter α reflects the prior belief on the fraction of clean data. Smaller values for α means more data points are rejected when modelling zc, which offers more robustness. So we have σ , σc, σd and α as hyper-parameters.

heading: 4.2 VARIATIONAL MODEL
text: We consider separate encoders for zc and zd, and make y depend on zc and zd. The idea is that parameters of each encoder focus on different aspects, i.e. clean or dirty patterns respectively. The model factorizes as
q(zc, zd, z , y|x) = qφy (y|zc, zd)qφc(zc|x)qφd(zd|x)qσ (z ), (5) and
qφc(zc|x) = N (zc|µφc(x),σ2φc(x)), qφd(zd|x) = N (zd|µφd(x),σ 2 φd (x)), (6)
1We tried using z = 0 at training, however this did not work as well as setting z to random noise.
qφy (y|zc, zd) = Bernoulli(y|πφy ([zc; zd]), qσ (z ) = pσ (z ), (7) where for distributions in eq. (6): {µφc(.),σφc(.)} is a neural network with parameters φc; similarly for {µφd(.),σφd(.)} with φd.We have σφc(x) and σφd(x) being diagonal covariance matrices. The distribution for random noise z is the same as in the generative model. In eq. (7), the πφy (.) parametrizes the Bernoulli distribution qφy (y|zc, zd), and is a neural network with parameters φy .We found that using qφy (y|zc, zd) yielded better results than using qφy (y|zd) or qφy (y|x). Since zc provides important context on the clean patterns present, which in turn allows zd to better focus on modelling dirty patterns. In practice, to stabilize the optimization procedure in a few cases, we used πφy ([sg(zc); zd]) in qφy (y|zc, zd) – eq. (7). Note that sg(zc) stands for stop gradient operator applied to zc; and this prevents zc from being updated with dirty pattern information early in the training. A more detailed discussion on qφy (y|zc, zd) is presented in Annex E.

heading: 4.3 TRAINING LOSS
text: Our model is trained to maximize an objective function with three terms, which accounts for our semi-supervised setting. The first term L(x) is the evidence lower bound (ELBO) for the unlabelled part of the data. The second term L(x, y) is the ELBO for the trusted set. The third term LWCE(x, y) is the weighted cross-entropy loss which ensures that qφy (y|zc, zd) correctly predicts the trusted set labels y.
The ELBO for the unlabelled (unsupervised) part is L(x) =Eqφc (zc|x)qφd (zd|x)pσ (z ) [ πφy ([zc; zd]) log pθ(x| [zc; z ]) (8)
+ (1− πφy ([zc; zd])) log pθ(x| [zc; zd])−DKL ( qφy (y|zc, zd)||pα(y) ) ] −DKL (qφc(zc|x)||pσc(zc))−DKL (qφd(zd|x)||pσd(zd)) ,
where qσ (z ) = pσ (z ) and so they cancel each other. The expectations are obtained via MonteCarlo (MC) estimation via reparameterization trick (Kingma & Welling, 2014; Rezende et al., 2014).
For the trusted set (supervised) part the ELBO is L(x, y) =Eqφc (zc|x)qφd (zd|x)pσ (z ) [ y log pθ(x| [zc; z ]) + (1− y) log pθ(x| [zc; zd]) ] (9)
+ log pα(y)−DKL (qφc(zc|x)||pσc(zc))−DKL (qφd(zd|x)||pσd(zd)) . Lastly, we need to define the weighted cross-entropy LWCE(x, y), which is
LWCE(x, y) = −y. log q(y = 1|x)− ωimb.(1− y). log (1− q(y = 1|x)) (10) where
ωimb = max { 1, Nl1 Nl0 } , Nl1 = Nl∑ i=1 yi, Nl0 = Nl −Nl1 , (11)
and ωimb compensates for trusted set class imbalance.2 However, we do not have access to q(y = 1|x) and thus we cannot estimate LWCE(x, y) directly. Still, we can minimize an upper-bound
LWCE(x, y) ≤ L̃WCE(x, y) =Eqφc (zc|x)qφd (zd|x) [ − y log q(y = 1|zc, zd) (12)
− ωimb.(1− y). log (1− q(y = 1|zc, zd)) ] ,
which is obtained by applying Jensen’s inequality.
Combining the three terms defined above, we minimize the overall loss
I = − 1 N  ∑ x∈Xu L(x) + ∑ (x,y)∈Xl×Yl L(x, y) + β 1 Nl ∑ (x,y)∈Xl×Yl L̃WCE(x, y) (13)
with respect to the generative and variational parameters. The hyperparameter β value controls the amount of up-sampling and importance relative to the other terms, which tends to be moderately high due to how small the trusted set is.
2Useful when the number of labelled outliers outnumbers the inliers. Such a case does not reflect the common dataset composition, i.e. the number of inliers is larger than outliers.

heading: 4.4 DISTANCE CORRELATION PENALTY
text: Ideally, zc captures clean patterns only, whilst zd captures dirty patterns. Therefore zc and zd should have low mutual information (MI). However, in more challenging scenarios, e.g. small trusted set or higher dataset corruption, obtaining this solution may not be guaranteed. Enforcing a constraint encouraging low MI between zc and zd will lead to better model performance and stability in challenging scenarios, improving repair quality.
We would like to introduce a constraint that minimizes MI between zc and zd. However, approximating MI properly can be complex. Instead, we use distance correlation (DC) as a surrogate for MI (Székely et al., 2007), which is easier to compute and can measure non-linear dependencies between vector variables. Other works have used DC as a surrogate for MI, e.g. (Chen et al., 2021). Further, DC can also measure dependence between vector variables of different dimensions, which is often the case with zc and zd. For the data batch (zc, zd) ∈ (Zc,Zd) where Zc ∈ RN×q and Zd ∈ RN×p, we can define the empirical estimate of DC as dCorrN (Zc,Zd). The definition of the estimator dCorrN (Zc,Zd) can be seen in Annex F. Essentially, DC is the standard correlation between the elements of the double centered pairwise distance matrices of each data batchZc and Zd. The range is 0 ≤ dCorrN (Zc,Zd) ≤ 1, where 0 means variables are independent, and 1 implies that zc and zd are strongly correlated.
Enforcing this constraint means adding a penalty to the model training loss. Hence, reusing the loss defined in eq. (13) we now have
min φc,φd,φy,θ
I + λt dCorrN (Zc,Zd), (14)
where λt increases every epoch from 0 until it reaches a maximum value λT , which is then maintained. The rate of increase of λt and λT are hyper-parameters. This strategy is a type of penalty method as used in constrained optimization.

heading: 4.5 OUTLIER DETECTION AND REPAIR PROCESS
text: After training, we proceed with outlier detection and automated repair, as in Section 3. The detection task is to discover the ground-truth labels y for each x ∈ X , using inferred labels ŷ. A score A(x) and threshold γ ≥ 0 are used to get the set of outliers O = {x ∈ X | A(x) ≥ γ}, where a higherA(x) means the more likely x is an outlier. The inferred label ŷ is obtained as: (inlier) ŷ = 1 if x /∈ O ; (outlier) ŷ = 0 if x ∈ O. For our model, we use a score based on the negative log probability of inlier given the latent subspaces, which is
A(x) = − log πφy ([µφc(x);µφd(x)]), x ∈ X . (15) The threshold γ can be chosen as γ ≈ − log(0.5) assuming qφy (y = 1|.) = πφy (.) is near calibrated, or it can be user-defined. The repair task is to obtain an inferred reconstruction x̂ from the outlier x ∈ O such that it is close to the inlier ground-truth x̃. The repair is generated using the most likely reconstruction under our model for y = 1 (inliers), which means only the clean subspace zc is used. This is the maximum a posteriori estimate for a VAE, where one approximates pθ(zc|x) by qφ(zc|x), and then uses the means of qφ(zc|x), p (z ) and pθ(x | [zc; z ]) in the estimate. Hence, we have
x̂ = µθ([µφc(x); 0]), x ∈ O. (16)

heading: 5 EXPERIMENTS
text: We evaluate two tasks: outlier detection, and automated repair. Our experiments use three image datasets: Frey-Faces3, Fashion-MNIST (Xiao et al., 2017), Synthetic-Shapes. Synthetic-Shapes is a synthetic dataset built around four different shapes (classes): a circle, a rectangle, an ellipse and a triangle. These are colored white and set in a black background. We corrupt datasets with synthetic systematic errors, since public real-world datasets with ground-truth repairs and respective labels are difficult to find, as seen in Eduardo et al. (2020); Krishnan et al. (2016); Liu et al. (2020). We compare our model (CLSVAE) with baselines ranging from completely supervised to unsupervised.4
3http://www.cs.nyu.edu/˜roweis/data/frey_rawface.mat 4Code for models and experiments will be released upon paper acceptance.

heading: Dataset Data Type No. Data Classes No. Error Classes Error Types
text: Evaluation For outlier detection we use AVPR (Average Precision) (Everingham et al., 2015) to measure detection quality, which is a surrogate for the area under the precision-recall curve (Hendrycks & Gimpel, 2016). This metric is preferred since it is insensitive to label imbalance, typical in outlier detection. AVPR score is between [0, 1] and higher means better. For automated repair we want to quantify the quality of the repair, for outlier instances. We report the standardized mean squared error (SMSE) between pixels of the ground-truth (inlier) instance and that of the proposed repair. We report SMSE separately for the dirty pixels (those affected by the systematic error) and for the clean pixels (those unaffected). The first measures repair performance, while the latter measures distortion that the repair process causes to clean pixels. In both cases, a lower SMSE means better. Note in the case of binary pixels, the SMSE is just the Brier score, and thus is in [0, 1].
Datasets and Corruption Process In our experiments we take an uncorrupted dataset and inject it with systematic errors. These systematic errors are synthetic, designed to seem like reasonable image corruptions, e.g. occlusion or failing of a camera sensor. The types of systematic errors used across datasets are either lines or squares. Lines (two diagonal, one vertical, and one horizontal) cross the image from side to side, and may have their color set at random (black / white). These lines always affect the same pixels, and have thickness of one pixel. Squares are randomly uniformly placed, so is their fill-in color, with size 6 × 6 pixels. We use different noise levels so we can study their impact, we use [15%, 25%, 35%, 45%] of dataset. The systematic error corruption process is done by picking uniformly at random an instance, and then applying that systematic error. We use a range of trusted set sizes by defining the number of instances labelled per systematic error class (type) and per data class. The latter is the underlying classes in the dataset, e.g. item labels in Fashion-MNIST. For each class, either systematic error or data class, we provide label y for a few instances at random obtaining a trusted set. We use the range TSsize = [5, 10, 25, 50] labelled samples per class, which results in different trusted set sizes depending on number of classes. Particularly, we have the trusted set ranges: Synthetic-Shapes with [40, 80, 200, 400] total samples; Frey-Faces with [25, 50, 125, 250] total samples; Fashion-MNIST with [90, 180, 450, 900] total samples. We create five examples (different random seeds) per noise level and per trusted set size, and train the models on them. The results are then averaged. Table 1 describes datasets and their corruption (error types), number of systematic errors and data classes. More details on this experimental setup in Annex G.
Comparative Methods Our baselines are VAEs since most of the relevant work in sparse semisupervision is of this type (Ilse et al., 2020; Joy et al., 2020), and the task of repair is related with that of manipulating the reconstruction. We have four baselines: VAE-L2, CVAE, VAEGMM, CCVAE. For details on model architecture and hyperparameters see Annex H. The VAE-L2 model is an unsupervised method tackling the issue of corruption by applying strong regularization (`2 regularization on weights). VAE-L2 uses the reconstruction likelihood for detection, more details in Annex A. CVAE is the supervised version of the semi-supervised M2 model (Kingma et al., 2014), and should have better repair quality than M2. CVAE uses the reconstruction likelihood as an detection score. We found a smaller variance for p(z) to be beneficial, for details see Annex B. VAEGMM, based on (Willetts et al., 2020), is an improved version of M2 for the sparse semi-supervision setting. In this setting the M2 model tends to have posterior collapse issues with q(y|x), picking one class over others. VAEGMM overcomes this issue, improving clustering and classification performance. Hence, we expect competitive detection performance from VAEGMM, for details see Annex C. The CCVAE (Joy et al., 2020) is a state-of-the-art (SotA) semi-supervised disentanglement model, allowing attribute manipulation in semi-supervised settings. For repair, we follow the automatic attribute manipulation procedure proposed by (Joy et al., 2020) (see Annex D). We adapted their code to our pipeline. Contrary to Joy et al. (2020), we found performance was superior when using a large up-sampling coefficient for the classifier, i.e. like β in CLSVAE. We provide two versions of our model, with and without distance correlation penalty in Section 4.4. So for CLSVAE-NODC use eq. (13) as training loss, whilst for CLSVAE use eq. (14).

heading: 5.1 DISCUSSION OF RESULTS
text: In Figure 2(a) we show performance as a function of the size of the trusted set, i.e. sweep of trusted set sizes, for a 35% noise level. Similar performance is seen for other datasets (see Annex J). Table 2(b) shows the results for all datasets for a 35% noise level and trusted set size of 10 labelled samples per class. Results for all trusted set sizes and noise levels are found in Annex J, with similar analysis on performance. Figure 1 shows some examples of image repairs for all datasets (larger version Annex I). Additional examples of repairs are seen in Annex K, including inlier instances, 45% noise level, and 5 samples per class (trusted set size) for Synthetic-Shapes since its an easier dataset.
Outlier Detection Looking at Figure 2, we see that on average both CLSVAE (our model) and CLSVAE-NODC have the highest AVPR, registering the best detection performance, with similar scores. We see that CCVAE, the previous SotA, has similar detection performance as CLSVAE.
VAEGMM, also semi-supervised, lags behind both likely because its designed for slightly larger trusted sets. All semi-supervised models (CLSVAE, CLSVAE-NODC, CCVAE, VAEGMM) improve their detection performance as the trusted set grows larger. CVAE and VAE-L2 do not use a trusted set, and thus have the same performance throughout all the trusted set range in Figure 2(a). These two observations about the trusted set range are also seen, for all datasets and noise levels, in Annex J. CVAE is supervised, still it shows poor performance, this may be due to: issues linked to (decoder) likelihood-based scores (Eduardo et al., 2020; Lan & Dinh, 2020); poor fitting to the data, thus impacting negatively the score. VAE-L2 uses a likelihood score, and is unsupervised, so poorer detection performance is understandable. This highlights semi-supervision as being important in systematic error detection. VAE-L2 registering good performance in Synthetic-Shapes (see Figure 2(b)) is likely due to this dataset being easier. Lastly, we note that CLSVAE tends to have better detection performance in higher noise levels relative to other methods (see Annex J). Therein, CCVAE has close to or similar detection performance as CLSVAE.
Automated Repair In Figure 2, we see that on average CLSVAE (our model) is best at automated repair (lowest SMSE on dirty pixels). We also see that distortion (repair clean pixels, SMSE) is relatively low, but not the lowest. This results in CLSVAE overall being the best repair method, not only replacing pixel values of the systematic error, but also inferring correctly the structure of the ground-truth repair (both clean and dirty pixels). This is confirmed in Figure 1, where reconstructions (repairs) by CLSVAE show the best quality: replacing the error values of affected pixels and recovering the underlying ground-truth, whilst preserving the uncorrupted image portion (i.e. low distortion). CLSVAE has slightly better repair than CLSVAE-NODC on average, but most importantly, it has better performance stability than CLSVAE-NODC, which can be seen in Annex J for repair (dirty pixels). Further, in Annex J, repair with CLSVAE is more advantageous relative to other models at higher noise levels. As expected, semi-supervised models (CLSVAE, CLSVAENODC, CCVAE) improve their repair of dirty pixels as the trusted set increases (see Figure 2(a)). Both VAE-L2 and CVAE do not use a trusted set, so performance is static. CCVAE has the ability to perform good repair, registering the second best repair for dirty pixels after CLSVAE, but often with higher distortion. CCVAE suffers from two issues that account for its worse performance relative to CLSVAE. For one, looking at Figure 1, CCVAE can sometimes fail to replace the pixel values from systematic errors. Secondly, and more often, it can fail to recover the underlying ground-truth even when replacing erroneous pixel values. Similarly, it has difficulty preserving the uncorrupted image portion (higher distortion). So some information about inlier appearance is being lost. This is explained by the fact that CCVAE latent space is not disentangled regarding the clean and dirty patterns. CVAE can repair some outliers well, but it fails to deal with other systematic errors, which leads to an overall poor repair performance. This is maybe due to the binary latent variable used for y making it harder to model multiple systematic errors, for more discussion see Joy et al. (2020). VAE-L2 is able to repair some errors, but overall has worse repair than CLSVAE. Its strong regularization, optimized mostly for detection, leads to higher distortion and loss of detail (see Figure 1). Its higher standard error (erratic repairs) is due to not being able to distinguish between clean and dirty patterns. VAEGMM does not do well in repair. This is likely due to it being better suited for classification or clustering tasks.

heading: 6 CONCLUSION
text: We have proposed a novel semi-supervised VAE (CLSVAE) for outlier detection and automated repair, in the presence of systematic errors. Our model exploits the fact that systematic errors are predictable by high capacity models, unlike random errors. Thus, CLSVAE partitions the latent space into two subspaces: one for clean patterns, and another for dirty ones. Inliers are only modelled by the clean pattern subspace, whilst outliers use both subspaces. We encourage low mutual information between these subspaces through a penalty, improving performance stability. Empirically this encourages higher fidelity repairs by the model, without human in the loop or other post-processing. We show CLSVAE only needs a small trusted set, requiring the user to label less data. We show that unsupervised models may not be able to distinguish between clean patterns and systematic errors, and strong regularization leads to a lower quality repair. Experimentally, CLSVAE showed superior repair quality and performance compared to other semi-supervised models, including a SotA disentanglement model. Experiments were carried out on image data, and in the future, we would like to explore other types of systematic errors and data types (tabular, sensor, natural language).

heading: B CVAE (FULLY SUPERVISED)
text: The CVAE (Conditional VAE) model (Kingma et al., 2014; Sohn et al., 2015) needs the entire train set to be labelled, i.e. like an oracle where the ground-truth for y is observed. This is generally not possible in practice, particularly for our model, as it would be cheating under the problem formulation. But it does provide a valuable baseline about what kind of performance this type of model, given all advantages, can obtain. A semi-supervised version of this model is in fact the M2 model (Kingma et al., 2014), which will always register lower performance than its supervised counter-part the CVAE. The ELBO for this model is
log p(x, y) ≥ L(x, y) = Eqφ(z|x,y) [pθ(x|z, y)]−DKL (qφ(z|x, y)||pσ(z)) , (20)
where θ and φ are the decoder and encoder parameters. Further, note that in a standard CVAE we usually have p(z) = N (z|0, I). However, we found empirically that stronger regularization for the encoder was needed for CVAE to do well in repair. Otherwise, it would often overfit to the systematic errors. Hence we used a modified prior pσ(z) = N (z|0, σ2I) where σ ∈ [0.1, 0.8], thus enforcing stronger regularization (lower message capacity) on latent code z. Even though it is not standard, we also tried qφ(z|x) but always ended up with a bit worse performance than qφ(z|x, y). In terms of outlier detection, we used an anomaly score based on the reconstruction (negative loglikelihood) of the CVAE model. This type of anomaly score is typical in VAE type models. For each
d ∈ {0, ..., D} features, we have
A(x) = − D∑ d log pθ(xd|µφ(x), y = 1), x ∈ X , (21)
where the outlier set of points is O = {x ∈ X | A(x) ≥ γ}, where γ is decided by the user. We followed the CVAE attribute manipulation strategy from (Klys et al., 2018) when defining repair of an outlier. We encode using the original label (i.e. y = 0), then switch label in latent space (i.e. y = 1) and decode it. In other words, for repair we have
x̂ = µθ(µφ(x, y = 0), y = 1), x ∈ O. (22)
C VAEGMM: ALTERNATIVE TO M2 (SPARSE SEMI-SUPERVISED)
This formulation is based on Willetts et al. (2020), specifically GM-DGM (Gaussian Mixture Deep Generative Model). The work focused on severe sparse semi-supervision (some classes do not even have labels). This formulation is relevant, in sparse semi-supervision, since posterior collapse of q(y|x) can occur for the original M2 model (Kingma et al., 2014). This in turn leads to poor performance in classification and clustering tasks. Hence, one can think of Willetts et al. (2020) as an improvement on M2 model for the issue above. We now present the version of GM-DGM from Willetts et al. (2020) applied to our problem.

heading: C.1 GENERATIVE MODEL
text: The generative model is defined as
pθ(x|z)pτ (z|y)pα(y), (23)
where pα(y) = Bernoulli(y|α), (24)
pτ (z|y) = y N (z|0, σ2y=1I) + (1− y) N (z|0, σ2y=0I), (25)
pθ(x|z) = N (x|µθ(z),Σθ(z)), (26)
which defines a 2-component Gaussian Mixture Model w.r.t. z, and τ = {σy=1, σy=0} and σy=1 < σy=0. A sensible range for τ is: σy=1 = [0.2, 1]; σy=1 = [2, 8]. This prior defines a 2-component Richter distribution (Gales & Olsen, 1999), expressing a type of heavy-tailed distribution on z. This type of distribution is common in robust statistics (Eduardo et al., 2020), as a means to robustify (regularize) the parameters of a model to outliers. The main idea here is that inlier samples z|y = 1 will be regularized more strongly compared to outliers z|y = 0, reflected by σy=1 < σy=0. We initially tried learning the parameters of the Gaussian components. However, we obtained much better results by fixing their parameters (i.e. mean vector and covariance matrix), particularly when it came to outlier detection with smaller trusted sets. Lastly α reflects the initial belief on the fraction of clean data.
C.2 VARIATIONAL MODEL
As for the variational model, we use the standard formulation provided in Willetts et al. (2020), which is the same as the original M2 model (Kingma et al., 2014). So we have
qφ(z|x, y)qφ(y|x), (27)
qφ(y|x) = Bernoulli(y|πφ(x)), (28)
qφ(z|x, y) = N (z|µφ(x, y),Σφ(x, y)), (29)
where πφ(x), µφ(x, y) and Σφ(x, y) are neural networks.

heading: C.3 TRAINING LOSS
text: We can write the ELBO (Evidence Lower Bound) for the unsupervised part of the dataset, i.e. x ∈ Xu, where labels y are not observed as
log p(x) ≥ L(x) = ∫ qφ(z|x, y)qφ(y|x) log
pθ(x|z)pτ (z|y)pα(y) qφ(z|x, y)qφ(y|x) dzdy (30)
L(x) =Eqφ(y|x)qφ(z|x,y) [log pθ(x|z)]− (31) − Eqφ(y|x) [DKL(qφ(z|x, y)||pτ (z|y))]−DKL(qφ(y|x)||pα(y)).
One can rewrite the above ELBO in a different way as L(x) = πφ(x)G(x, y = 1) + ( 1− πφ(x) ) G(x, y = 0)−DKL(qφ(y|x)||pα(y)), (32)
where G(x, y) = Eqφ(z|x,y) [log pθ(x|z)]−DKL(qφ(z|x, y)||pτ (z|y)), (33) and πφ(x) = qφ(y = 1|x) is the probability a data instance is clean.
The ELBO for the labelled part of the dataset (i.e. trusted set), where we have (x, y) ∈ Xl ,Yl , is written as
log p(x, y) ≥ L(x, y) = G(x, y) + log pα(y). (34)
Given the aforementioned definitions for the model and ELBOs (labelled and unlabelled), the overall training loss is the same as eq. (13) from the main paper.

heading: C.4 OUTLIER DETECTION AND REPAIR PROCESS
text: After training the model, then we can proceed with the outlier detection and repair process. First, we need to define a score for use in detection. So at test time we use
A(x) = − log qφ(y = 1|x), x ∈ X , (35) which is based on the probability of the instance being an inlier. Usually, the user defines a threshold γ to classify instances into inliers or outliers. The set of outlier instances is given by O = {x ∈ X | A(x) ≥ γ}, where γ ≥ 0. Assuming qφ(y = 1|x) is somewhat calibrated, then one can use a γ ≈ − log(0.5). After obtaining the setO, and conditioning on y = 1, we produce a repair using
x̂ = µθ(µφ(x, y = 1)), x ∈ O. (36)

heading: D CCVAE (SEMI-SUPERVISED DISENTANGLEMENT)
text: In Joy et al. (2020) the latent space z is split into two subspaces: the style (or agnostic) part z\c that is meant to model unlabelled patterns present in the instance; the characteristics part zc that is meant to model the labelled attributes (characteristics). Formally, we have z = [ zc; z\c ] where [ ; ] is the concatenation operation. In our case, following Joy et al. (2020) and their implementation code, we modelled zc as single variable (univariate) for our binary label: y = 1 for inliers and y = 0 for outliers.

heading: D.1 GENERATIVE MODEL
text: From Joy et al. (2020), using a similar notation, we have p(x, z, y) =pθ(x|z)pψ(zc|y)p(z\c)p(y), (37)
p(y) =Bernoulli(y|α), (38) p(z\c) =N (z\c|0, I), (39)
pψ(zc|y) =N (zc|µψ(y),σ2ψ(y)), (40) where pθ(x|z) is a neural network based decoder, and θ are the parameters of the neural network. In this case α has the same meaning as in CLSVAE, where it expresses the prior belief on the fraction of inliers present in the dataset. Note that the above model expresses a two-component mixture model on zc, with y as gating variable. In fact we have a mean and variance for each y value as it pertains to zc, i.e. µψ(y = 1) and σψ(y = 1) for inliers and similar for outliers (y = 0).
D.2 VARIATIONAL MODEL
The variational distribution is factorized as q(y,z|x) =qϕ,φ(y|x) qϕ,φ(z|x, y), (41) qϕ(y|zc) =Bernoulli(y|πϕ(zc)), (42) qφ(z|x) =N (z|µφ(x),σ2φ(x)), (43)
qϕ,φ(y|x) = ∫ qϕ(y|zc)qφ(z|x)dz, (44)
qϕ,φ(z|x, y) = qϕ(y|zc)qφ(z|x)
qϕ,φ(y|x) , (45)
where qϕ(y|zc) and qφ(z|x) are neural network based encoders, with ϕ and φ as neural network parameters.

heading: D.3 TRAINING LOSS
text: The training loss used in Joy et al. (2020) is defined by the ELBO, with one term for the labelled data (trusted set) and a second term the unlabelled data. The total ELBO is written as∑
x∈Xu
LCCVAE(x) + ∑
(x,y)∈Xl×Yl
LCCVAE(x, y), (46)
where the labelled part of the ELBO is LCCVAE(x, y) = Eqφ(z|x) [ qϕ(y|zc) qϕ,φ(y|x) log ( pθ(x|z)pψ(zc|y) qϕ(y|zc)qφ(z|x) )] +β log qϕ(y|x) + log p(y), (47)
where β is the hyperparameter controlling amount of up-sampling and importance relative to other terms, like in CLSVAE. In Joy et al. (2020), for their application, they found that setting β = 1 brought good results and found no need to tune it further. In our case, we found that we obtained better performance by using larger values for β. This is probably because the application is different, i.e. outlier detection and subsequent repair, and since the trusted set (labelled set) is quite small. The unlabelled part of the ELBO is not as important for our analysis here, please see the original paper.

heading: D.4 OUTLIER DETECTION AND REPAIR
text: After training the model, we proceed with outlier detection and automated repair. We need to define a score for use in detection, and for that we use the classifier given by the variational model. As such, we have A(x) = − logEqφ(z|x) [qϕ(y = 1|zc)] , x ∈ X , (48) which is the approximate negative log-probability of an instance being an inlier. The user may define a threshold γ to classify instances into inliers or outliers. The set of outlier instances is given by O = {x ∈ X | A(x) ≥ γ}, where γ ≥ 0. Assuming the classifier is somewhat calibrated, then one can use the a γ ≈ − log(0.5). From the perspective of CCVAE, the repair of an outlier instance is just attribute manipulation via zc subspace. Once the appropriate zc is found, then one uses the decoder for reconstruction obtaining a repair. Under our problem definition, automated repair is very much like conditional generation as seen in Joy et al. (2020), where samples for zc are drawn from the conditional prior whilst reusing z\c obtained from encoding the outlier instance. Specifically, we have z = [ zc ∼ pψ(zc|y); z\c ] , where depending on the y value it forces the generated samples to have, or not have, the presence of the attribute (e.g. inlier / outlier). In our case, we are interested in automated repair, and thus limiting human interaction apart from building the trusted set. That means that exploring zc with user interaction to pick the best repair (reconstruction) for each outlier instance is not realistic. So we wish to obtain the most likely reconstruction under y = 1, i.e. the maximum a posteriori, thus defining an automated repair for the outlier instance. Hence, the repair is given by
x̂ = µθ ([ µψ(y = 1);µφ(x)\c ]) , x ∈ O, (49)
where µψ(y = 1) is the mean for the inlier component of pψ(zc|y), and µφ(x)\c is the mean of qφ(z\c|x), which excludes the characteristic (labelled) latent subspace.

heading: E DISCUSSION ABOUT CLASSIFIER IN CLSVAE
text: We found that using qφy (y|zc, zd) leads to better performance in more challenging scenarios, e.g. smaller trusted set or when higher data corruption is present. One could have used qφy (y|zd), which works, but it is still worse than the former. This is mostly because zc provides important context on the clean patterns present in the instance, which in turn allows for zd to contain less information about clean patterns. Hence, zd will be able to focus more on the dirty patterns, and so the mutual information (MI) between zc and zd will be lower.
So the idea of using the stop gradient operator in qφy (y|zc, zd) = Bernoulli(y|πφy ([sg(zc); zd]) has to do with preventing zc from having dirty pattern information. By using sg(zc), when executing back-propagation, we prevent the gradient flow from qφy (y|zc, zd) to influence the learning of parameters φc. As such, φc will only be affect by gradients stemming from the reconstruction of inlier instances, and not the decision on y. This trick is useful early on in the training, stabilizing the outcome of the optimization procedure, leading to better repair. Generally, only a few cases benefit from this trick, and usually when small trusted sets are used.
Further, qφy (y|zc, zd) allows for the cross-entropy loss used for the trusted set to directly bias the latent space. This means labels y ∈ Yl can help in separating the latent inlier and outlier representations (manifolds). Other works have tried biasing the latent space in the same fashion, e.g. (Locatello et al., 2019; Ilse et al., 2020; Joy et al., 2020).

heading: F EMPIRICAL DISTANCE CORRELATION
text: For a random data batch of n samples, such as (Z,Z ′) = {(zk, z′k) : k = 1, ..., n}, one can define the empirical estimate for the distance correlation between multivariate random variables z ∈ Rq and z′ ∈ Rp. Note that q and p can be of different dimensions, i.e. q 6= p. Given this data batch, using the definition from (Székely et al., 2007), we first define Akl and Bkl as
akl = ||zk − zl||2 ,
āk· = 1
n n∑ l=1 akl , ā·l = 1 n n∑ k=1 akl , ā·· = 1 n2 n∑ k,l=1 akl ,
Akl = akl − āk· − ā·l + ā·· , and similarly, using bkl = ||z′k − z′l||2 we define
Bkl = bkl − b̄k· − b̄·l + b̄·· .
Now we are ready to define the empirical distance covariance as
dCovn(Z,Z ′) = 1
n2 n∑ k,l=1 AklBkl ,
and the following empirical distance variances for each random variable as
dVarn(Z) = 1
n2 n∑ k,l=1 A2kl , dVarn(Z ′) = 1 n2 n∑ k,l=1 B2kl .
Finally, combining the above measures we can write the distance correlation as
dCorrn(Z,Z ′) == dCovn(Z,Z ′)√
dVarn(Z) dVarn(Z ′) ,
where 0 ≤ dCorrn(Z,Z ′) ≤ 1. Note that if dCorrn(Z,Z ′) = 0 then z and z′ are statistically independent random variables. Otherwise, if dCorrn(Z,Z ′) = 1 then they are strongly correlated. The distance correlation measure can capture non-linear dependencies, whilst the more common Pearson correlation can only capture linear dependencies. Further, a value of 0 for Pearson correlation does not imply independence, unlike in distance correlation which it does.

heading: G EXPERIMENTS: DETAILS ON DATASETS, CORRUPTION, AND TRUSTED SETS
text: Here we have a detailed description of the datasets and their corruption. For each noise level corruption in NLsize = [15%, 25%, 35%, 45%], we instantiate five different examples of the same dataset using different random seeds. For each of those examples we build several trusted sets using the sizes in TSsize = [5, 10, 25, 50], where labelled instances of smaller trusted sets are reused in bigger ones. The models are run on each example, for each trusted set, and results are then averaged.
Synthetic-Shapes This is a synthetic image dataset. It is meant to test the models in a simpler setting as it relates to the clean dataset. We treat the pixel values as a Bernoulli variable. The underlying clean dataset is composed of four different shapes (classes): a circle, a rectangle, an ellipse and a triangle. The shapes are filled by white pixels and the background is black. For each instance, the shape is placed uniformly at random inside the 28 × 28 black background. The systematic errors have four types, all are white lines that cross the square image from one side to another side. We have 4 fixed-in-place lines (two diagonal, one vertical, and one horizontal), affecting the same pixels, and sometimes intersecting with the shapes. Hence, we have a total of eight underlying classes for trusted set constitution, i.e. four data and four systematic error classes.
The size of the images is 28 × 28, and the pixel values are binary {0, 1}, i.e. black and white. Overall, we have a dataset size of N = 5000, with the following split: train (80%), validation (10%), test (10%). Given the eight underlying classes, the size dataset N , and TSsize, we have the trusted set range: [40, 80, 200, 400] total number of labelled instances, which corresponds to [0.8%, 1.6%, 4%, 8%] of the entire dataset
Frey-Faces This is a gray-scale image dataset consisting of the same person with different facial expressions. We treat pixel values as continuous. In terms of data classes, we only have one (monolithic), since no labels for the expressions are provided. We have four systematic error classes, which consist of four randomly uniformly placed squares of 6 × 6 pixels. The place and color of these squares is defined by the random seed, when the corruption example is created. After that, always the same features are affected. Hence, in total we have five underlying classes for trusted set constitution, i.e. one data and four systematic error classes.
The size of images is 28× 20, and the pixel values range from [0, 256], i.e. gray-scale. The size of the entire dataset is N = 1965, with the following split: train (80%), validation (10%), test (10%). Given the above and TSsize, the trusted set range is: [25, 50, 125, 250] total number of labelled instances, which corresponds to [1.3%, 2.5%, 6.4%, 12.7%] of the entire dataset.
Fashion-MNIST This is a gray-scale image dataset, which consists of images of different types of clothing and accessories from an online merchant. There are 10 existing data classes, provided with the dataset. We have 4 fixed-in-place lines (two diagonal, one vertical, and one horizontal) where the color (black or white) for each depends on the random seed. Then we have 4 squares of size 6× 6 placed randomly uniformly with random color, dependent on random seed. Hence, we have a total of 18 underlying classes for trusted set constitution, i.e. 10 data and 8 systematic error classes.
The size of the images is 28 × 28, and the pixel values are continuous with range [0, 1], i.e. grayscale. The original train set is of size 60000 instances, which we split for our actual train set of 54000 (90%) and validation set of 6000 (10%). We use the same test set of 10000 instances, soN = 70000. Given TSsize and the train set size N , the trusted set range is: [90, 180, 450, 900] total number of labelled instances, which corresponds to [0.12%, 0.25%, 0.64%, 1.28%] of the entire dataset.

heading: H EXPERIMENTS: MODEL ARCHITECTURES, HYPER-PARAMETERS AND OPTIMIZATION

heading: Hyperparameter Selection
text: In our experiments, we tuned model hyperparameters according to outlier detection performance, which means highest AVPR (average precision). This was evaluated on the trusted set, the only labelled part of the dataset. Often, we would look at the repairs (reconstructions) offered by each model for the trusted set. This way we confirm that the repair process is reasonable enough, and no adjustment is needed on the hyperparameters side. In the case of VAE-L2 we had to not just
account for AVPR in the trusted set, but also check repair performance via SMSE (standardized mean squared error). This is because strong regularization, higher `2 coefficient, often leads to better outlier detection performance, but that comes at the cost of repair quality due to the VAE collapsing to mean behaviour. For each model, the hyperparameter search was carried out for each dataset at a noise level of 35%, and with 25 samples per class.
• VAE-L2 In Synthetic-Shapes: 200 epochs; KL divergence annealing used; `2 coefficient is 35.0. In Frey-Faces: 300 epochs; KL divergence annealing used; `2 coefficient is 100.0. In Fashion-MNIST: 100 epochs; KL divergence annealing used; `2 coefficient is 100.0.
• VAEGMM In Synthetic-Shapes: 200 epochs; KL divergence annealing used; fraction of clean data α = 0.6; (trusted set) up-sampling coefficient β = 1000; σy=1 = 0.9 and σy=0 = 5.0. In Frey-Faces: 300 epochs; KL divergence annealing used; fraction of clean data α = 0.6; (trusted set) up-sampling coefficient β = 1000; σy=1 = 0.6 and σy=0 = 5.0. In Fashion-MNIST: 100 epochs; KL divergence annealing used; fraction of clean data α = 0.6; (trusted set) up-sampling coefficient β = 100; σy=1 = 0.5 and σy=0 = 5.0.
• CVAE In Synthetic-Shapes: 200 epochs; KL divergence annealing used; σ = 0.5. In FreyFaces: 300 epochs; KL divergence annealing used; σ = 0.2. In Fashion-MNIST: 100 epochs; KL divergence annealing used; σ = 0.5.
• CCVAE In Synthetic-Shapes: 200 epochs; fraction of clean data α = 0.6; (trusted set) up-sampling coefficient β = 50000.0. In Frey-Faces: 300 epochs; fraction of clean data α = 0.6; (trusted set) up-sampling coefficient β = 10000.0. In Fashion-MNIST: 100 epochs; fraction of clean data α = 0.6; (trusted set) up-sampling coefficient β = 250000.0.
• CLSVAE In Synthetic-Shapes: 200 epochs; fraction of clean data α = 0.6; KL divergence annealing used; (trusted set) up-sampling coefficient β = 1000.0. σ = 0.5; σc = 0.5; σd = 5.0; distance correlation (DC) penalty used; λt annealing ratio of 0.5 (DC penalty); λT maximum value of 100.0 (DC penalty). In Frey-Faces: 300 epochs; fraction of clean data α = 0.6; KL divergence annealing used; (trusted set) up-sampling coefficient β = 1000.0. σ = 0.6; σc = 0.2; σd = 5.0; distance correlation (DC) penalty used; λt annealing ratio of 0.5 (DC penalty); λT maximum value of 1000.0 (DC penalty). In Fashion-MNIST: 100 epochs; fraction of clean data α = 0.6; KL divergence annealing used; (trusted set) up-sampling coefficient β = 100.0. σ = 0.1; σc = 0.2; σd = 5.0; distance correlation (DC) penalty used; λt annealing ratio of 0.5 (DC penalty); λT maximum value of 1000.0 (DC penalty).
• CLSVAE-NODC Same hyperparameter options as CLSVAE but without the distance correlation penalty.
Note the annealing ratio above for the KL divergence and for the DC penalty is inspired by Fu et al. (2019). We use only one cycle (monotonic), and the R (or ratio) is the proportion used to increase the penalty coefficient (or KL term coefficient) – e.g. 0.5.

heading: Optimization
text: We used the PyTorch framework to code all our models, and trained on a GeForce TITAN X GPU. All models were trained using the Adam optimizer, with an initial learning rate of 0.001.

heading: Model Architectures
text: For continuous type data, i.e. Fashion-MNIST and Frey-Faces, we used the Gaussian distribution as the likelihood of each pixel in the reconstruction loss. The variance of the Gaussian distribution is shared amongst all the pixels in the image, and it is learnt as a parameter of the model. This was done for all models. For binary type data, i.e. Synthetic-Shapes, we treated each pixel as a Bernoulli variable, and used the log-likelihood of this distribution for each pixel in the reconstruction loss. This was done for all models.
We used very similar encoder and decoder architectures for all models, so as to be fair and the results comparable. In the case of CLSVAE we used two encoders, one for the clean subspace zc, and the other for the dirty subspace zd. This architecture yielded better results for us in terms of repair in the trusted set. CLSVAE architecture can be seen in Table 5 for the encoders and decoder, and the classifier can be seen in 7. In Table 2, we see the neural architecture for the encoder and decoder of VAE and VAEGMM. The classifier architecture for the VAEGMM can be seen in Table 6. In Table
3, we find the architecture for the encoder and decoder of CVAE. In Table 4, we find the architecture for the encoder and decoder of CCVAE. The classifier architecture of CCVAE is just zc multiplied by a weight parameter plus a bias parameter, and then a sigmoid non-linearity is applied – like in Joy et al. (2020).

heading: I LARGER VERSION OF RECONSTRUCTIONS (REPAIRS) FIGURE

heading: J RESULTS FOR ALL NOISE LEVELS AND TRUSTED SET SIZES (ALL SWEEPS)

heading: K.1 SYNTHETIC-SHAPES

heading: K ADDITIONAL RECONSTRUCTIONS (REPAIRS) FOR ALL DATASETS

heading: K.2 FREY-FACES

heading: K.3 FASHION-MNIST

heading: L TESTING COMPRESSION HYPOTHESIS: ENTROPY OF CLEAN VS. CORRUPTED DATA
text: In this section, we experimentally compare the entropy of clean (without corruption) training data, and the entropy of corrupted training data. A larger entropy means that a dataset has larger variance overall. Note that in the setup of our problem, i.e. repairing systematic errors (see Section 3), only corrupted data is used for training. In Figure 22, for Frey-Faces, we compare the estimated entropy of clean training data against one that has been corrupted (35 % noise, corruption as in Table 1).
We estimate the entropy by first training a standard VAE model on the dataset (clean or corrupted), and then after training, we compute a tight bound on the marginal log-likelihood of that dataset. We compute this tight bound via IWAE estimator (importance weighted autoencoders, (Burda et al., 2016)), where we use K = 250 samples. Note that entropy is H(x) = −Epθ(x) [log pθ(x)], and hence marginal log-likelihood is just −H(x). We vary the dimension of VAE latent space (z) in order to see how well the model can learn the training data. For this experiment, the VAE is not regularized.
The main idea is to prove empirically that corrupted data has larger variance than clean data, and hence has more information to be modelled. Specifically, when using the type of masking corruptions used in our experiments. If a dataset has larger estimated entropy, then the data can be seen has having larger variance. A dataset with larger variance is a dataset with more diversity in terms of the patterns it contains, and hence it has more information to be compressed (or modelled). In particular, in a VAE this means more capacity (e.g. larger latent space) is needed to model a larger variance dataset.
Our claim, supported in literature (Eduardo et al., 2020; Ruff et al., 2019), is that a dataset that has been corrupted has larger variance because the added outliers (e.g. systematic errors) increase data pattern diversity, hence increasing entropy. Looking at Figure 22, we see that overall the entropy of corrupted data is larger than clean data, for all sizes of z. Hence, corrupted data has larger variance than clean data. For the smaller dimensions of z, in range [2, 10] units, we see that the VAE has less trouble learning the clean data compared to the corrupted data. This is also evidenced by how more quickly the entropy decreases for clean data relative to corrupt data, as we increase the latent space size in [2, 10]. We conclude that a VAE only needs a smaller latent space (subspace) to model clean data (inliers), and that corrupted data needs a larger latent space to be modelled properly.

Review summaries:
This paper presents a neural network model based on Variational Autoencoders (VAE) that learn an implicit representation separating outliers with systematic "errors" from inliers using a small labelled subset of the training data set (trusted set). More specifically, clean data and recurring systematic errors are represented in separate latent subspaces, where outliers are supposed to be a (linear) combination of such clean and "dirty" patterns. After training the model, there are two tasks: outlier detection and automated repair, removing the "dirty" pattern(s) from the detected outliers. 

The paper presents a Clean Subspace Variational Autoencoder (CLSVAE) model for systematic outlier detection. Building on VAE for latent variable modelling, the paper proposes a semi-supervised learning to infer the potential outlier.   The author made the following assumptions: 1. outliers exist therein, the inliers are still the majority; 2. Outliers are systematic with predictable recurring patterns; 3.  Compression hypothesis. Inlier data can be compressed further than outlier data. 4. Outliers are a combination of clean and dirty patterns.  On the model side, a mixture model is proposed parametrized by Bernoulli mixing weight and decoder p(x|z), while the encoder q(z|x) is applied for sampling latent variable z. A standard ELBO is formulated to maximise the log likelihood for both y known and unknown scenarios. For outlier repair, the author proposed to minimize the correlation between noise and clean signal.  Experimentally, the author evaluate two tasks: outlier detection, and automated repair on Frey-Faces3 , Fashion-MNIST (Xiao et al., 2017), Synthetic-Shapes.

This paper develops a deep learning based method for detecting and repairing systematic outliers in data (before performing model learning on the data).  The method is called Clean Subspace Variational Autoencoder (CLSVAE). CLSVAE takes as input a trusted set provided by the user (the trusted set contains a set of inliers and outliers, but not requiring information about how the outliers are corrupted), and seeks to learn a variational autoencoder where the latent code is a concatenation of two subspaces, a clean subspace and a dirty subspace. CLSVAE is semi-supervised because the labelled trusted set is much smaller than the unlabelled set.  The paper performs outlier detection and repair benchmarks on three image datasets with systematic outliers, and shows that CLSVAE achieves favorable results.