Repairing Systematic Outliers by Learning Clean Subspaces in VAEs
Authors: Simao Eduardo
Kai Xu
Alfredo Nazabal
Charles Sutton
Venue: ICLR 2022 Submitted (2022)
Keywords: variational autoencoder
deep generative models
outlier detection
data repair

Abstract: Data cleaning often comprises outlier detection and data repair.
Systematic errors result from nearly
deterministic transformations that occur repeatedly in the data, 
e.g. specific image pixels being set to default values or watermarks.
Consequently, models with enough capacity easily overfit to these
errors, making detection and repair difficult.
Seeing as a systematic outlier is a combination of patterns of a clean instance
and systematic error patterns, our main insight is that inliers can be
modelled by a smaller representation (subspace) in a model than outliers.
By exploiting this,
we propose \emph{Clean Subspace Variational Autoencoder (CLSVAE)}, 
a novel semi-supervised model for detection and automated repair of
systematic errors.
The main idea is to partition the latent space and model inlier and 
outlier patterns separately.
CLSVAE is effective with much less labelled data compared to previous related
models, often with less than 2\% of the data.
We provide experiments using three image datasets in scenarios with
different levels of corruption and labelled set sizes, comparing to relevant baselines.
CLSVAE provides superior repairs without human intervention, 
e.g. with just 0.25\% of labelled data we see a relative error 
decrease of 58\% compared to the closest baseline.

Under review as a conference paper at ICLR 2022
REPAIRING SYSTEMATIC OUTLIERS BY LEARNING CLEAN SUBSPACES IN VAES
Anonymous authors Paper under double-blind review
ABSTRACT
Data cleaning often comprises outlier detection and data repair. Systematic emrors result from nearly deterministic transformations that occur repeatedly in the data, e.g. specific image pixels being set to default values or watermarks. Consequently, models with enough capacity easily overfit to these errors, making detection and repair difficult. Sceing systematic outlier is combination of patterns of a clean instance and systematic error patterns, our main insight is that inliers can be modelled by smaller representation (subspace) in model than outliers. By exploiting this, we propose Clean Subspace Variational Autoencoder (CLSVAE), novel supervspe model for detection and automated repair of systematic errors. The main idea is to partition the latent space and model inlier and outlier patterns separately. CLSVAE is effective with much less labelled data compared to previous related models, often with less than 2% of the data. We provide exper- iments using three image datasets in scenarios with different levels of corruption and labelled set sizes, comparing to relevant baselines. CLSVAE provides supe- rior repairs without human intervention, e.g. with just 0.25% of labelled data we see relative error decrease of 58% compared to the closest baseline.
INTRODUCTION
Often practitioners have to deal with dirty datasets before they can start applying machine learning (ML) models. We focus on datasets where some instances have been corrupted by noise, producing outliers. The corresponding clean instances prior to corruption are called inliers, as well as any other instances that have not been corrupted. Because the presence of outliers can degrade the performance of MI methods (Krishnan et al. 2016 Liu et al. 2020), a standard option is to resort to a data cleaning pipeline before applying any model. This pipeline includes two key tasks: i) outlier detection (Ruff et al. 22221, detecting all outliers; ii) repair those outlier instances (Neutatz) et al. 2021 Wan et al., 2020), recovering the underlying inlier instance. Ultimately, the goal is to propose method that performs these steps automatically. i.e. automatic detection and repair.
Generally, we can consider two types of errors present in an outlier: random or systematic TTy- [or 1997 Liu al. 2020). Random errors corrupt each instance independently, and feature value changes are sampled from an unknown distribution. This noising cannot be replicated in a repeatable manner. For continuous features, a common example of this type of error are those well-modelled by additive noise with zero-mean. Systematic errors result from deterministic, or nearly deterministic, transformations that occur repeatedly in the data. Examples of systematic errors include watermarks or deterministic pixel corruption (e.g. artifacts) in images; additive offsets or replacement by default values (e.g. NaN) in sensor data; deterministic change categories (mislabelling) or of name for- mats in categorical features in tabular data. Usually, the same features are affected, but not always. In most cases, this noising can be replicated.
These two types of errors have different impact when it comes to models performing detection or repair. Random errors do not show a distinct pattern across outliers and thus are not predictable. As a result, unsupervised models using regularization
data reweighting (Zhou & Paffenroth) 2017]] Akrami et al.) 2019: Eduardo et al. (020)) can avoid overfitting to random errors. On the contrary, a systematic error shows pattern across outliers, a result of the (nearly) deterministic transforma- tion, making them predictable (Liu et al., 2020). This property makes higher capacity models (e.g.
deep learning) prone overfitting these errors, even in the presence of regularization. As

---

Under review as a conference paper at ICLR 2022
sult, models for outlier detection and repair in the presence of systematic errors more easily conflate outliers with inliers. In this work we study how to develop a model for data cleaning robust to the effect of systematic errors.
One solution is to provide some supervision, so the model can distinguish between inliers and out- liers with systematic errors. This supervision can be provided in different forms, such as logic rules (Rekatsinas et al., 2017) or programs (Lew et al. (02T) describing the underlying clean data. How- ever, these may require expert knowledge or substantial effort to formalize. Hence, is easier and less time consuming for the practitioner to simply provide a trusted set as form of supervision. A trusted set is small labelled subset of the data, which can be used to train a method for detection and repair. The user labels the instances either inlier or outlier, providing a few examples (e.g. 10 instances) per type of systematic error to repair. Overall, this might correspond to less than 2% of the entire dataset (sparse semiisupervisioo). Equally important, labelling does not require the user to manually repair the instances in the trusted set.
Deep generative models (DGM) have high capacity (flexible) and thus can easily overfit to system- atic errors. This motivates us to propose a method for detection and repair of such errors based on a semi-supervised DGM, which to the best of our knowledge has not been explored. We postulate that inlier data needs a smaller representation relative to outlier data when being represented by DGMs which we call the compression hypothesis. In addition, claim that outliers are combination of representation descrihing the inlier portion of an instance, and a representation describing the type of systematic error.
Given these insights, we propose Clean Subspace Variational Autoencoder (CLSVAE), a novel semi- supervised model for detection and automated repair of systematic errors. This model deviates from a standard VAE (Kingma & Welling 2014), in two ways. First, the latent representation is parti- tioned into two subspaces: one that describes the data if it were an inlier (clean subspace), and the other that describes what systematic error (if any) has been applied (dirty subspace). The model is encouraged to learn a disentangled representation using a simple yet effective approach: for outliers the decoder will use clean subspace concatenated with the dirty subspace; in turn, for inliers the decoder will reuse the same clean subspace but concatenated with random noise. Then, at repair time the decoder will only need the clean subspace to reconstruct the underlying inlier. Secondly, we introduce semi-supervision through a trusted set, in so helping the model distinguish between in- liers and outliers with systematic errors. Additionally, to encourage the clean subspace represent inlier data, and the dirty subspace to represent systematic errors, we introduce a novel penalty term minimizing their mutual information (MI). This penalty is based on the distance correlation (DC) (Székely et al. 2007). and it improves model performance (stability) and repair quality. Compared to baselines we provide superior repairs, particularly, we show significant advantage in smaller trusted sets or when more of the dataset is corrupted.
RELATED WORK
For random errors, several works have explored detection AArami et al. 2019 Ruff al. 2019 Liu eet al., 20200 Lai et al. 2019). Some have proposed methods for detection and automated repair of random errors (Eduardo al.
2020 Zhou & Paffenroth 2017 Krishnan et al. 2016). A few works
have explored outlier detection for systematic outliers, both semi-supervised (Ruff et al. 2019) and
2020 but these works do not consider automated repair. unsupervised (Liu et al. 2020 Lai et al.
For tabular data, methods have been proposed that can do repair after detection has been performed; for an overview, see (Ilyas & Chul 2019 Chu et al., 2016). These use probabilistic models melded with logic rules, i.e. probabilistic relational models (Rekatsinas et al. 2017). or user-written pro- grammatic deseriptions of the data, e.g. in a probabilistic programming language (Lew et al.) 2021). They have the potential to capture systematic errors. However, these require the user to provide rules or programs that characterize clean data, which requires manual effort and expertise. In contrast, labelling few inlier and outlier instances to build trusted set may be more user-friendly
The idea of using unsupervised models for outlier detection is not new (Schölkopf et al., 1999 Liu
et al., 2008). Unsupervised removal from dataset of small fraction of instances suspected of being outliers has been explored in Koh et al. (2018): Diakonikolas et al. (2018); Liu et al. 2220)) when used against adversarial errors it is called data sanitization (Koh et al." 2018). However, if
2

---

Under review as a conference paper at ICLR 2022
one wants to repair existing dirty data, common unsupervised approach for moderate corruption is to apply enough regularization or data reweighting to an autoencoder (Eduardo et al. 2020 Zhou & Paffenroth] 2017 Akrami et al. 2019), hopefully repairing the errors at reconstruction. This has proven successful for random errors, though as we show, this is less effective for systematic errors. Often regularization is too strong leading to bad repair quality, since reconstruction is collapsing to mean behaviour - e.g. blurry image samples, missing details.
Attribute manipulation models (Klys et al. 2018 Choi al. 2020) usually rely on extensive labelled data, usually fully supervised. Data cleaning can be seen as an attribute manipulation problem with one attribute: either the instance is an inlier or outlier. Some models like CVAE (Sohn et al. 2015) may use discrete latent variables, instead of continuous, to model attributes. These may lack the capacity to capture diversity in the same attribute (Joy et al. 2020), e.g. distinct types of systematic errors in the data, and thus offer a poor fit to the data. Later, this may result in poor repair.
Disentanglement models, unsupervised (Locatello et al.] 2019. Tonolini et al., 2019) and semi- supervised (Isee et al., 2020 Locatello et 2019 Joy et al., (020), encourage individual (continu- ous) latent variables to capture different attributes of the data instances. theory, a disentanglement model could isolate the inlier / outlier attribute into latent variable, then after training one could find a value for this variable that repairs the outlier. In practice, this is much simpler with semi- supervised models, since we know which variable corresponds the attribute, but these tend to more labelled data than we consider. In either case, these models usually need additional processing, often requiring human in the loop to explore the latent variable and find the best repair. Conversely, once our model is trained, further human intervention is not needed, so that repair is automated.
3 PROBLEM DEFINITION
We assume the user knows that systematic errors have corrupted dataset X, and thus outliers exist therein, but the inliers are still the majority. The user also has idea of what patterns constitute sys- tematic errors, and is able to recognize them. Intuitively, we think of inlier data being characterized by a set of patterns, which we call clean patterns, and those that constitute (systematic) errors, called dirty patterns. An outlier is an instance that has been corrupted by systematic errors, where prior to that would be constituted by clean patterns only. Formally, for an underlying inlier instance, an outlier is defined by æ fer( (m). We define for as a general transformation that corrupts the inlier instance with systematic errors, and not necessarily invertible. Each type of systematic error usually affects specific features of æ in the same predictable way, repeatedly in the dataset. This is unlike random errors, which both the feature and changed values are at random throughout instances.
The user builds small trusted set by labelling few of the inliers and outliers in the train set X, forming the labelled subset X₂ { So we have the dirty dataset X' X2 U Xy, where Xx fIx,, ) is the unlabelled part. The overall size of the train set is N N, + N., Each 2n E Xr is associated with a label ya E {0, 1}. which indicates whether on, is an inlier (yn = 1) or an outlier (yn- = 0). We write V/- = {yr } and thus the trusted set is formally defined by (X, M1). The trusted set should be representative of the inliers and outliers in the data. Note that there is set of different corrupting transformations, i.e. systematic error types, each applied to several instances. Hence, the trusted set should provide at least a few labelled samples per type of systematic error. This is important so as to help the model distinguish between inliers and outliers. In our problem, the labelled portion of the dataset X (trusted set) is significantly smaller than the unlabelled portion Xz, c.g. 0.5% of N.. Given how small the trusted set is, we refer to this sparse semi-supervision
In this work, the main task is to first perform outler detection on X, in order to discover y for each instance The second task is repair those instances that are considered to be outliers (y 0). Specifically, the model needs to provide a repair transformation gr such that a = (x) where a z & (i.e. repair is close enough to the underlying inlier instance). Thus, we want model that performs these two steps automatically, i.e. without a human in the loop.
4 OUR PROPOSAL: CLEAN SUBSPACE VAE (CLSVAE)
In this section, we introduce the generative and variational models for our proposal. Our Variational Autoencoder (VAE) model is motivated by three observations:
3

---

Under review as a conference paper at ICLR 2022
Outliers are systematic. Outlier instances can be described by number of predictable recurring patterns, some of which are considered dirty (e.g. black patch on an image). Our assumption is that the latter patterns can be well represented by a DGM.
Compression hypothesis. Inlier data can be compressed further than outlier data. We assume that less capacity (parameters or variables) is needed by DGMs to represent inliers.
Outliers are combination of clean and dirty patterns. Because clean and dirty patterss produce different visible effects in the instance, the effects of clean and dirty patterns can be modelled separately. Hence, this DGM only requires the parameters (or variables) of the clean patterns to generate a repair.
We call our model the sparse supe-vipe Clean Subspace VAE (CLSVAE). This reflects the fact that our model has a partitioned latent space: a subspace for clean patterns, and subspace for dirty patterns. Thus at test time only the subspace for clean patterns used generate a repair. A standard VAE (Kingma & Welling, 2014), with weight decay, is provided in Annex A for readers unfamiliar with this type of model.
4.1 GENERATIVE MODEL
The main idea is that if the compression hypothesis holds inlier samples (y 1) will have smaller representation relative to outliers (y = 0), whilst outliers need an extra representation to model errors. In our model, 2e will represent inliers, zu will represent the error pattern for outliers, and z, is random noise. The overall latent code is z [ y Ze + (1 - y) za]. where [; defines vector concatenation, leading
latent space that partitioned into two subspaces. If generating an inlier (y = 1) then 2e E RR is by itself responsible for modelling all the clean patterns present in an instance, encouraging a lower dimensional manifold for inlier data. This due to z, being a random noise vector of the same dimension as E IP, devoid of additional information. If generating an outlier (y = 0) both ze and z, are used, where z models the different types of dirty patterns that can be detected in an instance. In fact, using during inlier generation encourages zd to model dirty patterns only, encouraging a higher dimensional manifold for outlier data.
Our generative model is therefore defined by the joint distribution between y and latent subspaces z. and za. The main idea expressed above can be written as two component mixture model for the decoder, see eq. (4), where y is the gating variable. Hence we have
po (x (3,
te Ze ze: v) pa zes Zd, ze: y)Po (ze)Pod (x.),
(1)
where
(v(y) Berouuli(yla),
(2)
Po. Po. (Zo) N2zz00,001 Paa (=a) = _2, o.4) Po, (ze) 0Z
(3)
po Rc, 2d, Ze, v) p= [ze; z.) po [ze: 21))
(4)
where the density po (a: [:; .)) is parameterized by a neural network. We assume inlier data has smaller variance whole than outlier data, we use a a. that is much smaller than σa- Note that ze is just Gaussian random noise"] with < 0g. Hence, after training, the region around 0 (zero mean) for this subspace encourages ( [: -I) to only use ze for reconstruction, obtaining a repaired instance. The parameter reflects the prior belief on the fraction of clean data. Smaller values for a means more data points are rejected when modelling ze, which offers more robustness. So we have o,, Je, Ja and a as hhper- parameters.
4.2 VARIATIONAL MODEL
We consider separate encoders for z. and 2a, and make y depend on z. and za. The idea that parameters of each encoder focus different aspects, i.e. clean or dirty patterns respectively. The model factorizes as
q(Z., Net Ze, x) Io. (yl Zec zz)9s. (x.12))0 (2.)) (z.(z) (z.),
(5)
and
Go. (ze xx) = A NZ up. (z)), 0 (z)), (0.(2al0) = NVzzal (x)), (2)))
(6)
We tried using z. 0 training, however this did work as well setting z, random noise.
4

---

Under review as a conference paper at ICLR 2022
4oh Ze, za) = Bernoulli TT, (lze) zal), 9o. (ze) Po. (ze).
(6 {ug. oe is neural network with parameters de: simi- larly for (M,,()) (-), oe (.)) with Qd. We have 0o (x) and (,)()) being diagonal covariance matri- ces. The distribution for random noise Ze the same as in the generative model. In eq. 7 the
where for distributions in eq.
To, (.) parametrizes the Bernoulli distribution doy (y) Re; Ea), and is neural network with parame- ters By.We We found that using qo, (y) Zey Za) yielded better results than using yo, (y) ) or yo, (y/ a:). Since ze provides important context the clean patterns present, which in turn allows zd to better focus on modelling dirty patterns. In practice, to stabilize the optimization procedure in a few cases, used TT, ([sg( &c); =a]) in go, [y/=e. =d) eq. 7 Note that sg(zz) stands for stop gradient op- erator applied to zz; and this prevents ze from being updated with dirty pattern information early in the training. A more detailed discussion 4o, (yl =e. za) is presented in Annex E
4.3 TRAINING Loss
Our model is trained to maximize an objective function with three terms, which accounts for our semi-supervised setting. The first term ((x) is the evidence lower bound (ELBO) for the unlabelled part of the data. The second term [(x, y) is the ELBO for the trusted set. The third term LwcE y) the weighted cross-entropy loss which ensures that qo, (ylze, za) correctly predicts the trusted set labels y.
The ELBO for the unlabelled (unsupervised) part is
(xx) gBg (* (#.) o ([=o: za)) log (ac [2o)
(8)
+ (1 -o (1zo) znl)) lo (a: (Zci zal) - DEL (gon (u) PPp (v))
DD (1x, (2c| I|P (ze)) ) DEL (zala))
where qo, (z.) Po. (z.) and they cancel each other. The expectations are obtained via Monte-
Carlo (MC) estimation via reparameterization trick (Kingma & Welling 2014 Rezende et al. 2014).
For the trusted set (supervised) part the ELBO is
L(x, 3) =g (2 (z.l=)) (xal:)))___p)p... (=.) po (zei zel) (1 3) logpe((t [Zc)
(9)
log Pa (v) D 1, (1o, (zel (2) (zp/ (ze)) DEL. (Za( (Pa
Lastly, need define the weighted cross-entropy Lwck (Ir, y), which is
LWCE(2: (z, y) y. y. log g(1 1|x)) Wimb- (1 y). log (1 a(y= [1x))
(10)
where
N Ni
Wimb max 1,
No
Vi No. N₁ No,
(11)
and Wimb compensates for trusted set class imbalance x)) and thus we cannot estimate LwCB(x, y) directly. Still, we can minimize an upper-bound
However, do not have access to q(y
Lwce(æ, v) < ÉwCr(æ,, v) =E (z. w) (za us)
log g = 1|zc, za)
(12)
Wimb (1 - log (1 g(v = 1| za))
which is obtained by applying Jensen's 's inequality. Combining the three terms defined above, we minimize the overall loss
I=
1 N
Σ C(x) x
Σ
C(x, y)
+ 9 N
1
Σ
Ewen(a, (13)
(
(x.))
with respect the generative and variational parameters. The hyperparameter 3 value controls the amount up-sampling and importance relative to the other terms, which tends to be moderately high due how small the trusted set is.
Useful when the number of labelled outliers outnumbers the inliers. Such case does not reflect the common dataset composition, i.c. the number of inliers is larger than outliers.
5

---

Under review as a conference paper at ICLR 2022
4.4 DISTANCE CORRELATION PENALTY
Ideally, Ze captures clean patterns only, whilst za captures dirty patterns. Therefore 2, and Zn should have low mutual information (MI). However, in more challenging scenarios, e.g. small trusted set or higher dataset corruption, obtaining this solution may not be guaranteed. Enforcing a constraint encouraging low MI between &e and zd will lead to better model performance and stability in challenging scenarios, improving repair quality.
We would like introduce constraint that minimizes MI between 2. and Zd However, approxi- mating MI properly can be complex. Instead, we use distance correlation (DC) as a surrogate for MI (Székely al.] 2007). which is easier to compute and can measure non-linear dependencies be- tween vector variables. Other works have used DC as surrogate for MI, e.g. (Chen et al.) 2021). Further, DC can also measure dependence between vector variables of different dimensions, which is often the case with &e and zd. For the data batch (Zc, =a) (Ze, Za) where Ze RNxx xg and Za E R N xp, we can define the empirical estimate of DC as dCorrn Ze. Za). The definition of the estimator dCorrn ZZe Za) can be seen in Annex [F Essentially, DC is the standard correlation be- tween the elements of the double centered pairwise distance matrices of each data batch Z. and Zd- The range is 0 I dCorr eZ. Za) S 1, where 0 means variables are independent, and 1 implies that 2e and 2a are strongly correlated.
Enforcing this constraint means adding penalty the model training loss. Hence, reusing the loss defined in eq. (13) we now have
min I+ de dCorrn (Ze, Za), 0er
(14)
where A increases every epoch from until it reaches a maximum value Ay. which is then main- tained. The rate of increase of X and \y are hyper-parametent This strategy is a type of penalty method as used in constrained optimization.
4.5 OUTLIER DETECTION AND REPAIR PROCESS
After training, we proceed with outlier detection and automated repair, as in Section 3] The detec- tion task is to discover the ground-trut labels for each x E X, using inferred labels ÿ. A score A(r)) and threshold y 2 0 are used to get the set of outliers higher A(ær) means the more likely x is an outlier. The inferred label j is obtained as: (inlier) ú 1 x: & O (outlier) ý 0 if æ E O. For our model, we use a score based on the negative log probability of inlier given the latent subspaces, which is
{ar E X A(x) 2 7), where a
A(x)= = log Tæ, (1H+, (z); (x))), x x EEX.
(15)
The threshold can be chosen as ry x - log (0.5) assuming ge, 11. Te. (.) is near calibrated, or it can be user-defined. The repair task is to obtain an inferred reconstruction & from the outlier E O such that it is close to the inlier ground-truth ă. The repair is generated using the most likely reconstruction under our model for y = 1 (inliers), which means only the clean subspace 2. is used. This is the maximum posteriori estimate for VAE, where one approximates (= (x) by qo (ze(zz) and then uses the means of go (ze x), Pe (z.) and (x [zei z..) in the estimate. Hence, we have
a = M (z):
x E O.
(16)
EXPERIMENTS
We evaluate two tasks: outlier detection, and automated repair. Our experiments use three image datasets: Frey-Faces") Fashion-MNIST (Xiao et al. (017). Synthetic-Shapes. Synthelic-Shapes is synthetic dataset built around four different shapes (classes): circle, a rectangle, an ellipse and a triangle. These are colored white and set in black background. We corrupt datasets with synthetic systematic errors, since public real-world datasets with ground-truth repairs and respective labels difficult find, as seen in Eduardo et al. (2020); Krishnan et al. (2016); Liu et al. (2020). We compare our model (CLSVAE) with baselines ranging from completely supervised to unsupervised
http:: /ww.wwww...ooo cs. nyu. edu/ rowei s/ data/ / weeeyrawface...o
"oode for models and experiments will be released upon paper acceptance.

---

Under review as a conference paper at ICLR 2022
Dataset
Data Type
No. Data Classes
No. Error Classes
Error Types
Syntherie-Shupe
28 x binary
(black white)
4
lines
28 28 continuous
Fashion-MNISS
10
lines & 4 squares
(grey greyescale)
28 20 continuous
Frey-Faces
4 squares
(grey scale)
Table I: Description dataset and its corruption.
Evaluation For outlier detection we use AVPR (Average Precision) (Everingham et al., 2013) to measure detection quality, which is a surrogate for the area under the precision-recall curve (Hendrycks & Gimpel, 2016). This metric preferred since it is insensitive to label imbalance, typical in outlier detection. AVPR score is between [0, 1] and higher means better. For automated repair we want to quantify the quality of the repair, for outlier instances. We report the standard- ized mean squared error (SMSE) between pixels of the ground-trud proposed repair. We report SMSE separately for the dirty pixels (those affected by the systematic error) and for the clean pixels (those unaffected). The first measures repair performance, while the latter measures distortion that the repair process causes to clean pixels. In both cases, a lower SMSE means better. Note in the case of binary pixels, the SMSE is just the Brier score, and thus is in [0, 1].
(inlier) instance and that of the
Datasets and Corruption Process In our experiments we take an uncorrupted dataset and inject it with systematic errors. These systematic errors are synthetic, designed to seem like reasonable image corruptions, e.g. occlusion or failing of a camera sensor. The types of systematic errors used across datasets are either lines or squares. Lines (two diagonal, one vertical, and one horizontal) cross the image from side to side, and may have their color set at random (black / white). These lines always affect the same pixels, and have thickness of one pixel. Squares are randomly uni- formly placed, so is their fill-in color, with size 6 x 6 pixels. We use different noise levels so we can study their impact, we use [15%, 25%, 35%, 45%] of dataset. The systematic error corruption process is done by picking uniformly at random an instance, and then applying that systematic error. We use a range of trusted set sizes by defining the number of instances labelled per systematic error class (type) and per data class. The latter is the underlying classes in the dataset, e.g. item labels in Fashion- -NIST. For each class, either systematic error or data class, we provide label y for a few in- stances at random obtaining a trusted set. We use the range TSae = [5, 10, 25, 50) labelled samples per class, which results in different trusted set sizes depending on number of classes. Particularly, we have the trusted set ranges: Synthetic-Shape
with [40, 80, 200, 400] total samples; Frey-Faces with with [90, 180, 450, 900] total samples. We create
[25, 50, 125, 250] total samples; Fashion-MNISS five examples (different random seeds) per noise level and per trusted set size, and train the models on them. The results are then averaged. Table [ describes datasets and their corruption (error types), number of systematic errors and data classes. More details on this experimental setup in Annex G
Comparative Methods Our baselines are VAEs since most of the relevant work in sparse semi- supervision is of this type (Ilse et al. 2020 Joy et al., 2020). and the task of repair is related with that of manipulating the reconstruction. We have four baselines: VAE-1.2, CVAF, VAEGMM, CC- VAE. For details on model architccture and hyperparameters see Annex H The VAE-L2 model an unsupervised method tackling the issue of corruption by applying strong regularization (l, regular- ization on weights). VAE-L2 uses the reconstruction likelihood for detection, more details in Annex
A CVAE is the supervised version of the semi-supervised M2 model (Kingma et al., 2014). and should have better repair quality than M2. CVAE uses the reconstruction likelihood as an detection score. We found a smaller variance for p(z) be beneficial, for details see Annex B VAEGMM, based on (Willetts et al. 2020), is an improved version of M2 for the sparse semi-supervision setting. In this setting the M2 model tends to have posterior collapse issues with q(y)2z). picking one class over others. VAEGMM overcomes this issuc, improving clustering and classification performance. Hence, we expect competitive detection performance from VAEGMM, for details see Annex C The CCVAE (Joy et al. 2020) is state-of-the-an (SotA) semi-supervised disentanglement model, allowing attribute manipulation in semi-supervised settings. For repair, we follow the automatic attribute manipulation procedure proposed by (Joy et al.) 2020) (see Annex D). We adapted their code to our pipeline. Contrary to Joy et al. (2020), we found performance was superior when using a large up-sampling coefficient for the classifier, i.c. like 3 in CLSVAE We provide two versions of our model, with and without distance correlation penalty in Section 4.4 So for ELSVAE-NODC use eq. (13) as training loss, whilst for CLSVAE use eq. 14).
7

---

Under review as a conference paper at ICLR 2022
(a) Syntheric- Shapes: 35% noise, 10 labels per class (1.6% of dataset).
(b) Bachhon-MNISST 35% noise, 10 labels per class (0.25% of dataset).
(c) Frey-Facee 35% noise, 10 labels per class (2.5% of dataset)
Figure I: Images for model repair (reconstruction), outlier (corrupted) and inlier (uncorrupted). (a) Original (Outlier); (b) Ground-Trun (Inlier); (c) VAE-L2; (d) VAEGMM: (e) CVAE; (1) CCVAE; (g) CLSVAE-NODC; (h) CLSVAE. A larger version of this figure is found in Annex
Notterttnt SHOE
03
VAE-22
CLEVAS MOWNE-MOOO
0.7
60
Hompptee Class)
a
Outiiz (ANPR ()
Repuir Dary Piack (SMSF ()
Repair on
Plack (SMSF ()
CVAE Kinbin
0.47 (0.03)
0.003 4)
CCVAE
0,56 (0.03)
0.005 (0.001)
CLSVAL NODC 10605
0,99 (3e-4)
(0.024)
0,002
CLSVAE (Ours)
0.99(0..2)
I0014 (0.008)
0.045 (0.004)
Fashoon MWVS7
VAE L2 IK Kinama Welling 2014
0,49-00023
1.362 (1.140)
0.175 (0.004)
VAEGMA Nemiliie di 2010
0.66:00 (66-0.113)
3.161 (1.032)
0J895 (0.001)
CYAE Kanurrere 2014
0.36:0001)
1.732 (0.335)
0,899 (0.201)
CCVAE Poy am 201a1
0:51 (0.09)
1.119 (0.556)
0.135 (0.0033
CLSVAE-NODC T0uas!
0:54/0.10)
0354 (0.214)
0.107 (0.002)
(Curs)
(0/0077)
Kinann Welling 2014
CVAR
0.270
CLSVAL-NODC
0,85 (3)
(00079)
0.172 (0.025)
CLSVAE (Ours)
0.99:00033
0
0.177 (0,033)
(b)
Figure 2: Outlier detection uses AVPR score where highest is best. Repair for dirty pixels, and for clean pixels (distortion), SMSE where lowest is best. Trusted range sweep for Rishion-MNIST" where 10. 0.25, 0.64, 1.28)
dataset, 35 % noise level. (b) Table for results 35% noise level, and 10 labelled samples per class for the trusted set. Boldface corresponds to the best performances within a standard error, and green color to best mean performance overall. Standard error in brackets.
5.1 DISCUSSION OF RESULTS
In Figure 2 (a) we show performance function of the size of the trusted set, i.e. sweep trusted set sizes, for a 35% noise level. Similar performance is seen for other datasets (see Annex D}. Table b) shows the results for all datasets for a 35% noise level and trusted set size of 10 labelled samples per class. Results for all trusted set sizes and noise levels are found in Annex [] with similar analysis image repairs for all datasets (larger version Annex Additional examples of repairs seen in Annex K including inlier instances, 45% noise level, and samples per class (trusted set size) for
on performance. Figure [ shows some examples
since its an easier dataset.
Outlier Detection Looking at Figure 2
that average both CLSVAE (our model) and CLSVAE-NODC have the highest AVPR, registering the best detection performance, with similar scores. We that CCVAF, the previous SotA, has similar detection performance as CLSVAR.
8

---

Under review as a conference paper at ICLR 2022
VAEGMM, also sumi-supervised, lags behind both likely because its designed for slightly larger trusted sets. All sumi-super models (CLSVAE, CLSVAE-NODC, CCVAE, VAEGMM) im- prove their detection performance the trusted set grows larger. CVAE and VAB-L2 do not use trusted set, and thus have the same performance throughout all the trusted set range in Tigure[ja) These two observations about the trusted set range are also seen, for all datasets and noise levels, in Annex J CVAE is supervised, still it shows poor performance, this may be due to: issues linked to (decoder) likelihood-based scores (Eduardo et al. 2020 Lan & Dinh 2020); poor fitting to the data, thus impacting negatively the score. VAE-L2 uses a likelihood score, and is unsupervised, so poorer detection performance is understandable. This highlights sumerviperiiii as being impor- tant in systematic error detection. VAE-L2 registering good performance in Symthetic-Shupes (see Figure b)) is likely due to this dataset being easier. Lastly, we note that CLSVAE tends to have better detection performance in higher noise levels relative to other methods (see Annex Therein, CCVAE has close to or similar detection performance as CLSVAE.
Automated Repair In Figure 2 we see that on average CLSVAE (our model) is best at automated repair (lowest SMSE on dirty pixels). We also see that distortion (repair clean pixels, SMSE) is relatively low, but not the lowest. This results in CLSVAE overall being the best repair method, not only replacing pixel values of the systematic error, but also inferring correctly the structure of the ground-truth repair (both clean and dirty pixels). This is confirmed in Figure [ where recon- structions (repairs) by CLSVAE show the best quality: replacing the error values of affected pixels and recovering the underlying ground-truth, whilst preserving the uncorrupted image portion (i.e. low distortion). CLSVAE has slightly better repair than CLSVAE-NODC on average, but most im- portantly, has better performance stability than CLSVAE-NODC, which can be seen in Annex []] for repair (dirty pixels). Further, in Annex J repair with CLSVAE is more advantageous relative to other models at higher noise levels. As expected, semi- cemi-suerviied models (CLSVAE, CLSVAE- NODC, CCVAE) improve their repair of dirty pixels the trusted set increases (see Figure aaa).. Both VAE-L2 and CVAE do not use a trusted set, so performance is static. CCVAE has the ability to perform good repair, registering the second best repair for dirty pixels after CLSVAE, but often with higher distortion. CCVAE suffers from two issues that account for its worse performance relative to CLSVAE For one, looking at Figure D CCVAE can sometimes fail to replace the pixel values from systematic errors. Secondly, and more often, it can fail to recover the underlying ground-truth even when replacing erroneous pixel values. Similarly, it has difficulty preserving the uncorrupted image portion (higher distortion). So some information about inlier appearance is being lost. This explained by the fact that CCVAE latent space is not disentangled regarding the clean and dirty patterns. CVAE can repair some outliers well, but it fails to deal with other systematic errors, which leads to an overall poor repair performance. This is maybe due to the binary latent variable used for y making it harder to model multiple systematic errors, for more discussion see Joy et al. (2020). VAE-L22 is able to repair some errors, but overall has worse repair than CLSVAE. Its strong regular- ization, optimized mostly for detection, leads to higher distortion and loss of detail (see Figure Its higher standard error (erratic repairs) is due to not being able to distinguish between clean and dirty patterns. VAEGMM does not do well in repair. This is likely due to it being better suited for classification or clustering tasks.
6 CONCLUSION
We have proposed a novel semi-supervised VAE (CLSVAE) for outlier detection and automated re- pair, in the presence of systematic errors. Our model exploits the fact that systematic errors are pre- dictable by high capacity models, unlike random errors. Thus, CLSVAE partitions the latent space into two subspaces: one for clean patterns, and another for dirty ones. Inliers are only modelled by the clean pattern subspace, whilst outliers use both subspaces. We encourage low mutual informa- tion between these subspaces through a penalty, improving performance stability. Empirically this encourages higher fidelity repairs by the model, without human in the loop or other post-processing. We show CLSVAE only needs a small trusted set, requiring the user to label less data. We show that unsupervised models may not be able to distinguish between clean patterns and systematic errors, and strong regularization leads to a lower quality repair. Experimentally, CLSVAE showed superior repair quality and performance compared to other sumi-supe models, including SotA disen- tanglement model. Experiments were carried out on image data, and in the future, we would like to explore other types of systematic errors and data types (tabular, sensor, natural language).

---

Under review as a conference paper at ICLR 2022
REFERENCES
H. Akrami, Anand A. Joshi, J. Li, and R. Leahy. Robust variational autoencoder.
ArXiv,
www/1905.0991... 2019.
Yuri Burda, Roger B Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In ICLR (Poster), 2016.
Yanzhi Chen, Dinghuai Zhang, Michael U. Gutmann, Aaron Courville, and Zhanxing Zhu. Neural approximate sufficient statistics for implicit models. In Niuth International Conference on Learn- ing Representations (ICLR 2021), May 2021. URL https://iclr.cr Con Conffrenceees.... 2021/Dates Ninth International Conference on Learning Representations 2021, ICLR 2021;
Conference date: 04-05-2083 ---2--2221 Through 07-05-2021.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-War Woo Ha. Stargan v2: for multiple domains. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020.
Diverse image synthesis
(CVPR), pp.
Xu Chu, Ihab F lyyas, Sanjay Krishnan, and Jiannan Wang. Data cleaning: Overview and emerging challenges. In Proceedings of the 2016 international conference on management of data, pp- 2201-2206, 2016.
Ilias Diakonikolas, Gautam Kamath, Daniel M Kane, Jerry Li, Jacob Steinhardt, and Alistair Stew- art. Sever: A robust meta-algorithm for stochastic optimization. arXiv preprint 1003:v:8803002..55 2018.
Simao Eduardo, A. Nazábal, Christopher K. I. Williams, and Charles Sutton. Robust variational autoencoders for outlier detection and repair of mixed-type data. In AISTATS, 2020.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98-11) 136, 2015.
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Çelikyilmaz, and Lawrence Carin. Cyclical annealing schedule: A simple approach mitigating kl vanishing. In NAACL, 2019.
M. J. F. Gales and Peder A. Olsen. Tail distribution modelling using the richter and power ponential distributions. In Sixth European Conference on Speech Communication and Tech- nology, EUROSPEECH 1999, Budapest, Hungary, September 5-9, 1999. ISCA, 1999. URL http:///ww. isca- speccchasppehhh.hcm org
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and cut-of-distriuution examples in neural networks. arXiv preprint arXiv: 1tt.a/ppeeeec199/99999999 9999/99999999 1wte:999_11__h__1_1
2016.
Maximilian Ilse, Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. Medical Imaging with Deep Learning, pp. 322-348. PMLR, 2020.
Ihab F Ilyas and Xu Chu. Data cleaning. Morgan Claypool, 2019.
Tom Joy, Sebastian Schmon, Philip Torr, N Siddharth, and Tom Rainforth. Capturing label charac- teristics in vaes. In International Conference Learning Representations, 2020.
Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp- 3581-3589, 2014.
Jack Klys, Snell, and R. Zemel. Learning latent subspaces in variational autoencoders. In NeurlPS, 2018.
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. arXiv preprint arXiv: 1.00741, 2018.
10

---

Under review as a conference paper at ICLR 2022
Sanjay Krishman, Jiannan Wang. Eugene Wu, Michael J Franklin, and Ken Goldberg. Activeclean: Interactive data cleaning for statistical modeling. Proceedings of the VLDB Endowment, 9(12): 948-959, 2016.
Chieh-Hsin Lai, Dongmian Zou, and Gilad Lerman. Robust subspace recovery layer for unsuper- vised anomaly detection. In International Conference on Learning Representations, 2019.
Chieh-Hsin Lai, Dongmian Zou, and Gilad Lerman. Novelty detection via robust variational au- toencoding, arXiv preprint arXiv:2006.00 05534, 2020.
Charline Le Lan and Laurent Dinh. Perfect density models cannot guarantee anomaly detection.
arXiv preprint arXiv:20/2.03808, 2020.
Alexander Lew, Monica Agrawal, David Sontag, and Vikash Mansinghka. Pelean: Bayesian data cleaning at scale with domain-specific probabilistic programming. In International Conference on Artificial Intelligence and Statistics, pp. 1927-1935. PMLR, 2021.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. 2008 eighth ieee international conference on data mining, pp. 413-422. IEEE, 2008
Zifan Liu, Zhechun Zhou, and Theodoros Rekatsinas. Picket: Guarding against corrupted data in tabular data during learning and inference. arXiv preprint arXiv:2006.04730, 2020.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In International Conference on Learning Representutiins, 2019.
Felix Neutatz, Binger Chen, Ziawasch Abedjan, and Eugene Wu. From cleaning before ml clean- ing for ml. Data Engineering, pp. 24, 2021.
Theodoros Rekatsinas, Xu Chu, Ihab F Ilyas, and Christopher Ré. Holoclean: Holistic data repairs with probabilistic inference. arXiv preprint arXiv: 00820, 2017.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference machine learning, pp. 1278-1286. PMLR, 2014.
Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus- Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694, 2019.
Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojeiech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 2021.
Bernhard Schölkopf, Robert C Williamson, Alexander J Smola, John Shawe-Taylor, John C Platt, et al. Support vector method for novelty detection. In NIPS, volume 12, pp. 582-588. Citeseer, 1999.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28: 3483-3491, 2015.
Gábor J Székely, Maria Rizzo, and Nail K Bakirov. Measuring and testing dependence by corre- lation of distances. The annals of statisties, 35(6):2769-2777 2007.
John Taylor. Introduction to error analysis, the study of uncertainties in physical measurements. 1997.
F. Tonolini, B. S. Jensen, and R. Murray-Smith. Variational sparse coding. In UAI, 2019.
Ziyu Wan, Bo Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Jing Liao, and Fang Wen. Bringing old photos back to life. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2747-2757, 2020.

---

Under review as a conference paper at ICLR 2022
Matthew Willetts, Stephen Roberts, and Chris Holmes. Semi Semi-unsupervisedd learning: Clustering and classifying using ultra-sparse labels. In 2020 IEEE International Conference on Big Data (Big Data), pp. 5286-5295. IEEE, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Chong Zhou and Randy C Paffenroth. Anomaly detection with robust deep autoencoders. In Pro- ceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 665-674, ACM, 2017.
A VAE-L2 (UNSUPERVISED)
The unsupervised VAE can be used to perform both outlier detection and repair. However, without regularization it will overfit to systematic errors (outliers). So, here regularize both the encoder and decoder weights via lg regularization (weight decay). We can write the modified ELBO as
p(x) 2 [(x) (=(m)) (pe(x|z)) DKL (gs (z x [pp()))) + Xe
Σ WWE
1,,,11
(17)
where 0 and & are the parameters for the decoder and encoder respectively. Further, Ae, defines the regularization strength. The w, are the parameters for a layer indexed by i, belonging to the set of all VAE layer parameters WAF. Note that Ar is picked by using trusted set performance in outlier detection (AVPR) and repair (SMSE) metrics. Using repair metrics is not possible in practice, under our problem formulation, since we only have labels about which instances are outliers / inliers. However we relaxed this assumption here, since we wanted to obtain the best results possible with VAE-L2.
In terms of outlier detection, we used an anomaly score based on the reconstruction (negative log- likelihood) of the VAE. This type of anomaly score is typical in VAEs. For each d E {o..... D} features, we have
A(x) - log Loggo(Eaaluo E X,
(18)
where the outlier set of points is O fxx X A(x) 2 7) where ? decided by the user. The repair process for the outliers is
a = ==((((((z)), x 2EO.
(19)
B CVAE (FULLY SUPERVISED)
The CVAE (Conditional VAE) model (Kingma et al. 2014 Sohn et al. 2015) needs the entire train set to be labelled, i.e. like an oracle where the ground-true -tuuh for y is observed. This is generally not possible in practice, particularly for our model, as it would be cheating under the problem formulation. But it does provide a valuable baseline about what kind of performance this type of model, given all advantages. can obtain. A supe-supe version of this model is in fact the M2 model (Kingma et al. 2014), which will always register lower performance than its supervised counter-part part the CVAE. The ELBO for this model is
log p(x, y) 2 C(x, y) = Bgg(=|m.)) [po(z]2,))) y)] DeL. (gg(2|z, ()lp(())))
(20)
where / and 6 are the decoder and encoder parameters. Further, note that in a standard CVAE we usually have p(z) (/ 0, I). However, we found empirically that stronger regularization for the encoder was needed for CVAE to do well in repair. Otherwise, would often overfit to the systematic errors. Hence we used a modified prior (z(z)
(2 where E [0.1, 0.8), thus enforcing stronger regularization (lower message capacity) on latent code z. Even though is not standard, we also tried 4, ax) but always ended up with bit worse performance than ( |z. v).
In terms outlier detection, we used an anomaly score based on the reconstruction (negative log- likelihood) of the CVAE model. This type of anomaly score is typical in VAE type models. For each
12

---

Under review as a conference paper at ICLR 2022
d E {o, D) features, we have
D log Po (o) = 1), a E A,
A(x) = Σ
(21)
where the outlier set of points is fxr E EX| A(x) 2 2 where y is decided by the user. We followed the CVAE attribute manipulation strategy from (Klys et al.] (018) when defining repair of
an outlier. We encode using the original label (i.e. = 0), then switch label in latent space (i.e.
y 1) and decode it. In other words, for repair we have
x #EO.
(22)
C VAEGMM: ALTERNATIVE TO M2 (SPARSE SEMI-SUPERVISED)
This formulation is based on Willetts et al. (2020), specifically GM-DGM (Gaussian Mixture Deep Generative Model). The work focused on severe sparse semi-supervision (some classes do not even have labels). This formulation is relevant, in sparse sumervupeeiision, since posterior collapse of (()(z)) occur for the original M2 model (Kingma et al. 2014). This in turn leads to poor performance in classification and clustering tasks. Hence, one can think of Willetts et al. (2020) as improvement on M2 model for the issue above. We now present the version of GM-DGM from Willetts et al. (2020) applied to our problem.
C.1 GENERATIVE MODEL.
The generative model is defined
po (o(x)))p--(ll))p_0
(23)
where
(y( Bernouli(yla),
(24)
pr (zly)) N 0, o (1- Oz o
(25)
po(x = (x Ez))))
(26)
which defines a 2-component Gaussian Mixture Model w.r.t. 2, and r (0y=1, dy=o] and dy=1 <
y==0- A sensible range for is: a d==1 (0.2, 1]: 0= [2,8). This prior defines 2-component Richter distribution (Gales & Olsen 1999). expressing a type of heavy-tailed distribution on z. This type distribution common robust statistics (Eduardo et al. 2020), as means robustify (regularize) the parameters of a model to outliers. The main idea here is that inlier samples z/y = will be regularized more strongly compared outliers = y O, reflected by 0y=1 < 0y=o- We initially tried learning the parameters of the Gaussian components. However, we obtained much better results by fixing their parameters (i.e. mean vector and covariance matrix), particularly when it came to outlier detection with smaller trusted sets. Lastly reflects the initial belief on the fraction of clean data.
C.2 VARIATIONAL MODEL
As for the variational model, we the standard formulation provided in Willetts al. which is the same as the original M2 model (Kingma et al. 2014). So we have
2020
(s(z|æ, vy)x)
(27)
(y x)) Bermoulliyllss(an go( (z) lz, y) = W/=>/u//aa,,,...a (2, (x. y)),
(28)
(29)
where (T) (2, v) and Ex y) neural networks.
13

---

Under review as a conference paper at ICLR 2022
C.3 TRAINING LOSS
We can write the ELBO (Evidence Lower Bound) for the unsupervised part of the dataset, i.e. Xx, where labels y are observed as
p(z) 2 [(x) - / 2 20(2] yy x)) log po(a)2)p, (v) dzdy
(30)
((x) Lo (=)o(=) [log ( (vla)) [ (9 x, v) (
(31)
One can rewrite the above ELBO in a different way as ((x) md(m) (x. 1) (1 ( g(z,
0)
(32)
where
(ax, v) = (=|o.v) [log p [ D (z) (
(33)
and (T(=) = go = 1|x) the probability a data instance is clean.
The ELBO for the labelled part of the dataset (i.e. trusted set), where we have (x. E Nt, V/. written as
log p( v) 2 L(x, v) = G (z, y) + log Po (y))
(34)
Given the aforementioned definitions for the model and ELBOs (labelled and unlabelled), the overall training loss is the same eq. (13) from the main paper.
C.4 OUTLIER DETECTION AND REPAIR PROCESS
After training the model, then we can proceed with the outleer detection and repair process. First, we need to define score for use in detection. So at test time we use
A(x)
log (y = 1|x),
X'
(35)
which is based on the probability of the instance being an inlier. Usually, the user defines threshold y to classify instances into inliers or outliers. The of outlier instances is given by O = { E X A(x) 2 ?), where y 2 O. Assuming ( (x) is somewhat calibrated, then one can use y log( (0.5). After obtaining the set O, and conditioning on y = 1, we produce repair using
i (o(u y = 1)), a O.
(36)
D CCVAE (SEMI-SUPERVISED DISENTANGLEMENT)
In Joy et al. 2220)) the latent space z is split into two subspaces: the style (or agnostic) part zie that
meant to model unlahelled patterns present in the instance; the characteristics part z, that is meant have
to model the labelled attributes (characteristics). Formally,
[zei Fte] where [ is the concatenation operation. In our case, following Joy et al. (2020) and their implementation code, modelled z single variable (univariate) for our binary label: y 1 for inliers and y 0 for outliers.
D.I GENERATIVE MODEL
From Joy al. 2020), using similar notation, have
p(x, z, =p ( > (x.c) ply) Bernoulli(yy))))
(37)
(38)
p(z,e) -V 0.I),
(39)
Pe (z.lv) = (V) (v))
(40)
where po pa( ( is neural network based decoder, and 0 are the parameters of the neural network. In this case a has the same meaning as in CLSVAE, where it expresses the prior belief on the fraction of inliers present in the dataset. Note that the above model expresses two-component mixture model on x,, with y as gating variable. In fact we have a mean and variance for each value as pertains Za, i.e. pyl 1) and a (y 1) for inliers and similar for outliers (y 0).
14

---

Under review as a conference paper at ICLR 2022
D.2 VARIATIONAL MODEL
The variational distribution is factorized as
( z (z) - Fo.o (y x)) (2la,
(41)
(p(y [e) = ()
(42)
20 =V (e) (2)), o (z)),
(43)
/ go (vl (z))
(44)
q:p (
ae (2|a,, (2,
(45)
ae ar)
where 9pp (y)ze) and (x(x)))) are neural network based encoders, with and & as neural network parameters.
D.3 TRAINING LOSS
The training loss used in Joy et al. (2020) is defined by the ELBO, with one term for the labelled data (trusted set) and a second term the unlabelled data. The total ELBO written as
Σ
LOCVAE(x)+
LoOVAE(<, v),
(46)
A
(0.0)
where the labelled part of the ELBO is
q (M Rc)
pa (=]()))) x
(ZolV)
LoCvAE(æe, y) = Eg (z(
log
+ 3 log go (ular) + logg(v)) (47)
Up.0 (x))
(p( 2.) 2.)
where |3 is the hyperparameter controlling amount of up-sampling and importance relative to other terms, like in CLSVAE In Joy al. (2020), for their application, they found that setting = brought good results and found no need to tune it further. In our case, we found that we obtained better performance by using larger values for B. This probably because the application is different, i.c. outlier detection and subsequent repair, and since the trusted set (labelled set) is quite small. The unlabelled part of the ELBO is not as important for our analysis here, please see the original paper.
D.4 OUTLIER DETECTION AND REPAIR
After training the model, we proceed with outlier detection and automated repair. We need to define a score for use in detection, and for that we use the classifier given by the variational model. As such, we have
(48) which is the approximate negative log pre-oobbbbiltyy of an instance being an inlier. The user may define a threshold y to classify instances into inliers or outliers. The of outlier instances given by O = fxr E X A(x) 2 7). where 2 O. Assuming the classifier is somewhat calibrated, then log(0.
A(x) = - log (=)==) [g = 1|ze)] x X.
one can use the a 7
From the perspective of CCVAE, the repair of an outlier instance is just attribute manipulation via z. subspace. Once the appropriate z. is found, then one uses the decoder for reconstruction obtaining repair. Under our problem definition, automated repair is very much like conditional generation as seen in Joy et al. (2020), where samples for ze are drawn from the conditional prior whilst reusing F/e obtained from encoding the outlier instance. Specifically, where depending on the value it forces the generated samples to have, or not have, the presence of the attribute (e.g. inlier / outlier). In our case, limiting human interaction apart from building the trusted set. That means that exploring z. with user interaction pick the best repair (reconstruction) for each outlier instance not realistic. So we wish to obtain the most likely reconstruction under = 1, i.e. the maximum a posteriori, thus defining an automated repair for the outlier instance. Hence, the repair is given by
have
[ze Pa (xel)))) 2
interested in automated repair, and thus
i μ4 ([m (y 1): (o(x)))))
O,
(49)
where (y = 1) the mean for the inlier component of p (zelv), and ps (2) is the mean of (s(2)ela:) (z), which excludes the characteristic (labelled) latent subspace.
15

---

Under review as a conference paper at ICLR 2022
E DISCUSSION ABOUT CLASSIFIER IN CLSVAE
We found that using qø, (yl zd) leads to better performance in more challenging scenarios, e.g. smaller trusted set or when higher data corruption is present. One could have used g6 (yl za) which works, but it is still worse than the former. This is mostly because &e provides important context on clean patterns present in the instance, which in turn allows for Za to contain less information about clean patterns. Hence, Za will be able to focus more on the dirty patterns, and so the mutual information (MI) between &e and zd will be lower.
So the idea of using the stop gradient operator in qá, qá, (y) za) Bernoulliyy/ TT, (lsg(ze): =a]) has to do with preventing 2, from having dirty pattern information. By using sg(ze). when execut- ing back-propagation,
prevent the gradient flow from gø, (y/lz., zd) influence the learning of parameters Φe- As such, de will only be affect by gradients stemming from the reconstruction of inlier instances, and not the decision on y. This trick useful early in the training, stabilizing the outcome of the optimization procedure, leading to better repair. Generally, only a few cases benefit from this trick, and usually when small trusted
used.
Further, Jo Ze, zd) allows for the cross-entropy loss used for the trusted set to directly bias the latent space. This means labels y 3% can help in separating the latent inlier and outlier repre- sentations (manifolds). Other works have tried biasing the latent space in the same fashion, e.g.
(Locatello al. 2019 llse et 2020 Joy et al. 2020
F EMPIRICAL DISTANCE CORRELATION
For random data batch of n samples, such as (Z,Z") ( ((xx,()) k= =,.,,, one can define the empirical estimate for the distance correlation between multivariate random variables 2 R and x' E R. Note that and can be of different dimensions, i.c. # p. Given this data batch, using the definition from (Székely al. 2007), we first define Agı and B3l as
Okl
||
21
1 n
Σ Okl
Σ
n
Ok.
agl
0.1
=
a. =
ag!
n 1=
n
1=1
R.==1
Agt 0kl
ūk.
0hd1 + C...
and similarly, using but = ||
z/1le we define
Bui bus bR. T. + b...
Now we are ready to define the empirical distance covariance as and the following empirical distance variances for each random variable as
dCov, (Z, z' 2 k.
Asa Bat
n
daarn (Z)
A
dVar (Z')
Σ B
n
k,1=1
k.i=1
Finally, combining the above measures write the distance correlation
dCov, (Z.v (Z,Z')
dCorr, (2,rr,(Z,2 (Z,Z') /dVar, (Z) dVar,,
where 0 < dCorr,, (Z, (Z,Z)) < 1. Note that if dCorr, Z') 0 then and z' are statistically inde- pendent random variables. Otherwise, dCorr, (Z, Z') = then they are strongly correlated. The distance correlation measure capture non-linear dependencies, whilst the more common Pear- son correlation can only capture linear dependencies. Further, a value of 0 for Pearson correlation does not imply independence, unlike in distance correlation which it does.
16

---

Under review as a conference paper at ICLR 2022
G EXPERIMENTS: DETAILS ON DATASETS, CORRUPTION, AND TRUSTED SETS
Here we have detailed description of the datasets and their corruption. For each noise level corrup- tion in NLsize = [15%, 25%, 35%, 45%]. we instantiate five different examples of the same dataset using different random seeds. For each of those examples we build several trusted sets using the sizes in Tis [5, 10, 25, 50), where labelled instances of smaller trusted sets are reused in bigger ones. The models are run on each example, for each trusted set, and results are then averaged.
Synthetic-Shapes This is synthetic image dataset. It is meant to test the models in a simpler setting as it relates the clean dataset. We treat the pixel values as a Bernoulli variable. The underlying clean dataset is composed of four different shapes (classes): a circle, rectangle, an ellipse and a triangle. The shapes are filled by white pixels and the background is black. For each instance, the shape is placed uniformly at random inside the 28 x 28 black background. The systematic errors have four types, all are white lines that cross the square image from one side to another side. We have 4 fixed-in-place lines (two diagonal, one vertical, and one horizontal), affecting the same pixels, and sometimes intersecting with the shapes. Hence, we have a total of eight underlying classes for trusted set constitution, i.e. four data and four systematic error classes.
The size of the images is 28 x 28, and the pixel values are binary {0, 13, i.e. black and white. Overall, we have a dataset size of N = 5000, with the following split: train (80%), validation (10%), test (10%). Given the eight underlying classes, the size dataset N, and TS we have the trusted set range: [40, 80, 200, 400] total number of labelled instances, which corresponds to [0.8%, 1.6%, 4%, 8%] of the entire dataset
Frey-Faces This is gray-scale image dataset consisting of the same person with different facial expressions. We treat pixel values as continuous. In terms of data classes, we only have one (mono- lithic), since no labels for the expressions are provided. We have four systematic error classes, which consist of four randomly uniformly placed squares of 6 x 6 pixels. The place and color of these squares is defined by the random seed, when the corruption example is created. After that, always the same features are affected. Hence, in total we have five underlying classes for trusted set constitution, i.e. one data and four systematic error classes.
The size of images is 28 x 20, and the pixel values range from [0, 256], i.e. gray-scale. The size of the entire dataset N 1965, with the following split: train (80%), validation (10%), test (10%). Given the above and TS the trusted set range is: [25, 50, 125, 250] total number of labelled instances, which corresponds to [1.3%, 2.5,, 6.4%, 126 of the entire dataset.
This is a gray-scale image dataset, which consists of images of different types of clothing and accessories from an online merchant. There are 10 existing data classes, provided with the dataset. We have 4 fixed-in-plas inpacee lines (two diagonal, one vertical, and one horizontal) where the color (black or white) for each depends on the random seed. Then we have 4 squares of size 6 x 6 placed randomly uniformly with random color, dependent random seed. Hence, we have total of 18 underlying classes for trusted set constitution, i.e. 10 data and 8 systematic error classes.
Fashion-MNIST
The size of the images is 28 28, and the pixel values continuous with range [0, 1). i.e. gray- scale. The original train set is of size 60000 instances, which we split for our actual train set of 54000 (90%) and validation set of 6000 (10%). We use the same test set of 10000 instances, so N 70000. Given TSsine and the train set size N, the trusted set range is: [90, 180, 450, 900] total number of labelled instances, which corresponds to [0.12%, 0.25%, 0.64%, 1.28%] of the entire dataset.
H EXPERIMENTS: MODEL ARCHITECTURES, HYPER- HYPER-PARAMETERS AND OPTIMIZATION
Hy Hyperaaraeeter Selection
In our experiments, we tuned model hyperparameters according to outlier detection performance, which means highest AVPR (average precision). This was evaluated on the trusted set, the only labelled part of the dataset. Often, would look at the repairs (reconstructions) offered by each model for the trusted set. This way we confirm that the repair process is reasonable enough, and adjustment is needed on the hyperparameters side. In the case of VAE-L22 we had to not just
17

---

Under review as a conference paper at ICLR 2022
account for AVPR in the trusted set, but also check repair performance via SMSE (standardized mean squared error). This is because strong regularization, higher lg coefficient, often leads to better outlier detection performance, but that comes the cost of repair quality due to the VAE collapsing to mean behaviour. For each model, the hyperparameter search was carried out for each dataset a noise level of 35%, and with 25 samples per class.
VAE-L2 In Synthetic-Shapes 200 epochs; KL divergence annealing used; & coefficient is 35.0. In Frey-Faces: 300 epochs; KL divergence annealing used; &2 coefficient is 100.0. In Fashion-MNIST! 100 epochs; KL divergence annealing used; lz coefficient is 100.0.
VAEGMM In Symthetic-Shapes: 200 epochs; KL divergence annealing used: fraction of clean data a 0.6; (trusted set) up-sampling coefficient
1000; 0y=1 0.9 and
yy=0 = 5.0. In Frey-Faces: 300 epochs; KL divergence annealing used; fraction of clean data = 0.6; (trusted set) up-sampling coefficient / 1000; 0 d-m 0.6 and σy=0 =
5.0. In Fashion-MNIST: 100 epochs; KL divergence annealing used; fraction of clean data a = 0.6; (trusted set) up-sampling coefficient = 100; 0y=1 = 0.5 and y== = 5.0.
CVAE In Synthetic-Shapes: 200 epochs; KL divergence annealing used; a 0.5. In Frey- Faces: 300 epochs; KL divergence annealing used; o = 0.2. In Fashion-MNIST: 100 epochs; KL divergence annealing used;
0.5.
CCVAE In Symthetic-Shapes 200 epochs; fraction of clean data = 0.6; (trusted set) up-sampling coefficient 3 = 50000.0. In Frey-Faces: 300 epochs; fraction of clean data = 0.6; (trusted set) up-sampling coefficient 3 = 10000.0. In Fashion-MNIST: 100 epochs; fraction of clean data o = 0.6; (trusted set) up-sampling coefficient 3 = 250000.0.
CLSVAE In Synthetic- Shapes: 200 epochs; fraction of clean data a 0.6; KL diver- gence annealing used; (trusted set) up-sampling coefficient B = 1000.0.
= 0.5;
a. = 0.5: aa = 5.0; distance correlation (DC) penalty used; A annealing ratio of 0.5 (DC penalty); ÀT maximum value of 100.0 (DC penalty). In Frey-Faces: 300 epochs; fraction of clean data a 0.6; KL divergence annealing used; (trusted set) up-sampling coefficient 1000.0.
0.6; o. = 0.2; σa 5.0; distance correlation (DC) penalty used; Ae annealing ratio of 0.5 (DC penalty); 2y maximum value of 1000.0 (DC penalty). In MaNS:: 100 epochs; fraction of clean data
0.6; KL divergence annealing
used; (trusted set) up-sampling coefficient 3 = 100.0. a, = 0.1; a. = 0.2; = 5.0; dis- tance correlation (DC) penalty used; X annealing ratio of 0.5 (DC penalty); Ar maximum value of 1000.0 0 (DC penalty).
CLSVAE-NODC Same hyperparameter options as CLSVVE but without the distance cor- relation penalty.
Note the annealing ratio above for the KL divergence and for the DC penalty is inspired by Fu et al. (2019). We use only one cycle (monotonic), and the R (or ratio) is the proportion used increase the penalty coefficient (or KL term coefficient) e.g. 0.5.
Optimization
We used the Py PyTorch framework to code all our models, and trained on a GeForce TITAN X GPU. All models were trained using the Adam optimizer, with an initial learning rate of 0.001.
Model Architectures
For continuous type data, i.e. Fashion-MNIST and Frey-Faces, we used the Gaussian distribution as the likelihood of each pixel in the reconstruction loss. The variance of the Gaussian distribution is shared amongst all the pixels in the image, and it is learnt as a parameter of the model. This was done for all models. For binary type data, i.e. Synthetic-Shuper we treated each pixel as a Bernoulli variable, and used the log-likedn likeliikl of this distribution for each pixel in the reconstruction loss. This was done for all models.
We used very similar encoder and decoder architectures for all models, so as to be fair and the results comparable. In the case of CLSVAE we used two encoders, one for the clean subspace z., and the other for the dirty subspace Zd. This architecture yielded better results for us in terms of repair in the trusted set. CLSVAE architecture can be seen in Table for the encoders and decoder, and the classifier can be seen in / In Table [ we see the neural architecture for the encoder and decoder of VAE and VAEGMM. The classifier architecture for the VAEGMM can be seen in Table 6 In Table
18

---

Under review as a conference paper at ICLR 2022
B find the architecture for the encoder and decoder of CVAE. In Table 4
find the architecture
for the encoder and decoder of CCVAE. The classifier architecture of CCVAE is just &e multiplied by weight parameter plus bias parameter, and then sigmoid non-linearity is applied like in
Joy et al. (2020).
Encoder
Decoder
(img_size, 200) +
(15, 50)
ReLU +
- ReLU
(200, 100)
+ (50, 100)
ReLU +
- ReLU
(100, 50) +
(100, 200)
ReLU -
+ ReLU
(50, 15)
+ (200, img_size)
Table 2:
Architecture of encoder and decoder for VAE and VAEGMM. Further, for binary pixels the decoder
will use a Sigmoid non- lincariy the end.
Encoder
Decoder
(img size, 200)
(16, 50)
ReLU -
- ReLU 7
(200, 100)
+ (50, 100)
ReLU +
+ ReLU
(100, 50) +
(100, 200)
ReLU -
+ ReLU
(50, 15)
+ (200, img size)
Table 3: Architecture of encoder and decoder for CVAE. Further, for binary pixels the decoder will use Sigmoid non lincariyy at the end.
Encoder
Decoder
(img_size, 200)
(16, 50)
ReLU +
+ ReLU +
(200, 100)
+ (50, 100)
ReLU +
- ReLU
(100, 50)
(100, 200)
ReLU -
+ ReLU
(50, 16)
- (200, img size)
Table 4: Architecture of encoder and decoder for CCVAE. Further, for binary pixels the decoder will use Sigmoid non-linearity the end.
Encoder z
Encoder zd
Decoder
(img_size, 200)
(img_size, 200)
(15, 50)
+ ReLU +
+ ReLU +
+ ReLU +
(200, 100) -
+ (200, 100)
(50, 100)
ReLU +
+ ReLU +
+ ReLU +
(100, 50) +
+ (100, 50)
(100, 200)
ReLU +
+ ReLU
ReLU (200, amg-size)
2 x (50, 10)
+ x (50, 5)
Table 5: Architecture of encoder and decoder for CLSVAE. Note for CLSVAE latent space of size 15 is split:
10 for 2: and for zu. Further, for binary pixels the decoder will use a Sigmoid non- ooneiiicrrtyy the end.
19

---

Under review as a conference paper at ICLR 2022
Classifier
(img size, 200)
ReLU
(200, 100)
ReLU
+ (100, 50) +
ReLU + (50, 1)
Sigmoid
Table 6: Architecture of classifier VAEGMM.
Classifier
(15, 7) +
ReLU
(7,
ReLU
(5, Sigmoid
Table 7: Architecture of classifier CLSVAE, input is z.
20

---

Under review as a conference paper at ICLR 2022
I LARGER VERSION OF RECONSTRUCTIONS (REPAIRS) FIGURE
(g)
(a) Saathetic-hhaper 35% noise, 10 labels per class (1.6% of dataset).
[a]
(b) Fashion MNIST: 35% noise, labels per class (0.25% of dataset).
[8]
[b]
tcl
(d)
(e)
(g)
(h)
(c) Frey-Faces: 35% noise, 10 labels per class
(2.5% dataset)
Figure 3: Images for model repair (roconstruction), outlier (corrupted) and inlier (uncorrupted): (a) Original
(Outlier): (b) Ground-Trun (Inlier); (c) VAE-L2; (d) VAEGMM; (c) CVAE; (f) CCVAE; (g) NLDCAE-NODC; (b) CLSVAE.
21

---

Under review as a conference paper at ICLR 2022
J RESULTS FOR ALL NOISE LEVELS AND TRUSTED SET SIZES (ALL SWEEPS)
Noise Level (Percent)
Noise iPercent)
10
Outlier Detection AVPR
0.9
0.8
0.7
0.6
0.3
0.4
0.3
Model Type
0.2
VAE.L2
WAEGMM
(Percent)
CVAE
2.0
OLSVAE-OOD
CLSVAE
Outler Detection AVR
0.9
OOVAE
0.8
0.7
0.6
0.4
0.3
0.2
10
20
30
40
10
20
40 50
Trusted Set (Samples per Class)
Trusted Set (samples Class!
Figure 4: Synthetic-Shape Outlier detection (AVPR) where higher is better. Trusted set range sweep where
TS size [5, 10, 25, 50] samples per class, i.e. (0.8%, 1.6%, 4%, 8%] of the entire dataset.
Noise Level (Percent) 15
Noise Level (Percent] 25
10c
Repair on Dirty Fixels SMSEE
10-1
Typa
VAE
Noise Level (Percent) 35
Noise Level (Percent) 45
CVAE
Repair on Dirty Pixels SMSEE
ELSWAE
COVAE
10-1
10
20
Trustad Set (Samples per Class) Trusted (Samples per Class)
Figure 5: Synthetic-Shape Repair of dirty pixels in outliers (SMSE), where lower better. Trusted set range sweep where TS
[5, 10, 25, 50] samples per class, i.c. [0.8%, 1.6%, 4%, 8%] the entire dataset.
22

---

Under review as a conference paper at ICLR 2022
(Percent)
(Percent)
0.07
Rapair on Clean Poels. SMSE
0.05
0.04
0.02
0.01
Model Type
VAE-22
VAEGMM
Noise Level (Percent) 35
Noise Level (Percent) 45
CVAE
0.07
CLSWAE-NODC
Repair on Clean Pixels 5MEE
CLSVAE
0.05
0.03
0.02
0.00
20
30
50
10
20
50
Trusted (Samples per Class)
(Samples per Class)
Figure 6: Synthetic-Shapes Repair of clean pixels in outliers (SMSE), i.c. distortion, where lower is better.
Trusted set range sweep where TS [5. 10, 25, 50] samples per class, i.c. [0.8%, 1.6%, 4%, 8%) of the
entire dataset.
Noise Level (Percent) 15
Noise Level (Percent) 25
1.0
Outiler betoction AVPR
0.8
0.7
0.6
8.5
0.4
Model Type
0.2
WAF-L2
VAEGMM
Nolse Level (Percent) 35
Noise Level (Percent)
1.0
CLSVAE NODE
Outlier Detection AVRR
0.9
OOVAC
0.8
0.7
0.5
0.4
0.3
0.2
30
10
Trusted Sct (Samp es per Class)
Trusted Set (Sarples per Classl
Figure 7: Frey-Faces. Outlier detection (AVPR) where higher is better. Trusted set range sweep where TSame [5, 10, 25, 50) samples per class, [1.3%, 2.5%, 6.4%, 12.7%) the entire dataset.
23

---

Under review as a conference paper at ICLR 2022
(Percent)
(Percent)
Repair on Diry Pixels. SMSE
10
Model Type
WEE-12
VAEGMM
Ncise Level (Percent)
Noise Leval (Percent)
CVAE
CLSVAE-NODC
Repair on Dirty Pieels 5MSE
DOVAZ
10"
10
(Samples per Figure 8: Frey-Faces. Repair of dirty pixels in outliers (SMSE), where lower is better. Trusted set range sweep
(Sarppees per Class)
where TS = [5, 10, 25, 50] samples class, i.c. [1.3%, 2. 2.5%, 6.4%, 12.7%) of the entire dataset.
Noise Level (Percent) = 15
Noise Level iPercent) =
Repair on Clean Pixels SWSE
0.7
0.6
0.4
07.
0.1
Model Type
WAE-17
VAEGMM
Noise Level (Percent) 35
Noise Level iPercent) 45
CVAE
Repair on Clean Poxels SMSE
0.
CLSWAE
CLSVAENNODC
0.b
OCVAE
0.5
3.4
0.2
20
30
40
100
Trusted (Samples ess per Class)
(Samples per Classi
Figure 9; Frey-Faces. Repair of clean pixels in outliers (SMSE), i.c. distortion, where lower is better. Trusted set range sweep where TSeee [5, 10, 25, 50) samples per class, i.e. [1.3%, 2.5%, 6.4%, 12.7%) of the entire datasct.
24

---

Under review as a conference paper at ICLR 2022
Level (Percent)
(Percent)
100
Outler Detaction AVPR
0.8
0.6
0.4
0.2
Model Type
WAE-L2
Noise Level (Percent] = 35
Noise Level iPercent) =
CVAE
1.0
CLSVAE
Outller Detection AVR
0.8
0.6
0.4
0.2
20
30
40
10
20
40
50
Trusted (Samples Class)
Trusted Set (Samples Class!
Figure 10: Fashion-MNIST. Outlier detection (AVPR) where higher is better. Trusted set range sweep where
TSsi
[5, 10, 25, 50] samples per class, i.e. 12%, 0.25%, 0.64%, 1.28% of the entire dataset.
Noise Level (Percent) 15
Noise Level (Percent) 25
Repair on Dirty Pixels 545E
Model Type
WAZGMM
Noise Level (Percent) 15
Noise Level (Percent) 45
OLSVAE-NODC
Repair on Dirty Hoxlss SMSE
CLSVAA
OCVAE
10
40
50
10
20
50
(Samples per Class)
Trusted
(Samples per Class)
Figure 11: FasHion-MNIST. Repair of dirty pixels in outliers (SMSE), where lower is better. Trusted set
range sweep where TSae = 10, 25, 50] samples per class, i.c. 10., 0.25%, 0.64%, 1.28%) of the entire datasct.
25

---

Under review as a conference paper at ICLR 2022
(Percent)
(Percent)
Repair an Clean Piells 5MSS
0.20
0.16
0.14
Model Type
0.10
VAE-22
VAEGMM
Noise Level (Percent) 35
Noise Level (Percent) 45
CVAE
CLSWAE-NODC
Repair on Clean Pixels. 5MEE
CLSVAE
0.18
0.16
0.12
0.10
10
20
30
50
10 20
50
Trusted (Samples per Class)
(Samples per Class)
Figure 12: Fashion-MNIST. Repair of clean pixels in outliers (SMSE), i.e. distortion, where lower is better. Trusted set range sweep where T of the entire dataset.
[5, 10, 25, 50) samples per class, i.e. 10%, 0.25%, 0.64%, 1.28%)
K ADDITIONAL RECONSTRUCTIONS (REPAIRS) FOR ALL DATASETS
K.1 SYNTHETIC-SHAPES
Figure 13: Images for model repair (reconstruction). outlier (corrupted) and inlier (uncorrupted): (a) Original (Outlier); (b) Ground-The Truth (Inlier); (c) VAE-L2; (d) VAEGMM: (c) CVAE; (f) CCVAB: (g) NLSVAE-NODC; (h) CLSVAE The first two rows inlier examples, the others being outliers. Synthetic-Shapes: 35% noise, 5 labels per class 8% of dataset).
26

---

Under review as a conference paper at ICLR 2022
Figure 14: Images for model repair (reconstruction), outlier (corrupted) and inlier (uncorrupedd:: (a) Original (Outlier); (b) Ground- Truth (Inlier); (c) VAE-L2: (d) VAEGMM; (e) CVAE; (f) CCVAE; (g) CLSVAE-NODC; (h) CLSVAE. The first two rows inlier examples, the others being outliers. Syuthetic-Shaper 35% % noise, 50 labels per class (8% of dataset).
Figure 15: Images for model repair (reconstruction). outlier (corrupted) and inlier (uncorrupted): (a) Original (Outlier); (b) Ground-The Truth (Inlier); (c) VAE-L2; (d) VAEGMM: (c) CVAE; (f) CCVAB: (g) NLSCAE-NODC; (h) CLSVAE The first two rows inlier examples, the others being outliers. Synthetic-Shapes: 45% noise, 5 labels per class 8% of dataset).
27

---

Under review as a conference paper at ICLR 2022
K.2 FREY-FACES
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Figure 16: Images for model repair (reconstruction), outlier (corrupted) and inlier (uncorrupted: (a) Original (Outlier; (b) Ground- Truth (Inlier); (c) VAE-L2; (d) VAEGMM; (c) CVAE; (f) CCVAB; (g) NODC; (h) CLSVAE The first two rows inlier examples, the others being outliers. Frey-Faces: 35 % noise, 10 labels per class (2.5% of dataset).
28

---

Under review as a conference paper at ICLR 2022
(a)
(b)
(C)
(d)
(e)
(f)
(g)
(h)
Figure 17: Images for model repair (reconstruction). outlier (corrupted) and inlier (uncorrutedd: (a) Original (Outlier); (b) Ground- Truth (Inlier); (c) VAE-L2; (d) VAEGMM; (c) CVAE; (f) CCVAB; (g) NOSC: (h) CLSVAE. The first two rows inlier examples, the others being outliers. Frey-Faces: 35% % noise, 50 labels per class (12.7% of dataset).
29

---

Under review as a conference paper at ICLR 2022
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Figure 18: Images for model repair (reconstruction). outlier (corrupted) and inlier (uncorrupedd: (a) Original (Outlier); (b) Ground Truth (Inlier); (c) VAE-L2; (d) VAEGMM; (c) CVAE; (f) CCVAB; (g) CLSVAE-NODC; (h) CLSVAE. The first two rows inlier examples, the others being outliers. Frey-Faces: 45% % noise, 10 labels per class (2.5% of dataset).
30

---

Under review as a conference paper at ICLR 2022
K.3 FASHION-MNIST
Figure 19: Images for model repair (reconstruction), outlier (corrupted) and inlier (uncorrupted): (a) Original (Outlier); (b) Ground- Truth (Inlier); (c) VAE-L2; (d) VAEGMM: (e) CVAE; (f) CCVAE: (g) CLSVAE-NODC; (h) CLSVAE. The first two rows are inlier examples, others being outliers. Kushion-MNIS % noise, 10 labels per class (0.25% of dataset).
Figure 20: Images for model repair (reconstruction), outlier (corrupted) and inlier (uncorrupted:: (a) Original (Outlier); (b) Ground- Truth (Inlier); (c) VAE-L2; (d) VAEGMM; (c) CVAE; (f) CCVAB; (g) NLSVAE-NODC; (h) CLSVAE. The first two rows are inlier examples, the others being outliers. Fashion-MNIST: MNIST: 35 % noise, 50 labels per class (1. (1.28% of datasct).
31

---

Under review as a conference paper at ICLR 2022
(g)
Figure 21: Images for model repair (reconstruction), outlier (corrupted) and inlier (uncorrupted): (a) Original (Outlier); (b) Ground- Truth (Inlier); (c) VAE-L2: (d) VAEGMM; (e) CVAE; (f) CCVAE: (g) CLSVAE-NODC; (h) CLSVAE. The first two rows are inlier examples, others being outliers. Fashion-MNIST 45 % noise, 10 labels per class (0.25% of dataset).
L TESTING COMPRESSION HYPOTHESIS: ENTROPY OF CLEAN VS. CORRUPTED DATA
Entropy of Clean vs. Dirty Training Data: Frey-Faces 35% Noise
Ground-Truth Data (Clean) Corrupted (Dirty)
330
Entropy: -oo log mox!
-450
-550
-600
-700
-750
VAE Latent Space Dimension (x)
Figure 22: Entropy of ground-truth training data (clean: without corruption) vs the entropy of corrupted training data (as in T.. Entropy estimation via IWAE (Burda et al., 2016). using a standard VAE (not regularized). VAE uses same architecture of Annex [ except dimension of z (latent space) now has the range [2, 4, 6, 8, 10, 15, 20, 25. 35, 50] (x-axis). Frey-Faces training data, with 35% noise level for corrupted dataset.
In this section, experimentally compare the entropy of clean (without corruption) training data, and the entropy corrupted training data. A larger entropy means that a dataset has larger variance overall. Note that in the setup of our problem, i.c. repairing systematic errors (see Section B) only corrupted data is used for training. In Figure 22] for Frey- Faces, we compare the estimated entropy of clean training data against one that has been corrupted (35 % noise, corruption in Table
32

---

Under review as a conference paper at ICLR 2022
We estimate the entropy by first training standard VAE model on the dataset (clean or corrupted), and then after training, we compute a tight bound on the marginal log- likelihood of that dataset. We compute this tight bound via IWAE estimator (importance weighted autoencoders, (Burda al. 2016), where we use = 250 samples. Note that entropy is H(xx) = - pottz) [log pa (2) J and hence marginal log-likelihood is just -H()). We vary the dimension of VAE latent space (z) in order see how well the model learn the training data. For this experiment, the VAE is not regularized.
The main idea is to prove empirically that corrupted data has larger variance than clean data, and hence has more information to be modelled. Specifically, when using the type of masking corrup- tions used in experiments. If dataset has larger estimated entropy, then the data be has having larger variance. A dataset with larger variance is a dataset with more diversity in terms of the patterns it contains, and hence has more information to be compressed (or modelled). In particular, in VAE this means more capacity (e.g. larger latent space) is needed to model a larger variance dataset.
Our claim, supported in literature (Eduardo et al. 2020: Ruff et al. 2019), is that a dataset that has been corrupted has larger variance because the added outliers (e.g. systematic errors) increase data pattern diversity, hence increasing entropy. Looking at Figure 22 we see that overall the entropy of corrupted data is larger than clean data, for all sizes of Hence, corrupted data has larger variance than clean data. For the smaller dimensions of in range [2, 10] units, we see that the VAE has less trouble learning the clean data compared to the corrupted data. This is also evidenced by how more quickly the entropy decreases for clean data relative corrupt data,
increase the latent space size [2, 10]. We conclude that a VAE only needs a smaller latent space (subspace) to model clean data (inliers), and that corrupted data needs larger latent space to be modelled properly.
33


Reviews:
[{"review_id": "trw1laDkpVv", "reviewer": "Reviewer_Hvcm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe problem is well motivated and clearly presented, very easy to follow. The formulation is very clear.\nThe performance of outlier removal is satisfactory on simple datasets.\n\nWeakness:\nThe idea and formulation is a little incremental and similar to RVAE [Eduardo et.al.]. The main difference could be the semi-supervised mixture model.\n\nSome thinking:\nHow correlated is the denoising quality and the downstream task, e.g., classification? could it be formulated together?\nHow is the model related to blind-source separation and ICA? Could the author gave some insight?", "summary_of_the_paper": "The paper presents a Clean Subspace Variational Autoencoder (CLSVAE) model for systematic outlier detection. Building on VAE for latent variable modelling, the paper proposes a semi-supervised learning to infer the potential outlier. \n\nThe author made the following assumptions:\n1. outliers exist therein, the inliers are still the majority;\n2. Outliers are systematic with predictable recurring patterns;\n3.  Compression hypothesis. Inlier data can be compressed further than outlier data.\n4. Outliers are a combination of clean and dirty patterns.\n\nOn the model side, a mixture model is proposed parametrized by Bernoulli mixing weight and decoder p(x|z), while the encoder q(z|x) is applied for sampling latent variable z. A standard ELBO is formulated to maximise the log likelihood for both y known and unknown scenarios. For outlier repair, the author proposed to minimize the correlation between noise and clean signal.\n\nExperimentally, the author evaluate two tasks: outlier detection, and automated repair on Frey-Faces3 , Fashion-MNIST (Xiao et al., 2017), Synthetic-Shapes.", "summary_of_the_review": "The author presented CLSVAE to handle systematic outlier. The problem is well motivated and clearly presented, very easy to follow. The formulation is very clear. The performance of outlier removal is satisfactory on simple datasets.\n\nThe idea and formulation is a little incremental and similar to RVAE [Eduardo et.al.] with the main difference could be the semi-supervised mixture model.\n\nAs mentioned above, I wish the author could answer the two following questions:\n1. How correlated is the denoising quality and the downstream task, e.g., classification? could it be formulated together?\n2. How is the model related to blind-source separation and ICA? Could the author gave some insight?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "p971XlEzvwM", "reviewer": "Reviewer_5JVH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The paper is well motivated and generally well written. The key idea of subspace learning is intuitive and easy to understand.\n- Results look to be good and support the claim.\n\nWeaknesses:\n- While I found Section. 3 to be well written, but Section 4, which contains the math and equations, seems to be difficult to follow, especially for a person that is not necessarily familiar with VAEs and generative models. Perhaps a simple introduction of those could help.\n\n- An issue of the paper, from my perspective, is that the experiments are simulated and not real. Therefore, I am not fully convinced that the developed method will have practical impact.  In particular, is it possible to test the method on some dataset that is well-known to be corrupted by systematic outliers (rather than simulating the outliers)?  One example that I am aware of, is that, in robot perception, wrong correspondences is a big challenge. For example, in point cloud registration, establishing reliable matches between a pair of point clouds is a very challenging task. There is a huge amount of literature on this problem, but perhaps [ref1] and [ref2] could be good references.  This maybe a future work direction for the authors to consider.\n\n- I also think an ablation study could help, where the impact of the size of the trusted set on the performance could be investigated. If the authors increase the size of the trusted set from <10% to 50%, do we expect a gain in the performance?\n\n\n[Ref1] Yang, Heng, Jingnan Shi, and Luca Carlone. \"Teaser: Fast and certifiable point cloud registration.\" IEEE Transactions on Robotics 37, no. 2 (2020): 314-333.\n\n[Ref2] Yi, Kwang Moo, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. \"Learning to find good correspondences.\" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2666-2674. 2018.", "summary_of_the_paper": "This paper develops a deep learning based method for detecting and repairing systematic outliers in data (before performing model learning on the data).\n\nThe method is called Clean Subspace Variational Autoencoder (CLSVAE). CLSVAE takes as input a trusted set provided by the user (the trusted set contains a set of inliers and outliers, but not requiring information about how the outliers are corrupted), and seeks to learn a variational autoencoder where the latent code is a concatenation of two subspaces, a clean subspace and a dirty subspace. CLSVAE is semi-supervised because the labelled trusted set is much smaller than the unlabelled set.\n\nThe paper performs outlier detection and repair benchmarks on three image datasets with systematic outliers, and shows that CLSVAE achieves favorable results.\n\n", "summary_of_the_review": "I am not an expert in this field. I tend to weak accept this paper, but I will also see if other reviewers have serious concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Jdwfmrpje6", "reviewer": "Reviewer_7fHt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well written and the contributions are clearly formulated.\nHowever, I have several concerns with it:\n- The intuition behind the assumption that inliers can be represented in a subspace of the clean and dirty patterns is not so obvious, especially given the datasets that have been used in the experiments. These systematic errors of adding or masking out lines or squares at fixed positions actually \"simplify\" the data, and the variance is reduced.\n- The problem is clearly stated, but to me it is not evident what would be a practical application of this algorithm and a real use case. First of all, the fact that a labelled (trusted) data set is needed (albeit small) constrains the practical usage. Second, the use case of detecting images where some pixels are set to default values (e.g. from a deficient camera) would be dependent on a specific sensor. Finally, the application of watermarks is mentioned and makes more sense. However, the experimental evaluation does not include this more complex type of outliers.\n- There is an extensive appendix, which would be Ok. But the main paper misses some explanations and details that are only mentioned in the apendices (for example appendix E and F)\n\nSome minor remarks: \n- Section 4.2. on the variational model mixes the theoretical model with some implementation aspects, like \"stop gradient\" (which is only related to training/optimisation). Also some parts are not explained, like the distribution \"pi\".\n- The figures and tables are too small.\n", "summary_of_the_paper": "This paper presents a neural network model based on Variational Autoencoders (VAE) that learn an implicit representation separating outliers with systematic \"errors\" from inliers using a small labelled subset of the training data set (trusted set).\nMore specifically, clean data and recurring systematic errors are represented in separate latent subspaces, where outliers are supposed to be a (linear) combination of such clean and \"dirty\" patterns.\nAfter training the model, there are two tasks: outlier detection and automated repair, removing the \"dirty\" pattern(s) from the detected outliers.\n", "summary_of_the_review": "The paper presents and interested approach that is well formulated. However, the main motivations and assumptions are not well comprehensible, theoretical and practical implementation aspects are mixed, and the presentation is lacking clarity in some places.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline paper with 2 marginally above and a marginally below acceptance recommendations. While the authors provided valid responses to some of the criticism, I still find some of the motivation and assumptions not sufficiently clear, theoretical and practical issues are mixed, and the validation on only synthetic data raises practical questions.
Decision: Reject