Title: Unsupervised Anomaly Detection in The Presence of Missing Values
Venue: NeurIPS 2024
Accepted: yes
Keywords: missing data

anomaly detection

deep learning
Abstract: Anomaly detection methods typically require fully observed data for model training and inference and cannot handle incomplete data, while the missing data problem is pervasive in science and engineering, leading to challenges in many important applications such as abnormal user detection in recommendation systems and novel or anomalous cell detection in bioinformatics, where the missing rates can be higher than 30\% or even 80\%. In this work, first, we construct and evaluate a straightforward strategy, ''impute-then-detect'', via combining state-of-the-art imputation methods with unsupervised anomaly detection methods, where the training data are composed of normal samples only. We observe that such two-stage methods frequently yield imputation bias from normal data, namely, the imputation methods are inclined to make incomplete samples ''normal", where the fundamental reason is that the imputation models learned only on normal data and cannot generalize well to abnormal data in the inference stage. To address this challenge, we propose an end-to-end method that integrates data imputation with anomaly detection into a unified optimization problem. The proposed model learns to generate well-designed pseudo-abnormal samples to mitigate the imputation bias and ensure the discrimination ability of both the imputation and detection processes. Furthermore, we provide theoretical guarantees for the effectiveness of the proposed method, proving that the proposed method can correctly detect anomalies with high probability. Experimental results on datasets with manually constructed missing values and inherent missing values demonstrate that our proposed method effectively mitigates the imputation bias and surpasses the baseline methods significantly. The source code of our method is available at https://github.com/jicongfan/ImAD-Anomaly-Detection-With-Missing-Data.

Unsupervised Anomaly Detection in The Presence of Missing Values

Abstract

Anomaly detection methods typically require fully observed data for model training and inference and cannot handle incomplete data, while the missing data problem is pervasive in science and engineering, leading to challenges in many important applications such as abnormal user detection in recommendation systems and novel or anomalous cell detection in bioinformatics, where the missing rates can be higher than 30% or even 80%. In this work, first, we construct and evaluate a straightforward strategy, 'impute-then-detect', via combining state-of-the-art imputation methods with unsupervised anomaly detection methods, where the training data are composed of normal samples only. We observe that such twostage methods frequently yield imputation bias from normal data, namely, the imputation methods are inclined to make incomplete samples 'normal', where the fundamental reason is that the imputation models learned only on normal data and cannot generalize well to abnormal data in the inference stage. To address this challenge, we propose an end-to-end method that integrates data imputation with anomaly detection into a unified optimization problem. The proposed model learns to generate well-designed pseudo-abnormal samples to mitigate the imputation bias and ensure the discrimination ability of both the imputation and detection processes. Furthermore, we provide theoretical guarantees for the effectiveness of the proposed method, proving that the proposed method can correctly detect anomalies with high probability. Experimental results on datasets with manually constructed missing values and inherent missing values demonstrate that our proposed method effectively mitigates the imputation bias and surpasses the baseline methods significantly. The source code of our method is available at https:// github.com/jicongfan/ImAD-Anomaly-Detection-With-Missing-Data .

1 Introduction

Anomaly detection (AD) [Breunig et al., 2000, Schölkopf et al., 2001, Liu et al., 2008, Pevn` y, 2016, Zong et al., 2018, Ruff et al., 2018, Cai and Fan, 2022, Fu et al., 2024, Zhang et al., 2024, Xiao et al., 2025], aiming at identifying anomalous or novel samples in data, is a crucial machine learning problem. It finds extensive applications in many high-stakes fields such as biology, healthcare, finance, and cybersecurity. Data missing or incompleteness, a persistent and unavoidable issue in many real-world situations, often arises during the processes of data collection, transmission, and storage. Moreover, in fields like bioinformatics (e.g. single-cell RNA sequencing) [Zhang and Zhang, 2018], psychology (e.g. questionnaire data) [Schlomer et al., 2010], and recommendation systems (e.g. user-item interaction data) [Shani and Gunawardana, 2011, Fan et al., 2024], the data missing rates are often higher than 30% or even 80% . Indeed, the missing data problems lead to

many challenges for anomaly detection, such as detecting anomalous cells or rare cell types based on incomplete single-cell RNA sequencing data [Fa et al., 2021] and identifying abnormal users in recommendation systems [Yang and Cai, 2017]. Regrettably, most existing AD methods necessitate complete data in both the training and test sets, rendering them ill-equipped to handle datasets with missing values. Consequently, addressing the AD challenge in the context of incomplete data becomes both necessary and inevitable.

A naive strategy is filling the missing values by statistical characteristics such as mean or median and then performing anomaly detection. Taking two real-world datasets 'Adult' and 'KDD'as examples, we consider the mechanism missing completely at random and fill the missing entries with the variable means and then perform a classical AD methods Isolation Forest [Liu et al., 2008]) and two deep learning based AD methods (Deep SVDD [Ruff et al., 2018] and NeutraL AD [Qiu et al., 2021]). The results are shown in Figure 1. The detection accuracies of the four methods degrade signifi-



























IForest

Deep SVDD

NeuTraL AD

























IForest

Deep SVDD

NeuTraL AD

cantly with the missing rate increases. This verified the failure of the naive strategy and the difficulty of unsupervised anomaly detection with missing values. Besides the naive imputation, one may consider using more powerful imputation algorithms [Dempster et al., 1977, Pigott, 2001, Candes and Recht, 2012, Stekhoven and Bühlmann, 2012, Gondara and Wang, 2018, Yoon et al., 2018, Fan et al., 2020, Muzellec et al., 2020] to recover the missing values and subsequently implementing AD algorithms on imputed data. We refer to this strategy as 'impute-then-detect'.

It is worth noting that, for unsupervised anomaly detection, where the training set is composed of only normal samples, such 'impute-then-detect' methods would yield imputation bias for normal data, i.e., the imputation methods are inclined to recover an abnormal sample with missing values as 'normal' as possible during the inference, which leads to lower recall or higher false negative rate. Figure 2 clearly shows the negative impacts of the imputation bias, which is worth studying and addressing. The main challenge is that the training set and test set do not satisfy the condition of identical distribution and the imputation model trained only on incomplete normal data does not generalize well to incomplete ab-





















complete data

imputed by MissForest

imputed by GAIN





















complete data

imputed by MissForest

imputed by GAIN

normal data. In Section 4.2, we quantitatively and comprehensively evaluate the 'impute-then-detect' methods using state-of-the-art imputation algorithms and AD algorithms.

To tackle the aforementioned problem, in this paper, we propose an end-to-end method, called ImAD, for unsupervised anomaly detection on incomplete data. The main idea of ImAD is to integrate data im putation and a nomaly d etection into a unified optimization objective and alleviate imputation bias by automatically learning to generate pseudo-abnormal samples. Note that the pseudo-abnormal samples are by-products of the training process and we do not use any extra data in all experiments. Our contributions are summarized as follows.

We study the imputation bias problem of the 'impute-then-detect' pipeline and quantitatively evaluate their detection performance.

We propose a novel method ImAD for AD on incomplete data. To the best of our knowledge, it is the first end-to-end unsupervised AD method in the presence of missing value.

We provide theoretical guarantees for ImAD, proving that it can correctly detect anomalies with high probability.

We compare ImAD with more than 9 baselines on 11 real datasets of various domains, covering datasets with manually constructed missing values and datasets with inherent missing values.

2 Related Work

Missing Data Imputation Data imputation fills missing data with plausible values and provides imputed data for downstream tasks such as classification, clustering, and visualization. As the missing data problem is prevalent in many fields, the study on missing data imputation is extensive, and many algorithms have been proposed in the past decades. Mayer et al. [2019] pointed out that there are approximately 150 implementations available to handle missing data. These methods can be roughly organized into three categories. The first category is based on the iterative regression model, such as well-known methods Multiple Imputation by Chained Equations (MICE) [Royston and White, 2011] and MissForest [Stekhoven and Bühlmann, 2012] that trains random forests on observed data through an iterative imputation scheme. The second category is the matrix completion methods [Candes and Recht, 2012, Mazumder et al., 2010, Fan and Chow, 2018, Fan et al., 2019, 2020]. The third category is based on deep learning especially deep generative models [Fan and Chow, 2017, Yoon et al., 2018, Li et al., 2019, Muzellec et al., 2020]. For instance, Yoon et al. [2018] proposed generative adversarial imputation network (GAIN) based on generative adversarial network (GAN) [Goodfellow et al., 2014] and [Tashiro et al., 2021] proposed conditional score-based diffusion models for probabilistic time-series imputation (CSDI) based diffusion model [Sohl-Dickstein et al., 2015]. Indeed, these deep imputation methods often achieve state-of-the-art performance in the tasks of missing data imputation, when the distributions of the training data and testing data are identical. However, their performance in recovering the missing values for unsupervised AD is rarely studied.

Anomaly detection on incomplete data The research on anomaly detection in the presence of missing values is very limited. To the best of the authors' knowledge, [Zemicheal and Dietterich, 2019] is the first work evaluating the detection performance of anomaly detection methods combined with different data imputation techniques. Their experiments of anomaly detection on a few UCI datasets with missing values showed that implementations of unsupervised anomaly detection methods such as Isolation Forest [Liu et al., 2008] on incomplete data should always include algorithms for handling missing values and the imputation contributes to improving the detection performance of anomaly methods. Fan et al. [2022] studied the problem of statistical process monitoring with missing values and proposed a fast incremental nonlinear matrix completion method for online and sequential imputation. Sarda et al. [2023] provided a study of existing unsupervised anomaly detection methods on GAN-imputed data.

It's worth noting that the strategies used in [Zemicheal and Dietterich, 2019, Fan et al., 2022, Sarda et al., 2023] are two-stage methods, where the imputation models are trained on the training dataset that does not contain any abnormal data or only contains very few unlabeled outliers. As a result, the imputation model will not generalize well on abnormal data during the inference and will use the learned pattern of normal data to fill the missing values of abnormal data, which makes the abnormal data similar to normal data and hence lowers detection accuracy. In contrast, our method integrates data imputation and anomaly detection into a unified process, and alleviates the imputation bias via introducing pseudo-abnormal samples, and hence achieves superior detection accuracy.

3 Proposed Method

3.1 Problem Formulation and Our Motivation

Given n samples x 1 , x 2 , · · · , x n drawn from an unknown distribution D x ⊆ R m , the goal of unsupervised AD is to learn a decision function f : R m → { 0 , 1 } by utilizing only these n samples, such that f ( x ) = 0 if x ∈ D x and f ( x ) = 1 if x / ∈ D x . We consider the scenario that X := [ x ⊤ 1 , x ⊤ 2 , · · · , x ⊤ n ] ⊤ ∈ R n × m contains missing values. Let M ∈ { 0 , 1 } n × m be a mask matrix determined by some missing mechanism M such as MCAR, MAR, or MNAR, where M i,j = 1 means X i,j is observed and M i,j = 0 means X i,j is missing. Then the observed incomplete matrix is

where ⊙ is the Hadamard product. Equation (1) implies that the missing values of X are temporarily filled with zeros. In many scenarios such as gene expression data analysis, recommendation systems, and questionnaire surveys, the data missing rate in ˘ X is often high. Training an anomaly detection model f on ˘ X and using it to detect anomalies in new incomplete data has practical significance

such as detecting anomalous cells or rare cell types in bioinformatics, identifying abnormal users in recommendation systems, and recognizing unusual subjects using questionnaires of psychology.

As mentioned before, conventional AD methods are vulnerable to missing values and a good imputation algorithm can raise the detection accuracy of an anomaly detection method to some extent. However, the strategy 'impute-then-detect' is inclined to make incomplete abnormal samples normal and hence cannot provide satisfactory detection performance. Therefore, in this work, we aim to provide an end-to-end unsupervised anomaly detection method in the presence of missing values to mitigate the imputation bias and improve the detection accuracy. The most challenging problem is that the imputation model (denoted as I ) trained only on incomplete normal data cannot generalize well to incomplete abnormal data. To solve the challenge, we take the following strategy and consideration.

We propose to learn a model that can generate some pseudo-abnormal samples, and then learn an imputation model from both the original normal data and the generated pseudo-abnormal samples. Thus, the learned imputation model can generalize well to incomplete abnormal data during inference and recover the missing values with high accuracy, which further improves the accuracy of anomaly detection. However, we encounter the following issues.

It is non-trivial to generate meaningful pseudo-abnormal samples that are similar enough to real ones. The reason is that the distribution (i.e., D x ) of training data is unknown and the data dimension m is often high.

The incompleteness of X further increases the difficulty of generating pseudo-abnormal samples.

On the other hand, the generated pseudo-abnormal samples should not be too far from the normal data, where a large gap will make the learned imputation model fail to impute the abnormal samples close to normal data and cause the abnormal samples to be hard to detect.

The generating model, imputation model, and detection model should be coordinated with each other and as a whole to ensure the reliability of the inference.

3.2 Learning Framework of ImAD

To address the aforementioned challenges, we propose to find a d -dimensional latent space Z where the normal data are lying and then generate pseudo-abnormal samples around the normal samples in Z . The samples in Z will be mapped back by a neural network to the original data space, yielding reliable pseudo-abnormal data.

We define D z as the latent distribution of the normal data in Z and define D ˜ z as the latent distribution of pseudo-abnormal data in Z . Since the patterns of normality are limited and the patterns of abnormality are unlimited, we let D z be a truncated Gaussian distribution (a hyperball denoted by B , with radius r 1 ) in Z and assume that the remaining region of Z is the abnormal region, denoted as Z \ B . It should be pointed out that there is no need to define D ˜ z in the entire space Z \ B , which will be explained in the discussion for Theorem 3.2(b) and further supported by Theorem 3.4 in Section

3.4. Instead, we only need to define D ˜ z in a small region of Z \ B that encloses B , which will reduce the uncertainty of random sampling (or samples size equivalently) and make it easier for mapping the samples back to the original data space. Thus, we define D ˜ z as a hypershell surrounding B and let D ˜ z be a truncated Gaussian. The radii of the two hyperspheres forming the hypershell are r 1 and r 2 respectively, where r 2 > r 1 . An illustration of D z and D ˜ z in 2-D space is shown in Figure 3, where D z and D ˜ z are truncated Gaussian from N ( 0 , 0 . 5 2 · I 2 ) and N ( 0 , I 2 ) respectively. The theoretical analysis for sampling from D z and D ˜ z is in Appendix A. We learn a reconstructor R : R d → R m to transform the samples drawn from D z to the original data distribution D x , i.e.,

R is actually a reconstruction model that recovers the original data from the latent space Z . With D ˜ z and R , we can obtain a distribution D ˜ x of pseudo-abnormal data in the original data space as

The samples (denoted by ˜ x ) drawn from D ˜ x are reasonable pseudo-abnormal samples, which will be explained by the discussion for Theorem 3.2(a) in Section 3.4.

GLYPH(cmap:d835)11

0

GLYPH(cmap:d835)23

GLYPH(cmap:d835)22

GLYPH(cmap:d835)12

GLYPH(cmap:d835)25

GLYPH(cmap:d835)15

GLYPH(cmap:d835)14

GLYPH(cmap:d835)33

GLYPH(cmap:d835)31

GLYPH(cmap:d835)13

GLYPH(cmap:d835)11

GLYPH(cmap:d835)23

GLYPH(cmap:d835)22

GLYPH(cmap:d835)12

GLYPH(cmap:d835)25

GLYPH(cmap:d835)15

GLYPH(cmap:d835)14

GLYPH(cmap:d835)33

GLYPH(cmap:d835)31

GLYPH(cmap:d835)21

GLYPH(cmap:d835)35

GLYPH(cmap:d835)34

GLYPH(cmap:d835)32

GLYPH(cmap:d835)24

GLYPH(cmap:d835)24

GLYPH(cmap:d835)22

GLYPH(cmap:d835)12

GLYPH(cmap:d835)35

GLYPH(cmap:d835)15

GLYPH(cmap:d835)13

GLYPH(cmap:d835)33

GLYPH(cmap:d835)31

GLYPH(cmap:d835)24

GLYPH(cmap:d835)22

GLYPH(cmap:d835)12

GLYPH(cmap:d835)35

GLYPH(cmap:d835)15

GLYPH(cmap:d835)13

GLYPH(cmap:d835)33

GLYPH(cmap:d835)

31 GLYPH(cmap:d835)32

GLYPH(cmap:d835)14

GLYPH(cmap:d835)21

GLYPH(cmap:d835)23

GLYPH(cmap:d835)34

GLYPH(cmap:d835)25

GLYPH(cmap:d835)11

GLYPH(cmap:d835)(X) ≈ Z

GLYPH(cmap:d835)(X) ≈ Z

: starting point of normal data

: starting point of pseudo-abnormal data

GLYPH(cmap:d835)

GLYPH(cmap:d835)

X

X

X = GLYPH(cmap:d835)(X)

X = GLYPH(cmap:d835)(X)

X ≈ ℛ(GLYPH(cmap:d835)(X))⨀M

ℛ(Z)⨀M

Reconstructor

Projector

Imputer

GLYPH(cmap:d835)

GLYPH(cmap:d835)

0

0

0

0

0

0

0

0

0

0

0

0

Reconstructor

Now we use a projector P : R m → R d to transform D x and D ˜ x into D z and D ˜ z respectively, i.e.,

However, the training set ˘ X = M ( X ) is incomplete, and we need to learn an imputation model I to recover the missing values, i.e., ˆ X = I ( ˘ X ) . More generally, we denote

We hope that the imputation model is also able to recover the missing values of the generated pseudo-abnormal samples if they have, though they are complete. We thus remove some values of the generated pseudo-abnormal samples ˜ x ∼ D ˜ x using missing mechanism ˜ M and let D ˘ ˜ x = ˜ M ( D ˜ x ) .

The missing values are then recovered by

This step mitigates the problem of imputation bias encountered by the 'impute-then-detect' methods.

Let E I , E P , and E R denote some distance or discrepancy measure between distributions. We here show how to achieve the goals of (2), (3), (4), (5), and (6) in a unified optimization problem. First, for normal data, we solve

For the generated pseudo-abnormal data, we solve

Let ̂ E · be a finite-sample estimation of E · . Combining (7) and (8), we obtain the objective of ImAD:

where ˘ ˜ X = R ( ˜ Z ) ⊙ ˜ M , ˆ ˜ X = I ( ˘ ˜ X ) , and [ · ; · ] denotes the row-wise concatenation of two matrices. In (9), the samples in Z are drawn from D z and the samples in ˜ Z are drawn from D ˜ z . The roles of the three parts of the objective function in (9) are analyzed as follows.

L ( DI ) denotes the data imputation loss. With this loss, the imputation model will be able to recover the missing values of normal data and abnormal data.

L ( AD ) denotes the anomaly detection loss. With this loss, the anomaly detection model will be discriminative and be able to project normal data and abnormal data into different regions in Z .

L ( RE ) denotes the reconstruction loss. This loss is to ensure that D z and D ˜ z are meaningful.

We see that our method ImAD couples data imputation with anomaly detection to a unified optimization objective. Figure 4 depicts the overall framework of ImAD, where the green and red arrows show the flow paths of normal data (starting from ˘ X ) and pseudo-abnormal data (starting from ˜ Z ) respectively. The reconstructors in Figure 4 share parameters.

3.3 Specific Implementation of ImAD

We use three neural networks h ψ , f θ and g ϕ with parameters ψ, θ, ϕ to model I , P and R respectively. For E , we consider two different cases. If the samples are pair-wise, we directly use the square loss, which is simple and efficient. Thus, in L DI and L RE , we use the square loss, and the square loss for L RE is masked by M . When the samples are not pair-wise, we take advantage of the Sinkhorn distance [Cuturi, 2013] derived from the optimal transport theory. The Sinkhorn distance between two distributions D U and D V supported by their finite samples U = { u 1 , u 2 , · · · , u n u } ∼ D u and V = { v 1 , v 2 , · · · , v n v } ∼ D v is defined as

where P ∈ R n u × n v is the transport plan and C ∈ R n u × n v is the metric cost matrix. The two probability vectors a and b satisfy a T 1 = 1 , b T 1 = 1 , and η ≥ 0 is a trade-off between the Wasserstein distance and entropy regularization.

By applying h ψ , f θ , g ϕ , square loss, and Sinkhorn distance to (9), we obtain the following problem:

Solving the problem (11), we get well trained imputer h ψ ∗ and projector f θ ∗ . For a new sample ˘ x new containing missing values, we define an anomaly score s ( · ) by

which is the distance to the origin in the latent space. If s (˘ x new ) > r 1 , ˘ x new is detected as abnormal. Otherwise, ˘ x new is treated as a normal sample.

3.4 Theoretical Guarantees for ImAD

WLOG, we assume f θ , g ϕ , and h ψ all have L layers, where θ = { W f 1 , W f 2 , . . . , W f L } , ϕ = { W g 1 , W g 2 , . . . , W g L } , and ψ = { W h 1 , W h 2 , . . . , W h L } . Denote the spectral norm and ℓ 2 , 1 -norm of a matrix as ∥ · ∥ σ and ∥ · ∥ 2 , 1 respectively. We also make the following assumptions.

Assumption 3.1. For f θ , g ϕ , and h ψ , the following conditions hold: 1) ∥ W f l ∥ σ ≤ α f , ∥ W g l ∥ σ ≤ α g , ∥ W h l ∥ σ ≤ α h , ∀ l ∈ [ L ] ; 2) ∥ W f l ∥ 2 , 1 ≤ b f , ∥ W g l ∥ 2 , 1 ≤ b g , ∥ W h l ∥ 2 , 1 ≤ b h , ∀ l ∈ [ L ] ; 3) all activation functions in f θ , g ϕ , and h ψ are ρ -Lipschitz continuous; 4) the maximum width of the layers in f θ , g ϕ , and h ψ is ¯ d .

The following theorem can be used to obtain some deterministic guarantee for ImAD.

Theorem 3.2. Under Assumption 3.1, we have:

Theorem 3.2(a) indicates that in the latent space Z , if an abnormal sample ˜ z ∼ D ˜ z is close to a normal sample z ∼ D z , in the original data space, the corresponding abnormal sample ˜ x is still close to the normal sample x provided that α g is not too large. This means the generated pseudo-abnormal

samples are practical and useful. For Theorem 3.2(b), let's consider an incomplete abnormal sample ˘ ˜ x and assume that its closest incomplete pseudo-abnormal sample generated by the ˜ z on the outer hypersphere (shown in Figure 3) is ˘ ˜ x ∗ , where ∥ ˘ ˜ x -˘ ˜ x ∗ ∥ = β . Then in the latent space, we have ∥ ˜ z -˜ z ∗ ∥ ≤ ρ 2 L α L f α L h β . Let the radii of the inner and outer hyperspheres be r 1 and r 2 respectively. Now we can conclude that if r 2 -r 1 > ρ 2 L α L f α L h β , ˜ z is outside the decision region given by the inner hypersphere and hence ˘ ˜ x is successfully detected as an abnormal sample.

Now we study the theoretical guarantees for our ImAD in the sense of expectation. Let r 1 be the thresholds for the anomaly score defined by (12) to determine whether a sample is normal or not. Let r 2 be the radius of the outer hypersphere enclosing D ˜ z . Let ¯ s ˘ x be the average anomaly score of the (incomplete) normal training data, i.e., ¯ s 2 ˘ x = 1 n ∑ n i =1 s (˘ x i ) 2 . Let ¯ ε 2 ˘ ˜ x := 1 n ∑ n i =1 | r 2 2 -s ( ˘ ˜ x i ) 2 | , where ˘ ˜ x i are the (incomplete) pseudo-abnormal samples generated during the training stage. With these definitions and Assumption 3.1, the following (proved in Appendix C) presents the theoretical generalization ability of our ImAD.

Theorem 3.3. Suppose the squared anomaly score s (˘ x ) 2 of normal data is always upper-bounded by γ , | r 2 2 -s ( ˘ ˜ x ) 2 | of the pseudo-abnormal data is always upper-bounded by ˜ γ , and the absolute output of f θ is always upper-bounded by ϑ . Suppose the samples in ˘ X and ˘ ˜ X are independently drawn D ˘ x and D ˘ ˜ x respectively. Define κ = α L f α L h , ζ = ( 1 + L ( b f α f ) 2 / 3 + L ( b h α h ) 2 / 3 ) 3 / 2 , ∆ = r 2 1 -¯ s 2 ˘ x , and ˜ ∆ = r 2 2 -r 2 1 -¯ ε 2 ˘ ˜ x .

(a) For normal data from D ˘ x , over the randomness of ˘ X ,

(b) For abnormal data from D ˘ ˜ x , over the randomness of ˘ ˜ X ,

Theorem 3.3(a) means that a normal sample, in expectation, is detected as anomalous with probability at almost δ , where δ is close to zero under some mild conditions such as L is not too large and n is not too small. In other words, a false alarm happens with low probability. Theorem 3.3 (b) means that an abnormal sample drawn from D ˘ ˜ x , in expectation, can be successfully detected with probability at least 1 -δ , where δ is close to zero under some mild conditions. Theorem 3.3(b) also indicates that a larger r 2 is better. It is worth noting that here we only focus on D ˘ ˜ x , which is defined by D ˜ z , ˜ M , and g ϕ . D ˘ ˜ x can be regarded as a distribution of difficult anomalous data that are close to normal data. The anomalous samples drawn from space out of D ˘ ˜ x are much easier to detect, which is further supported by the following theorem (proved in Appendix D).

Theorem 3.4. Let c be a constant satisfying ∥ f θ ◦ h ψ ◦ g ϕ ( z ) -f θ ◦ h ψ ◦ g ϕ ( z ′ ) ∥ ≥ c ∥ z -z ′ ∥ for any z , z ′ and assume that ∥ f θ ◦ h ψ ◦ g ϕ ( 0 ) -0 ∥ ≤ ε . Any samples drawn from the space out of D ˘ ˜ x can be correctly detected if cr 2 -ε > r 1 .

4 Experiments

4.1 Datasets, Baselines, and Implementation Details

We compare ImAD with 'impute-then-detect' methods on 11 publicly available tabular datasets from various fields, including seven datasets with manually constructed missing values and four datasets with inherent missing values. In all experiments, only incomplete normal data are used in the training stage, but there are both incomplete normal and abnormal data during the inference. The statistics of all datasets are in Table 1 and a detailed description of all datasets is in Appendix J. Considering the 'impute-then-detect' strategy, for data imputation, we use MissForest [Stekhoven and Bühlmann, 2012] and GAIN [Yoon et al., 2018]. For anomaly detection, we use Isolation Forest [Liu et al.,

2008], Deep SVDD [Ruff et al., 2018], NeutraL AD [Qiu et al., 2021] and DPAD [Fu et al., 2024]. The pairwise combination between the imputation and anomaly detection methods yields eight 'impute-then-detect' baselines.

We use MLPs to construct the three modules of ImAD, Adam [Kingma and Ba, 2015] as the optimizer and set coefficient η of entropy regularization term in Sinkhorn distance to 0.1 in all experiments. Other experimental hyper-parameters are provided in Appendix J. Sensitivity analysis of hyper-parameters is provided in Appendix I. A detailed description of distinct missing mechanisms, including MCAR, MAR, and MNAR, is provided in Appendix J. In this study, we let the missing rate mr be 0.2 or 0.5, which is consistent with the previous data imputation works [Yoon et al., 2018, Muzellec et al., 2020]. We use the AUROC (Area Under the Receiver Operating Characteristic curve) and AUPRC (Area Under the Precision-Recall curve) to evaluate the detection performance. ALL experiments were conducted on 20 Cores Intel(R) Xeon(R) Gold 6248 CPU with one NVIDIA Tesla V100 GPU, CUDA 12.0. We report the average results of five runs.

4.2 Experimental Results on Datasets with Manually Constructed Missing Values

Before presenting the numerical results, we show the effectiveness of the generated pseudo-abnormal samples learned for the Botnet dataset in Figure 5, where we directly let the latent space Z be 2-D for convenient visualization. We see that the pseudo-abnormal samples cover the region of real abnormal samples, which matches our motivation and expectation.

abnormlsample

normalsample

missingrate=0.2

pseudoabnormalsample

missingrate=0.5

The results of anomaly detection with missing data under the setting of MCAR are shown in Table 2 and more results under MCAR are provided in Appendix K. In Table 2, 'Mean-Filling' denotes that the missing values are filled with feature means.

We have the following observations from the Table 2:

The detection performance of 'impute-then-detect' methods does not decrease with the increasing of missing rate from 0 . 2 to 0 . 5 in some cases (emphasized by underline), which indicates the adverse impact of imputation bias for the detection algorithm. The main reason is that a

lower missing rate implies a simpler imputation task, leading to a more pronounced imputation bias from normal data, which makes the abnormal data more 'normal', thereby increasing the difficulty of detection for such two-stage methods.

The 'impute-then-detect' methods with 'MissForest' (simple and shallow imputation algorithms) achieve better detection performance than those with 'GAIN' (generative and deep imputation model) in most cases, suggesting that a sophisticated imputation module may not contribute positively to subsequent anomaly detection because the identical distribution assumption does not hold here. The outstanding recovery ability leads to a pronounced imputation bias and further affects the detection task.

Compared with all baselines, ImAD achieves better detection performance in almost all cases. Besides, different from the 'impute-then-detect' methods, the performance of ImAD increases with the changes of missing rate from 0 . 5 to 0 . 2 in all cases. This indicates that the imputation module of ImAD generalizes well on incomplete abnormal data and the generated pseudoabnormal samples can alleviate the bias.

4.3 Experimental Results on Datasets with Inherent Missing Values

We report experimental results on the four datasets with inherent missing values in Table 3, where the naive imputation methods 'Zero-Filling' and 'Mean-Filling' are also considered. Observing Table 3, we notice that the naive imputation methods are insufficient for subsequent detection tasks when facing high missing rates and the imputation bias impacts the detection accuracy of 'impute-thendetect' methods. Our ImAD outperforms all baselines in all cases. It indicates that our proposed method is practical and effective for real-world anomaly detection with missing data.

4.4 Impact of Different Missing Mechanisms

Given a real dataset, the missing mechanism is usually unknown and difficult to estimate. It is expected that when the missing mechanism ˜ M in generating (incomplete) pseudo-normal samples is closer to the missing mechanism M in the real data, the performance of ImAD should be better. In this section, we analyze the impact of different ˜ M on the detection performance of ImAD. Note that

for the synthetic incomplete data, we accurately know the missing mechanism. The experimental results are reported in Table 4. On real incomplete data, our method is robust to the setting of missing mechanism ˜ M and has better overall performance when ˜ M is MCAR. Therefore, based on Occam's Razor principle and the empirical results, we recommend using MCAR as the missing mechanism for the generated pseudo-abnormal samples when the real missing mechanism is unknown. On the other hand, as shown in Table 4, on synthetic incomplete data, detection performance degrades when ˜ M is different from M .

4.5 More Experimental Results

The appendices contain the following additional results: I. Performance gain from pseudo-abnormal samples (Appendix G); II. Influence of the constrained radii r 1 , r 2 (Appendix H); III. Sensitivity analysis of hyperparameters (Appendix I); IV. Impact of different missing rates for training and test set(Appendix K); V. Results of MAR and MNAR (Appendix K).

5 Conclusion

This paper proposed ImAD, the first end-to-end unsupervised anomaly detection method on incomplete data. ImAD integrates data imputation with anomaly detection into a unified optimization objective and automatically generates pseudo-abnormal samples to alleviate the imputation bias. We theoretically proved the effectiveness of ImAD and empirically evaluated ImAD on multiple real-world datasets. The results showed that ImAD mitigates imputation bias from normal data and provides an effective solution for unsupervised anomaly detection in the presence of missing values. One limitation of this work is that we haven't considered the applications on incomplete image data and incomplete time series.

Acknowledgments

This work was supported by the General Program of Natural Science Foundation of Guangdong Province under Grant No.2024A1515011771, the National Natural Science Foundation of China under Grant No.62376236, the Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (2023B1212010001), Shenzhen Science and Technology Program ZDSYS20230626091302006, Shenzhen Stability Science Program 2023, and Hetao ShenzhenHong Kong Science and Technology Innovation Cooperation Zone Project (No.HZQSWS-KCCYB2024016). The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems , 30, 2017.

Barry Becker and Ronny Kohavi. Adult. UCI Machine Learning Repository, 1996. DOI: https://doi.org/10.24432/C5XW20.

Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying densitybased local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data , pages 93-104, 2000.

Jinyu Cai and Jicong Fan. Perturbation learning based anomaly detection. Advances in Neural Information Processing Systems , 35, 2022.

Emmanuel Candes and Benjamin Recht. Exact matrix completion via convex optimization. Communications of the ACM , 55(6):111-119, 2012.

Zhenpeng Chen, Jie M Zhang, Max Hort, Mark Harman, and Federica Sarro. Fairness testing: A comprehensive survey and analysis of trends. ACM Transactions on Software Engineering and Methodology , 2023.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems , 26, 2013.

Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological) , 39(1): 1-22, 1977.

Botao Fa, Ting Wei, Yuan Zhou, Luke Johnston, Xin Yuan, Yanran Ma, Yue Zhang, and Zhangsheng Yu. Gapclust is a light-weight approach distinguishing rare cells from voluminous single cell expression profiles. Nature Communications , 12(1):4197, 2021.

Jicong Fan and Tommy Chow. Deep learning based matrix completion. Neurocomputing , 266: 540-549, 2017.

Jicong Fan and Tommy WS Chow. Non-linear matrix completion. Pattern Recognition , 77:378-394, 2018.

Jicong Fan, Lijun Ding, Yudong Chen, and Madeleine Udell. Factor group-sparse regularization for efficient low-rank matrix recovery. Advances in neural information processing Systems , 32, 2019.

Jicong Fan, Yuqian Zhang, and Madeleine Udell. Polynomial matrix completion for missing data imputation and transductive learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3842-3849, 2020.

Jicong Fan, Tommy W. S. Chow, and S. Joe Qin. Kernel-based statistical process monitoring and fault detection in the presence of missing data. IEEE Transactions on Industrial Informatics , 18 (7):4477-4487, 2022. doi: 10.1109/TII.2021.3119377.

Jicong Fan, Rui Chen, Zhao Zhang, and Chris H.Q. Ding. Neuron-enhanced autoencoder matrix completion: Theory and practice. In The Twelfth International Conference on Learning Representations , 2024.

Dazhi Fu, Zhao Zhang, and Jicong Fan. Dense projection for anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 8398-8408, 2024.

Lovedeep Gondara and Ke Wang. Mida: Multiple imputation using denoising autoencoders. In Advances in Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part III 22 , pages 260-272. Springer, 2018.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

Soyeon Caren Han, Taejun Lim, Siqu Long, Bernd Burgstaller, and Josiah Poon. Glocal-k: Global and local kernels for recommender systems. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 3063-3067, 2021.

Xiao Han, Lu Zhang, Yongkai Wu, and Shuhan Yuan. Achieving counterfactual fairness for anomaly detection. In Pacific-Asia Conference on Knowledge Discovery and Data Mining , pages 55-66. Springer, 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations , 2015.

Steven Cheng-Xian Li, Bo Jiang, and Benjamin Marlin. Misgan: Learning from incomplete data with generative adversarial networks. arXiv preprint arXiv:1902.09599 , 2019.

Lichman. Uci machine learning repository, 2013. URL http://archive.ics.uci.edu/ml .

Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining , pages 413-422. IEEE, 2008.

Imke Mayer, Aude Sportisse, Julie Josse, Nicholas Tierney, and Nathalie Vialaneix. R-miss-tastic: a unified platform for missing values methods and workflows. arXiv preprint arXiv:1908.04822 , 2019.

Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. Spectral regularization algorithms for learning large incomplete matrices. The Journal of Machine Learning Research , 11:2287-2322, 2010.

Yair Meidan, Michael Bohadana, Yael Mathov, Yisroel Mirsky, Dominik Breitenbacher, Asaf, and Asaf Shabtai. detection_of_iot_botnet_attacks_n_baiot. UCI Machine Learning Repository, 2018. DOI: https://doi.org/10.24432/C5RC8J.

Boris Muzellec, Julie Josse, Claire Boyer, and Marco Cuturi. Missing data imputation using optimal transport. In International Conference on Machine Learning , pages 7130-7140. PMLR, 2020.

Tomáš Pevn` y. Loda: Lightweight on-line detector of anomalies. Machine Learning , 102:275-304, 2016.

Therese D Pigott. A review of methods for missing data. Educational research and evaluation , 7(4): 353-383, 2001.

Chen Qiu, Timo Pfrommer, Marius Kloft, Stephan Mandt, and Maja Rudolph. Neural transformation learning for deep anomaly detection beyond images. In International Conference on Machine Learning , pages 8703-8714. PMLR, 2021.

Shebuti Rayana. Odds library, 2016. URL https://odds.cs.stonybrook.edu .

Patrick Royston and Ian R White. Multiple imputation by chained equations (mice): implementation in stata. Journal of statistical software , 45:1-20, 2011.

Donald B Rubin. Inference and missing data. Biometrika , 63(3):581-592, 1976.

Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning , pages 4393-4402. PMLR, 2018.

Kisan Sarda, Amol Yerudkar, and Carmen Del Vecchio. Unsupervised anomaly detection for multivariate incomplete data using gan-based data imputation: A comparative study. In 2023 31st Mediterranean Conference on Control and Automation (MED) , pages 55-60. IEEE, 2023.

Nicholas Schaum, Jim Karkanias, Norma F Neff, Andrew P May, Stephen R Quake, Tony WyssCoray, Spyros Darmanis, Joshua Batson, Olga Botvinnik, Michelle B Chen, et al. Single-cell transcriptomics of 20 mouse organs creates a tabula muris: The tabula muris consortium. Nature , 562(7727):367, 2018.

Gabriel L Schlomer, Sheri Bauman, and Noel A Card. Best practices for missing data management in counseling psychology. Journal of Counseling psychology , 57(1):1, 2010.

Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. Estimating the support of a high-dimensional distribution. Neural computation , 13(7):1443-1471, 2001.

Åsa Segerstolpe, Athanasia Palasantza, Pernilla Eliasson, Eva-Marie Andersson, Anne-Christine Andréasson, Xiaoyan Sun, Simone Picelli, Alan Sabirsh, Maryam Clausen, Magnus K Bjursell, et al. Single-cell transcriptome profiling of human pancreatic islets in health and type 2 diabetes. Cell metabolism , 24(4):593-607, 2016.

Guy Shani and Asela Gunawardana. Evaluating recommendation systems. Recommender systems handbook , pages 257-297, 2011.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

Daniel J Stekhoven and Peter Bühlmann. Missforest-non-parametric missing value imputation for mixed-type data. Bioinformatics , 28(1):112-118, 2012.

Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems , 34:24804-24816, 2021.

Dmitry Usoskin, Alessandro Furlan, Saiful Islam, Hind Abdo, Peter Lönnerberg, Daohua Lou, Jens Hjerling-Leffler, Jesper Haeggström, Olga Kharchenko, Peter V Kharchenko, et al. Unbiased classification of sensory neuron types by large-scale single-cell rna sequencing. Nature neuroscience , 18(1):145-153, 2015.

Feng Xiao, Jianfeng Zhou, Kunpeng Han, Haoyuan Hu, and Jicong Fan. Unsupervised anomaly detection using inverse generative adversarial networks. Information Sciences , 689:121435, 2025. ISSN 0020-0255. doi: https://doi.org/10.1016/j.ins.2024.121435.

Zhihai Yang and Zhongmin Cai. Detecting abnormal profiles in collaborative filtering recommender systems. Journal of Intelligent Information Systems , 48:499-518, 2017.

Jinsung Yoon, James Jordon, and Mihaela Schaar. Gain: Missing data imputation using generative adversarial nets. In International conference on machine learning , pages 5689-5698. PMLR, 2018.

Tadesse Zemicheal and Thomas G. Dietterich. Anomaly detection in the presence of missing values for weather data quality control. In Proceedings of the 2nd ACM SIGCAS Conference on Computing and Sustainable Societies , COMPASS '19, page 65-73, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367141. doi: 10.1145/3314344.3332490. URL https://doi.org/10.1145/3314344.3332490 .

Lihua Zhang and Shihua Zhang. Comparison of computational methods for imputing single-cell rna-sequencing data. IEEE/ACM transactions on computational biology and bioinformatics , 17(2): 376-389, 2018.

Yunhe Zhang, Yan Sun, Jinyu Cai, and Jicong Fan. Deep orthogonal hypersphere compression for anomaly detection. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=cJs4oE4m9Q .

Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International conference on learning representations , 2018.

A Analysis for Sampling in Latent Space

To project the normal data onto target distribution and generate pseudo-abnormal data, our proposed method involves sampling from two latent distributions D z , D ˜ z ∼ N ( 0 , σ 2 I d ) . In this section, we provide a lower bound for the constrained sampling radius r , given a sampling probability p that provides a probabilistic guarantee, namely, to obtain N points from a truncated Gaussian, we need to sample N/p times from a Gaussian distribution . Subsequently, we perform sampling within the truncated Gaussian distribution with a constrained radius r .

For target distribution D z , we expect that it is compact and can be easily sampled, in which the compactness is to ensure a clear and reliable decision boundary between normal and abnormal data. Therefore, we select truncated Gaussian from N ( 0 , σ 2 I d ) as target distribution D z and bound D z in a d -dimensional radius r hyperball centering at origin. For radius r , we have the following proposition.

Proposition A.1. Let F d denote the cumulative distribution function (CDF) of the chi-square distribution χ 2 ( d ) . For a given probability 0 < p < 1 , when r ≥ σ √ F -1 d ( p ) , the sampling probability in D z satisfies P ( ∥ z ∥ 2 < r 2 ) ≥ p where z = [ z 1 , z 2 , · · · , z d ] and z 1 , . . . , z d i.i.d. ∼ N (0 , σ 2 ) .

Proof. We have

Let Y = ∑ d i =1 z 2 i σ 2 , we get

Therefore,

According to the analysis in Section 3.2, we select truncated Gaussian from N ( 0 , ˜ σ 2 I d ) as target distribution D ˜ z and bound D ˜ z between two d -dimensional hyperspheres with radii r 1 , r 2 respectively, centering at origin, where r 2 > r 1 . For radius r 1 , r 2 , we have the following proposition.

Proposition A.2. Let F d denote the cumulative distribution function (CDF) of the chi-square distribution χ 2 ( d ) . For a given probability 0 < p < 1 , when r 1 ≤ ˜ σ √ F -1 d ( p 1 ) , r 2 ≥ ˜ σ √ F -1 d ( p 2 ) and satisfies p = p 2 -p 1 , the sampling probability in D ˜ z satisfies P ( r 2 1 < ∥ z ∥ 2 < r 2 2 ) ≥ p where z = [ z 1 , z 2 , · · · , z d ] and z 1 , . . . , z d i.i.d. ∼ N (0 , ˜ σ 2 ) .

Proof. According the proof for Proposition A.1, we have

Therefore, we get P ( ∥ z ∥ 2 < r 2 2 ) -P ( ∥ z ∥ 2 < r 2 1 ) = P ( r 2 1 < ∥ z ∥ 2 < r 2 2 ) ≥ p 2 -p 1 = p .

As shown in Figure 3, we set radius r 1 of D ˜ z equals to radius r of D z . Also, we maintain the settings r 1 = r in all experiments to make the introduced pseudo-abnormal samples are not far from the normal data.

B Proof for Theorem 3.2

Proof. Recall that g ϕ was defined as

Then for any z , ˜ z ∈ R d , we have

This finished the proof for part (a) of the theorem. The proof for part (b) is similar and omitted here for simplicity.

C Proof for Theorem 3.3

We define the following model class

where f θ and h ψ satisfy the Assumption 3.1 and π is the sum of squares of the outputs of f θ ◦ h ψ , corresponding to the definition of the anomaly score, meaning s ( x ) 2 = π ( f θ ( h ψ ( x ))) . The following lemma (proved by Appendix E) provides the covering number bound of F .

Lemma C.1. Under Assumption 3.1, for any ϵ > 0 , it holds that

Suppose the loss function is µ -Lipschitz, it follows from Lemma C.1 that

With the covering number, we can bound the Rademacher complexity by the following Dudley entropy integral:

Lemma C.2 (Lemma A.5 of [Bartlett et al., 2017], reformulated) . Let F γ := ℓ ◦F ˘ X be a real-valued function class taking values in [0 , γ ] , and assume that 0 ∈ F γ . Then

Combining (22) and Lemma C.2, and letting R 2 := µ 2 ∥ ˘ X ∥ 2 F ln(2 ¯ d 2 ) ρ 4 L (2 ϑ ) 2 dκ 2 ζ 2 , we obtain

where we have chosen α = 1 √ n .

The following lemma is the classical generalization error bound based on the Rademacher complexity.

Lemma C.3. Given hypothesis function space F mapping x ∈ X to R d and γ > 0 , define F γ := { ( x , y ) ↦→ l γ ( f ( x ) , y ) : f ∈ F} , where l γ (ˆ y, y ) ≤ γ . Then, with probability at least 1 -δ over a sample X of size n , every f ∈ F satisfies L γ ( f ) ≤ ˆ L γ ( f ) + 2 R X ( F γ ) + 3 γ √ ln (2 /δ ) 2 n .

Now using Lemma C.3 and inequality (23), we have

For Theorem 3.3(a), the loss function is ℓ (ˆ y, y ) = | ˆ y -y | = ˆ y , where y ≡ 0 and ˆ y = s (˘ x ) 2 due to the definition of the anomaly score. This also means that the Lipschitz constant µ of ℓ is 1 . We assume that the squared anomaly scores on the normal training data are upper bounded by γ . Let ˆ L γ ( f ) = 1 n ∑ n i =1 s (˘ x i ) 2 and L γ ( f ) = E D ˘ x [ s ( x ) 2 ] . It follows from (24) that

Let r 1 be the threshold determined by the training data to judge whether a sample is anomalous or not. We let and solve for δ :

where ¯ s 2 = 1 n ∑ n i =1 s (˘ x i ) 2 . Then we rewrite (25) as

which holds with probability at least 1 -δ . In other words,

because both s (˘ x ) and r 1 are nonnegative. We complete the proof for Theorem 3.3(a).

For Theorem 3.3(b), R 2 := µ 2 ∥ ˘ ˜ X ∥ 2 F ln(2 ¯ d 2 ) ρ 4 L (2 ϑ ) 2 dκ 2 ζ 2 . We consider the following loss function

The Lipshitz constant of this loss is 1 . Suppose the loss is upper bounded by ˜ γ . Similar to the proof for Theorem 3.3 (a), we have

which implies

Let and solve for δ :

where ¯ ε = 1 n ∑ n i =1 ℓ ( ˘ ˜ x i ) . Then we rewrite (32) as

which holds with probability at least 1 -δ . In other words,

Since | r 2 2 -s ( ˘ ˜ x ) 2 | ≤ τ implies that s ( ˘ ˜ x ) 2 ≥ r 2 2 -τ , letting τ = r 2 2 -r 2 1 , we arrive at

which also means because both s ( ˘ ˜ x ) and r 1 are nonnegative. Renaming δ as ˜ δ , we finish the proof. Note that in the theorem of the main paper, we have put outside the constant 4 in R 2 . That's why the constant 24 becomes 48 .

D Proof for Theorem 3.4

Proof. The anomalous samples (denoted by ˘ x ) drawn from space out of D ˘ x are much easier to detect. The reason is that, in the latent space, these anomalous samples (denoted by ˘ z ′ ) are sufficiently far from the normal region ( D z ). According to the definition of D ˘ x , we have ˘ z ′ = f ◦ h (˘ x ) = f ◦ h ◦ g (˘ z ) . According to the definition of the anomaly score, we need to measure ∥ ˘ z ′ -0 ∥ . We have

where c is some constant depending on the networks and we have assumed that ∥ f ◦ h ◦ g (0) -0 ∥ ≤ ε . Now suppose that r 2 is sufficiently large such that cr 2 -ε > r 1 , then ∥ ˘ z ′ -0 ∥ > r 1 , meaning that ˘ z ′ is outside the inner hypersphere and hence can be detected successfully. Nevertheless, determining an exact c is still an open problem for neural networks.

E Proof for Lemma C.1

Proof. In F , π can be regarded as an additional layer of the neural network, where the activation function for each element of z is square, the weight matrix for the output is a vector consisting of d ones, and the activation function for the final output is linear. Thus, π ◦ f θ ◦ g ψ has 2 L +1 layers. For the square activation function, the Lipschitz constant is 2 ϑ . For the final output layer, the spectral norm of the weights is √ d , which is equal to the ℓ 2 . 1 norm because it is a vector.

Lemma E.1 (Theorem 3.3 of [Bartlett et al., 2017]) . Let fixed nonlinearities ( σ 1 , . . . , σ L ) and reference matrices ( M 1 , . . . , M L ) be given, where σ i is ρ i -Lipschitz and σ i (0) = 0 . Let spectral norm bounds ( s 1 , . . . , s L ) , and matrix (2 , 1) norm bounds ( b 1 , . . . , b L ) be given. Let data matrix X ∈ R n × d be given, where the n rows correspond to data points. Let H X denote

the family of matrices obtained by evaluating X with all choices of network F A : H X := { F A ( X T ) : A = ( A 1 , . . . , A L ) , ∥ A i ∥ σ ≤ s i , ∥ ∥ A ⊤ i -M ⊤ i ∥ ∥ 2 , 1 ≤ b i } , where each matrix has dimension at most W along each axis. Then for any ϵ > 0 ,

Then using Lemma E.1 and Assumption 3.1, we can obtain

F Time Complexity Analysis

The notations used in the complexity analysis are explained as follows:

n, m to denote the number of samples of the training phase and inference phase, respectively.

Missforest is a well-known data imputation algorithm based on random forest ( O ( t 1 · v · n log n ) ) where t 1 denotes the number of trees, v denotes the number of attributes.

T, T g , T d , T ae , T oc denote the iterations of corresponding methods.

¯ L and ¯ d denote the number of layers of the neural network and the maximum width of the layers of the corresponding models, respectively.

t 2 denotes the number of trees of I-Forest and t is the maximum iterations of the Sinkhorn algorithm.

p, ψ, K denote the key parameters of the corresponding methods.

G Gain of Detection Performance from Pseudo-Abnormal Samples

G.1 ImAD Benefits from Learning Pseudo-abnormal Samples

In this section, we explore the influences of introduced pseudo-abnormal samples for detection performance. On all datasets used in our experiments, we remove the pseudo-abnormal samples in the training process and only use incomplete normal data to train ImAD. The experimental results are shown in Table 6 and Table 7, where the detection performance of ImAD is improved in all the cases when introducing pseudo-abnormal samples into the training process. This indicates that the generated pseudo-abnormal samples are practical and effective for anomaly detection on incomplete data.

G.2 ImAD's Pseudo-abnormal Samples Can Improve 'impute-then-detect' Methods

Furthermore, we save the generated pseudo-abnormal data from the training process of ImAD on Titanic and Bladder, and then we add them into the training set for data imputation of 'impute-thendetect' methods. The related results are provided in Table 8. We see that the pseudo-abnormal

samples learned by our ImAD can improve the performance of 'impute-then-detect' methods. The reason is that with the pseudo-abnormal data, the imputation algorithms, i.e., MissForest and GAIN, generalize better on the test data. These results further confirm the effectiveness of the generative module of our ImAD.

H Influence of Constrained Radii r 1 , r 2 for Detection Performance

In this section, we explore the influences of constrained radii r 1 , r 2 for detection performance. We change the latent dimension d = { 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 } and carry out related experiments. Detailed experimental settings and results are provided in Table 9 and Figure 6 and Figure 7, respectively.































missing_rate=0.2

missing_rate=0.5































missing_rate=0.2

missing_rate=0.5



























missing_rate=0.2

missing_rate=0.5



























missing_rate=0.2

missing_rate=0.5





























missing_rate=0.2

missing_rate=0.5





























missing_rate=0.2

missing_rate=0.5

As showed in Table 9, we change the dimension d of latent space and then get r = σ √ F -1 d ( p ) (See Proposition A.1) and set target distribution D z ∼ N ( 0 , 0 . 5 2 · I d ) , D ˜ z ∼ N ( 0 , I d ) and set p = 0 . 9 .

Figure 6 and Figure 7 shows the fluctuation of detection performance with different latent dimension d . It can be observed that our method is not very sensitive to changes in the radii r 1 and r 2 , but its performance degrades with a reduction in the latent dimension. This is reasonable since the smaller latent dimension results in more information loss.

I Ablation Study and Sensitivity Analysis of Hyperparameters

For hyper-parameters α, β, λ used in our experiments, we vary them in a large range to analyze the sensitivity of ImAD under MCAR. For hyper-parameter β , it cannot be set to 0 because the imputation module is an indispensable part in the presence of missing values. The average results are shown in Figure 8.































missing_rate=0.2

missing_rate=0.5































missing_rate=0.2

missing_rate=0.5





























missing_rate=0.2

missing_rate=0.5





























missing_rate=0.2

missing_rate=0.5



























missing_rate=0.2

missing_rate=0.5



























missing_rate=0.2

missing_rate=0.5































missing_rate=0.2

missing_rate=0.5































missing_rate=0.2

missing_rate=0.5













































































































































































J Detailed Experimental Implementations

J.1 Dataset Description

Adult 2 [Becker and Kohavi, 1996] is from the 1994 Census Income database with 14 variables including both categorical and continuous variables. The samples of income ≤ 50K are regarded as normal data, and the samples of income > 50K are regarded as abnormal data. Data preparation follows the previous work [Han et al., 2023].

KDD 3 [Lichman, 2013] is the KDDCUP99 10 percent dataset from the UCI repository and contains 121 variables including both categorical and continuous variables. The attack samples are regarded as normal data, and the non-attack samples are regarded as abnormal data.

Arrhythmia 4 [Rayana, 2016] is an ECG dataset. It was used to identify arrhythmic samples in five classes and contains 452 instances with 274 attributes.

Speech 5 [Rayana, 2016] consists of 3686 segments of English speech spoken with different accents and is represented by 400-dimensional so-called i-vectors which are widely used stateof-the-art features for speaker and language recognition.

Segerstolpe 6 [Segerstolpe et al., 2016] is a scRNA-seq dataset of human pancreas islets which includes six cell types: 'alpha', 'beta', 'delta', 'ductal', 'endothelial' and 'gamma'. In our experiments, 'alpha' is regarded as normal data and 'beta' is regarded as abnormal data.

Usoskin 7 [Usoskin et al., 2015] is a dataset employed for the analysis of sensory neuron cells, specifically originating from the mouse lumbar dorsal root ganglion. The dataset encompasses four distinct cell types: non-peptidergic nociceptor cells (NP), peptidergic nociceptor cells (PEP), neurofilament-containing cells (NF), and tyrosine hydroxylase-containing cells (TH). In our experiment, TH is regarded as normal data and PEP is abnormal data.

Botnet 8 [Meidan et al., 2018] is a public botnet datasets for the IoT. it was gathered from 9 commercial IoT devices authentically infected by Mirai and BASHLITE. There are 7,062,606 instances in the original datasets. In our experiments, we use 'Ecobee_Thermostat' subset of the original data, in which 'benign_traffic' is regarded as normal data and 'gafgyt_attacks' is regarded as abnormal data. The 'gafgyt_attacks' has five attack types and we randomly select 1,000 samples from each type as abnormal data of the test set.

Titanic 9 [Chen et al., 2023] is a classification dataset to detect the survival on the Titanic. In our experiments, we use nine features including 'gender, ticket, cabin, age, sibsp, parch, fare, embarked and pclass'. The instances that did not survive are considered normal samples and those that survived are considered abnormal(or unusual) samples.

MovieLens1M 10 Han et al. [2021] contains 1,000,209 anonymous rating of approximately 3,900 movies made by 6,040 MovieLens users who joined MovieLens in 2000. Due to the missing rate of the original dataset is near 95%, we remove some columns with quite high missing rate and obtain a new dataset with 82% missing rate. Since the age of all samples is divided into seven groups, we chose the middle five groups (18 < age < 56) as normal samples and the remaining as abnormal samples.

Bladder 11 is a cell transcriptome data from the model organism Mus musculus, in which contains 4 cell types (bladder cell, bladder urothelial cell, endothelial cell and leukocyte). We use the instance from 'bladder cell' as normal samples and those from ' leukocyte' as abnormal samples.

Seq2-Heart 12 [Schaum et al., 2018] is a single cell transcriptome data from the model organism Mus musculus, containing nearly 100,000 cells from 20 organs and tissues. There are 8 cell types in this data. We use the instances with 'fibroblast' type as normal samples and those with 'myofibroblast' type as abnormal samples.

J.2 Missing Mechanisms

In this work, we evaluate the detection performance of all the baselines under the three distinct missing mechanisms and we follow the previous work [Muzellec et al., 2020] to set the missing value generation mechanism.

A detailed explanation of our implementation is provided as follows.

MCAR : missing completely at random if the missingness is independent of the data. In our implementation, each entry is masked according to the realization of a Bernoulli random variable with parameter p = { 0 . 2 , 0 . 5 } .

MAR : missing at random if the missingness depends only on the observed values. In the MAR setting, for all experiments, a fixed subset of variables that cannot have missing values is sampled. Then, the entries from the remaining variables are masked according to a logistic model with random weights, which takes the non-missing variables as inputs. A bias term is fitted using line search to attain the desired proportion of missing values.

MNAR : missing not at random if the missingness depends on both the observed values and the unobserved values. In the MNAR setting, first, we sample a subset of variables whose values in the lower and upper p-th percentiles are masked according to a Bernoulli random variable, and the values in-between are left not missing.

J.3 Sampling in Target Distribution

In our experiments, we select two truncated Gaussian distribution N ( 0 , σ 2 I d ) with different σ as target distribution D , D and set σ = 0 . 5 , σ = 1 . 0 respectively. For target distribution D ∼

z ˜ z z N ( 0 , 0 . 5 2 · I d ) , according to the Proposition A.1, we set constrained radius r = 0 . 5 √ F -1 d ( p ) where d denotes the latent dimension and set p = 0 . 9 . Similarity, for target distribution D ˜ z ∼ N ( 0 , I d ) , we set r 1 = r and r 2 = √ F -1 d ( p ) and set p = 0 . 9 .

J.4 All Baselines

For the data imputation method used in our experiments, GAIN 13 , MissOT 14 , we use official code and the hyperparameters are fine-tuned as suggested in the original paper. For MissForest, we use missingpy 15 which is a library for missing data imputation in Python to implement the MissForest [Stekhoven and Bühlmann, 2012] algorithm. For anomaly detection method, Deep SVDD 16 [Rubin, 1976], NeutraL AD 17 [Qiu et al., 2021] and DPAD [Fu et al., 2024], we use official code and the hyperparameters are fine-tuned as suggested in the original paper. For Isolation Forest, we use scikit-learn 18 to implement the Isolation Forest [Liu et al., 2008] algorithm.

J.5 Hyper-parameter Settings

The hyperparameters used in our experiments are provided in Table 10.

K More Experimental Results

In this section, we conduct experiments on the Speech dataset with a missing rate mr ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 } . The related results are visualized in Figure 9, where the detection performance of 'impute-then-detect' methods does not degrade and some of them even improve with the increasing of missing rate from 0.1 to 0.8. Moreover, our proposed method outperforms all baselines in almost all cases.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8



25

30

35

40

45

50

55

60

65



0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8



35

40

45

50

55

60



MeanF --> IForest

MeanF --> DSVDD

MeanF --> NLAD

MeanF --> DPAD

MissF --> IForest

MissF --> DSVDD

MissF --> NLAD

MissF --> DPAD

GAIN --> IForest

GAIN --> DSVDD

GAIN --> NLAD

GAIN --> DPAD

ImAD (Ours)

In some real scenarios, it is possible that the missing rates of training and test sets are not equal. In this section, we conduct related experiments on the Speech dataset. In these experiments, we keep the missing rate mr = 0 . 5 on the training set and change the missing rate from 0.2 to 0.8 on the test set. We visualize the experimental results in Figure 10.

0.2

0.3

0.4

0.5

0.6

0.7

0.8



25

30

35

40

45

50

55

60



0.2

0.3

0.4

0.5

0.6

0.7

0.8



35

40

45

50

55

60



MeanF --> IForest

MeanF --> DSVDD

MeanF --> NLAD

MeanF --> DPAD

MissF --> IForest

MissF --> DSVDD

MissF --> NLAD

MissF --> DPAD

GAIN --> IForest

GAIN --> DSVDD

GAIN --> NLAD

GAIN --> DPAD

ImAD (Ours)

With such an experimental setup, the performance of the methods based on Mean-Filling and MissForest fluctuates significantly. Our proposed method outperforms all baselines in almost all cases in spite of also showing some degree of performance fluctuation.

The experimental results on the Segerstolpe and Usoskin with MCAR are provided in Table 11. The experimental results on the Botnet dataset with MCAR are provided in Table 12. In addition to the missing mechanism MCAR, we also compare ImAD with 'impute-then-detect' baselines under missing mechanism MAR and MNAR. Note that we did not employ GAIN [Yoon et al., 2018] under MAR and MNAR because GAIN was proposed under the assumption of MCAR. Instead, we utilize MissOT [Muzellec et al., 2020] as the imputation method under MAR and MNAR. For all baselines, the experimental results on the Adult dataset under MAR and MNAR are shown in Table 13 and Table 14, respectively, in which, our proposed method outperforms all two-stage baselines in all cases.

L Limitations and Future Work

Anomaly detection and data imputation are ubiquitous tasks across various data types. In this work, we primarily focus on incomplete tabular data. However, other data types, such as image and timeseries data, also need to be studied in similar scenarios. Therefore, we will conduct further studies on more data types in future work based on ImAD.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See the abstract part.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Section 3.4 and Appendix A,B,C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 4.1 and Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All the datasets used are publicly available and we provide the download URLs (see Appendix I) for each dataset. We provide experimental code in the supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not

including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 4.1 and Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Section 4.2 and 4.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All details are included in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See the Section 4.1 and Appendix I.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn anomaly detection, where training data consists only of normal instances, conventional missing value imputation approaches may cause imputation bias, meaning that imputations are inclined to make anomalous incomplete instances appear normal. This study addressed this issue by proposing an end-to-end training method that incorporates missing value imputation and anomaly detection into a unified optimization problem. The experimental results demonstrated that the proposed method can mitigate imputation bias, thereby outperforming conventional \"impute-then-detect\" methods in various anomaly detection benchmarks.\n\nSTRENGTHS:\n- Anomaly detection in the presence of missing values is a very important problem in practice, while few existing studies have addressed this issue.\n- This study addresses the issue properly using a novel approach.\n- The manuscript is overall well written, providing detailed justification for the approach.\n\nWEAKNESSES:\n- The implementation details are obscure in the manuscript.\n- Discussion on when the proposed method is more effective is needed.\n- Evaluation on realistic scenarios that motivated this study is needed.\n\nQUESTIONS:\n- How does the imputer network look? The network may not directly process missing values. How were the missing values handled before being fed into the network? This should be clarified.\n- In Table 8, the hyperparameter settings differ between datasets. It is important to explain how they were chosen. It is very inappropriate if the authors chose them based on test performance. Additionally, how is overfitting prevented?\n- I suppose the proposed method may work well when the distribution of each feature is significantly different between normal and anomalous data. It would be great if the authors investigated when the proposed method was more effective compared to \"impute-then-detect\" baselines.\n- In the real world, the mechanism of how missing values appear in data is usually unknown, and we cannot guess it from data alone; it can only be conjectured based on domain knowledge. If we don't know the actual missing mechanism of the data, how do we choose the missing mechanism M M when generating pseudo-abnormal samples? Is the effectiveness sensitive to the suitability of the missing mechanism we choose? For example, what will happen if the actual mechanism was NMAR and we set M to MCAR?\n- As motivated by cases of abnormal user detection in recommendation systems and novel or anomalous cell detection in bioinformatics, where the missing rates can be higher than 30% or even 80%, the effectiveness of the proposed method should be evaluated under such conditions. The benchmark datasets used in the experiments do not cover such conditions.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

comments: SUMMARY:
This study addresses the challenge of anomaly detection in the presence of missing data, which is common in various fields like recommendation systems and bioinformatics. Traditional methods struggle with missing data, leading to biased imputations and ineffective anomaly detection. The study proposes an integrated approach that combines data imputation with anomaly detection in a unified optimization framework. By generating pseudo-abnormal samples during training, the method mitigates imputation biases and enhances anomaly detection performance. The approach is supported by theoretical guarantees and outperforms baseline methods in experimental evaluations on diverse datasets.

STRENGTHS:
- The paper is well-written
- The proposed method is novel
- Extensive experiments are conducted to robustly support the claims

WEAKNESSES:
- There is no report on computational time.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ImAD, an end-to-end approach to anomaly detection in the presence of missing data. It addresses the imputation bias observed in the traditional impute-then-detect approaches, where the imputation model trained only on normal data tends to normalize incomplete abnormal samples. The proposed method generates pseudo-abnormal samples in the latent space and uses them in joint learning of the imputor, reconstructor, and projector.\n\nSTRENGTHS:\n- It proposes a novel approach that combines data imputation and anomaly detection in a single framework, which has not been explored much in previous studies. \n- The idea of generating pseudo-abnormal samples in the latent space and using them for joint learning of the imputor, projector, and reconstructor makes sense and shows strong empirical performance.\n\nWEAKNESSES:\n- Although the authors provide extensive details on the data and the experimental setup, some crucial information is still missing. For instance, it does not explain how the optimal hyperparameters for each method (both the proposed and others) are selected for the results shown in the tables and figures (whether by using a validation set, or referring to the performance on the test set, or using other methods), which can significantly affect the results and generalizability. Additionally, the composition of the training and test set split is not explained. \n- The results from using only two missing rate values (0.2 and 0.5) may not provide enough evidence about its robustness and effectiveness. In the previous study referred as a reference to this setting (Yoon et al., 2018), a broader range from 0.1 to 0.8 was explored. Therefore, certain claims regarding the imputation bias (e.g., the authors state that the detection performance of “impute-then-detect” methods does not decrease with the increasing of missing rate from 0.2 to 0.5 because of the imputation bias issue) are not fully supported empirically.\n\nQUESTIONS:\n- Are the missing rates used for training and test data the same? How would it affect the performance if different values are used? \n- Could you provide more information on how to select the optimal hyperparameters in the given experiments and also in general situations? \n- I am curious to see how the performance would change to different missing rates in comparison with other methods (for a wider range of values). \n- Is the imputor learned by this model only effective for the task of anomaly detection, or can it be applied to other tasks as well?  \n- What are the specific details of the architectures for MLPs used for each main component? How would different architectures affect the performance? \n- Is the method applicable to non-tabular data?\n\nLIMITATIONS:\nLimited experimental setting and ablation study.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper proposes a unified framework to find anomalies in data with missing attribute values. Instead of relying on impute-then-detect approach, which could lead to imputation bias, authors proposed a multi-objective learning framework in which the imputation and data modeling are done together. The core assumption is that there is a latent dimension in which the normal data lies within a sphere and anomalous data lies in the region outside this sphere, and sampling from this anomalous region will provide anomalous samples in the original space. This is done by learning two-way mappings between the latent space and original space. An imputation map is also learnt for the anomalous data. Details of a neural network based implementation is given. \n\nExperimental results on several benchmark data sets are given to showcase two capabilities: the proposed method can indeed identify anomalies when the data has missing attribute values better than other methods which do not impute, and the proposed method has better performance than impute-then-detect scheme.\n\nSTRENGTHS:\n- This is an interesting paper with an innovative approach. The idea of combining imputation and detection in a joint optimization framework is novel. \n\n- Authors have supported their assumptions with theoretical results.\n\n- The experimental evaluation is detailed and robust and largely supports the claims made in the paper.\n\nWEAKNESSES:\n- Experimental results are only marginally better than other solutions. I must admit that the method consistently outperforms the best existing approach so it might be better to use this approach.\n\nQUESTIONS:\n- How sensitive is the performance to the choice of $r_1$ and $r_2$? I feel that this choice could be very impactful.\n- I am not clear on the need for Sinkhorn distance. When will the samples not be pairwise?\n\nLIMITATIONS:\nLimitations are discussed, though more discussion on the sensitivity to parameter choices might be included. The paper does not have any immediate societal concerns.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-22"}