Title: UNSUPERVISED REPRESENTATION LEARNING BY PREDICTING RANDOM DISTANCES Anonymous

Abstract: Deep neural networks have gained tremendous success in a broad range of machine learning tasks due to its remarkable capability to learn semantic-rich features from high-dimensional data. However, they often require large-scale labelled data to successfully learn such features, which significantly hinders their adaption into unsupervised learning tasks, such as anomaly detection and clustering, and limits their applications into critical domains where obtaining massive labelled data is prohibitively expensive. To enable downstream unsupervised learning on those domains, in this work we propose to learn features without using any labelled data by training neural networks to predict data distances in a randomly projected space. Random mapping is a theoretical proven approach to obtain approximately preserved distances. To well predict these random distances, the representation learner is optimised to learn genuine class structures that are implicitly embedded in the randomly projected space. Experimental results on 19 real-world datasets show our learned representations substantially outperform state-of-the-art competing methods in both anomaly detection and clustering tasks.

1. INTRODUCTION:

Unsupervised representation learning aims at automatically extracting expressive feature representations from data without any manually labelled data. Due to the remarkable capability to learn semantic-rich features, deep neural networks have been becoming one widely-used technique to empower a broad range of machine learning tasks. One main issue with these deep learning techniques is that a massive amount of labelled data is typically required to successfully learn these expressive features. As a result, their transformation power is largely reduced for tasks that are unsupervised in nature, such as anomaly detection and clustering. This is also true to critical domains, such as healthcare and fintech, where collecting massive labelled data is prohibitively expensive and/or is impossible to scale. To bridge this gap, in this work we explore fully unsupervised representation learning techniques to enable downstream unsupervised learning methods on those critical domains.

In recent years, many unsupervised representation learning methods (Mikolov et al., 2013a; Le & Mikolov, 2014; Misra et al., 2016; Lee et al., 2017; Gidaris et al., 2018) have been introduced, of which most are self-supervised approaches that formulate the problem as an annotation free pretext task. These methods explore easily accessible information, such as temporal or spatial neighbourhood, to design a surrogate supervisory signal to empower the feature learning. These methods have achieved significantly improved feature representations of text/image/video data, but they are often inapplicable to tabular data since it does not contain the required temporal or spatial supervisory information. We therefore focus on unsupervised representation learning of high-dimensional tabular data. Although many traditional approaches, such as random projection (Li et al., 2006) , principal component analysis (PCA) (Rahmani & Atia, 2017) , manifold learning (Donoho & Grimes, 2003; Hinton & Roweis, 2003) and autoencoder (Vincent et al., 2010) , are readily available for handling those data, many of them (Donoho & Grimes, 2003; Hinton & Roweis, 2003; Rahmani & Atia, 2017) are often too computationally costly to scale up to large or high-dimensional data. Approaches like random projection and autoencoder are very efficient but they often fail to capture complex class structures due to its underlying data assumption or weak supervisory signal.

In this paper, we introduce a Random Distance Prediction (RDP) model which trains neural networks to predict data distances in a randomly projected space. When the distance information captures in-trinsic class structure in the data, the representation learner is optimised to learn the class structure to minimise the prediction error. Since distances are concentrated and become meaningless in high dimensional spaces (Beyer et al., 1999) , we seek to obtain distances preserved in a projected space to be the supervisory signal. Random mapping is a highly efficient yet theoretical proven approach to obtain such approximately preserved distances. Therefore, we leverage the distances in the randomly projected space to learn the desired features. Intuitively, random mapping preserves rich local proximity information but may also keep misleading proximity when its underlying data distribution assumption is inexact; by minimising the random distance prediction error, RDP essentially leverages the preserved data proximity and the power of neural networks to learn globally consistent proximity and rectify the inconsistent proximity information, resulting in a substantially better representation space than the original space. We show this simple random distance prediction enables us to achieve expressive representations with no manually labelled data. In addition, some task-dependent auxiliary losses can be optionally added as a complementary supervisory source to the random distance prediction, so as to learn the feature representations that are more tailored for a specific downstream task. In summary, this paper makes the following three main contributions.

• We propose a random distance prediction formulation, which is very simple yet offers a highly effective supervisory signal for learning expressive feature representations that optimise the distance preserving in random projection. The learned features are sufficiently generic and work well in enabling different downstream learning tasks. • Our formulation is flexible to incorporate task-dependent auxiliary losses that are complementary to random distance prediction to further enhance the learned features, i.e., features that are specifically optimised for a downstream task while at the same time preserving the generic proximity as much as possible. • As a result, we show that our instantiated model termed RDP enables substantially better performance than state-of-the-art competing methods in two key unsupervised tasks, anomaly detection and clustering, on 19 real-world high-dimensional tabular datasets.

2. RANDOM DISTANCE PREDICTION MODEL:



2.1. THE PROPOSED FORMULATION AND THE INSTANTIATED MODEL:

We propose to learn representations by training neural networks to predict distances in a randomly projected space without manually labelled data. The key intuition is that, given some distance information that faithfully encapsulates the underlying class structure in the data, the representation learner is forced to learn the class structure in order to yield distances that are as close as the given distances. Our proposed framework is illustrated in Figure 1 . Specifically, given data points x i , x j ∈ R D , we first feed them into a weight-shared Siamese-style neural network φ(x; Θ). φ : R D → R M is a representation learner with the parameters Θ to map the data onto a M -dimensional new space.

Then we formulate the subsequent step as a distance prediction task and define a loss function as:

EQUATION

where η is an existing projection method and l is a function of the difference between its two inputs.

Here one key ingredient is how to obtain trustworthy distances via η. Also, to efficiently optimise the model, the distance derivation needs to be computationally efficient. In this work, we use the inner products in a randomly projected space as the source of distance/similarity since it is very efficient and there is strong theoretical support of its capacity in preserving the genuine distance information. Thus, our instantiated model RDP specifies L rdp (x i , x j ) as follows1 :

EQUATION

)

where φ is implemented by multilayer perceptron for dealing with tabular data and η : R D → R K is an off-the-shelf random data mapping function (see Sections 3.1 and 3.2 for detail). Despite its simplicity, this loss offers a powerful supervisory signal to learn semantic-rich feature representations that substantially optimise the underlying distance preserving in η (see Section 3.3 for detail). The proposed random distance prediction (RDP) framework. Specifically, a weight-shared two-branch neural network φ first projects x i and x j onto a new space, in which we aim to minimise the random distance prediction loss L rdp , i.e., the difference between the learned distance φ(x i ; Θ), φ(x j ; Θ) and a predefined distance η(x i ), η(x j ) (η denotes an existing random mapping). L aux is an auxiliary loss that is optionally applied to one network branch to learn complementary information w.r.t. L rdp . The lower right figure presents a 2-D t-SNE (Hinton & Roweis, 2003) visualisation of the features learned by RDP on a small toy dataset optdigits with 10 classes.

2.2. FLEXIBILITY TO INCORPORATE TASK-DEPENDENT COMPLEMENTARY AUXILIARY LOSS:

Minimising L rdp learns to preserve pairwise distances that are critical to different learning tasks. Moreover, our formulation is flexible to incorporate a task-dependent auxiliary loss L aux , such as reconstruction loss (Hinton & Salakhutdinov, 2006) for clustering or novelty loss (Burda et al., 2019) for anomaly detection, to complement the proximity information and enhance the feature learning.

For clustering, an auxiliary reconstruction loss is defined as:

EQUATION

)

where φ is an encoder and φ : R M → R D is a decoder. This loss may be optionally added into RDP to better capture global feature representations.

Similarly, in anomaly detection a novelty loss may be optionally added, which is defined as:

EQUATION

)

By using a fixed η, minimising L ad aux helps learn the frequency of underlying patterns in the data (Burda et al., 2019) , which is an important complementary supervisory source for the sake of anomaly detection. As a result, anomalies or novel points are expected to have substantially larger (φ(x; Θ ) -η(x))

2 than normal points, so this value can be directly leveraged to detect anomalies.

Note since L ad aux involves a mean squared error between two vectors, the dimension of the projected space resulted by φ and η is required to be equal in this case. Therefore, when this loss is added into RDP, the M in φ and K in η need to be the same. We do not have this constraint in other cases.

3. THEORETICAL ANALYSIS OF RDP:

This section shows the proximity information can be well approximated using inner products in two types of random projection spaces. This is a key theoretical foundation to RDP. Also, to accurately predict these distances, RDP is forced to learn the genuine class structure in the data.

3.1. WHEN LINEAR PROJECTION IS USED:

Random projection is a simple yet very effective linear feature mapping technique which has proven the capability of distance preservation. Let X ⊂ R N ×D be a set of N data points, random projection uses a random matrix A ⊂ R K×D to project the data onto a lower K-dimensional space by X = AX . The Johnson-Lindenstrauss lemma (Johnson & Lindenstrauss, 1984) guarantees the data points can be mapped to a randomly selected space of suitably lower dimension with the distances between the points are approximately preserved. More specifically, let ∈ (0, 1 2 ) and K = 20 log n 2 . There exists a linear mapping f : R D → R K such that for all x i , x j ∈ X :

(1 -

EQUATION

)

Furthermore, assume the entries of the matrix A are sampled independently from a Gaussian distribution N (0, 1). Then, the norm of x ∈ R D can be preserved as:

EQUATION

)

Under such random projections, the norm preservation helps well preserve the inner products:

EQUATION

) where x is a normalised x such that ||x|| ≤ 1.

The proofs of Eqns. ( 5), ( 6) and ( 7) can be found in (Vempala, 1998) .

Eqn. (7) states that the inner products in the randomly projected space can largely preserve the inner products in the original space, particularly when the projected dimension K is large.

3.2. WHEN NON-LINEAR PROJECTION IS USED:

Here we show that some non-linear random mapping methods are approximate to kernel functions which are a well-established approach to obtain reliable distance/similarity information. The key to this approach is the kernel function k : X ×X → R, which is defined as k(x i , x j ) = ψ(x i ), ψ(x j ) , where ψ is a feature mapping function but needs not to be explicitly defined and •, • denotes a suitable inner product. A non-linear kernel function such as polynomial or radial basis function (RBF) kernel is typically used to project linear-inseparable data onto a linear-separable space.

The relation between non-linear random mapping and kernel methods is justified in (Rahimi & Recht, 2008) , which shows that an explicit randomised mapping function g : R D → R K can be defined to project the data points onto a low-dimensional Euclidean inner product space such that the inner products in the projected space approximate the kernel evaluation:

EQUATION

Let A be the mapping matrix. Then to achieve the above approximation, A is required to be drawn from Fourier transform and shift-invariant functions such as cosine function are finally applied to Ax to yield a real-valued output. By transforming the two data points x i and x j in this manner, their inner product g(x i ) • g(x j ) is an unbiased estimator of k(x i , x j ).

3.3. LEARNING CLASS STRUCTURE BY RANDOM DISTANCE PREDICTION:

Our model using only the random distances as the supervisory signal can be formulated as:

EQUATION

)

where

y ij = η(x i ) • η(x j ).

Let Y η ∈ R N ×N be the distance/similarity matrix of the N data points resulted by η. Then to minimise the prediction error in Eqn. (9), φ is optimised to learn the underlying class structure embedded in Y. As shown in the properties in Eqns. ( 7) and (8), Y η can effectively preserve local proximity information when η is set to be either the random projectionbased f function or the kernel method-based g function. However, those proven η is often built upon some underlying data distribution assumption, e.g., Gaussian distribution in random projection or Gaussian RBF kernel, so the η-projected features can preserve misleading proximity when the distribution assumption is inexact. In this case, Y η is equivalent to the imperfect ground truth with partial noise. Then optimisation with Eqn. ( 9) is to leverage the power of neural networks to learn consistent local proximity information and rectify inconsistent proximity, resulting in a significantly optimised distance preserving space. The resulting space conveys substantially richer semantics than the η projected space when Y η contains sufficient genuine supervision information.

4. EXPERIMENTS:

This section evaluates the learned representations through two typical unsupervised tasks: anomaly detection and clustering. Some preliminary results of classification can be found in Appendix H.

4.1. PERFORMANCE EVALUATION IN ANOMALY DETECTION:



4.1.1. EXPERIMENTAL SETTINGS:

Our RDP model is compared with five state-of-the-art methods, including iForest (Liu et al., 2008) , autoencoder (AE) (Hinton & Salakhutdinov, 2006) , REPEN (Pang et al., 2018) , DAGMM (Zong et al., 2018) and RND (Burda et al., 2019) . iForest and AE are two of the most popular baselines.

The other three methods learn representations specifically for anomaly detection.

As shown in Table 1 , the comparison is performed on 14 publicly available datasets of various domains, including network intrusion, credit card fraud detection, disease detection and bank campaigning. Many of the datasets contain real anomalies, including DDoS, Donors, Backdoor, Creditcard, Lung, Probe and U2R. Following (Liu et al., 2008; Pang et al., 2018; Zong et al., 2018) , the rare class(es) is treated as anomalies in the other datasets to create semantically real anomalies. The Area Under Receiver Operating Characteristic Curve (AUC-ROC) and the Area Under Precision-Recall Curve (AUC-PR) are used as our performance metrics. Larger AUC-ROC/AUC-PR indicates better performance. The reported performance is averaged over 10 independent runs. Our RDP model uses the optional novelty loss for anomaly detection task by default. Similar to RND, given a data point x, its anomaly score in RDP is defined as the mean squared error between the two projections resulted by φ(x; Θ ) and η(x). Also, a boosting process is used to filter out 5% likely anomalies per iteration to iteratively improve the modelling of RDP. This is because the modelling is otherwise largely biased when anomalies are presented. In the ablation study in Section 4.1.3, we will show the contribution of all these components.

4.1.2. COMPARISON TO THE STATE-OF-THE-ART COMPETING METHODS:

The AUC-ROC and AUC-PR results are respectively shown in Tables 1 and 2 . RDP outperforms all the five competing methods in both of AUC-ROC and AUC-PR in at least 12 out of 14 datasets. This improvement is statistically significant at the 95% confidence level according to the two-tailed sign test (Demšar, 2006) . Remarkably, RDP obtains more than 10% AUC-ROC/AUC-PR improvement over the best competing method on six datasets, including Donors, Ad, Bank, Celeba, Lung and U2R. RDP can be thought as a high-level synthesis of REPEN and RND, because REPEN leverages a pairwise distance-based ranking loss to learn representations for anomaly detection while RND is built using L ad aux . In nearly all the datasets, RDP well leverages both L rdp and L ad aux to achieve significant improvement over both REPEN and RND. In very limited cases, such as on datasets Backdoor and Census where RND performs very well while REPEN performs less effectively, RDP is slightly downgraded due to the use of L rdp . In the opposite case, such as Probe, on which REPEN performs much better than RND, the use of L ad aux may drag down the performance of RDP a bit. 

4.1.3. ABLATION STUDY:

This section examines the contribution of L rdp , L ad aux and the boosting process to the performance of RDP. The experimental results in AUC-ROC are given in Table 3 , where RDP\X means the RDP variant that removes the 'X' module from RDP. In the last two columns, Org SS indicates that we directly use the distance information calculated in the original space as the supervisory signal, while SRP SS indicates that we use SRP to obtain the distances as the supervisory signal. It is clear that the full RDP model is the best performer. Using the L rdp loss only, i.e., RDP\L ad aux , can achieve performance substantially better than, or comparably well to, the five competing methods in Table 1 . This is mainly because the L rdp loss alone can effectively force our representation learner to learn the underlying class structure on most datasets so as to minimise its prediction error. The use of L ad aux and boosting process well complement the L rdp loss on the other datasets. In terms of supervisory source, RDP and SRP SS perform substantially better than Org SS on most datasets. This is because the distances in both the non-linear random projection in RDP and the linear projection in SRP SS well preserve the distance information, enabling RDP to effectively learn much more faithful class structure than that working on the original space. (Li et al., 2006) in random projection, autoencoder (AE) (Hinton & Salakhutdinov, 2006) The clustering performance in the original feature space, denoted as Org, is used as a baseline. As shown in Table 4 , five high-dimensional real-world datasets are used. Some of the datasets are image/text data. Since here we focus on the performance on tabular data, they are converted into tabular data using simple methods, i.e., by treating each pixel as a feature unit for image data or using bag-of-words representation for text data2 . The reported NMI score and F-score are averaged over 30 times to address the randomisation issue in K-means clustering. In this section RDP adds the reconstruction loss L clu aux by default, but RDP also works very well without the use of L clu aux .

4.2.2. COMPARISON TO THE-STATE-OF-THE-ART COMPETING METHODS:

Table 4 shows the NMI and F-score performance of K-means clustering. Our method RDP enables K-means to achieve the best performance on three datasets and ranks second in the other two datasets. RDP-enabled clustering performs substantially and consistently better than that based on AE in terms of both NMI and F-score. This demonstrates that the random distance loss enables RDP to effectively capture some class structure in the data which cannot be captured by using the reconstruction loss. RDP also consistently outperforms the random projection method, SRP, and the robust PCA method, COP. It is interesting that K-means clustering performs best in the original space on Sector. This may be due to that this data contains many relevant features, resulting in no obvious curse of dimensionality issue. Olivetti may contain complex manifolds which require extensive neighbourhood information to find them, so only HLLE can achieve this goal in such cases. Nevertheless, RDP performs much more stably than HLLE across the five datasets. 

Similar to anomaly detection, this section examines the contribution of the two loss functions L rdp and L clu aux to the performance of RDP, as well as the impact of different supervisory sources on the performance. The F-score results of this experiment are shown in Table 5 , in which the notations have exactly the same meaning as in Table 3 . The full RDP model that uses both L rdp and L clu aux performs more favourably than its two variants, RDP\L rdp and RDP\L clu aux , but it is clear that using L rdp only performs very comparably to the full RDP. However, using L clu aux only may result in large performance drops in some datasets, such as R8, 20news and Olivetti. This indicates L rdp is a more important loss function to the overall performance of the full RDP model. In terms of supervisory source, distances obtained by the non-linear random projection in RDP are much more effective than the two other sources on some datasets such as Olivetti and RCV1. Three different supervisory sources are very comparable on the other three datasets.

5. RELATED WORK:

Self-supervised Learning. Self-supervised learning has been recently emerging as one of the most popular and effective approaches for representation learning. Many of the self-supervised methods learn high-level representations by predicting some sort of 'context' information, such as spatial or temporal neighbourhood information. For example, the popular distributed representation learning techniques in NLP, such as CBOW/skip-gram (Mikolov et al., 2013a) and phrase/sentence embeddings in (Mikolov et al., 2013b; Le & Mikolov, 2014; Hill et al., 2016) , learn the representations by predicting the text pieces (e.g., words/phrases/sentences) using its surrounding pieces as the context. In image processing, the pretext task can be the prediction of a patch of missing pixels (Pathak et al., 2016; Zhang et al., 2017) or the relative position of two patches (Doersch et al., 2015) . Also, a number of studies (Goroshin et al., 2015; Misra et al., 2016; Lee et al., 2017; Oord et al., 2018) explore temporal contexts to learn representations from video data, e.g., by learning the temporal order of sequential frames. Some other methods (Agrawal et al., 2015; Zhou et al., 2017; Gidaris et al., 2018) are built upon a discriminative framework which aims at discriminating the images before and after some transformation, e.g., ego motion in video data (Agrawal et al., 2015; Zhou et al., 2017) and rotation of images (Gidaris et al., 2018) . There have also been popular to use generative adversarial networks (GANs) to learn features (Radford et al., 2015; Chen et al., 2016) . The above methods have demonstrated powerful capability to learn semantic representations. However, most of them use the supervisory signals available in image/video data only, which limits their application into other types of data, such as traditional tabular data. Although our method may also work on image/video data, we focus on handling high-dimensional tabular data to bridge this gap.

Other Approaches. There have been several well-established unsupervised representation learning approaches for handling tabular data, such as random projection (Arriaga & Vempala, 1999; Bingham & Mannila, 2001; Li et al., 2006) , PCA (Wold et al., 1987; Schölkopf et al., 1997; Rahmani & Atia, 2017) , manifold learning (Roweis & Saul, 2000; Donoho & Grimes, 2003; Hinton & Roweis, 2003; McInnes et al., 2018) and autoencoder (Hinton & Salakhutdinov, 2006; Vincent et al., 2010) . One notorious issue of PCA or manifold learning approaches is their prohibitive computational cost in dealing with large-scale high-dimensional data due to the costly neighbourhood search and/or eigen decomposition. Random projection is a computationally efficient approach, supported by proven distance preservation theories such as the Johnson-Lindenstrauss lemma (Johnson & Lindenstrauss, 1984) . We show that the preserved distances by random projection can be harvested to effectively supervise the representation learning. Autoencoder networks are another widely-used efficient feature learning approach which learns low-dimensional representations by minimising reconstruction errors. One main issue with autoencoders is that they focus on preserving global information only, which may result in loss of local structure information. Some representation learning methods are specifically designed for anomaly detection (Pang et al., 2018; Zong et al., 2018; Burda et al., 2019) . By contrast, we aim at generic representations learning while being flexible to incorporate optionally task-dependent losses to learn task-specific semantic-rich representations.

6. CONCLUSION:

We introduce a novel Random Distance Prediction (RDP) model which learns features in a fully unsupervised fashion by predicting data distances in a randomly projected space. The key insight is that random mapping is a theoretical proven approach to obtain approximately preserved distances, and to well predict these random distances, the representation learner is optimised to learn consistent preserved proximity information while at the same time rectifying inconsistent proximity, resulting in representations with optimised distance preserving. Our idea is justified by thorough experiments in two unsupervised tasks, anomaly detection and clustering, which show RDP-enabled anomaly detectors and clustering substantially outperform their counterparts on 19 real-world datasets. We plan to extend RDP to other types of data to broaden its application scenarios. 

A IMPLEMENTATION DETAILS:

RDP-enabled Anomaly Detection. The RDP consists of one fully connected layer with 50 hidden units, followed by a leaky-ReLU layer. It is trained using Stochastic Gradient Descent (SGD) as its optimiser for 200 epochs, with 192 samples per batch. The learning rate is fixed to 0.1. We repeated the boosting process 30 times to obtain statistically stable results. In order to have fair comparisons, we also adapt the competing methods AE, REPEN, DAGMM and RND into ensemble methods and perform the experiments using an ensemble size of 30.

RDP-enabled Clustering. RDP uses a similar network architecture and optimisation settings as the one used in anomaly detection, i.e., the network consists of one fully connected layer, followed by a leaky-ReLU layer, which is optimised by SGD with 192 samples per batch and 0.1 learning rate. Compared to anomaly detection, more semantic information is required for clustering algorithms to work well, so the network consists of 1,024 hidden units and is trained for 1,000 epochs. Clustering is a significant yet common analysis method, which aims at grouping samples close to each other into the same clusters and separating far away data points into different clusters. Compared to anomaly detection that often requires pattern frequency information, clustering has a higher requirement of the representation expressiveness. Therefore, if the representative ability of a model is strong enough, it should also be able to learn representations that enable clustering to work well on the projected space.

Note that the representation dimension M in the φ function and the projection dimension K in the η function are set to be the same to alleviate parameter tuning. This means that M = K = 50 is used in anomaly detection and M = K = 1024 is used in clustering. We have also tried deeper network structures, but they worked less effectively than the shallow networks in both anomaly detection and clustering. This may be because the supervisory signal is not strong enough to effectively learn deeper representations. We show in Appendix E that RDP performs stably w.r.t. a range of representation dimensions in both anomaly detection and clustering tasks.

The runtime of RDP at the testing stage is provided in Appendix F with that of the competing methods as baselines. For both anomaly detection and clustering tasks, RDP achieves very comparable time complexity to the most efficient competing methods (see Tables 10 and 11 in Appendix F for detail).

B DATASETS:

The statistics and the accessible links of the datasets used in the anomaly detection and clustering tasks are respectively presented in Tables 6 and 7 . DDoS is a dataset containing DDoS attacks and normal network flows. Donors is from KDD Cup 2014, which is used for detecting a very small number of outstanding donors projects. Backdoor contains backdoor network attacks derived from the UNSW-NB15 dataset. Creditcard is a credit card fraud detection dataset. Lung contains data records of lung cancer patients and normal patients. Probe and U2R are derived from KDD Cup 99, in which probing and user-to-root attacks are respectively used as anomalies against the normal network flows. The above datasets contain real anomalies. Following (Liu et al., 2008; Pang et al., 2018; Zong et al., 2018) , the other anomaly detection datasets are transformed from classification datasets by using the rare class(es) as the anomaly class, which generates semantically real anomalies.

R8, 20news, Sector and RCV1 are widely used text classification benchmark datasets. Olivetti is a widely-used face recognition dataset. The experimental results of AUC-PR performance of RDP and its variants in the anomaly detection task are shown in Table 8 . Similar to the results shown in Table 3 , using the L rdp loss only, our proposed RDP model can achieve substantially better performance over its counterparts. By removing the L rdp loss, the performance of RDP drops significantly in 11 out of 14 datasets. This demonstrates that the L rdp loss is heavily harvested by our RDP model to learn high-quality representations from random distances. Removing L ad aux from RDP also results in substantial loss of AUC-PR in many datasets. This indicates both the random distance prediction loss L rdp and the task-dependent loss L ad aux are critical to RDP. The boosting process is also important, but is not as critical as the two losses. Consistent with the observations derived from Table 3 , distances calculated in non-linear and linear random mapping spaces are more effective supervisory sources than that in the original space.

D NMI PERFORMANCE OF ABLATION STUDY IN CLUSTERING:

Table 9 shows the NMI performance of RDP and its variants in the clustering task. It is clear that our RDP model with the L rdp loss is able to achieve NMI performance that is comparably well to the full RDP model, which is consistent to the observations in Table 5 . Without using the L rdp loss, the performance of the RDP model has some large drops on nearly all the datasets. This reinforces the crucial importance of L rdp to RDP, which also justifies that using L rdp alone RDP can learn expressive representations. Similar to the results in Table 5 , RDP is generally more reliable supervisory sources than Org SS and SRP SS in this set of results.

E SENSITIVITY W.R.T. THE DIMENSIONALITY OF REPRESENTATION SPACE:

This section presents the performance of RDP using different representation dimensions in its feature learning layer. The sensitivity test is performed for both anomaly detection and clustering tasks. This demonstrates the general stability of our RDP method on different application domains. On the other hand, the flat trends also indicate that, as an unsupervised learning source, the random distance cannot provide sufficient supervision information to learn richer and more complex representations in a higher-dimensional space. This also explains the performance on quite a few datasets where the performance of RDP decreases when increasing the representation dimension. In general, the representation dimension 50 is recommended for RDP to achieve effective anomaly detection on datasets from different domains.

E.2 SENSITIVITY TEST IN CLUSTERING:

Figure 4 presents the NMI and F-score performance of RDP-enabled K-means clustering using different representation dimensions on all the five datasets in the clustering task. Similar to the sensitivity test results in the anomaly detection task, on all the five datasets, K-means clustering performs stably in the representation space resulted by RDP with different representation dimensions. The clustering performance may drop a bit when the representation dimension is relatively low, e.g., 512. Increasing the representation to 1,280 may help RDP gain better representation power in some datasets but is not a consistently better choice. Thus, the representation dimension 1,024 is generally recommended for clustering. Recall that the required representation dimension in clustering is normally significantly higher than that in anomaly detection, because clustering generally requires significantly more information to perform well than anomaly detection.

F COMPUTATIONAL EFFICIENCY:

The runtime of RDP is compared with its competing methods in both anomaly detection and clustering tasks. because RDP has a base runtime of its boosting process. Therefore, the runtime of RDP seems to be almost constant across the datasets. This is a very desired property for handling high-dimensional and large-scale datasets. Since RDP relies on distance information as its supervisory signal, one interesting question is that, can RDP still work when the presented data is raw data in a non-Euclidean space, such as raw text and image data? One simple and straightforward way to enable RDP to handle those raw data is, as what we did on the text and image data used in the evaluation of clustering, to first convert the raw texts/images into feature vectors using commonly-used methods, e.g., TF-IDF (Aizawa, 2003) for text data and treating each pixel as a feature unit for image data, and then perform RDP on these vector spaces. A further question is that, do we need RDP in handling those data since there are now a large number of advanced representation learning methods that are specifically designed for raw text/image datasets? Or, how is the performance of RDP compared to those advanced representation learning methods for raw text/image datasets? This section provides some preliminary results in the clustering task for answering these questions.

G.1 ON RAW TEXT DATA On the raw text datasets R8 and 20news, we first compare RDP with the advanced document representation method Doc2Vec3 as in (Le & Mikolov, 2014) . Recall that, for RDP, we first use the bag-of-words model information (e.g., TF-IDF) to simply convert documents into high-dimensional feature vectors and then perform RDP using the feature vectors. Doc2Vec leverages the idea of distributed representations to directly learn representations of documents. We further derive a variant of RDP, namely Doc2Vec+RDP, which performs RDP on the Doc2Vec projected representation space rather than the bag-of-words vector space. All RDP, Doc2Vec and Doc2Vec+RDP project data onto a 1,024-dimensional space for the subsequent learning tasks. Note that, for the method Doc2Vec+RDP, to better examine the capability of RDP in exploiting the Doc2Vec projected space, we first use Doc2Vec project raw text data onto a higherdimensional space (5,120 dimensions for R8 and 10,240 dimensions for 20news), and RDP further learns a 1,024-dimensional space from this higher-dimensional space.

The comparison results are shown in Table 12 . Two interesting observations can be seen. First, RDP can significantly outperform Doc2Vec on R8 or performs comparably well on 20news. This may be due to the fact that the local proximity information learned in RDP is critical to clustering; although the word prediction approach in Doc2Vec helps learn semantic-rich representations for words/sentences/paragraphs, the pairwise document distances may be less effective than RDP since Doc2Vec is not like RDP that is designed to optimise this proximity information. Second, Doc2Vec+RDP can achieve substantially better performance than Doc2Vec, especially on the dataset 20news where Doc2Vec+RDP achieves a NMI score of 0.198 while that of Doc2Vec is only 0.084. This may be because, as discussed in Section 3.3, RDP is equivalent to learn an optimised feature space out of its input space (Doc2Vec projected feature space in this case) using imperfect supervision information. When there is sufficient accurate supervision information, RDP can learn a substantially better feature space than its input space. This is also consistent with the results in Table 4, in which clustering based on the RDP projected space also performs substantially better than that working in the original space 'Org'. The results are shown in Table 14 . Similar to the performance in clustering and anomaly detection, our model using only the random distance prediction loss L rdp , i.e., RDP\L clu aux , performs very favourably and stably on all the five datasets. The incorporation of \L clu aux into the model, i.e., RDP, helps gain some extra performance improvement on datasets like 20news, but it may also slightly downgrade the performance on other datasets. An extra hyperparameter may be added to control the importance of these two losses. 

:

Since we operate on real-valued vector space, the inner product is implemented by the dot product. The dot product is used hereafter to simplify the notation.

RDP can also build upon advanced representation learning methods for the data transformation, for which some interesting preliminary results are presented in Appendix G.

We use the implementation of Doc2Vec in a popular text mining python package gensim available at https://radimrehurek.com/gensim/index.html

The released code of RotNet is available at https://github.com/gidariss/FeatureLearningRotNet.

Figures and Tables:

Figure1: The proposed random distance prediction (RDP) framework. Specifically, a weight-shared two-branch neural network φ first projects x i and x j onto a new space, in which we aim to minimise the random distance prediction loss L rdp , i.e., the difference between the learned distance φ(x i ; Θ), φ(x j ; Θ) and a predefined distance η(x i ), η(x j ) (η denotes an existing random mapping). L aux is an auxiliary loss that is optionally applied to one network branch to learn complementary information w.r.t. L rdp . The lower right figure presents a 2-D t-SNE(Hinton & Roweis, 2003) visualisation of the features learned by RDP on a small toy dataset optdigits with 10 classes.

Figure 4: NMI and F-score performance of RDP-enabled K-means using different representation dimensions on all the five datasets used in clustering.



Table content: <table><tr><td/><td colspan="2">Data Characteristics</td><td/><td/><td colspan="4">Our Method RDP and Its Five Competing Methods</td></tr><tr><td>Data</td><td>N</td><td colspan="2">D Anomaly (%)</td><td>iForest</td><td>AE</td><td>REPEN</td><td>DAGMM</td><td>RND</td><td>RDP</td></tr><tr><td>DDoS</td><td colspan="2">464,976 66</td><td>3.75%</td><td/><td/><td/><td/><td/></tr></table>

AUC-PR (mean±std) performance of RDP and its five competing methods on 14 datasets. ± 0.020 0.248 ± 0.001 0.300 ± 0.012 0.038 ± 0.000 0.110 ± 0.015 0.301 ± 0.028 Donors 0.124 ± 0.006 0.138 ± 0.007 0.120 ± 0.032 0.070 ± 0.024 0.201 ± 0.033 0.432 ± 0.061 Backdoor 0.045 ± 0.007 0.065 ± 0.004 0.129 ± 0.001 0.034 ± 0.023 0.433 ± 0.015 0.305 ± 0.008

Table content: <table><tr><td>Data</td><td>iForest</td><td>AE</td><td>REPEN</td><td>DAGMM</td><td>RND</td><td>RDP</td></tr><tr><td colspan="7">DDoS 0.141 Ad 0.363 ± 0.061 0.479 ± 0.000 0.600 ± 0.002 0.140 ± 0.000 0.473 ± 0.009 0.726 ± 0.007</td></tr><tr><td>Apascal</td><td colspan="6">0.015 ± 0.002 0.023 ± 0.001 0.041 ± 0.001 0.023 ± 0.009 0.021 ± 0.005 0.042 ± 0.003</td></tr><tr><td>Bank</td><td colspan="6">0.293 ± 0.023 0.264 ± 0.001 0.276 ± 0.001 0.150 ± 0.020 0.258 ± 0.006 0.364 ± 0.013</td></tr><tr><td>Celeba</td><td colspan="6">0.060 ± 0.006 0.082 ± 0.001 0.081 ± 0.001 0.037 ± 0.017 0.068 ± 0.010 0.104 ± 0.006</td></tr><tr><td>Census</td><td colspan="6">0.071 ± 0.004 0.072 ± 0.000 0.064 ± 0.005 0.061 ± 0.001 0.081 ± 0.001 0.086 ± 0.001</td></tr><tr><td colspan="7">Creditcard 0.145 ± 0.031 0.382 ± 0.004 0.359 ± 0.014 0.010 ± 0.012 0.290 ± 0.012 0.363 ± 0.011</td></tr><tr><td>Lung</td><td colspan="6">0.379 ± 0.092 0.565 ± 0.022 0.429 ± 0.005 0.042 ± 0.003 0.381 ± 0.104 0.705 ± 0.028</td></tr><tr><td>Probe</td><td colspan="6">0.923 ± 0.011 0.964 ± 0.002 0.964 ± 0.000 0.409 ± 0.153 0.609 ± 0.014 0.955 ± 0.002</td></tr><tr><td>R8</td><td colspan="6">0.076 ± 0.018 0.097 ± 0.006 0.083 ± 0.000 0.019 ± 0.011 0.134 ± 0.031 0.146 ± 0.017</td></tr><tr><td>Secom</td><td colspan="6">0.106 ± 0.007 0.093 ± 0.000 0.091 ± 0.001 0.066 ± 0.002 0.086 ± 0.002 0.096 ± 0.001</td></tr><tr><td>U2R</td><td colspan="6">0.180 ± 0.018 0.230 ± 0.004 0.116 ± 0.007 0.025 ± 0.019 0.217 ± 0.011 0.261 ± 0.005</td></tr></table>

AUC-ROC results of anomaly detection (see Appendix C for similar AUC-PR results).

Table content: <table><tr><td/><td/><td colspan="2">Decomposition</td><td/><td colspan="2">Supervision Signal</td></tr><tr><td>Data</td><td>RDP</td><td>RDP\L rdp</td><td>RDP\L ad aux</td><td>RDP\Boosting</td><td>Org SS</td><td>SRP SS</td></tr><tr><td>DDoS</td><td colspan="6">0.942 ± 0.008 0.852 ± 0.011 0.931 ± 0.003 0.866 ± 0.011 0.924 ± 0.006 0.927 ± 0.005</td></tr><tr><td>Donors</td><td colspan="6">0.962 ± 0.011 0.847 ± 0.011 0.737 ± 0.006 0.910 ± 0.013 0.728 ± 0.005 0.762 ± 0.016</td></tr><tr><td>Backdoor</td><td colspan="6">0.910 ± 0.021 0.935 ± 0.002 0.872 ± 0.012 0.943 ± 0.002 0.875 ± 0.002 0.882 ± 0.010</td></tr><tr><td>Ad</td><td colspan="6">0.887 ± 0.003 0.812 ± 0.002 0.718 ± 0.005 0.818 ± 0.002 0.696 ± 0.003 0.740 ± 0.008</td></tr><tr><td>Apascal</td><td colspan="6">0.823 ± 0.007 0.685 ± 0.019 0.732 ± 0.007 0.804 ± 0.021 0.604 ± 0.032 0.760 ± 0.030</td></tr><tr><td>Bank</td><td colspan="6">0.758 ± 0.007 0.690 ± 0.006 0.684 ± 0.004 0.736 ± 0.009 0.684 ± 0.002 0.688 ± 0.015</td></tr><tr><td>Celeba</td><td colspan="6">0.860 ± 0.006 0.682 ± 0.029 0.709 ± 0.005 0.794 ± 0.017 0.667 ± 0.033 0.734 ± 0.027</td></tr><tr><td>Census</td><td colspan="6">0.653 ± 0.004 0.661 ± 0.003 0.626 ± 0.006 0.661 ± 0.001 0.636 ± 0.006 0.560 ± 0.006</td></tr><tr><td colspan="7">Creditcard 0.957 ± 0.005 0.945 ± 0.001 0.950 ± 0.000 0.956 ± 0.003 0.947 ± 0.001 0.949 ± 0.003</td></tr><tr><td>Lung</td><td colspan="6">0.982 ± 0.006 0.867 ± 0.031 0.911 ± 0.006 0.968 ± 0.018 0.884 ± 0.018 0.928 ± 0.008</td></tr><tr><td>Probe</td><td colspan="6">0.997 ± 0.000 0.975 ± 0.000 0.998 ± 0.000 0.978 ± 0.001 0.995 ± 0.000 0.997 ± 0.001</td></tr><tr><td>R8</td><td colspan="6">0.902 ± 0.002 0.883 ± 0.006 0.867 ± 0.003 0.895 ± 0.004 0.830 ± 0.005 0.904 ± 0.005</td></tr><tr><td>Secom</td><td colspan="6">0.57 ± 0.004 0.541 ± 0.006 0.544 ± 0.011 0.563 ± 0.008 0.512 ± 0.007 0.530 ± 0.016</td></tr><tr><td>U2R</td><td colspan="6">0.986 ± 0.001 0.981 ± 0.001 0.987 ± 0.000 0.988 ± 0.002 0.987 ± 0.000 0.981 ± 0.002</td></tr><tr><td colspan="2">#wins/draws/losses (RDP vs.)</td><td>13/0/1</td><td>13/0/1</td><td>12/0/2</td><td>10/2/2</td><td>6/0/8</td></tr><tr><td colspan="4">4.2 PERFORMANCE EVALUATION IN CLUSTERING</td><td/><td/><td/></tr><tr><td colspan="3">4.2.1 EXPERIMENTAL SETTINGS</td><td/><td/><td/><td/></tr></table>

Two widely-used clustering performance metrics, Normalised Mutual Info (NMI) score and F-score, are used. Larger NMI or F-score indicates better performance.

Table content: <table/>

NMI and F-score performance of K-means on the original space and projected spaces.

Table content: <table><tr><td colspan="3">Data Characteristics</td><td/><td/><td colspan="2">NMI Performance</td><td/><td/></tr><tr><td>Data</td><td>N</td><td>D</td><td>Org</td><td>HLLE</td><td>SRP</td><td>AE</td><td>COP</td><td>RDP</td></tr><tr><td>R8</td><td colspan="8">7,674 17,387 0.524 ± 0.047 0.004 ± 0.001 0.459 ± 0.031 0.471 ± 0.043 0.025 ± 0.003 0.539 ± 0.040</td></tr><tr><td colspan="9">20news 18,846 130,107 0.080 ± 0.004 0.017 ± 0.000 0.075 ± 0.002 0.075 ± 0.006 0.027 ± 0.040 0.084 ± 0.005</td></tr><tr><td colspan="2">Olivetti 400</td><td colspan="7">4,096 0.778 ± 0.014 0.841 ± 0.011 0.774 ± 0.011 0.782 ± 0.010 0.333 ± 0.018 0.805 ± 0.012</td></tr><tr><td colspan="9">Sector 9,619 55,197 0.336 ± 0.008 0.122 ± 0.004 0.273 ± 0.011 0.253 ± 0.010 0.129 ± 0.014 0.305 ± 0.007</td></tr><tr><td colspan="7">RCV1 20,242 47,236 0.154 ± 0.000 0.006 ± 0.000 0.134 ± 0.024 0.146 ± 0.010</td><td>N/A</td><td>0.165 ± 0.000</td></tr><tr><td colspan="3">Data Characteristics</td><td/><td/><td colspan="2">F-score Performance</td><td/><td/></tr><tr><td>Data</td><td>N</td><td>D</td><td>Org</td><td>HLLE</td><td>SRP</td><td>AE</td><td>COP</td><td>RDP</td></tr><tr><td>R8</td><td colspan="8">7,674 17,387 0.185 ± 0.189 0.085 ± 0.000 0.317 ± 0.045 0.312 ± 0.068 0.088 ± 0.002 0.360 ± 0.055</td></tr><tr><td colspan="9">20news 18,846 130,107 0.116 ± 0.006 0.007 ± 0.000 0.109 ± 0.006 0.083 ± 0.010 0.009 ± 0.004 0.119 ± 0.006</td></tr><tr><td colspan="2">Olivetti 400</td><td colspan="7">4,096 0.590 ± 0.029 0.684 ± 0.024 0.579 ± 0.022 0.602 ± 0.023 0.117 ± 0.011 0.638 ± 0.026</td></tr><tr><td colspan="9">Sector 9,619 55,197 0.208 ± 0.008 0.062 ± 0.001 0.187 ± 0.009 0.184 ± 0.010 0.041 ± 0.004 0.191 ± 0.007</td></tr><tr><td colspan="7">RCV1 20,242 47,236 0.519 ± 0.000 0.342 ± 0.000 0.508 ± 0.003 0.514 ± 0.057</td><td>N/A</td><td>0.572 ± 0.003</td></tr></table>



Table content: <table><tr><td/><td/><td>Decomposition</td><td/><td colspan="2">Supervision Signal</td></tr><tr><td>Data</td><td>RDP</td><td>RDP\L rdp</td><td>RDP\L clu aux</td><td>Org SS</td><td>SRP SS</td></tr><tr><td>R8</td><td colspan="5">0.360 ± 0.055 0.312 ± 0.068 0.330 ± 0.052 0.359 ± 0.028 0.363 ± 0.046</td></tr><tr><td colspan="6">20news 0.119 ± 0.006 0.083 ± 0.010 0.117 ± 0.005 0.111 ± 0.005 0.111 ± 0.007</td></tr><tr><td colspan="6">Olivetti 0.638 ± 0.026 0.602 ± 0.023 0.597 ± 0.019 0.610 ± 0.022 0.601 ± 0.023</td></tr><tr><td>Sector</td><td colspan="5">0.191 ± 0.007 0.184 ± 0.010 0.217 ± 0.007 0.181 ± 0.007 0.186 ± 0.009</td></tr><tr><td>RCV1</td><td colspan="5">0.572 ± 0.003 0.514 ± 0.057 0.526 ± 0.011 0.523 ± 0.003 0.532 ± 0.001</td></tr></table>

Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1058-1067, 2017. Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851-1858, 2017.

Table content: <table><tr><td>Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng</td></tr><tr><td>Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In ICLR,</td></tr><tr><td>2018.</td></tr></table>

Datasets used in the clustering task

Table content: <table><tr><td>Data</td><td>N</td><td>D</td><td colspan="2">#Classes Link</td></tr><tr><td>R8</td><td>7,674</td><td>17,387</td><td>8</td><td>http://csmining.org/tl files/Project Datasets/r8 r52/r8-train-all-terms.txt</td></tr><tr><td colspan="3">20news 18,846 130,107</td><td>20</td><td>https://scikit-learn.org/0.19/datasets/twenty newsgroups.html</td></tr><tr><td>Olivetti</td><td>400</td><td>4,096</td><td>40</td><td>https://scikit-learn.org/0.19/datasets/olivetti faces.html</td></tr><tr><td>Sector</td><td>9,619</td><td>55,197</td><td>105</td><td>https://www.csie.ntu.edu.tw/∼cjlin/libsvmtools/datasets/multiclass.html#sector</td></tr><tr><td>RCV1</td><td colspan="2">20,242 47,236</td><td>2</td><td>https://www.csie.ntu.edu.tw/∼cjlin/libsvmtools/datasets/binary.html#rcv1.binary</td></tr><tr><td colspan="5">C AUC-PR PERFORMANCE OF ABLATION STUDY IN ANOMALY DETECTION</td></tr></table>

AUC-PR performance of RDP and its variants in the anomaly detection task.

Table content: <table><tr><td/><td/><td colspan="2">Decomposition</td><td/><td colspan="2">Supervision Signal</td></tr><tr><td>Data</td><td>RDP</td><td>RDP\L rdp</td><td>RDP\L ad aux</td><td>RDP\Boosting</td><td>Org SS</td><td>SRP SS</td></tr><tr><td>DDoS</td><td colspan="6">0.301 ± 0.028 0.110 ± 0.015 0.364 ± 0.013 0.114 ± 0.001 0.363 ± 0.007 0.380 ± 0.030</td></tr><tr><td>Donors</td><td colspan="6">0.432 ± 0.061 0.201 ± 0.033 0.104 ± 0.007 0.278 ± 0.040 0.099 ± 0.004 0.113 ± 0.010</td></tr><tr><td>Backdoor</td><td colspan="6">0.305 ± 0.008 0.433 ± 0.015 0.142 ± 0.006 0.537 ± 0.005 0.143 ± 0.005 0.154 ± 0.028</td></tr><tr><td>Ad</td><td colspan="6">0.726 ± 0.007 0.473 ± 0.009 0.491 ± 0.014 0.488 ± 0.008 0.419 ± 0.015 0.530 ± 0.007</td></tr><tr><td>Apascal</td><td colspan="6">0.042 ± 0.003 0.021 ± 0.005 0.031 ± 0.002 0.028 ± 0.003 0.016 ± 0.003 0.035 ± 0.007</td></tr><tr><td>Bank</td><td colspan="6">0.364 ± 0.013 0.258 ± 0.006 0.266 ± 0.018 0.278 ± 0.007 0.262 ± 0.016 0.265 ± 0.021</td></tr><tr><td>Celeba</td><td colspan="6">0.104 ± 0.006 0.068 ± 0.010 0.060 ± 0.004 0.072 ± 0.008 0.050 ± 0.009 0.065 ± 0.010</td></tr><tr><td>Census</td><td colspan="6">0.086 ± 0.001 0.081 ± 0.001 0.075 ± 0.001 0.087 ± 0.001 0.077 ± 0.002 0.064 ± 0.001</td></tr><tr><td colspan="7">Creditcard 0.363 ± 0.011 0.290 ± 0.012 0.414 ± 0.02 0.329 ± 0.007 0.362 ± 0.016 0.372 ± 0.024</td></tr><tr><td>Lung</td><td colspan="6">0.705 ± 0.028 0.381 ± 0.104 0.437 ± 0.083 0.542 ± 0.139 0.361 ± 0.054 0.464 ± 0.053</td></tr><tr><td>Probe</td><td colspan="6">0.955 ± 0.002 0.609 ± 0.014 0.952 ± 0.007 0.628 ± 0.011 0.937 ± 0.005 0.959 ± 0.011</td></tr><tr><td>R8</td><td colspan="6">0.146 ± 0.017 0.134 ± 0.031 0.109 ± 0.006 0.173 ± 0.028 0.067 ± 0.016 0.134 ± 0.019</td></tr><tr><td>Secom</td><td colspan="6">0.096 ± 0.001 0.086 ± 0.002 0.096 ± 0.006 0.090 ± 0.001 0.088 ± 0.004 0.093 ± 0.004</td></tr><tr><td>U2R</td><td colspan="6">0.261 ± 0.005 0.217 ± 0.011 0.266 ± 0.007 0.238 ± 0.009 0.187 ± 0.013 0.239 ± 0.023</td></tr><tr><td colspan="2">#wins/draws/losses (RDP vs.)</td><td>13/0/1</td><td>11/0/3</td><td>11/0/3</td><td>12/0/2</td><td>5/0/9</td></tr></table>

NMI performance of RDP and its variants in the clustering task.

Table content: <table><tr><td/><td/><td>Decomposition</td><td/><td colspan="2">Supervision Signal</td></tr><tr><td>Data</td><td>RDP</td><td>RDP\L rdp</td><td>RDP\L clu aux</td><td>Org SS</td><td>SRP SS</td></tr><tr><td>R8</td><td colspan="5">0.539 ± 0.040 0.471 ± 0.043 0.505 ± 0.037 0.567 ± 0.021 0.589 ± 0.039</td></tr><tr><td colspan="6">20news 0.084 ± 0.005 0.075 ± 0.006 0.081 ± 0.002 0.075 ± 0.002 0.074 ± 0.003</td></tr><tr><td colspan="6">Olivetti 0.805 ± 0.012 0.782 ± 0.010 0.784 ± 0.010 0.795 ± 0.011 0.787 ± 0.011</td></tr><tr><td>Sector</td><td colspan="5">0.305 ± 0.007 0.253 ± 0.010 0.340 ± 0.007 0.295 ± 0.009 0.298 ± 0.008</td></tr><tr><td>Rcv1</td><td colspan="5">0.165 ± 0.000 0.146 ± 0.010 0.168 ± 0.000 0.154 ± 0.002 0.147 ± 0.000</td></tr><tr><td colspan="4">E.1 SENSITIVITY TEST IN ANOMALY DETECTION</td><td/><td/></tr><tr><td colspan="6">Figures 2 and 3 respectively show the AUC-ROC and AUC-PR performance of RDP using different</td></tr><tr><td colspan="6">representation dimensions on all the 14 anomaly detection datasets used in this work. It is clear</td></tr><tr><td colspan="6">from both performance measures that RDP generally performs stably w.r.t. the use of different</td></tr><tr><td colspan="3">representation dimensions on diverse datasets.</td><td/><td/><td/></tr></table>

Since training time can vary significantly using different training strategies in deep learning-based methods, it is difficult to have a fair comparison of the training time. Moreover, theThe testing runtime in seconds of RDP and its five competing anomaly detection methods on 14 anomaly detection datasets are provided in Table10. Since most of the methods integrate representation learning and anomaly detection into a single framework, the runtime includes the execution time of feature learning and anomaly detection for all six methods. In general, on most large datasets, RDP runs comparably fast to the most efficient methods iForest and RND, and is faster the two recently proposed deep methods REPEN and DAGMM. Particularly, RDP runs faster than REPEN and DAGMM by a factor of around five on high-dimensional and large-scale datasets like Donors and Census. RDP is slower than the competing methods in processing small datasets. This is mainly

Table content: <table><tr><td>F-score Performance</td><td>0.2 0.3 0.4 0.5 0.6 0.7 0.8</td><td>AUC-ROC</td><td colspan="2">0.75 0.8 0.85 0.9 0.95 1</td><td/><td>R8 20news Olivetti Sector Rcv1</td><td>F-score Performance</td><td>0.2 0.3 0.4 0.5 0.6</td><td>DDoS Donors Back Ad Bank Celeba Census Credit Lung Probe R8 Secom U2r Apascal</td><td>R8 20news Olivetti Sector Rcv1</td></tr><tr><td/><td>512 0.1</td><td/><td colspan="2">768 0.7</td><td>1024</td><td>1280</td><td/><td>512 0.1</td><td>768</td><td>1024</td><td>1280</td></tr><tr><td/><td/><td colspan="4">0.65 Representation Dimension</td><td/><td/><td/><td>Representation Dimension</td></tr><tr><td/><td/><td/><td/><td>0.6</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td>25</td><td>50</td><td/><td>75</td><td>100</td></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="3">Representation Dimension</td></tr><tr><td/><td colspan="9">Figure 2: AUC-ROC results of RDP w.r.t. different representation dimensions on 14 datasets.</td></tr><tr><td/><td/><td/><td/><td>1</td><td/><td/><td/><td/><td>DDoS</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>Donors</td></tr><tr><td/><td/><td/><td/><td>0.9</td><td/><td/><td/><td/><td>Back Ad</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>Apascal</td></tr><tr><td/><td/><td/><td/><td>0.8</td><td/><td/><td/><td/><td>Bank Celeba</td></tr><tr><td/><td/><td/><td/><td>0.7</td><td/><td/><td/><td/><td>Census Credit</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>Lung</td></tr><tr><td/><td/><td/><td/><td>0.6</td><td/><td/><td/><td/><td>Probe R8</td></tr><tr><td/><td/><td colspan="2">AUC-PR</td><td>0.5</td><td/><td/><td/><td/><td>Secom U2r</td></tr><tr><td/><td/><td/><td/><td>0.4</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td>0.3</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td>0.2</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td>0.1</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td>0</td><td>25</td><td>50</td><td/><td>75</td><td>100</td></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="3">Representation Dimension</td></tr><tr><td/><td colspan="9">Figure 3: AUC-PR results of RDP w.r.t. different representation dimensions on 14 datasets.</td></tr><tr><td colspan="10">models can often be trained offline. Thus, we focus on comparing the runtime at the testing stage.</td></tr><tr><td colspan="10">All the runtime experiments below were done on a computing server node equipped with 32 Intel</td></tr><tr><td colspan="10">Xeon E5-2680 CPUs (2.70GHz) and 128GB Random Access Memory.</td></tr><tr><td colspan="9">F.1 TESTING RUNTIME IN ANOMALY DETECTION</td></tr></table>

Testing runtime (in seconds) on 14 anomaly detection datasets.

Table content: <table><tr><td colspan="3">Data Characteristics</td><td colspan="5">RDP and Its Five Competing Methods</td></tr><tr><td>Data</td><td>N</td><td colspan="6">D iForest AE REPEN DAGMM RND RDP</td></tr><tr><td>DDoS</td><td colspan="2">464,976 66</td><td colspan="3">54.06 86.86 172.47</td><td colspan="2">197.85 31.86 28.93</td></tr><tr><td>Donors</td><td colspan="2">619,326 10</td><td colspan="3">28.17 52.31 226.14</td><td colspan="2">194.45 44.31 36.84</td></tr><tr><td colspan="6">Backdoor 95,329 196 26.51 51.66 36.43</td><td colspan="2">187.61 12.26 29.95</td></tr><tr><td>Ad</td><td colspan="4">3,279 1,555 6.71 14.71</td><td>3.24</td><td>31.54</td><td>8.12 30.83</td></tr><tr><td>Apascal</td><td colspan="2">12,695 64</td><td>6.53</td><td>4.27</td><td>6.30</td><td>69.35</td><td>3.62 22.88</td></tr><tr><td>Bank</td><td colspan="2">41,188 62</td><td>9.72</td><td>6.87</td><td>17.25</td><td colspan="2">170.56 9.31 28.47</td></tr><tr><td>Celeba</td><td colspan="2">202,599 39</td><td colspan="3">20.54 26.70 71.60</td><td colspan="2">223.70 18.05 33.91</td></tr><tr><td>Census</td><td colspan="5">299,285 500 155.77 225.29 121.08</td><td colspan="2">236.21 42.83 57.74</td></tr><tr><td colspan="3">Creditcard 284,807 29</td><td colspan="3">22.45 29.38 103.18</td><td colspan="2">235.93 20.97 30.84</td></tr><tr><td>Lung</td><td colspan="4">145 3,312 6.20 13.11</td><td>2.16</td><td>39.75</td><td>1.44 24.29</td></tr><tr><td>Probe</td><td colspan="2">64,759 34</td><td colspan="3">9.55 10.06 28.14</td><td colspan="2">131.40 9.90 29.61</td></tr><tr><td>R8</td><td colspan="4">3,974 9,467 59.70 45.48</td><td>7.81</td><td>31.99</td><td>8.26 14.33</td></tr><tr><td>Secom</td><td colspan="2">1,567 590</td><td>7.32</td><td>5.78</td><td>2.83</td><td>18.22</td><td>3.23 22.52</td></tr><tr><td>U2R</td><td colspan="2">60,821 34</td><td>8.95</td><td>9.38</td><td>26.55</td><td colspan="2">185.88 9.90 28.10</td></tr><tr><td colspan="4">F.2 TESTING RUNTIME IN CLUSTERING</td><td/><td/><td/></tr></table>

shows the testing runtime of RDP and its four competing methods in enabling clustering on five datasets. Since exactly the same K-means clustering is used on the features in all the five cases, we exclude the runtime of the K-means clustering for more straightforward comparison. The results show that RDP runs comparably fast to the very efficient methods SRP and AE since they do not involve complex computation at the testing stage; RDP runs about five orders of magnitude faster than HLLE since HLLE takes a huge amount of time in its nearest neighbours searching. Note that 'Org' indicates the clustering performed on the original space, so it involves no feature learning and does not take any time.

Table content: <table/>

Testing runtime (in seconds) on five clustering datasets.

Table content: <table><tr><td colspan="6">Data Characteristics RDP and Its Four Competing Methods</td></tr><tr><td>Data</td><td>N</td><td>D</td><td colspan="2">Org HLLE SRP AE</td><td>RDP</td></tr><tr><td>R8</td><td colspan="2">7,674 17,387</td><td>-</td><td>9,658.85 1.16 1.08</td><td>0.89</td></tr><tr><td colspan="5">20news 18,846 130,107 -94,349.20 2.26 11.49</td><td>6.85</td></tr><tr><td colspan="2">Olivetti 400</td><td>4,096</td><td>-</td><td>166.02 0.73 0.03</td><td>0.03</td></tr><tr><td colspan="3">Sector 9,619 55,197</td><td colspan="2">-24,477.80 1.40 4.28</td><td>2.87</td></tr><tr><td colspan="3">RCV1 20,242 47,236</td><td colspan="2">-47,584.79 2.80 8.91</td><td>5.04</td></tr></table>

NMI and F-score performance of K-means clustering using RDP, Doc2Vec, and Doc2Vec+RDP based feature representations of the text datasets R8 and news20. ,387 0.317 ± 0.014 0.360 ± 0.055 0.316 ± 0.007 20news 18,846 130,107 0.115 ± 0.006 0.119 ± 0.006 0.126 ± 0.009

Table content: <table><tr><td colspan="3">Data Characteristics</td><td/><td colspan="2">NMI Performance</td></tr><tr><td>Data</td><td>N</td><td>D</td><td>Doc2Vec</td><td>RDP</td><td>Doc2Vec+RDP</td></tr><tr><td>R8</td><td colspan="5">7,674 17,387 0.241 ± 0.022 0.539 ± 0.040 0.250 ± 0.003</td></tr><tr><td colspan="6">20news 18,846 130,107 0.080 ± 0.003 0.084 ± 0.005 0.198 ± 0.009</td></tr><tr><td colspan="3">Data Characteristics</td><td/><td colspan="2">F-score Performance</td></tr><tr><td>Data</td><td>N</td><td>D</td><td>Doc2Vec</td><td>RDP</td><td>Doc2Vec+RDP</td></tr><tr><td>R8</td><td>7,674 17</td><td/><td/><td/><td/></tr></table>

F-score performance of classification on five real-world datasets.

Table content: <table><tr><td>Data</td><td>HLLE SRP</td><td>AE</td><td colspan="2">COP RDP\L clu aux</td><td>RDP</td></tr><tr><td>R8</td><td colspan="3">0.246 0.895 0.874 0.860</td><td>0.900</td><td>0.906</td></tr><tr><td>20news</td><td colspan="3">0.005 0.733 0.709 0.718</td><td>0.735</td><td>0.753</td></tr><tr><td colspan="4">Olivetti 0.895 0.899 0.820 0.828</td><td>0.900</td><td>0.896</td></tr><tr><td>Sector</td><td colspan="3">0.037 0.671 0.645 0.689</td><td>0.690</td><td>0.696</td></tr><tr><td>RCV1</td><td colspan="3">0.766 0.919 0.918 N/A</td><td>0.940</td><td>0.926</td></tr></table>

Peer review (iclr2020):
This paper proposed a method of unsupervised representation by transforming a set of data points into another space while maintaining the pairwise distance as good as possible. The paper is well structured with background literatures, formula, as well as experiments to show the advantage of the proposed method. I find it generally interesting, with the following major concerns. 1. Representation or dimension reduction? If the original space is a structured space like Euclidean space, then effectively this paper*s method coincides with regular distance preserving method in dimension reduction, and Johnson-Lindenstrauss theories. If the original space is not structured or doesn*t naturally have a good distance measure, then the proposed method cannot work. For example, if the original dataset is a set of documents, and the task is to do representation learning to convert each document into a compact vector. However, there*s no good distance metric for the document space. If TF-IDF is used, then the representation space also inherits TF-IDF type features which is not desired. If more advanced similarity is used for the document space, then the role of representation learning is not essential anymore as that similarity measure can already help the downstream tasks. 2. Section 3 the theoretical analysis. This part seems like a collection of previous works and contains minimal information about the proposed method. 3. Some writing issues, like page 4 line 7 about the equation numbering.