Unsupervised Representation Learning by Predicting Random Distances
Authors: Hu Wang
Guansong Pang
Chunhua Shen
Congbo Ma
Venue:  (2020)
Keywords: representation learning
unsupervised learning
anomaly detection
clustering

Abstract: Deep neural networks have gained tremendous success in a broad range of machine learning tasks due to its remarkable capability to learn semantic-rich features from high-dimensional data. However, they often require large-scale labelled data to successfully learn such features, which significantly hinders their adaption into unsupervised learning tasks, such as anomaly detection and clustering, and limits their applications into critical domains where obtaining massive labelled data is prohibitively expensive. To enable downstream unsupervised learning on those domains, in this work we propose to learn features without using any labelled data by training neural networks to predict data distances in a randomly projected space. Random mapping is a highly efficient yet theoretical proven approach to obtain approximately preserved distances. To well predict these random distances, the representation learner is optimised to learn class structures that are implicitly embedded in the randomly projected space. Experimental results on 19 real-world datasets show our learned representations substantially outperform state-of-the-art competing methods in both anomaly detection and clustering tasks.

Under review as a conference paper at ICLR 2020
UNSUPERVISED REPRESENTATION LEARNING BY PREDICTING RANDOM DISTANCES
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks have gained tremendous success in a broad range of ma- chine learning tasks due its remarkable capability learn semantic-rich features from high- sigh--imenssonal data. However, they often require large-scale labelled data to successfully learn such features, which significantly hinders their adaption into unsupervised learning tasks, such anomaly detection and clustering, and limits their applications into critical domains where obtaining massive labelled data is prohibitively expensive. To enable downstream unsupervised learning those domains, in this work we propose to learn features without using any labelled data by training neural networks to predict data distances in randomly projected space. Random mapping is a theoretical proven approach to obtain approximately preserved distances. To well predict these random distances, the representation learner is optimised to learn genuine class structures that are implicitly embedded in the randomly projected space. Experimental results on 19 real-world datasets show our learned representations substantially outperform state-of-the-art com- peting methods in both anomaly detection and clustering tasks.
INTRODUCTION
Unsupervised representation learning aims at automatically extracting expressive feature represen- tations from data without any manually labelled data. Due to the remarkable capability to learn semantic-rich features, deep neural networks have been becoming one widely-used technique em- power a broad range of machine learning tasks. One main issue with these deep learning techniques that a massive amount of labelled data is typically required to successfully learn these expressive features. As a result, their transformation power is largely reduced for tasks that are unsupervised in nature, such as anomaly detection and clustering. This is also true to critical domains, such as healthcare and fintech, where collecting massive labelled data is prohibitively expensive and/or is impossible to scale. To bridge this gap, in this work we explore fully unsupervised representation learning techniques enable downstream unsupervised learning methods on those critical domains.
In recent years, many unsupervised representation learning methods (Mikolov et al., 2013a Le & Mikolov 2014 Misra et al. 2016 Lee al., 2017 Gidaris al. 2018) have been introduced, of which most are self-supervised approaches that formulate the problem as an annotation free pretext task. These methods explore easily accessible information, such temporal or spatial neighbour- hood, to design surrogate supervisory signal to empower the feature learning. These methods have achieved significantly improved feature representations of text/image/video data, but they are often inapplicable to tabular data since it does not contain the required temporal or spatial supervisory in- formation. We therefore focus on unsupervised representation learning of high- fiigh--imessioall tabular data. Although many traditional approaches, such as random projection (Li et al. (2066), principal component analysis (PCA) (Rahmani & Atial
Atia|[2017), (017)) manifold learning (Donoho Grimes 2003)
Hinton & Roweis, 2003) and autoencoder (Vincent et al.) 2010). are readily available for handling those data, many of them (Donoho & Grimes 20033 Hinton & Roweis 2003 Rahmani & Atia 2017))
often too computationally costly to scale up to large high-dimensional data. Approaches like random projection and autoencoder are very efficient but they often fail to capture complex class structures due to its underlying data assumption weak supervisory signal.
In this paper, we introduce a Random Distance Prediction (RDP) model which trains neural networks to predict data distances in randomly projected space. When the distance information captures in-

---

Under review as a conference paper at ICLR 2020
trinsic class structure in the data, the representation learner is optimised to learn the class structure to minimise the prediction error. Since distances are concentrated and become meaningless in high dimensional spaces (Beyer et al. 1999p, we seek to obtain distances preserved in projected space to be the supervisory signal. Random mapping is a highly efficient yet theoretical proven approach to obtain such approximately preserved distances. Therefore, we leverage the distances in the ran- domly projected space to learn the desired features. Intuitively, random mapping preserves rich local proximity information but may also keep misleading proximity when its underlying data distribu- tion assumption is inexact; by minimising the random distance prediction error, RDP essentially leverages the preserved data proximity and the power of neural networks to learn globally consis- tent proximity and rectify the inconsistent proximity information, resulting in substantially better representation space than the original space. We show this simple random distance prediction en- ables us to achieve expressive representations with no manually labelled data. In addition, some lusk-dependent auxiliary losses can be optionally added as a complementary supervisory source to the random distance prediction, as to learn the feature representations that are more tailored for specific downstream task. In summary, this paper makes the following three main contributions.
We propose a random distance prediction formulation, which is very simple yet offers highly effective supervisory signal for learning expressive feature representations that optimise the distance preserving in random projection. The learned features are sufficiently generic and work well in enabling different downstream learning tasks.
Our formulation is flexible incorporate task-dependent auxiliary losses that are comple- mentary to random distance prediction to further enhance the learned features, i.e., features that are specifically optimised for downstream task while the same time preserving the generic proximity as much as possible.
As result, we show that our instantiated model termed RDP enables substantially bet- ter performance than state-of-the-art competing methods in two key unsupervised tasks, anomaly detection and clustering, on 19 real-world high-dimensional tabular datasets.
2 RANDOM DISTANCE PREDICTION MODEL
2.1 THE PROPOSED FORMULATION AND THE INSTANTIATED MODEL
We propose to learn representations by training neural networks to predict distances in a randomly projected space without manually labelled data. The key intuition is that, given some distance information that faithfully encapsulates the underlying class structure in the data, the representation learner is forced learn the class structure in order to yield distances that are as close as the given distances. Our proposed framework is illustrated in Figure Specifically, given data points x,, X, e RR, we first feed them into a weight-shared Siamese-style neural network o(x; θ). Φ ; RR + RM a representation learner with the parameters θ to map the data onto a -ddimensional new space. Then we formulate the subsequent step as a distance prediction task and define loss function as:
Tuddp (x,, x,) = I((x(x;; +), (x,;; 0)), (n(x.), n(x)))),
where 77 is an existing projection method and / is a function of the difference between its two inputs.
Here one key ingredient is how to obtain trustworthy distances via 1,. Also. to efficiently optimise the model, the distance derivation needs to be computationally efficient. In this work, we use the inner products in randomly projected space the source of distance/similariy since it is very efficient and there is strong theoretical support of its capacity in preserving the genuine distance information. Thus, our instantiated model RDP specifies Lnd (X,, x,) as follows
Lydp (x,, x,) = (ø(x;; +) ((x;; θ) (xx)) x)x,)))²
(2)
where ò is implemented by multilayer perceptron for dealing with tabular data and 77 1R + R is an off-the-shelf random data mapping function (see Sections Sectionss3.]]and]]] for detail). Despite its sim- plicity, this loss offers a powerful supervisory signal to learn semantic-rich feature representations that substantially optimise the underlying distance preserving in (see Section 3.3 for detail).
Since operate on real-valued vector space, the inner product is implemented by the dot product. The dot product used hereafter simplify the notation.
2

---

Under review as a conference paper at ICLR 2020
Xi
(x (1) (n(x)),(3))))
X11 Zig
Axp
(xx;)
Random Distance
Neural Network
Shared weights
Lrdp
X
(x²; 0)
8ys Ay2
Xjp
Neural Network
50
-10-40-00-50
Figure 1: The proposed random distance prediction (RDP) framework. Specifically, weight-shared two-branch neural network & first projects X; and X, onto a new space, in which we aim to min- imise the random distance prediction loss Lndp. i.c.. the difference between the learned distance (Ø(x;; θ), (xy; θ)) and predefined distance (17(x)),,(x)
(1 denotes an existing random map- ping). Laur an auxiliary loss that is optionally applied one network branch to learn comple- mentary information w.r.t Lrdp- The lower right figure presents 2-D t-SNE (Hinton & Roweis 2003) visualisation of the features learned by RDP on a small toy dataset optdigits with classes.
2.2 FLEXIBILITY TO INCORPORATE TASK-DEPENDENT COMPLEMENTARY AUXILIARY LOSS
Minimising Lngg learns to preserve pairwise distances that are critical to different learning tasks. Moreover, our formulation is flexible to incorporate a task-dependent auxiliary loss Lauze such as reconstruction loss (Hinton & Salakhutdinov (206)) for clustering or novelty loss (Burda et al. 2019) for anomaly detection, complement the proximity information and enhance the feature learning.
For clustering, an auxiliary reconstruction loss is defined as:
I. clu (x) (x &' ();
(3)
where is an encoder and d' R RDP to better capture global feature representations.
R D is a decoder. This loss may be optionally added into
Similarly, in anomaly detection novelty loss may be optionally added, which is defined as:
Lad (x) = (ő(x; - -y(x))²
(4)
By using a fixed n, minimising Lad helps learn the frequency of underlying patterns in the data (Burda et al., 2019), which is important complementary supervisory source for the sake of anomaly detection. As result, anomalies or novel points are expected to have substantially larger (6(x;)))-xxx) than normal points, so this value can be directly leveraged to detect anomalies.
Note since L ad involves mean squared error between vectors, the dimension of the projected space resulted by ó and 17 is required to be equal in this case. Therefore, when this loss is added into RDP, the M in and K 17 need to be the same. We do not have this constraint in other cases.
THEORETICAL ANALYSIS OF RDP
This section shows the proximity information can be well approximated using inner products in two types of random projection spaces. This is a key thcorctical foundation to RDP. Also, to accurately predict these distances, RDP is forced to learn the genuine class structure in the data.
3.1 WHEN LINEAR PROJECTION IS USED
Random projection is simple yet very effective linear feature mapping technique which has proven the capability of distance preservation. Let X' R x be set of N data points, random projection

---

Under review as a conference paper at ICLR 2020
uses random matrix A ART. The Johnson-Linde Lindenstrauss lemma (Johnson & Lindenstrauss, 1984) guarantees the data points be mapped to randomly selected space of suitably lower dimension with the distances between the points approximately preserved. More specifically, let (0, 2) and K 20 There exists linear mapping f R. ? + IR K such that for all X,,Y, X: (1-c) xy1|² & lx,(x,) fxx;))1 s (1 +c]xx x,11².
RXX to project the data onto lower K-dimensional space by X'
(5)
Furthermore, assume the entries of the matrix A are sampled independently from Gaussian distri- bution (0, 1). Then, the norm of x RD can be preserved as:
Pr (1 (1-)) |x|
Ax| S (1 + +]]x 2 2e VK
(6)
Under such random projections, the norm preservation helps well preserve the inner products:
33x
Pr [[x & -(x)) f(x))) 2 <) v 4e
where & is normalised x such that | 1.
The proofs of Eqns.
6 and
can be found in Vempala 1998).
Eqn. 7 states that the inner products in the randomly projected space can largely preserve the inner products in the original space, particularly when the projected dimension is large.
3.2 WHEN NON-LINEAR PROJECTION Is USED
Here we show that some non-linear random mapping methods are approximate to kernel functions which are a cell-established approach to obtain reliable distance/similarity information. The key to this approach is the kernel function A X' x X I, which is defined as xxx,,x,) = (v(x), (xxx)) )), where & is a feature mapping function but needs not to be explicitly defined and (. denotes a suitable inner product. A non-linear kernel function such polynomial or radial basis function (RBF) kernel is typically used to project linear- inceariiseeearale data onto a linear-separable space.
The relation between non-linear random mapping and kernel methods is justified in (Rahimi & Recht 2008). which shows that an explicit randomised mapping function RR. + R can be defined to project the data points onto low-dimensional Euclidean inner product space such that the inner products in the projected space approximate the kernel evaluation:
k(x;, xy) (xxx,,) (xx)) g(x:) g(x;).
Let A be the mapping matrix. Then to achieve the above approximation, is required be drawn from Fourier transform and shift-invariant functions such as cosine function are finally applied to Ax to yield a real-valued output. By transforming two data points x, and x, in this manner. their inner product gxx:) g ((x;) is unbiased estimator of k(x;,
3.3 LEARNING CLASS STRUCTURE BY RANDOM DISTANCE PREDICTION
Our model using only the random distances as the supervisory signal can be formulated as:
argmin Σ
(o(x;;) o(x;; Q) - yiz))
X, M, EX
where Vig .x.)) (x)). Let Y, RNN be the distance/similarity matrix of the N data points resulted by Then to minimise the prediction error in Eqn. ( á is optimised to learn the underlying class structure embedded Y. As shown in the properties in Eqns. (7) and (8). Y,, can effectively preserve local proximity information when 17 is set to be either the random projection- based f function or the kernel method-based g function. However, those proven v) often built upon some underlying data distribution assumption, e.g., Gaussian distribution in random projection or Gaussian RBF kernel, so the proojeeceed features can preserve misleading proximity when the distribution assumption is inexact. In this case, Y, is equivalent to the imperfect ground truth with partial noise. Then optimisation with Eqn. (9) is to leverage the power of neural networks to learn consistent local proximity information and rectify inconsistent proximity, resulting in a significantly optimised distance preserving space. The resulting space conveys substantially richer semantics than the 7 projected space when Y,, contains sufficient genuine supervision information.
4

---

Under review as a conference paper at ICLR 2020
4
EXPERIMENTS
This section evaluates the learned representations through two typical unsupervised tasks: anomaly detection and clustering. Some preliminary results of classification can be found in Appendix HH
4.1
PERFORMANCE EVAL.CATION IN ANOMALY DETECTION
4.1.1
EXPERIMENTAL SETTINGS
Our RDP model is compared with five state-of-the-art methods, including iForest (Liu et al. 2008 autoencoder (AE) (Hinton & Salakhutdinov 2006), REPEN (Pang al. 2018). DAGMM (Zon et al. 2018) and RND (Burda et al.) 2019). Forest and AE are two of the most popular baselines. The other three methods learn representations specifically for anomaly detection.
As shown in Table T the comparison is performed on 14 publicly available datasets of various domains, including network intrusion, credit card fraud detection, disease detection and bank cam- paigning. Many of the datasets contain real anomalies, including DDoS, Donors, Backdoor, Cred- itcard, Lung, Probe and U2R. Following (Liu et al. 2008 Pang et al., [0118] Zong et al., (018)) the rare class(es) is treated as anomalies in the other datasets to create semantically real anomalies. The Area Under Receiver Operating Characteristic Curve (AUC-ROC) and the Area Under Precision- Recall Curve (AUC-PR) used as performance metries. Larger LUC-ROC/AUC-PE indicates better performance. The reported performance is averaged over 10 independent runs.
Table 1: AUC-ROC (meanttstd) performance of RDP and its five competing methods on 14 datasets.
Data Characteristics
Our Method RDP and Its Five Competing Methods
Data
N
D
Anomaly (%)
iForest
AE
REPEN
DAGMM
RND
RDP
DDo8
464,976
66
3.75%
0,880 = 0.018 0.901 = 0.000
0.933 09933=0.002 0.766 2 0.019 0.852 = 0.011 0.942 : 0.008
Donors
619,326
5.92%
0.774 0.010 0.812
0001 0.777 2.77700005
0.763+00.1
0.847 0.011 0.962
0.011
Backdoor
95,329
196
2.44%
0.723 = 0.029 0.806
0.007 0857 0:57:000 0.001
3.813=0.00
0,935 = 0.002 0.910 : 0.021
3,279
1,555
13.99%
0.687 + 0.021 0,703
0.000 0.853 3653==00m
0.500 0.000
0.812 0.002 0.887 0.003
Apascal
12,695
64
1.38%
0,514 + 0.051 0.623
0.005
0.813 0.004
0.710 + 0.020 0.685 + 0.019 0.823 0.007
Bank
41.
62
11.26%
0,713 0.021 0.666
0.000
0.681 = 0.001 0.616
0.690 + 0.006 0.758 0.007
Celeba
202,599
39
2.24%
0.693 + 0.014 0.735
0.002
0.802
0.002
0.680 + 0.067 0.682 + 0.029
0.860 0.006
Census
299,285
500
6,20%
0.599 1 0.019 0,602
0.000
0.542
0,003 0.502 1 0.023 0,661 0,003 0.653 0.004
Creditcard
284,807
29
0,17%
0,948 + 0.005 0,948
0.000
0,950
0.001 0.877 + 0.005 0.945 + 0.001 0.957 0.005
Lung
145
3,312
4.13%
0.893 1 0.057 0,953
0.004
0,949
0,002
0.830 1 0.087 0.867 1 0.031
0.982 0.006
Probe
64.759
34
6.43%
0,995 + 0.001 0,997
0.000
0.997
= 0.000
0.953 : 0.008 0.975 : 0.000
0.997 0.000
R8
3,974
9.467
1,28%
0.841 1 0.023 0,835
0.000
0.910
0.000 0.760 1 0.066
0.883 0.006
0,902 0.002
Secom
1,567
590
6.63%
0,548 + 0.019 0.526 0.000
0.510
0.004 0.513
: 0.010 0.541
- 0.006
0.570 0.004
U2R
60,821
34
0.37%
0.988 = 0.001 0.987 0.000
0.978 = 0.000 0.945 = 0.028
0.981 + 0.001
0.986 0.001
Our RDP model uses the optional novelty loss for anomaly detection task by default. Similar to RND, given data point x, its anomaly score in RDP is defined as the mean squared error between the two projections resulted by o(x; θ*) and (x)). Also, a boosting process is used to filter out 5% likely anomalies per iteration to iteratively improve the modelling of RDP. This is because the modelling is otherwise largely biased when anomalies are presented. In the ablation study in Section will show the contribution of all these components.
4.1.2 COMPARISON TO THE STATE-OF -HET COMPETING METHODS
The AUC-ROC and AUC-PR results are respectively shown in Tables I and 2. RDP outperforms all the five competing methods in both of AUC-ROC and AUC-PR in at least 12 out of 14 datasets. This improvement is statistically significant at the 95% confidence level according to the two-tailed sign test (Demšar 2006). Remarkably, RDP obtains more than 10% AUC-ROC/AUC-PR improvement over the best competing method on six datasets, including Donors, Ad, Bank, Celeba, Lung and U2R. RDP can be thought as a high-level synthesis of REPEN and RND, because REPEN leverages pairwise distance-based ranking loss to learn representations for anomaly detection while RND is built using Lad In nearly all the datasets, RDP well leverages both Lydp and Lad to achieve datasets significant improvement over both REPEN and RND. In very limited cases, such Backdoor and Census where RND performs very well while REPEN performs less effectively, RDP is slightly downgraded due to the of Lrdp- In the opposite case, such Probe, on which REPEN performs much better than RND, the of I, ouz may drag down the performance of RDP bit.
5

---

Under review as a conference paper at ICLR 2020
Table 2: AUC-PR (ttan--std) performance of RDP and its five competing methods on 14 datasets..
Data
1Forest
AE
REPEN
DAGMM
RND
RDP
DDoS
0.141 = 0.020 0.248 = 0.001
0.300 0.012 0.038 = 0.000
0.110
0.015
0.301
0.028
Donors
0.124
0.138 0.007
0
0,070 0.024
0.201 = 0.003 0.432 + 0.061
Backdoor
0.045 0.0007 0.065 0.004
0.129
0.001
0.433
0.015
0.305 0.008
Ad
0.363 0.066 0.479 + 0.000 0.600
0,140=0.000
0.473
= 0.009
0.726 0.007
Apascal
0.015 0.002 0.023 0.001 0.041
0.001
0.009
0.021
0.005
0.042 0.003
Bank
0.293 1: 0.023 0.264 1 0.001 0,276
0.001 0,150 1 0.020
0.258
1 0.006 0.364
0.013
Celeba
0.060 0.006 0.082
0.001
0.081
0.001
0.068
0.010
0.104
0.006
Census
0,071 1: 0.004 0.072 1 0.000 0.064
0.005
0.061
J. 0.001
0.081
1 0.001 0.086
0.001
Creditcard
0.145 + 0.031
0.004 0,359
0,010
0.2%0
0.012
0,363
0.011
Lung
0.379 1: 0.092 0,565 1 0.022 0.429 0.005 0.042 1 0.003 0.381 1 0.104 0.705 0.028
Probe
0.923 + 0.011 0,964
0.002
0.609
0.014
0.002
R8
0.076 + 0.018 0.097
= 0.006 0.083 = 0.000 0.019 = 0.011
0. 134 = 0.031
0.146 + 0.017
Secom
0.106 + 0.007 0.053 0.000 0,091
0,066 0.002
0.086 0.002
0.001
U2R
0.180 + 0.018 0.230
= 0.004
0.116
= 0.007 0.025 = 0.019
0.217
= 0.011
0.261 + 0.005
4.1.3 ABLATION STUDY
This section examines the contribution of Lydps Lad and the boosting process to the performance of RDP. The experimental results in AUC-ROC are given in Table where RDP X means the RDP variant that removes the 'X' module from RDP. In the last two columns, Org-SS indicates that we directly use the distance information calculated in the original space as the supervisory signal, while SRP SS indicates that we use SRP to obtaín the distances as the supervisory signal. is clear that the full RDP model is the best performer. Using the Lrdp loss only, i.e., RDP(Led. L can achieve performance substantially better than, or comparably well to, the five competing methods in [ This is mainly because the Lydp loss alone can effectively force our representation learner to learn the underlying class structure on most datasets so as to minimise its prediction error. The use of Lad and boosting process well complement the Lrdp loss on the other datasets.
In terms of supervisory source, RDP and SRP_SS perform substantially better than Org-SS on most datasets. This is because the distances in both the non-linear random projection in RDP and the linear projection in SRP.SS well preserve the distance information, enabling RDP to effectively learn much more faithful class structure than that working the original space.
Table 3: AUC-ROC results of anomaly detection (see Appendix for similar AUC-PR results).
Decompositon
Supervision Signal
Data
RDP
RDP Ledaa
RDP
RDP Beosting
Org SS
SRP SS
DDaS
0.008
0.011
0,931
0.003
0.011
1.924
0,927 0.005
Donors
0.962 + 0.011
0.847
0.011 0.737
+ 0.006
0.910
0.013
0.728 + 0.005 0.762 + -0016
Backdoor
0.021
0.872
0.012
0.943
0.002
0.875
0,882
0.010
Ad
0.887 1 0.003
0.812 1 0.002
0.718
1: 0.005
0.818
: 0.002
0.696
1: 0.003 0.740
: 0,008
Apascal
0.007
0.732
0.007
0.021
0,760
0.030
Bank
0.758 1 0.007
0.690
1: 0.006 0.684 1: 0.004
0.736
2 0.009
0.684
1: 0.002 0.688 : 0.015
Celeba
0,860
0.006
0.682
0.709 + 0.005
0.794
0.667
0.734 0.027
Census
0.653 :1: 0.004
0.661 1: 0.003 0.626 1 0.006
0.661
- 0.001
0.636 1: 0.006 0,560 : 0.006
Creditcard
0.957 0.005
0.945
: 0.001 0.950 + 0.000
0.956
0.947
0.949 0.003
Lung
0.982 - 0.006
0.867 + 0.031 0.911 + 0.006
0.968 = 0.018
0.884 +: 0.018 0.928 = 0.008
Probe
0.997 0.000
0.975
0,998 + 0.000
0.978
0.995
0.997 0.001
R8
0.902 - 0.002
0.883
+ 0.006
0.867
+ 0.003
0.895
= 0.004
0.830 + 0.005 0.904 = 0.005
Secom
0.57 0.004
0.541
0.006
0.544
+- 0.011
0.563
0.008
0.512 0.007
0.530
0.016
U2R
0.986 1. 0.001
0.981
0.001
0.987
1 0,000
0.988
0.002
0.987 1. 0.000
0.981
1 0.002
wwwns/draws//neee
(RDP vs.)
13/V1
13/11
12NY2
10/22
60/
4.2 PERFORMANCE EVALUATION IN CLUSTERING
4.2.1
EXPERIMENTAL SETTINGS
For clustering, RDP is compared with four state-of-the-art unsupervised representation learning methods in four different areas, including HLLE (Donoho & Grimes) 2003) in manifold learning, Sparse Random Projection (SRP) (Li et al. 2006) in random projection, autoencoder (AE) (Hinton & Salakbutdinov 2006) in data reconstruction-bart based neural network methods and Coherence Pursuit (COP) (Rahmani & Atial 2017) in robust PCA. These representation learning methods are first used to yield the new representations, and K-means (Hartigan & Wong 1979) is then applied to the

---

Under review as a conference paper at ICLR 2020
representations to perform clustering. Two widely-used clustering performance metrics, Normalised Mutual Info (NMI) score and F-score, are used. Larger NMI or F-score indicates better performance. The clustering performance in the original feature space, denoted Org, is used shown in Table 4 five high-dimensional real-world datasets are used. Some of the datasets are image/text data. Since here we focus on the performance tabular data, they are converted into tabular data using simple methods, i.c., by treating each pixel as feature unit for image data or using bag-of-words representation for text dat: The reported NMI score and F-score are averaged over 30 times to address the randomisation issue in K-means clustering. In this section RDP adds the reconstruction loss L L by default, but RDP also works very well without the use of L clu axe.
baseline. As
4.2.2 COMPARISON TO THE-STATE-OF-THE--AN COMPETING METHODS
Table shows the NMI and F-score performance of K-means clustering. Our method RDP en- ables K-means to achieve the best performance on three datasets and ranks second in the other two datasets. RDP-enabled clustering performs substantially and consistently better than that based on AE in terms of both NMI and F-score. This demonstrates that the random distance loss enables RDP to effectively capture some class structure in the data which cannot be captured by using the reconstruction loss. RDP also consistently outperforms the random projection method, SRP, and the robust PCA method, COP. It is interesting that K-means clustering performs best in the original space on Sector. This may be due to that this data contains many relevant features, resulting in no obvious curse of dimensionality issue. Olivetti may contain complex manifolds which require ex- tensive neighhourhood information to find them, so only HILE can achieve this goal in such cases. Nevertheless, RDP performs much more stably than HLLE across the five datasets.
Table 4: NMI and F-score performance of K-means on the original space and projected spaces.
Data Characteristics
NMI Performance
Data
D
Org
HLLEE
SRP
COP
RDP
R8
7,674
17.387
0,524
= 0,047
0.004
0.00 0.459 0.031 0.471 1 0.043
0.025 = 0.003
0.539 = 0.040
20news
18.846
130,107
0.080
0.004
0.017 + 0.000
0.075 + 0.002 0.075
0.006
0.027
0.040
0.084
0.005
Olivetti
400
4,096
0.778
0.014
0.841
+ 0.011
0.774
0.011
0.782 + 0.0100
0.333
+ 0.018
0.805
+ 0.012
Sector
9,619
55.197
0.336
0.088
0.122
+ 0.004 0273
0.253 0 0.010 0.129
0.129
+ 0.014
0,305 0.007
RCVI
20,242
47,236
0.154
0.000
0.006
+ 0.000 0.134 + 0.024 0,146 +
+-0000 0.010
N/A
0.165 0,000
Data haracteristies
F-score Performance
Data
N
D
Org
HLLE
SRP
AE
COP
RDP
R8
7,674
17.387
0.185 0.189
0.085 1 0.000
0.317 1 0:045 0.312 1 00068 0.088 1 0.002
0.360 :: 0.055
20news
18,846
130,107
0,116
0.006
0.007 : 0.000
0.109 : 0.006 0.083 + 0.010 0.009 = 0.004
0.119
0.006
Olivelti
400
4.096
0.590
0.029
0.684 1 0.024
0.579 1 0.022 0.602 0.023
0.117 1: 0.011
0.638 :: 0.026
Sector
9.619
55.197
0,208
0,008 0.062
: 0.001
0.187 : 0.009 0,184 + 0.010 0.041 # 0.004
0.191 0.007
RCVI
20,242
47,236
0.519
0.000
0.342 + 0.000
0.508
- 0.003
+ 0.057
N/A
0.572 = 0.003
Table 5: F-score performance of K-means clustering (see similar NMI results in Appendix D
Decompositon
Supervision Signal
Data
RDP
Lod
RDP RDP,Z20
Org S8
SRP SS
R8
0.360
0.055
0.312 1 0.068
0.330 0.052
0.359 0.028
0.363 1 0.046
20news
0,119
0.006
0.083
1 0.010
0.117 1 0.005
0.111
1 0.000
0.111 1 0.007
Olivetti
0.026
0.602
0.023
0.597 0.019
0.022
0.601
0.023
Sector
0,191 = 0.007
0.184
+ 0.010
0.217
= 0.007
0.181
- 0.007
0.186
+ 0.009
RCVI
0.003
0.514
0.057
0.526
0.011
0.003
0.532 0.001
4.2.3 ABLATION STUDY
Similar to anomaly detection, this section examines the contribution of the two loss functions Lydp and Lew to the performance of RDP, well the impact of different supervisory sources on the performance. The F-score results of this experiment are shown in Table 5 in which the notations have exactly the same meaning in Table 3 The full RDP model that uses both Lrdp and Lgla performs more favourably than its two variants, RDP RDP', and RDP Lady but it is clear that using T.ddp only performs very comparably to the full RDP. However, using T- only may result in large
RDP can also build upon advanced representation learning methods for the data transformation, for which some interesting preliminary results presented in Appendix G
7

---

Under review as a conference paper at ICLR 2020
performance drops in some datasets, such as R8, 20news and Olivetti. This indicates Lrdp is more important loss function to the overall performance of the full RDP model. In terms of supervisory source, distances obtained by the non-linear random projection in RDP are much more effective than the two other sources on some datasets such as Olivetti and RCVI. Three different supervisory sources are very comparable the other three datasets.
RELATED WORK
Self-supervised Learning. Self-supervised learning has been recently emerging as one of the most popular and effective approaches for representation learning. Many of the self self-suuer methods learn high-level representations by predicting some sort of 'context' information, such as spatial or temporal neighbourhood information. For example, the popular distributed representation learning techniques NL., such as CBOW/skip-geee gramm (Mikolov et al. 2013a) and phrase/sentence embed- dings in (Mikolov et al., 2013b, Le & Mikolov] [2114] Hill et al. (016), learn the representations by predicting the text pieces (c.g..
using its surrounding pieces as the con- text. In image processing, the pretext task can be the prediction of patch of missing pixels (Pathak) eet al. 2016) Zhang et al. (2017) or the relative position of two patches (Doersch et al. (015)) Also,
number of studies (Goroshin al. 2015 Misra et al. 2016 Lee et al.) 2017 Oord et al.) 2018)
explore temporal contexts to learn representations from video data, e.g., by learning the temporal order of sequential frames. Some other methods (Agrawal et al. 2015 Zhou al.. 2017 Gidaris et al.] (018)) are built upon a discriminative framework which aims at discriminating the images be- fore and after some transformation, e.g., ego motion in video data (Agrawal et al.) 2015 Zhou al. 2017] and rotation of images (Gidaris et al.] 2018). There have also been popular to use generative adversarial networks (GANs) learn features (Radford et al.) 2013 Chen et al. 2016). The above methods have demonstrated powerful capability to learn semantic representations. However, most of them use the supervisory signals available in image/video data only, which limits their application into other types of data, such as traditional tabular data. Although our method may also work image/video data, we focus on handling high-dimensional tabular data to bridge this gap.
Other Approaches. There have been several well-established unsupervised representation learn- ing approaches for handling tabular data, such as random projection (Arriaga & Vempala, 1999,
Bingham & Mannila 2001 Li al. 2006), PCA (Wold et al. 1987 Schölkopf et al.; 1997| Rah-
mani & Atia (2017). manifold learning (Roweis & Saul 2000]] Donoho & Grimes [203]] Hinton &
Roweis 2003 McInnes al.) 2018 and autoencoder (Hinton & Salakhutdinov [2066] Vincent al.)
2010). One notorious issue of PCA or manifold learning approaches is their prohibitive computa-
tional cost in dealing with large-scale high nigh-dimensional data due to the costly neighbourhood search and/or eigen decomposition. Random projection is computationally efficient approach, supported lemma (Johnson & Lin-
by proven distance preservation theories such as the Jahnson-L.nneentrern denstrauss 1984). We show that the preserved distances by random projection can be harvested to effectively supervise the representation learning. Autoencoder networks are another widely-used efficient feature learning approach which learns low-dimensional representations by minimising construction errors. One main issue with autoencoders is that they focus on preserving global infor- mation only, which may result in loss of local structure information. Some representation learning methods are specifically designed for anomaly detection (Pang et al.) 2018 Zong et al. 20018) Burda al.) 2019). By contrast, we aim generic representations learning while being flexible incor- porate optionally task-dependent losses to learn task-specific semantic-rich representations.
6 CONCLUSION
We introduce a novel Random Distance Prediction (RDP) model which learns features in fully unsupervised fashion by predicting data distances in a randomly projected space. The key insight is that random mapping is theoretical proven approach to obtain approximately preserved distances, and to well predict these random distances, the representation learner is optimised to learn consistent preserved proximity information while at the same time rectifying inconsistent proximity, resulting in representations with optimised distance preserving. Our idea is justified by thorough experiments in two unsupervised tasks, anomaly detection and clustering, which show RDP-enabled anomaly datasets. We
detectors and clustering substantially outperform their counterparts on 19 plan to extend RDP to other types of data to broaden its application scenarios.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In Proceedings of the IEEE International Conference on Computer Vision, pp. 37-45, 2015.
Akiko Aizawa. An information-theeeee Management, 39(1):45-65, 2003.
perspective of tf-idf measures. Information Processing &
Rosa I Arriaga and Santosh Vempala. An algorithmic theory of learning: Robust concepts and random projection. 40th Annual Symposium on Foundations of Computer Science, pp. 616 623. IEEE, 1999.
Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft. When is "nearest neighbor" meaningful? International Conference on Database Theory, pp. 217-235. Springer, 1999.
Ella Bingham and Heikki Mannila. Random projection in dimensionality reduction: applications to image and text data. Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 245-250. ACM, 2001.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In ICLR, 2019.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7(Jan):1-30 2006.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp- 1422-1430, 2015.
David L Donoho and Carrie Grimes. Hessian cigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences, 100(10)55511-556, 2003.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, and Yann LeCun. Unsupervised learn- ing of spatiotemporally coherent metrics. Proceedings of the IEEE international conference on computer vision, pp. 4086-4093, 2015.
John Hartigan and Manchek A Wong. Algorithm as 136: k-means clustering algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):100-108, 1979.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In 15th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3367-1377,20 2016.
Geoffrey E Hinton and Sam T Roweis. Stochastic neighbor embedding. Advances in neural information processing systems, pp. 857-864, 2003.
Geoffrey E Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 31 313(5786):504-500) 2006.
William Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics, 26(189-206):1, 1984.
Quoc Le and Tomas Mikolov. Distributed representations sentences and documents. Interna- fional conference machine learning, pp. 2014.

---

Under review as a conference paper at ICLR 2020
Hsin-
Hsin-Ying Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Hsuan Yang. Unsupervised represen- tation learning by sorting sequences. In Proceedings of the IEEE International Conference on Computer Vision, pp. 667-676, 2017.
Ping Li, Trevor Hastie, and Kenneth W Church. Very sparse random projections. In Proceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 287-296. ACM, 2006.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 Eighth IEEE Interna- tional Conference on Data Mining. pp. 413-422. IEEE, 2008
Leland McInnes, John Healy, and James Melville. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXi::1802.03422 2018.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint
2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen. Greg Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 3111-3119, 2013b.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European Conference on Computer Vision, pp. 527-544. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv:1807.03748, 2018.
Guansong Pang, Longbing Cao, Ling Chen, and Huan Liu. Learning representations ultrahigh- dimensional data for random iittace-based outlier detection. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2041-2050. ACM, 2018.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 2536-2544, 2016.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2015.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in neural information processing systems, pp. 1177-1184, 2008.
Mostafa Rahmani and George Atia. Coherence pursuit: Fast, simple, and robust subspace recov- ery. In Proceedings of the 34th International Conference on Machine Learning, pp. 2864-2873. JMLR. org, 2017.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embed- ding. Science, 290655000:2323-22222 2000.
Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Kernel principal component anal- ysis. In International conference on artificial neural networks, pp. 583-588. Springer, 1997.
Santosh Vempala. Random projection: A new approach to vlsi layout. In Proceedings 39th Annual Symposium on Foundations of Computer Science, pp. 389-395, IEEE, 1998.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of machine learning research, 1 11:3371-3408, 2010.
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.
10

---

Under review as a conference paper at ICLR 2020
Richard Zhang. Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1058-1067, 2017.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1851-1858, 2017.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In ICLR, 2018.
A IMPLEMENTATION DETAILS
RDP-enabled Anomaly Detection. The RDP consists of one fully connected layer with 50 hidden units, followed by leaky-ReI.U layer. It is trained using Stochastic Gradient Descent (SGD) as its optimiser for 200 epochs, with 192 samples per batch. The learning rate fixed to 0.1, We repeated the boosting process 30 times to obtain statistically stable results. In order to have fair comparisons, we also adapt the competing methods AE, REPEN, DAGMM and RND into ensemble methods and perform the experiments using an ensemble size of 30.
RDP-enabled Clustering. RDP uses a similar network architecture and optimisation settings as the one used in anomaly detection, i.c., the network consists of one fully connected layer, followed by a Ieaky-ReLU layer, which is optimised by SGD with 192 samples per batch and 0.1 learning rate. Compared to anomaly detection, more semantic information is required for clustering algorithms to work well, so the network consists of 1,024 hidden units and is trained for 1, 1,000 epochs. Clus- tering is a significant yet common analysis method, which aims grouping samples close each other into the same clusters and separating far away data points into different clusters. Compared to anomaly detection that often requires pattern frequency information, clustering has a higher re- quirement of the representation expressiveness. Therefore, if the representative ability of a model is strong enough, should also be able to learn representations that enable clustering to work well on the projected space.
Note that the representation dimension M in the Φ function and the projection dimension K in the n function are set to be the same to alleviate parameter tuning. This means that M = K 50 is used in anomaly detection and M = K = 1024 is used in clustering. We have also tried deeper network structures, but they worked less effectively than the shallow networks in both anomaly detection and clustering. This may be because the supervisory signal is not strong enough to effectively learn deeper representations. We show in Appendix E that RDP performs stably w.r.t. range of representation dimensions in both anomaly detection and clustering tasks.
The runtime of RDP the testing stage is provided in Appendix F with that of the competing meth- ods as baselines. For both anomaly detection and clustering tasks, RDP achieves very comparable time complexity the most efficient competing methods (see Tables [10 and 11 Appendix F for detail).
B DATASETS
The statistics and the accessible links of the datasets used in the anomaly detection and clustering tasks are respectively presented in Tables and 7 DDoS is dataset containing DDoS attacks and normal network flows. Donors is from KDD Cup 2014. which is used for detecting a very small number of outstanding donors projects. Backdoor contains backdoor network attacks derived from the UNSW-NB15 dataset. Creditcard is credit card fraud detection dataset. Lung contains data records of lung cancer patients and normal patients. Probe and U2R are derived from KDD Cup 99, in which probing and user-to-root attacks are respectively used as anomalies against the normal network flows. The above datasets contain real anomalies. Following {Liu et al., 2008 Pang et al. 2018 Zong et al., 2018), the other anomaly detection datasets transformed from classification datasets by using the rare class(es) as the anomaly class, which generates semantically real anomalies.
11

---

Under review as a conference paper at ICLR 2020
Table 6: Datasets used in the anomaly detection task
Data
N
D
Anomaly (%)
Link
DDOS
464,976
66
3.75%
htp://w wtt:://ww.essmiin www.ws/iiig..g///cc.oiioooo
8index
Donors
619.326
5.92%
https///www wttp:///wwwwwglco.comk cow/c///d////o//ooe//oooooooo cup- http://kddcu--------eedddiiii-id -iep200---rediiiiinggeeeee exciteeent--t--oooeeeee
Backdoor
95,329
196
2.44%
hups://www, unsw,ad adfa. edu, wt/////wwcbbbbrr--ybbrreeeeeeeee
Ad
3,279
1,555
13.99%
/ttchv/ www. wei.edu'mii htw.////aattoiimmnnttaaaavttoote
Apascal
12,695
64
1.38%
http://isiin.cc uiuc. edu/attributes/
Bank
41,188
62
11 11.26%%
hups://arccivee ics uci hatem//aaaaattaaa-aaaaaaecee
Celeba
202.599
39
2.24%
http://mmlaal cuhk edu. wwt.ee//o.ojjeeee..
https //arcbive uci. htcpprooect//Celeb......
Census
299,285
500
6.20%
Creditcard
284,807
29
0.17%
hups://www. www.ke..co..gnggene. cow/mllleeee whw.hhheehccrrddruuudt
Lung
145
3,312
4.13%
/tarc/// uci eeiieedummi http:////aaatttssuunaua Laagassseettnnnnnnneeeeee
Probe
64,759
34
6.43%
hup://kddd.ee ies, eduudaaaaaasss/ wasaaaaaaaseedeeaeeeeee wwwpee//wwccuuuss//dduuugg....
R8
3,974
9,467
1.28%
hup://ssmining. orrgt files/Project, Dafasets/8
/wt////ttaiinalll----eoaoo train-all--erms.tt
Secom
1.567
590
6.63%
https /farchiv uci wwt./////aattssss/sscooom
U2R
60,821
34
0.37%
hup://kdd ies www.eeaaaaaeeeeekeeeeeeeee hwwp://wwww...c.ku.o////........
R8, 20news, Sector and RCVI are widely used text classification benchmark datasets. Olivetti is widely-used face recognition dataset.
Table 7: Datasets used in the clustering task
Data
N
D
#alasses
Link
R8
7,674
17,387
http://ssmining.oo org/tl hwws///rojeet..c.ccccecc Datasets/88 Bacrr___322 152/8 ferms.tKt
20news
18.846
130.107
20
https://cikki-lcarn.ooo 19daaasetsstwentt newsgroups.html
Olivetti
400
4,096
https://scikii org/0. 19/datasets/olivetii faces html
Sector
9.619
55.197
105
https://www.csip esi..ntu. htwp://wwwwgiiliiisttooooo///o/e www.aa/attttttttiiaaa.... htmilssctor
RCVI
20,242
47,236
https:/www.csiecc ntu. wtt.://www...........o//eteemma. htwprrryyyrrrvv wawlllecll..iimii. .iinary
C AUC-PR PERFORMANCE OF ABLATION STUDY IN ANOMALY DETECTION
The experimental results of AUC-PR performance of RDP and its variants in the anomaly detec- tion task are shown in Table 8 Similar to the results shown in Table 3 using the Lndp loss only, our proposed RDP model can achieve substantially better performance over its counterparts. By removing the Lrdp loss, the performance of RDP drops significantly in 11 out of 14 datasets. This demonstrates that the Lndp loss heavily harvested by our RDP model learn high-quality rep- resentations from random distances. Removing Lad from RDP also results in substantial loss of AUC-PR in many datasets. This indicates both the random distance prediction loss Trdp and the lask-dependent loss Lad are critical to RDP. The boosting process is also important, but is not as critical as the two losses. Consistent with the observations derived from Table distances calculated in non-linear and linear random mapping spaces are more effective supervisory sources than that in the original space.
D NMI PERFORMANCE OF ABLATION STUDY IN CLUSTERING
Table shows the NMI performance of RDP and its variants in the clustering task. is clear that our RDP model with the Teddp loss is able to achieve NMI performance that is comparably well to the full RDP model, which is consistent to the observations in Table 5 Without using the Lydp loss, the performance of the RDP model has some large drops nearly all the datasets. This reinforces the crucial importance of Lydp to RDP, which also justifies that using Lrdp alone RDP can learn expressive representations. Similar to the results in Table RDP is generally more reliable supervisory sources than Org SS and SRP SS in this set of results.
E SENSITIVITY W.R.T. THE DIMENSIONALITY OF REPRESENTATION SPACE
This section presents the performance RDP using different representation dimensions in its feature learning layer. The sensitivity test is performed for both anomaly detection and clustering tasks.
12

---

Under review as a conference paper at ICLR 2020
Table 8: AUC-PR performance of RDP and its variants in the anomaly detection task.
Decomposition
Supervision Signal
Data
RDP
RDP,
RDP,L. L2
RDP:
Boostingg
Org. SS
SRP .S
DDoS
0.301 = 0.028
0.110 1 0.015 0.364 = 0.013
0.114 = 0.001
0.30. = 0.007 0,380 1 0.030
Donors
0.432
0.201 +: 0.033 0.104 +
0.278
0.099 + 0.004
0.113 0.010
Backdoor
0.305 0.008
0,433 + 0.015 0.142 = 0.006
0.537 - 0.005
0.143 - 0.005
0.154 + 0.028
Ad
0,726 0.007
0.473 +: 0.009 0.491 + 0.014
0.488 + 0.008
0.419 + 0.015 0,530 + 0.007
Apascal
0.042
0.021 +: 0.005 0.031
0.002
0.028 0.003
0.016 0.003 0.035
0.007
Bank
0.364 0.013
0,258 + 0.006 0.266 = 0.018
0,278 = 0.007
0.262 = 0.016 0.265 + 0.021
Celebu
0.104 0.006
0.068 +: 0.010 0.060 + 0.004
0.072 + 0.008
0.050 + 0.009 0.065 + 0.010
Census
0.086
0.087
0.064
Crediteard
0.363 0.011
0.290 + 0.012 0.414 + 0.02
0.329 + 0.007
0.362 + 0.016 0.372 + 0.024
Lung
0.705 0.028
0.381 + 0.104 0.437 + 0.083
0.542 + 0.139
0,361 + 0.054 0.464 0.053
Probe
0.955 0.002
0.609 + 0.014 0.952 - 0.007
0.628 0.011
0,937 0,005 0.959
R8
0.146 0.017
0,134 +: 0.031 0.109
+ 0.006
0.173 + 0.028
0,067 + 0.016 0.134 + 0.019
Secom
0.096
0.001
0.086
+: 0.002 0.096
0.006
0.090
+ 0.001
0
+ 0.004 0.093 0.004
U2R
0.261
0.005
0,217 0.011
0.266
+ 0.007
0.238 + 0.009
0.187 + 0.013 0.239 0.023
Pwins/druws/ Josses (RDP vs.)
1370/1
11703
11703
12AW2
370/9
Table 9: NMI performance of RDP and its variants in the clustering task.
Decompostion
Supervision Signal
Data
RDP
RDP Lndo
RDP RDP,L
Org SS
SRP SS
0.539
0.040
0.471 0.043
0.037
0.567 0.021
0.033
20news
0.084 + 0.005
0.075
+ 0.006
0.081 : 0.002
0.075 + 0.002
0.074 : 0.003
Olivetti
0.805 + 0.012
0.782 - 0.010
0.784 = 0.010
0.795 : 0.011
0.787 1 0.011
Sector
0,305 + 0.007
0.253 0.010
0.007
0.295
+ 0.009
0.298 0.008
Rey1
0.165 0.000
0.146 + 0.010
0.168
0.000
0.154 + 0.002
0.147 0.000
E. SENSITIVITY TEST IN ANOMALY DETECTION
Figures 2 and respectively show the AUC-ROC and AUC-PR performance of RDP using different representation dimensions on all the 14 anomaly detection datasets used in this work. It is clear from both performance measures that RDP generally performs stably w.r.t. the use of different representation dimensions on diverse datasets. This demonstrates the general stability of our RDP method on different application domains. On the other hand, the flat trends also indicate that, as an unsupervised learning source, the random distance cannot provide sufficient supervision information to learn richer and more complex representations in a higher-dimensions, space. This also explains the performance on quite few datasets where the performance of RDP decreases when increasing the representation dimension. In general, the representation dimension 50 is recommended for RDP to achieve effective anomaly detection datasets from different domains.
E.2 SENSITIVITY TEST IN CLUSTERING
Figure presents the NMI and F-score performance of RDP-enabled K-means clustering using different representation dimensions all the five datasets in the clustering task. Similar to the sensitivity test results in the anomaly detection task, on all the five datasets, K-means clustering per- forms stably in the representation space resulted by RDP with different representation dimensions. The clustering performance may drop a bit when the representation dimension is relatively low, e.g., 512. Increasing the representation 1,280 may help RDP gain better representation power in some datasets but is not consistently better choice. Thus, the representation dimension 1,024 is gener- ally recommended for clustering. Recall that the required representation dimension clustering normally significantly higher than that in anomaly detection, because clustering generally requires significantly more information to perform well than anomaly detection.
F COMPUTATIONAL EFFICIENCY
The runtime of RDP is compared with its competing methods in both anomaly detection and clus- tering tasks. Since training time can vary significantly using different training strategies in deep beasn methods, it is difficult to have fair comparison of the training time. Moreover, the
13

---

Under review as a conference paper at ICLR 2020
Dous
0.95
Coleba
Census
0.85
Lung
Credi
AUC-ROC
Probe
0.8
RB
Sexom
U21
0.75
0.7
0.6
+
Representation Dimension
Figure 2: AUC-ROC results of RDP w.r.t. different representation dimensions on 14 datasets.
0.9
Back
Ad
Apancal
0.8
Bank
Celeta
Census
lung
0.6
PR
0.4
0.3
0.2
0.1
50
75
100
Representation
Figure 3: AUC-PR results of RDP w.r.t. different representation dimensions on 14 datasets.
models can often be trained offline. Thus, we focus on comparing the runtime the testing stage. All the runtime experiments below were done on computing server node equipped with 32 Intel Xeon E5-2680 CPUs (2.70GHz) and 128GB Random Access Memory.
F.1 TESTING RUNTIME ANOMALY DETECTION
The testing runtime seconds of RDP and its five competing anomaly detection methods on 14 anomaly detection datasets are provided in Table TO Since most of the methods integrate represen- tation learning and anomaly detection into a single framework, the runtime includes the execution time of feature learning and anomaly detection for all six methods. In general, most large datasets, RDP runs comparably fast to the most efficient methods iForest and RND, and is faster the two re- cently proposed deep methods REPEN and DAGMM. Particularly, RDP runs faster than REPEN and DAGMM by a factor of around five on nigh-dimensional and large-scale datasets like Donors and Census. RDP is slower than the competing methods in processing small datasets. This is mainly
14

---

Under review as a conference paper at ICLR 2020
Gliveti
0.7
Rivi
0.6
F-score Performance
0.5
0.5
0.4
0.3
0.3
0.2
0.1
512
708
1024
1200
512
708
1024
1280
Representation Dimension
Representation Dimension
Figure 4: NMI and F-score performance RDP-enabled K-means using different representation dimensions all the five datasets used in clustering.
because RDP has base runtime of its boosting process. Therefore, the runtime of RDP seems be almost constant across the datasets. This is very desired property for handling high-dimensional and large-scale datasets.
Table 10: Testing runtime (in seconds) on 14 anomaly detection datasets.
Data Characteristies
RDP and Its Five Competing Methods
Data
N
D
Foorett
AE
REPEN
DAGMM
RND
RDP
DDoS
464,976
66
54.06
86.86
172.47
197.85
31.86
28.93
Donors
619,326
10
28.17
$2.31
226.14
194.45
44.31
36.84
Backdoor
95,329
196
26.51
$1.66
36.43
187.61
12.26
29.95
Ad
3,279
1.555
6.71
14.71
3.24
31.54
8.12
30.83
Apascal
12,695
6.53
4.27
6.30
69.35
3.62
22.88
Bank
41,188
62
9.72
6.87
17.25
170.56
9.31
28.47
Celeba
202.599
39
20.54
26.70
71.60
223.70
18.05
33.91
Census
299,285
500
155.77
225.29
121.08
236.21
42.83
57.74
Creditcard
284,807
29
22.45
29.38
103.18
235.93
20.97
30.84
Lung
145
3,312
6.20
13.11
2.16
39.75
1.44
24.29
Probe
64,759
9.55
10.06
28.14
131.40
29.61
R8
3,974
9.467
59,70
45.48
7,81
31.99
8.26
14,33
Secom
1,567
590
7.32
5.78
2.83
18.22
3.23
22.52
U2R
60,821
34
8.95
9.38
26.55
185.88
9.90
28.10
F.2 TESTING RUNTIME IN CLUSTERING
Table 11 shows the testing runtime of RDP and its four competing methods in enabling clustering on five datasets. Since exactly the same K-means clustering is used on the features in all the five cases, we exclude the runtime of the K-means clustering for more straightforward comparison. The results show that RDP runs comparably fast to the very efficient methods SRP and AE since they do not involve complex computation at the testing stage; RDP runs about five orders of magnitude faster than HLLE since HLLE takes a huge amount of time in its nearest neighbours searching. Note that "Org' indicates the clustering performed on the original space, so it involves no feature learning and does not take any time.
Table 11: Testing runtime (in seconds) on five clustering datasets.
Data Characteristics
RDP and Its Four Competing Methods
Data
N
D
Org
HIL.E.
SRP
AE
RDP
R8
7,674
17.387
9,658.85
1.16
1.08
0.89
20news
18,846
130.107
94,349.20
2.26
11.49
Olivetti
400
4,096
166.02
0.73
0.03
0.03
Sector
9,619
55.197
24.477.80
1.40
4.28
2.87
RCVI
20,242
47.236
47,584.79
2.80
8.91
5.04
15

---

Under review as a conference paper at ICLR 2020
G COMPARISON TO -FATE-OF-TH--ART REPRESENTATION LEARNING METHODS FOR RAW TEXT AND IMAGE DATA
Since RDP relies on distance information as its supervisory signal, one interesting question is that, can RDP still work when the presented data is raw data in non- -on-Euclidean space, such as raw text and image data? One simple and straightforward way to enable RDP to handle those raw data is, as what we did on the text and image data used in the evaluation of clustering, to first convert the raw texts/images into feature vectors using commonly-used methods, e.g., TF-IDF (Aizawal 2003) for text data and treating each pixel as feature unit for image data, and then perform RDP these vector spaces. A further question is that, do we need RDP in handling those data since there are now large number of advanced representation learning methods that are specifically designed for raw text/image datasets? Or, how is the performance of RDP compared to those advanced representation learning methods for raw text/image datasets? This section provides some preliminary results in the clustering task for answering these questions.
G.1 ON RAw TEXT DATA
On the raw text datasets R8 and 20news, we first compare RDP with the advanced document rep- resentation method Doc2 as in (Le Mikolov) 2014]. Recall that, for RDP, we first use the bag-of-words model and document frequency information (e.g.. TF-IDF) to simply convert doc- uments into high-dimensional feature vectors and then perform RDP using the feature vectors. Doc2Vec leverages the idea of distributed representations to directly learn representations of doc- uments. We further derive a variant of RDP, namely Doc2Vec+RDP, which performs RDP on the Doc2Vec projected representation space rather than the bag-of-words vector space. All RDP, Doc2Vec and Doc2Vec+RRD project data onto a .024-dimensional space for the subsequent learn- ing tasks. Note that, for the method Doc2VecRRDP, to better examine the capability of RDP in exploiting the Doc2Vec projected space, we first use Doc2 Vec project raw text data onto higher- dimensional space (5,120 dimensions for R8 and 10,240 dimensions for 20news), and RDP further space.
lears dimesiimalaa space from this higher-dimension
The comparison results are shown in Table 12] Two interesting observations can be seen. First, RDP can significantly outperform Doc2 Vec on R8 or performs comparably well on 20news. This may be due to the fact that the local proximity information learned in RDP is critical to cluster- ing; although the word prediction approach in Doc2Vec helps learn semantic-rich representations for words/senteecces/paaagaaa since Doc2Vec is not like RDP that is designed to optimise this proximity information. Second, Doc2Vec+RDP can achieve substantially better performance than Doc2Vec, especially on the dataset 20news where Doc2VecRRDP achieves a NMI score of 0.198 while that of Doc2 is only 0.084. This may be because, as discussed in Section 3.3 RDP is equivalent to learn an optimised feature space out of its input space (Doe2Vec projected feature space in this case) using imperfect super- vision information. When there sufficient accurate supervision information, RDP can learn a substantially better feature space than its input space. This is also consistent with the results in Ta- ble in which clustering based on the RDP projected space also performs substantially better than that working in the original space 'Org'.
the pairwise document distances may be less effective than RDP
Table 12: NMI and F-score performance of K-means clustering using RDP, Doc2Vec, and Dec2Vec+RDP based feature representations of the text datasets R8 and news20.
Data Characteristics
NMI Performance
Data
N
D
Doc2Vec
RDP
RDc
R8
7,674
17,387
0.241 + 0.022 0.339 + 0.040 0.250 + 0.00
20news 18,846
130.107
0.080 + 0.003 0.084 + 0.005 0.198 + 0.009
Data Characteristics
F F-score Performance
Data
D
RDP 0.317 = 0.014 0.360 0.055 0.316 0.007 0.007
Doc2Vec
Doc2Vec+RDP
R8
7,674
17.387
20news 18,846 130,107
0.115 1 0.006 0.119 0.006 0.126 0.009
We use the implementation of Voc2Vec in a popular text mining python package gensim available http:///wwwactctccoooooommnnnnna
16

---

Under review as a conference paper at ICLR 2020
G.2 ON RAW IMAGE DATA
On the raw image dataset Oliverti, we compare RDP with the advanced representation learning method for raw images, RotNet (Gidaris et al. 2018). RDP uses each image pixel as feature unit and performs on 64 x 64 vector space. RotNet directly learns representations of images by predicting whether a given image is rotated or not. Similar to the experiments on raw text data, we also evaluate the performance of RDP working on the RotNet projected space, i.c., RotNet+RDP. All RDP, RotNet and RotNet+RDP first learn a 1,024 representation space, and then K-means is applied to the learned space to perform clustering. In the case of RotNet+RDP, the raw image data is first projected onto a -i4n-iiinni space, and then RDP is applied to this nigher-dimensional space learn .024-dimensional representation space.
We use the implementation of RotNet released by its authors" Note that the original RotNet is applied to large image datasets and has deep network architecture, involving four convolutional blocks with three convolutional layers for each block. We found directly using the original architec- ture is too deep for Olivetti and performs ineffectively as the data contains only 400 image samples. Therefore, we simplify the architecture of RotNet and derive four variants of RotNet, including RotNet1x2. RotNetax RotNetyx1 and RotNet2x1- Here RotNetaxe represents RotNet with a con- volutional blocks and convolutional layers for each block. Note that RotNet2x1 is the simplest variant we can derive that works effectively. We evaluate the original RotNet, its four variants and the combination of these five RotNets and RDP.
Table 13: NMI and F-score performance of K-means clustering using RDP, RotNet, and Rot- Re based feature representations of the image dataset Olivetti.
NMI Performance
F-score Performance
Org
0.778 = 0.014
0.590 = 0.029
RDP
0.805 + 0.012
0.638 1 0.026
RotNet
0.467 + 0.014
0.243 + 0.014
RoINet+RDP
0.472 1 0.011
0.242 1 0.011
RotNet422
0.318 = 0.010
0.281 = 0.014
2o
0.517 1: 0.010
0.282 - 0.014
RotNelxx1
0.519 0.010
0.283 0:014
RotNel +RDP
0,536 1 0.010
0.298
0.011
RotNetss1
0.526 1: 0.014
0.303 1 0.018
RotNetsw +RDP
0.567 + 0.010
0.336 = 0.015
RoiNelas1
0.561 1 0.010
0.339 1 0.016
RotNet2x +DDP
0.587 + 0.009
0.374 = 0.015
The evaluation results are presented in Table RotNet and all its four variants on Olivetti. is interesting that Org (i.e., performing K-means clustering on the original 64 x 64 vector space) also obtains a similar superiority over the RotNet family. This may be because Olivetti is too small to provide sufficient training samples for RotNet and its variants to learn its underlying semantic abstractions. This conjecture can also explain the increasing performance of RotNet variants with decreasing complexity of the RotNet architecture. Similar to the results the raw text data, applying RDP the RotNet projected spaces can also leam substantially more expressive representations than the representations yielded by RotNet and its variants, especially when the RotNet methods work well, such as the two cases: RotNetsx vs. RotNetyx1+RDD and RoiNetzx1 vs. RotNet2x1+RDD
13 Impressively, RDP can significantly outperform
H PERFORMANCE EVALUATION IN CLASSIFICATION
We also performed some preliminary evaluation of the learned representations in classification tasks using a feed-forwart lhyee-layer neural network model as the classifier. We used the same datasets as in the clustering task. Specifically, the representation learning model first outputs the new rep- resentations of the input data, and then the classifier performs classification the learned repre- sentations. RDP is compared with the same competing methods HLLE, SRP, AE and COP as in clustering. F-score used the performance evaluation metric here.
*The released code of RotNet available http://gghaaa.ooooddddiiraeeeeee
17

---

Under review as a conference paper at ICLR 2020
The results shown in Table 14 Similar to the performance in clustering and anomaly detection, our model using only the random distance prediction loss Lrdp- i.c., RDP Lelu performs very favourably and stably on all the five datasets. The incorporation of a into the model, i.c., RDP, helps gain some extra performance improvement datasets like 20news, but it may also slightly downgrade the performance on other datasets. An extra hyperparameter may be added to control the importance of these two losses.
Table 14: F-score performance of classification five real-world datasets.
Data
HILLE
SRP
AE
COP
RDP: 1//en
RDP
R8
0.246
0.895
0.874
0.860
0.900
0.906
20mews
0.005
0.733
0.709
0.718
0.735
0.753
Olivetti
0.895
0.899
0.820
0.828
0.900
0.896
Sector
0.037
0.671
0.645
0.689
0.690
0.696
RCV1
0.766
0.919
0,918
N/A
0,940
0,926
18


Reviews:
[{"review_id": "rkxisntMor", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n###### Overall Recommendation\nI vote for the “Weak Accept” decision for this paper. \n\n### Summary\nThis paper introduces a novel model termed Random Distance Prediction model, it can predict data distances in randomly projected space. The distance in projected space is used as the supervisory signal to learn representations without any manually labeled data, avoiding the concentration and inefficiency problem when dealing with high-dimensional tabular data. Their main contribution is extending the random distance in projected spaces to approximate the original distance information of the hight-dimensional tabular data effectively. Overall, the idea in this paper is interesting and effective, the experiment results in two typical unsupervised tasks (anomaly detection and clustering) also look very promising. However, the writing sometimes has unclear descriptions, given these clarifications in an author's response, I would be willing to increase the score.\n\n### Strengths\n1. The illustration of the authors' idea is clear and concise.\n2. The theoretical analysis of the proposed method is solid and systematic, the validity of the subparts have been proven previously.\n3. The experiment part is well organized. The RDP model are compared with several state-of-the-art unsupervised learning methods in 19 real-world datasets of various domains. The experimental setup is solid with realistic considerations, the results are very convincing and promising.\n4. This paper provides sufficient detail for reproducing the results.\n\n### Weaknesses\nLack of systematic description of the authors’ major contribution. The proposed model looks more like a combination of previous conclusions, which makes readers feel the core parts of this paper build heavily on previous work.\n\n### Questions\n1. What is the relation between “random distance prediction loss” and “task-dependent auxiliary loss”?\n2. Are there any solutions and references about how to choose the task-dependent loss L_aux? \n3. Why you shade the second part of the loss function in Figure 1; \n4. How long is it for training the proposed model and getting the experiment results? Does the RDP model still outperform the other algorithms?\n\n### Suggestions to improve the paper\n1.  It would be better to reorganize Section 1 and Section 2, please describe the contribution in a more systematic way.\n2. Add details for the architecture of the model, please give more descriptions about Figure 1.\n3. It might also help to add an algorithm comparison box for the test time for the proposed method.\n\n### Minor Edit Suggestions\n1. It would be better to give more descriptions about Figure 1; the lower right part in Figure 1 is not explained in the caption; the shadow part in Figure 1 is not precise.\n2. Figure 1 was bad organized, please make the legend readable size. \n3. I don't think there exist the proofs of Eqns. (2)-(4) in the reference paper (Vempala, 1998), which was written in Page 4. The number should be revised.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyeWNBsjFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a method of unsupervised representation by transforming a set of data points into another space while maintaining the pairwise distance as good as possible. The paper is well structured with background literatures, formula, as well as experiments to show the advantage of the proposed method. I find it generally interesting, with the following major concerns.\n\n1. Representation or dimension reduction? If the original space is a structured space like Euclidean space, then effectively this paper's method coincides with regular distance preserving method in dimension reduction, and Johnson-Lindenstrauss theories. If the original space is not structured or doesn't naturally have a good distance measure, then the proposed method cannot work. For example, if the original dataset is a set of documents, and the task is to do representation learning to convert each document into a compact vector. However, there's no good distance metric for the document space. If TF-IDF is used, then the representation space also inherits TF-IDF type features which is not desired. If more advanced similarity is used for the document space, then the role of representation learning is not essential anymore as that similarity measure can already help the downstream tasks.\n\n2. Section 3 the theoretical analysis. This part seems like a collection of previous works and contains minimal information about the proposed method.\n\n3. Some writing issues, like page 4 line 7 about the equation numbering.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJxM3ExoFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors discuss the a novel technique, called Random Distance Prediction, to learn rich features from domains where massive data are hard to produce; in particular, they focus on the two tasks of anomaly detection and clustering.\nThe paper is well written and understandable by a non-specialistic audience; introduction and references are adequate, and the theoretical analysis is reasonably explained, although the two optional losses should have been discussed more deeply.\nResults are fairly supporting the authors' claim: however, the improvement in performances w.r.t. alternative approaches are limited in most cases, and the contribute of the optional losses is somehow unclear and inconsistent across different datasets. It would be interesting also to check how stable is the method (i.e. the losses) for different underlying network architectures, and also how RDP compares to very basic approaches employing dimensionality reduction algorithms such as t-SNE or UMAP.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that this is an interesting paper but it required major modifications. After rebuttal, thee paper is much improved but unfortunately not above the bar yet. We encourage the authors to iterate on this work again.
Decision: Reject