user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality

Paper content:
Deep Neural Networks (DNNs) have recently been shown to be vulnerable against adversarial examples, which are carefully crafted instances that can mislead DNNs to make errors during prediction.
To better understand such attacks, a characterization is needed of the properties of regions (the so-called `adversarial subspaces') in which adversarial examples lie.
We tackle this challenge by characterizing the dimensional properties of adversarial regions, via the use of Local Intrinsic Dimensionality (LID).
LID assesses the space-filling capability of the region surrounding a reference example, based on the distance distribution of the example to its neighbors.
We first provide explanations about how adversarial perturbation can affect the LID characteristic of adversarial regions, and then show empirically that LID characteristics can facilitate the distinction of adversarial examples generated using state-of-the-art attacks.
As a proof-of-concept, we show that a potential application of LID is to distinguish adversarial examples, and the preliminary results show that it can outperform several state-of-the-art detection measures by large margins for five attack strategies considered in this paper across three benchmark datasets.
Our analysis of the LID characteristic for adversarial regions not only motivates new directions of effective adversarial defense, but also opens up more challenges for developing new attacks to better understand the vulnerabilities of DNNs.
Deep Neural Networks (DNNs) are highly expressive models that have achieved state-of-the-art performance on a wide range of complex problems, such as speech recognition and image classification BID18 .
However, recent studies have found that DNNs can be compromised by adversarial examples (Szegedy et al., 2013; BID8 BID27 .
These intentionally-perturbed inputs can induce the network to make incorrect predictions at test time with high confidence, even when the examples are generated using different networks BID24 BID3 BID29 .
The amount of perturbation required is often small, and (in the case of images) imperceptible to human observers.
This undesirable property of deep networks has become a major security concern in real-world applications of DNNs, such as self-driving cars and identity recognition BID5 BID34 .
In this paper, we aim to further understand adversarial attacks by characterizing the regions within which adversarial examples reside.Each adversarial example can be regarded as being surrounded by a connected region of the domain (the 'adversarial region' or 'adversarial subspace') within which all points subvert the classifier in a similar way.
Adversarial regions can be defined not only in the input space, but also with respect to the activation space of different DNN layers (Szegedy et al., 2013) .
Developing an understanding of the properties of adversarial regions is a key requirement for adversarial defense.
Under the assumption that data can be modeled in terms of collections of manifolds, several works have attempted to characterize the properties of adversarial subspaces, but no definitive method yet exists which can reliably discriminate adversarial regions from those in which normal data can be found.
Szegedy et al. (2013) argued that adversarial subspaces are low probability regions (not naturally occurring) that are densely scattered in the high dimensional representation space of DNNs.
However, a linear formulation argues that adversarial subspaces span a contiguous multidimensional space, rather than being scattered randomly in small pockets BID8 Warde-Farley et al., 2016) .
Tanay & Griffin (2016) further emphasize that adversarial subspaces lie close to (but not on) the data submanifold.
Similarly, it has also been found that the boundaries of adversarial subspaces are close to legitimate data points in adversarial directions, and that the higher the number of orthogonal adversarial directions of these subspaces, the more transferable they are to other models (Tramèr et al., 2017) .
To summarize, with respect to the manifold model of data, the known properties of adversarial subspaces are: (1) they are of low probability, (2) they span a contiguous multidimensional space, (3) they lie off (but are close to) the data submanifold, and (4) they have class distributions that differ from that of their closest data submanifold.Among adversarial defense/detection techniques, Kernel Density (KD) estimation has been proposed as a measure to identify adversarial subspaces BID7 .
BID2 demonstrated the usefulness of KD-based detection, taking advantage of the low probability density generally associated with adversarial subspaces.
However, in this paper we will show that kernel density is not effective for the detection of some forms of attack.
In addition to kernel density, there are other density-based measures, such as the number of nearest neighbors within a fixed distance, and the mean distance to the k nearest neighbors (k-mean distance).
Again, these measures have limitations for the characterization of local adversarial regions.
For example, in FIG0 the three density measures fail to differentiate an adversarial example (red star) from a normal example (black cross), as the two examples are locally surrounded by the same number of neighbors FORMULA8 , and have the same k-mean distance (KM=0.19) and kernel density (KD=0.92).As
an alternative to density measures, FIG0 leads us to consider expansion-based measures of intrinsic dimensionality as a potentially effective method of characterizing adversarial examples. Expansion
models of dimensionality assess the local dimensional structure of the data -such models have been successfully employed in a wide range of applications, such as manifold learning, dimension reduction, similarity search and anomaly detection BID0 BID13 . Although
earlier expansion models characterize intrinsic dimensionality as a property of data sets, the Local Intrinsic Dimensionality (LID) fully generalizes this concept to the local distance distribution from a reference point to its neighbors BID13 BID7 -the dimensionality of the local data submanifold in the vicinity of the reference point is revealed by the growth characteristics of the cumulative distribution function. In this
paper, we use LID to characterize the intrinsic dimensionality of adversarial regions, and attempt to test how well the estimates of LID can be used to distinguish adversarial examples. Note that
the main goal of LID is to characterize properties of adversarial examples, instead of being applied as a pure defense method, which requires stronger assumptions on the current threat model. In FIG0 ,
the estimated LID of the adversarial example (LID ≈ 4.36) is much higher than that of the referenced normal data sample (LID ≈ 1.53), illustrating that the estimated LID can efficiently capture the intrinsic dimensional properties of adversarial regions. In this paper
, we aim to study the LID properties of adversarial examples generated using state-of-the-art attack methods. In particular
, our contributions are:• We propose LID for the characterization of adversarial regions of deep networks. We discuss how
adversarial perturbation can affect the LID characteristics of an adversarial region, and empirically show that the characteristics of test examples can be estimated effectively using a minibatch of training data.• Our study reveals
that the estimated LID of adversarial examples considered in this paper 1 is significantly higher than that of normal data examples, and that this difference becomes more pronounced in deeper layers of DNNs.• We empirically demonstrate
that the LID characteristics of adversarial examples generated using five state-of-the-art attack methods can be easily discriminated from those of normal examples, and provide a baseline classifier with features based on LID estimates that generally outperforms several existing detection measures on five attacks across three benchmark datasets. Though the adversarial examples
considered here are not guaranteed to be the strongest with careful parameter tuning, these preliminary results firmly demonstrate the usefulness of LID measurement.• We show that the adversarial regions
generated by different attacks share similar dimensional properties, in that LID characteristics of a simple attack can potentially be used to detect other more complex attacks. We also show that a naive LID-based detector
is robust to the normal low confidence Optimization-based attack of BID2 .
In this paper, we have addressed the challenge of understanding the properties of adversarial regions, particularly with a view to detecting adversarial examples.
We characterized the dimensional properties of adversarial regions via the use of Local Intrinsic Dimensionality (LID), and showed how these could be used as features in an adversarial example detection process.
Our empirical results suggest that LID is a highly promising measure for the characterization of adversarial examples, one that can be used to deliver state-of-the-art discrimination performance.
From a theoretical perspective, we have provided an initial intuition as to how LID is an effective method for characterizing adversarial attack, one which complements the recent theoretical analysis showing how increases in LID effectively diminish the amount of perturbation required to move a normal example into an adversarial region (with respect to 1-NN classification) BID1 .
Further investigation in this direction may lead to new techniques for both adversarial attack and defense.In the learning process, the activation values at each layer of the LID-based detector can be regarded as a transformation of the input to a space in which the LID values have themselves been transformed.
A full understanding of LID characteristics should take into account the effect of DNN transformations on these characteristics.
This is a challenging question, since it requires a better understanding of the DNN learning processes themselves.
One possible avenue for future research may be to model the dimensional characteristics of the DNN itself, and to empirically verify how they influence the robustness of DNNs to adversarial attacks.Another open issue for future research is the empirical investigation of the effect of LID estimation quality on the performance of adversarial detection.
As evidenced by the improvement in perfor-mance observed when increasing the minibatch size from 100 to 1000 ( Figure 5 in Appendix A.3), it stands to reason that improvements in estimator quality or sampling strategies could both be beneficial in practice.
FIG3 illustrates LID characteristics of the most effective attack strategy known to date, Opt, on the MNIST and SVHN datasets.
On both datasets, the LID scores of adversarial examples are significantly higher than those of normal or noisy examples.
In the right-hand plot, the LID scores of normal examples and its noisy counterparts appear superimposed due to their similarities.
Figure 5 shows the discrimination power (detection AUC) of LID characteristics estimated using two different minibatch sizes: the default setting of 100, and a larger size of 1000.
The horizontal axis represents different choices of the neighborhood size k, from 10% to 90% percent to the batch size.
We note that the peak AUC is higher for the larger minibatch size.

assistant: text: We characterize the dimensional properties of adversarial subspaces in the neighborhood of adversarial examples via the use of Local Intrinsic Dimensionality (LID).