Manifold-aware Training: Increase Adversarial Robustness with Feature Clustering
Authors: Ting-An Yen
Chun-Shien Lu
Pau-Choo Chung
Venue:  (2021)
Keywords: Adversarial Attacks
Adversarial Defense
Robustness
Convolutional Neural Network
Feature Compactness

Abstract: The problem of defending against adversarial attacks has attracted increasing attention in recent years. While various types of defense methods ($\textit{e.g.}$, adversarial training, detection and rejection, and recovery) were proven empirically to bring robustness to the network, their weakness was shown by later works. Inspired by the observation from the distribution properties of the features extracted by the CNNs in the feature space and their link to robustness, this work designs a novel training process called Manifold-Aware Training (MAT), which forces CNNs to learn compact features to increase robustness.  The effectiveness of the proposed method is evaluated via comparisons with existing defense mechanisms, $\textit{i.e.}$, the TRADES algorithm, which has been recognized as a representative state-of-the-art technology, and the MMC method, which also aims to learn compact features. Further verification is also conducted using the attack adaptive to our method. Experimental results show that MAT-trained CNNs exhibit significantly higher performance than state-of-the-art robustness.

Under review as a conference paper at ICLR 2021
MANIFOLD-AWARE TRAINING: INCREASE ADVER- SARIAL ROBUSTNESS WITH FEATURE CLUSTERING
Anonymous authors Paper under double-blind review
ABSTRACT
The problem of defending against adversarial attacks has attracted increasing at- tention in recent years, While various types of defense methods (e.g. adversarial training, detection and rejection, and recovery) were proven empirically to bring robustness to the network, their weakness was shown by later works. Inspired by the observation from the distribution properties of the features extracted by the CNNs in the feature space and their link to robustness, this work designs novel training process called Manifold Aware Training (MAT), which forces CNNs to learn compact features to increase robustness. The effectiveness of the proposed method is evaluated via comparisons with existing defense mechanisms, i.e., the representative state-of-the
TRADES algorithm, which has been recognized art technology, and the MMC method, which also aims to learn compact features. Further verification is also conducted using the attack adaptive our method. Experimental results show that MAT-trained CNNs exhibit significantly higher performance than state-of-the- robustness.
INTRODUCTION
1.1 BACKGROUND
Convolutional neural networks (CNNs) are increasingly used in recent years due to their high adap- tivity and flexibility. However. Szegedy et al. (2014) discovered that by maximizing the loss of CNN model w.r.t the input data, one can find a small and imperceptible perturbation which causes misclassification errors of the CNN. The proposed method for constructing such perturbations was designated as the Fast Gradient Sign Method (FGSM), while the corrupted data (with perturbation) were referred to as adversarial examples. Since that time, many algorithms for constructing such perturbations have been proposed, where these algorithms are referred to generally as adversarial attack methods (e.g.. MMdry et al. (2018): Carlini & Wagner (2017): Rony et al. (2019); [rendel et al. (2018); Chen et al. (2017); Alzantot al. (2019); Ru et al. (2020); Al-Dujii & O Reilly (2020)). ). Among them, Projected Gradient Descent (PGD) (Kurakin et al. 2017) and Carlini & Wagner (C&W) W) (Carlini & Wagner 2017) attacks are the most widely-used methods. Specifically, PGD is a multi-step variant of FGSM, which exhibits higher attack success rate, and C&W attack leverages an objective function designed jointly minimize the perturbation norm and likelihood of the input being correctly classified. The existence of adversarial examples implies an underly- ing inconsistency between the decision-making processes of CNN models and humans, and can be catastrophic in life-and-dedat death applications, such as automated vehicles or medical diagnosis systems, in which unpredictable noise may cause the CNN to misclassify the inputs.
Various countermeasures for thwarting adversarial attacks, known as adversarial defenses, have been proposed. One of the most common forms in adversarial defense to augment the training dataset with adversarial examples so as to increase the generalization ability of the CNN toward these pat-
terns. Such a technique is known as adversarial training (Shaham et al., 2015 Zhang et al.) 2019] Wang et al. 2020). However, while such methods can achieve state-of-the-an art robustness in terms of robust accuracy (i.e., the accuracy of adversarial examples), training robust classifiers is non-trivin voivialiial task. For example, Nakkiran (2019) showed that significantly higher model capacity is required for robust training. while (Schmidt et al. (2018) proved that robust training requires a significantly larger number of data instances than natural training.

---

Under review as a conference paper at ICLR 2021
1.2 MOTIVATION AND CONTRIBUTIONS
The study commences by observing the distribution properties of the features learned by existing training methods order to obtain a better understanding the adversarial example problem. In particular, the t-SNE dimension reduction method (van der Maaten & Hinton 2008) is used to visu- alize the extracted features, as illustrated in Figures The observation results reveal the following feature distribution properties:
Non-clustering: same-class features are not always clustered together (i.e., some points leave the clusters of their respective colors in Figure I. which is at odds with intuition, which expects that the representative (for classification) features of same-class samples should be similar to one another.
ceafusiig-distancee closeness between samples in the feature space does not imply resem- blance of their prediction (especially for adversarial examples, as there many triangles colored differently than surrounding points in Figure (b)).
72
25-
-75
-75
Figure 1: Features learned by models trained for MNIST using TRADES (Zangg et al. 2019) train- ing, where the clean samples (points) are colored by predicted classes and the adversarial examples (triangles) are colored by (a) true labels or (b) predicted classes.
We confirm the validity of these observations through a numerical analysis of the matching rate be- tween the closest cluster dominant class and the predicted class. Clustering analysis algorithms (e.g. Ward's Hierarchical Clustering algorithm (Ward 1963)) are leveraged to find the clusters formed by the CNN-leamed features. Additionally, the match rate defined as Eg dom((()))= where ((*) is the closest cluster to (C(#) arg ming dist(C, x)) and dom(C) evaluates the dominant class of a cluster (dom( (C) = arg max, m m(C), where m(C) E N produces a cluster mapping vec- tor describing the number of members of each class prediction). Table [ summarizes the analysis results and confirms that both properties exist. Intuitively, a good feature extractor for classification purpose should produce similar features for all samples within the same class.
On the other hand, according to Tang et al. (2019), the existence of adversarial examples results from mismatch between features used by human and those used by CNNs. Therefore, one intuitive ap- proach for increasing CNN robustness is simply to drive CNN-learned features toward human-used features. However, is impossible understand and predict human-used features with any absolute degree of certainty. Thus, an alternative approach is to force the CNN-leamed features to have some expected properties that human-used features should also have. For example, as mentioned above, features for the classification of objects belonging to the same class should be similar to one another.
Based on the above observations, the present study proposes a novel training process, designated as Manifold-Aware Training (MAT), for learning features which are both representative and compact. The experimental results confirm that models trained with MAT exhibit significantly higher robust- ness than existing state-of-the-art models. It would be clear later that our idea is, in some sense, similar to Pang et al. (2020) in that the authors proposed the Max-Maaalanobis Center (MMC) loss. which minimizes the distance of features to their assigned preset class centers. By showing that robustness of the model using MMC with adversarial training is higher than that using simply adversarial training, the authors claimed that high feature compactness results in locally sufficient
2

---

Under review as a conference paper at ICLR 2021
Table I: Match rates of clean data and adversarial examples, which generated using PGD and Decoupled Direction and Norm (DDN) attack (Rony et al. 2019), in naturally trained models (-N) and TRADES-trained models (-T).
Data Type
MNIST-N
CIFARIO-N
MNIST-T CIFARTO-T
Clean
99.61%
99.31%
99.67%
91.65%
PGD 46.84%
74.44%
98,43%
77.42%
DDN
27.66%
51.53%
63.64%
56.96%
samples, which are beneficial for robust training according to Nakkiran (2019). Their conclusion of feature compactness helps robustness matches our idea.
The main contributions of our study can be summarized as follows:
A better understanding of the relationship between robustness and the distance between adversarial examples and clean samples in the MAT-leamndd feature space.
Our method improves the state-of-the-ee performance from 57% to 80% for CIFARIO and from 96% to 99% for MNIST in terms of robust accuracy.
The main notations used in present study are summarized Appendix
RELATED WORKS
One family of adversarial defense methods, referred to as adderssaiial-taiinng--bar methods, aug- ments the training data with adversarial examples. For example, ZZhng et al. (2019) defined the so-called TRADES loss, which is based on the trade-off between the clean accuracy (i.e., the accu- racy of clean images) and the robustness accuracy and has the form
1 ( N
min
max
where B. (x())) denotes the lp boundedd ball (z) fx' || =I1p e)). and ;(,:;)) and J'(,, 0) are implemented using the cross-entropy loss and the Kullback-Leible divergence crite- rion, respectively. Ding al. (2000) proposed the Max Margin Adversarial (MMA) training method, which minimizes the cross-entropy loss for misclassified samples and maximizes the margin be- tween them and the nearest decision boundaries for correctly classified samples. The authors proved that by minimizing the cross- entropy loss of the samples on the decision boundaries, the margin can be maximized. Wang et al. (2020) found that misclassified samples bring more robustness when used in adversarial training than correctly classified ones. Inspired by this observation, the authors proposed an objective function that encouraged stability around misclassified samples and optimized the classification loss for adversarial examples.
based methods achieve state-of-the-art robust accuracy, they inherit some limitations (Nakkiran] 2019; Schhmidt al. 2018). Some studies try to alleviate these lim- itations to further improve the CNN robustness. For examples, according to the work of Schmidt et al.) (2018), robust training requires significantly more samples than natural training. However, high sample density may make the training samples locally sufficient for robust training. Therefore, Pang et al. (2020) proposed the Max- Max-Mahalanobis Center (MMC) loss follows:
While the adversarial-trainee
LmMc(6(#). 9) [l(()) (
where (b-) is CNN feature extractor, and S denotes the pre-set class center for class y. Addition- ally, the authors analyzed the limitations of traditional softmax and cross-entropy (SCE) loss based training methods regarding the induced sample density in the feature space. The authors then proved that the feature density induced by the MMC loss is guaranteed to be high around the centers (μ (o)).
To our knowledge, however, improving the robustness of CNNs by manipulating the feature space attracts little attention. In view of this, propose novel defense method by training CNNs with
3

---

Under review as a conference paper at ICLR 2021
loss functions designed to induce feature compactness and improve the ofate-of-the-ar robustness significantly.
3 PROPOSED METHOD
The proposed Manifold-Awel Aware Training (MAT) algorithm is introduced in Subsection 3.1 and two additional loss functions further improve the performance of MAT are described in Subsection 3.2 Moreover, a transformation technique designed to architecturally transform MAT-trained CNNs into traditional CNNs for compatibility with existing techniques is presented in Subsection 3.3
3.1 MANIFOLD-AWARE TRAINING (MAT)
Recall the observed properties of features learned by traditional CNNs, i.e., the non-clustering prop- erty and the confusing-distance property (see Section DT This subsection introduces proposed loss function to achieve intra-class feature compactness and inter-class feature dispersion. Specifi- cally, the loss function minimizes the distance of CNN output features to their corresponding class centers while the class centers are far from one another. According to to|Pang et al. (2020), fixing class centers during the training phase makes better performance than updating them while learning for another objective (e.g., intra-class feature compactness) since the CNN can focus on one objective rather than seeking trade-off between multiple objectives. To ensure the pre-defined class centers are far from one another (to ensure inter-class dispersion), the distance between the samples is mea- sured using the cosine-distance (CD(a, b) and the class centers are generated using the algorithm proposed by Pang et al. (2018). i.e., the Max Max-Mahalanobis Distribution (MMD) centers. The generated center vectors are the vertices of an L-dimensional simplex such that the included angles between class center vectors are maximized,
cos(a, b). where cos b) (a b)/ ( (
ie.,
(1 (1)) arg min max (x))) (1()), 1 (0 CV1 < iデj
Note that the details of the generation algorithm and examples of MMD centers are provided in Appendix B Accordingly, the proposed loss function for intra-class compactness, i.e., Feature To Center (FTC) loss, is defined as
LFrC(#, v) CD()(#), 1 (1)).
(4)
In addition, in the inference phase, an input images æ is predicted to belong to class ú as:
ÿ = f(x) = arg min CD()()), (1)) arg max cos cos(p((r), (k)). ISKSL
(5)
1SkLL
3.2 AUXILIARY LOSS FUNCTIONS TO ENHANCE ROBUSTNESS
This subsection introduces two auxiliary loss functions which are proposed to further improve the robustness of MAT.
3.2.1 SECOND-ORDER (SO) Loss
Inspired by the approach proposed by Yan al. (2018). this sub-section proposes an objective func- tion which minimizes the magnitude of the gradient of the classification loss and thus improves the stability of the classification results. Generally speaking, backpropagation for gradient value for an arbitrary CNN is difficult and expensive to compute. However, since the computation operations of the FTC loss after the feature layer are fixed, the gradient w.r.t the features can be computed simply by applying fixed series of operations through forwarding. Note that rather than using ordinary classification loss (e.g., the cross-entropy loss), the FTC loss is used when applying this technique to MAT since the stability of the FTC loss implies both the stability of the output features and the classification performance. Therefore, the magnitude of the gradient of the FTC loss w.r.t the features can be formulated the following Second-order (SO) loss, i.e..
Lso(x, y) = V LrTc B = | ||
1 ((())
cos(Ø, (1)
Φ
11
(6)
1011
4

---

Under review as a conference paper at ICLR 2021
The corresponding derivation is provided below
V LFTC = V [1 - cos( (0. [ = - V cos Φ. de (v))
Φ
Φ pe())
V
|
(v)) ll2
| 2
|
1
10
Φ (v)
10b12
1))
(os(o,
11
1
| l2 |u
|
cos( Φ 1 (v)
|| |1012
110112
3.2.2 BOUNDED- ROUNDED-INPUT-BBUUEDDD-UUUUTT (BIBO) Loss
Conceptually, the adversarial example problem is a consequence of the local instability of CNNs. Specifically, the output of CNN may change significantly when small perturbation is added to the input. Intuitively, therefore, if the local instability is alleviated, the adversarial robustness should be correspondingly improved. This subsection thus proposes Output (BIBO) loss to minimize the difference in the feature space between samples with small perturbations and clean sample, i.e..
CBIBO () max CD(6(=).
x'EB,(æ)
Specifically, the maximization practically achieved using PGD optimizationm, which was also adopted in the second term of the TRADES loss (Eq. IT.
3.2.3 TOTAL LOSS OF MAT
The total loss of MAT is defined by:
LMAT = LFTC + a Lso + B LBIBO-
The SO loss ensures the local stability of the feature distance to class center (LFTC) w.r.t the features while the BIBO loss ensures the local stability of the features w.r.t the input image. Therefore, the total loss stablizes the feature distance to class center when the input is subject to small perturbations.
3.3 MAT TR MAATOOOOAAIIONN
The MAT-trained models described above produce features rather than class scores (logits), as for traditional CNNs, and this difference may make the former incompatible with existing theories or techniques designed for traditional CNNs. For example. the objective functions of most adversarial attacks are based on logits, which MAT-trained CNNs do not produce. However, the MAT models can be transformed to be architecturally identical to traditional CNNs by using the cosine similarities each class center as the logits. Additionally, since the 1 no-ormms of the center vectors are the same, the computation of the scores can be simplified to one of computing inner-products, as proved below:
Proof. Let (9) and (1) be center vectors of two different classes. We can derive:
cos(E) ())))
ES cos(12, (1)
(10)
Φ (0)
Φ ( _ Φ (0)
(11)
2
2
Note that the simplified logit computation can be achieved by adding a linear layer to the back end of the CNN with the weights set as the class centers and biases set as zero (h W W (1(1)) fe(2) ..-. (L.)T where h are the logits.)
5

---

Under review as a conference paper at ICLR 2021
Robustness comparison. For Clean and PGD, the accuracy is reported. For C&W and DDN, the average perturbation norm of adversarial examples is reported.
Table 2:
CIFARIO MNIST
Defense
SO
BIBO
Clean
PGD
C&W
DDN
Natural
95.76% 99.54%
00.03% / 10.19%
0.432/ /0.677
0.134/0.33
AT
78.49% 99.30%
45,93% / 88.88.2 88.26%
1.054 2.545
TRADES
O
84.92% / 99.34%
56.61% % / 94.38%
0.722 00.442
0.823 / 2.852
X
X
94.93% /99.64%
03.45% / 61.62%
0.488 0.389
0.235 / 1.305
XXXX
85.04% 999.34%
53.58% 99.30%
1.234 2.313
0,924 6.103
MAT
DOWD
X
95.21% 999.68%
79.92% / 82.06%
0.576 0.418
0.595 1.570
O
83.80% 99.32%%
80.55% 199.8%%
2.457 2.063
2.750 4.723
4 EXPERIMENTAL RESULTS
This section evaluates the robustness of the proposed MAT algorithm. First, the assumptions made in the threat model are described in Subsection[41] In (. the defense capability of our method is evaluated and compared with state-of-the-art, known TRADES (Zhang al.) 2019) and MMC (Pang et al., 20200), respectively, along with verification against adaptive attacks in Subsection 4.4
Finally, we examine the effectiveness of feature compactness in Subsection 4.5
4.1 THREAT MODEL
According to Carlini et al. (2019), defining threat models essential to achieve fair and reliable evaluation and comparison. Since the motivation of our work is to test the worst-case robustnes, the goal of the adversary is assumed to be that of causing misclassification errors, iee, non-targeted adversaries. In pursuing this goal, the adversary is further assumed to have full knowledge of the victim model, including the architecture, parameters, and datasets used. In other words, white-box attacks were considered in this paper, where the perturbation of bounded attacks is constrained to have bounded nn-norm while the mean 1p-norm of perturbation rather than robust accuracy is adopted for evaluation against unbounded attacks.
4.2 ROBUSTNESS OF MAT-TRAINED CNNS
This subsection compares the robustness of our proposed MAT with that of existing mechanisms. TRADES (Zhang al.) 2019) was selected as the state-of-the-an art method because the authors leased their code and trained models, which makes it casy to reproduce the experiments, and we note that both MMA (Ding al. 2020) and MART (Wang al., 2020) only yield comparable performance with TRADES. The basic adversarial training (AT) approach, which minimizes the cross-entropy loss for adversarial examples rather than clean samples, was chosen as a baseline. Furthermore, to perform an ablation study, all variants of MAT with various combination of loss functions empirically compared explore their capabilities in defending against attacks. The experiments use two datasets, namely MNIST and CIFAR10 10 (as in Zhang et al. (2019)), associ- ated with VGG Net (Simonyan & Zisserman 2015) and wide ResNet (Zagoruyko & Komodakis 2016), respectively. A library called fool box (Rauber et al.) 2017) was utilized to construct the adversarial examples. Three common white-box attack methods were used for evaluation purpose: PGD (Madry et al. 2018), C&W (Carlini & Wagner 2017), and DDN (Rony et al. 2019). where C&W and DDN belong to unbounded attacks. The parameters with respect to these attacks are shown in Appendix C Before applying adversarial attacks to MAT-trained models, the models are transformed into traditional architecture using the approach described in Subsection 3.3] The robust accuracy under different attacks and the clean accuracy (the accuracy of clean test data) are shown in Table 2 We can observe that the MAT-trained CNN (with all loss functions applied) exhibit significantly higher robustness than TRADEST
'The second term in Eq. 1 BIBO-like loss, where the distance is measured by Bullhack-Leiiler diver- gence rather than cosine-distance.

---

Under review as a conference paper at ICLR 2021
Table 3: Classification accuracy (%) comparison with MMC (Pang et al. 2020) for CIFARIO. The superscripts un and tar denote untargeted and targeted PGD attacks, respectively, and the subscripts of PGD denote the number of iterations for conducting such attacks. Note that S denotes the robust accuracy which is under 1 %.
8
/55
< 16
255
Defense
Clean
PGD in
PGD
PGD
PGD 50
PGD
PGD in
PGD 30
PGD
50
Natural
92.9
3.7
< [
3.6
<
2.9
2.6
AT
80.9
69.8
55.4
69.4
53.9
53.3
34.1
38.5
21.5
MMC
81.8
70.8
56.3
70.
55.0
54.7
37.4
39.9
27.7
MAT
83.8
80.1
80.4
76.1
80.4
79.0
79.9
60.4
79.8
Furthermore, as Carlini et al. (019)) stated, for bounded attacks, the accuraey-errsu-pprtubbaaiio strength curve more indicative of the robustness than the accuracy of the single perturbation strength. Therefore, the related curves are presented in Figure 2 (Note that the detailed values are presented Appendix D)).
MNIST
CIFAR10
1.0
robust accuracy FC/
robust accuracy
0.8
0.6
0.4
TRADES
TRADES
MAT
MAT
0.1
0.2
0.3
0.4
0.5
0.011
0.021
0.031
0.041
0.051
perturbation strength
perturbation strength
Figure 2: Robust accuracy vs. PGD with different perturbation strength (c). Note that here MAT employs the total loss function.
4.3 COMPARISON WITH MMC
As similar approach to MAT, MMC (Pang et al. 2020) was chosen to be another bascline. To make fair comparison, the attack settings in MMC were used. The classification accuracies obtained from natural training (Natural), AT, and MMC were directly from Pang et al. (2020). Note that the model architecture for MAT wide ResNet (Zagoruyko & Komodakis (016) while the chosen in MMC was ResNet32. As shown in Table the MAT model yields remarkably higher robustness than the MMC model, despite the underlying difference in model architectures.
4.4 ADAPTIVE ATTACKS
Carlini al. 2019) stated that performing robustness evaluation against adaptive attacks, i.e., tacks with objective functions which take the defense details into account, is essential to prevent a false sense of robustness. Thus, adaptive PGD attacks for MAT-trained models can be achieved by directly maximizing the training loss, including the regularization terms (SO loss or BIBO loss). Note that the BIBO loss is result of maximization (using PGD). However, conducting attacks by optimizing an objective function incorporating such maximized result suffers quadratic time com- plexity, which may not realistic. Nonetheless, the inner maximization can actually be simplified as the inner cosine-distance function since the outer optimization is also maximization process. The same technique can also be applied to the TRADES loss to conduct adaptive attack to TRADES models. In the experiments, the threat model of this adaptive attack is the section 4.] The robust accuracy under such attack TRADES models shown in Table The best performance still comparable with the state-of-the-art performance (of TRADES).
described in Sub-

---

Under review as a conference paper at ICLR 2021
Table 4: Robust accuracy under the adaptive attack.
CIFARTO
MNIST
Defense TRADES
SO
BIBO
PGD
Adaptive-PGD
PGD
Adaptive-PGD
0
56.61% %
56.26%
94.38%
93.06%
X
03.45%
00.45%
76.79%
09.78%
X
0
53.58%
52.27%
99.30%
92.23%
MAT
0
X
79.92%
01.27%
82.06%
09.79%
0
O
80.55%
53.65%
99.28%
92.66%
Table 5: Cosine-similarity to the predicted class center.
CIFARTO
MNIST
Defense
SO
BIBO
Clean
PGD
Clean
PGD
X
X
0.993=00003 0.935#0.102
0.999=100008
0.843:1..15
X
0
0.879:10.12
0.615:10.068
0.995=10.023
0.727:100000
MAT
0
X
0.993110.034
0.996::0.015
0.999=0.008
0.790110.116
0
O
0.845=10.133
0.569+0.075
0.996+00021
0.762++0.055
Note that the adversarial training works (Zhang et al. 2019) Ding et al. 2020: Wang et al. 2020) did
consider adaptive attack evaluations, while adaptive attack method for TRADES (Zhang al.,
2019) is proposed in our work, and it is intuitive that similar approach can be used for MMA (Ding al. 2020) and MART (Wang al. 2020) construct adaptive adversarial examples. Additionally, Tramer et al. (2000) have managed to conduct adaptive attacks toward MMC (Pang et al.) 2020)
models and reduce robust accuracy to under 0.5% by using the MMC loss as the objective function
of PGD, which is the same as our strategy for MAT and TRADES.
4.5 FEATURE CLUSTERING PERFORMANCE BY MAT
The effectiveness of MAT in performing feature clustering was examined by evaluating the cosine- similarity (mean + standard deviation) to the class center of the predicted class. For each class, the mean and standard deviation were computed to quantify the level of clustering within class and over different classes, respectively. As shown in Table 5 the models with lower cosine-simg similarity for clean data have lower clean accuracy (Table 2 than the other models. However, the difference between their adversarial (PGD) similarity and the clean similarity is higher. Therefore, the cosine- similarity of clean data and the difference between the adversarial and clean similarity can be taken as indicators of the clean accuracy and robust accuracy, respectively.
CONCLUSION
To defend against adversarial attacks, this work analyzes the feature distribution of traditionally- trained CNNs for gaining more knowledge about adversarial examples. Two properties, iee., the non-clustering property and confusing-distant means of I-SNE visualization and clustering analysis (showing the limitations regarding representa- tiveness). In this paper, by exploiting feature compactness, a novel training process for improving model robustness, designated as MAT, is proposed, in which the FTC loss aims to induce intra- class feature compactness and inter-class feature dispersion, while two auxiliaries functions (the SO loss and the BIBO loss) of MAT are introduced. The experimental results show that the MAT-trained model with all loss functions applied exhibits significantly higher robustness than the state-of sate-of-ther -f-the-art beethod-trinned (TRADES) model. Additionally, the effectiveness of feature clustering and defense against adaptive attack are also evaluated. Our results reveal that the compactness of the clean features provides useful indication of the clean accuracy, while the distance between the clean compactness and the adversarial compactness provides an indication of the robust accuracy.
property, of the feature distribution are identified by
Overall, this study gives useful insight into the relation between feature compactness and bustmess and provides a novel training process for improving the adversarial robustness of CNNs compared to current -fee----t-h--rr--- methods.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Abdullah Al-Dujaili and Una-May O'Reilly. Sign bits are all you need for black-box attacks. In International Conference on Learning Representations, 2020. URL httph::/oppeneviieeett
net Ftre/Eormm? id SygWCTEFwH
Moustafa Alzantot, Yash Sharma, Supriyo Chakraborty, Huan Zhang, Cho-Jui Hsieh, and Mani B. Srivastava. Genattack: Practical black-box attacks with gradient-free optimization. In Pro- ceedings of the Genetic and Evolutionary Computation Conference (GECCO), GECCO '19, pp. 111 1111-1119, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361118. doi: 10.1145/3321777.33212222 URL hattps:///oi.... 10t5//00.11///// 1145/
5321707. 3321749
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Re- liable attacks against black-box machine learning models. In International Conference on Learning Representations (ICLR), 2018. URL http::: /ttp:s/////eeeeieew.. wwt.forummffocooo SyZIOGwCZ
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57, 2017.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: 2019.
Pin-Yu Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order op- timization based black-box attacks deep neural networks without training substitute models. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Securily, 2017.
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. In International Conference Learning Representations (ICLR), 2020. URL AkeryxBtPB
Reuben Feinman, Ryan R. Curtin, Saurabh Shintre, and Andrew B. Gardner. Detecting Adversarial Samples from Artifacts. arXiv e-prints 1703:01703.04100 art. arXiv: 10fiv:1003.000 703.00440, March 2017.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In International Conference on Learning Representations (ICLR), 2017.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distribut samples and adversarial attacks. In International Conference on Neural In- formation Processing Systems (NeurIPS), NIPS' 18, pp. 7167-7177, Red Hook, NY, USA, 2018. Curran Associates Inc.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M. Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Michael E. Houle, Dawn Song, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In International Conference on Learning Representations (ICLR), 2018.
www.teetttoooee ferss//foummiioe d= 9i
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations (ICLR), 2018. URL
net http://eeeffftooooeeoee/o wttpp//nrweeeee.eee/oroommmmam.m foreet/fooummddo
EJZIBIZAD
Preetum Nakkiran. Adversarial robustness may be at odds with simplicity. arXiv preprint 1rX1v:1901.00532. hiss/9911.0532. 2019. BRLLhttp://axxi..om wttp://g//ggg//......o
Tianyu Pang, Chao Du, and Jun Zhu. Max-mahalanone linear discriminant analysis networks. In International Conference Machine Learning (ICML), 2018
Tianyu Pang, Kun Xu, Yinpeng Dong, Chao Du, Ning Chen, and Jun Zhu. Rethinking softmax cross- entropy loss for adversarial robustness. In International Conference on Learning Representations (ICLR), 2020. URL htlL/hhtss://ppeeeet ew. htw.:eeeeeeeooommmiiiaa.g.gao...

---

Under review as a conference paper at ICLR 2021
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark the robustness of machine learning models. In Reliable Machine Learning in the Wild Workshop, 34th International Conference on Machine Learning, 2017. URL http://aaxttttt org/ abs/
1707. 770700133 04131
Jérôme Rony, Luiz Gustavo Hafemann, Luiz Soares de Oliveira, Ismail Ben Ayed, Robert Sabourin, and Eric Granger. Decoupling direction and norm for efficient gradient-based 12 adversarial at- tacks and defenses. In Conference on Computer Vision and Pattern Recognition (CVPR), pp- 4317-4325, 06 2019, doi: 2..11099CVPR..211..04444
Binxin Ru, Adam Cobb, Arno Blaas, and Yarin Gal. Bayesopt adversarial attack. In International Conference Learning Representations, 2020. URLhttps:/ www. net//frum?? forum?
1d id-Hkee--rtty Irtvi
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. In Conference on Neural Information Pro- cessing Systems (NeurIPS), 2018.
Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing local stability of neural nets through robust optimization. Neurocomputing, 11 2015. doi: 10.1016/ j.neucom.22 2018 0..0.027.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), International Conference on Learning Representations (ICLR), 2015. URL
org/ abs 1409 1556
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014. /ttp://::/axxxv. hotgg//bs/3313 6199
Sanli Tang, Xiaolin Huang, Mingjian Chen, Chengjin Sun, and Jie Yang. Adversarial attack type i: Cheat classifiers by significant changes. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP:1-1, 08 2019. doi: 10. 1099/TPAML.201.22933399
Florian Tramèr, N. Carlini, W. Brendel, and A. Madry. On adaptive attacks to adversarial example defenses. arXiv preprint arXiv:2002.083477 abs/2002.083477 2020.
L.J.P. van der Maaten and G.E. Hinton. Visualizing high -igeedimessional data using t-sne. Journal of Machine Learning Research, 9(nov):2579-266: 2008. ISSN 1532-4435. Pagination: 27.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma. and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representations (ICLR), 2020. URL https:/ /ttpnreeee//pereeee.... net htt/ffommm?ic
rk rk10g6EFWS
Joe H. Ward. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58(301):236 224,244, 1963. ISSN 01621459. uttpttt:: www.. istor.
org/ Sog/stablee/2829G
Ziang Yan, Yiwen Guo, and Changshui Zhang. Deep defense: Training dnns with improved adversarial robustness. In Advances in Neural Information Process-
ing Systems, pp. 417-426, 2018. URL http: httt::/papers.... http:::/papeer...ps...o..o///// wttps.ccpc.c.re treenitraiinnngg -inns - with www.onoodaaeeeerrriiaaeeeoeee robssattiss--ssssseesss....
pdf
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.). Proceedings of the British Machine Vision Conference (BMVC), pp. 87.1-87.12 BMVA Press. September 2016. ISBN 1-901725-59-6 doi: 10.5244/C. 30.87. URL /ttpps::/da dx. doi org / 10. 5244/2.42220 30.87
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference of Machine Learning (ICML), 2019.
10


Reviews:
[{"review_id": "zzQzZ9CO3wY", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a defense that combines prior work on learning features that are compact for samples from the same but dispersed for samples from different classes (MMD by Pang et al.) with (a) a method to find better class centers, (b) a gradient-norm regularization and (c) an adversarial training regularization.\n\nUnfortunately, the reported results on the robustness of the defense are clearly wrong. For one, the core part of this defense by Pang et al. was broken by [1] which is not mentioned here. More importantly, the adversarial attacks employed here are not suited for finding minimal adversarial perturbations against the proposed defense. This can be seen most clearly in Figure 2 (or Table 10 in the appendix): If we allow a perturbation with L-infinity norm of 0.5 on MNIST, then we can always find an adversarial perturbation simply by setting the whole image to a flat gray value of 0.5. In turn, any effective adversarial attack should drive network performance down to at least random baseline performance (10%) for epsilon = 0.5. Instead, the paper reports > 99% accuracy for this value under a PGD attack, which means that PGD is totally ineffective against the given defense and a very different adaptive attack would be needed to accurately measure its robustness. Similarly, in Table 3 the attack success of targeted attacks is often higher than for untargeted attacks, again a clear sign for ineffective attacks. The work also uses an adaptive attack which works better for some versions of MAT but performs similar to PGD in other cases. Hence, the adaptive attack employed here are not good enought.\n\nThe reason why the proposed attacks fail against the defense are probably simple: for one, the attacks optimise a different classificatioon loss then what is actually used by the model. Second, both auxiliary losses may give rise to gradient masking, the most common issue for gradient-based attacks to fail against a defense. I highly suggest the authors study [1] to get familiar with how to engineer strong adaptive attacks.\n\n[1] On Adaptive Attacks to Adversarial Example Defenses, Florian Tramer, Nicholas Carlini, Wieland Brendel, Aleksander Madry, NeurIPS 2020, https://arxiv.org/abs/2002.08347", "extra_scores": {}, "extra_text": {}}, {"review_id": "tD4cYRakmsO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper tackles the problem of training models that are robust to adversarial inputs. The authors starts by observing that previous models generate embeddings that can both (i) place same-class embeddings in different clusters and (ii) different-class embeddings in close proximity. They then introduce new loss functions that penalize these behaviors and design a training procedure (MAT) around these new losses. Finally, they show favorable performance of MAT compared to state-of-the-art techniques for addressing adversarial robustness.\n\nReasons for score:\nOverall, I vote for accepting. Training adversarially robust models is an important problem, and the paper’s experimental validation that the features of prior methods (TRADES) exhibit (i) non-clustering and (ii) confusing distance motivates the approach they take. The loss functions are explicitly designed to combat these issues, and the experimental results clearly show the favorability of the MAT procedure. In addition, the ablation study of the various components of the loss functions also adds some insight into the results. The paper is also very well written.\n\nCons:\nIt would be of interest to have some theoretical justification for the approach. Regarding the loss functions, it seems that BIBO should be a consequence of penalizing FTC loss and SO loss, and should not be explicitly needed (this is also somewhat consistent with the results of Table 2). Finally, some of the experimental results can be explored further. For example, in the ablation study, some of the experiments perform better without one of the loss functions, and it may help to explain such behavior. \n\nClarity / Typos:\nThe paper is very well written. A couple of minor points:\nFeature compactness - Maybe explain this phrase better in the introduction (explained well in Section 3 introduction)\nEqn 1: Maybe write J(f(x), y) and J(f(x), f(x’)) instead\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "oxM2g9iZ9IY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Results: To defend against adversarial attacks, this work experimentally analyzes the feature distribution of traditionally- trained CNNs for gaining more knowledge about adversarial examples. Two properties, i.e., the non-clustering property and confusing-distance property, of the feature distribution are identified by means of t-SNE visualization and clustering analysis (showing the limitations regarding representativeness) in Figure 1. The authors introduce a loss function which separates out cluster centers of CNN output features, setting them as far as possible - so that model accuracy is preserved while strengthening robustness. They test on two datasets: CIFAR10, MNIST, and show improvements in \"robustness\" of the model. \n\nStrong points: The experiments presented are promising in terms of increasing robustness of the learned models. \n\nWeak points: Experiments are only conducted on two datasets, it's unclear how generalization these results are. Further, there is no theoretical development regarding manifolds in the feature space.  \n\nMinor typing errors: \n\"indication of the clean accuracy\" \n\"using PGD optimizationm,\"\n\"an input images x\"\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VdxOoQuC6FI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThe authors propose a novel training process called Manifold-Aware\nTraining (MAT) to increase the robustness of the CNN against adversarial\nexamples. The authors compare MAT against the state-of-the-art in\ndefenses against adversarial evasion attacks (i.e., TRADES and MCC) and\nshow their approach outperforms it.\n\n# Strengths\n\n+  Interesting intuition of performing training \"in the\" manifold\n+  Interesting intuition to support SO and BIBO losses\n\n# Weaknesses\n\n-  Lack of comparison with a similar approach\n-  Lack of conclusive remarks / actionable points\n\n# Comments\n\nI praise the authors intuition of exploring the possibility of training\na classifier by exploiting knowledge of the manifold - its immediate\nimplication is that of focusing on lower dimensions of compact features\nthat would be more robust to manipulation (and thus adversarial\nattacks). I also particularly appreciate the threat model and the fact\nthe approach is evaluated in a white-box setting, according to Carlini\net al. (2019). While the authors' intuition is interesting, I wonder how\neasy this is to achieve in practice. In general, we have no knowledge of\nthe underlying manifold and I thus wonder what guarantees this approach\nwould provide. The results seem to show no clear loss-dependent trend\nand I thus wonder whether we can easily draw conclusive remarks. (For\ninstance, should we use SO and BIBO always? From a theoretical\nperspective, it seems so, but experiments seem to show otherwise.)\n\nFigure 1 is interesting as it shows that the representative features of\nsame-class samples are not always similar to one another. Wasn't this\nalready explored in Szegedy et al.? Perhaps not visually, but the fact\nthat objects close in the input space get eventually separated in the\nlatent space across the layers of the CNN is quite known. Also, a\nsimilar approach to the authors' proposal seems to be explored by Crecchi\net al. [1]. It would be interesting to compare and position MAT against\nthis.\n\n## Additional Comments\n\nIn Section 3.2, the authors propose two auxiliary loss functions to\nfurther improve the robustness of MAT. I wonder whether the BIBO loss\nwould just suffice for the purpose, instead of relying on the\nsecond-order loss too. I appreciate the explanation in Section 3.2.3 but\nit would be interesting to understand how one should expect to tune\nalpha and beta accordingly. \n\nResults on CIFAR10 seem less stable than compared to those on MNIST. In\nparticular, there is no trend that shows that relying on SO and BIBO on\na clean dataset provides better results than with a plain FTC loss:\n94%->85%->95%->83%; why the 95%? Is that expected? Similar reasoning\ncan actually be applied to MNIST too when one looks at PGD:\n61%->99%->82->99; why 82%? Is this expected? In contrast, TRADES seem to\nshow an expected trend (even when BIBO loss is considered).\n\nThe authors rely on the library 'foolbox' - my impression was that\ncleverhans [2] represented the state-of-the-art when it comes to\nexperimenting with adversarial ML attacks. What advantages does foolbox\nhave compared to cleverhans?\n\nAlthough off-topic for this work, it would be interesting to understand\nwhether MAT would be beneficial in defending against adversarial attacks\nthat consider realizable attacks (in the problem space). Figure 2 shows\nthe stability of MAT robustness for increasing values of perturbations.\nAdversarial attacks in the problem-space might need to consider\nadditional constraints while being non-necessarily constrained in a\nlp-norm [3].\n\n[1] Crecchi et al. Detecting Adversarial Examples through Nonlinear\nDimensionality Reduction. ESANN 2019\n(https://pralab.diee.unica.it/sites/default/files/crecchi19-esann.pdf)\n\n[2] http://www.cleverhans.io/\n\n[3] https://s2lab.kcl.ac.uk/projects/intriguing/ (IEEE S&P 2020)\n\n### Minor Typos\n\n\"optimizationm\" -> \"optimization\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "DiDrtOmSKyF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to leverage the manifold aware training to learn compact representation. The authors proposes to enforce the learned representation along with generated vectors for different clusters, which implicitly enlarge the margin of the prediction.\nHowever the technical contribution as using the three-term loss to improve robustness is limited. In particular, it's is unclear what the equation (10) and (11) try to prove without a concrete theorem or lemma statement. \n\nFrom the empirical performance, it looks promising from table 2 but it's also quite clear that the TRADES loss BIBO dominates the performance, and without adding this loss, the proposed MAT training cannot achieve high robustness. This is as expected and also render the proposed method less effective.  \nIn addition, TRADES is evaluated on ImageNet and it would be good for the work to evaluate on ImageNet to demonstrate the generalization ability and scalability. It would also be good to explain why without the BIBO loss, the robustness against adaptive attack of MAT is almost 0 which again shows the weakness of the main proposed method.\n\nIt would also be necessary to provide analysis for the properties of the learned representation. For instance, if it is compact features, whether its rank is indeed lower, and whether the entropy of the learned features is indeed low in order to claim the consistent and compact feature representation. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers expressed clear concerns about the paper but the authors did not provide any response. 
Decision: Reject