With False Friends Like These, Who Can Have Self-Knowledge?

1 INTRODUCTION . Deep neural networks ( DNNs ) have achieved breakthroughs in a variety of challenging problems such as image understanding ( Krizhevsky et al. , 2012 ) , speech recognition ( Graves et al. , 2013 ) , and automatic game playing ( Mnih et al. , 2015 ) . Despite these remarkable successes , their pervasive failures in adversarial settings , the phenomenon of adversarial examples ( Biggio et al. , 2013 ; Szegedy et al. , 2014 ) , have attracted significant attention in recent years ( Athalye et al. , 2018 ; Carlini et al. , 2019 ; Tramer et al. , 2020 ) . Such small perturbations on inputs crafted by adversaries are capable of causing well-trained models to make big mistakes , which indicates that there is still a large gap between machine and human perception , thus posing potential security concerns for practical machine learning ( ML ) applications ( Kurakin et al. , 2016 ; Qin et al. , 2019 ; Wu et al. , 2020b ) . An adversarial example is “ an input to a ML model that is intentionally designed by an attacker to fool the model into producing an incorrect output ” ( Goodfellow & Papernot , 2017 ) . Following the definition of adversarial examples on classification problems ( Goodfellow et al. , 2015 ; Papernot et al. , 2016 ; Elsayed et al. , 2018 ; Carlini et al. , 2019 ; Zhang et al. , 2019 ; Wang et al. , 2020b ; Zhang et al. , 2020 ; Tramèr et al. , 2020 ) , given a DNN classifier f and a correctly classified example x with class label y ( i.e. , f ( x ) = y ) , an adversarial example xadv is generated by perturbing x such that f ( xadv ) 6= y and xadv ∈ B ( x ) . The neighborhood B ( x ) denotes the set of points within a fixed distance > 0 of x , as measured by some metric ( e.g. , the lp distance ) , so that xadv is visually the “ same ” for human observers . Then , an imperfection of the classifier is highlighted by Gadv = Acc ( D ) −Acc ( A ) , the performance gap between the accuracy ( denoted by Acc ( · ) ) evaluated on clean set sampled from data distribution D and adversarially perturbed set A . An adversary could construct such a perturbed setA that looks no different from D but can severely degrade the performance of even state-of-the-art DNN models . From direct attacks in the digital space ( Goodfellow et al. , 2015 ; Carlini & Wagner , 2017 ) to robust attacks in the physical world ( Kurakin et al. , 2016 ; Xu et al. , 2020 ) , from toy classification problems ( Chen et al. , 2020 ; Dobriban et al. , 2020 ) to complicated perception tasks ( Zhang & Wang , 2019 ; Wang et al. , 2020a ) , from the high dimensional nature of the input space ( Goodfellow et al. , 2015 ; Gilmer et al. , 2018 ) to the framework of ( non ) -robust features ( Jetley et al. , 2018 ; Ilyas et al. , 2019 ) , many efforts have been devoted to understanding and mitigating the risk raised by adversarial examples , thus closing the gap Gadv . Previous works mainly concern the adversarial risk on correctly classified examples . However , they typically neglect a risk on misclassified examples themselves which will be formalized in this work . In this paper , we first investigate an intriguing , yet far overlooked phenomenon , where given a DNN classifier f and a misclassified example x with class label y ( i.e. , f ( x ) 6= y ) , we can easily perturb x to xhyp such that f ( xhyp ) = y and xhyp ∈ B ( x ) . Such an example xhyp looks harmless , but actually can be maliciously utilized by a false friend to fool a model to be self-satisfied . Thus we name them hypocritical examples ( see Figure 1 for a comparison with adversarial examples ) . Adversarial examples and hypocritical examples are two sides of the same coin . On the one side , a well-performed but sensitive model becomes unreliable in the existence of adversaries . On the other side , a poorly performed but sensitive model behaves well with the help of friends . With false friends like these , a naturally trained suboptimal model could have state-of-the-art performance , and even worse , a randomly initialized model could behave like a well-trained one ( see Section 2.1 ) . It is natural then to wonder : Why should we care about hypocritical examples ? Here we give two main reasons : 1 . This is of scientific interest . Hypocritical examples are the opposite of adversarial examples . While adversarial examples are hard test data to a model , hypocritical examples aim to make it easy to do correct classification . Hypocritical examples warn ML researchers to think carefully about high test accuracy : Does our model truly achieve human-like intelligence , or is it just simply because the test data prefers the model ? 2 . There are practical threats . A variety of nefarious ends may be achievable if the mistakes of ML systems can be covered up by hypocritical attackers . For instance , before allowing autonomous vehicles to drive on public roads , manufacturers must first pass tests in specific environments ( closed or open roads ) to obtain a license ( Administration et al. , 2016 ; Briefs , 2015 ; Lei , 2018 ) . An attacker may add imperceptible perturbations on the test examples ( e.g. , the “ stop sign ” on the road ) stealthily without human notice , to hypocritically help an ML-based autonomous vehicle to pass the tests that might otherwise fail . However , the high performance can not be maintained on public roads without the help of the attacker . Thus , the potential risk is underestimated and traffic accidents might happen unexpectedly when the vehicle driving on public roads . In such a case , if the examples used to evaluate a model are falsified by a false friend , the model will manifest like a perfect one ( on hypocritical examples ) , but it actually may not be well performed even on clean examples , not to mention adversarial examples . Thus a new imperfection of the classifier can be found in Ghyp = Acc ( F ) −Acc ( D ) , the performance gap between the accuracy evaluated on clean set sampled from D and hypocritically perturbed set F . Still , F looks no different from D but can stealthily upgrade the performance . Once a deployer trusts the hypocritical performance carefully designed by a false friend and uses the “ well-performed ” model in real-world applications , potential security concerns appear even in benign environments . Thus we need methods to defend our models from false friends , that is , making our models have self-knowledge . We propose a defense method by improving model robustness against hypocritical perturbations . Specifically , we formalize the hypocritical risk and minimize it via a differentiable surrogate loss ( Section 3 ) . Experimentally , we verify the effectiveness of our proposed attack ( Section 2.1 ) and defense ( Section 4.1 ) . Further , we study the transferability of hypocritical examples across models trained with various methods ( Section 4.2 ) . Finally , we conclude our paper by discussing and summarizing our results ( Section 5 and Section 6 ) . Our main contributions are : • We give a formal definition of hypocritical examples . We demonstrate the unreliability of standard evaluation process in the existence of false friends and show the potential security risk on the deployment of a model with high hypocritical performance . • We formalize the hypocritical risk and analyze its relation with natural risk and adversarial risk . We propose the first defense method specialized for hypocritical examples by minimizing the tradeoff between the natural risk and an upper bound of hypocritical risk . • Extensive experiments verify the effectiveness of our proposed methods . We also examine the transferability of hypocritical examples . We show that the transferability is not always desired by the attackers , which depends on their purpose . 2 FALSE FRIENDS AND ADVERSARIES . Better an open enemy than a false friend ! Only by being aware of the potential risk of the false friend can we prevent it . In this section , we expose a kind of false friends , who are capable of manipulating model performance stealthily during the evaluation process , thus making the evaluation results unreliable . We consider a classification task with data ( x , y ) ∈ Rd × { 1 , . . . , C } from a distribution D. Denote by f : Rd → { 1 , ... , C } the classifier which predicts the class of an input example x : f ( x ) = arg maxk pk ( x ) , where pk ( x ) is the kth component of p ( x ) : Rd → ∆C ( e.g. , the output after softmax activation ) , in which ∆C = { u ∈ RC | 1Tu = 1 , u ≥ 0 } is the probabilistic simplex . Adversarial examples are malicious inputs crafted by an adversary to induce misclassification . We first give the commonly accepted definition of adversarial examples as follows : Definition 1 ( Adversarial Examples ) . Given a classifier f and a correctly classified input ( x , y ) ∼ D ( i.e. , f ( x ) = y ) , an -bounded adversarial example is an input x∗ ∈ Rd such that : f ( x∗ ) 6= y and x∗ ∈ B ( x ) . The assumption underlying this definition is that inputs satisfying x∗ ∈ B ( x ) preserve the label y of the original input x . The reason for the existence of adversarial examples is that a model is overly sensitive to non-semantic changes . Next , we formalize a complementary phenomenon to adversarial examples , called hypocritical examples . Hypocritical examples are malicious inputs crafted by a false friend to stealthily correct the prediction of a model : Definition 2 ( Hypocritical Examples ) . Given a classifier f and a misclassified input ( x , y ) ∼ D ( i.e. , f ( x ) 6= y ) , an -bounded hypocritical example is an input x∗ ∈ Rd such that : f ( x∗ ) = y and x∗ ∈ B ( x ) . The same as adversarial examples , hypocritical examples are bounded to preserve the label of the original input , and are another consequence that arises from excessive sensitivity of a classifier . As a false friend , a hypocritical example can be generated from a misclassified example by maximizing max x′∈B ( x ) 1 ( f ( x′ ) = y ) , ( 1 ) which is equivalent to minimizing min x′∈B ( x ) 1 ( f ( x′ ) 6= y ) , ( 2 ) where 1 ( · ) is the indicator function . Similar to Madry et al . ( 2018 ) ; Wang et al . ( 2020b ) , in practice , we leverage the commonly used cross entropy ( CE ) loss as the surrogate loss of 1 ( f ( x′ ) 6= y ) and minimize it by projected gradient descent ( PGD ) . Note that Equation 2 looks similar to but conceptually differs from the known targeted adversarial attack ( Carlini & Wagner , 2017 ) , which generates a kind of adversarial examples defined on correctly classified clean inputs and targeted to wrong classes . The hypocritical examples here are defined on misclassified inputs and are targeted to their right classes .

Summary: 1. The premise of the paper is that the adversary can perturb the *test* set so that the model is shown to perform better that it really is capable of. And in Section 7 (Conclusion) the paper claims that it exposes this new risk. However, remember that this risk is already mitigated in practice by keeping the test data *independent* of the model/classifier (e.g., see Kaggle competitions where the test set is hidden). Therefore, the perceived risk is not even present. In the context that the technique has been introduced, it seems like the [malicious] actor would only be fooling him/her self rather than fooling the model/classifier.