Adversarial Attacks on Binary Image Recognition Systems

1 INTRODUCTION . In this paper we study adversarial attacks on models designed to classify binary ( i.e . black and white ) images . Models for binary image classification are heavily used across a variety of applications that include receipt processing , passport recognition , check processing , and license plate recognition , just to name a few . In such applications , the text recognition system typically binarizes the input image ( e.g . check processing ( Jayadevan et al. , 2012 ) , document extraction ( Gupta et al. , 2007 ) ) and trains a model to classify binary images . In recent years there has been an overwhelming interest in understanding the vulnerabilities of AI systems . In particular , a great deal of work has designed attacks on image classification models ( e.g . ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ; Moosavi-Dezfooli et al. , 2016 ; Kurakin et al. , 2016 ; Papernot et al. , 2016 ; Madry et al. , 2017 ; Carlini & Wagner , 2017 ; Chen et al. , 2017 ; Ilyas et al. , 2018a ; b ; Tu et al. , 2019 ; Guo et al. , 2019 ; Li et al. , 2019 ) ) . Such attacks distort images in a manner that is virtually imperceptible to the human eye and yet cause state-of-the-art models to misclassify these images . Although there has been a great deal of work on attacking image classification models , these attacks are designed for colored and grayscale images . These attacks hide the noise in the distorted images by making minor perturbations in the color values of each pixel . Somewhat surprisingly , when it comes to binary images , the vulnerability of state-of-the-art models is poorly understood . In contrast to colored and grayscale images , the search space of attacks on binary images is extremely restricted and noise can not be hidden with minor perturbations of color values in each pixel . As a result , existing attack algorithms on machine learning systems do not apply to binary inputs . Since binary image classifiers are used in high-stakes decision making and are heavily used in banking and other multi-billion dollar industries , the natural question is : Are models for binary image classification used in industry vulnerable to adversarial attacks ? In this paper we initiate the study of attacks on binary image classifiers . We develop an attack algorithm , called SCAR , designed to fool binary image classifiers . SCAR carefully selects pixels to flip to the opposite color in a query efficient manner , which is a central challenge when attacking black-box models . We first show that SCAR outperforms existing attacks that we apply to the binary setting on multiple models trained over the MNIST and EMNIST datasets , as well as models for handwritten strings and printed word recognition . We then use SCAR to demonstrate the vulnerability of text recognition systems used in industry . We fool commercial check processing systems used by US banks for mobile check deposits . One major challenge in attacking these systems , whose software we licensed from providers , is that there are two independent classifiers , one for the amount written in words and one for the amount written in numbers , that must be fooled with the same wrong amount . Check fraud is a major concern for US banks , accounting for $ 1.3 billion in losses in 2018 ( American Bankers Association , 2020 ) . Since check fraud occurs at large scale , we believe that the vulnerability of check processing systems to adversarial attacks raises a serious concern . We also show that no attack can obtain reasonable guarantees on the number of pixel inversions needed to cause misclassification as there exist simple classifiers that are provably robust to large perturbations . There exist classifiers for d-dimensional binary images such that every class contains some image that requires Ω ( d ) pixel inversions ( L0 distance ) to change the label of that image and such that for every class , a random image in that class requires Ω ( √ d ) pixel inversions in expectation . Related work . The study of adversarial attacks was initiated in the seminal work by Szegedy et al . ( 2013 ) that showed that models for image classification are susceptible to minor perturbations in the input . There has since then been a long line of work developing attacks on colored and greyscale images . Most relevant to us are L0 attacks , which iteratively make minor perturbations in carefully chosen pixels to minimize the total number of pixels that have been modified ( Papernot et al. , 2016 ; Carlini & Wagner , 2017 ; Schott et al. , 2018 ; Guo et al. , 2019 ) . We compare our attack to two L0 attacks that are applicable in the black-box binary setting ( Schott et al. , 2018 ; Guo et al. , 2019 ) . Another related area of research focuses on developing attacks that query the model as few times as possible ( Chen et al. , 2017 ; Ilyas et al. , 2018a ; b ; Guo et al. , 2019 ; Li et al. , 2019 ; Tu et al. , 2019 ; Al-Dujaili & O ’ Reilly , 2019 ) . We discuss below why most of these attacks can not be applied to the binary setting . There has been previous work on attacking OCR systems ( Song & Shmatikov , 2018 ) , but the setting deals with grayscale images and white-box access to the model . Attacks on colored and grayscale images employ continuous optimization techniques and are fundamentally different than attacks on binary images which , due to the binary nature of each pixel , employ combinatorial optimization approaches . Previous work has formulated adversarial attack settings as combinatorial optimization problems , but in drastically different settings . Lei et al . ( 2018 ) consider attacks on text classification for tasks such as sentiment analysis and fake news detection , which is a different domain than OCR . Moon et al . ( 2019 ) formulate L∞ attacks on colored image classification as a combinatorial optimization problem where the search space for the change in each pixel is { −ε , ε } instead of [ −ε , ε ] . Finally , we also note that binarization , i.e . transforming colored or grayscale images into black and white images , has been studied as a technique to improve the robustness of models ( Schott et al. , 2018 ; Schmidt et al. , 2018 ; Ding et al. , 2019 ) . Previous attacks are ineffective in the binary setting . Previous attacks on grayscale ( or colored ) images are not directly applicable to our setting since they cause small perturbations in pixel values , which is not possible with binary images . One potential approach to use previous attacks is to relax the binary values to be in the grayscale range . However , the issue with this approach is that small changes in the relaxed grayscale domain are lost when rounding the pixel values back to being a valid binary input for the classifier . Another approach is to increase the step size of an attack such that a small change in a grayscale pixel value instead causes a binary pixel value to flip . This approach is most relevant to L0 attacks since they perturb a smaller number of pixels . However , even for the two L0 attacks which can be applied to the binary setting with this approach ( Guo et al. , 2019 ; Schott et al. , 2018 ) , this results in a large and visible number of pixel inversions , as shown in Section 6 . 2 PROBLEM FORMULATION . Binary images and OCR systems . Binary images x ∈ { 0 , 1 } d are d-dimensional images such that each pixel is either black or white . An m-class classifier F maps x to a probability distribution F ( x ) ∈ [ 0 , 1 ] m where F ( x ) i corresponds to the confidence that image x belongs to class i . The predicted label y of x is the class with the highest confidence , i.e. , y = arg maxi F ( x ) i. Optical Character Recognition ( OCR ) systems convert images of handwritten or printed text to strings of characters . Typically , a preprocessing step of OCR systems is to convert the input to a binary format . To formalize the problem of attacking OCR systems , we consider a classifier F where the labels are strings of characters . Given a binary image x with label y , we wish to produce an adversarial example x′ which is similar to x , but has a predicted label y′ 6= y . For example , given an image x of license plate 23FC6A , our goal is to produce a similar image x′ that is recognized as a different license plate number . We measure the similarity of an adversarial image x′ to the original image x with a perceptibility metric Dx ( x′ ) . For binary images , a natural metric is the number of pixels where x and x′ differ , which corresponds to the L0 distance between the two images . Finding an adversarial example can thus be formulated as the following optimization problem : min x′∈ { 0,1 } d ‖x−x′‖0≤k F ( x′ ) y where k is the maximum dissimilarity tolerated for adversarial image x′ . For targeted attacks with target label yt , we instead maximize F ( x′ ) yt . Since there are at least ( d k ) feasible solutions for x′ , which is exponential in k , this is a computationally hard problem . Check processing systems . A check processing system F accepts as input a binary image x of a check and outputs confidence scores F ( x ) which represent the most likely amounts that the check is for . Check processing systems are a special family of OCR systems that consist of two independent models that verify each other . Models FC and FL for Courtesy and Legal Amount Recognition ( CAR and LAR ) classify the amounts written in numbers and in words respectively . If the predicted labels of the two models do not match , the check is flagged . For example , if the CAR and LAR of a valid check read 100 and “ one hundred ” , the values match and the check is processed . The main challenge with attacking checks is to craft an adversarial example x′ with the same target label for both FC and FL . Returning to the previous example , a successful adversarial check image might have the CAR read 900 and the LAR read “ nine hundred ” . For this targeted attack , the optimization problem is : max x′∈ { 0,1 } d , yt 6=y ||x−x′||0≤k FC ( x ′ ) yt + FL ( x ′ ) yt subject to yt = argmaxiFC ( x ′ ) i = argmaxiFL ( x ′ ) i The attacker first needs to select a target amount yt different from the true amount y , and then attack FC and FL such that both misclassify x′ as amount yt . Since check processing systems also flag checks for which the models have low confidence in their predictions , we want to maximize both the probabilities FC ( x′ ) yt and FL ( x ′ ) yt . In order to have x ′ look as similar to x as possible , we also limit the number of modified pixels to be at most k. Check processing systems are configured such that FC and FL only output the probabilities for a limited number of their most probable amounts . This limitation makes the task of selecting a target amount challenging , as aside from the true amount , the most probable amounts for each of FC and FL may be disjoint sets . Black-box access . We assume that we do not have any information about the OCR model F and can only observe its outputs , which we formalize with the score-based black-box setting where an attacker only has access to the output probability distributions of a model F over queries x′ .

Summary: The main question this paper aims to answer is how vulnerable binary image classification systems are.  This is an important question because of the application of such binary image classifications for check processing, invoice processing, and license plate registration. One also would think that such systems are less vulnerable to adversarial attacks given the simplicity of their inputs and the fact that most adversarial attacks are based on color or grey scale images.  The authors propose an adversarial attack algorithm called SCAR that efficiently flips the binary pixels with reasonable number of queries in order to confuse the classifier to return a desirable label with high confidence. The authors show that the proposed method outperforms the existing baselines on multiple data sets. Very interestingly, they also showed that their algorithm is able to attack the online deposit systems of US bank with a high success rate. Their example of a check with the amount of $401 that is minimally modified for the amount of $701  is quite significant given that the model had to change both the word and numbers on the check.