Adversarial Attacks on Binary Image Recognition Systems
Authors: Eric Balkanski
Harrison Chase
Kojin Oshiba
Alexander Rilee
Yaron Singer
Richard Wang
Venue:  (2021)
Keywords: Adversarial attacks
Binary images
Image Recognition
Check processing systems

Abstract: We initiate the study of adversarial attacks on models for binary (i.e. black and white) image classification. Although there has been a great deal of work on attacking models for colored and grayscale images, little is known about attacks on models for binary images. Models trained to classify binary images are used in text recognition applications such as check processing, license plate recognition, invoice processing, and many others. In contrast to colored and grayscale images, the search space of attacks on binary images is extremely restricted and noise cannot be hidden with minor perturbations in each pixel. Thus, the optimization landscape of attacks on binary images introduces new fundamental challenges.

In this paper we introduce a new attack algorithm called Scar, designed to fool classifiers of binary images. We show that Scar significantly outperforms existing L0 attacks applied to the binary setting and use it to demonstrate the vulnerability of real-world text recognition systems. Scar’s strong performance in practice contrasts with hardness results that show the existence of worst-case classifiers for binary images that are robust to large perturbations. In many cases, altering a single pixel is sufficient to trick Tesseract, a popular open-source text recognition system, to misclassify a word as a different word in the English dictionary. We also demonstrate the vulnerability of check recognition by fooling commercial check processing systems used by major US banks for mobile deposits. These systems are substantially harder to fool since they classify both the handwritten amounts in digits and letters, independently. Nevertheless, we generalize Scar to design attacks that fool state-of-the-art check processing systems using unnoticeable perturbations that lead to misclassification of deposit amounts. Consequently, this is a powerful method to perform financial fraud.

Under review as a conference paper at ICLR 2021
ADVERSARIAL ATTACKS ON BINARY IMAGE RECOGNITION SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
We initiate the study of adversarial attacks on models for binary (i.e. black and white) image classification. Although there has been great deal of work on attacking models for colored and grayscale images, little is known about attacks on models for binary images. Models trained to classify binary images are used text recognition applications such as check processing, license plate recognition, invoice processing, and many others. In contrast to colored and grayscale images, the search space of attacks binary images is extremely restricted and noise cannot be hidden with minor perturbations in each pixel. Thus, the optimization landscape of attacks binary images introduces new fundamental challenges.
In this paper we introduce new attack algorithm called SCAR, designed fool classifiers of binary images. We show that SCAR significantly outperforms existing Lg attacks applied to the binary setting and use to demonstrate the vulnerability of real-world text recognition systems. SCAR's 's strong performance in practice contrasts with hardness results that show the existence of worst-case classifiers for binary images that are robust to large perturbations. In many cases, altering a single pixel is sufficient to trick Tesseract, a popular open-source text recognition system, misclassify word as a different word in the English dictionary. We also demonstrate the vulnerability of check recognition by fooling commercial check processing systems used by major US banks for mobile deposits. These systems are substantially harder to fool since they classify both the handwritten amounts in digits and letters, independently. Nevertheless, we generalize SCAR to design attacks that fool state-of-the-at check processing systems using unnoticeable perturbations that lead to misclassification of deposit amounts. Consequently, this is powerful method to perform financial fraud.
INTRODUCTION
In this paper we study adversarial attacks models designed to classify binary (i.e. black and white) images. Models for binary image classification are heavily used across a variety of applications that include receipt processing, passport recognition, check processing, and license plate recognition, just to name a few. In such applications, the text recognition system typically binarizes the input image (e.g. check processing (Jayadevan al., 2012), document extraction (Gupta et al., 2007)) and trains a model to classify binary images.
In recent years there has been an overwhelming interest in understanding the vulnerabilities of AI systems. In particular, great deal of work has designed attacks on image classification models (e.g. (Szegedy et al., 2013; Goodfellow al., 2014; Moosavi- Dezfooli et al., 2016; Kurakin al., 2016; Moosavi-Dezfooli Papernot et al., 2016; Madry et al., 2017; Carlini & Wagner, 2017; Chen et al., 2017; llyas et al., b0;;;;; Tu et al., 2019; Guo al., 2019; Li et al., 2019)). Such attacks distort images a manner that is virtually imperceptible to the human eye and yet cause state-of-the-as ahe-art models to misclassify these images. Although there has been a great deal of work on attacking image classification models, these attacks are designed for colored and grayscale images. These attacks hide the noise in the distorted images by making minor perturbations in the color values of each pixel.
Somewhat surprisingly, when it comes to binary images, the vulnerability of ofate-of offte--f-hh-aattt models is poorly understood. In contrast to colored and grayscale images, the search space of attacks on binary images is extremely restricted and noise cannot be hidden with minor perturbations of color values in

---

Under review as a conference paper at ICLR 2021
each pixel. As a result, existing attack algorithms on machine learning systems do not apply to binary inputs. Since binary image classifiers are used in high-staker decision making and are heavily used in banking and other multi-billion dollar industries, the natural question is:
Are models for binary image classification used in industry vulnerable to adversarial attacks?
In this paper we initiate the study of attacks on binary image classifiers. We develop an attack algorithm, called SCAR, designed to fool binary image classifiers. SCAR carefully selects pixels to flip to the opposite color in a query efficient manner, which is a central challenge when attacking black-box models. We first show that SCAR outperforms existing attacks that we apply the binary setting on multiple models trained over the MNIST and EMNIST datasets, as well as models for handwriten strings and printed word recognition. We then use SCAR demonstrate the vulnerability of text recognition systems used in industry. We fool commercial check processing systems used by US banks for mobile check deposits. One major challenge in attacking these systems, whose software we licensed from providers, is that there are two independent classifiers, one for the amount written in words and one for the amount written in numbers, that must be fooled with the same wrong amount. Check fraud is major concern for US banks, accounting for $1.3 3..3 billion in losses in 2018 (American Bankers Association, 2020). Since check fraud occurs at large scale, we believe that the vulnerability of check processing systems to adversarial attacks raises serious concern.
We also show that no attack can obtain reasonable guarantees on the number of pixel inversions needed to cause misclassification as there exist simple classifiers that are provably robust to large perturbations. There exist classifiers for d-dimensional binary images such that every class contains some image that requires (dd) pixel inversions (Lo distance) to change the label of that image and such that for every class, a random image in that class requires 2 (d) pixel inversions in expectation.
Related work. The study of adversarial attacks was initiated in the seminal work by Szegedy et al. (2013) that showed that models for image classification are susceptible to minor perturbations in the input. There has since then been a long line of work developing attacks on colored and greyscale images. Most relevant to us are Lg attacks, which iteratively make minor perturbations in carefully chosen pixels to minimize the total number of pixels that have been modified (Papernot et al., 2016; Carlini & Wagner, 2017; Schott et al., 2018; Guo et al., 2019). We compare our attack to two Lo attacks that are applicable in the black-box binary setting (Schott et al., 2018; Guo et al., 2019). Another related area of research focuses on developing attacks that query the model as few times as possible (Chen et al., 2017; llyas et al., 201 8a;b; Guo et al., 2019; Li et al., 2019; Tu et al., 2019; Al-Dujaili & O' O'Reilly. 2019). We discuss below why most of these attacks cannot be applied to the binary setting. There has been previous work attacking OCR systems (Song & Shmatikov, 2018), but the setting deals with grayscale images and white-box access to the model.
Attacks on colored and grayscale images employ continuous optimization techniques and are fun- damentally different than attacks on binary images which, due to the binary nature of each pixel, employ combinatorial optimization approaches. Previous work has formulated adversarial attack settings as combinatorial optimization problems, but in drastically different settings. Lei et al. (2018) consider attacks on text classification for tasks such as sentiment analysis and fake news detection, which is a different domain than OCR. Moon et al. (2019) formulate Lx attacks on colored image classification as a combinatorial optimization problem where the search space for the change in each pixel is (-e,e} instead of [- Finally, we also note that binarization, i.e. transforming colored or grayscale images into black and white images, has been studied as a technique to improve the robustness of models (Schott et al., 2018; Schmidt et al., 2018; Ding et al., 2019).
Previous attacks are ineffective in the binary setting. Previous attacks on grayscale (or colored) images are not directly applicable to our setting since they cause small perturbations in pixel values, which is not possible with binary images. One potential approach to use previous attacks is to relax the binary values to be in the grayscale range. However, the issue with this approach is that small changes in the relaxed grayscale domain are lost when rounding the pixel values back to being a valid binary input for the classifier. Another approach is to increase the step size of attack such that a small change in a grayscale pixel value instead causes a binary pixel value to flip. This approach is most relevant to Lo attacks since they perturb a smaller number of pixels. However, even for the two Lo attacks which can be applied to the binary setting with this approach (Guo et al., 2019; Schott et al., 2018), this results in a large and visible number of pixel inversions, as shown in Section 6.
2

---

Under review as a conference paper at ICLR 2021
2 PROBLEM FORMULATION
Binary images and OCR systems. Binary images x E {0. are d-dimensional images such that each pixel is either black or white. An m-class classifier F maps x to a probability distribution F(x) E [0, 1 where F(x); corresponds to the confidence that image x belongs to class i. The predicted label y of x is the class with the highest confidence, i.e., y arg max; F(x);- Optical Character Recognition (OCR) systems convert images of handwritten characters. Typically, a preprocessing step of OCR systems is to convert the input to a binary format. To formalize the problem of attacking OCR systems, we consider a classifier F' where the labels are strings of characters. Given a binary image x with label y. we wish to produce an adversarial example x' which is similar to but has predicted label y # y. For example, given image of license plate 23FC6A, our goal is to produce a similar image x' that is recognized as a different license plate number. We measure the similarity of adversarial image x' to the original image x with perceptibility metric Dx(x') For binary images, natural metric is the number of pixels where x and x' differ, which corresponds to the Lo distance between the two images. Finding an adversarial example can thus be formulated as the following optimization problem:
printed text to strings of
min F(x')
x' x'E
where k is the maximum dissimilarity tolerated for adversarial image x'. For targeted attacks with target label yr, we instead maximize F(x')y. Since there are at least (+) feasible solutions for x', which is exponential in k, this is a computationally hard problem.
Check processing systems. A check processing system F accepts as input a binary image x of a check and outputs confidence scores F(x) which represent the most likely amounts that the check is for. Check processing systems are a special family of OCR systems that consist of two independent models that verify each other. Models Fc and Fr. for Courtesy and Legal Amount Recognition (CAR and LAR) classify the amounts written in numbers and in words respectively. If the predicted labels of the two models do not match, the check is flagged. For example, if the CAR and LAR of valid check read 100 and "one hundred", the values match and the check is processed. The main challenge with attacking checks is craft adversarial example x' with the same target label for both Fc and Fi. Returning to the previous example, a successful adversarial check image might have the CAR read 900 and the LAR read "nine hundred". For this targeted attack, the optimization problem is:
max x x'E(0.1)" Me x'xx'llsSk
Fx(x') + FL(x')x
subject
yt = argmax,Fooxx)
argmax,Fr.xx"
The attacker first needs to select target amount ye different from the true amount y, and then attack Fc and Fz such that both misclassify x' as amount yt- Since check processing systems also flag checks for which the models have low confidence in their predictions, we want to maximize both the In order to have x' look as similar to x as possible, we also
probabilities F=(x'),, y, and limit the number of modified pixels be most k. Check processing systems are configured such that Fc and Fz only output the probabilities for limited number of their most probable amounts. This limitation makes the task of selecting a target amount challenging, as aside from the true amount, the most probable amounts for each of Fc and Fz may be disjoint sets.
Black-box access. We assume that we do not have any information about the OCR model F and can only observe its outputs, which we formalize with the score- based black-box setting where an attacker only has access to the output probability distributions of a model F over queries x'.
3 EXISTENCE OF PROVABLY ROBUST CLASSIFIERS FOR BINARY IMAGES
We first show the existence of binary image classifiers that are provably robust to any attack that modifies a large, bounded, number of pixels. This implies that there is no attack that can obtain reasonable guarantees on the number of pixel inversions needed to cause misclassification. Our first result is that there exists an m-class linear classifier F for binary images such that every class contains some image whose predicted label according to F cannot be changed with o(d) pixel flips, i.e., every
3

---

Under review as a conference paper at ICLR 2021
class contains at least one image which requires number of pixel flips that is linear in the number of pixels to be attacked. The analysis, which is in the appendix, uses probabilistic argument.
Theorem 1. There exists an m-class linear classifier F for d-dimensional binary images s.t. for all classes there exists at least one binary image x in i that is robust to /44 /2d log m/2 pixel changes, ie., all x' s.1. ||x - xlloo S d/4 /2d log m arg max, F(x'); i.
This robustness result holds for all m classes, but only for the most robust image in each class. We also show the existence of a classifier robust to attacks on an image drawn uniformly at random. There exists 2-class classifier s.t. for both classes, uniformly random image in that class requires, in expectation, 52(v pixel flips to be attacked. The analysis relies on onti-concentration bounds.
Theorem 2. There exists 2-class linear classifier F for d-dimensional binary images such that for both classes uniformly random binary image in that class i is robust v pixel changes in expectation, i.e. Ex-(((i) [ming/:arg max, F(x'), #/ x x'llo| 2 vd/8.
These hardness results hold for worst-case classifiers. Experimental results in Section show that, in practice, classifiers for binary images are highly vulnerable and that the algorithms that present next require a small number of pixel flips to cause misclassification.
4 ATTACKING BINARY IMAGES
In this section, we present SCAR, our main attack algorithm. We begin by describing a simplified version of SCAR, Algorithm I, then discuss the issues of hiding noise in binary images and optimizing the number of queries, and finally describe SCAR. At each iteration, Algorithm 1 finds the pixel in input image x such that flipping Tp to the opposite color causes the largest decrease in F(x'), y, which is the confidence that this perturbed input x' is classified as the true label y. It flips this pixel and repeats this process until either the perturbed input is classified as label y' * y or the maximum Lo distance k with the original image is reached. Because binary images are such that E {0, 1, we implicitly work in Zg. In particular, with c1..... the image x' with pixel p flipped.
......... Cd as the standard basis vectors, x' + Cpp represents
Algorithm 1 A combinatorial attack OCR systems. input model F, image x, label x' f-x
while = arg max, F(x'); and ||x' - xllo S l do
p + arg min, p F(x' ep)y
x'tx'' + eep
return
Although the adversarial images produced by Algorithm I successfully fool models and have small Lo distance the original image, it suffers two aspects: the noise added to the inputs is visible to the human eye, and the required number of queries to the model is large.
Hiding the noise. Attacks on images in a binary domain are fundamentally different from attacks on colored or grayscale images. In the latter two cases, the noise is often imperceptible because the change to any individual pixel is small relative to the range of possible colors. Since attacks on binary images can only invert a pixel's color or leave it untouched, noisy pixels are highly visible their colors contrast with that of their neighboring pixels. This is a shortcoming of Algorithm 1, which results in noise with small Lo distance but that highly visible (for example, see Figure 1). To address this issue, we impose new constraint that only allows modifying pixels on the boundary of black and white regions in the image. A pixel is on a boundary if it white and at least one of its eight neighboring pixels is black (or vice-versa). Adversarial examples produced under this constraint have a greater To distance to their original images, but the noise is significantly less noticeable.
Optimizing the number of queries. An attack may be computationally expensive if it requires many queries to a black-bor model. For paid services where a model is hidden behind an API, running attacks can be financially costly well. Several works have proposed techniques to reduce the number of queries. Many of these are based on gradient estimation (Chen et al., 2017; Tu et al., 2019; Ilyas et al., 201 8a;b; Al-Dujaili & O O'Reilly, 2019). Recently, several gradient-free black-box
4

---

Under review as a conference paper at ICLR 2021
attacks have also been proposed. Li et al. (2019) and Moon et al. (2019) propose two such approaches, but these rely on taking small steps of size E in direction which modifies all pixels. SIMBA (Guo et al., 2019), another gradient-free -ree attack, can be extended to the binary setting and is evaluated in the context of binary images in Section 6. We propose two optimization techniques to exploit correlations between pixels both spatially and temporally. We define the gain from flipping pixel p at point x' as the following discrete derivative of F' in the direction of p:
F(x', - F(x' + ep)y
We say a pixel has large gain if this value is larger than threshold
Spatial correlations. Pixels in the same spatial regions are likely to have similar discrete derivatives (e.g. Figure in appendix). At every iteration, we prioritize evaluating the gains of the eight pixels (p) neighboring the pixel which was modified in the previous iteration of the algorithm. If one of these pixels has large gain, then we flip it and proceed to the next iteration.
Temporal correlations. Pixels with large discrete derivatives at one iteration are likely to also have large discrete derivatives in the next iteration (e.g. Figure 5 in appendix). At each iteration, we first consider pixels that had large gain in the previous iteration. one of these pixels still produces large gain in the current iteration, we flip it and proceed to the next iteration.
SCAR. In order to improve on the number of queries, SCAR (Algorithm 2) prioritizes evaluating the discrete derivatives at pixels which are expected to have large gain according the spatial and temporal correlations. If one of these pixels has large gain, then it is flipped and the remaining pixels are not evaluated. If none of these pixels have large gain, we then consider all pixels on the boundary (xx) of black and white regions in the image x. In this set, the pixel with the largest gain is flipped regardless of whether has gain greater than т. before, we denote the standard basis vector the direction of coordinate i with e; We keep track of the gain of each pixel with vector g.
Algorithm 2 SCAR, Shaded Combinatorial Attack on Recognition sytems.
input model F, image x, label y, threshold т, budget x' <- x. g 0
while y arg max, F F(x'); ;; and ||x' - x1lo S l: do
for p : 9p
E (2y)) do
9p + F(x'),
F(x' + eply
if maxp 9p < r then
for p E (x'') do
9p 5 F(x')y F(x' + oply
// + arg max, 9p
x' x+xx x' + Cp
return
Algorithm is an untargeted attack which finds x' which is classified as label y' * y by F. It can easily be modified into targeted attack with target label yt by changing the first condition in the while loop from y arg max; F(x'); to yt * arg max, F(x'); and by computing the gains Ip as F(x+ + cp)y. F(x),. g. instead of F(x), F(x ep)y- Even though SCAR performs well in practice, there exists simple classifiers for which any algorithm requires a large number of pixel inversions to find an adversarial example x', as shown in Section 3.
5 SIMULTANEOUS ATTACKS
There are two significant challenges to attacking check processing systems. In the previous section, we discussed the challenge caused by the preprocessing step that binarizes check images (Jayadevan et al., 2012). The second challenge is that check processing systems employ two independent models that verify the output of the other model: Fe and Fr, classify the amount written in numbers and in letters respectively. We thus propose an algorithm which tackles the problem of attacking two separate OCR systems simultaneously. A natural approach is to search for a target amount at the intersection of what Fc and Fz determines are probable amounts. However, the models are often highly confident of the true amount, and other amounts have extremely small probability. To increase the likelihood of choosing target amount which will result in adversarial
unmodified checks,
5

---

Under review as a conference paper at ICLR 2021
2 3
1.625
1. 1.5
1.625. 1. 1.625.
1 1.625.
test
test
down down
Figure 1: Examples of attacks CNN trained over MNIST (top left). CNN trained over EMNIST (top right), an LSTM for handwritten numbers (center), and Tesseract for typed words (bottom). The images correspond to, from left to right, the original image, the outputs of SCAR, "ANILLA-SCAM POINTWISE, and SIMBA, The predicted labels are in light gray below each image. For Tesseract attacks (bottom), we show the original image and SCAR's 's output.
example, we first proceed with an untargeted attack on both Fc and FL using SCAR, which returns image x" with reduced confidence in the true amount y. Then we choose the target amount y to be the amount with the maximum value min( Fx(x"),, (x)")),, since our goal is to attack both Fo and Fy. Then we run T-SCAR, which is the targeted version of SCAR, twice to perform targeted attacks on both Fc and FL over image x".
Algorithm 3 The attack on check processing systems. input check image x, models Fc and FL, label y xC, X extract CAR and LAR regions of x
xX, xX t- SCAR (Fo, (c), Xc),SCAR((((())) SEAR(FL FL,,x1) XL)
Ve maxizy (xx)FF(x))); FL (x(x))))
X xi t T-SCAR (F,, (F, Mt), T-SCAR( xH,
replace CAR, LAR regions of x with x return x
6 EXPERIMENTS
We demonstrate the effectiveness of SCAR for attacking text recognition systems. We attack, in increasing order of model complexity, standard models for single handwritten character classification (Section 6.2), LSTM model for handwritten numbers classification (Section 6.3), widely used open source model for typed (printed) text recognition called Tesseract (Section 6.4), and finally commercial check processing systems used by banks for mobile check deposit (Section 6.5).
6.1 EXPERIMENTAL SETUP
Benchmarks. We compare four attack algorithms. SCAR Algorithm 2 with threshold 0.1. VANILLA-SCAR is Algorithm 1, We compare SCAR to Algorithm I to demonstrate the importance of hiding the noise and optimizing the number of queries SIMBA Algorithm in (Guo al.. 2019) with the Cartesian basis and = 1. SIMBA is an algorithm for attacking (colored) images in black-box settings using small number of queries. and takes a step towards Eq -Eq if one of these improves the objective. In the setting where is sampled from the Cartesian basis and E = 1. SIMBA A corresponds to an Lo attack on binary images which iteratively chooses random pixel and flips it if doing so results in decrease in the confidence of the true label. POINTWISE (Schott et al., 2018) first applies random salt and pepper noise until the image is misclassified. It then greedily returns each modified pixel to its original color if the image remains misclassified. We use the implementation available in Foolbox (Rauber et al., 2017).
every iteration, samples direction q
Metries. To evaluate the performance of each attack over a model F and test set X, we use three metries. The success rate of A is the fraction of images E X for which the output image x' = A(x) is adversarial, i.e. the predicted label y/ of is different from the true label y of x. We only attack images which are initially correctly classified by F. We use the Lo distance to measure how similar an image x' = A(x) is to the original image x, which is the number of pixels where x and x' differ. The number of queries model F to obtain output image x' A(x).

---

Under review as a conference paper at ICLR 2021
NNIITT (CNN)
(LNNNS))
Tessenec
1.0
SU0000s rit
rote
5.0008/006
0:80
0.
S0AR
0.
0.5
0.23
00
0.0
0.00
10
Daeberoo
Distance
Docbence
MUISI (CNN))
1.0
1.0
LSTH
0.75
Tesseract
Success rote
0.6
0.5
0.5
0.00
0.25
00
0.0
0.0
2000
0000
7500
10000
800
1000
1500
2000
59k
103k
133K
200K
000
1003
1200
2000
quetios
Numzer queries
Numbor of quuur
Numbw qqaetia
Figure 2: Success rate by To distance and numher of queries for a CNN model on MNIST, a LeNet5 model on EMNIST, an LSTM model on handwritten numbers, and Tesseract over printed words.
The distance constraint k. We seek a principled approach to selecting the maximum Lo distance k. For an image x with label y, the Lo constraint is k = ax(x)/ where F(x) counts the number of pixels in the foreground of the image, a E [0, 1] is a fixed fraction, and yy] represents the number of characters in y, e.g. |23FC6A| 6. In other words, & is fixed fraction of the average number of pixels per character in x. In our experiments, we set a 1/5.
6.2 DIGIT AND CHARACTER RECOGNITION SYSTEMS
For each experiment, we provide further details about the datasets and models in the appendix. We train models over binarized versions of the MNIST digit (LeCun et al., 2010) and EMNIST letter (Cohen al., 2017) datasets. We binarize each dataset with the map I + LT We additionally preprocess the EMNIST letter dataset to only include lowercase letters. We consider five models: logistic regression model (LogReg). a 2-layer perceptron (MLP2), convolutional neural network (CNN), a neural network from (LeCun et al., 1998) (LeNet5), and a support vector machine (SVM). Their Top-1 accuracies are given in the appendix.
We discuss the results of the attacks on the CNN model trained over MNIST and on the LeNet5 model trained over EMNIST. The full results for the remaining 8 models are in the appendix. In Figure 2, we observe that for fixed Lo distances K S k. VANILLA-SCA has the largest number of successful attacks with an Lo distance most k on the CNN model, For example, 80% of the images were successfully attacked by flipping at most 7 of the 784 pixels of an MNIST image. SCAR is very close but requires significantly fewer queries and, as shown in Figure its noise less visible even though its Lo distance is slightly larger. SIMBA requires very few queries to attack between 40% and 65% of the image, but the attacked images have large Ly distances. The success rate does not increase past 40% and 65% because the noise constraint k is reached. POINTWISE obtains success rate close to 85% and 98% on the CNN and LeNet5, respectively. The average Lo distance of the images produced by POINTWISE is between SCAR and SIMBA. Overall, SCAR obtains the best number of queries and Lo distance combination. It is the only attack, together with VANILLA-SCAR, which consistently obtains a success rate close to 100% on MNIST and EMNIST models.
6.3 LSTM ON HANDWRITTEN NUMBERS
We train an OCR model on the ORAND-CAR. dataset, part of the HDSRC 2014 competition on handwritten strings (Diem et al., 2014). This dataset consists of 5793 images from real bank checks taken from a Uruguayan bank. Each image contains between 2 and 8 numeric characters. We implement the OCR model described in (Mor & Wolf, 2018). The trained model achieves precision score of 85.7% on the test set of ORAND-CAR-A, which would have achieved first place in the HDSRC 2014 competition. The results are similar to the attacks on the CNN-MNIST model. SIMBA A has less than 20% success rate. POINTWISE obtains a high success rate with a small number of queries, but is outperformed by SCAR and VANILLA-SCAF being high-ding sigh-dimensional (d x 50, 000) and consisting of multiple digits, the reason why SIMBA performs poorly is that the flipped pixels are spread out over the different digits (see Figure 1).
terms of Lo distance. Due to the images
7

---

Under review as a conference paper at ICLR 2021
four
S/PIH
143
4
A-hlo
Legal
fur honged one.
1-60
Amoun:: Confidence: 0.897
Amount: 701 Confidence: 0.845
Finel
Amount: 701
Confidence: 0.809
Figure 3: An example of a check for $401 attacked by Algorithm 3 that is misclassified with high confidence as S701 by check processing system used by US banks.
6.4 TESSERACT ON PRINTED WORDS
Tesseract is popular open- open-source text recognition system designed for printed text that sponsored by Google (Smith, 2007). We attacked 100 images of random printed English words of length four (the full list of words, together with the misclassified labels, can be found the appendix). Tesseract does not recognize any word and reject input images with excess noise. Since the goal is to misclassify images as words with a different meaning, an attack is successful the adversarial image is classified as word in the English dictionary. The main result for the attacks Tesseract is that, surprisingly, for around half the images, flipping a single pixel results in the image being classified as a different word in the English dictionary (see Figure 2). SCAR again produces attacks with Lo distance close to VANILLA MANILLA-SCAR, but with fewer queries. Unlike the other models, SCAR and VANILLA- MALLLA-SCRR do not reach close to 100% accuracy rate. We hypothesize that this is due to the fact that, unlike digits, not every combination of letters forms a valid label, so many words have an edit distance of multiple characters get the closest different label. In these experiments, POINTWISE obtains the highest success rate. In the appendix, we consider SCAR attacking Tesseract on the word "idle" and analyze the spatial and temporal correlations between pixels in that example.
6.5 CHECK PROCESSING SYSTEMS
We licensed software from providers of check processing systems to major US banks and applied the attack described in Algorithm 3. This software includes the prediction confidence as part of their output. Naturally, access to these systems is limited and the cost per query is significant. We confirm the findings from the previous experiments that SCAR, which is used as a subroutine by Algorithm 3, is effective in query-limited settings and showcase the vulnerability of OCR systems used in the industry. Check fraud is major concern for US banks; it caused over $1.3 billion in losses in 2018 (American Bankers Association, 2020). We obtained 17.1% success rate (19 out of 111 checks) when attacking check processing systems used by banks for mobile check deposits. As previously mentioned, check is successfully attacked when both amounts on the check are misclassified as the same wrong amount (see Figure 3). Since check fraud occurs at large scale, we believe that this vulnerability raises serious concerns."
Classifier
Queries 1615
Lo distance
CAR (Fc)
11.77
LAR (FL)
8757
14.85
We say that check is misclassified with high confidence if the amounts written in number and words are each classified with confidence at least 50% for the wrong label. We obtained high confidence misclassification for 76.5% of the checks successfully attacked. In Figure 3, we show the output of check for $401 that has both amounts classified as 701 with confidence at least 80%. On average, over the checks for which we obtained high confidence misclassification, Algorithm 3 flipped 11.77 and 14.85 pixels and made 1615 and 8757 queries for the amounts in numbers and words respectively. The checks are high resolution, with widths of size 1000. Additional examples of checks misclassified with high confidence can be found in the appendix.
Regarding physical realizability: instead of printing an adversarial check in high resolution, an attacker can redirect the camera input of a mobile phone to arbitrary image files, which avoids printing and taking a picture of an adversarial check. This hacking of the camera input is easy perform Android.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Abdullah Al-Dujaili and Una-May O'Reilly. There are no bit parts for sign bits in black-box attacks. arXiv preprint MXXiv:900.06660 2019.
American
Bankers
survey. 2020. attps://wwin aba http://www...o/eeeeeeeeeeeaereee ant
Association.
Deposit account
fraud
URL
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
IEEE symposium on security and privacy (SP), pp. 39-57. IEEE, 2017.
Pin-Ye Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsich. Zoo: Zeroth order optimization based black-box attacks deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 15-26, 2017.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. EMNIST: Extending MNIST to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN) pp. 29266. IEEE, 2017.
Markus Diem, Stefan Fiel, Florian Kleber, Robert Sablatnig. Jose M Saavedra, David Contreras, Juan Manuel Barrios, and Luiz Oliveira. ICFHR 2014 competition on handwritten digit string recognition in challenging datasets (HDSRC 2014). In 2014 14th International Conference on Frontiers in Handwriting Recognition, pp. 779-784. IEEF, 2014.
Gavin Weiguang Ding, Kry Yik Chau Lui, Xiaomeng Jin, Luyu Wang, and Ruitong Huang, On the sensitivity of adversarial robustness to input data distributions. In ICLR (Poster), 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1rXiv:1412.65772 2014.
Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Q Weinberger.
Simple black-box adversarial attacks. arXiv preprint arXiv:1905.07121, 2019.
Maya Gupta, Nathaniel P Jacobson, and Eric K Garcia. OCR binarization and image pre-processing for searching historical documents. Pattern Recognition, 40(2):389 2007.
Andrew llyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. arXiv preprint vrxiv:1804.0859. 2018a.
Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. arXiv preprint arXiv:1807.00 07998,, 2018b.
R Jayadevan, Satish R Kolhe, Pradeep M Patil, and Umapada Pal. Automatic processing of handwrit- ten bank cheque images: a survey. International Journal on Document Analysis and Recognition (IJDAR), 15(4):267-296 2012.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprini arXiv: 1ate::607.0223. 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. 2010.
Qi Lei, Lingfei Wu, Pin-Yu Chen, Alexandros G Dimakis, Inderjit Dhillon, and Michael Witbrock. Discrete adversarial attacks and submodular optimization with applications to text classification. arXiv preprint arXiv: /8/::///2/ 82200/551 2018.
Yandong Li, Lijun Li, Liqiang Wang. Tong Zhang, and Boqing Gong, Nattack: Learning the distributions of adversarial examples for an improved black-box attack on deep neural networks. arXiv preprint arXiv: arXiv: (00009905500441 2019.

---

Under review as a conference paper at ICLR 2021
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: (eb:::7700000000 06083, 2017.
Seungyong Moon, Gaon An, and Hyun Oh Song. Parsimonious black-box adversarial attacks via efficient combinatorial optimization. arXiv preprinl S0Kiv::905506633. 2019.
Seyed-Mohae Gbosavi-Dezzfooly Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574-2582, 2016.
Noam Mor and Lior Wolf. Confidence prediction for lexicon-free OCR. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 218.225. IEEE, 2018.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on security and privacy (EuroS&P), pp. 372-387. IEEE, 2016.
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark the robustness machine learning models. arXiv preprint arXiv: 7777000.11.. 2017.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. In Advances in Neural Information Processing Systems, pp- 5014-5026, 2018.
Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on MNIST. arXiv preprint 18X5v 09090, 2018
Ray Smith. An overview the Tesseract OCR engine. Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), volume 2, pp. 629-633. IEEE, 2007.
Congzheng Song and Vitaly Shmatikov. Fooling OCR systems with adversarial text images, arXiv 2018.
preprint /rXiv:8802.000
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Chun-Chen Tu, Paishun Ting, Pin-Yu Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, and Shin-Ming Cheng. Autozoom: Autoencoder-bast based zeroth order optimization method for attacking black-box neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 742-749, 2019.
10

---

Under review as a conference paper at ICLR 2021
A MISSING ANALYSIS FROM SECTION 3
Theorem 1. There exists an m-class linear classifier F for d-dimensional binary images s.t. for all classes there exists at least one binary image in i that is robust to d/4 / log m/2 pixel changes, for all x' s.t. ||x x'1lo - d/4 12 m/2. arg max, F(x'); ==
=
Proof. In this proof, we assume that binary images have pixel values in {-1, instead of {0, 1). We consider linear classifier Fwi....... such that the predicted label y of binary image x = arg max; x x
We wish show the existence of weight vectors wi...., w that all have large pairwise Lo distance. This is closely related to error-correction codes in coding theory which, in order to detect and reconstruct a noisy code, also aims to construct binary codes with large pairwise distance.
We do this using the probabilistic method. Consider m binary weight vectors WI,.... Wm chosen uniformly at random, and independently, among all d-dimensional binary vectors w E f-1, By the Chernoff bound, for all i.j E [m], we have that for 0 < 5 < 1,
Pr[l wyllo < (1 8)d/2] S e d544
There are (22) < m2 pairs of images (t, By a union bound and with 5 = /8log m/d. we get that
Pr [lw; - wyllo > d/2 v 2d logm: for all i,j E [m]] +
m=--0= 80a/4 > 0.
Thus, by the probabilistic method, there exists wi...., w such that l| -wilo > d/2
12d m for all j.j E [m].
remains to show that the linear classifier Fw......"".,, satisfies the condition of the theorem statement. For class i, consider the binary image x; = w.- Note that for binary images x E f-1,1y we have x <Tw"- d 2 x w; llo. Thus, x] w d and arg max , x] w3 < 2d m, and get x. w - arg max, jæz x. W[ > 2 log m. Each pixel change reduces this difference by at most 4. Thus, for all x' such that |x; - x'Ilo < (d 2/dd log m)/4 = d/4 v 2d log m/2, we have x'T arg max j4z x'T w > 0 and the predicted label of x' is
D
Theorem 2. There exists 2-class linear classifier F for d-dimensional binary images such that for both classes i, uniformly random binary image in that class i is robust to vd/8 pixel changes in expectation, i.e. Bx)((() [ming carg max, F(xc)szti I|x x'|lo] 2 vd/8.
Proof. Consider the following linear classifier
O F(x) {
f x0/2 <
otherwise
Informally, this a classifier which assigns label if ||x||o < d/2 and label 1 if ||x|o > d/2. The classifier tiebreaks the ||x||o = d/2 case depending whether or not the first position in is 1 or Notice that this classifier assigns exactly half the space the label 0, and the other half the label 1.
Consider class 0 and let ((0) be the uniform distribution over all x in class O. We have
1(dd) d
Pr [||x||o (0
8
when 8 < d/2 and Pixeu(()) x||0 = T (9) when s d/2. The binomial coefficient (.) is maximized when d/2. For all d E zτ, Stirling's approximation gives lower and upper bounds of / 2 2e -d S d! < dd d-1, Since dl even, we get
d
d!
S e2d
d/2
111)
क vVd
Therefore, we have that for all 8,
1
d
s e2d
Pr |||x||0 = x] xE
2d
d/2)
v
11

---

Under review as a conference paper at ICLR 2021
which implies
vvd 2V
Pr (
||x||| d/21
2
e
de
4e
x vVd 2
The same argument follows similarly for members of class 1. Therefore, for either class, at least half of the images 전 of that class are such that |||x||o d/21 2 zyd 2 v/d
These images require at least
4
vd pixel flips in order to change the predicted label according to F, and we obtain the bound in the theorem statement.
B ADDITIONAL DESCRIPTION OF DATASETS AND MODELS
B.1 DIGIT AND CHARACTER RECOGNITION SYSTEMS
The datasets. We preprocess the EMNIST letter dataset to only include lowercase letters, since an uppercase letter which is misclassified as the corresponding lowercase letter does not change the semantic meaning of the overall word. We randomly select 10 correctly-classifee samples from each class in MNIST and EMNIST lowercase letters form two datasets attack.
Models. We consider the following five models, trained in the same manner for the MNIST and EMNIST datasets. For each model, we also list their Top-1 accuracies on MNIST and EMNIST.
LogReg: We create a logistic regression model by flattening the input and follow this with fully connected layer with softmax activation. (MNIST: 91.87% EMNIST: 80.87%
MLP2: We create a 2-layer MLP by flattening the input, followed by two sets of fully connected layers of size 512 with ReLU activation and dropout rate 0.2. We then add a fully connected layer with softmax activation. (MNIST: 98.01% % EMNIST: 93.46%)
CNN: We use two convolutional layers of 32 and 64 filters of size 3 x 3, each with ReLU activation. The latter layer is followed by a 2 x 2 Max Pooling layer with dropout rate 0.25. (MNIST: 99.02% EMNIST: 95.04%)
This output is flattened and followed by a fully connected layer of size 128 with ReLU activation and dropout rate 0.5. We then add fully connected layer with softmax activation.
LeNet 5: We use the same architecture in (LeCun et al., 1998). (MNIST 99.01% % EMNIST: 94.33%)
SVM: We use the sklearn implementation with default parameters. (MNIST: 94.11%/ EMNIST: 87.53%)
Except for the SVM, we train each model for 50 epochs with batch size 128, using the Adam optimizer with learning rate of 10-3 The experimental results for CNN on MNIST and LeNet5 on EMNIST are shown in Section 5.
B.2 LSTM ON HANDWRITEN NUMBERS
The dataset. We train an OCR model on the ORAND-CAR-A dataset, part of the HDSRC 2014 competition handwritten strings (Diem al., 2014). This dataset consists of 5793 images from real bank checks taken from a Uruguayan bank. The characters in these images consist of numeric characters (0-9) and each image contains between 2 and 8 characters. These images also contain some background noise due to the real nature of the images. We observe the train/test split given in the initial competition, meaning that we train our model on 2009 images and attack only a randomly selected subset from the test set (another 3784 images). The images as presented in the competition were colored, but we binarize them similar preprocessing step as done for MNIST/EMNIST datasets.
The LSTM model. We implement the OCR model described in (Mor & Wolf, 2018), which consists of convolutional layer, followed by 3-layer deep bidirectional LSTM, and optimizes for CTC loss. CTC decoding was done using a beam search of width 100. The model was trained with the Adam optimizer using learning rate of 10-1, and was trained for 50 epochs. The trained model
12

---

Under review as a conference paper at ICLR 2021
1IdEST (1 (LogFeg)
Bassy
1.775
6--
0.5
25
0.50
SCAR
10
20
13
20
20
30
1.0
1I4ST (LeNetS)
0.75
0.
0250
0.25
2003 9000 1520
18860
2000
5000
7090
10000
2000 1000
7530
13550
1500
3000
3536
19931
quoles
Figure 4: Success rate by Lo distance and by number of queries for four different models on MNIST.
achieves a precision score of .857 on the test set of ORAND-CAR-A, which would have achieved first place in that competition.
B.3 TESSERACT ON PRINTED WORDS
The model. We use Tesseract version 4.1.1 trained for the English language. Tesseract 4 is based on an LSTT model (see (Song Shmatikov, 2018) for detailed description of the architecture of Tesseract's model).
The dataset. We attack images of single printed English word. Tesseract supports a large number of languages, and we use the version of Tesseract trained for the English language. We picked words of length four in the English dictionary. We then rendered these words in black over white background using the Arial font in size 15. We added 10 white pixels for padding on each side of the word. The accuracy rate over 1000 such images of English words of length four chosen at random 0.965 and the average confidence among words correctly classified 0.906. Among the words correctly classified by Tesseract, we selected 100 at random to attack.
For some attacked images with a lot of noise, Tesseract does not recognize any word and rejects the input. Since the goal of these attacks is misclassify images as words with a different meaning, only consider an attack to be successful if the adversarial image produced is classified as a word in the English dictionary. For example, consider attacked image of the word "one". If Tesseract does not recognize any word in this image, or recognizes "oe" or ":one", we do not count this image as a successful attack.
We restricted the attacks to pixels that were at distance at most three of the box around the word. Since our algorithm only considers boundary pixels, this restriction avoids giving an unfair advantage to our algorithm in terms of total number of queries. In some cases, especially images with a lot of noise, Tesseract does not recognize any word and rejects the input. Since the goal of these attacks is to misclassify images as words with different meaning than the true word, we only consider an attack be successful if the adversarial image produced is classified as a word in the English dictionary. For example, consider an image with the text "one". If Tesseract does not recognize any word in this image, or recognizes "oe" or ""one", we do not count this image as a successful attack.
36/6 36/6
3676
36/6
36/6
Figure 6: Examples of attacks on the LSTM for handwritten numbers. The images correspond to, from left to right, the original image, the outputs of SCAR, VANILLA-SCAR POINTWISE. and SIMBA.
13

---

Under review as a conference paper at ICLR 2021
IXANEST (Legfeg))
1.0
1.-
60-
66
SCAR
13
10
ENNIST (CNB)
0.
2003 2000
1920
17860
2000
5000
7990
10000
200 7500
1000
13533
2500
5000
7534
1993
Figure 5: Success rate by Lo distance and by number of queries for four different models on EMNIST.
C ADDITIONAL EXPERIMENTAL RESULTS
In Figure 4 and Figure 5, we provide additional experimental results on the MNIST and EMNIST datasets. In Figure 6, we give additional examples of attacks on the LSTM model for handwritten number. Table 1, we list the 100 English words length we attacked together with the word label of the image resulting from running SCAR.
Spatial and temporal correlations. In Figure 7 we plot a separate line for each pixel p and the corresponding decrease confidence from flipping that pixel at each iteration. We first note the pixels with the smallest gains at some iteration are often among the pixels with the smallest gains the next iteration, which indicates temporal correlations. Most of the gains are negative, which implies that, surprisingly, for most pixels, flipping that pixel increases the confidence of the true label. Thus, randomly choosing which pixel to flip, as in SIMBA, ineffective.
0.5
Average
Marginal Effect
-0.5
1
2
3
4
5
lteration
Figure 7: The gain from each pixel for the five iterations took to successfully attack the word "idle" on Tesseract.
As discussed in Section 4, SCAR exploits spatial and temporal correlations to optimize the number of queries needed. As an example, we consider SCAR attacking Tesseract on the word "idle".
Figure 8 again shows the gain from flipping each pixel, but this time as heatmap for the gains the first iteration. We note that most pixels with a large gain have at least one neighboring pixel that also has large gain. This heatmap illustrates that first querying the neighboring pixels of the previous pixel flipped is an effective technique to reduce the number queries needed to find a high gain pixel.
14

---

Under review as a conference paper at ICLR 2021
0.10
0.05
idie.
0.00
-0.05
-0.10
Figure 8: Heatmap of the gains from flipping a pixel on the word "idle" with Tesseract.
Original
Label
Original
Label
Original
Label
word
from SCAR
word
from SCAR
word
from Scar
down
dower
race
rate
punt
pant
fads
fats
nosy
rosy
mans
mans
pipe
pie
serf
set
cram
ram
soft
soft
dare
dare
cape
tape
pure
pure
hood hoot
bide
hide
zoom
zoom
yarn
yam
full
fall
lone
tone
gorp
gore
lags
fags
fuck
fucks
fate ate
dolt
dot
fist
fist
mags
mays
mods
mots
went
weal
oust
bust
game
game
omen men
rage
rage
taco
idle
die
moth
math
ecol
col
yeah
yeah
woad
woad
deaf
deaf
feed
feet
aged
ed
vary
vary
nuns runs
dray
ray
tell
tel
educ educ
ency
ency
avow
vow
gush
gust
pres
press
wits
wits
news
news
deep
sleep
weep
ween
swim
swim
bldg
bid
vile
vie
hays
nays
warp
war
nets
tube
lube
lost
lo
smut snout
lure
hare
sqrt
sat
mies miles
romp
romp
okay
okay
boot hoot
comp
camp
kept
sept
yipe
vie
pith
pithy
herb
herbs
hail fail
ploy
pro
show
how
saga
gaga
toot foot
hick
nick
drat
rat
boll
boil
tout
foul
limo
lino
elev ale
blur
bur
idem
idler
dank dank
biog
dog
twin
twins
gild
ail
lain
fain
slip
sip
waxy
waxy
gens
gents
yeti
yet
test
fest
mega
mega
loge toge
pups
pups
Table 1: The 100 English words of length we attacked together with the word label the image
resulting from running SCAR.
Finally, in Figure 6, show additional examples of our attacks check processing systems.
15

---

Under review as a conference paper at ICLR 2021
5 Five
7
2
seventy
Iux
Figure 9: First digit and word of the CAR and LAR amount of checks for $562, $72, and $2 misclassified as $862, $92, and $3 by a check processing system. The pixels in red correspond to pixels whose colors differ between the original and attacked image.
16


Reviews:
[{"review_id": "VKbkCFx1_UA", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper describes adversarial attacks on binary (black/white) image classifiers, flipping a classifier result by changing just a few pixels. The authors go on to show that these attacks work on a real-world financial application. \n\nTechnical contributions: \n\n* A query-optimized method for finding adversarial attacks (SCAR).\n* An empirical demonstration of the method\n* A proof (via a straightforward argument) that there are certain types of binary image classifiers that are provably somewhat robust to attack. \n\nRecommendation: reject, but with weak confidence.\n\nReason for score: I see two main issues with this paper, both of which may be cleared up with further discussion.\n\n* The authors make the case that this is a realistic attack on a high-value target. I believe it is worth opening a discussion on whether this raises ethical issues for publication.\n* What is the relation to previous work on sparse adversarial attacks? For example, \"SparseFool: a few pixels make a big difference\" (Modas et al., CVPR 2019) and \"One Pixel Attack for Fooling Deep Neural Networks,\" (Su et al., https://arxiv.org/pdf/1710.08864.pdf). While these do not relate to binary images, they seem to explore a closely related direction. The SCAR algorithm is different, but it seems important to compare with this work. The existence of other sparse methods also calls into question whether this represents a fundamentally new advance.\n\nAreas for improvement:\n* The numeric results (e.g., section 6.2) are hard to read. Putting them in a table would be helpful.\n* A key goal of SCAR is \"hiding the noise\" (p. 4). It would be nice to have more discussion of this goal. If L_0 distance isn't the goal, why not? If there's an implicit perceptual metric at play, why not make it explicit? \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KI6euUN2-K", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nAdversarial attacks for binary image classification are unique from traditional attacks on color images due to its limited available space for perturbation.  This paper proposes an algorithm that efficiently searches for valid attacks (both targeted and untargeted) which cause minimum flipped pixels. The proposed method is evaluated on digit classification, letter classification, and check processing systems. The baselines compared are mostly methods that were originally developed for color images.\n\n\n##########################################################################\n\nReasons for score:\n\nThis is a borderline paper which I tend to vote for rejection. The problem is well-defined and the method is clearly introduced. However, the evaluation of the proposed method is flawed. In particular, the classifiers being attacked seem to be simple and the perceptibility metric is questionable.\n\n\n##########################################################################\n\nPros:\n\nThis paper proposes a novel problem with a non-trivial impact. The problem of binary image classification is also well-motivated. The authors provide a clear explanation of how attacks on the binary images are different from those on color images.\n \nThe authors provide theoretical analysis about the upper limit of pixels that need to be flipped in order to confuse the classifier.\n\nThe related work section is well-organized and the authors clearly state why previous works on color images do not apply to the binary image settings.\n\nBoth the problem formulation and description of the method are well-organized. \n\n\n\n##########################################################################\n\nCons:\n\nIn the problem formulation, one constraint is that the attacks should be imperceptible to humans. This is measured by D_x(x'). However, D_x is parameterized as L0 distance which is not convincing. For instance, slightly increasing the font size of the same letter would incur a large L0 distance while making the modification imperceptible. On the other hand, transforming the letter \"I\" to \"T\" might incur a very small L0 distance while causing perceptible change. The authors should justify the use of L0 or consider experimenting with other parameterizations of D_x.\n\nWhile the classifier used in section 6.3 is carefully selected and well justified, the classifiers used in section 6.2 seems too simple. In addition, instead of attacking plain classifiers, it would make the case stronger if simple defense algorithms are also considered.\n\nThe results in Section 6.5 do not have baseline algorithms to compare with. Also, it would be helpful if a reference to the check processing system is provided.\n\nThis paper lacks a conclusion.\n\n\n##########################################################################\n\nSuggestions & Questions:\n\nIn section 6.1, tau is set to 0.1. How is tau selected and how does tau impact the performance?\n\nPlease consider rewriting the last sentence of the abstract.\n\nPlease address the comments in the cons.\n\n\n#########################################################################", "extra_scores": {}, "extra_text": {}}, {"review_id": "5pfEZ1-sJj", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##################################################################\nSummary:\nThis paper targets at a challenging task of attacking binary image classification. A score-based black-box attack algorithm SCAR is designed to fool Tesseract and US banks’ commercial check processing systems.\n\n##################################################################\nPros:\n(1) This paper introduces a challenging task of attacking binary image classifiers.\n(2) The proposed method SCAR is simple yet effective, which successfully fools Tesseract and the state-of-the-art check processing systems. \n(3) The paper reads smooth and is mostly well-written.\n\n##################################################################\nCons:\n(1)\tThe task of attacking binary image classifiers is a special case of L0 attacks, which also aim at perturbing a small number of pixels. The reviewer thinks that most L0 attack algorithms can be adapted to this binary attack task. The reviewer has concerns that since L0 attack has been widely explored, do we really need to study a much narrow task of binary attack? \n(2)\tThe novelty of the proposed method is somewhat limited. The proposed SCAR greedily selects the modified pixels, by considering spatial and temporal correlations. However, this method is not new enough. Similar heuristic algorithms have also been developed in previous black-box attack algorithms.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3B2lt5tJyn", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main question this paper aims to answer is how vulnerable binary image classification systems are.  This is an important question because of the application of such binary image classifications for check processing, invoice processing, and license plate registration. One also would think that such systems are less vulnerable to adversarial attacks given the simplicity of their inputs and the fact that most adversarial attacks are based on color or grey scale images.  The authors propose an adversarial attack algorithm called SCAR that efficiently flips the binary pixels with reasonable number of queries in order to confuse the classifier to return a desirable label with high confidence. The authors show that the proposed method outperforms the existing baselines on multiple data sets. Very interestingly, they also showed that their algorithm is able to attack the online deposit systems of US bank with a high success rate. Their example of a check with the amount of $401 that is minimally modified for the amount of $701  is quite significant given that the model had to change both the word and numbers on the check. \n\nThe authors also provided some general theories on the existence of binary image classifier provably robust to any attack that modifies large, bounded number of pixels. However, it is not clear to me how this is related to the rest of paper’s discussion. The connection is missing in the paper. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was referred to the ICLR 2021 Ethics Review Committee based on concerns about a potential violation of the ICLR 2021 Code of Ethics (https://iclr.cc/public/CodeOfEthics) raised by reviewers. The paper was carefully reviewed by two committee members, who provided a binding decision. The decision is "Significant concerns (Do not publish)". Details are provided in the Ethics Meta Review. As a result, the paper is rejected based Ethics Review Committee's decision .

The technical review and meta reviewing process moved proceeded independently of the ethics review. The result is as follows:

This paper considers sparse (L0) attacks against binary images analysis systems, in particular OCR.  The major concern of the reviewers seems to be similarity to other methods in the literature, but reviewers did not specify any specific methods to compare to.  Because it was not possible for reviewers to address such vague concerns, and because I believe the authors did a good job differentiating their work in the rebuttal, I think the paper is of good merit.  
Decision: Reject