SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhuoqun Huang
University of Melbourne zhuoqun@unimelb.edu.au
Neil G. Marchant
University of Melbourne  
nmarchant@unimelb.edu.au
Keane Lucas
Carnegie Mellon University keanelucas@cmu.edu
Lujo Bauer
Carnegie Mellon University  
1bauer@cmu.edu
Olga Ohrimenko
University of Melbourne oohrimenko@unimelb.edu.au
Benjamin I. P. Rubinstein
University of Melbourne brubinstein@unimelb.edu.au
Randomized smoothing is a leading approach for constructing classifiers that are certifiably robust against adversarial examples. Existing work on randomized smoothing has focused on classifiers with continuous inputs, such as images, where  $\ell_p$ -norm bounded adversaries are commonly studied. However, there has been limited work for classifiers with discrete or variable-size inputs, such as for source code, which require different threat models and smoothing mechanisms. In this work, we adapt randomized smoothing for discrete sequence classifiers to provide certified robustness against edit distance-bounded adversaries. Our proposed smoothing mechanism randomized deletion (RS-Del) applies random deletion edits, which are (perhaps surprisingly) sufficient to confer robustness against adversarial deletion, insertion and substitution edits. Our proof of certification deviates from the established Neyman-Pearson approach, which is intractable in our setting, and is instead organized around longest common subsequences. We present a case study on malware detection—a binary classification problem on byte sequences where classifier evasion is a well-established threat model. When applied to the popular MalConv malware detection model, our smoothing mechanism RS-Del achieves a certified accuracy of  $91\%$  at an edit distance radius of 128 bytes.
Neural networks have achieved exceptional performance for classification tasks in unstructured domains, such as computer vision and natural language. However, they are susceptible to adversarial examples—inconspicuous perturbations that cause inputs to be misclassified [1, 2]. While a multitude of defenses have been proposed against adversarial examples, they have historically been broken by stronger attacks. For instance, six of nine defense papers accepted for presentation at ICLR2018 were defeated months before the conference took place [3]; another tranche of thirteen defenses were circumvented shortly after [4]. This arms race between attackers and defenders has led the community to investigate certified robustness, which aims to guarantee that a classifier's prediction cannot be changed under a specified set of adversarial perturbations [5, 6].
Most prior work on certified robustness has focused on classifiers with continuous fixed-dimensional inputs, under the assumption of  $\ell_p$ -norm bounded perturbations. A variety of certification methods have been proposed for this setting, including deterministic methods which bound a network's output by convex relaxation [5, 7, 6, 8] or composing layerwise bounds [9, 8, 10-12], and randomized smoothing which obtains high probability bounds for a base classifier smoothed under random noise [13-15]. Randomized smoothing has achieved state-of-the-art certified robustness against  $\ell_2$ -norm bounded perturbations on ImageNet, correctly classifying  $71\%$  of a test set under perturbations with  $\ell_2$ -norm up to  $127/255$  (half maximum pixel intensity) [16]. However, real-world attacks go beyond continuous modalities: a range of attacks have been demonstrated against models with discrete inputs, such as binary executables [17-20], source code [21], and PDF files [22].
Unfortunately, there has been limited investigation of certified robustness for discrete modalities, where prior work has focused on perturbations with bounded Hamming  $(\ell_0)$  distance [23-25], in some cases under additional constraints [26-30]. While this work covers attackers that overwrite content, it does not cover attackers that insert/delete content, which is important for instance in the malware domain [17, 31]. One exception is Zhang et al.'s method for certifying robustness under string transformation rules that may include insertion and deletion, however their approach is only feasible for small rule sets and is limited to LSTMs [32]. Liu et al. [33] also study certification for a threat model that includes insertion/deletion in the context of point cloud classification.
In this paper we develop a comprehensive treatment of edit distance certifications for sequence classifiers. We consider input sequences of varying length on finite domains. We cover threat models where an adversary can arbitrarily perturb sequences by substitutions, insertions, and deletions or any subset of these operations. Moreover, our framework encompasses adversaries that apply edits to blocks of tokens at a time. Such threat models are motivated in malware analysis, where attackers are likely to edit semantic objects in an executable (e.g., whole instructions), rather than at the byte-level. We introduce tunable decision thresholds, to adjust decision boundaries while still forming sound certifications. This permits trading-off misclassification rates and certification radii between classes, and is useful in settings where adversaries have a strong incentive to misclassify malicious instances as benign [34, 35].
We accomplish our certifications using randomized smoothing [13, 14], which we instantiate with a general-purpose deletion smoothing mechanism called RS-Del. Perhaps surprisingly, RS-Del does not need to sample all edit operations covered by our certifications. By smoothing using deletions only, we simplify our certification analysis and achieve the added benefit of improved efficiency by querying the base classifier with shorter sequences. RS-Del is compatible with arbitrary base classifiers, requiring only oracle query access such as via an inference API. To prove our robustness certificates, we have to deviate from the standard Neyman-Pearson approach to randomized smoothing, as it is intractable in our setting. Instead, we organize our proof around a representative longest common subsequence (LCS): the LCS serves as a reference point in the smoothed space between an input instance and a neighboring instance, allowing us to bound the confidence of the smoothed model between the two.
Finally, we present a comprehensive evaluation of RS-Del in the malware analysis setting<sup>1</sup>. We investigate tradeoffs between accuracy and the size of our certificates for varying levels of deletion, and observe that RS-Del can achieve a certified accuracy of  $91\%$  at an edit distance radius of 128 bytes using on the order of  $10^{3}$  model queries. By comparison, a brute force certification would require in excess of  $10^{308}$  model queries to certify the same radius. We also demonstrate asymmetric certificates (favouring the malicious class) and certificates covering edits at the machine instruction level by leveraging chunking and information from a disassembler. Finally, we assess the empirical robustness of RS-Del to several attacks from the literature, where we observe a reduction in attack success rates.
Sequence classification Let  $\mathcal{X} = \Omega^{\star}$  represent the space of finite-length sequences (including the empty sequence) whose elements are drawn from a finite set  $\Omega$ . For a sequence  $x \in \mathcal{X}$ , we denote its length by  $|x|$  and the element at position  $i$  by  $x_{i}$  where  $i$  runs from 1 to  $|x|$ . We consider classifiers that map sequences in  $\mathcal{X}$  to  $K$  classes in  $\mathcal{Y} = \{0, \dots, K - 1\}$ . For example, in our case study on
<sup>1</sup>Our implementation is available at https://github.com/Dovermore/randomized-deletion.
malware detection, we take  $\mathcal{X}$  to be the space of byte sequences (binaries), and  $\Omega$  to be the set of bytes and  $\mathcal{Y}$  to be  $\{0,1\}$  where 0 and 1 denote benign and malicious binaries respectively.
Robustness certification Given a classifier  $f \colon \mathcal{X} \to \mathcal{Y}$ , an input  $x \in \mathcal{X}$  and a neighborhood  $\mathcal{N}(x) \subset \mathcal{X}$  around  $x$ , a robustness certificate is a guarantee that the classifier's prediction is constant in the neighborhood, i.e.,
$$
f (\boldsymbol {x}) = f \left(\boldsymbol {x} ^ {\prime}\right) \quad \forall \boldsymbol {x} ^ {\prime} \in \mathcal {N} (\boldsymbol {x}). \tag {1}
$$
When the neighborhood corresponds to a ball of radius  $r$  centered on  $x$ , we make the dependence on  $r$  explicit by writing  $\mathcal{N}_r(x)$ . We adopt the paradigm of conservative, probabilistic certification, which is a natural fit for randomized smoothing [14]. Under this paradigm, a certifier may either assert that (1) holds with high probability, or decline to assert whether (1) holds or not.
Edit distance robustness Most existing work on robustness certification focuses on classifiers that operate on fixed-dimensional inputs in  $\mathbb{R}^d$ , where the neighborhood of certification is an  $\ell_p$ -ball [6, 5, 13-15]. However,  $\ell_p$  robustness is not well motivated for sequence classifiers:  $\ell_p$  neighborhoods are limited to constant-length sequences, and the norm is ill-defined for sequences with non-numeric elements. For example, an  $\ell_p$  neighborhood around a byte sequence like  $x = (78, 7A, 2D, 00)$  must include sequences additively perturbed by real-valued sequences like  $\delta = (-0.09, 0.07, 0.01, 0.1)$ , which clearly results in a type mismatch. Even if one focuses on robustness for length-preserving sequence perturbations, the  $\ell_p$  neighborhood is a poor choice because it excludes sequences that are even slightly misaligned. Motivated by these shortcomings, we consider edit distance robustness.
Edit distance is a natural measure for comparing sequences. Given a set of elementary edit operations (ops)  $O$ , the edit distance  $\mathrm{dist}_O(\boldsymbol{x}, \boldsymbol{x}')$  is the minimum number of ops required to transform sequence  $\boldsymbol{x}$  into  $\boldsymbol{x}'$ . We consider three ops: delete a single element (del), insert a single element (ins), and substitute one element with another (sub). For generality, we allow  $O$  to be any combination of these ops. When  $O = \{\mathrm{del}, \mathrm{ins}, \mathrm{sub}\}$  the edit distance is known as Levenshtein distance [36]. A primary goal of this paper is to produce edit distance robustness certificates, where the neighborhood of certification is an edit distance ball  $\mathcal{N}_r(\boldsymbol{x}) = \{\boldsymbol{x}' \in \mathcal{X} : \mathrm{dist}_O(\boldsymbol{x}', \boldsymbol{x}) \leq r\}$ .
Threat model We consider an adversary that has full knowledge of our base and smoothed models, source of randomness, certification scheme, and possesses unbounded computation. The attacker makes edits from  $O$  up to some budget, in order to misclassify a target  $x$ . In the context of our experimental case study on malware detection, edit distance is a reasonable proxy for the cost of running evasion attacks that iteratively apply localized functionality-preserving edits (e.g., [19, 37, 38, 17, 39]). Since the edit distance scales roughly linearly with the number of attack iterations, the adversary has an incentive to minimize edit distance for these attacks.
3 RS-Del: Randomized deletion smoothing
In this section, we propose a method for constructing sequence classifiers that are certifiably robust under bounded edit distance perturbations. Our method RS-Del extends randomized smoothing with a novel deletion mechanism and tunable decision thresholds. We review randomized smoothing in Section 3.1 and describe our deletion mechanism in Section 3.2, along with its practical aspects in Section 3.3. We summarize the certified robustness guarantees of our method in Table 1 and defer their derivation to Section 4.
3.1 Randomized smoothing
Let  $f_{\mathrm{b}}: \mathcal{X} \to \mathcal{Y}$  be a base classifier and  $\phi: \mathcal{X} \to \mathcal{D}(\mathcal{X})$  be a smoothing mechanism that maps inputs to a distributions over (perturbed) inputs. Randomized smoothing composes  $f_{\mathrm{b}}$  and  $\phi$  to construct a new smoothed classifier  $f: \mathcal{X} \to \mathcal{Y}$ . For any input  $x \in \mathcal{X}$ , the smoothed classifier's prediction is
$$
f (\boldsymbol {x}) := \underset {y \in \mathcal {Y}} {\arg \max } \left\{p _ {y} (\boldsymbol {x}; f _ {\mathrm {b}}, \phi) - \eta_ {y} \right\}, \tag {2}
$$
where  $\pmb {\eta} = \{\eta_y\}_{y\in \mathcal{Y}}$  is a set of real-valued tunable decision thresholds and
$$
p _ {y} (\boldsymbol {x}; f _ {\mathrm {b}}, \phi) = \Pr_ {\boldsymbol {z} \sim \phi (\boldsymbol {x})} [ f _ {\mathrm {b}} (\boldsymbol {z}) = y ] \tag {3}
$$
is the probability that the base classifier  $f_{\mathrm{b}}$  predicts class  $y$  for a perturbed input drawn from  $\phi(\boldsymbol{x})$ . We omit the dependence of  $p_y$  on  $f_{\mathrm{b}}$  and  $\phi$  where it is clear from context.
The viability of randomized smoothing as a method for achieving certified robustness is strongly dependent on the smoothing mechanism. Ideally, the mechanism should be chosen to yield a smoothed classifier with improved robustness under the chosen threat model, while minimizing any drop in accuracy compared to the base classifier. The mechanism should also be amenable to analysis, so that a tractable robustness certificate can be derived.
Remark 1. Previous definitions of randomized smoothing (e.g., [13, 14]) do not incorporate decision thresholds, and effectively assume  $\eta = 0$ . We introduce decision thresholds as a way to trade off error rates and robustness between classes. This is useful when there is asymmetry in misclassification costs across classes. For instance, in our case study on malware detection, robustness of benign examples is less important because adversaries have limited incentive to trigger misclassification of benign examples [34, 35, 40]. We note that the base classifier may also be equipped with decision thresholds, which provide another degree of freedom to trade off error rates between classes.
3.2 Randomized deletion mechanism
We propose a smoothing mechanism that achieves certified edit distance robustness. Our smoothing mechanism perturbs a sequence by deleting elements at random, and is called randomized deletion, or RS-Del for short.
Consider a sequence  $\pmb{x} \in \mathcal{X}$  whose elements are indexed by the set  $[\pmb{x}] = \{1, \dots, |\pmb{x}|\}$ . We specify the distribution of  $\phi(\pmb{x})$  for RS-Del in two stages. In the first stage, a random edit  $\epsilon$  is drawn from a distribution  $G(\pmb{x})$  over the space of possible edits to  $\pmb{x}$ , denoted  $\mathcal{E}(\pmb{x})$ . Since we only consider deletions for smoothing, any edit can be represented by the set of element indices in  $[\pmb{x}]$  that remain after deletion. Hence  $\mathcal{E}(\pmb{x})$  is taken as the powerset of  $[\pmb{x}]$ . We specify edit distribution  $G(\pmb{x})$  so that each element is deleted i.i.d. with probability  $p_{\mathrm{del}} \in (0, 1)$ :
$$
\Pr [ G (\boldsymbol {x}) = \epsilon ] = \prod_ {i = 1} ^ {| \boldsymbol {x} |} p _ {\mathrm {d e l}} ^ {\mathbf {1} _ {i \notin \epsilon}} (1 - p _ {\mathrm {d e l}}) ^ {\mathbf {1} _ {i \in \epsilon}}, \tag {4}
$$
where  $\mathbf{1}_A$  denotes the indicator function, which returns 1 if  $A$  evaluates to true and 0 otherwise. In the second stage, the edit  $\epsilon$  is applied to  $\pmb{x}$  to yield the perturbed sequence:
$$
\boldsymbol {z} = \operatorname {a p p l y} (\boldsymbol {x}, \epsilon) := \left(x _ {\epsilon_ {(i)}}\right) _ {i = 1 \dots | \epsilon |}, \tag {5}
$$
where  $\epsilon_{(i)}$  denotes the  $i$ -th smallest index in  $\epsilon$ . The perturbed sequence  $\mathbf{z}$  is guaranteed to be a subsequence of  $\mathbf{x}$ . Putting both stages together, the distribution of  $\phi(\mathbf{x})$  is
$$
\Pr [ \phi (\boldsymbol {x}) = \boldsymbol {z} ] = \sum_ {\epsilon \in \mathcal {E} (\boldsymbol {x})} \Pr [ G (\boldsymbol {x}) = \epsilon ] \mathbf {1} _ {\text {a p p l y} (\boldsymbol {x}, \epsilon) = \boldsymbol {z}}. \tag {6}
$$
Remark 2. It may be surprising that we are proposing a smoothing mechanism for certified edit distance robustness that does not use the full set of edit ops  $O$  covered by the threat model. It is a misconception that randomized smoothing requires perfect alignment between the mechanism and the threat model. All that is needed from a robustness perspective, is for the mechanism to return distributions that are statistically close for any pair of inputs that are close in  $O$  edit distance; this can be achieved solely with deletion. In fact, perfect alignment is known to be suboptimal for some  $\ell_p$  threat models [15]. Our deletion mechanism leads to a tractable robustness certificate covering the full set of edit ops (see Section 4). Moreover while benefiting robustness, our empirical results show that our deletion mechanism has only a minor impact on accuracy (see Section 5). Finally, our deletion mechanism reduces the length of the input, which is beneficial for computational efficiency (see Appendix G). This is not true in general for mechanisms employing insertions/substitutions.
3.3 Practical considerations
We now discuss considerations for implementing and certifying RS-Del in practice. In doing so, we reference theoretical results for certification, which are covered later in Section 4.
1 Sample perturbed inputs for prediction estimate:  $S\gets \{\pmb {z}_i\sim \phi (\pmb {x})\}_{i = 1\dots n_{\mathrm{pred}}}$
2 Confidence scores:  $\hat{\mu}_y\gets \frac{1}{|S|}\sum_{z\in S}\mathbf{1}_{f_{\mathrm{b}}(z)} = y$
3 Prediction:  $\hat{y} \gets \arg \max_{y \in \mathcal{Y}} \{\hat{\mu}_y - \eta_y\}$
4 Sample perturbed inputs for lower confidence bound (LCB):  $S^{\prime}\gets \{\pmb {z}_i\sim \phi (\pmb {x})\}_{i = 1\dots n_{\mathrm{bnd}}}$
5 LCB:  $\underline{\mu}_{\hat{y}} \gets \operatorname{BinLCB}(\sum_{z \in S'} \mathbf{1}_{f_{\mathrm{b}}(z) = \hat{y}}, |S'|, \alpha)$
6 Compute certified radius  $r^{\star}$  using Table 1
7 if  $\mu_{\hat{y}} < \eta_{\hat{y}}$  then return abstain
8 else return prediction  $\hat{y}$ , radius  $r^{\star}$
Figure 1: Probabilistic certification of RS-Del. Here  $\pmb{x}$  is the input sequence,  $f_{\mathrm{b}}$  is the base classifier,  $p_{\mathrm{del}}$  is the deletion probability,  $\pmb{\eta}$  is the set of decision thresholds,  $\alpha$  is the significance level, and  $n_{\mathrm{pred}}, n_{\mathrm{bind}}$  are sample sizes.  $\operatorname{BinLCB}(k, n, \alpha)$  returns a lower confidence bound for  $p$  at level  $\alpha$  given  $k \sim \operatorname{Bin}(n, p)$ .
Edit ops O	Certified radius r*
del	ins	sub
✓	[log 1-μy/1-νy/log pdel]
✓	[log μy/μy/log pdel]
✓	✓	[log μy/μy/log pdel]
✓	✓	✓	[log(1+νy-μy)/log pdel]
✓	[log(1+νy-μy)/log pdel]
✓	✓	[log(1+νy-μy)/log pdel]
✓	✓	[log(1+νy-μy)/log pdel]
Table 1: Edit distance robustness certificates for RS-Del as a function of the edit ops  $O$  used to define the edit distance. Here  $\mu_y$  is the confidence for predicted class  $y$  and  $\nu_y$  is a threshold derived from  $\eta$  defined in (10).
Probabilistic certification Randomized smoothing does not generally support exact evaluation of the classifier's confidence  $\mu_y\coloneqq p_y(\pmb {x})$  , which is required for exact prediction and exact evaluation of the certificates we develop in Section 4. While  $\mu_{y}$  can be evaluated exactly for RS-Del by enumerating over the possible edits  $\mathcal{E}(\pmb {x})$  , the computation scales exponentially in  $|\pmb {x}|$  (see Appendix A). Since this is infeasible for even moderately-sized  $\pmb{x}$  , we follow standard practice in randomized smoothing and estimate  $\mu_{y}$  with a lower confidence bound using Monte Carlo sampling [14]. This procedure is described in pseudocode in Figure 1: lines 1-3 estimate the predicted class  $y = f(x)$  , lines 4-5 compute a  $1 - \alpha$  lower confidence bound on  $\mu_{y}$  , and line 6 uses this information and the results in Table 1 to compute a probabilistic certificate that holds with probability  $1 - \alpha$  . If the lower confidence bound on  $\mu_{y}$  exceeds the corresponding detection threshold  $\eta_{y}$  , the prediction and certificate are returned (line 8), otherwise we abstain due to lack of statistical significance (line 7).
Training While randomized smoothing is compatible with any base classifier, it generally performs poorly for conventionally-trained classifiers [13]. We therefore train base classifiers specifically to be used with RS-Del, by replacing original sequences with perturbed sequences (drawn from  $\phi$ ) at training time. This has been shown to achieve good empirical performance in prior work [14].
Sequence chunking So far, we have described edit distance robustness and RS-Del assuming edits are applied at the level of sequence elements. However in some applications it may be reasonable to assume edits are applied at a coarser level, to contiguous chunks of sequence elements. For example, in malware analysis, one can leverage information from a disassembler to group low-level sequence elements (bytes) into more semantically meaningful chunks, such as machine instructions, addresses and header fields (see Appendix C.3). Our methods are compatible with chunking—we simply reinterpret the sequence as a sequence of chunks, rather than a sequence of lower-level elements. This can yield a tighter robustness certificate, since edits within chunks are excluded.
4 Edit distance robustness certificate
We now derive an edit distance robustness certificate for RS-Del. We present the derivation in three parts: Section 4.1 provides an outline, Section 4.2 derives a lower bound on RS-Del's confidence score and Section 4.3 uses the bound to complete the derivation. All proofs are presented in Appendix B.
4.1 Derivation outline
Following prior work [14, 13, 41], we derive an edit distance robustness certificate that relies on limited information about RS-Del. We allow the certificate to depend on the input  $\pmb{x}$ , the smoothed prediction  $y = f(\pmb{x})$ , the confidence score  $\mu_y = p_y(\pmb{x};f_{\mathrm{b}})$ , the decision threshold  $\eta_y$ , and the architecture of  $f$ , including the deletion smoothing mechanism  $\phi$ , but excluding the architecture of
the base classifier  $f_{\mathrm{b}}$ . Limiting the dependence in this way improves tractability and ensures that the certificate is applicable for any choice of base classifier  $f_{\mathrm{b}}$ . Formally, the only information we assume about  $f_{\mathrm{b}}$  is that it is some classifier in the feasible base classifier set:
$$
\mathcal {F} (\boldsymbol {x}, \mu_ {y}) = \left\{h \in \mathcal {X} \rightarrow \mathcal {Y}: \mu_ {y} = p _ {y} (\boldsymbol {x}; h) \right\}. \tag {7}
$$
Recall that an edit distance robustness certificate at radius  $r$  for a classifier  $f$  at input  $\pmb{x}$  is a guarantee that  $f(\pmb{x}) = f(\bar{\pmb{x}})$  for any perturbed input  $\bar{\pmb{x}}$  in the neighborhood  $\mathcal{N}_r(\pmb{x}) = \{\bar{\pmb{x}} \in \mathcal{X} : \mathrm{dist}_O(\bar{\pmb{x}}, \pmb{x}) \leq r\}$ . We observe that this guarantee holds for RS-Del in the limited information setting iff the  $\pmb{\eta}$ -adjusted confidence for predicted class  $y$  exceeds the  $\pmb{\eta}$ -adjusted confidence for any other class for all perturbed inputs  $\bar{\pmb{x}}$  and feasible base classifiers  $h$ :
$$
p _ {y} (\bar {\boldsymbol {x}}; h) - \eta_ {y} \geq \max  _ {y ^ {\prime} \neq y} \left\{p _ {y ^ {\prime}} \left(\bar {\boldsymbol {x}}; h\right) - \eta_ {y ^ {\prime}} \right\} \quad \forall \bar {\boldsymbol {x}} \in \mathcal {N} _ {r} (\boldsymbol {x}), h \in \mathcal {F} (\boldsymbol {x}, \mu_ {y}). \tag {8}
$$
To avoid dependence on the confidence of the runner-up class, which is inefficient for probabilistic certification, we work with the following more convenient condition.
Proposition 3. A sufficient condition for (8) is  $\rho (\pmb {x},\mu_y)\geq \nu_y(\pmb {\eta})$  where