SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[87] Microsoft Corporation. PE format, June 2022. URL https://docs.microsoft.com/en-us/windows/win32-debug/pe-format.
[88] VirusShare.com. VirusShare.com. URL https://virusshare.com/.
[89] Matthew G. Schultz, Eleazar Eskin, Erez Zadok, and Salvatore J. Stolfo. Data mining methods for detection of new malicious executables. In IEEE Symposium on Security and Privacy, S&P, 2001.
[90] J. Zico Kolter and Marcus A. Maloof. Learning to Detect and Classify Malicious Executables in the Wild. Journal of Machine Learning Research, 7(99):2721-2744, 2006.
[91] Chocolatey Software. Chocolatey software. URL https://chocolatey.org/.
[92] Chocolatey Software. Chocolately software docs | security. URL https://docs.chocolate.org/en-us/information/security.
[93] Luca Demetrio and Battista Biggio. secml-malware: Pentesting windows malware classifiers with adversarial EXEmples in python. arXiv preprint:2104.12848, 2021.
[94] Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers. In Advances in Neural Information Processing Systems, NeurIPS, 2019.
[95] Daniel Lowd and Christopher Meek. Good word attacks on statistical spam filters. In Second Conference on Email and Anti-Spam, CEAS, 2005.
[96] Yan Scholten, Jan Schuchardt, Simon Geisler, Aleksandar Bojchevski, and Stephan Gunnemann. Randomized message-interception smoothing: Gray-box certificates for graph neural networks. In Advances in Neural Information Processing Systems, NeurIPS, 2022.
[97] Jinyuan Jia, Binghui Wang, Xiaoyu Cao, and Neil Zhenqiang Gong. Certified robustness of community detection against adversarial structural perturbation via randomized smoothing. In Proceedings of The Web Conference 2020, WWW, pages 2718-2724, 2020. doi: 10.1145/3366423.3380029.
A Brute-force edit distance certification
In this appendix, we show that an edit distance certification mechanism based on brute-force search is computationally infeasible. Suppose we are interested in issuing an edit distance certificate at radius  $r$  for a sequence classifier  $f$  at input  $\mathbf{x}$ . Recall from (1) that in order to issue a certificate, we must show there exists no input  $\bar{\mathbf{x}}$  within the edit distance neighborhood  $\mathcal{N}_r(\mathbf{x})$  that would change  $f$ 's prediction. This problem can theoretically be tackled in a brute-force manner, by querying  $f$  for all inputs in  $\mathcal{N}_r(\mathbf{x})$ . In the best case, this would take time linear in  $|\mathcal{N}_r(\mathbf{x})|$ , assuming  $f$  responds to queries in constant time. However the following lower bound [68], shows that the size of the edit distance neighborhood is too large even in the best case:
$$
| \mathcal {N} _ {r} (\boldsymbol {x}) | \geq \sum_ {i = 0} ^ {r} 2 5 5 ^ {i} \sum_ {j = i - r} ^ {r} \binom {| \boldsymbol {x} | + j} {i} \geq 2 5 5 ^ {r}.
$$
For example, applying the loosest bound that is independent of  $\mathbf{x}$ , we see that brute-force certification at radius  $r = 128$  would require in excess of  $255^r \approx 10^{308}$  queries to  $f$ . In contrast, our probabilistic certification mechanism (Figure 1) makes  $n_{\mathrm{pred}} + n_{\mathrm{bnd}}$  queries to  $f$ , and we can provide high probability guarantees when the number of queries is of order  $10^3$  or  $10^4$ .
In this appendix, we provide proofs of the theoretical results stated in Section 4.
B.1 Proof of Proposition 3
A sufficient condition for (8) is
$$
\min  _ {\bar {x} \in \mathcal {N} _ {r} (\boldsymbol {x})} \min  _ {h \in \mathcal {F} (\boldsymbol {x})} p _ {y} (\bar {x}; h) \geq \max  _ {\bar {x} \in \mathcal {N} _ {r} (\boldsymbol {x})} \max  _ {h \in \mathcal {F} (\boldsymbol {x})} \left(\eta_ {y} + \max  _ {y ^ {\prime} \neq y} p _ {y ^ {\prime}} (\bar {x}; h) - \min  _ {y ^ {\prime} \neq y} \eta_ {y ^ {\prime}}\right). \tag {17}
$$
We first consider the multi-class case where  $|\mathcal{V}| > 2$ . If  $\eta_y \geq \min_{y' \neq y} \eta_{y'}$ , then  $p_y(\bar{x}; h) \geq \max_{y' \neq y} p_{y'}(\bar{x}; h)$  by (17) and we can upper-bound  $\max_{y' \neq y} p_{y'}(\bar{x}; h)$  by  $\frac{1}{2}$ . On the other hand, if  $\eta_y \geq \min_{y' \neq y} \eta_{y'}$ , we can only upper-bound  $\max_{y' \neq y} p_{y'}(\bar{x}; h)$  by 1. Thus when  $|\mathcal{V}| > 2$  (17) implies
$$
\min_{\bar{\boldsymbol{x}}\in \mathcal{N}_{r}(\boldsymbol {x})}\min_{h\in \mathcal{F}(\boldsymbol {x})}p_{y}(\bar{\boldsymbol{x}};h)\geq \left\{ \begin{array}{ll}\frac{1}{2} +\eta_{y} - \min_{y^{\prime}\neq y}\eta_{y^{\prime}}, & \eta_{y}\geq \min_{y^{\prime}\neq y}\eta_{y^{\prime}},\\ 1 + \eta_{y} - \min_{y^{\prime}\neq y}\eta_{y^{\prime}}, & \eta_{y} <   \min_{y^{\prime}\neq y}\eta_{y^{\prime}}. \end{array} \right.
$$
Next, we consider the binary case where  $|\mathcal{Y}| = 2$ . Since the confidence sum to 1, we have  $\max_{y' \neq y} p_{y'}(\bar{x}; h) = 1 - p_y(\bar{x}; h)$ . Putting this in (17) implies
$$
\min  _ {\bar {\boldsymbol {x}} \in \mathcal {N} _ {r} (\boldsymbol {x})} \min  _ {h \in \mathcal {F} (\boldsymbol {x})} p _ {y} (\bar {\boldsymbol {x}}; h) \geq \frac {1 + \eta_ {y} - \min  _ {y ^ {\prime} \neq y} \eta_ {y ^ {\prime}}}{2}.
$$
Let  $r_S \colon S \to \{1, \dots, |S|\}$  be a bijection that returns the rank of an element in an ordered set  $S$ . Let  $\dot{r}_S \colon 2^S \to 2^{\{1, \dots, |S|\}}$  be an elementwise extension of  $r_S$  that returns a set of ranks for an ordered set of elements—i.e.,  $\dot{r}_S(U) = \{r_S(i) : i \in U\}$  for  $U \subseteq S$ . We claim  $m(\bar{\epsilon}) = \dot{r}_{\epsilon^\star}^{-1}(\dot{r}_{\bar{\epsilon}^\star}(\bar{\epsilon}))$  is a bijection that satisfies the required property.
To prove the claim, we note that  $m$  is a bijection from  $2^{\bar{\epsilon}^{\star}}$  to  $2^{\epsilon^{\star}}$  since it is a composition of bijections  $\dot{r}_{\bar{\epsilon}^{\star}}: 2^{\bar{\epsilon}^{\star}} \to 2^{\{1,\dots,l\}}$  and  $\dot{r}_{\epsilon^{\star}}^{-1}: 2^{\{1,\dots,l\}} \to 2^{\epsilon^{\star}}$  where  $l = |\bar{\epsilon}^{\star}| = |\epsilon^{\star}|$ . Next, we observe that  $\dot{r}_{\bar{\epsilon}^{\star}}(\bar{\epsilon})$  relabels indices in  $\bar{\epsilon}$  so they have the same effect when applied to  $z^{\star}$  as  $\bar{\epsilon}$  on  $\bar{x}$  (this also holds for  $\dot{r}_{\epsilon^{\star}}$  and  $\epsilon$ ). Thus
$$
\begin{array}{l} \operatorname {a p p l} (\bar {x}, \bar {\epsilon}) = \operatorname {a p p l} (z ^ {\star}, \dot {r} _ {\bar {\epsilon} ^ {\star}} (\bar {\epsilon})) \\ = \operatorname {a p p l y} \left(\boldsymbol {z} ^ {\star}, \dot {r} _ {\epsilon^ {\star}} \left(\dot {r} _ {\epsilon} ^ {- 1} \left(\dot {r} _ {\bar {\epsilon} ^ {\star}} (\bar {\epsilon})\right)\right)\right) \\ = \operatorname {a p p l y} (\boldsymbol {x}, m (\bar {\epsilon})) \\ \end{array}
$$
as required. To prove the final statement, we use (4), (5) and (12) to write
$$
\begin{array}{l} \frac {s (\bar {\epsilon} , \bar {\boldsymbol {x}} ; h)}{s (\epsilon , \boldsymbol {x} ; h)} = \frac {\mathbf {1} _ {h \left(\operatorname {a p p l y} (\bar {\boldsymbol {x}} , \bar {\epsilon})\right) = y} p _ {\mathrm {d e l}} ^ {| \bar {\boldsymbol {x}} | - | \bar {\epsilon} |} (1 - p _ {\mathrm {d e l}}) ^ {| \bar {\epsilon} |}}{\mathbf {1} _ {h \left(\operatorname {a p p l y} (\boldsymbol {x} , \epsilon)\right) = y} p _ {\mathrm {d e l}} ^ {| \boldsymbol {x} | - | \epsilon |} (1 - p _ {\mathrm {d e l}}) ^ {| \epsilon |}} \\ = \frac {p _ {\mathrm {d e l}} ^ {| \overline {{\boldsymbol {x}}} | - | \boldsymbol {z} |} (1 - p _ {\mathrm {d e l}}) ^ {| \boldsymbol {z} |} \mathbf {1} _ {h (\boldsymbol {z}) = y}}{p _ {\mathrm {d e l}} ^ {| \boldsymbol {x} | - | \boldsymbol {z} |} (1 - p _ {\mathrm {d e l}}) ^ {| \boldsymbol {z} |} \mathbf {1} _ {h (\boldsymbol {z}) = y}} \\ = p _ {\mathrm {d e l}} ^ {| \bar {\boldsymbol {x}} | - | \boldsymbol {x} |}, \\ \end{array}
$$
where the second last line follows from the fact that apply  $(\bar{x},\bar{\epsilon}) = \mathrm{apply}(\pmb {x},\epsilon) = \pmb{z}$
B.3 Proof of Theorem 5
Let  $\overline{\epsilon}^{\star}$  and  $\epsilon^{\star}$  be defined as in Lemma 4. We derive an upper bound on the sum over  $\epsilon \in 2^{\epsilon^{\star}}$  that appears in (13). Observe that
$$
\begin{array}{l} \sum_ {\epsilon \notin 2 ^ {\epsilon^ {*}}} s (\epsilon , \boldsymbol {x}; h) \leq \sum_ {\epsilon \notin 2 ^ {\epsilon^ {*}}} \Pr \left[ G (\boldsymbol {x}) = \epsilon \right] \\ = 1 - \sum_ {\epsilon \in 2 ^ {\epsilon^ {\star}}} \Pr \left[ G (\boldsymbol {x}) = \epsilon \right] \\ = 1 - p_{\mathsf{del}}^{|\boldsymbol {x}| - |\epsilon^{\star}|}\sum_{|\epsilon | = 0}^{|\epsilon^{\star}|}\binom {|\epsilon^{\star}|}{|\epsilon|}p_{\mathsf{del}}^{|\epsilon^{\star}| - |\epsilon |}(1 - p_{\mathsf{del}})^{|\epsilon |} \\ = 1 - p _ {\mathrm {d e l}} ^ {| \boldsymbol {x} | - | \epsilon^ {\star} |}, \tag {18} \\ \end{array}
$$
where the first line follows from the inequality  $\mathbf{1}_{h(\mathrm{apply}(\boldsymbol{x},\epsilon) = y)} \leq 1$ ; the second line follows from the law of total probability; the third line follows by constraining the indices  $\{1,\dots ,|\boldsymbol {x}|\} \setminus \overline{\epsilon}^{\star}$  to be deleted; and the last line follows from the normalization of the binomial distribution. Putting (18) and  $\sum_{\bar{\epsilon}\in 2^{\bar{\epsilon}^{\star}}}s(\bar{\epsilon},\bar{\boldsymbol{x}};h)\geq 0$  in (13) gives
$$
\begin{array}{l} p _ {y} (\bar {\boldsymbol {x}}; h) \geq p _ {\mathrm {d e l}} ^ {| \bar {\boldsymbol {x}} | - | \boldsymbol {x} |} \left(\mu_ {y} - 1 - p _ {\mathrm {d e l}} ^ {| \boldsymbol {x} | - | \epsilon^ {\star} |}\right) \\ = p _ {\mathrm {d e l}} ^ {| \bar {\boldsymbol {x}} | - | \boldsymbol {x} |} \left(\mu_ {y} - 1 - p _ {\mathrm {d e l}} ^ {\frac {1}{2} (\operatorname {d i s t} _ {\mathrm {L C S}} (\bar {\boldsymbol {x}}, \boldsymbol {x}) + | \boldsymbol {x} | - | \bar {\boldsymbol {x}} |)\right). \tag {19} \\ \end{array}
$$
In the second line above we use the following relationship between the LCS distance and the length of the LCS  $|z^{\star}| = |\epsilon^{\star}|$ :
$$
\operatorname {d i s t} _ {\mathrm {L C S}} (\bar {\boldsymbol {x}}, \boldsymbol {x}) = | \bar {\boldsymbol {x}} | + | \boldsymbol {x} | - 2 | \boldsymbol {z} ^ {\star} |.
$$
Since (19) is independent of the base classifier  $h$ , the lower bound on  $\rho(\bar{\pmb{x}}, \pmb{x}, \mu_y)$  follows immediately.
B.4 Proof of Corollary 6
Since the length of  $\pmb{x}$  can only be changed by inserting or deleting elements in  $\bar{\pmb{x}}$ , we have
$$
\left| \boldsymbol {x} \right| - \left| \bar {\boldsymbol {x}} \right| = n _ {\text {i n s}} - n _ {\text {d e l}}. \tag {20}
$$
We also observe that the LCS distance can be uniquely decomposed in terms of the counts of insertion ops  $m_{\mathrm{ins}}$  and deletion ops  $m_{\mathrm{del}}$ :  $\mathrm{dist}_{\mathrm{LCS}}(\bar{\boldsymbol{x}}, \boldsymbol{x}) = m_{\mathrm{del}} + m_{\mathrm{ins}}$ . These counts can in turn be related to the given decomposition of edit ops counts for generalized edit distance. In particular, any substitution must be expressed as an insertion and deletion under LCS distance, which implies  $m_{\mathrm{ins}} = n_{\mathrm{ins}} + n_{\mathrm{sub}}$  and  $m_{\mathrm{del}} = n_{\mathrm{del}} + n_{\mathrm{sub}}$ . Thus we have
$$
\operatorname {d i s t} _ {\mathrm {L C S}} (\bar {\boldsymbol {x}}, \boldsymbol {x}) = n _ {\mathrm {d e l}} + n _ {\mathrm {i n s}} + 2 n _ {\mathrm {s u b}}. \tag {21}
$$
Substituting (20) and (21) in (14) gives the required result.
Eliminating  $n_{\mathrm{sub}}$  from (16) using the constraint  $n_{\mathrm{sub}} = r - n_{\mathrm{del}} - n_{\mathrm{ins}}$ , we obtain a minimization problem in two variables:
$$
\min  _ {n _ {\text {i n s}}, n _ {\text {d e l}} \in \mathbb {N} _ {0}} \psi (n _ {\text {i n s}}, n _ {\text {d e l}})
$$
$$
s. t. \quad 0 \leq n _ {\text {i n s}} + n _ {\text {d e l}} \leq r
$$
where  $\psi (n_{\mathrm{ins}},n_{\mathrm{del}}) = p_{\mathrm{del}}^{n_{\mathrm{del}} - n_{\mathrm{ins}}}\left(\mu_y - 1 + p_{\mathrm{del}}^{r - n_{\mathrm{del}}}\right)$ . Observe that  $\psi$  is monotonically increasing in  $n_{\mathrm{ins}}$  and  $n_{\mathrm{del}}$ :
$$
\frac {\psi \left(n _ {\text {i n s}} + 1 , n _ {\text {d e l}}\right)}{\psi \left(n _ {\text {i n s}} , n _ {\text {d e l}}\right)} = \frac {1}{p _ {\text {d e l}}} \geq 1
$$
$$
\frac {\psi \left(n _ {\text {i n s}} , n _ {\text {d e l}} + 1\right)}{\psi \left(n _ {\text {i n s}} , n _ {\text {d e l}}\right)} = \frac {\left(\mu_ {y} - 1\right) p _ {\text {d e l}} ^ {n _ {\text {d e l}} + 1} + p _ {\text {d e l}} ^ {r}}{\left(\mu_ {y} - 1\right) p _ {\text {d e l}} ^ {n _ {\text {d e l}}} + p _ {\text {d e l}} ^ {r}} \geq 1,
$$
where the second inequality follows since we only consider  $r$  and  $\mu_y$  such that the numerator and denominator are positive. Thus the minimizer is  $(n_{\mathrm{ins}}^{\star}, n_{\mathrm{del}}^{\star}, n_{\mathrm{sub}}^{\star}) = (0, 0, r)$  and we find  $\rho(\pmb{x}; \mu_y) = \mu_y - 1 + p_{\mathrm{del}}^r$ . The expression for the certified radius follows by solving  $\rho(\pmb{x}; \mu_y) \geq \nu_y(\pmb{\eta})$  for non-negative integer  $r$ .
Recall that Corollary 6 gives the following lower bound on the classifier's confidence at  $x$ :
Observe that we can replace  $\mu_y$  by a lower bound  $\underline{\mu}_y$  that holds with probability  $1 - \alpha$  (as is done in lines 4-6 of Figure 1) and obtain a looser lower bound  $\tilde{\rho} (\bar{\boldsymbol{x}},\boldsymbol {x},\underline{\mu}_y)\leq \tilde{\rho} (\bar{\boldsymbol{x}},\boldsymbol {x},\mu_y)$  that holds with probability  $1 - \alpha$ . Crucially, this looser lower bound has the same functional form, so all results depending on Corollary 6, namely Theorem 7 and Table 1, continue to hold albeit with probability  $1 - \alpha$ .
C Background for malware detection case study
In this appendix, we provide background for our case study on malware detection, including motivation for studying certified robustness of malware detectors, a formulation of malware detection as a sequence classification problem, and a threat model for adversarial examples.
Malware (malicious software) detection is a vital capability for proactively defending against cyberattacks. Despite decades of progress, building and maintaining effective malware detection systems remains a challenge, as malware authors continually evolve their tactics to bypass detection and exploit new vulnerabilities. One technology that has lead to advancements in malware detection, is the application of machine learning (ML), which is now used in many commercial systems [69, 44, 46, 47] and continues to be an area of interest in the malware research community [70, 71, 45, 50]. While traditional detection techniques rely on manually-curated signatures or detection rules, ML allows a detection model to be learned from a training corpus, that can potentially generalize to unseen programs.
Although ML has an apparent advantage in detecting previously unseen malware, recent research has shown that ML-based static malware detectors can be evaded by applying adversarial perturbations [18-20, 37, 51, 31, 52, 17, 39]. A variety of perturbations have been considered with different effects at the semantic level, however all of them can be modeled as inserting, deleting and/or substituting bytes. This prompts us to advance certified robustness for sequence classifiers within this general threat model—where an attacker can perform byte-level edits.
C.2 Related work
Several empirical defense methods have been proposed to improve robustness of ML classifiers [63, 64]. Incer Romeo et al. [34] compose manually crafted Boolean features with a classifier that is constrained to be monotonically increasing with respect to selected inputs. This approach permits a combination of (potentially vulnerable) learned behavior with domain knowledge, and thereby aims to mitigate adversarial examples. Demontis et al. [63] show that the sensitivity of linear support vector machines to adversarial perturbations can be reduced by training with  $\ell_{\infty}$  regularization of weights. In another work, Quiring et al. [64] take advantage of heuristic-based semantic gap detectors and an ensemble of feature classifiers to improve empirical robustness. Compared to our work on certified adversarial defenses, these approaches do not provide formal guarantees.
Binary normalization [72-75] was originally proposed to defend against polymorphic/metamorphic malware, and can also be seen as a mitigation to certain adversarial examples. It attempts to sanitize binary obfuscation techniques by mapping malware to a canonical form before running a detection algorithm. However, binary normalization cannot fully mitigate attacks like Disp (see Table 9), as deducing opaque and evasive predicates are NP-hard problems [17].
Dynamic analysis can provide additional insights for malware detection. In particular, it can record a program's behavior while executing it in a sandbox (e.g., collecting a call graph or network traffic) [76-80]. Though detectors built on top of dynamic analysis can be more difficult to evade, as the attacker needs to obfuscate the program's behavior, they are still susceptible to adversarial perturbations. For example, an attacker may insert API calls to obfuscate a malware's behavior [81-84]. Applying RS-Del to certify detectors that operate on call sequences [80] or more general dynamic features would be an interesting future direction.
C.3 Static ML-based malware detection
We formulate malware detection as a sequence classification problem, where the objective is to classify a file in its raw byte sequence representation as malicious or benign. In the notation of Section 2, we assume the space of input sequences (files) is  $\mathcal{X} = \Omega^{\star}$  where  $\Omega = \{0,1,\dots ,255\}$  denotes the set of bytes, and we assume the set of classes is  $\mathcal{V} = \{0,1\}$  where 1 denotes the 'malicious' class and 0 denotes the 'benign' class. Within this context, a malware detector is simply a classifier  $f:\mathcal{X}\to \mathcal{V}$ .
Detector assumptions Malware detectors are often categorized according to whether they perform static or dynamic analysis. Static analysis extracts information without executing code, whereas dynamic analysis extracts information by executing code and monitoring its behavior. In this work, we focus on machine learning-based static malware detectors, where the ability to extract and synthesize information is learned from data. Such detectors are suitable as base classifiers for RS-Del, as they can learn to make (weak) predictions for incomplete files where chunks of bytes are arbitrarily removed. We note that dynamic malware detectors are not compatible with RS-Del, since it is not generally possible to execute an incomplete file.
Incorporating semantics In Section 3.3, we noted that our methods are compatible with sequence chunking where the original input sequence is partitioned into chunks, and reinterpreted as a sequence of chunks rather than a sequence of lower-level elements. In the context of malware detection, we can partition a byte sequence into semantically meaningful chunks using information from a disassembler, such as Ghidra [85]. For example, a disassembler can be applied to a Windows executable to identify chunks of raw bytes that correspond to components of the header, machine instructions, raw data, padding etc. Applying our deletion smoothing mechanism at the level of semantic chunks, rather than raw bytes, may improve robustness as it excludes edits within chunks that may be semantically invalid. It also yields a different chunk-level edit distance certificate, that may cover a larger set of adversarial examples than a byte-level certificate of the same radius. Figure 3 illustrates the difference between byte-level and chunk-level deletion for a Windows executable, where chunks correspond to machine instructions (such as push ebp) or non-instructions (NI).
Figure 3: Illustration of the deletion smoothing mechanism applied to an executable file at the byte-level versus chunk-level. Left: An executable file where the elementary byte sequence representation is shown in the 2nd column and chunks that correspond to machine instructions are shown in the 3rd column (sourced from the Ghidra [85] disassembler). Bytes that do not correspond to machine instructions are marked NI. Shading represents bytes (light gray) or instruction chunks (dark gray) that are deleted in the corresponding perturbed file to the right. Middle: A perturbed file produced by the deletion mechanism operating at the byte level (BYTE). Notice that individual instructions may be partially deleted. Right: A perturbed file produced by the deletion mechanism operating at the chunk-level (INSN).
C.4 Threat model
We next specify the modeled attacker's goals, capabilities and knowledge for our malware detection case study [86].
Attacker's objective We consider evasion attacks against a malware detector  $f: \mathcal{X} \to \mathcal{Y}$ , where the attacker's objective is to transform an executable file  $x$  so that it is misclassified by  $f$ . To ensure the attacked file  $\bar{x}$  is useful after evading detection, we require that it is functionally equivalent to the original file  $x$ . We focus on evasion attacks that aim to misclassify a malicious file as benign in our experiments, as these attacks dominate prior work [52]. However, the robustness certificates derived in Section 4 also cover attacks in the opposite direction—where a benign file is misclassified as malicious.
Attacker's capability We measure the attacker's capability in terms of the number of elementary edits they make to the original file  $\mathbf{x}$ . If the attacker is capable of making up to  $c$  elementary edits, then they can transform  $\mathbf{x}$  into any file in the edit distance ball of radius  $c$  centred on  $\mathbf{x}$ :
$$
\mathcal {A} _ {c} (\boldsymbol {x}) = \{\bar {\boldsymbol {x}} \in \mathcal {X}: \operatorname {d i s t} _ {O} (\boldsymbol {x}, \bar {\boldsymbol {x}}) \leq c \}.
$$
Here  $\mathrm{dist}_O(\pmb{x},\bar{\pmb{x}})$  denotes the edit distance from the original file  $\pmb{x}$  to the attacked file  $\bar{\pmb{x}}$  under the set of edit operations (ops)  $O$ . We assume  $O$  consists of elementary byte-level or chunk-level deletions (del), insertions (ins) and substitutions (sub), or a subset of these operations.
We note that edit distance is a reasonable proxy for the cost of running evasion attacks that iteratively apply localized functionality-preserving edits (e.g., [19, 37, 38, 17, 39]). For these attacks, the edit distance scales roughly linearly with the number of attack iterations, and therefore the attacker has an incentive to minimize edit distance. While attacks do exist that make millions of elementary edits in the malware domain (e.g., [31]), we believe that an edit distance-constrained threat model is an important step towards realistic threat models for certified malware detection. (To examine the effect of large edits on robustness we include the GAMMA attack [31] in experiments covered in Appendix F.)
Remark 9. The set  $\mathcal{A}_c(\pmb{x})$  overestimates the capability of an edit distance-constrained attacker, because it may include files that are not functionally equivalent to  $\pmb{x}$ . For example,  $\mathcal{A}_c(\pmb{x})$  may include files that are not malicious (assuming  $\pmb{x}$  is malicious) or files that are invalid executables. This poses no problem for certification, since overestimating an attacker's capability merely leads to a stronger certificate than required. Indeed, overestimating the attacker's capability seem necessary, as functionally equivalent files are difficult to specify, let alone analyze.
Table 3: Summary of datasets.
Dataset	Label	Number of samples
Train	Validation	Test
Sleipnir2	Benign	20948	7012	6999
Malicious	20768	6892	6905
VTFeed	Benign	111258	13961	13926
Malicious	111395	13870	13906