SlowGuess/NAACL_2021_OCR huggingface dataset:
$$
\widehat {\mathbb {E}} ^ {l b} \left[ f _ {\mathcal {M} _ {E}} ^ {\mathbf {y} _ {c}} (\mathbf {X}) \right] > e ^ {2 \epsilon} \max  _ {i \neq c} \widehat {\mathbb {E}} ^ {u b} \left[ f _ {\mathcal {M} _ {E}} ^ {\mathbf {y} _ {i}} (\mathbf {X}) \right], \tag {7}
$$
the prediction score vector  $\widehat{\mathbb{E}}[f_{\mathcal{M}_E}^{\mathbf{y}}(\mathbf{X})]$ -based classification is certified robust with probability  $\eta$  to  $L$ -adversary word substitution attack on  $\mathbf{X}$ .
4.3 Simulated Exponential Mechanism
Simulated Exponential Mechanism. The conceptual exponential mechanism in Definition 4.2 is computationally impractical. The bottleneck is the need to enumerate the entire  $\mathcal{S}(L)$  in order to calculate the probability distribution of  $\mathbb{P}_{\mathbf{X}'}$  for each  $\mathbf{X}' \in \mathcal{S}(L)$  and the normalization factor  $\rho$ , which essentially requires us to perform inference for  $\mathcal{S}(L) \gg n$  times ( $n$  is the number of samples) for certifying a single input sentence  $\mathbf{X}$ .
In the following, we show that we can significantly reduce the computation cost by sampling via a simulated exponential mechanism, which suffices to sample  $n$  candidate  $L$ -substitution sentences
and calculate only  $n$  times, i.e., the same repetitions as the Monte Carlo estimation. The key insight is based on the different purpose of applying the exponential mechanism between the conventional scenario for achieving DP and our certified robustness scenario. For the former, in order to ensure DP of the final output  $f_{\mathcal{M}_E}(\mathbf{X}_{\tau}')$ , the intermediate  $\mathbf{X}_{\tau}'$  is forced to satisfy DP, i.e., drawn from the exact probability distribution designated by the exponential mechanism. For the latter, while the derivation of the certified robustness relied on the randomness of DP and the exponential mechanism, we do not actually require the DP of the intermediate  $\mathbf{X}_{\tau}'$ . As a result, it allows us to sample  $\mathbf{X}_{\tau}'$  from other simpler distributions without calculating the probability distribution of the exponential mechanism, as long as the alternative approach can obtain the equivalent  $\widehat{\mathbb{E}}[f_{\mathcal{A}}^{\mathbf{y}}(\mathbf{X})]$  for robustness certification.
We develop a simulated exponential mechanism via uniform sampling and re-weighted average prediction score calculation. Figure 2 shows the simulated mechanism in contrast to the conceptual mechanism. In detail, we sample from  $\mathcal{S}(L)$  with uniform probability, which can be efficiently implemented without generating  $\mathcal{S}(L)$ . Denoting a sample by  $\mathbf{X}_{\tau}^{\prime}$ , we calculate its scaled exponential mechanism probability by
$$
\bar {\mathbb {P}} _ {\mathbf {X} _ {\tau} ^ {\prime}} = \exp \left(\frac {\epsilon \cdot u \left(\mathcal {S} (L) , \mathbf {X} _ {\tau} ^ {\prime}\right)}{2 \Delta u}\right), \tag {8}
$$
which can be obtained via a single inference on  $\mathbf{X}_{\tau}^{\prime}$  and the inference on  $\mathbf{X}$  due to the omission of the normalization factor  $\rho$  that requires the entire  $\mathcal{S}(L)$ . The inference on  $\mathbf{X}$  only needs to be computed once and shared by all  $n$  Monte Carlo repetitions. Such uniform sampling and scaled probability calculation is repeated for  $n$  times, which requires only  $n + 1$  inferences. Finally, we use the following re-weighted average prediction score (weighted by the scaled exponential mechanism probability) for certified robust prediction,
$$
\bar {\mathbb {E}} \left[ f _ {\mathcal {M} _ {E}} ^ {\mathbf {y}} (\mathbf {X}) \right] = \sum_ {\tau = 1} ^ {n} \bar {\mathbb {P}} _ {\mathbf {X} _ {\tau} ^ {\prime}} \cdot f _ {\mathcal {M} _ {E}} ^ {\mathbf {y}} \left(\mathbf {X} _ {\tau} ^ {\prime}\right). \tag {9}
$$
The following theorem shows that  $\mathbb{E}[f_{\mathcal{M}_E}^{\mathbf{y}}(\mathbf{X})]$ -based prediction guarantees certified robustness and the conceptual exponential mechanism-based inference in Proposition 4.2 is certified robust provided  $\mathbb{E}[f_{\mathcal{M}_E}^{\mathbf{y}}(\mathbf{X})]$  is so.
Theorem 4.1. For any input  $\mathbf{X}$ , let  $\overline{\mathbb{E}}[f_{\mathcal{M}_E}^{\mathbf{y}}(\mathbf{X})]$  be calculated by eq.(9). Denote  $\overline{\mathbb{E}}^{lb}[f_{\mathcal{M}_E}^{\mathbf{y}}(\mathbf{X})]$  and  $\overline{\mathbb{E}}^{ub}[f_{\mathcal{M}_E}^{\mathbf{y}}(\mathbf{X})]$  be  $\eta$ -confidence lower and up-
per bounds, respectively, i.e.,  $\mathbb{E}^{lb}[f_{\mathcal{M}_E}^{\mathbf{y}}(\mathbf{X})] = \bar{\mathbb{E}} [f_{\mathcal{M}_E}^{\mathbf{y}}(\mathbf{X})] - \sqrt{\frac{1}{2n}\ln\left(\frac{2C}{1 - \eta}\right)}$  and  $\mathbb{E}^{ub}[f_{\mathcal{M}_E}^{\mathbf{y}}(\mathbf{X})] = \bar{\mathbb{E}} [f_{\mathcal{M}_E}^{\mathbf{y}}(\mathbf{X})] + \sqrt{\frac{1}{2n}\ln\left(\frac{2C}{1 - \eta}\right)}$ . If there exists a label  $c$  such that
$$
\bar {\mathbb {E}} ^ {l b} \left[ f _ {\mathcal {M} _ {E}} ^ {y _ {c}} (\mathbf {X}) \right] > e ^ {2 \epsilon} \max  _ {i \neq c} \bar {\mathbb {E}} ^ {u b} \left[ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X}) \right], \tag {10}
$$
the prediction score vector  $\bar{\mathbb{E}}[f_{\mathcal{M}_E}^{\mathbf{y}}(\mathbf{X})]$ -based classification is certified robust with probability  $\eta$  to  $L$ -adversary word substitution attack on  $\mathbf{X}$ .
The proof of Theorem 4.1 requires the following lemma, which is adapted from Lemma 4.1 from the accurate expectation of  $\mathbb{E}[f_{\mathcal{M}_E}^{\mathbf{y}}(\mathbf{X})]$  to the simulated expectation  $\overline{\mathbb{E}}[f_{\mathcal{M}_E}^{\mathbf{y}}(\mathbf{X})]$ . We stress that during both proofs, we do not use the DP property of  $\overline{\mathbb{E}}[f_{\mathcal{M}_E}^{\mathbf{y}}(\cdot)]$ , but only its equivalent relation to  $\widehat{\mathbb{E}}[f_{\mathcal{M}_E}^{y_i}(\cdot)]$ .
Lemma 4.3. For any label  $i \in [C]$  and any  $\mathbf{X}_1, \mathbf{X}_2 \in \mathcal{S}(L)$ , let  $\overline{\mathbb{E}}[f_{\mathcal{M}_E}^{\mathbf{Y}}(\mathbf{X})]$  be computed by eq.(9). Then, we have
$$
\bar {\mathbb {E}} \left[ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} \left(\mathbf {X} _ {1}\right) \right] \leq e ^ {\epsilon} \bar {\mathbb {E}} \left[ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} \left(\mathbf {X} _ {2}\right) \right]. \tag {11}
$$
Proof. First, we notice that for any  $X' \in \mathcal{S}(L)$ , it has  $\overline{\mathbb{E}}[f_{\mathcal{M}_E}^{y_i}(\mathbf{X}')] = \frac{\rho}{|\mathcal{S}(L)|} \widehat{\mathbb{E}}[f_{\mathcal{M}_E}^{y_i}(\mathbf{X}')]$  by  $\overline{\mathbb{P}}[\mathbf{X}] = \rho \mathbb{P}[\mathbf{X}]$  and the uniform sampling probability  $\frac{1}{|\mathcal{S}(L)|}$ . Second, since  $\widehat{\mathbb{E}}[f_{\mathcal{M}_E}^{y_i}(\mathbf{X}')]$  is  $\epsilon$ -WordDP, we can show that it satisfies Lemma 4.1 by switching  $\mathbb{E}[f_{\mathcal{M}_E}^{y_i}(\cdot)]$  there to  $\widehat{\mathbb{E}}[f_{\mathcal{M}_E}^{y_i}(\cdot)]$  here. It follows that:
$$
\begin{array}{l} \mathbb {\bar {E}} [ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X} _ {1}) ] = \widehat {\mathbb {E}} [ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X} _ {1}) ] \cdot (\frac {\rho}{| \mathcal {S} (L) |}) \\ \leq e ^ {\epsilon} \mathbb {E} [ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X} _ {2}) ] \cdot (\frac {\rho}{| \mathcal {S} (L) |}) = e ^ {\epsilon} \mathbb {E} [ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X} _ {2}) ], \\ \end{array}
$$
Proof. (Proof of Theorem 4.1) For any  $\mathbf{X}' \in \mathcal{S}(L)$ , by eq.(11), we have
$$
\begin{array}{l} e ^ {\epsilon} \bar {\mathbb {E}} [ f _ {\mathcal {M} _ {E}} ^ {y _ {c}} (\mathbf {X} ^ {\prime}) ] \geq \bar {\mathbb {E}} [ f _ {\mathcal {M} _ {E}} ^ {y _ {c}} (\mathbf {X}) ] \\ > \overline {{\mathbb {E}}} [ f _ {\mathcal {M} _ {E}} ^ {y _ {c}} (\mathbf {X}) ] - \sqrt {\frac {1}{2 n} l n (\frac {2 C}{1 - \eta})} = \overline {{\mathbb {E}}} ^ {l b} [ f _ {\mathcal {M} _ {E}} ^ {y _ {c}} (\mathbf {X}) ]; \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X} ^ {\prime}) ] \leq e ^ {\epsilon} \overline {{\mathbb {E}}} [ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X}) ] \leq e ^ {\epsilon} \max  _ {i \neq c} \overline {{\mathbb {E}}} [ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X}) ] \\ \leq e ^ {\epsilon} \max  _ {i \neq c} (\bar {\mathbb {E}} [ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X}) ] + \sqrt {\frac {1}{2 n} \ln \left(\frac {2 C}{1 - \eta}\right)}) \\ = e ^ {\epsilon} \max _ {i \neq c} \overline {{\mathbb {E}}} ^ {u b} [ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X}) ]. \\ \end{array}
$$
Equipped with the above two relations, we can prove the claim in Theorem 4.1. We show that  $\overline{\mathbb{E}}[f_{\mathcal{M}_E}^{y_i}(\mathbf{X})]$  is certified robust for any  $\mathbf{X}' \in \mathcal{S}(L)$ , as follows,
$$
\begin{array}{l} \overline {{\mathbb {E}}} [ f _ {\mathcal {M} _ {E}} ^ {y _ {c}} (\mathbf {X} ^ {\prime}) ] > \overline {{\mathbb {E}}} ^ {l b} [ f _ {\mathcal {M} _ {E}} ^ {y _ {c}} (\mathbf {X}) ] / e ^ {\epsilon} \\ > e ^ {\epsilon} \max  _ {i \neq c} \bar {\mathbb {E}} ^ {u b} \left[ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X}) \right] > e ^ {\epsilon} \max  _ {i \neq c} \bar {\mathbb {E}} \left[ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} \left(\mathbf {X} ^ {\prime}\right) \right]. \tag {12} \\ \end{array}
$$
which is  $\overline{\mathbb{E}}[f_{\mathcal{M}_E}^{y_c}(\mathbf{X}')] > e^{2\epsilon} \max_{i \neq c} \overline{\mathbb{E}}[f_{\mathcal{M}_E}^{y_i}(\mathbf{X})]$ . For completeness, we can also show that the certified robustness of  $\overline{\mathbb{E}}[f_{\mathcal{A}}^{\mathbf{y}}(\mathbf{X})]$  implies the certified robustness of  $\widehat{\mathbb{E}}[f_{\mathcal{A}}^{\mathbf{y}}(\mathbf{X})]$ :
$$
\begin{array}{l} \widehat {\mathbb {E}} [ f _ {\mathcal {M} _ {E}} ^ {y _ {c}} (\mathbf {X} ^ {\prime}) ] = (\frac {| \mathcal {S} (L) |}{\rho}) \cdot \overline {{\mathbb {E}}} [ f _ {\mathcal {M} _ {E}} ^ {y _ {c}} (\mathbf {X} ^ {\prime}) ] \\ > (\frac {| \mathcal {S} (L) |}{\rho}) \bar {\mathbb {E}} ^ {l b} [ f _ {\mathcal {M} _ {E}} ^ {y _ {c}} (\mathbf {X}) ] / e ^ {\epsilon} \\ > (\frac {| \mathcal {S} (L) |}{\rho}) e ^ {\epsilon} \max _ {i \neq c} \overline {{\mathbb {E}}} ^ {u b} [ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X}) ] \\ > (\frac {| \mathcal {S} (L) |}{\rho}) \max _ {i \neq c} \overline {{\mathbb {E}}} [ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X} ^ {\prime}) ] = \max _ {i \neq c} \widehat {\mathbb {E}} [ f _ {\mathcal {M} _ {E}} ^ {y _ {i}} (\mathbf {X} ^ {\prime}) ], \\ \end{array}
$$
which proves  $\widehat{\mathbb{E}}[f_{\mathcal{M}_E}^{y_c}(\mathbf{X}')] > \max_{i \neq c} \widehat{\mathbb{E}}[f_{\mathcal{M}_E}^{y_i}(\mathbf{X}')]$ .
Training procedure. To achieve a better certification result, we involve randomness in the training stage, which is also adopted by almost all certified robustness approaches. To do so, we use the data augmentation strategy that utilizes the perturbed sentences for training, i.e.,  $\mathbf{X}' \in \mathcal{S}(L) \setminus \mathbf{X}$  given the original training sample  $\mathbf{X}$ . In practice, we first train the model without data augmentation for several epochs to achieve a reasonable performance, followed by training with perturbed  $\mathbf{X}'$ . For each training data point, we randomly draw one neighbour sentence during training (as opposed to multiple draws during certified inference).
We evaluate WordDP on two classification datasets: Internet Movie Database (IMDB) (Maas et al., 2011) and AG News corpus (AGNews) (Zhang et al., 2015). IMDB is a binary sentiment classification dataset containing 50000 movie reviews. AGNews includes 30,000 news articles categorized into four classes. The target model architecture we select is a single-layer LSTM model with size of 128. We use Global Vectors for Word Representation (GloVe) (Pennington et al., 2014a) for word embedding. The LSTM model achieves  $88.4\%$  and  $91.8\%$  clean accuracy on IMDB and AGNews, respectively. We use PWWS (Ren et al., 2019) to generate adversarial examples on the test dataset. PWWS is a state-of-the-art attack method which uses WordNet to build synonym set and incorporates word saliency to replace selected named entities (NEs) with their synonyms in order to flip the prediction. The details about the datasets, model training and attack algorithm are in Appendix C.
5.1 Evaluation Metrics and Baselines
We use four metrics to evaluate the effectiveness of WordDP: certified ratio, certified accuracy, conditional accuracy, and conventional ac
curacy. Certified Ratio represents the fraction of testing set that the prediction satisfies the certification criteria:  $\frac{\sum_{t=1}^{T} \text{certifiedCheck}(\mathbf{X}_{t}, L, \epsilon)}{T}$ , where certifiedCheck returns 1 if Theorem 4.1 is satisfied and  $T$  is the size of the test dataset. Certified accuracy (CertAcc) denotes the fraction of the clean testing set on which the predictions are both correct and satisfy the certification criteria. This is a standard metric to evaluate certified robust model (Lecuyer et al., 2019). Formally, it is defined as:  $\frac{\sum_{t=1}^{T} \text{certifiedCheck}(\mathbf{X}_{t}, L, \epsilon) \& \text{corrClass}(\mathbf{X}_{t}, L, \epsilon)}{T}$ , where corrClass returns 1 if the classification output is correct. When the accuracy of a model is close to  $100\%$ , certified accuracy largely reflects certified ratio. Conventional accuracy (ConvAcc) is defined as the fraction of testing set that is correctly classified,  $\frac{\sum_{t=1}^{T} \text{corrClass}(\mathbf{X}_{t}, L, \epsilon)}{T}$ , which is a standard metric to evaluate any deep learning systems. Note that the input  $\mathbf{X}_{t}$  can be both adversarial or clean inputs. We use this metric to evaluate how WordDP empirically works on adversarial examples.
Besides the above standard metrics, we introduce a new accuracy metric called Conditional accuracy (CondAcc) to evaluate the following: when a clean input  $\mathbf{X}_t$  is certified within bound  $L$ , whether its corresponding  $L$ -word substitution adversarial example  $\mathbf{X}_t^{adv}$  is indeed correctly classified. The CondAcc can be formulated as:  $\frac{\sum_{t=1}^{T} \text{certifiedCheck}(\mathbf{X}_t, L, \epsilon) & \text{corrClass}(\mathbf{X}_t^{adv}, L, \epsilon)}{\sum_{t=1}^{T} \text{certifiedCheck}(X_t, L, \epsilon)}$ . While certified accuracy is typically evaluated on clean inputs in the literature to show the certified robustness property, conditional accuracy is evaluated on adversarial inputs and provides an informative measure of the classification result of adversarial examples when its counterpart clean input can be certified. This metric is aligned with the definition and purpose of certified robustness. Ideally, if a clean example is successfully certified, adversarial examples created from this clean example should have the same prediction. Therefore, the accuracy of adversarial examples is influenced by the ConvAcc of clean examples.
Comparison Methods. We compare WordDP with the state-of-the-art certified robust method SAFER for text classification. We note that SAFER only reports certified accuracy, without accuracy on adversarial examples. To conduct a fair comparison with WordDP, we rerun SAFER on the adversarial examples and report the comparison
(a) CertAcc on IMDB
(b) CondAcc on IMDB
(c) ConvAcc on IMDB
(d) CertAcc on AGNews
(e) CondAcc on AGNews
(f) ConvAcc on AGNews
Figure 3: Certified Accuracy, Conditional Accuracy and Conventional Accuracy on IMDB and AGNews
in CertAcc and CondAcc. Besides SAFER, we also compare the ConvAcc on adversarial examples with two state-of-the-art defense methods, i.e., IBP (Jia et al., 2019) and DNE (Zhou et al., 2020), which do not provide certified robustness guarantee. Thus, their defense may be broken by more powerful word substitution attacks in the future.
5.2 Certified Results
Certified Accuracy. Figure 3 presents the CertAcc, CondAcc and ConvAcc under different  $\epsilon$  and  $L$ , respectively. Each line in the figures represents a certified bound  $L$ , which allows  $L$  number of words to be substituted. The first row is the results on IMDB, and the second row is on AGNews.
Figures 3(a) and 3(d) show the certified accuracy on the two datasets. Since the conventional accuracy on the clean examples of our mechanisms is close to  $100\%$  (as shown in Figures 3(c) and 3(f)), the certified accuracy mainly reflects the certified ratio (which we skip in the results). As shown, higher  $\epsilon$  can result in lower CertAcc. This is intuitive as the condition in Theorem 4.1 is more difficult to satisfy when given higher epsilon, i.e. weaker requirement of indistinguishability of the output, hence results in lower certified ratio. As illustrated in 3(a), when  $\epsilon$  is around 1.5, the mechanism will approach 0 certified ratio. This indicates that  $\epsilon$  can only be searched within a limited range.
Comparing each line in 3(a) and 3(d), we note that greater  $L$  results in higher CertAcc in most cases for the AGNews dataset. This can be ex
ADV	IBP	DNE	SAFER	WordDP
IMDB	0.172	0.722	0.823	0.727	0.972
AGNews	0.194	0.823	0.909	0.647	0.719
Table 1: Empirical comparison on accuracy
(b) AGNews
Figure 4: Certified Ration vs. Conditional Accuracy plained by the fact that a greater  $L$  means more word substitutions and randomness are introduced in WordDP, making it easier to ensure the indistinguishability of the output, and hence a higher certified ratio.
Accuracy on Adversarial Examples. Figures 3(b), 3(e), 3(c) and 3(f) present CondAcc and ConvAcc of the two datasets on adversarial examples, respectively. Note that we only test the adversarial examples that are within the  $L$  bound. We also show the CondAcc and ConvAcc for both clean and adversarial examples without any defense mechanisms as a reference. In addition, we show ConvAcc of WordDP with varying parameters on clean examples to show the impact of the mechanism on clean examples.
As shown in the figures, WordDP achieves significantly higher accuracy on adversarial examples compared to no defense while maintaining the close to  $100\%$  accuracy on clean examples. Conditional
(a) Fixed attack power 40
(b) Fixed defense power 40
Figure 5: The trend on accuracy under different defense and attack power
accuracy is higher than conventional accuracy as expected, since it is computed only on those adversarial examples with a certified counterpart clean example. Besides, we can observe that with higher  $\epsilon$ , higher CondAcc on adversarial examples can be achieved. This is because less randomness is introduced in the inference.
In addition, by comparing different  $L$  bound under the same  $\epsilon$ , larger  $L$  can yield more accuracy improvement on adversarial examples but less on clean examples. Intuitively, using the aggregated prediction of more distant neighbouring sentences (higher  $L$ ) can benefit adversarial examples more than clean examples.
Trader-off between Certified Ratio and CondAcc. We can see that  $\epsilon$  has an opposite impact on certified accuracy (certified ratio) and CondAcc, we present the trade-off between the certified ratio and CondAcc of WordDP in Figure 4 in comparison with the baseline method SAFER. Ideally, we want both high certified ratio and high condAcc to contribute to overall high accuracy. The black dot represents the baseline SAFER, since the neighbouring sentence generating method of SAFER does not depend on  $L$  or  $\epsilon$ . As illustrated on these two datasets, with  $L = 20$  and  $L = 40$ , WordDP can dominate SAFER and achieve a much better performance in both certified ratio and condAcc.
Relation between certified bound  $L$  and adversarial attack power  $L_{adv}$ . Figure 5 presents the three accuracy metrics under different attack power and defense power. In Figure 5(a), we fix the attack power  $L_{adv}$  to 40, which means allowing less than 40 word substitutions, and adjust the WordDP defense power by using different certified bound  $L$ . As discussed in Section 4, certified bound  $L$  determines the size of neighbouring set. Greater  $L$  leads to higher randomness and thus can benefit the CondAcc and ConvAcc on adversarial examples. On the other hand, greater  $L$  also makes the certified condition more difficult to be satisfied, which result in lower CertAcc.
In Figure 5(b), we fix the certified bound  $L$  to 40, which means using the same power of  $WordDP$  to defend against adversarial examples generated by varying attack power  $L_{adv}$ . As shown in the figure, the performance increases with higher attack power. This is because the adversarial examples with more word changes (higher  $L_{adv}$ ) are more difficult to generate but easier to defend (due to the nature of PWWS attack algorithm).
Comparison with Empirical Defense. Besides certified robust method SAFER, we also compare CondAcc of WordDP with baseline empirical defense methods, IBP (Jia et al., 2019) and DNE (Zhou et al., 2020). Table 1 compares the highest CondAcc achieved by WordDP with the conventional accuracy reported by the baselines (ADV corresponds to no defense). WordDP achieves a much higher accuracy on IMDB dataset compared to IBP, DNE and SAFER. For AGNews, the accuracy of WordDP outperforms SAFER, but is lower than the two empirical defenses. We stress, however, the empirical defense methods do not provide any rigorous certified robustness guarantees and the performance can be significantly dependent on datasets and specific attacks.
Efficiency Comparison. We also compare the efficiency of WordDP with SAFER by computing the average time cost for certifying one input and producing the Monte Carlo sampling-based output. It takes WordDP 6.25s and 3.21s on IMDB and AGNews, respectively. As a comparison, it costs SAFER 230.35s and 96.68s. Thus, WordDP achieves more than  $30 \times$  efficiency improvement.
We proposed WordDP, a certified robustness method to adversarial word substitution attacks with the exponential mechanism-based algorithm. Compared with previous work, WordDP achieves notable accuracy improvement and  $30 \times$  efficiency improvement. In the future, it would be interesting to expand WordDP to other kinds of textual adversarial examples, such as character-level attacks. It is also worthwhile to study other certified approaches such as random smoothing.
We sincerely thank all anonymous reviewers for their constructive comments. This work is partially supported by the National Science Foundation (NSF) CNS-1952192, IIS-1838200, and National Institutes of Health (NIH) CTSA Award UL1TR002378.