SlowGuess/ACL_2020_OCR huggingface dataset:
Case 4  $p_A < q$  and  $p_B + q > 1$  We choose set  $U_1$  and  $U_2$  such that  $U_1 \subseteq S_{\mathbf{X}} - S_{\mathbf{X}^*}$ ;  $U_2 \in S_{\mathbf{X}} - S_{\mathbf{X}^*}$ ;  $U_1 \cap U_2 = \emptyset$ ;  $|U_1| / |S_{\mathbf{X}}| = p_A$  and  $|U_2| / |S_{\mathbf{X}}| = p_B - (1 - q)$ . Notice that the intersect of  $U_1$  and  $U_2$  can be empty as  $|U_1| / |S_{\mathbf{X}}| + |U_2| / |S_{\mathbf{X}}| = p_A + p_B - (1 - q) \leq 1 - (1 - q) = q = |S_{\mathbf{X}} - S_{\mathbf{X}^*}| / |S_{\mathbf{X}}|$ . We define the classifier:
$$
f ^ {*} (\mathbf {Z}) = \left\{ \begin{array}{l l} y & \text {i f} \mathbf {Z} \in U _ {1} \\ y _ {B} & \text {i f} \mathbf {Z} \in U _ {2} \cup S _ {\mathbf {X} ^ {*}} \\ \text {o t h e r c l a s s} (c \neq y \text {o r} y _ {B}) & \text {i f} \mathbf {Z} \in (S _ {\mathbf {X}} - S _ {\mathbf {X} ^ {*}}) - (U _ {1} \cup U _ {2}) \\ \text {a n y c l a s s} (c \in \mathcal {Y}) & \text {o t h e r w i s e} \end{array} \right.
$$
This classifier is well defined for binary classification because  $S_{\mathbf{X}} - S_{\mathbf{X}^*} - (U_1 \cup U_2) = \emptyset$ .
It can be easily verified that for each case, the defined classifier satisfies all the conditions in Theorem 2.
We set  $R = L$  in adversarial attacking, that is, all words in the sentence can be perturbed simultaneously by the attacker. We use 5,000 random draws in the Monte Carlo estimation of  $\Delta_{\mathbf{X}}$ , and use the same method in Jia et al. (2019) to tune the hyper-parameters when training the base models e.g. learning rate, batch size and the schedule of loss function. For the IMDB dataset, we train the IBP models and ours for 60 and 10 epochs, respectively. For the Amazon dataset, we train the IBP models and ours for 100 and 20 epochs, respectively.
We test our algorithm on two different datasets, IMDB and Amazon. The IMDB movie review dataset (Maas et al., 2011) is a sentiment classification dataset. It consists of 50,000 movie review comments with binary sentiment labels. The Amazon review dataset (McAuley, 2013) is an extremely large dataset that contains 34,686,770 reviews with 5 different types of labels. Similar to Cohen et al. (2019), we test the models on randomly selected subsets of the test set with 1,250 and 6,500 examples for IMDB and Amazon dataset, respectively.