SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm of RCS. Therefore, we use a greedy search algorithm via batch-wise selection for RCS. Algorithm 1 iterates the batch-wise selection  $\lfloor k|X| / \beta \rfloor$  times. At each iteration, it finds the minibatch  $B$  that has the largest gain  $\hat{G}_{\theta}(B|S)$  based on the parameters updated by the previously selected coreset, and then adds this minibatch into the final coreset. RCS via greedy search needs to calculate the gradient for each minibatch  $\lceil N / \beta \rceil$  times (Line 6 in Algorithm 1) and the gradient on the validation set  $\lfloor kN / \beta \rfloor$  times (Line 10 in Algorithm 1). In total, RCS needs to calculate the loss gradient  $\lceil N / \beta \rceil +\lfloor kN / \beta \rfloor \ll \binom {N}{kN}$  times, which is significantly more efficient than the native solution. We approximate the marginal gain function using the Taylor expansion as follows:
$$
\hat {G} _ {\theta} (B | S) \approx \eta \nabla_ {\theta} \mathcal {L} _ {\mathrm {R D}} (U; \theta - \eta \nabla_ {\theta} \mathcal {L} _ {\mathrm {A C L}} (S; \theta)) ^ {\top} \nabla_ {\theta} \mathcal {L} _ {\mathrm {A C L}} (B; \theta) + \beta \sigma , \tag {7}
$$
where  $B$  is a minibatch and  $\beta$  is the batch size. The derivation of Eq. (7) is in Appendix A.3. It enables us to efficiently calculate the marginal gain for each minibatch (Line 13 in Algorithm 1). We omit this term in Algorithm 1 since  $\beta \sigma$  is a constant. Intuitively, RCS selects the data whose training loss gradient and validation loss gradient are of the most similarity. In this way, the model can minimize the RD (validation loss) after updating its parameters by minimizing the ACL loss (training loss) on the coreset selected by RCS, thus helping improve the adversarial robustness of the representation.
3.4 Efficient ACL via RCS
We show an efficient ACL procedure via RCS in Algorithm 2. ACL with RCS trains the model on the previously selected coreset for  $I \in \mathbb{N}$  epochs, and for every  $I$  epochs a new coreset is selected.
Algorithm 2 Efficient ACL via RCS
1: Input: Unlabeled training set  $X$ , unlabeled validation set  $U$ , total training epochs  $E$ , learning rate  $\eta'$ , batch size  $\beta$ , warmup epoch  $W$ , epoch interval for executing RCS  $I$ , subset fraction  $k$ , learning rate for RCS  $\eta$   
2: Output: Adversarily pre-trained feature extractor  $f_{\theta'}$   
3: Initialize parameters of model  $g \circ f_{\theta'}$   
4: Initialize training set  $S \leftarrow X$   
5: // Warmup training for  $W$  epochs; Training on the coreset for  $(E - W)$  epochs  
6: for  $e = 0$  to  $(E - 1)$  do  
7: if  $e\% I == 0$  and  $e \geq W$  then  
8:  $\theta \leftarrow \text{copy}(\theta')$   
9:  $S \leftarrow \text{RCS}(X, U, \beta, g \circ f_{\theta}, \eta, k) //$  by Algorithm 1  
10: end if  
11: for batch  $m = 1, \ldots, \lceil |S| / \beta \rceil$  do  
12: Sample a minibatch  $B_m$  from  $S$   
13: Update  $\theta' \leftarrow \theta' - \eta' \nabla_{\theta'} \mathcal{L}_{\text{ACL}}(B_m; \theta')$   
14: end for  
15: end for
The pre-training procedure is repeated until the required epoch  $E \in \mathbb{N}$  is reached. We provide four tricks with three tricks as follows and one trick that enables efficient RCS on large-scale datasets with limited GPU memory in Appendix B.1.
Warmup on the entire training set. We take  $W$  epochs to train the model on the entire training set as the warmup. Warmup training enables the model to have a good starting point to provide informative gradients used in RCS. For example, we use  $10\%$  of the total training epochs for warmup.
Last-layer gradients. It is computationally expensive to compute the gradients over full layers of a deep model due to a tremendous number of parameters in the model. To tackle this issue, we utilize a last-layer gradient approximation, by only considering the loss gradients of the projection head  $g$  during RCS.
Adversarial data approximation. Calculating adversarial data during CS is extremely time-consuming since generating adversarial data needs to iteratively perform BP  $T$  times. We let  $T_{\mathrm{ACL}}$  be PGD steps,  $\epsilon_{\mathrm{ACL}}$  be the adversarial budget, and  $\rho_{\mathrm{ACL}}$  be the step size for PGD during ACL. Similarly,  $T_{\mathrm{RCS}}$ ,  $\epsilon_{\mathrm{RCS}}$ , and  $\rho_{\mathrm{RCS}}$  denote PGD configurations during RCS. To mitigate the above issue, we decrease  $T_{\mathrm{RCS}}$  (i.e.,  $0 < T_{\mathrm{RCS}} \leq T_{\mathrm{ACL}}$ ) and  $\rho_{\mathrm{RCS}} = \frac{\rho_{\mathrm{ACL}} \cdot T_{\mathrm{ACL}}}{T_{\mathrm{RCS}}}$  for efficiently generating adversarial data during CS. Note that the same adversarial budget is used for ACL and RCS, i.e.,  $\epsilon_{\mathrm{RCS}} = \epsilon_{\mathrm{ACL}}$ .
In this section, we first validate that our proposed RCS can significantly accelerate ACL [14] and its variant [17, 15] on various datasets [6, 26] with minor degradation in transferability to downstream tasks. Then, we apply RCS to a large-scale dataset, i.e., ImageNet-1K [1] to demonstrate that RCS enhances the scalability of ACL. Lastly, we demonstrate extensive empirical results. Extensive experimental details are in Appendix B.2.
Efficient pre-training configurations. We leverage RCS to speed up ACL [14] and DynACL [17] using ResNet-18 backbone networks. The pre-training settings of ACL and DynACL exactly follow their original paper and we provide the details in Appendix B.2. For the hyperparameters of RCS, we set  $\beta = 512$ ,  $\eta = 0.01$ , and  $T_{\mathrm{RCS}} = 3$ . We took  $W = 100$  epochs for warmup, and then CS was executed every  $I = 20$  epoch. We used different subset fractions  $k \in \{0.05, 0.1, 0.2\}$  for CS. The KL divergence was used as the distance function to calculate  $\mathcal{L}_{\mathrm{RD}}(\cdot)$  for all the experiments in Section 4. We used the pre-trained weights released in ACL's and DynACL's GitHub as the pre-trained encoder to reproduce their results on the entire set. We repeated all the experiments using different random seeds three times and report the median results to exclude the effect of randomization.
Finetuning methods. We used the following three kinds of finetuning methods to evaluate the learned representations: standard linear finetuning (SLF), adversarial linear finetuning (ALF), and
Figure 2: Cross-task adversarial robustness transferability from CIFAR-10 to CIFAR-100 (upper row) and STL-10 (bottom row). The number after the dash line denotes subset fraction  $k \in \{0.05, 0.1, 0.2\}$ . ACL with RCS and DynACL with RCS correspond to the red and orange solid lines, respectively. ACL with Random and DynACL with Random correspond to the blue and green dotted lines, respectively.
Figure 3: Cross-task adversarial robustness transferability from CIFAR-100 to CIFAR-10 (upper row) and STL-10 (bottom row). The number after the dash line denotes subset fraction  $k \in \{0.05, 0.1, 0.2\}$ .
adversarial full finetuning (AFF). SLF and ALF will keep the learned encoder frozen and only finetune the linear classifier using natural or adversarial samples, respectively. AFF leverages the pre-trained encoder as weight initialization and trains the whole model using the adversarial data. The finetuning settings exactly follow DynACL [17]. More details in Appendix B.2.
Evaluation metrics. The reported robust test accuracy (dubbed as "RA") is evaluated via AutoAttack (AA) [19] and the standard test accuracy (dubbed as "SA") is evaluated on natural data. In practice, we used the official code of AutoAttack [19] for implementing evaluations. In Appendix B.5, we provide robustness evaluation under more diverse attacks [19, 55, 56].
Baseline. We replaced RCS (Line 9 in Algorithm 2) with the random selection strategy (dubbed as "Random") to obtain the coreset  $S$  during ACL as the baseline. The implementation of Random exactly follows that in the coreset and data selection library [31].
Speed-up ratio. The speed-up ratio denotes the ratio of the running time of the whole pre-training procedure on the entire set (dubbed as "Entire") to that of ACL with various CS strategies (i.e., Random and RCS). We report the running time of Entire in Table 3.
4.1 Effectiveness of RCS in Efficient ACL
Cross-task adversarial robustness transferability. Figures 2 and 3 demonstrate the cross-task adversarial robustness transferability from CIFAR-10 and CIFAR-100 to downstream tasks, respectively. First, we observe that ACL/DynACL with RCS (red/orange lines) always achieves significantly higher robust and standard test accuracy than ACL/DynACL with Random (blue/green lines) among
Figure 4: Self-task adversarial robustness transferability evaluated on the CIFAR-10 (left three panels) and CIFAR-100 (right three panels) dataset. The number after the dash line denotes subset fraction  $k \in \{0.05, 0.1, 0.2\}$ .
various subset fractions while achieving a similar speed-up ratio to Random. Besides, compared with ACL/DynACL on the entire set (black/brown squares), ACL/DynACL with RCS (red/orange lines) can almost maintain robust and standard test accuracy on downstream tasks. Notably, RCS-0.2 almost maintains robust test accuracy on various downstream tasks via various finetuning methods. It validates the effectiveness of our principled method RCS in maintaining robustness transferability while speeding up pre-training.
Self-task adversarial robustness transferability. Figure 4 shows the self-task adversarial robustness transferability evaluated on CIFAR-10 and CIFAR-100 datasets where pre-training and finetuning were conducted on the same datasets. In Appendix B.3, we provide the performance on the STL-10 [26] task in Tables 4 and 5. Apparently, ACL/DynACL with RCS (red/orange lines) always yields better performance across tasks than ACL/DynACL with Random (blue/green lines). Besides, RCS (solid lines) can almost maintain robust test accuracy via various finetuning methods compared with Entire (squares) while having a high speed-up ratio. It validates that RCS is effective in efficiently learning robust representations.
4.2 Benchmarks on ImageNet-1K
To the best of our knowledge, we are the first to apply ACL [14] on ImageNet-1K efficiently via RCS. As shown in Table 1, ACL on the entire set of ImageNet-1K will need about 650.2 hours using 4 NVIDIA RTX A5000 GPUs, which is extremely time-consuming and unmanageable for us to conduct ACL on the entire set. Thus, we do not provide the results of ACL on the entire set.
Tables 1 and 2 demonstrate the cross-task adversarial robustness transferability of pre-trained WideResNet [57] with width 10 and depth 28 (WRN-28-10) from ImageNet-1K of  $32 \times 32$  resolution to CIFAR-10 and CIFAR-100, respectively. Experimental details of pre-training are in Appendix B.2. The two tables demonstrate that ACL with RCS consistently leads to better robust and standard test accuracy on downstream tasks than ACL with Random. Besides, we can observe that ACL with RCS often obtains better robustness transferability compared with standard CL. Surprisingly, ACL with RCS even consumed less running time (111.8 hours) than standard CL (147.4 hours) while achieving better transferability. It validates that RCS effectively enhances the scalability of ACL, and efficient ACL via RCS can yield useful and robust representations.
4.3 Extensive Experimental Results
Ablation study. We provide the results of RCS using various distance functions in Appendix B.8, RCS with various warmup epochs  $W$  in Appendix B.9, RCS with various epoch intervals  $I$  for executing RCS in Appendix B.10, RCS with various batch sizes during CS in Appendix B.11, and RCS for another variant AdvCL [15] in Appendix B.12.
Comparison between ACL with FGSM and ACL with RCS. To the best of our knowledge, no work has studied how to use FGSM (one-step PGD) [24] to speed up ACL. In Table 7, we demonstrate
Table 1: Cross-task adversarial robustness transferability from ImageNet-1K to CIFAR-10.
Pre-training	Runing time (hours)	SLF	ALF	AFF
SA (%)	RA (%)	SA (%)	RA (%)	SA (%)	RA (%)
Standard CL	147.4	84.36±0.17	0.01±0.01	10.00±0.00	10.00±0.00	86.63±0.12	49.71±0.16
ACL on entire set	650.2	-	-	-	-	-	-
ACL with Random	94.3	68.75±0.06	15.89±0.06	59.57±0.28	27.14±0.19	84.75±0.18	50.12±0.21
ACL with RCS	111.8	70.02±0.12	22.45±0.13	63.94±0.21	31.13±0.17	85.23±0.23	52.21±0.14
Table 2: Cross-task adversarial robustness transferability from ImageNet-1K to CIFAR-100.
Pre-training	Runing time (hours)	SLF	ALF	AFF
SA (%)	RA (%)	SA (%)	RA (%)	SA (%)	RA (%)
Standard CL	147.4	57.34±0.23	0.01±0.01	9.32±0.01	0.06±0.01	61.33±0.12	25.11±0.15
ACL on entire set	650.2	-	-	-	-	-	-
ACL with Random	94.3	38.53±0.15	10.50±0.13	28.44±0.23	11.93±0.21	59.63±0.33	25.46±0.26
ACL with RCS	111.8	40.28±0.17	14.55±0.10	33.15±0.26	14.89±0.16	60.25±0.18	28.24±0.13
that FGSM cannot effectively learn robust representations and consumes more running time than RCS. Therefore, our proposed RCS is more efficient and effective in speeding up ACL than FGSM.
RCS for efficient supervised robust pre-training. In Appendix B.13.2, following Hendrycks et al. [58] and Salman et al. [48], we apply RCS to speed up pre-training WRN-28-10 and ResNet-50 via SAT [34] on ImageNet-1K [1] and demonstrate that SAT with RCS can almost maintain transferability. Therefore, RCS can be a unified and effective framework for efficient robust pre-training.
Comparison between RCS and ACS [33] in speeding up supervised AT including Fast-AT [24], SAT [34], and TRADES [35]. We provide the comparison between RCS and ACS [33] in Appendix B.13.1, which validates that our proposed RCS, without using labels during CS, is more efficient and effective than ACS in terms of speeding up supervised AT [24, 34, 35].
Analysis of the coreset selected by RCS. We provide comprehensive quantitative and visualization analyses of the coreset selected by RCS in Appendix B.6. The analyses demonstrate that RCS helps minimize RD and tends to select a coreset that is closer to the entire training set compared to Random, thus helping ACL obtain useful and adversarially robust representations.
This paper proposed a robustness-aware coreset selection (RCS) framework for accelerating robust pre-training. RCS found an informative subset that helps minimize the representational divergence between natural data and their adversarial counterparts. We theoretically showed that RCS can be efficiently solved by greedy search approximately with an optimality guarantee. RCS does not require label information and is thus applicable to ACL as well as supervised AT. Our experimental results validated that RCS can significantly speed up both ACL and supervised AT while slightly hurting the robustness transferability.
One of the limitations is that RCS still requires spending a particular amount of running time during coreset selection in calculating the loss gradients, although we theoretically propose the greedy-search algorithm with an optimality guarantee to make RCS efficient. We leave how to further improve the efficiency of RCS, such as by leveraging better submodular function optimization methods [59, 60], as the future work.
This research is supported by the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative, Australian Research Council (ARC) under Award No. DP230101540 and NSF and CSIRO Responsible AI Program under Award No. 2303037. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
[59] Andreas Krause and Daniel Golovin. Submodular function maximization. Tractability, 3 (71-104):3, 2014.
[60] Daniel Golovin and Andreas Krause. Adaptive submodularity: Theory and applications in active learning and stochastic optimization. Journal of Artificial Intelligence Research, 42: 427-486, 2011.
[61] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 819-828, 2020.
[62] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
[63] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
[64] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[65] Jovana Mitrovic, Brian McWilliams, Jacob C Walker, Lars Holger Buesing, and Charles Blundell. Representation learning via invariant causal mechanisms. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=9p2ekP904Rs.
[66] Peter Buhlmann. Invariance, causality and robustness. 2020.
Theorem 1 (restated) We define a proxy set function  $\hat{G}_{\theta}(S) \triangleq G_{\theta}(S) + |S| \sigma$ , where  $\sigma = 1 + \nu_1 + \nu_2 L_2 + \eta M L_2 (L_1 + \eta k N (L_1 L_4 + L_2 L_3))$ ,  $\nu_1 \to 0^+$ , and  $\nu_2 > 0$  are positive constants. Given Assumption 1,  $\hat{G}_{\theta}(S)$  is monotone and  $\gamma$ -weakly submodular where  $\gamma > \gamma^* = \frac{1}{2\sigma - 1}$ .
Proof. We first provide proof of monotonicity of  $\hat{G}_{\theta}(S)$ . We use Taylor expansion two times to convert the marginal gain function  $\hat{G}_{\theta}(x|S)$  as follows,
$$
\begin{array}{l} \hat {G} _ {\theta} (x | S) = \hat {G} _ {\theta} (S \cup \{x \}) - \hat {G} _ {\theta} (S) \\ = - \mathcal {L} _ {\mathrm {R D}} (U; \theta - \eta \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (S; \theta) - \eta \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{x \}; \theta)) + \mathcal {L} _ {\mathrm {R D}} (U; \theta - \eta \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (S; \theta)) + \sigma \\ \xlongequal {\text {T a y l o r e x p a n s i o n}} \eta \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {R D}} (U; \theta - \eta \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (S; \theta)) ^ {\top} \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{e \}; \theta) + \xi_ {1} (\theta) + \sigma \\ \xlongequal {\text {T a y l o r e x p a n s i o n}} \eta \frac {\partial}{\partial_ {\theta}} \left[ \mathcal {L} _ {\mathrm {R D}} (U; \theta) - \eta \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {R D}} (U; \theta) ^ {\top} \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (S; \theta) + \xi_ {2} (\theta) \right] ^ {\top} \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{x \}; \theta) \\ + \xi_ {1} (\theta) + \sigma \\ = \eta \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {R D}} (U; \theta) ^ {\top} \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{x \}; \theta) - \eta^ {2} \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {R D}} (U; \theta) ^ {\top} \frac {\partial^ {2}}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (S; \theta) \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{x \}; \theta) \\ - \eta^ {2} \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (S; \theta) ^ {\top} \frac {\partial^ {2}}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {R D}} (U; \theta) \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{x \}; \theta) + \frac {\partial}{\partial_ {\theta}} \xi_ {2} (\theta) ^ {\top} \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{x \}; \theta) \\ + \xi_ {1} (\theta) + \sigma , \tag {8} \\ \end{array}
$$
where  $\xi_1(\theta) \to 0$  and  $\xi_2(\theta) \to 0$  are the remainder terms of Taylor series. Recall that  $\mathcal{L}_{\mathrm{RD}}(U; \theta) = \sum_{x_i \in U} \ell_{\mathrm{RD}}(x_i; \theta)$ ,  $\mathcal{L}_{\mathrm{ACL}}(S; \theta) = \sum_{x_i \in S} \ell_{\mathrm{ACL}}(x_i; \theta)$ ,  $|U| = M$ , and  $|X| = N$ . According to Assumption 1, we have the following results:
$$
\left\| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {R D}} (U; \theta) \right\| \leq | U | L _ {1} = M L _ {1}; \tag {9}
$$
$$
\left\| \frac {\partial^ {2}}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {R D}} (U; \theta) \right\| \leq | U | L _ {3} = M L _ {3}; \tag {10}
$$
$$
\left\| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (S; \theta) \right\| \leq | S | L _ {2} \leq k | X | L _ {2} = k N L _ {2}; \tag {11}
$$
$$
\left\| \frac {\partial^ {2}}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (S; \theta) \right\| \leq | S | L _ {4} \leq k | X | L _ {4} = k N L _ {4}. \tag {12}
$$
We assume that  $\| \xi_1(\theta)\| \leq \nu_1$  and  $\| \frac{\partial}{\partial_{\theta}}\xi_2(\theta)\| \leq \nu_2$  where  $\nu_{1}\rightarrow 0^{+}$  and  $\nu_{2} > 0$  are two positive constant. We can transform Eq. (8) as follows,
$$
\begin{array}{l} \hat {G} _ {\theta} (x | S) \geq - \| \hat {G} _ {\theta} (x | S) \| \\ \geq - \eta \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {R D}} (U; \theta) \| \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{x \}; \theta) \| - \eta^ {2} \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {R D}} (U; \theta) \| \| \frac {\partial^ {2}}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (S; \theta) \| \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{x \}; \theta) \| \\ - \eta^ {2} \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (S; \theta) \| \| \frac {\partial^ {2}}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {R D}} (U; \theta) \| \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{e \}; \theta) \| - \| \frac {\partial}{\partial_ {\theta}} \xi_ {2} (\theta) \| \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{x \}; \theta) \| \\ - \left\| \xi_ {1} (\theta) \right\| + \sigma \\ \geq - \eta M L _ {1} L _ {2} - \eta^ {2} M L _ {1} k N L _ {4} L _ {2} - \eta^ {2} k N L _ {2} M L _ {3} L _ {2} - \nu_ {2} L _ {2} - \nu_ {1} + \sigma \\ \geq - \eta M L _ {2} \left(L _ {1} + \eta k N \left(L _ {1} L _ {4} + L _ {2} L _ {3}\right)\right) - \nu_ {2} L _ {2} - \nu_ {1} + \sigma . \\ \end{array}
$$
Since  $\sigma = \eta ML_2(L_1 + \eta kN(L_1L_4 + L_2L_3)) + \nu_1 + \nu_2L_2 + 1$ , we can obtain that
$$
\hat {G} _ {\theta} (x \mid S) \geq 1. \tag {13}
$$
Eq. (13) shows that  $\hat{G}_{\theta}(S)$  is monotone, i.e.,  $\hat{G}_{\theta}(x|S) > 0$
Then, we provide proof of  $\gamma$ -weakly submodularity of  $\hat{G}_{\theta}(S)$ . The sketch of proof is that we first prove  $\hat{G}_{\theta}(S)$  is  $\alpha$ -submodular. Then, according to Lemma 1, we can conclude that  $\hat{G}_{\theta}(S)$  is also  $\gamma$ -weakly submodular.
Lemma 1 (Proposition 4 of Gatemiry and Gomez-Rodriguez [38]). The set function  $G(S):2^{X}\to \mathbb{R}$  that is  $\alpha$ -submodular is  $\gamma$ -weakly submodular with the submodularity ratio  $\gamma \geq 1 - \alpha$ .
The following is the definition of  $\alpha$ -submodularity.
Definition 2 ( $\alpha$ -submodularity [38]). A function is called  $\alpha$ -submodular if the marginal gain of adding an element  $x$  to set  $A$  is  $1 - \alpha$  times greater than or equals to the gain of adding an element  $x$  to set  $B$  where  $A \subseteq B$ . i.e.,
$$
\underset {A, B | A \subseteq B} {\forall} G _ {\theta} (x | A) \geq (1 - \alpha) G _ {\theta} (x | B).
$$
According to Eq. (13), we have obtained the lower bound of the marginal gain
$$
\hat {G} _ {\theta} (x | S) \geq 1 = \underline {{\hat {G}}} _ {\theta} (S). \tag {14}
$$
We can obtain the upper bounder of marginal gain based on Eq. (8) as follows,