SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xilie Xu $^{1*}$ , Jingfeng Zhang $^{2,3*†}$ , Feng Liu $^{4}$ , Masashi Sugiyama $^{2,5}$ , Mohan Kankanhalli $^{1}$
$^{1}$  School of Computing, National University of Singapore
2 RIKEN Center for Advanced Intelligence Project (AIP)
3 School of Computer Science, The University of Auckland
$^{4}$  School of Computing and Information Systems, The University of Melbourne
<sup>5</sup> Graduate School of Frontier Sciences, The University of Tokyo  
xuxilie@comp.nus.edu.sg jingfeng.zhang@auckland.ac.nz  
fengliu.ml@gmail.com sugi@k.u-tokyo.ac.jp
mohan@comp.nus.edu.sg
Adversarial contrastive learning (ACL) does not require expensive data annotations but outputs a robust representation that withstands adversarial attacks and also generalizes to a wide range of downstream tasks. However, ACL needs tremendous running time to generate the adversarial variants of all training data, which limits its scalability to large datasets. To speed up ACL, this paper proposes a robustness-aware coreset selection (RCS) method. RCS does not require label information and searches for an informative subset that minimizes a representational divergence, which is the distance of the representation between natural data and their virtual adversarial variants. The vanilla solution of RCS via traversing all possible subsets is computationally prohibitive. Therefore, we theoretically transform RCS into a surrogate problem of submodular maximization, of which the greedy search is an efficient solution with an optimality guarantee for the original problem. Empirically, our comprehensive results corroborate that RCS can speed up ACL by a large margin without significantly hurting the robustness transferability. Notably, to the best of our knowledge, we are the first to conduct ACL efficiently on the large-scale ImageNet-1K dataset to obtain an effective robust representation via RCS. Our source code is at https://github.com/GodXuxilie/Efficient_ACL_via_RCS.
The pre-trained models can be easily finetuned to downstream applications, recently attracting increasing attention [1, 2, 3, 4]. Notably, vision Transformer [5] pre-trained on ImageNet-1K [1] can achieve state-of-the-art performance on many downstream computer-vision applications [6, 7]. Foundation models [8] trained on large-scale unlabeled data (such as GPT [9] and CLAP [10]) can be adapted to a wide range of downstream tasks. Due to the prohibitively high cost of annotating large-scale data, the pre-trained models are commonly powered by the techniques of unsupervised learning [11, 12] in which contrastive learning (CL) is the most effective learning style to obtain the generalizable feature representations [13, 14].
Adversarial CL (ACL) [14, 15, 16, 17], that incorporate adversarial data with contrastive loss, can yield a robust representation that is adversarily robust. Compared with the standard CL [13], ACL can output a robust representation that can transfer some robustness to the downstream tasks against
*The first two authors have made equal contributions.
Figure 1: We learn a representation using CIFAR-10 [6] dataset (without requiring labels) via ACL [14] and DynACL [17]. Then, we evaluate the representation's robustness transferability to CIFAR-100 [6] and STL-10 [26] (using labels during finetuning) via standard linear finetuning. We demonstrate the running time of robust pre-training w.r.t. different coreset selection (CS) strategies and report the robust test accuracy under AutoAttack [19]. Our proposed RCS (star shapes) always achieves a higher robust test accuracy than random selection (triangles) while consuming much less running time than pre-training on the entire set (squares). Experimental details are in Appendix B.4.
adversarial attacks [18, 19] via finetuning. The robustness transferability is of great importance to a pre-trained model's practicality in safety-critical downstream tasks [20, 21, 22, 23].
However, ACL is computationally expensive, which limits its scalability. At each training epoch, ACL first needs to conduct several backward propagations (BPs) on all training data to generate their adversarial variants and then train the model with those adversarial data. Even if we use the techniques of fast adversarial training [24, 25] that significantly reduce the need for conducting BPs per data, ACL still encounters the issue of the large scale of the training set that is commonly used for pre-training a useful representation.
Coreset selection (CS) [27, 28] that selects a small yet informative subset can reduce the need for the whole training set, but CS cannot be applied to ACL directly. Mirzasoleiman et al. [29] and Killamsetty et al. [30, 31, 32] have significantly accelerated the standard training by selecting an informative training subset. Recently, Dolatabadi et al. [33] proposed an adversarial CS method to accelerate the standard adversarial training [34, 35, 24] by selecting a representative subset that can accurately align the gradient of the full set. However, the existing CS methods require the label information of training data, which are not applicable to the ACL that learns from the unlabeled data.
To accelerate the ACL, this paper proposes a robustness-aware coreset selection (RCS) that selects a coreset without requiring the label information but still helps ACL to obtain an effective robust representation. RCS searches for a coreset that minimizes the representational divergence (RD) of a representation. RD measures the representation difference between the natural data and their virtual adversarial counterparts [36], in which virtual adversarial data can greatly alter the output distribution in the sense of feature representation.
Although solving the RCS via traversing all possible subsets is simple, it is computationally prohibitive. Therefore, we transform the RCS into a proxy problem of maximizing a set function that is theoretically shown monotone and  $\gamma$ -weakly submodular (see Theorem 1) subject to cardinality constraints [37, 38]. Then, we can apply the greedy search [37] to efficiently find a coreset that can minimize the RD. Notably, Theorem 2 provides the theoretical guarantee of optimality of our greedy-search solution.
Empirically, we demonstrate RCS can indeed speed up ACL [14] and its variants [17, 15]. As shown in Figure 1, RCS can speed up both ACL [14] and DynACL [17] by 4 times and obtain an effective representation without significantly hurting the robustness transferability (details in Section 4.1). Notably, to the best of our knowledge, we are the first to apply ACL on the large-scale ImageNet-1K [1] dataset to obtain a robust representation via RCS (see Section 4.2). Besides, we empirically show that RCS is compatible with standard adversarial training [34, 35, 24] as well in Section 4.3. Our comprehensive experiments corroborate that our proposed RCS is a unified and principled framework for efficient robust learning.
In this section, we introduce related work and preliminaries about contrastive learning [13, 14, 17].
Contrastive learning (CL). CL approaches are frequently used to leverage large unlabeled datasets for learning useful representations. Previous unsupervised methods that map similar samples to similar representations [39] have the issue of collapsing to a constant representation. CL addresses the representational collapse by introducing the negative samples [12]. Chen et al. [13] presented SimCLR that leverages the contrastive loss for learning useful representations and achieved significantly improved accuracy on the standard suit of downstream tasks. Recently, adversarial contrastive learning (ACL) [40, 14, 41, 15, 16, 42, 17, 43] that incorporates adversarial training with the contrastive loss [13] has become the most effective unsupervised approaches to learn robust representations. Jiang et al. [14] showed that ACL could exhibit better adversarial robustness on downstream tasks compared with standard CL (i.e., SimCLR [13]). Luo et al. [17] proposed to dynamically schedule the strength of data augmentations to narrow the gap between the distribution of training data and that of test data, thus enhancing the performance of ACL. Xu et al. [43] improved ACL from the lens of causality [44, 45] and proposed an adversarial invariant regularization that achieves new SOTA results.
However, ACL needs large computational resources to obtain useful representations from large-scale datasets. ACL needs to spend a large amount of running time generating adversarial training data during pre-training. Fast adversarial training (Fast-AT) [24] uses the fast gradient descent method (FGSM) [18] to accelerate the generation procedure of adversarial training data, thus speeding up robust training. A series of recent works [46, 47] followed this line and further improved the performance of fast AT. However, the time complexity of ACL is still proportional to the size of the training set. Besides, the large-scale training sets (e.g., ImageNet-1K [1] contains over 1 million training images) further contribute to the inefficiency of adversially pre-training procedures [48, 49]. To enable efficient ACL, we propose a novel CS method that can speed up ACL by decreasing the amount of training data.
Coreset selection (CS). CS aims to select a small yet informative data subset that can approximate certain desirable characteristics (e.g., the loss gradient) of the entire set [28]. Several studies have shown that CS is effective for efficient standard training in supervised [29, 31, 30] and semi-supervised [32] settings. Dolatabadi et al. [33] proposed adversarial coreset selection (ACS) that selects representative coresets of adversarial training data that can estimate the adversarial loss gradient on the entire training set. However, ACS is only adapted to supervised AT [34, 35]. Thus, ACS requires the label information and is not applicable to ACL. In addition, previous studies did not explore the influence of CS on the pre-trained model's transferability. To this end, we propose a novel CS method that does not require label information and empirically demonstrate that our proposed method can significantly accelerate ACL while only slightly hurting the transferability.
Here, we introduce the preliminaries of contrastive learning [13, 14, 17]. Let  $(\mathcal{X}, d_{\infty})$  be the input space  $\mathcal{X}$  with the infinity distance metric  $d_{\infty}(x, x') = \| x - x' \|_{\infty}$ , and  $\mathcal{B}_{\epsilon}[x] = \{x' \in \mathcal{X} \mid d_{\infty}(x, x') \leq \epsilon\}$  be the closed ball of radius  $\epsilon > 0$  centered at  $x \in \mathcal{X}$ .
ACL [14] and DynACL [17]. We first introduce the standard contrastive loss [13]. Let  $f_{\theta}:\mathcal{X}\to \mathcal{Z}$  be a feature extractor parameterized by  $\theta$ ,  $g:\mathcal{Z}\rightarrow \mathcal{V}$  be a projection head that maps representations to the space where the contrastive loss is applied, and  $\tau_i,\tau_j:\mathcal{X}\to \mathcal{X}$  be two transformation operations randomly sampled from a pre-defined transformation set  $\mathcal{T}$ . Given a minibatch  $B\sim \mathcal{X}^{\beta}$  consisting of  $\beta$  samples, we denote the augmented minibatch  $B^{\prime} = \{\tau_{i}(x_{k}),\tau_{j}(x_{k})\mid \forall x_{k}\in B\}$  consisting of  $2\beta$  samples. We take  $h_\theta (\cdot) = g\circ f_\theta (\cdot)$  and  $x_{k}^{u} = \tau_{u}(x_{k})$  for any  $x_{k}\sim \mathcal{X}$  and  $u\in \{i,j\}$ . The standard contrastive loss of a positive pair  $(x_k^i,x_k^j)$  is as follows:
$$
\ell_ {\mathrm {C L}} (x _ {k} ^ {i}, x _ {k} ^ {j}; \theta) = - \sum_ {u \in \{i, j \}} \log \frac {e ^ {\operatorname* {s i m} \left(h _ {\theta} (x _ {k} ^ {i}) , h _ {\theta} (x _ {k} ^ {j})\right) / t}}{\sum_ {x \in B ^ {\prime} \setminus \{x _ {k} ^ {u} \}} e ^ {\operatorname* {s i m} \left(h _ {\theta} (x _ {k} ^ {u}) , h _ {\theta} (x)\right) / t}},
$$
where  $\mathrm{sim}(p,q) = p^\top q / \| p\| \| q\|$  is the cosine similarity function and  $t > 0$  is a temperature parameter.
Given an unlabeled set  $X \sim \mathcal{X}^N$  consisting of  $N$  samples, the loss function of ACL [14] is  $\mathcal{L}_{\mathrm{ACL}}(X; \theta) = \sum_{k=1}^{N} \ell_{\mathrm{ACL}}(x_k; \theta)$  where
$$
\ell_{\mathrm{ACL}}(x_{k};\theta) = \big(\max_{\substack{\tilde{x}_{k}^{i}\in \mathcal{B}_{\epsilon}[x_{k}^{i}]\\ \tilde{x}_{k}^{j}\in \mathcal{B}_{\epsilon}[x_{k}^{j}]}}(1 + \omega)\cdot \ell_{\mathrm{CL}}(\tilde{x}_{k}^{i},\tilde{x}_{k}^{j};\theta)\big) + (1 - \omega)\cdot \ell_{\mathrm{CL}}(x_{k}^{i},x_{k}^{j};\theta),
$$
in which  $\omega \in [0,1]$  is a hyperparameter and  $\tilde{x}_k^i$  and  $\tilde{x}_k^j$  are adversarial data generated via projected gradient descent (PGD) [34] within the  $\epsilon$ -balls centered at  $x_k^i$  and  $x_k^j$ . Note that ACL [14] fixes  $\omega = 0$  while DynACL [17] dynamically schedules  $\omega$  according to its dynamic augmentation scheduler that gradually anneals from a strong augmentation to a weak one. We leave the details of the data augmentation scheduler [17] in Appendix B.2 due to the limited space.
Given an initial positive pair  $(x^{i,(0)}, x^{j,(0)})$ , PGD step  $T \in \mathbb{N}$ , step size  $\rho > 0$ , and adversarial budget  $\epsilon \geq 0$ , PGD iteratively updates the pair of data from  $\tau = 0$  to  $T - 1$  as follows:
$$
x ^ {i, (\tau + 1)} = \Pi_ {\mathcal {B} _ {\epsilon} [ x ^ {i, (0)} ]} \big (x ^ {i, (\tau)} + \rho \operatorname {s i g n} \left(\nabla_ {x ^ {i, (\tau)}} \ell_ {\mathrm {C L}} \left(x ^ {i, (\tau)}, x ^ {j, (\tau)}\right)\right),
$$
$$
x ^ {j, (\tau + 1)} = \Pi_ {\mathcal {B} _ {\epsilon} [ x ^ {j, (0)} ]} \big (x ^ {j, (\tau)} + \rho \mathrm {s i g n} (\nabla_ {x ^ {j, (\tau)}} \ell_ {\mathrm {C L}} (x ^ {i, (\tau)}, x ^ {j, (\tau)}) \big),
$$
where  $\Pi_{\mathcal{B}_{\epsilon}[x]}$  projects the data into the  $\epsilon$ -ball around the initial point  $x$ .
ACL is realized by conducting one step of inner maximization on generating adversarial data via PGD and one step of outer minimization on updating  $\theta$  by minimizing the ACL loss on generated adversarial data, alternatively. Note that the parameters of the projection head  $g$  are updated as well during ACL. Here we omit the parameters of  $g$  for notational simplicity since we only use the parameters of feature extractor  $f_{\theta}$  on downstream tasks after completing ACL.
3 Robustness-Aware Coreset Selection
In this section, we first introduce the representational divergence (RD), which is quantified by the distance of the representation between natural data and their virtual adversarial variants, as the measurement of the adversarial robustness of a representation. Then, we formulate the learning objective of the Robustness-aware Coreset Selection (RCS). Next, we theoretically show that our proposed RCS can be efficiently solved via greedy search. Finally, we give the algorithm of efficient ACL via RCS.
3.1 Representational Divergence (RD)
Adversarial robustness is the most significant property of adversarially pre-trained models, which could lead to enhanced robustness transferability. Inspired by previous studies [35, 50], we measure the adversarial robustness of the representation in an unsupervised manner using the representational divergence (RD). Given a natural data point  $x \sim \mathcal{X}$  and a model  $g \circ f_{\theta}: \mathcal{X} \to \mathcal{V}$  composed of a feature extractor  $f_{\theta}$  and a projector head  $g$ , RD of this data point  $\ell_{\mathrm{RD}}(x; \theta)$  is quantified by the distance between the representation of the natural data and that of its virtual adversarial counterpart [36], i.e.,
$$
\ell_ {\mathrm {R D}} (x; \theta) = d \left(g \circ f _ {\theta} (\tilde {x}), g \circ f _ {\theta} (x)\right) \quad \text {s . t .} \quad \tilde {x} = \underset {x ^ {\prime} \in \mathcal {B} _ {\epsilon} [ x ]} {\arg \max } d \left(g \circ f _ {\theta} \left(x ^ {\prime}\right), g \circ f _ {\theta} (x)\right), \tag {1}
$$
in which we can use PGD method [51] to generate adversarial data  $\tilde{x}$  within the  $\epsilon$ -ball centered at  $x$  and  $d(\cdot, \cdot): \mathcal{V} \times \mathcal{V} \to \mathbb{R}$  is a distance function, such as the Kullback-Leibler (KL) divergence [35], the Jensen-Shannon (JS) divergence [52], and the optimal transport (OT) distance [50]. We denote the RD on the unlabeled validation set  $U$  as  $\mathcal{L}_{\mathrm{RD}}(U; \theta) = \sum_{x_i \in U} \ell_{\mathrm{RD}}(x_i; \theta)$ . The smaller the RD is, the representations are of less sensitivity to adversarial perturbations, thus being more robust.
3.2 Learning Objective of Robustness-Aware Coreset Selection (RCS)
Our proposed RCS aims to select an informative subset that can achieve the minimized RD between natural data and their adversarial counterparts, thus expecting the selected coreset to be helpful in improving the adversarial robustness of representations. Therefore, given an unlabeled training set  $X \sim \mathcal{X}^N$  and an unlabeled validation set  $U \sim \mathcal{X}^M$  ( $M \ll N$ ), our proposed Robustness-aware Coreset Selection (RCS) searches for a coreset  $S^*$  such that
$$
S ^ {*} = \underset {S \subseteq X, | S | / | X | \leq k} {\arg \min } \mathcal {L} _ {\mathrm {R D}} (U; \underset {\theta} {\arg \min } \mathcal {L} _ {\mathrm {A C L}} (S; \theta)). \tag {2}
$$
Note that the subset fraction  $k \in (0,1]$  controls the size of the coreset, i.e.,  $|S^{*}| \leq kN$ . RCS only needs to calculate RD on the validation set (i.e.,  $\mathcal{L}_{\mathrm{RD}}(U)$ ) and ACL loss on the subset (i.e.,  $\mathcal{L}_{\mathrm{ACL}}(S)$ ), which can be computed in an unsupervised manner. Thus, RCS is applicable to ACL on unlabeled datasets, and compatible with supervised AT on labeled datasets as well, such as Fast-AT [24], SAT [34] and TRADES [35] (details in Appendix B.13). Intuitively, the coreset  $S^{*}$  found by RCS can make the pre-trained model via minimizing the ACL loss  $\mathcal{L}_{\mathrm{ACL}}(S^{*})$  achieve the minimized  $\mathcal{L}_{\mathrm{RD}}(U)$ , thus helping attain adversarially robust presentations, which could be beneficial to the robustness transferability to downstream tasks.
To efficiently solve the inner minimization problem of Eq. (2), we modify Eq. (2) by conducting a one-step gradient approximation as the first step, which keeps the same practice as [31, 30], i.e.,
$$
S ^ {*} = \underset {S \subseteq X, | S | / | X | \leq k} {\arg \min } \mathcal {L} _ {\mathrm {R D}} (U; \theta - \eta \nabla_ {\theta} \mathcal {L} _ {\mathrm {A C L}} (S; \theta)), \tag {3}
$$
where  $\eta > 0$  is the learning rate. We define a set function  $G: 2^X \to \mathbb{R}$  as follows:
$$
G _ {\theta} (S \subseteq X) \triangleq - \mathcal {L} _ {\mathrm {R D}} (U; \theta - \eta \nabla_ {\theta} \mathcal {L} _ {\mathrm {A C L}} (S; \theta)). \tag {4}
$$
We fix the size of the coreset, i.e.,  $|S^{*}| = k|X|$  being a constant. Then, the optimization problem in Eq. (2) can be reformulated as
$$
S ^ {*} = \underset {S \subseteq X, | S | / | X | = k} {\arg \max } G _ {\theta} (S). \tag {5}
$$
3.3 Method—Greedy Search
Solving Eq. (5) is formulated as maximizing a set function subject to a cardinality  $|S|$  constraint [53, 31, 32]. A naive solution to this problem is to traverse all possible subsets of size  $kN$  and select the subset  $S$  which has the largest value  $G_{\theta}(S)$ . But this naive solution is computationally prohibitive since it needs to calculate the loss gradient  $\nabla_{\theta}\mathcal{L}_{\mathrm{ACL}}(S;\theta)$  for  $\binom{N}{kN}$  times. Fortunately, Das and Kempe [37] and Gatmiry and Gomez-Rodriguez [38] proposed the greedy search algorithm to efficiently solve this kind of problem approximately if the set function is monotone and  $\gamma$ -weakly submodular. Note that the greedy search algorithm shown in Algorithm 1 only needs to calculate the loss gradient  $\lceil N/\beta \rceil + \lfloor kN/\beta \rfloor \ll \binom{N}{kN}$  times where  $\beta$  is batch size.
Definition 1 (Monotonicity and  $\gamma$ -weakly submodularity [37, 54]). Given a set function  $G:2^{X}\to \mathbb{R}$ , the marginal gain of  $G$  is defined as  $G(x|A)\triangleq G(A\cup \{x\}) - G(A)$  for any  $A\subset X$  and  $x\in X\setminus A$ . The set function  $G$  is monotone if  $G(x|A)\geq 0$  for any  $A\subset X$  and  $x\in X\setminus A$ . The set function  $G$  is called  $\gamma$ -weakly submodular if  $\sum_{x\in B}G(x|A)\geq \gamma [G(A\cup B) - G(A)]$  for some  $\gamma \in [0,1]$  and any disjoint subset  $A,B\subseteq X$ .
Assumption 1. The first-order gradients and the second-order gradients of  $\ell_{\mathrm{RD}}$  and  $\ell_{\mathrm{ACL}}$  are bounded w.r.t.  $\theta$ , i.e.,
$$
\| \frac {\partial}{\partial_ {\theta}} \ell_ {\mathrm {R D}} (x; \theta) \| \leq L _ {1}, \| \frac {\partial}{\partial_ {\theta}} \ell_ {\mathrm {A C L}} (x; \theta) \| \leq L _ {2}, \| \frac {\partial^ {2}}{\partial_ {\theta}} \ell_ {\mathrm {R D}} (x; \theta) \| \leq L _ {3}, \| \frac {\partial^ {2}}{\partial_ {\theta}} \ell_ {\mathrm {A C L}} (x; \theta) \| \leq L _ {4},
$$
where  $L_{1}, L_{2}, L_{3}$ , and  $L_{4}$  are positive constants.
Next, we theoretically show that a proxy set function  $\hat{G}_{\theta}(S)$  is monotone and  $\gamma$ -weakly submodular in Theorem 1.
Theorem 1. We define a proxy set function  $\hat{G}_{\theta}(S) \triangleq G_{\theta}(S) + |S| \sigma$ , where  $\sigma = 1 + \nu_1 + \nu_2 L_2 + \eta ML_2(L_1 + \eta k N(L_1 L_4 + L_2 L_3))$ ,  $\nu_1 \to 0^+$ , and  $\nu_2 > 0$  are positive constants. Given Assumption  $I$ ,  $\hat{G}_{\theta}(S)$  is monotone and  $\gamma$ -weakly submodular where  $\gamma > \gamma^* = \frac{1}{2\sigma - 1}$ .
The proof is in Appendix A.1. We construct a proxy optimization problem in Eq. (5) as follows:
$$
\hat {S} ^ {*} = \underset {S \subseteq X, | S | / | X | = k} {\arg \max } \hat {G} _ {\theta} (S). \tag {6}
$$
According to Theorem 1, a greedy search algorithm [37, 38] can be leveraged to approximately solve the proxy problem in Eq. (6) and it can provide the following optimality guarantee of the optimization problem in Eq. (5).
Algorithm 1 Robustness-aware Coreset Selection (RCS)
1: Input: Unlabeled training set  $X$ , unlabeled validation set  $U$ , batch size  $\beta$ , model  $g \circ f_{\theta}$ , learning rate for RCS  $\eta$ , subset fraction  $k \in (0,1]$   
2: Output: Coreset  $S$   
3: Initialize  $S \gets \emptyset, N \gets |X|$   
4: Split entire training set into minibatches  $\{B_m\}_{m=1}^{\lceil N/\beta \rceil}$   
5: for each minibatch  $B_m \subset X$  do  
6: Compute gradient  $q_m \gets \nabla_\theta \mathcal{L}_{\mathrm{ACL}}(B_m; \theta)$   
7: end for  
8: // Conduct greedy search via batch-wise selection  
9: for  $1, \ldots, \lfloor kN/\beta \rfloor$  do  
10: Compute gradient  $q_U \gets \nabla_\theta \mathcal{L}_{\mathrm{RD}}(U; \theta)$   
11: Initialize best_gain = -∞  
12: for each minibatch  $B_m \subset X$  do  
13: Compute marginal gain  $\hat{G}(B_m|S) \gets \eta q_U^\top q_m$  // refer to Eq. (7)  
14: if  $\hat{G}(B_m|S) > \text{best_gain}$  then  
15: Update  $s \gets m$ , best_gain  $\leftarrow \hat{G}(B_m|S)$   
16: end if  
17: end for  
18: Update  $S \gets S \cup B_s, X \gets X \setminus B_s$   
19: Update  $\theta \gets \theta - \eta q_s$   
20: end for
Theorem 2. Given a fixed parameter  $\theta$ , we denote the optimal solution of Eq. (5) as  $G_{\theta}^{*} = \sup_{S \subseteq X, |S| / |X| = k} G_{\theta}(S)$ . Then,  $\hat{S}^{*}$  in Eq. (6) found via greedy search satisfies
$$
G _ {\theta} \left(\hat {S} ^ {*}\right) \geq G _ {\theta} ^ {*} - \left(G _ {\theta} ^ {*} + k N \sigma\right) \cdot e ^ {- \gamma^ {*}}.
$$
Remark. The proof is in Appendix A.2. Theorem 2 indicates that the greedy search for solving the proxy problem in Eq. (6) can provide a guaranteed lower-bound of the original problem in Eq. (5), which implies that RCS via greedy search can help ACL to obtain the minimized RD and robust representations. We also empirically validate that ACL on the coreset selected by RCS can achieve a lower RD compared with ACL on the randomly selected subset in Figure 5 (in Appendix B.6), which empirically supports that RCS via greedy search is effective in achieving the minimized RD.