SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiawei Du , Qin Shi , Joey Tianyi Zhou
Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore {dujw, Joey_Zhou}@cfar.a-star.edu.sg, shiqin924924@gmail.com
Dataset distillation is a newly emerging task that synthesizes a small-size dataset used in training deep neural networks (DNNs) for reducing data storage and model training costs. The synthetic datasets are expected to capture the essence of the knowledge contained in real-world datasets such that the former yields a similar performance as the latter. Recent advancements in distillation methods have produced notable improvements in generating synthetic datasets. However, current state-of-the-art methods treat the entire synthetic dataset as a unified entity and optimize each synthetic instance equally. This static optimization approach may lead to performance degradation in dataset distillation. Specifically, we argue that static optimization can give rise to a coupling issue within the synthetic data, particularly when a larger amount of synthetic data is being optimized. This coupling issue, in turn, leads to the failure of the distilled dataset to extract the high-level features learned by the deep neural network (DNN) in the latter epochs. In this study, we propose a new dataset distillation strategy called Sequential Subset Matching (SeqMatch), which tackles this problem by adaptively optimizing the synthetic data to encourage sequential acquisition of knowledge during dataset distillation. Our analysis indicates that SeqMatch effectively addresses the coupling issue by sequentially generating the synthetic instances, thereby enhancing its performance significantly. Our proposed SeqMatch outperforms state-of-the-art methods in various datasets, including SVNH, CIFAR-10, CIFAR-100, and Tiny ImageNet. Our code is available at https://github.com/shqii1j/seqmatch.
Recent advancements in Deep Neural Networks (DNNs) have demonstrated their remarkable ability to extract knowledge from large-scale real-world data, as exemplified by the impressive performance of the large language model GPT-3, which was trained on a staggering 45 terabytes of text data [4]. However, the use of such massive datasets comes at a significant cost in terms of data storage, model training, and hyperparameter tuning.
The challenges associated with the use of large-scale datasets have motivated the development of various techniques aimed at reducing datasets size while preserving their essential characteristics. One such technique is dataset distillation [5, 6, 13, 22, 29, 35, 42, 48, 50, 51, 52, 53], which involves synthesizing a smaller dataset that effectively captures the knowledge contained within the original dataset. Models trained on these synthetic datasets have been shown to achieve comparable performance to those trained on the full dataset. In recent years, dataset distillation has garnered increasing attention from the deep learning community and has been leveraged in various practical applications, including continual learning [41, 52, 53], neural architecture search [21, 39, 40, 51, 53], and privacy-preserving tasks [12, 15, 31], among others.
Existing methods for dataset distillation, as proposed in [5, 11, 13, 33, 37, 38, 47, 51, 53], have improved the distillation performance through enhanced optimization methods. These approaches have achieved commendable improvements in consolidating knowledge from the original dataset and generating superior synthetic datasets. However, the knowledge condensed by these existing methods primarily originates from the easy instances, which exhibit a rapid reduction in training loss during the early stages of training. These easy instances constitute the majority of the dataset and typically encompass low-level, yet commonly encountered visual features (e.g. edges and textures [49]) acquired in the initial epochs of training. In contrast, the remaining, less frequent, but more challenging instances encapsulate high-level features (e.g. shapes and contours) that are extracted in the subsequent epochs and significantly impact the generalization capability of deep neural networks (DNNs). The findings depicted in Figure 1 reveal that an overemphasis on low-level features hinders the extraction and condensation of high-level features from hard instances, thereby resulting in a decline in performance.
Figure 1: Left: MTT [5] fails to extract adequate high-level features. The loss drop rate between easy and hard instances is employed as the metric to evaluate the condensation efficacy of low-level and high-level features. The upper solid lines represent the loss change of hard instances, while the lower dashed lines depict the loss change of easy instances. The inability to decrease the loss of hard instances indicates MTT's inadequacy in capturing high-level features. In contrast, our proposed SeqMatch successfully minimizes the loss for both hard and easy instances. Right: The consequent performance improvement of SeqMatch in CIFAR [25] datasets. Experiments are conducted with 50 images per class (ipc = 50).
In this paper, we investigate the factors that hinder the efficient condensation of high-level features in dataset distillation. Firstly, we reveal that DNNs are optimized through a process of learning from low-level visual features and gradually adapting to higher-level features. The condensation of high-level features determines the effectiveness of dataset distillation. Secondly, we argue that existing dataset distillation methods fail to extract high-level features because they treat the synthetic data as a unified entity and optimize each synthetic instance unvaryingly. Such static optimization makes the synthetic instances become coupled with each other easier in cases where more synthetic instances are optimized. As a result, increasing the size of synthetic dataset will over-condense the low-level features but fail to condense additional knowledge from the real dataset, let alone the higher-level features.
Building upon the insights derived from our analysis, we present a novel dataset distillation strategy, termed Sequential Subset Matching (Seq
Match), which is designed to extract both low-level and high-level features from the real dataset, thereby improving dataset distillation. Our approach adopts a simple yet effective strategy for reorganizing the synthesized dataset  $S$  during the distillation and evaluation phases. Specifically, we divide the synthetic dataset into multiple subsets and encourage each subset to acquire knowledge in the order that DNNs learn from the real dataset. Our approach can be seamlessly integrated into existing dataset distillation methods. The experiments, as shown in Figure 1, demonstrate that SeqMatch effectively enables the latter subsets to capture high-level features. This, in turn, leads to a substantial improvement in performance compared to the baseline method MTT, which struggles to compress higher-level features from the real dataset. Extensive experiments demonstrate that SeqMatch outperforms state-of-the-art methods, particularly in high compression ratio<sup>2</sup> scenarios, across a range of datasets including CIFAR-10, CIFAR-100, TinyImageNet, and subsets of the ImageNet.
In a nutshell, our contribution can be summarized as follows.
- we examine the inefficacy of current dataset distillation in condensing hard instances from the original dataset. We present insightful analyses regarding the plausible factors contributing to this inefficacy and reveal the inherent preference of dataset distillation in condensing knowledge.
- We thereby propose a novel dataset distillation strategy called Sequential Subset Matching (SeqMatch) to targetedly encourage the condensing of higher-level features. SeqMatch seamlessly integrates with existing dataset distillation methods, offering easy implementa
${}^{2}$  compression ratio  $=$  compressed dataset size / full dataset size [8]
tion. Experiments on diverse datasets demonstrate the effectiveness of SeqMatch, achieving state-of-the-art performance.
Coreset selection is the traditional dataset reduction approach by selecting representative prototypes from the original dataset[2, 7, 17, 43, 45]. However, the non-editable nature of the coreset limits its performance potential. The idea of synthesizing the "coreset" can be traced back to Wang et al. [47]. Compared to coreset selection, dataset distillation has demonstrated greatly superior performance. Based on the approach of optimizing the synthetic data, dataset distillation can be taxonomized into two types: data-matching methods and meta-learning methods [30].
Data-matching methods encourage the synthetic data to imitate the influence of the target data, involving the gradients, trajectories, and distributions. Zhao and Bilen [52] proposed distribution matching to update synthetic data. Zhao et al. [53] matched the gradients of the target and synthetic data in each iteration for optimization. This approach led to the development of several advanced gradient-matching methods[5, 20, 22, 51]. Trajectory-matching methods [5, 9, 13] further matched multi-step gradients to optimize the synthetic data, achieving state-of-the-art performance. Factorization-based methods [11, 29, 33] distilled the synthetic data into a low-dimensional manifold and used a decoder to recover the source instances from the factorized features.
Meta-learning methods treat the synthetic data as the parameters to be optimized by a meta (or outer) algorithm [3, 11, 32, 34, 37, 38, 54]. A base (or inner) algorithm solves the supervised learning problem and is nested inside the meta (or outer) algorithm with respect to the synthetic data. The synthetic data can be directly updated to minimize the empirical risk of the network. Kernel ridge regression (KRR) based methods [34, 37, 38] have achieved remarkable performance among meta-learning methods.
Both data-matching and meta-learning methods optimize each synthetic instance equally. The absence of variation in converged synthetic instances may lead to the extraction of similar knowledge and result in over-representation of low-level features.
Background Throughout this paper, we denote the target dataset as  $\mathcal{T} = \{(x_i,y_i)\}_{i = 1}^{|\mathcal{T}|}$ . Each pair of data sample is drawn i.i.d. from a natural distribution  $\mathcal{D}$ , and  $x_{i}\in \mathbb{R}^{d},y_{i}\in \mathcal{V} = \{0,1,\dots ,C - 1\}$  where  $d$  is the dimension of input data and  $C$  is the number of classes. We denote the synthetic dataset as  $\mathcal{S} = \{(s_i,y_i)\}_{i = 1}^{|S|}$  where  $s_i\in \mathbb{R}^d,y_i\in \mathcal{V}$ . Each class of  $\mathcal{S}$  contains ipc (images per class) data pairs. Thus,  $|\mathcal{S}| = \mathrm{ipc}\times C$  and ipc is typically set to make  $|\mathcal{S}|\ll |\mathcal{T}|$ .
We employ  $f_{\theta}$  to denote a deep neural network  $f$  with weights  $\theta$ . An ideal training progress is to search for an optimal weight parameter  $\hat{\theta}$  that minimizes the expected risk over the natural distribution  $\mathcal{D}$ , which is defined as  $L_{\mathcal{D}}(f_{\theta}) \triangleq \mathbb{E}_{(x,y) \sim \mathcal{D}}[\ell(f_{\theta}(x), y)]$ . However, as we can only access the training set  $\mathcal{T}$  sampled from the natural distribution  $\mathcal{D}$ , the practical training approach of the network  $f$  is to minimizing the empirical risk  $L_{\mathcal{T}}(f_{\theta})$  minimization (ERM) on the training set  $\mathcal{T}$ , which is defined as
$$
\hat {\theta} = \operatorname {a l g} \left(\mathcal {T}, \theta_ {0}\right) = \underset {\theta} {\arg \min } L _ {\mathcal {T}} \left(f _ {\theta}\right) \quad \text {w h e r e} \quad L _ {\mathcal {T}} \left(f _ {\theta}\right) = \frac {1}{| \mathcal {T} |} \sum_ {x _ {i} \in \mathcal {T}} \ell \left[ f _ {\theta} \left(x _ {i}\right), y _ {i} \right], \tag {1}
$$
where  $\ell$  can be any training loss function;  $\mathrm{alg}$  is the given training algorithm that optimizes the initialized weights parameters  $\theta_0$  over the training set  $\mathcal{T}$ ;  $\theta_0$  is initialized by sampling from a distribution  $P_{\theta_0}$ .
Dataset distillation aims to condense the knowledge of  $\mathcal{T}$  into the synthetic dataset  $S$  so that training over the synthetic dataset  $S$  can achieve a comparable performance as training over the target dataset  $\mathcal{T}$ . The objective of dataset distillation can be formulated as,
$$
\underset {(x, y) \sim \mathcal {D}, \theta_ {0} \sim P _ {\theta_ {0}}} {\mathbb {E}} \left[ \ell \left(f _ {\mathrm {a l g} (\mathcal {T}, \theta_ {0})} (x), y\right) \right] \simeq \underset {(x, y) \sim \mathcal {D}, \theta_ {0} \sim P _ {\theta_ {0}}} {\mathbb {E}} \left[ \ell \left(f _ {\mathrm {a l g} (\mathcal {S}, \theta_ {0})} (x), y\right) \right]. \tag {2}
$$
Gradient Matching Methods We take gradient matching methods as the backbone method to present our distillation strategy. Matching the gradients introduced by  $\mathcal{T}$  and  $S$  helps to solve  $S$
in Equation 2. By doing so, gradient matching methods achieve advanced performance in dataset distillation. Specifically, gradient matching methods introduce a distance metric  $D(\cdot, \cdot)$  to measure the distance between gradients. A widely-used distance metric [53] is defined as  $D(X, Y) = \sum_{i=1}^{I} \left(1 - \frac{\langle X_i, Y_i \rangle}{\|X_i\| \|Y_i\|}\right)$ , where  $X, Y \in \mathbb{R}^{I \times J}$  and  $X_i, Y_i \in \mathbb{R}^J$  are the  $i^{\text{th}}$  columns of  $X$  and  $Y$  respectively. With the defined distance metric  $D(\cdot, \cdot)$ , gradient matching methods consider solving
$$
\widehat {\mathcal {S}} = \underset { \begin{array}{c} \mathcal {S} \subset \mathbb {R} ^ {d} \times \mathcal {Y} \\ | \mathcal {S} | = \mathrm {i p c} \times C \end{array} } {\arg \min } \underset {\theta_ {0} \sim P _ {\theta_ {0}}} {\mathbb {E}} \left[ \sum_ {m = 1} ^ {M} \mathcal {L} (\mathcal {S}, \theta_ {m}) \right], \quad \text {w h e r e} \quad \mathcal {L} (\mathcal {S}, \theta) = D \left(\nabla_ {\theta} L _ {\mathcal {S}} (f _ {\theta}), \nabla_ {\theta} L _ {\mathcal {T}} (f _ {\theta})\right), \tag {3}
$$
where  $\theta_{i}$  is the intermediate weights which is continuously updated by training the network  $f_{\theta_0}$  over the target dataset  $\mathcal{T}$ . The methods employ  $M$  as the hyperparameter to control the length of teacher trajectories to be matched starting from the initialized weights  $\theta_0 \sim P_{\theta_0}$ .  $\mathcal{L}(\mathcal{S},\theta)$  is the matching loss. The teacher trajectory  $\{\theta_0,\theta_1,\dots ,\theta_M\}$  is equivalent to a series of gradients  $\{g_1,g_2,\dots ,g_M\}$ . To ensure the robustness of the synthetic dataset  $\mathcal{S}$  to different weights initializations,  $\theta_0$  will be sampled from  $P_{\theta_0}$  for many times. As a consequence, the distributions of the gradients for training can be represented as  $\{P_{g_1},P_{g_2},\dots ,P_{g_M}\}$ .
Algorithm 1 Training with SeqMatch in Distillation Phase.
Input: Target dataset  $\mathcal{T}$ ; Number of subsets  $K$ ; Iterations  $N$  in updating each subset; A base distillation method  $\mathcal{A}$ .  
1: Initialize the synthetic dataset  $\mathcal{S}_{\mathrm{all}}$   
2: Divide  $\mathcal{S}_{\mathrm{all}}$  into  $K$  subsets of equal size  $\lfloor \frac{|S_{\mathrm{all}}|}{K} \rfloor$ , i.e.,  $\mathcal{S}_{\mathrm{all}} = \mathcal{S}_1 \cup \mathcal{S}_2 \cup \dots \cup \mathcal{S}_K$   
3: for each  $\mathcal{S}_k$  do  
4: ▷ Optimize each subset  $\mathcal{S}_k$  sequentially:  
5: repeat  
6: if  $k = 1$  then  
7: Initialize network weights  $\theta_0^k \sim P_{\theta_0}$   
8: else  
9: Load network weights  $\theta_0^k \sim P_{\theta_N^{k-1}}$  saved in optimizing last subset  $\mathcal{S}_{k-1}$   
10: for  $i = 1$  to  $N$  do  
11: ▷ Update Network weights by subset  $\mathcal{S}_k$ :  
12:  $\theta_i^k = \operatorname{alg}(\mathcal{S}_k \cup \mathbb{S}^{(k-1)}, \theta_{i-1}^k)$   
13: ▷ Update  $\mathcal{S}_k$  by the base distillation method:  
14:  $\mathcal{S}_k \gets \mathcal{A}(\mathcal{T}, \mathcal{S}_k, \theta_i^k)$   
15: Record and save updated network weights  $\theta_N^{k-1}$   
16: until Converge  
Output: Distilled synthetic dataset  $\mathcal{S}_{\mathrm{all}}$
Increasing the size of a synthetic dataset is a straightforward approach to incorporating additional high-level features. However, our findings reveal that simply optimizing more synthetic data leads to an excessive focus on knowledge learned from easy instances. In this section, we first introduce the concept of sequential knowledge acquisition in a standard training procedure (refer to subsection 4.1). Subsequently, we argue that the varying rate of convergence causes certain portions of the synthetic data to abandon the extraction of further knowledge in the later stages (as discussed in Figure 4.2). Finally, we present our proposed strategy, Sequential Subset Matching (referred to as SeqMatch), which is outlined in Algorithm 1 in subsection 4.3.
4.1 Features Are Represented Sequentially
Many studies have observed the sequential acquisition of knowledge in training DNNs. Zeiler et al. [49] revealed that DNNs are optimized to extract low-level visual features, such as edges and textures, in the lower layers, while higher-level features, such as object parts and shapes, were
represented in the higher layers. Han et al. [16] leverages the observation that DNNs learn the knowledge from easy instances first, and gradually adapt to hard instances[1] to propose noisy learning methods. The sequential acquisition of knowledge is a critical aspect of DNN.
However, effectively condensing knowledge throughout the entire training process presents significant challenges for existing dataset distillation methods. While the synthetic dataset  $S$  is employed to learn from extensive teacher trajectories, extending the length of these trajectories during distillation can exacerbate the issue of domain shifting in gradient distributions, thereby resulting in performance degradation. This is primarily due to the fact that the knowledge extracted from the target dataset  $\mathcal{T}$  varies across different epochs, leading to corresponding shifts in the domains of gradient distributions. Consequently, the synthetic dataset  $S$  may struggle to adequately capture and consolidate knowledge from prolonged teacher trajectories.
To enhance distillation performance, a common approach is to match a shorter teacher trajectory while disregarding knowledge extracted from the latter epochs of  $\mathcal{T}$ . For instance, in the case of the CIFAR-10 dataset, optimal hyperparameters for  $M$  (measured in epochs) in the MTT [5] method were found to be 2, 20, 40 for ipc = 1, 10, 50 settings, respectively. The compromise made in matching a shorter teacher trajectory unexpectedly resulted in a performance gain, thereby confirming the presence of excessive condensation on easy instances.
Taking into account the sequential acquisition of knowledge during deep neural network (DNN) training is crucial for improving the generalization ability of synthetic data. Involving more synthetic data is the most straightforward approach to condense additional knowledge from longer teacher trajectory. However, our experimental findings, as illustrated in Figure 1, indicate that current gradient matching methods tend to prioritize the consolidation of knowledge derived from easy instances in the early epochs. Consequently, we conducted further investigations into the excessive emphasis on low-level features in existing dataset distillation methods.
4.2 Coupled Synthetic Dataset
The coupling issue within the synthetic dataset impedes its effectiveness in condensing additional high-level features. Existing dataset distillation methods optimize the synthetic dataset  $S$  as a unified entity, resulting in the backpropagated gradients used to update  $S$  being applied globally. The gradients on each instance only differ across different initializations and preassigned labels, implying that instances sharing a similar initialization within the same class will converge similarly. Consequently, a portion of the synthetic data only serves the purpose of alleviating the gradient matching error for the pre-existing synthetic data.
Consider a synthetic dataset  $S$  is newly initialized to be distilled from a target dataset  $\mathcal{T}$ . The distributions of the gradients for distillation are  $\{P_{g_1}, P_{g_2}, \dots, P_{g_M}\}$ , and the sampled gradients for training is  $\{g_1, g_2, \dots, g_M\}$ . Suppose that  $G$  is the integrated gradients calculated by  $S$ , by minimizing the loss function as stated in
Equation 3, the gradients used for updating  $s_i$  when  $\theta = \theta_m$  would be
Figure 2: The accuracy discrepancy between the networks trained using  $S^{+}$  and  $S^{-}$  separately. The discrepancy will increase with the magnitude of  $R(s_{i},f_{\theta_{m}})$ . These results verified the coupling issue between  $S^{+}$  and  $S^{-}$ , and our proposed method SeqMatch successfully mitigates the coupling issue. More experimental details can be found in subsection 5.3.
$$
\begin{array}{l} \nabla_ {s _ {i}} \mathcal {L} (\mathcal {S}, \theta_ {m}) = \frac {\partial \mathcal {L}}{\partial G} \cdot \frac {\partial G}{\partial \nabla_ {\theta_ {m}} \ell (f _ {\theta_ {m}} (s _ {i}) , y _ {i})} \cdot \frac {\partial \nabla_ {\theta_ {m}} \ell (f _ {\theta_ {m}} (s _ {i}) , y _ {i})}{\partial s _ {i}} \\ = \frac {\partial \mathcal {L}}{\partial G} \cdot R \left(s _ {i}, f _ {\theta_ {m}}\right), \quad \text {w h e r e} \quad R \left(s _ {i}, f _ {\theta_ {m}}\right) \triangleq \frac {\partial \nabla_ {\theta_ {m}} \ell \left(f _ {\theta_ {m}} \left(s _ {i}\right) , y _ {i}\right)}{\partial s _ {i}}. \tag {4} \\ \end{array}
$$
we have  $\frac{\partial G}{\partial\nabla_{\theta_m}\ell(f_{\theta_m}(s_i),y_i)} = 1$ , because  $G$  is accumulated by the gradients of each synthetic data, i.e.,  $G = \nabla_{\theta_m}L_S(f_{\theta_m}) = \sum_{i = 1}^{\| S\|}\nabla_{\theta_m}\ell (f_{\theta_m}(s_i),y_i)$ . Here we define the amplification function  $R(s_i,f_{\theta_m})\subset \mathbb{R}^d$ . Then, the gradients on updating synthetic instance  $\nabla_{s_i}\mathcal{L}(S,\theta_m)$  shares the same  $\frac{\partial\mathcal{L}}{\partial G}$  and only varies in  $R(s_i,f_{\theta_m})$ . The amplification function  $R(s_i,f_{\theta_m})$  is only affected by the pre-assigned label and initialization of  $s_i$ .
More importantly, the magnitude of  $R(s_{i},f_{\theta_{m}})$  determines the rate of convergence of each synthetic instance  $s_i$ . Sorted by the  $l_{1}$ -norm of amplification function  $\| R(s_i,f_{\theta_m})\| _1$  can be divided into two subsets  $S^{+}$  and  $S^{-}$ .  $S^{+}$  contains the synthetic instances with greater values of  $R(s_i,f_{\theta_m})$  than those in  $S^{-}$ . That implies that instances in  $S^{+}$  converge faster to minimize  $D(\nabla_{\theta_m}L_{S^+}(f_{\theta_m}),g_m)$ , and  $S^{+}$  is optimized to imitate  $g_{m}$ . On the other hand, the instances in  $S^{-}$  converge slower and are optimized to minimize  $D(\nabla_{\theta_m}L_{S^-}(f_{\theta_m}),\epsilon)$ , where  $\epsilon$  represents the gradients matching error  $\epsilon$  of  $S^{+}$ , i.e.,  $\epsilon = g_{m} - \nabla_{\theta_{m}}L_{S^{+}}(f_{\theta_{m}})$ . Therefore,  $S^{-}$  is optimized to imitate  $\epsilon$  and its effectiveness is achieved by compensating for the gradients matching error of  $S^{+}$ .  $S^{-}$  is coupled with  $S^{+}$  and unable to capture the higher-level features in the latter epochs.