SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xing Wei $^{1}$  Anjia Cao $^{1}$  Funing Yang $^{1}$  Zhiheng Ma $^{2*}$
$^{1}$ School of Software Engineering, Xi'an Jiaotong University
2Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences weixing@mail.xjtu.edu.cn zh.ma@siat.ac.cn {caoanjia7, moolink}@stu.xjtu.edu.cn
The success of deep learning relies heavily on large and diverse datasets, but the storage, preprocessing, and training of such data present significant challenges. To address these challenges, dataset distillation techniques have been proposed to obtain smaller synthetic datasets that capture the essential information of the originals. In this paper, we introduce a Sparse Parameterization for Epitomic datasEt Distillation (SPEED) framework, which leverages the concept of dictionary learning and sparse coding to distill epitomes that represent pivotal information of the dataset. SPEED prioritizes proper parameterization of the synthetic dataset and introduces techniques to capture spatial redundancy within and between synthetic images. We propose Spatial-Agnostic Epitomic Tokens (SAETs) and Sparse Coding Matrices (SCMs) to efficiently represent and select significant features. Additionally, we build a Feature-Recurrent Network (FReeNet) to generate hierarchical features with high compression and storage efficiency. Experimental results demonstrate the superiority of SPEED in handling high-resolution datasets, achieving state-of-the-art performance on multiple benchmarks and downstream applications. Our framework is compatible with a variety of dataset matching approaches, generally enhancing their performance. This work highlights the importance of proper parameterization in epitomic dataset distillation and opens avenues for efficient representation learning. Source code is available at https://github.com/MIV-XJTU/SPEED.
Deep learning has achieved remarkable success across diverse domains, thanks to its ability to extract insightful representations from large and diverse datasets [1-5]. Nevertheless, the storage, preprocessing, and training of these massive datasets introduce significant challenges that strain storage and computational resources. In response to these challenges, dataset distillation techniques have arisen as a means to distill a more compact synthetic dataset that encapsulates the pivotal information of the original dataset. The central concept of dataset distillation is the extraction of an epitomic representation, capturing the core characteristics and patterns of the original dataset while minimizing storage demands. By doing so, deep learning models trained on the distilled dataset can attain performance levels similar to those trained on the original dataset but with significantly reduced storage and training costs. This approach opens new horizons for applications requiring cost-effective storage solutions and expedited training times, such as neural architecture search [6-8] and continual learning [9-11].
The success of dataset distillation heavily relies on two essential factors: proper parameterization of the synthetic dataset and an effective design of the matching objective to align it with the original
*Zhiheng Ma is the corresponding author.
Figure 1: SPEEDED Overview. We take spatial-agnostic epitomic tokens as the shared dictionary of the dataset and perform multi-head sparse combinations to synthesize instance-specific features. Subsequently, we utilize feature-recurrent blocks to generate hierarchical representations for non-linear synthesis of image patches, while reusing the sparse features. In this way, we sparsely parameterize the dataset, alleviating the storage burden and producing highly representative synthetic images.
dataset. While many dataset distillation methods have primarily focused on optimizing the matching objective [12-18], the critical aspect of synthetic dataset parameterization has often been overlooked. Typically, these methods employ a naive image-independent parameterization approach, where each learnable parameter basis (synthetic image) is optimized independently. Although some studies have recognized the inefficiency of image-independent parameterization and explored the mutual coherence and relationship between different synthetic images to improve compression efficiency [19-22], none of the previous methods have fully considered the spatial redundancy that exists within individual images and between different images.
In this paper, we present an efficient parameterization framework, named Sparse Parameterization for Epitomic datasEt Distillation (SPEED), which addresses the aforementioned limitations. SPEED leverages principles from representation learning paradigms, including convolutional neural network (CNN) [23, 1], vision transformer (ViT) [3], dictionary learning [24-30], and sparse coding [31-36].
SPEED introduces Spatial-Agnostic Epitomic Tokens (SAETs) that are shared among all synthetic image patches, and employs the Sparse Coding Matrices (SCMs) to select the most significant tokens. Subsequently, these selected tokens are assembled sequentially to form higher-level representations that facilitate the reconstruction of synthetic patches via non-linear mapping. To further minimize storage requirements, we propose a Feature-Recurrent Network (FReeNet) that utilizes recurrent blocks to generate hierarchical features, with SAETs and SCMs shared by all blocks, while leveraging a multi-head mechanism to enhance feature diversity.
In comparison to previous methods, our approach demonstrates significant advantages in handling high-resolution real-world datasets with substantial spatial redundancy. Notably, it achieves outstanding performance on ImageNet subsets, surpassing most of the previous state-of-the-art methods [21, 37] by achieving an average improvement of  $11.2\%$  with 1 image per class storage space. Additionally, our sparse parameterization approach exhibits superior performance on unseen network architectures, outperforming previous state-of-the-art approaches [19, 21, 22], and even surpassing our own full-element baseline, highlighting the potential of sparse representation for storage efficiency and improved generalization abilities.
Our method demonstrates competitive results across three standard dataset distillation benchmarks, such as surpassing the previous state-of-the-art [22] on CIFAR100 by  $6.0\%$  and on TinyImageNet by  $10.9\%$  when using 1 image per class storage space. It also exhibits strong performance on downstream applications, such as continual learning. Furthermore, our framework is compatible with multiple existing matching objectives [13, 16, 17], generally enhancing their performance through the use of our sparse parameterization strategy.
In summary, our work highlights the importance of proper parameterization in epitomic dataset distillation and introduces the SPEED framework as a solution. We showcase its superiority in handling high-resolution datasets, achieving exceptional performance on benchmarks and downstream
applications. Our framework not only enhances storage efficiency but also improves generalization capabilities, opening new avenues for efficient representation learning in deep learning applications.
The purpose of dataset distillation [12] is to learn a synthetic dataset  $S = \{(\tilde{X}_i, y_i)\}_{i=1}^N$  that is much smaller in size than the original dataset  $\mathcal{T} = \{(X_i, y_i)\}_{i=1}^M$ , i.e.,  $N \ll M$ , while minimizing the loss of information. Formally, previous methods optimize the synthetic dataset by minimizing various matching objectives, all of which can be expressed using the following formulation:
$$
\mathcal {S} ^ {*} = \underset {\mathcal {S}} {\arg \min } \mathbb {E} _ {\theta \sim \Theta} \left[ \mathcal {D} \left(\varphi (\mathcal {T}, \theta), \varphi (\mathcal {S}, \theta)\right) \right], \tag {1}
$$
where  $\Theta$  represents the distribution used for initializing the network parameters,  $\theta$  parameterize the training network,  $\mathcal{D}(\cdot ,\cdot)$  is the dataset matching metric,  $\varphi (\cdot)$  maps the dataset to other informative spaces (e.g. gradient [13], feature [17, 18], and parameter spaces [16]).
However, rather than exclusively focusing on matching objectives, this paper introduces a universal parameterization method for synthetic datasets that can be seamlessly integrated with most existing matching objectives. The naive parameterization method, which optimizes each synthetic image  $\hat{X}$  independently [12-14, 16-18], fails to leverage shared information between images, resulting in unnecessary redundancy. In the following sections, we will present our parameterization framework, which decomposes the synthetic dataset into Spatial-Agnostic Epitomic Tokens, Sparse Coding Matrices, and a Feature-Recurrent Network. This framework significantly mitigates redundancy within and between images, irrespective of the spatial locations of features.
2.1 Spatial-Agnostic Recurrent Parameterization
Taking inspiration from the Vision Transformer (ViT) [3], we propose a more efficient parameterization approach applied at the patch level, utilizing cascade non-linear combinations to reduce complex and fine-level redundancy. To further minimize the storage footprint, we share Sparse Coding Matrices (SCMs) and Spatial-Agnostic Epitomic Tokens (SAETs) across all recurrent blocks. The main approach can be formulated as follows:
$$
\tilde {X} _ {i} = \Phi_ {\phi} (E, A _ {i}). \tag {2}
$$
Here  $\tilde{X}_i = [\tilde{x}_i^{(1)},\tilde{x}_i^{(2)},\dots,\tilde{x}_i^{(J)}]\in \mathbb{R}^{L\times J}$ , and  $\tilde{x}_i^j\in \mathbb{R}^L$  represents the  $j$ -th patch of  $\tilde{X}_i$ , which is flatten into a vector.  $L$  equals to the product of the patch height, width, and number of channels, and  $J$  is the total patch number. Similar to ViT, we divide the synthetic image into non-overlapping rectangular patches.  $E = [e_1,e_2,\dots,e_K]\in \mathbb{R}^{D\times K}$  is the Spatial-Agnostic Epitomic Tokens (SAETs) shared by all synthetic patches, where  $D$  is the feature dimension, and  $K$  is the total number of tokens.  $A_{i} = [a_{i}^{(1)},a_{i}^{(2)},\dots,a_{i}^{(J)}]\in \mathbb{R}^{K\times J}$  is the Sparse Coding Matrix (SCM) for the  $i$ -th synthetic image, where  $a_{i}^{(j)}\in \mathbb{R}^{K}$  is the specific coding vector for patch  $\tilde{x}_i^{(j)}$ .  $A_{i}$  will be further sparsified and saved in a storage-efficient format.  $\Phi_{\phi}(\cdot)$  is a non-linear recurrent transformer-style network that maps SAETs and the SCM to a synthetic image. Its learnable parameters are denoted as  $\phi$ . This network is referred to as the Feature-Recurrent Network (FReeNet), which is described in detail below.
Feature-Recurrent Network. In the Feature-Recurrent Network (FReeNet), each recurrent block shares the same SAETs and SCMs for the sake of parameter efficiency. However, this shared approach can lead to a lack of diversity in the resulting representations, as a single pool of SAETs must model multi-scale features. Drawing inspiration from the multi-head mechanism introduced in the transformer architecture, we introduce the concept of "multi-head" SAETs, aiming to strike a balance between storage efficiency and feature diversity. Initially, the original SAETs are split along the feature dimension to create multiple SAET pools, denoted as  $\{E^h\}_{h=1}^H$ , where  $E^h \in \mathbb{R}^{\frac{D}{H} \times K}$ . Additionally, each pool is assigned an independent SCM, denoted as  $\{A_i^h\}_{h=1}^H$ , where  $A_i^h \in \mathbb{R}^{K \times J}$ . The coding matrix  $A_i^h$  will undergo further sparsification to select the most significant tokens. We refer to the mechanism that combines the multi-head SAETs and SCMs as Multi-Head Sparse Coding (MHSC), which can be formulated as follows:
$$
\mathrm {M H S C} _ {r} \left(\left\{E ^ {h} \right\} _ {h = 1} ^ {H}, \left\{A _ {i} ^ {h} \right\} _ {h = 1} ^ {H}\right) = W _ {r} \left[ E ^ {1} A _ {i} ^ {1}, E ^ {2} A _ {i} ^ {2}, \dots , E ^ {H} A _ {i} ^ {H} \right] + b _ {r}, \tag {3}
$$
where  $W_{r} \in \mathbb{R}^{D \times D}$  is a linear projection and  $b_{r}$  is the bias, both of which are specific to each recurrent block and not shared across blocks. Using MHSC as the central component, we can construct the FReeNet in a recurrent manner, with SAETs and SCMs shared across different scales:
$$
Z _ {r} ^ {\prime} = \operatorname {L N} _ {r} ^ {1} \left(\operatorname {M H S C} _ {r} \left(\left\{E ^ {h} \right\} _ {h = 1} ^ {H}, \left\{A _ {i} ^ {h} \right\} _ {h = 1} ^ {H}\right)\right) + Z _ {r - 1}), \quad r = 1, 2, \dots , R,
$$
$$
Z _ {r} = \mathrm {L N} _ {r} ^ {2} \left(\mathrm {M L P} _ {r} \left(Z _ {r} ^ {\prime}\right) + Z _ {r} ^ {\prime}\right), \quad r = 1, 2, \dots , R, \tag {4}
$$
$$
\tilde {X} _ {i} = W Z _ {R} + b,
$$
where  $R$  is the total number of recurrent blocks, and  $Z_{R} \in \mathbb{R}^{D \times J}$  is the output of the last block.  $Z_{0}$  is initialized with a zero matrix.  $W \in \mathbb{R}^{L \times D}$  and  $b$  make up the final linear projection layer, and  $\tilde{X}_{i} \in \mathbb{R}^{L \times J}$  is the output synthetic image, which is then rearranged into its original shape. MLP stands for the multi-layer perceptron. In our implementation, we set the MLP to have one hidden layer with the same dimension as the input, and incorporate layer normalization (LN) [38] and residual connection [2] in each block. It is worth noting that the parameters other than the SAETs and SCMs are not shared between different blocks to ensure that each block processes a different scale of features. Despite the use of shared SCMs across different blocks, our analysis demonstrates that SCMs still occupy the majority of the stored parameters. Therefore, we introduce a sparsification method to further enhance the storage efficiency of SCMs, motivated by the theories and techniques of sparse coding [31, 32]. In the subsequent section, we will reuse  $\tilde{X}_{i} = \Phi_{\phi}\left(\left\{E^{h}\right\}_{h=1}^{H},\left\{A_{i}^{h}\right\}_{h=1}^{H}\right)$  to refer to the multi-head implementation of the FReeNet, without any ambiguity.
2.2 Training Objective and Feature Sparsification
Since the  $\ell_0$  norm is not differentiable and difficult to optimize [39-42], we instead adopt the  $\ell_1$  norm as the sparsity penalty function. By promoting sparsity in solutions, it can effectively remove redundant features [40, 43-45]. Our optimization objective can be expressed as follows:
$$
\mathcal {S} = \left\{\left(\Phi_ {\phi} (\{E ^ {h} \} _ {h = 1} ^ {H}, \{A _ {i} ^ {h} \} _ {h = 1} ^ {H}), y _ {i}\right) \right\} _ {i = 1} ^ {N},
$$
$$
\underset {\left\{E ^ {h} \right\} _ {h = 1} ^ {H}, \left\{\left. A _ {i} ^ {h} \right\} _ {h = 1} ^ {H} \right\} _ {i = 1} ^ {N}, \phi} {\arg \min } \mathbb {E} _ {\theta \sim \Theta} \left[ \mathcal {D} \left(\varphi (\mathcal {T}, \theta), \varphi (\mathcal {S}, \theta)\right) \right] + \lambda \sum_ {i = 1} ^ {N} \sum_ {h = 1} ^ {H} | | A _ {i} ^ {h} | | _ {1}, \tag {5}
$$
where  $||\cdot ||_1$  is the  $\ell_1$  norm of a matrix,  $\lambda$  controls the amount of regularization. Using this approach, we decompose synthetic images into a multi-head SAET  $\{E^h\}_{h = 1}^H$  and network parameters  $\phi$  that are shared by all synthetic images, and a multi-head SCM  $\{A_i^h\}_{i = 1}^H$  for each synthetic image.
Feature Sparsification with Global Semantic Preservation. Sparse codes allow for the ranking of features [33, 46], with higher coefficients indicating more important features. Therefore, we can select the most influential sub-parameters of SCM to achieve the desired sparsity  $||A_i^h ||_0 \leq k$ . This process reduces storage inefficiency while preserving the global semantics of synthetic images. Moreover, all existing matching objectives [13, 16-18] optimize synthetic images using a specific training network, but these synthetic images are then used to train various agnostic network architectures. By pruning unnecessary features of synthetic images, we can further enhance their generalization ability on unseen network architectures. We experimentally validate this claim in Sec. 3.3. Specifically, given a learned SCM  $A \in \mathbb{R}^{K \times J}$ , we assign a binary mask  $B \in \mathbb{R}^{K \times J}$  to it:
$$
B [ i, j ] = \left\{ \begin{array}{l l} 1, & A [ i, j ] \in \operatorname {t o p k} (\operatorname {a b s} (A)) \\ 0, & \text {o t h e r w i s e} \end{array} , \bar {A} = B \odot A, \right. \tag {6}
$$
where  $\mathrm{topk}(\cdot)$  obtains the largest  $k$  elements, and  $\mathrm{abs}(\cdot)$  takes the absolute value of each element of the input matrix. We operate a Hadamard product on the two to preserve the top- $k$  efficient elements of the SCM, i.e., select the most critical epitomic features. By operating on each learned SCM, we receive  $\{\{\tilde{A}_i^h\}_{h=1}^H\}_{i=1}^N$ , which can be directly applied to synthesize images.
For the compressed storage of SCMs, we adopt the widely-used coordinate (COO) format, utilizing the uint8 data type to store the row and column coordinates of non-zero elements as the size of the sparse matrix is always smaller than or equal to  $256 \times 256$  in our implementation. Consequently, the storage needed for each non-zero element is 1.5 times that of a single ffloat32 tensor. In this way, the storage complexity of SCMs can be greatly reduced from  $O(NHKJ)$  to  $O(NHk)$ , where  $k \ll KJ$ . The algorithm of our approach is summarized in Alg. 1.
Algorithm 1 Sparse Parameterization for Epitomic Dataset Distillation (SPEED).
Input:  $\mathcal{T}$ : original dataset;  $N$ : total number of synthetic images;  $H$ : number of heads;  $k$ : expected number of non-zero elements of SCM; SPARSIFY( $\cdot, \cdot$ ): feature sparsification.
1: Randomly initialize SAETs  $\{E^h\}_{h = 1}^H$  ,SCMs  $\{\{A_i^h\}_{h = 1}^H\}_{i = 1}^N$  , and the parameters  $\phi$  of FReeNet
2: for each distillation step... do
3: Get a random initialized training backbone for the matching strategy
4: Construct the synthetic dataset:  $\mathcal{S} = \left\{\left(\Phi_{\phi}\left(\{E^{h}\}_{h = 1}^{H},\{A_{i}^{h}\}_{h = 1}^{H}\right),y_{i}\right)\right\}_{i = 1}^{N}$
5: Inner optimization of the training backbone with respect to the matching strategy (if necessary)
6: Compute the objective function combined matching loss with sparsity penalty as Eq. (5)
7: Optimize  $\{E^h\}_{h = 1}^H,\{\{A_i^h\}_{h = 1}^H\}_{i = 1}^N$  ,  $\phi$  with respect to the objective function
8: end for
9: for  $i = 1$  to  $N$  do
0: for  $h = 1$  to  $H$  do
1: Select most significant features:  $\bar{A}_i^h\gets$  SPARSIFY  $(A_i^h,k)$  , according to Eq. (6)
2: Convert  $\bar{A}_i^h$  to compressed storage format
3: end for
4: end for
Output and Save:  $\{E^h\}_{h = 1}^H,\{\{\bar{A}_i^h\}_{h = 1}^H\}_{i = 1}^N$  and  $\phi$
Storage Analysis. Our storage budget is constrained by the upper bound determined by the storage requirements of the original synthetic images. Specifically, for a given budget of  $n$  images per class (IPC) [12, 13] with  $c$  total classes, the maximum budget is limited to  $cnLJ$ , where  $LJ$  is the storage requirement for a single synthetic image. Therefore, the following inequality must be satisfied:
Noticing that SAETs and FReeNet are shared by all synthetic images, and  $1.5Hk \ll LJ$ , we can synthesize a much larger amount of synthetic images than the original one. For instance, considering the IPC 1 storage budget on CIFAR100, i.e.,  $c = 100$  and  $n = 1$ , we set  $D = 96$ ,  $K = 64$ ,  $H = 3$ ,  $k = 48$ ,  $R = 2$ ,  $L = 48$ , and  $J = 64$ , resulting in a more informative synthetic dataset with a size of  $N = 1100$  (11 final images for each class).
In this section, we first evaluate our method and compare it with previous work. Then, we conduct generalization experiments and perform ablation studies. More results and detailed values for hyperparameters will be extensively discussed in the appendix. To quantify the performance and guarantee the fairness of the comparison, we use the default Conv-InstanceNorm-ReLU-AvgPool ConvNet with 128 channels as our training backbone, consistent with previous methods. We adopt trajectory matching [16] as our default matching objective.
3.1 Comparisons
Standard Benchmarks. We first conduct experiments on three standard dataset distillation benchmark datasets: CIFAR10 [48] and CIFAR100 [48] at a resolution of  $32 \times 32$ , and TinyImageNet [49] at a resolution of  $64 \times 64$ . To adhere to the standard protocol, we employ a 3-layer ConvNet and a 4-layer ConvNet for training and evaluating CIFAR and TinyImageNet, respectively. As illustrated in Tab. 1, SPEED achieves highly competitive results on all three datasets. Remarkably, we achieve a test accuracy of  $40.0\%$  and  $26.9\%$  on CIFAR100 and TinyImageNet, respectively, using 1 image per class (IPC) storage space, representing improvements of  $6.0\%$  and  $10.9\%$  over the previous state-of-the-art [22].
ImageNet Subsets. We evaluate the performance of SPEED on the high-resolution ImageNet [50] subsets and achieve new state-of-the-art results, as shown in Tab. 2. Consistent with [16, 21], we split
Dataset	CIFAR10	CIFAR100	TinyImageNet
IPC	1	10	50	1	10	50	1	10	50
Coreset	Random	14.4±2.0	26.0±1.2	43.4±1.0	4.2±0.3	14.6±0.5	30.0±0.4	1.4±0.1	5.0±0.2	15.0±0.4
Herding	21.5±1.3	31.6±0.7	40.4±0.6	8.4±0.3	17.3±0.3	33.7±0.5	2.8±0.2	6.3±0.2	16.7±0.3
K-Center	23.3±0.9	36.4±0.6	48.7±0.3	8.6±0.3	20.7±0.2	33.6±0.4	2.7±0.2	7.8±0.4	16.7±0.4
Forgetting	13.5±1.2	23.3±1.0	23.3±1.1	4.5±0.3	9.8±0.2	-	1.6±0.1	5.1±0.2	15.0±0.3
Matching	DC [13]	28.3±0.5	44.9±0.5	53.9±0.5	12.8±0.3	25.2±0.3	32.1±0.3	-	-	-
DSA [14]	28.8±0.7	52.1±0.5	60.6±0.5	13.9±0.3	32.3±0.3	42.8±0.4	-	-	-
KIP [47]	49.9±0.2	62.7±0.3	68.6±0.2	15.7±0.2	28.3±0.1	-	-	-	-
DM [17]	26.0±0.8	48.9±0.6	63.0±0.4	11.4±0.3	29.7±0.3	43.6±0.4	3.9±0.2	12.9±0.4	24.1±0.3
TM [16]	46.3±0.8	65.3±0.7	71.6±0.2	24.3±0.3	40.1±0.4	47.7±0.2	8.8±0.3	23.2±0.2	28.0±0.3
FRePo [37]	46.8±0.7	65.5±0.4	71.7±0.2	28.7±0.1	42.5±0.2	44.3±0.2	15.4±0.3	25.4±0.2	-
Parameters / Class	3,072	30,720	153,600	3,072	30,720	153,600	12,288	122,880	614,400
Param.	IDC [19]	50.0±0.4	67.5±0.5	74.5±0.1	-	44.8±0.2	-	-	-	-
HaBa [21]	48.3±0.8	69.9±0.4	74.0±0.2	33.4±0.4	40.2±0.2	47.0±0.2	-	-	-
RTP [22]	66.4±0.4	71.2±0.4	73.6±0.5	34.0±0.4	42.9±0.7	-	16.0±0.7	-	-
SPEED (Ours)	63.2±0.1	73.5±0.2	77.7±0.4	40.0±0.4	45.9±0.3	49.1±0.2	26.9±0.3	28.8±0.2	30.1±0.3
Whole Dataset	84.8±0.1	56.2±0.3	37.6±0.4
Table 1: Comparisons with previous dataset distillation and coreset selection methods on standard benchmarks. "Matching" refers to dataset distillation methods with specific matching objectives. "Param." refers to dataset distillation methods with synthetic data parameterization. The bold numbers represent the highest accuracy achieved in each category.
Dataset	ImageNette	ImageWoof	ImageFruit	ImageMeow	ImageSquawk	ImageYellow
IPC	1	10	1	10	1	10	1	10	1	10	1	10
TM [16]	47.7±0.9	63.0±1.3	28.6±0.8	35.8±1.8	26.6±0.8	40.3±1.3	30.7±1.6	40.4±2.2	39.4±1.5	52.3±1.0	45.2±0.8	60.0±1.5
FRePo [37]	48.1±0.7	66.5±0.8	29.7±0.6	42.2±0.9	-	-	-	-	-	-	-	-
Parameters / Class	49,152	491,520	49,152	491,520	49,152	491,520	49,152	491,520	49,152	491,520	49,152	491,520
HaBa [21]	51.9±1.7	64.7±1.6	32.4±0.7	38.6±1.3	34.7±1.1	42.5±1.6	36.9±0.9	42.9±0.9	41.9±1.4	56.8±1.0	50.4±1.6	63.0±1.6
SPEED (Ours)	66.9±0.7	72.9±1.5	38.0±0.9	44.1±1.4	43.4±0.6	50.0±0.8	43.6±0.7	52.0±1.3	60.9±1.0	71.8±1.3	62.6±1.3	70.5±1.5
Whole Dataset	87.4±1.0	67.0±1.3	63.9±2.0	66.7±1.1	87.5±0.3	84.4±0.6
Table 2: Comparisons with previous methods on high-resolution ImageNet subsets.
ImageNet into 6 subsets, namely, ImageNette, ImageWoof, ImageFruit, ImageMeow, ImageSquawk, and ImageYellow, each consisting of 10 classes with resolutions of  $128 \times 128$ . And a 5-layer ConvNet is employed as the model for both training and evaluation.