SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Donghyeok Shin* KAIST
tlsehdgur0@kaist.ac.kr
Seungjae Shin* KAIST
tmdwo0910@kaist.ac.kr
II-Chul Moon  
AIST, Summary.AI
icmoon@kaist.ac.kr
This paper presents FreD, a novel parameterization method for dataset distillation, which utilizes the frequency domain to distill a small-sized synthetic dataset from a large-sized original dataset. Unlike conventional approaches that focus on the spatial domain, FreD employs frequency-based transforms to optimize the frequency representations of each data instance. By leveraging the concentration of spatial domain information on specific frequency components, FreD intelligently selects a subset of frequency dimensions for optimization, leading to a significant reduction in the required budget for synthesizing an instance. Through the selection of frequency dimensions based on the explained variance, FreD demonstrates both theoretical and empirical evidence of its ability to operate efficiently within a limited budget, while better preserving the information of the original dataset compared to conventional parameterization methods. Furthermore, based on the orthogonal compatibility of FreD with existing methods, we confirm that FreD consistently improves the performances of existing distillation methods over the evaluation scenarios with different benchmark datasets. We release the code at https://github.com/sdh0818/FreD.
The era of big data presents challenges in data processing, analysis, and storage; and researchers have studied the concept of dataset distillation [20, 26, 14] to resolve these challenges. Specifically, the objective of dataset distillation is to synthesize a dataset with a smaller cardinality that can preserve the performance of original large-sized datasets in machine learning tasks. By distilling the key features from the original dataset into a condensed dataset, less computational resources and storage space are required while maintaining the performances from the original dataset. Dataset distillation optimizes a small-sized variable to represent the input, not the model parameters. This optimization leads the variable to store a synthetic dataset, and the variable is defined in the memory space with the constraint of limited capacity. Since dataset distillation involves the optimization of data variables with limited capacity, distillation parameter designs, which we refer to as parameterization, could significantly improve corresponding optimization while minimizing memory usage.
Some of the existing distillation methods, i.e. 2D image dataset distillations [24, 14, 3, 25, 27], naively optimize data variables embedded on the input space without any transformation or encoding. We will refer to distillation on the provided input space as spatial domain distillation, as an opposite concept of frequency domain distillation that is the focus of this paper. The main drawback of spatial domain distillation would be the difficulty in specifying the importance of each pixel dimension on the spatial domain, so it is necessary to utilize the same budget as the original dimension for representing a single instance. From the perspective of a data variable, which needs to capture the key information of the original dataset with a limited budget, the variable modeling with whole dimensions becomes a significant bottleneck that limits the number of distilled data instances. Various spatial domain-based
parameterization methods [7, 10] have been proposed to overcome this problem, but they are either highly vulnerable to instance-specific information loss [7] or require additional training with an auxiliary network [10].
This paper argues that the spatial domain distillation has limitations in terms of 1) memory efficiency and 2) representation constraints of the entire dataset. Accordingly, we propose a novel Frequency domain-based dataset Distillation method, coined FreD, which contributes to maintaining the task performances of the original dataset based on the limited budget. FreD employs a frequency-based transform to learn the data variable in the transformed frequency domain. Particularly, our proposed method selects and utilizes a subset of frequency dimensions that are crucial for the formation of an instance and the corresponding dataset. By doing so, we are able to achieve a better condensed representation of the original dataset with even fewer dimensions, which corresponds to significant efficiency in memory. Throughout the empirical evaluations with different benchmarks, FreD shows consistent improvements over the existing methods regardless of the distillation objective utilized.
2.1 Basic Notations
This paper primarily focuses on the dataset distillation for classification tasks, which is a widely studied scenario in the dataset distillation community [20, 26]. Given  $C$  classes, let  $\mathcal{X} \in \mathbb{R}^d$  denote the input variable space, and  $\mathcal{Y} = \{1, 2, \dots, C\}$  represent the set of candidate labels. Our dataset is  $D = \{(x_i, y_i)\}_{i=1}^N \subseteq \mathcal{X} \times \mathcal{Y}$ . We assume that each instance  $(x, y)$  is drawn i.i.d from the data population distribution  $\mathcal{P}$ . Let a deep neural network  $\phi_\theta : \mathcal{X} \to \mathcal{Y}$  be parameterized by  $\theta \in \Theta$ . This paper employs cross-entropy for the generic loss function,  $\ell(x, y; \theta)$ .
2.2 Previous Researches: Optimization and Parameterization of Dataset Distillation
Dataset Distillation Formulation. The goal of dataset distillation is to produce a cardinality-reduced dataset  $S$  from the given dataset  $D$ , while maximally retaining the task-relevant information of  $D$ . The objective of dataset distillation is formulated as follows:
$$
\min  _ {S} \mathbb {E} _ {(x, y) \in \mathcal {P}} [ \ell (x, y; \theta_ {S}) ] \text {w h e r e} \theta_ {S} = \arg \min  _ {\theta} \frac {1}{| S |} \sum_ {\left(x _ {i}, y _ {i}\right) \in S} \ell \left(x _ {i}, y _ {i}; \theta\right) \tag {1}
$$
However, the optimization of Eq. (1) is costly and not scalable since it is a bi-level problem [26, 17]. To avoid this problem, various proxy objectives are proposed to match the task-relevant information of  $D$  such as gradient [26], features [25], trajectory [3], etc. Throughout this paper, we generally express these objectives as  $\mathcal{L}_{DD}(S,D)$ .
Parameterization of  $S$ . Orthogonal to the investigation on objectives, other researchers have also examined the parameterization and corresponding dimensionality of the data variable,  $S$ . By parameterizing  $S$  in a more efficient manner, rather than directly storing input-sized instances, it is possible to distill more instances and enhance the representation capability of  $S$  for  $D$  [7, 10]. Figure 1 divides the existing methods into three categories.
As a method of Figure 1b, HaBa [10] proposed a technique for dataset factorization, which involves breaking the dataset into bases and hallucination networks for diverse samples. However, incorporating an additional network in distillation requires a separate budget, which is distinct from the data instances.
As a method of Figure 1c, IDC [7] proposed the utilization of an upsampler module in dataset distillation
(a) Dataset distillation on input-sized variable.
(b) Dataset distillation on variable with parameterized transform.
Figure 1: Comparison of different parameterization strategies for optimization of  $S$ .
to increase the number of data instances. This parameterization enables the condensation of a single instance with reduced spatial dimensions, thereby increasing the available number of instances. However, the compression still operates on the spatial domain, which results in a significant information loss per instance. Please refer to Appendix A.1 for the detailed literature reviews.
(a) Statistical properties of dataset on spatial and frequency domain.
(b) Tendency of dataset distillation loss.
Figure 2: Visualization of the information concentrated property. (a) On the frequency domain, a large proportion of both amplitude (left) and explained variance ratio (right) are concentrated in a few dimensions. (b) The magnitude of the gradient for  $\mathcal{L}_{DD}(S,D)$  is also concentrated in a few frequency dimensions. We utilize trajectory matching for  $\mathcal{L}_{DD}(S,D)$ . A brighter color denotes a higher value. Best viewed in color.
The core idea of this paper is to utilize a frequency domain to compress information from the spatial domain into a small number of frequency dimensions. Therefore, we briefly introduce the frequency transform, which is a connector between spatial and frequency domains. Frequency-transform function, denoted as  $\mathcal{F}$ , converts an input signal  $x$  to the frequency domain. It results in the frequency representation,  $f = \mathcal{F}(x) \in \mathbb{R}^{d_1 \times d_2}$ . The element-wise expression of  $f$  with  $x$  is as follows:
$$
f _ {u, v} = \sum_ {a = 0} ^ {d _ {1} - 1} \sum_ {b = 0} ^ {d _ {2} - 1} x _ {a, b} \phi (a, b, u, v) \tag {2}
$$
Here,  $\phi(a, b, u, v)$  is a basis function for the frequency domain, and the form of  $\phi(a, b, u, v)$  differs by the choice of specific transform function,  $\mathcal{F}$ . In general, an inverse of  $\mathcal{F}, \mathcal{F}^{-1}$ , exists so that it enables the reconstruction of the original input  $x$ , from its frequency components  $f$  i.e.  $x = \mathcal{F}^{-1}(\mathcal{F}(x))$ .
A characteristic of the frequency domain is that there exist specific frequency dimensions that encapsulate the major information of data instances from the other domain. According to [1, 21], natural images tend to exhibit the energy compaction property, which is a concentration of energy in the low-frequency region. Consequently, it becomes feasible to compress the provided instances by exclusively leveraging the low-frequency region. Remark 1 states that there exists a subset of frequency dimensions that minimize the reconstruction error of a given instance as follows:
Remark 1. ([16]) Given  $d$ -dimension data instance of a signal  $x = [x_0, \dots, x_d]^T$ , let  $f = [f_0, \dots, f_d]^T$  be its frequency representation with discrete cosine transform  $(DCT)$ , i.e.  $f = DCT(x)$ . Also, let  $f^{(k)}$  denote the  $k$  elements of  $f$  while other elements are 0. Then, the minimizer of the reconstruction error  $\|x - DCT^{-1}(f^{(k)})\|_2$  is  $[f_0, \dots, f_k, 0, \dots, 0]$ .
Figure 2a supports the remark. When we apply a frequency transform to an image, we can see that the coefficients in the low-frequency region have very large values compared to the coefficients in other regions. Also, as shown in Remark 1, the reconstructed image is quite similar to the original image while only utilizing the lowest frequency dimensions. Beyond the instance level, The right side of Figure 2a represents that certain dimensions in the frequency domain exhibit a concentration of variance. It suggests that the frequency domain requires only a small number of dimensions to
2Although some frequency transform handles the complex space, we use the real space for brevity.
(b) DC (0.0411)
(c) DSA (0.2683)
(g) Preservation property of trained  $S$
(d) DM (0.0534)
(e) IDC (0.1020)
(f) TM (0.0242)
Figure 3: (a)-(f) The log of explained variance ratio on each frequency domain dimension (surface) and the degree of overlap between  $D$  and  $S$  for the top-200 dimensions (floor). The parenthesis is  $L_{2}$  difference of explained variance ratio between  $D$  and  $S$ . (g) It can be observed that trained  $S$  mostly preserves the top- $k$  explained variance ratio dimensions of  $D$ , regardless of  $k$ .
describe the total variance of a dataset. On the other hand, the spatial domain does not exhibit this behavior; poor quality of reconstruction, and a similar variance ratio across all dimensions.
To investigate whether this tendency is maintained in the dataset distillation, we conducted a frequency analysis of various dataset distillation losses  $\mathcal{L}_{DD}(S,D)$ . Figure 2b shows the magnitude of gradient for  $\mathcal{L}_{DD}(S,D)$  i.e.  $\| \nabla_{S}\mathcal{L}_{DD}(S,D)\|$ . In contrast to the spatial domain, where the gradients w.r.t the spatial domain are uniformly distributed across each dimension; the gradients w.r.t frequency domain are condensed into certain dimensions. This result suggests that many dimensions are needed in the spatial domain to reduce the loss, but only a few specific dimensions are required in the frequency domain. Furthermore, we compared the frequency domain information of  $D$  and  $S$  which were trained with losses from previous research. As shown in Figures 3a to 3f, the distribution of explained variance ratios in the frequency domain is very similar across different distillation losses. Also, Figure 3g shows that trained  $S$  mostly preserves the top- $k$  explained variance ratio dimensions of  $D$  in the frequency domain, regardless of  $k$  and the distillation loss function. Consequently, if there exists a frequency dimension on  $D$  with a low explained variance ratio of frequency representations, it can be speculated that the absence of the dimension will have little impact on the optimization of  $S$ .
3.2 FreD: Frequency domain-based Dataset Distillation
We introduce the frequency domain-based dataset distillation method, coined FreD, which only utilizes the subset of entire frequency dimensions. Utilizing FreD on the construction of  $S$  has several advantages. First of all, since the frequency domain concentrates information in a few dimensions, FreD can be easy to specify some dimensions that preserve the most information. As a result, by reducing the necessary dimensions for each instance, the remaining budget can be utilized to increase the number of condensed images. Second, FreD can be orthogonally applied to existing spatial-based methods without constraining the available loss functions. Figure 4 illustrates the overview of FreD with the information flow. FreD consists of three main components: 1) Synthetic frequency memory, 2) Binary mask memory, and 3) Inverse frequency transform.
Synthetic Frequency Memory  $F$ . Synthetic frequency memory  $F$  consists of  $|F|$  frequency-label data pair, i.e.  $F = \{(f^{(i)},y^{(i)})\}_{i = 1}^{|F|}$ . Each  $f^{(i)}\in \mathbb{R}^d$  is initialized with a frequency representation acquired through the frequency transform of randomly sampled  $(x^{(i)},y^{(i)})\sim D$ , i.e.  $f^{(i)} = \mathcal{F}(x^{(i)})$ .
Binary Mask Memory  $M$ . To filter out uninformative dimensions in the frequency domain, we introduce a set of binary masks. Assuming the disparity of each class in the frequency domain, we utilize class-wise masks as  $M = \{M^{(1)},\dots,M^{(C)}\}$ , where each  $M^{(c)}\in \{0,1\}^d$  denotes the binary mask of class  $c$ . To filter out superfluous dimensions, each  $M^{(c)}$  is constrained to have  $k$  non-zero elements, with the remaining dimensions masked as zero. Based on  $M^{(c)}$ , we acquire the class-wise
(a) Overall structure of FreD.
(b) Inverse frequency transform with binary masking.
Figure 4: Visualization of the proposed method, Frequency domain-based Dataset Distillation, FreD.
filtered frequency representation,  $M^{(c)} \odot f^{(i)}$ , which takes  $k$  frequency coefficients as follows:
$$
M _ {u, v} ^ {(c)} \odot f ^ {(i)} = \left\{ \begin{array}{l l} f _ {u, v} ^ {(i)} & \text {i f} M _ {u, v} ^ {(c)} = 1 \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {3}
$$
PCA-based variance analysis is commonly employed for analyzing the distribution data instances. This paper measures the informativeness of each dimension in the frequency domain by utilizing the Explained Variance Ratio (EVR) of each dimension, which we denote as  $\eta_{u,v} = \frac{\sigma_{u,v}^2}{\sum_{u',v'}\sigma_{u',v'}^2}$ .
Here,  $\sigma_{u,v}^{2}$  is the variance of the  $u,v$ -th frequency dimension. We construct a mask,  $M_{u,v}^{(c)}$ , which only utilizes the top- $k$  dimensions based on the  $\eta_{u,v}$  to maximally preserve the class-wise variance of  $D$  as follows:
$$
M _ {u, v} ^ {(c)} = \left\{ \begin{array}{l l} 1 & \text {i f} \eta_ {u, v} \text {i s a m o n g t h e t o p - k v a l u e s} \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {4}
$$
This masking strategy is efficient as it can be computed solely using the dataset statistics and does not require additional training with a deep neural network. This form of modeling is different from traditional filters such as low/high-pass filters, or band-stop filters, which are common choices in image processing. These frequency-range-based filters pass only certain frequency ranges by utilizing the stylized fact that each instance contains varying amounts of information across different frequency ranges. We conjecture that this type of class-agnostic filter cannot capture the discriminative features of each class, thereby failing to provide adequate information for downstream tasks, i.e. classification. This claim is supported by our empirical studies in Section 4.3.
Inverse Frequency Transform  $\mathcal{F}^{-1}$ . We utilize the inverse frequency transform,  $\mathcal{F}^{-1}$ , to transform the inferred frequency representation to the corresponding instance on the spatial domain. The characteristics of the inverse frequency transform make it highly suitable as a choice for dataset distillation. First,  $\mathcal{F}^{-1}$  is a differentiable function that enables the back-propagation of the gradient to the corresponding frequency domain. Second,  $\mathcal{F}^{-1}$  is a static and invariant function, which does not require an additional parameter. Therefore,  $\mathcal{F}^{-1}$  does not require any budget and training that could lead to inefficiency and instability. Third, the transformation process is efficient and fast. For  $d$ -dimension input vector,  $\mathcal{F}^{-1}$  requires  $\mathcal{O}(d\log d)$  operation while the convolution layer with  $r$ -size kernel needs  $\mathcal{O}(dr)$ . Therefore, in the common situation, where  $\log d < r$ ,  $\mathcal{F}^{-1}$  is faster than the convolution layer. We provide an ablation study to compare the effectiveness of each frequency transform in Section 4.3 and Appendix D.8.
Learning Framework. Following the tradition of dataset distillation [20, 26, 7], the training and evaluation stages of FreD are as follows:
$$
F ^ {*} = \underset {F} {\arg \min } \mathcal {L} _ {D D} (\tilde {S}, D) \text {w h e r e} \tilde {S} = \mathcal {F} ^ {- 1} (M \odot F) \quad \text {(T r a i n i n g)} \tag {5}
$$
$$
\theta^ {*} = \underset {\theta} {\arg \min } \mathcal {L} \left(\tilde {S} ^ {*}; \theta\right) \text {w h e r e} \tilde {S} ^ {*} = \mathcal {F} ^ {- 1} \left(M \odot F ^ {*}\right) \quad (\text {E v a l u a t i o n}) \tag {6}
$$
3There are several researches which utilize this property to replace the convolution layer with a frequency transform [2, 12, 15].
Algorithm 1 FreD: Frequency domain-based Dataset Distillation
1: Input: Original dataset  $D$ ; Number of classes  $C$ ; Frequency transform  $\mathcal{F}$ ; Distillation loss  $\mathcal{L}_{DD}$ ; Dimension budget per instance  $k$ ; Number of frequency representations  $|F|$ ; Learning rate  $\alpha$
2: Initialize  $F = \emptyset$
3: for  $c = 1$  to  $C$  do
4:  $F^{(c)}\gets \{(\mathcal{F}(x^{(i)}),y^{(i)})\}_{i = 1}^{\frac{|F|}{C}}$  from a class-wise mini-batch  $\{(x^{(i)},y^{(i)})\}_{i = 1}^{\frac{|F|}{C}}\sim D^{(c)}$
5: Initialize  $M^{(c)}$  by Eq. (4) of the main paper
6: end for
7: repeat
8:  $F \gets F - \alpha \nabla \mathcal{L}_{DD}(\mathcal{F}^{-1}(M \odot B_F), B_D)$  from a mini-batch  $B_D \sim D$  and  $B_F \sim F$
9: until convergence
10: Output: Masked frequency representations  $M \odot F$
where  $M \odot F$  denotes the collection of instance-wise masked representation i.e.  $M \odot F = \{(M^{(y^{(i)})}(f^{(i)}), y^{(i)})\}_{i=1}^{|F|}$ . By estimating  $F^*$  from the training with Eq (5), we evaluate the effectiveness of  $F^*$  by utilizing  $\theta^*$ , which is a model parameter inferred from training with the transformed dataset,  $\tilde{S} = \mathcal{F}^{-1}(M \odot F^*)$ . Algorithm 1 provides the instruction of FreD.
Budget Allocation. When we can store  $n$  instances which is  $d$ -dimension vector for each class, the budget for dataset distillation is limited by  $n \times d$ . In FreD, we utilize  $k < d$  dimensions for each instance. Therefore, we can accommodate  $|F| = \lfloor n\left(\frac{d}{k}\right)\rfloor >n$  instances with the same budget. After the training of FreD, we acquire  $M \odot F^{*}$ , which actually shares the same dimension as the original image. However, we only count  $k$  non-zero elements on each  $f$  because storing 0 in  $d - k$  dimension is negligible by small bytes. Please refer to Appendix E.3 for more discussion on budget allocation.
3.3 Theoretic Analysis of FreD
This section provides theoretical justification for the dimension selection of FreD by EVR,  $\eta$ . For validation, we assume that  $\mathcal{F}$  is linearly bijective.
Proposition 1. Let domain  $A$  and  $B$  be connected by a linear bijective function,  $W$ . The sum of  $\eta$  over a subset of dimensions in domain  $A$  for a dataset  $X$  is equal to the sum of  $\eta$  for the dataset transformed to domain  $B$  using only the corresponding subset of dimensions.
Proposition 1 claims that if two different domains are linearly bijective, the sum of EVR that utilizes only specific dimensions remains the same even when transformed into a different domain. In other words, if the sum of EVR of specific dimensions in the frequency domain is high, this value can be maintained when transforming a dataset only with those dimensions into the other domain.
Corollary 1. Assume that two distinct domains,  $B$  and  $C$ , are linearly bijective with domain  $A$  by  $W_{B}$  and  $W_{C}$ . Let  $X$  be a dataset in domain  $A$ , and  $X_{B}$  and  $X_{C}$  be the datasets transformed to domains  $B$  and  $C$ , respectively. Let  $V_{B,k}^{*}$  and  $V_{C,k}^{*}$  be the set of  $k$  dimension indexes that maximize  $\eta$  in each domain. Let  $\eta_{B,k}^{*}$  and  $\eta_{C,k}^{*}$  be the corresponding sum of  $\eta$  for each domain. If  $\eta_{B,k}^{*} \geq \eta_{C,k}^{*}$ , then the sum of  $\eta$  for  $W_{V_{B,k}^{*}}X_{B}$  is greater than that of  $W_{V_{C,k}^{*}}X_{C}$ .
Corollary 1 claims that for multiple domains that have a bijective relationship with a specific domain, if a certain domain can obtain the high value of sum of EVR with the same number of dimensions, then when inverse transformed, it can improve the explanation in original domain. Based on the information concentration of the frequency domain, FreD can be regarded as a method that can better represent the distribution of the original dataset,  $D$ , based on  $S$ .
(a) Comparison of domain
(b) Efficiency of EVR selection
Figure 5: Empirical evidence for EVR-based selection.
PCA, which sets the principal components of the given dataset as new axes, can ideally preserve  $\eta$ .