SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[57] Zhenyi Wang, Li Shen, Le Fang, Qiuling Suo, Tiehang Duan, and Mingchen Gao. Improving task-free continual learning by distributionally robust memory evolution. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 22985–22998. PMLR, 2022.
[58] Zhenyi Wang, Enneng Yang, Li Shen, and Heng Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. arXiv preprint arXiv:2307.09218, 2023.
[59] Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 1121-1128, 2009.
[60] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014-3023, 2021.
[61] Enneng Yang, Li Shen, Zhenyi Wang, Shiwei Liu, Guibing Guo, and Xingwei Wang. Data augmented flatness-aware gradient projection for continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5630-5639, 2023.
[62] Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. The Eleventh International Conference on Learning Representations, 2023.
[63] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pages 7105–7114. PMLR, 2019.
[64] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages 12674-12685. PMLR, 2021.
[65] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6514-6523, 2023.
[66] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In Ninth International Conference on Learning Representations 2021, 2021.
[67] Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7856-7865, 2023.
[68] Zhu Fei, Zhang Xu-Yao, and Liu Cheng-Li. Class incremental learning: A review and performance evaluation. Acta Automatica Sinica, 49:1-26, 2023.
Experiment Details. We train a model using condensed synthetic images and measure the Top-1 accuracy on raw testing images. For a fair comparison, our main experimental settings follow the settings of DC, DSA and DM. Experiments are conducted on a 3-layer convolutional network [46] (ConvNet-3) with 128 filters, without any specific emphasis. The batch size of the raw images is set to 256 during the matching process. For the DC [66], DSA [64], LoDC, and LoDM methods, the outer loop is set to 1 and the inner loop is set to 1 when using 1 image per class in all experimental datasets. When using 10 images per class, the outer loop is set to 10 and the inner loop is set to 50. In the MNIST and CIFAR10 datasets, when using 50 images per class, the outer loop is set to 50 and the inner loop is set to 10. For the DM [65] and LoDM methods, the loop is set to 20,000 in all experiments. The optimizers all use SGD. When optimizing the condensed data, the learning rate for DC/DSA/LoDC/LoDSA is set to 0.1 by default, and the learning rate for DM/LoDM is set to 1.0 by default. When using condensed data to train the network, the update times of the network are set to 1,000, and the learning rate is set to 0.01.
Relationship between Memory and Rank. Suppose the shape of each channel of a real image is  $(H, W)$ , where  $H$  and  $W$  are the height and width of the image, respectively. The memory consumed by traditional dataset distillation can be recorded as  $N \times D \times H \times W$ , where  $N$  and  $D$  represent the number of images and the number of channels. The memory consumed by our low-rank dataset distillation method is  $N \times D \times (H \times r + r \times W)$ , where  $r$  is the rank size. For example, in the CIFAR10 dataset, both  $H$  and  $W$  are 32, and our  $r$  is generally set to 2, which means that it can save 8 times the storage compared to traditional dataset distillation methods (i.e., DC, DSA and DM).
B.1 Experimental Analysis
Compare with Other Advanced Methods. In our main paper, the low-rank plugin is mainly used in the traditional DC [66]/DSA [64] based on gradient matching and DM [65] based on distribution matching. Recently, a research work (i.e., MTT [8]) shows that better dataset condensation performance can be obtained based on finer-grained gradient trajectory matching. This is strictly orthogonal to our work. As shown in the Tab. 4, when our low-rank plugin is further applied to the MTT, our work achieved  $58.7\%$  accuracy on CIFAR10  $(\mathrm{Img} / \mathrm{Cls} = 1)$  and  $31.0\%$  on CIFAR100  $(\mathrm{Img} / \mathrm{Cls} = 1)$ . This is a significant improvement compared to MTT, which is  $46.3\%$  and  $24.3\%$ .
In addition, related to our work are recent works on efficient dataset distillation [23, 67, 33, 13]. Among them, (i) IDC-I/IDC [23] and IDM [67] consider the local similarity of image pixels, and they partition and expand the synthetic image to obtain more images. This is orthogonal to our work, i.e., we can perform partitioning and expansion operations on images reconstructed by our low-rank plugin to further increase the number of images under the same memory constraints. For example, on the CIFAR10 dataset, compared with IDC-I/IDC, our low-rank versions (i.e., LoIDC-I/LoIDC) bring performance improvements of  $12.5\%$  and  $6.6\%$ , respectively. (ii) HaBa [33] inputs the bases into the Hallucinator networks to reconstruct condensed images, while RememberThePast [13] shares bases among all classes and constructs condensed images by addressing matrices. We note that their basis is also equal to the original image size. Therefore, under the same memory cost, we can use our low-rank plugin on their bases to obtain more images. For example, on the CIFAR10 dataset, our LoHaBa and LoRememberThePast improved by  $17.8\%$  and  $4.0\%$ , respectively, compared to Haba and RememberThePast in Tab. 4.
Table 4: Compare with other advanced dataset condensation methods.
CIFAR10 (Img/Cls=1)	MTT 46.3%	IDC-I 36.7%	IDC 50.6%	HaBa 48.3%	RememberThePast 66.4%
LoMTT 58.7%	LoIDC-I 49.2%	LoIDC 57.2%	LoHaBa 66.1%	LoRememberThePast 68.4%
CIFAR100 (Img/Cls=1)	MTT 24.3%	IDC-I 16.6%	IDC 24.9%	HaBa 33.4%	RememberThePast -
LoMTT 31.0%	LoIDC-I 26.9%	LoIDC 33.1%	LoHaBa 36.1%	LoRememberThePast -
Principal Component Analysis of Images. In this section, we verify that both original images and images condensed by traditional dataset distillation methods (e.g., DSA [64] and DM [65]) have low-rank properties. Therefore, it is inefficient to perform dataset condensation directly in high-dimensional pixel space. Specifically, we randomly sample some images from the CIFAR10 dataset and further perform singular value decomposition (SVD) on each channel of these images. As shown in Fig. 5, the diagonal matrix represents the distribution of the average eigenvalues on all channels of all selected images. We can observe that both the original images (Fig. 5(a)) and the images synthesized by the traditional data distillation methods (e.g., DSA [64] in Fig. 5(b) and DM [65] in Fig. 5(c)) can be approximated by low rank, that is, the first few eigenvalues on the diagonal dominate. Therefore, it is reasonable to perform low-rank dataset distillation in this paper. Fig. 5(d) shows the distribution of eigenvalues of images produced by the proposed LoDM.
(a) Real
(b) DSA
(c) DM
(d) LoDM
Figure 5: Principal component analysis of images on CIFAR10 dataset: (a) Real images; (b) Synthetic images of DSA; (c) Synthetic images of DM; and (d) Synthetic images of LoDM  $(r = 2)$ .
B.2 Image Visualization
Training Process of Synthetic Data. Fig. 6 and Fig. 7 illustrate the intermediate state images of different ranks  $(r = \{2,8\})$  of our LoDM during the iterative process of synthesizing images on the CIFAR10 dataset. In addition, Fig. 10 and Fig. 11, we visualize the intermediate state images of different ranks  $(r = \{2,4\})$  of our LoDM in the iterative process of synthesizing images on the MNIST dataset. Visualization of Real and Synthetic Images. We provide more visualizations of the synthetic images from the different datasets: MNIST in Fig. 8, and CIFAR10 in Fig. 9. As the rank size increases, the synthesized image captures increasingly detailed features.
(a) 0-th It
Figure 6: Visualization of the synthetic image process of  $\mathrm{LoDM}(\mathrm{r} = 2)$  on the CIFAR10 dataset.
(b) 1000-th It
(c) 5000-th It
(d) 20000-th It
(a) 0-th It
(c) 5000-th It
Figure 7: Visualization of the synthetic image process of  $\mathrm{LoDM}(\mathrm{r} = 8)$  on the CIFAR10 dataset.
(a) DC
(d) Our LoDC(r=2)
(e) Our LoDSA(r=2)
(f) Our LoDM(r=2)
(g) Our LoDM(r=4)
Figure 8: Visualization on the MNIST dataset (10 Img/Cls).
(h) LoDM(r=8)
(i) Real
(c) Our LoDM(r=1)
(d) Our LoDM(r=2)
(e) Our LoDM(r=4)
(f) Our LoDM(r=8)
Figure 9: Visualization on the CIFAR10 dataset (10 Img/Cls).
(c) 5000-th It
Figure 10: Visualization of the synthetic image process of LoDM(r=2) on the MNIST dataset.
(a) 0-th It
Figure 11: Visualization of the synthetic image process of LoDM(r=4) on the MNIST dataset.