Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks
Authors: Zhiwei Deng
Olga Russakovsky
Venue: NeurIPS 2022 Accept (2022)
Keywords: Dataset Distillation
Bi-level optimization
Memories
Continual learning

Abstract: We propose an algorithm that compresses the critical information of a large dataset into compact addressable memories. These memories can then be recalled to quickly re-train a neural network and recover the performance (instead of storing and re-training on the full original dataset). Building upon the dataset distillation framework, we make a key observation that a shared common representation allows for more efficient and effective distillation. Concretely, we learn a set of bases (aka ``memories'') which are shared between classes and combined through learned flexible addressing functions to generate a diverse set of training examples. This leads to several benefits: 1) the size of compressed data does not necessarily grow linearly with the number of classes; 2) an overall higher compression rate with more effective distillation is achieved; and 3) more generalized queries are allowed beyond recalling the original classes. We demonstrate state-of-the-art results on the dataset distillation task across five benchmarks, including up to 16.5% and 9.7% accuracy improvement when distilling CIFAR10 and CIFAR100 respectively. We then leverage our framework to perform continual learning, achieving state-of-the-art results on four benchmarks, with 23.2% accuracy improvement on MANY.

Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks
Zhiwei Deng
Olga Russakovsky
Department of Computer Science
Princeton University
(zhiweid, olgarus) @cs. (ecs.princeton.ed
Abstract
We propose an algorithm that compresses the critical information of large dataset into compact addressable memories. These memories can then be recalled quickly re-train neural network and recover the performance (instead of storing and re-training on the full original dataset). Building upon the dataset distillation framework, we make a key observation that shared common representation allows for more efficient and effective distillation. Concretely, we learn a set of bases (aka "memories") which are shared between classes and combined through learned flexible addressing functions to generate a diverse set of training examples. This leads to several benefits: 1) the size of compressed data does not necessarily grow linearly with the number of classes; 2) an overall higher compression rate with effective distillation is achieved; and more generalized queries are allowed beyond recalling the original classes. We demonstrate state-of-the-an results on the dataset distillation task across six benchmarks, including up 16.5% and 9.7% in retained accuracy improvement when distilling CIFAR 10 and CIFAR 100 respec- tively. We then leverage our framework perform continual learning, achieving state-of-the-art results on four benchmarks, with 23.2% accuracy improvement on MANY. The code is released on our project webpage'.
Introduction
Compressing large amount of information into small memory storage space is one of the key components of human intelligence [1-3] - person can retrieve memories from the past and quickly recover the corresponding skills. Deep learning methods have made large strides in building task- specific models, but are shown to easily forget past knowledge when learning new tasks [4, 5].
To equip neural network learners with memorizing ability, dataset distillation [6] is proposed as potential solution. Concretely, compressed set of examples (memories) learned to summarize the key information in a dataset that affects model training: these examples can then be used to quickly retrain models and recover the corresponding skills. This differs from the standard reconstruction- based compression algorithms [7-9] and shows strong performance [10-13].
critical question in building powerful compressed memories is: what structures and representations should we use to build the memories? An effective structure and organization of memories can lead different fundamental assumptions about data and affect the compression and learning behaviours. Existing works [10. 11, 14, 12, 15, 13, 6] follow simple representation. where set of learnable examples assigned for each class. However, under this assumption, the size of the memories can linearly grow with the number of classes, making the distillation of datasets with large number of classes challenging. Naturally, this can potentially lead redundancies the learned memories,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

due to the separation of data among classes. Furthermore, this representation is less generalizable to continuous label space, where infinite number of label values exists.
In our paper, we make the observation that there is information shared between classes, and hypoth- esize that common and compact representation exists for all classes. Following this hypothesis, we propose to formulate the problem as memory addressing process, where the memories store a common set of bases shared by all classes, and the recombination of bases is performed through an addressing function. This decomposition between memories and addressing functions enables the possibility that all common information is stored in one part of the representation, and the accessing of the common information depends on the specific labels and is handled through an extra function. We find that this formulation can significantly improve both the compression rate and the performance.
We adopt the hack-propagatton through time learning framework to train the memories and addressing functions, and identify several critical factors that can improve the performance. Specifically, we find that adopting the momentum term, and performing long unrolls in the inner optimization loop are both critical. This differs from the common usage of bi-level optimization algorithm on this task [6. 16], and leads to strong performance outperforming single-step gradient matching methods [10, 11] even with the simple data representation.
In the experiments, we extensively evaluate our algorithm on six benchmarks of the Dataset Distilla- tion task, and show that it consistently outperforms previous state-of of anf-the-ar art by significant margin. For example, we achieve 66.4% accuracy on CIFARIO 10 with the storage space of image per class, improving over the previous state-of-the-art KIP method [12, 13] by 16.5%. We further demonstrate our method on the continual learning tasks, and show that a simple "compress-then-reaal using our framework leads to state-of-the-art results on four datasets. For example, we outperform all prior methods by 23.2% in retained accuracy on the challenging MANY [17] benchmark. Finally, we demonstrate the generality of our approach by extending to image-based (rather than tabel-based memory recall, and synthesizing new classifiers (unseen during training) from our distilled memories.
method
2 Related works
The task of dataset distillation fundamentally compression problem, with a different prioritization on the information contained in data. There have been several lines of methods, developed with different criteria prioritize information. Generalization loss with bi-level optimization framework [18-20] has been widely studied and is used in the early works of dataset distillation [6. 16]. It emphasizes on the loss at the final optimization state. Gracient-maachhnng or ccore-matching methods [10, 11, 15] are adopted to directly match the induced gradients from synthetic data. If ideally matched over the gradient field, the compressed dataset can naturally lead to the same model parameters with gradient descent. Kernel method [13, 12] shows that with the connection to Gaussian processes, a kernel inducing points method can be used to achieve strong performance, but with large computation costs. These are also connected with the recent progress on pragmatic compression methods [21, 22], which compress or match distributions based on a decision process (in dataset distillation's 's case, the gradient descent search process).
Dataset Distillation.
Continual learning. Broadening the learning paradigms, continual learning problem aims to build agents that learn through a stream of tasks and accrue knowledge along the process. "Catastrophic forgetting" [5, 23, 4] is well-known phenomenon in this setting, where the neural network forgets previous skills when learning new ones. Various methods [24-27, 17, 28-35] on regularization, replay, or dynamic model, have been proposed to alleviate the issue and address the "stability- plasticity dilemma" [36, 37]. Memory buffer has been critical component in the past methods [24- 27, 17, 28, 38, 39], but mainly relies on a random selection of real samples with different strategies. Recently, several works extend the usage of memory to storing random basis [39] (online setting) or SVD bases [38] (offline setting).
3 Background: dataset distillation
The task of Dataset Distillation [6] is proposed to compress the key information of large-scale training dataset into a small amount of learned data, which can be stored using limited memory space and retrieved through label indices or task information to recover the performance of a model.

---

Problem Setting, Formally, given large dataset Drr = { y; H containing N pairs of training data (z,, vi). where z; an image and y, is the corresponding label in C' classes, a small dataset D. = [(a], y ") No & N, can be synthesized or distilled, such that a model trained on D, can have the same generalization ability ones trained on Dzy:
E (( (m;; 0()),, v)] = B(a.))>Di. [m( (*()), y)
(I)
where g(*) and 0²(*) are the optimized parameters using Dtr and D, respectively, is metric, e.g., accuracy, and Die is the test dataset. The model is often a neural network classifier f(;;))
parameterized by and trained with loss function ((((;; 0), y).
Synthetic dataset representations. The synthetic dataset contains the core information that needs to be learned. The representation of the synthetic data affects the compactness and effectiveness of the distillation process. In existing methods [6, 10. 11, 14, 12. 15], the dataset D, is defined as collection of learnable data samples (ar', y). and the number of samples is separately and equally distributed across classes. This representation has several disadvantages: first, the number of synthetic data samples needed for dataset grows linearly with the number of classes, leading to limited applicability when the number of classes is large or undefined (e.g., language or other continuous labels); second, the potentially shared and common information across classes is ignored- this results in a less compact representation of the distilled information and lower compression rate; lastly, the representation not able generalize to new classes or tasks, due to the lack of common representation learned across classes.
4 Model
Overview. In this section, first, we present new perspective of the problem, where the Dataset Distillation problem is formulated as a memory addressing process: instead of learning synthetic images separately for each class, construct and learn a common memory representation that can be accessed through addressing matrices to construct synthetic datasets. Under this formulation, the number of synthetic images does not need to grow linearly with the number of classes, the shared information among classes can be exploited to reduce redundancies and improve compression rate, and datasets can be distilled with respect more generic queries. Second, we further show several critical empirical facets of back-propagation through time framework, which lead to drastic improvements on the performance and outperform the single-step gradient matching methods. This is in contrast with the current common observation that gradient matching outperforms back- pook-propagttton through time framework on dataset distillation tasks.
In the following, we present the two core components of our method, (1) the new formulation of dataset distillation, with memories and addressing matrices in Sec. 4.1, and (2) the learning framework under
through time in Sec. 4.2.
4.1 Dataset Distillation as memory addressing
E X' E V, we aim to learn a single compact and compressed representation M,, referred to as memories, that can be accessed through learned addressing function A(-). A(-) takes all possible values of y as input and recalls the corresponding synthetic data. With set of {y,}- J. a synthetic dataset recalled using the above process can train model fo X - y from scratch and obtain the same generalization ability as trained on D.r-
Problem formulation. Given a task-specific dataset Der = (0,,
Ideally, the memories M and the addressing function A can jointly capture the critical information that defines the task mapping from X' to y, such that the recalled synthetic data given query y; contains distinctive information that defines y; and the synthetic dataset recalled with {y.} can satisfy eqn. I when used for re-training. For example, in the standard classification tasks, we can enumerate all possible values in the label space (discrete) and address the memories, to construct the synthetic dataset that contains critical information. Under this formulation, since we are learning a single, shared and accessible representation for all yss the size of memories can be defined flexibly regardless of the number of classes, removing the linear growth limitation the standard distillation settings. There is also no constraint on the form of ys, which can be either discrete or continuous. The storage budget or compression rate is calculated by considering the storage space of parameters in both memories and addressing functions, which should be compact as possible.
3

---

0
0
Shared common bases
0
1
0
0
y
0
0
0
0
0
(memories)
0
Class
Label
Hard-coded addressing matrix
Query
Learnable addressing matrices
(label
Compressed images
(memories)
Both considered in the storage budget
Figure Distilling large-scale dataset into compressed memories. Left: the standard dataset distillation task under the formulation of memory addressing. The addressing matrices are hard-coded with Is and to fetch the corresponding compressed image memories. The memory size grows linearly with number of classes. Right: learnable addressing matrix with shared common bases (number of bases can flexibly deñned). The information sharing between classes is captured in this representation. The queries can be generalized to any vector representation, besides one-hot labels, i.c. for a general dataset from X' to V, can be distilled into memories for recall and model re-training.
Memory representation. We use set of bases to store in the memories M [b]. bx}; where each vector by Rd has the same dimension intrinsic components in a dataset that characterizes the task mapping from X' to y. Through re-using the bases, we can produce desired synthetic dataset for model re-training. Spatial redundancies in images: note that, as a special case, images can contain redundancies spatially and be stored in downsampled version to improve the compression rate. The downsampled image bases can be passed via a deterministic upsampling process (e.g., bilinear interpolation) to recover the original resolution.
Rdd and all vectors collectively define the
Memory addressing. For each query y, bases in the memories M. Similar to previous methods on accessing memories, we use A to produce of coefficients, and linearly combine the bases to produce synthetic data. Formally, to retrieve synthetic examples for each we define a set of matrices { [AI,... A, A, E R&,, where dy is the dimension size of y and r is the number of data samples that can be retrieved. With the memories M = (b),.., ........ bx) ), we define:
parameterized function A(y) re-combine the
x Y² A, bx]: bx]", Rdx1
where y E Rdyx1 is in vectorized form, such one-hot encoding of categorical labels, and vr y² A; corresponds to coefficient vector v that combines the bases. The produced synthetic data ar' is paired with y as the corresponding label. Our model is shown on the right of figure
Constructing a dataset. To construct synthetic dataset D. for model re-training, we are often given set of samples {y.). or can enumerate all possible values of y (if discrete). With the set {y:). we use eqn.2 to address and retrieve the synthetic dataset D v: = (2'r. vD) for each y;- The final dataset is the union of all DD U The dataset D. can be used in either minibatch form for stochastic gradient descent, or as a whole for batch gradient descent.
Generalized possibilities of queries. Another advantage of our formulation, besides a compact and shared representation, is the various possibilities of queries y. In principle, under this formulation, the label y can be flexibly defined as other forms, such as language or audio. where the representation resides in a continuous space or follows distribution p(h(y) defined by a feature extractor l((-). The set of } then can be sampled from the distribution p instead of having to enumerate all possible values. This provides a general way of compressing or distilling a large dataset without constraint on the forms of labels.
Connection with standard Dataset Distillation. the standard setting, dataset distillation is defined for classification tasks with discrete labels and each label owns its unique set of synthetic data. We can show that this is a special case of our formulation: if the bases are constructed as the collection of those label-specific synthetic data (K N'. N' is the total size of the synthetic dataset), the addressing matrices A, defined in space {0, 1] CxNN where A, A/mm. n] if equals

---

Algorithm
1: hyperparameters: Momentum rate So, 3,, learning rate oo, on for and Φ respectively.
2: imput: Dataset Dtr, memories M, addressing function A, loss function ((.,))
3: repeat
4: Sample subset of labels y"
5: Address the memories M and obtain synthetic dataset DD with eqn.
6: Randomly initialize model parameters 0u
7: Initialize momentum my
8:
for 1 to 7 do
Sample minibatch B {(rr y.)} from D
9:
Compute L = [. Ei {(fo... (2))),
10:
11:
Update momentum m, - + de
20:-1
12:
Update 0, = 0. Of
13:
end for
14:
Sample minibatch B {x y,)} from Dr, with labels in ) Σ 1 {[Ser (2:)), vi)
Compute J(Φ) = B
16:
Update Φ < OPT-STEP Φ J(),, a1, 31)
until Converge
m(N//C) i (ith item of mhth class) and A,m,,n) n] = 0 at other positions, then the "retrieval" process can also be defined as eqn. 2.
4.2 Learning framework: back-propagation through time
In this section, we build upon the back- propagattoonnn through time algorithm and discuss detail the learning framework that performs the distillation process from dataset to memories and addressing functions.
Starting from notations, we define the parameters contained in both memories and addressing functions as Φ which are collectively optimized. A loss function is E(..)) defined on a task-specific dataset. We denote an optimization algorithm OPT(,, :; oz, 33 e, T), where and 3 are the learning rate and the momentum rate, respectively, and T is the number of optimization steps. For a single step optimization, we denote it as OPT-STEP(:; 3)
To learn the parameters &- (M, follow standard bi-level optimization framework with back- propagation through time (BPTT), where the inner-loop uses the synthetic dataset D, to train randomly initialized model starting from seratch, and generalization loss is computed using a minibatch B = { y.)} sampled from D,,. The parameters & are implicitly contained in the synthetic dataset D. and optimized when minimizing the generalization loss. The bi-level optimization defines:
min ((6)
B Σ (a,; 0²), y:);
1
subject to 0 OPT (0., D.; op, So E, T,T)
(3)
where On represents the initializing parameters, 0" the optimized model parameters in the inner optimization loop, ag and Bu are the learning rate and momentum rate for opt(..-,, and J/ ((Ø) the generalization loss on minibatch B. In practice, for each inner loop training, we randomly sample subset )" from and retrieve the corresponding subset D. This reduces the computation cost inner loops. We empirically observe that equivalent results can achieved with faster runtime. The algorithm is summarized in Alg. 1, where lines 8-13 13 define the inner loop optimization process OPT(,, ; a, 3,, T), and the gradients of generalization loss (line 15) is paok-propagated through the inner loop to update Φ Note that, in principle, the inner loop optimization can be performed using any optimizer. In this paper, we mainly rely on the standard stochastic gradient descent with momentum to train the distilled data.
5

---

Critical factors in BPTT. Although being a natural choice in performing dataset distillation and adopted in the original work [6], the BPTT framework has been shown to underperform other algorithms, such as single-step gradient matching methods [10, 11], on various benchmarks. The underlying causes that hinder the performance of the algorithm are still underexplored. In our work, we investigate and identify the factors that can unleash the potential of back-propagation through time framework on dataset distillation and lead to significant performance boosts.
Momentum term. In previous dataset distillation works [6, 16], the usage of prok-propagation through time framework omits the momentum term in the inner loop optimization. Indeed, this has been a common practice in meta-learning tasks [18, 40]. Adding momentum terms in meta-learning can potentially even hurt the performance and lead less gradient diversity [40]. However, we observe that, in dataset distillation tasks, the momentum term is crucial for making BPTT excel, even in the relatively short inner loop optimization settings (e.g. 10 or 20 steps). We provide results and analysis in the experiments section.
Long unrolled trajectories. Another aspect in using BPTT in dataset distillation is the length of unrolled optimization trajectories in the inner loops. The previous usage of BPTT on this task [6, 16] adopts relatively short inner loop optimization trajectories (e.g. 10-30 steps). Instead, we show that unrolling the trajectories long enough (e.g. 200 steps) with momentum terms can potentially produce 0" that better summarizes the information contained in memories and addressing matrices, generating more effective gradients to learn the compressed representation.
5 Experiments
We thoroughly evaluate our model and demonstrate the benefits over previous methods. In section 5.1, we show that using a shared representation is critical to improving the distillation performance and compression rates. Specifically, we observe that there is strong evidence that there is information re-using across classes. We further show the benefits of our model on standard continual learning tasks in section 5.2. For example, we observe that a simple ccompres--tennrrcall method can achieve performance outperforming state-of-the-art continual learning models with complex designs. Finally, in section 5.3, we show that storing the compressed data enables synthesizing new classifiers (section 5.3.1) and the shared representation formulation allows more general queries (section 5.3.2), which can be continuous (e.g. image features).
5.1 Dataset Distillation
In this section, we follow the standard setting of dataset distillation, and perform dataset compression that can be recalled with discrete class labels.
Datasets. We test our models on six standard dataset distillation benchmarks: MNIST [41], Fash- ionMNIST [42], SVHN [43], CIFARIO [44], CIFARIOU [44], and Tiny [45]. MNIST contains 10 classes with 60,000 writing digit images as training and 10,000 as test set. The images gray-scale with shape of 28 x 28. FaShionMNIST is dataset with clothing and shoe images and consists of a training with size 60,000 and a test set with size 10,000. Each image is 28 x 28 in gray scale, and has a label from 10 classes. SVHN contains street digit images where each image has shape 32 x 32 x 3. The dataset contains 73,257 images for training and 26,032 images for testing. CIFAR10 and CIFAR 100 color image datasets, with 50,000 training images and 10,000 testing images on each. CIFARIO has 10 classes with 5,000 images per class, and CIFARI 100 has 100 classes with 500 images per class. TinylmageNet [45] contains 200 categories with images of resolution 64x64. The training and testing sets have 100,000 and 10,000 images respectively.
Experiment settings. We evaluate our distillation models under three different memory budgets for each dataset: 1/10/50 images per class. We focus on high compression rate scenarios and consider the 1 and 10 settings for CIFARI00. Following previous works [14, 11, 12, 15]. the main network architecture used in experiments is a simple convolutional network (ConvNet) with 3 x 3 filters, InstanceNorm, ReLU and x 2 average pooling. For the evaluation protocol, each model is evaluated on 20 randomly initialized models, trained for 300 epochs on a synthetic dataset, and tested on a held-out testing dataset. We use one GPU per experiment run.
Memory budget calculation. Since our model uses memories and addressing matrices to store the compressed information, we treat the total number of images as a memory storage budget. When

---

I/C
DC [10]
DSA
KIP (NN)
CAFE [46]
TM
DM [M41
Ours
91.7
88.720.6
90.120.1
93.1=0.3
89. 59.7.00.6
98.710.7
MNIST [41]
10
97.410.2
97.8:0.1
97,510.0
97.5=0.1
97.51..11
9.5
50
98.8=002
99.2=0.1
98,3+0.1
98.9=0.2
98.6+0.1
99.4+-0.4
70.5++0.6
70.6+0.6
73.5+00.5
77.1=0.9
88.5+00.1
F-MNIST [42]
82.3+0.4
84.6+0.3
8688+0.1
83.0-0.3
90.0+-0.7
50
83.6=0.4
88.7=0.2
88.0=0.11
88.2-0.3
91.2:10..3
31.2+1.4
27.5=114
57.3+0.1
42.9=3.0
07.3:10.11
SVHN (43)
76.1110.6
79.2.1..5
75.010.1
77.9-0.6
89.1.10.2
50
82.3+003
84,4+004
80.5+0.1
82.3-044
89.5+0.2
28.3++0.5
28.8+0.7
49,9+0.2
31.6-0.8
46.3+0.8
26.0+0.88
66.4+-0.4
CIFARIO [44]
10
44.9=0.5
52.1=0.5
62.7+0.3
50,9=0.5
65.3+
48.9
71.2::0.4
50
03.9+0.5
60.6=0.5
68.6+0.2
62.3=0.4
71.6+0.2
63.0+0.4
73.6+0..5
12.8+0.3
13,9-0.3
15.7+0.2
14.0-0.3
24.3++0.3
11,4+0.3
34.0+0.4
CIFARIO0 (44)
10
25.2110.3
32.3-0.3
28.310.1
31.5=0.2
40.14:0.4
29.740.3
42.910.7
TinylmageNer [45]
8.810.3
3.220.2
16.0:1..7
Table I: We compare our method with previous works ConvNet recovered accuracy. Our algorithm consistently outperforms all previous methods and achieves state-of-the-an "Note that we selected the best results from baseline model variants. I/C is images per class (storage budget eqn. 4).
comparing with N images for C' classes, we ensure
size(bases) + size(addressing matrices) As NCsize(image)
where size(.) is the total size of a tensor, assuming the numbers are stored as floats. For given number of bases, we calculate the corresponding maximum number of addressing matrices allowed using eqn. 4 and use the integer lowerbound as the final value.
Model details. We use bases with spatial resolution downsampled by factor of 2 in both height and width from the standard image size based on datasets. All models are trained for 50k iterations with SGD optimizer. For the inner loop optimization, we set the momentum rate as 0.9, and use 150 steps for small memory budgets I and 10, and 200 steps for budget 50. The number of bases is selected
Accuracy
CIFARJ0
SVHN
1601151 24151) sacui
# #bases (#aadd--mat) (1 VC)
on the held-out validation set (10% of training set), an example is shown in figure 2, where different numbers of bases and addressing matrices (calculated with eqn. 4) have impacts accuracies; more details in the appendix.
Figure Validation set
Result state-of-the-art accuracy. We compare our model with previous methods: Dataset Con- densation (DC) [10], Differentiable Siamese Augmentation (DSA) [11], Kernel Inducing Points (KIP) [12], Distribution Matching (DM) [14), Aligning Features (CAFE) [46] and Trajectory Match- ing (TM) [15]. The results are summarized in table 1. Following previous methods [11, 12, 15], we adopt simple data augmentations and preprocessing: flip and rotation on CIFAR 10 and CIFAR100 datasets, and ZCA on SVHN, CIFAR 10 and CIFAR 100. As shown in table 1, our model consistently outperforms previous methods, especially under high compression rate cases where only image allowed per class. For example, achieve 87.3% 3% and 66.4% SVHN and CIFAR1O 10 with I image per class, outperforming prior arts by 30% and 16.5% under the same storage budget, respectively, and even beat the performance of previous methods using 10 images per class.
Analysis: information sharing across classes. The core observation in our method is that a common representation can enable information sharing across classes and reduce redundancies. To verify this, we calculate the average coefficients δ + 11 y" A, for each class y and visualize the cosine similarities of e from two classes. The visualizations are shown in fig. 3. Higher cosine similarity scores indicate that two classes are utilizing similar bases components in the memories to produce synthetic images. For example, in CIFAR 100 (right one of figure 3), classes maple, oak, palm, pine and willow trees have strong sharing, while lawn mower and rocket are distinct from each other. Similar patterns can be found in CIFARIO dataset, shown in the left one of figure 3.
Result 2: back propagation through time is a strong baseline. In figure 4 and table 2, we show that a vanilla BPTT variant is already a strong baseline which outperforms previous single-step gradient methods [10] by 40.4% on SVHN and 20.3% on SVHN and CIFAR 10 under image per class. Note that the performance on SVHN has doubled the accuracy 31 11.2% obtained using single-step gradient matching methods [10]. In the vanilla BPTT variant, downsampling (ds) or
7

---

pickup truck
rack mower
airgiane
bird
cst
dog
frog
horv
ship
tuuk
mapia
palm tree
pine tree
willw tree
biyycl
matorayce
train
dreetcor
trastor
airplane
mapic
pal tree
SUNI BFTT
brd
willow tree
SVIt BITT.ii
bicycle
deer
22
motorrycle
pickup Train
frag
lawn mowar
herse
streetcar
truck
opt
Figure 3: Similarity matrices of learned addressing coefficients for the CIFAR1O dataset (left) and subset of CIFARION 100 classes (right).
Figure 4: Analysis on BPTT steps and momentums.
I/C
Single-step GM
Ours
Ours RPTT
Ours
Full
Ours
28.8+
49.1
55.2
64.2
66.4:100.4
CIFAR10
10
52.1
62.4-10.4
65.9-10.4
70.9:00.4
71.2++0.44
50
60.6:10.
70.5+0.4
71.1
72.1+0.5
73.8:10.4
1
13.9+0.3
21.3++0.6
25.9++0.4
33.5
34.0-F0.4
CIFARI00
10
32.3++0.3
34.7++0.5
36.5+004
40.6:H0.3
42.9+0077
Table 2: Ablation studies of every component and comparison with single-step gradient matching [10]. ds: downsampling. Aug.: data augmentation.
memory addressing formulation used. We also analyze the effects of long unrolls and momentum terms on vanilla BPTT in figure 4. It is observed that on both short inner loops (10 steps) and long ones (100 steps). adding momentum terms can consistently lead a strong performance boost, e.g. 7.0% and 9.2% on CIFAR10. Using longer inner loop trajectories can also increase the recovered accuracy, e.g. 18.2% and 42.3% on CIFARIO and SVHN, respectively, compared
step cases.
Ablation studies. To further analyze the effects of different components in our algorithm, we perform ablation studies on CIFARIO and CIFAR100. Besides the vanilla BPTT, the ablation results of components (downsampling, augmentation and memory addressing formulation) are summarized in table 2. We show that downsampling can indeed reduce spatial redundancies (e.g. improve results from 49.1% to 55.2% on CIFARIO 10 with 1 image further increase the recovered accuracy (from 55.2% to 64.2% on CIFARIO with image per class). It is also shown that our model is quite robust to the ablation of data augmentation, which has a small effect (1-2%) on the results. The resulting algorithm is simple and effective framework that uses memory addressing formulation and BPTT with long unrolls to distill datasets.
class), and memory addressing formulation can
Cross-architecture generalization. Our memories and addressing matrices are also generalizable across various architectures. We test our algorithm on ConvNet, ResNet12 and AlexNet for training and testing. The results summarized in the appendix, section ??, table ??.
5.2 Continual learning
One of the key usages of memories is prevent forgetting when model continually learns through tasks. In this section, we evaluate our algorithm on the standard continual learning benchmarks and show that, due the strong performance, simple "compress-then-ree can already rival with previous state-of-theearts with complex designs.
method with model
Efficient lifelong learning. Following [47], work with the problem where all tasks are streamed in mini-batches and learned in a single pass. learner is allowed to be equipped with small memory buffer. The data samples after seen will not be available unless stored in the buffer. We use mini-batch size of 10 to stream the data, following previous works [28, 24].
The learner's performance after learning on the task stream is commonly evaluated under two metrics: retained accuracy (RA) and backward-transfir and interference (BTI). RA the average accuracy of the final trained model on all tasks, and BTI measures the performance difference between after it was learned and after the full training process. Note that our algorithm does not perform actual learner training the data streams and BTI is not applicable.
Evaluation.

---

Rotations
Permutations
MANY
CIFAR-100
RA1
BTI
RA1
BTL
RA
BTL
RAT
BTL
ONLINE
53.38
-5.44
55.42 55.42=005
-13.76
32.62
-19.06
32.62
-19.06
EWC [48]
$7,96#1.33
-20.42
62.32=1.34
-13.32
33.10#0.14
-18.50
GEM [24]
67.3841.75
-18.02
55.42=1.10
-24.42
39.50=20.62
-17.50
48.27
-13.7
MER [17]
77.42
-5.60
73.46
-9.96
51.00
-13.57
51.38=11.0
-12.83
La-M [28]
77.42-40.60
-8.64
74.34-0.07
-7.60
50.43-40.21
-10.00
61.18-1.44
-9.00
sp-La [30]
77.77420.88
-8.16
76.88
-8.39
50.81
-13.73
Ours
80.32
N/A
78,48-20.76
74.07
N/A
62.58
N/A
Table 3: We show that Crompress-then-reees is a strong bascline that outperforms previous methods on four continual learning benchmarks. Baseline numbers are from [28] or obtained from public official repos.
Benchmarks. We evaluate our method on three tasks widely used in previous Continual Learning works. MNIST Rotations [24] contains 20 tasks with 1,000 samples in each. Every task consists of images rotated by a fixed angle from to 180 degrees. MNIST Permutations [48] has 20 tasks, and each task contains 1. 1,000 images generated through shuffling the image pixels by fixed permutation. MANY Permutations [17] is a longer variant with 100 tasks in total and 200 samples each. Incremental CIFAR-100 [29, 24] splits the CIFARI00 dataset into 20 5-way classification tasks as the task stream for learning.
Our model. Based on our distillation method, we adopt simple framework to perform continual learning: "compress then recall". During the training phase, we do not perform learning on neural networks, instead, the dataset of each task is distilled to memories and the paired addressing matrices. During test phase, we simply fetch the corresponding memories and addressing matrices for each task, and train new model from scratch to perform classification. Memory buffer designs. When new task starts, we use the full remaining memory buffer to store the samples and perform distillation with both buffer samples and streamed samples. After a task ends, the distilled memories and addressing matrices are stored in the buffer, taking 1/T of the space, where T is the total number of tasks. Namely, the buffer size keeps shrinking when more compressed representation of tasks is stored. Note that we compare our model with previous methods under the exact same memory sizes for fair comparisons. See the appendix for more details model and memory designs.
Results. We show that this simple method is already strong baseline that outperforms prior arts on four benchmarks, summarized in table 3. Our method is compared with: Online, EWC [48], GEM [24], MER [17], C-MAML [28], La-MAML [28] and Sparse-LaMAME [30]. For example, we can obtain 23% boost on MNIST MANY benchmark: from 50.81% 1% to 74.07%.
We further compare our model with previous works Kernel Continual Learning [39] and Stable SGD [37] following their settings, where each task in MNIST Rotation and MNIST Permutation contains 60,000 samples instead of 1,000 samples. Our model achieves 87.3 and 0.55 on Permutated MNIST and Rotated MNIST under their setting, outperforming both KCL (85.5 and 81.8-0.600 and Stable SGD (80.1 10.51 1 and 70.8 Interestingly. our results also are higher than the multitask upperbound (86.5+0.21 and 07.3=0.47) potentially due to that there is task interference in joint training, which can be naturally avoided in our method.
5.3 Synthesizing new classifiers after learning
If we want to memorize the past, what is the benefit of storing the compressed representation rather than a trained model? In this section, we show that our compressed representation can enable flcxible synthesis of new classifiers after the learning. Specifically, we demonstrate extrapolating between tasks to train new models, and performing memory recall with images instead of labels, showing the generalizability of our framework other query forms.
5.3.1 Extrapolating between tasks
In the real world, tasks often do not come together and a learner, therefore, cannot observe all tasks at once. In current machine learning paradigms, when models are separately trained for disjoint tasks, it has difficulty extrapolating between tasks to build new classifiers. This is different from human learning. We show that storing our compressed representation enables learner to extrapolate and synthesize new classifiers after learning separately each task. Specifically, we separate CIFAR100

---

into 20 disjoint 5-way classification tasks as training tasks. For testing, we select classes that are not seen together during training by randomly choosing tasks and picking class from each selected task. We use k = 2 and A = 5 to construct 2-wap 2-way and 5-way classification tasks, and sample 1,000 tasks each for evaluation. To train our models, we independently distill the datasets for 20 training tasks into corresponding memories. For each testing task, the class labels are used as queries to recall the synthetic data from the corresponding memories. The recalled data for each label, although not seen together during training. are used for re-training k-way classifier from scratch. We find that the compressed data can indeed train classifiers on new combinations, for example, we can achieve 72.53%=8.. 74 on 2-way classification, and 46.54% 66.4%% on 5-way classification, with I image per class storage budget. The upperbound with the full real dataset is 92.23% 76 and 82.72%
5.3.2 Dataset Distillation extension - recall the past with images
We extend the standard setting to recall the past with images: when the label information and task scopes are missing, but a few visual observations can be made, we would like to build classifiers based on the visual data. For example, when we see a bear image and a deer image, but cannot recall the exact word or category, can we recall the memories with images and build a classifier? This is possible with our problem formulation, where the forms of queries are not constrained to labels and we can distil a datasel to memories addressable by images.
We formulate the problem as follows. Formally, after observ- ing training dataset Dtr with y .O.. C-1], we would like to flexibly build classifiers for a subtask Y/ C. y/ based on visual observations X2 from V/. when the actual information of V/ is unknown. We work with 1-shot and 5-shot observation cases. As a baseline, we build a nearest neighbor classifier, which is pretrained on Der and takes features of few-shot data to classify test images. As a model variant, we could also cclassify-then-recall trained on D,,- to map the image shots into labels and turning into the standard setup. Benefiting from the general design of our formulation, we show that a model can directly perform "image addressing", where feature network can provide query vectors y in fig.1. The feature network, memories and addressing matrices can be jointly trained on Dtr. We evaluate the above models and basclines on CIFAR 100 and summarize the results in table 4. As shown in the table, our model is not only able to successfully perform the continuous query addressing with image feature vectors, but also outperforms two strong baselines constructing new classifiers. More analysis is in the appendix.
Methods
shot
shot
Nearest neighbor
48.55
61.72
classify-then--ecall
50.58
58.46
image addressing
55.74
71.20
Table 4: Few-shot perf. recovery.
using a classifier
6 Conclusion and limitations
In this paper, we propose a framework that distills a large dataset into compact addressable memories. This framework introduces several benefits, including removing the linear growth contraints on the compressed data size, allowing more general queries besides categorical labels, and most importantly, achieving high compression rate with strong re-training performance, outperforming previous state- of-the-arts in dataset distillation. We also demonstrate a Soompress-then-reann framework, leading to new state-of-the-arts in continual learning on four datasets. Our full model has potential limitations on the costly inner optimization loop, which might be time-consuming on larger models or datasets. This limitation might be solved by combining the memory formulation with different learning framework. One potential societal concern with dataset distillation in general is that the distilled dataset may not contain the full diversity of the original data distribution, causing the retrained classifier to perform especially poorly on minority populations; our method arguably takes a step towards mitigating that concern through improving the retrained accuracy.
method using our
7 Acknowledgements
This material is based upon work supported by the National Science Foundation under Grants No. 2107048 and 2112562. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We would also like to thank Vishvak Murahari, Sunny Cui, Ruth Fong, Vikram Ramaswamy, and Zeyu Wang for discussions.
10

---

References
[1] Timothy F Brady, Talia Konkle, and George A Alvarez. Compression in visual working memory: using statistical regularities to form more efficient memory representations. Journal of Experimental Psychology: General, 138(4):487, 2009.
[2] Geoffrey R Loftus and Elizabeth Loftus. Human memory: The processing of information. Psychology Press, 2019.
[3] John R Anderson and Gordon H Bower. Human associative memory. Psychology press, 2014.
[4] lan J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint 11X.v:13/2.6211, 2013.
[5] Michael McCloskey and Neal J Cohen. Catastrophic interference connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[6] Tongzhou Wang, Jun Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 10955,, 2018.
[7] Ian Goodfellow, Jean Pouget-Abadin Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[8] Ian Gemp, Brian McWilliams, Claire Vernade, and Thore Graepel. Eigengame: Pea as a nash equilibrium. In International Conference on Learning Representations, 2020.
[9] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 131iv:1312.61 14, 2013.
[10] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021.
[11] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning. 2021.
[12] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34, 2021.
[13] Timothy Nguyen. Zhourong Chen, and Jachoon Lee. Dataset meta-learing from kernel ridge-regression In International Conference on Learning Representations, 2021.
[14] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. arXiv preprint arXiv:2110.0411 2021.
[15] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Yan Zhu. Dataset distillation by matching training trajectories. Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition, pages 4750-4759 2022.
[16] Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distilla- tion. arXiv preprint arXivv 2019.
[17] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11 19001 2018.
[18] Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-1135. PMLR, 2017.
[19] Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning, pages 1165-1173. PMLR, 2017.

---

[20] Aniruddh Raghu, Maithra Raghu, Simon Kornblith, David Duvenaud, and Geoffrey Hinton. Teaching with commentaries. International Conference on Learning Representations, 2020.
[21] Sid Reddy, Anca Dragan, and Sergey Levine. Pragmatic image compression for human-in-the- loop decision-making. Advances in Neural Information Processing Systems, 34, 2021.
[22] Shengjia Zhao, Abhishek Sinha, Yutong He, Aidan Perreault, Jiaming Song, and Stefano Ermon. Comparing distributions by measuring differences that affect decision making. In International Conference on Learning Representations, 2021.
[23] Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
[24] David Lopez-Paz and Marc'Aurelis Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30:6467-6476, 2017.
[25] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10458 2019.
[26] Tom Veniat, Ludovic Denoyer, and Marc' Aurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. arXiv preprint arXiv:2012.126:: 2020.
[27] Cuong V Nguyen, Yingzhen Li, Thang Bui, and Richard E Turner. Variational continual learning. arXiv preprint
2017.
[28] Gunshi Gupta, Karmesh Yadav, and Liam Paull. La-maml: Look-ahead meta learning for continual learning. arXiv preprint XXXx:20077.1904 2020.
[29] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental elassifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[30] Johannes Von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, and João Sacramento. Learning where learn: Gradient sparsity in meta and continual learning. Advances in Neural Information Processing Systems, 34, 2021.
[31] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European conference on computer vision, pages 524-540. Springer, 2020.
[32] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting classification tasks. arXiv preprini arXiv: (tti:::9990008388 2(6), 2019.
[33] Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Con- tinual learning in deep neural networks. Trends in cognitive sciences, 24(12):1028-10) 1040, 2020.
[34] Andrei Rusu, Neil Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606.04600 2016
[35] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018.
[36] Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plastiiity dilemma: In- vestigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 4:504, 2013.
[37] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes continual learning. Advances Neural Information Processing Systems, 33:7308-7320, 2020.
12

---

[38] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2020.
[39] Mohammad Mahdi Derakhshani, Xiantong Zhen, Ling Shao, and Cees Snoek. Kernel continual learning. In International Conference on Machine Learning, pages 2621-2631. PMLR, 2021.
[40] Alex Nichol and John Schulman. Reptile: scalable metalearning algorithm. arXiv preprint arXiv:1803.02990 80999999 2(3):4, 2018.
[41] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-14)) 2012.
[42] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 7781v:1708..7777,, 2017.
[43] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[44] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[45] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[46] Kai Wang. Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe learning to condense dataset by aligning features. In IEEE/CVE Conference on Computer Vision and Pattern Recognition 2022, 2022.
[47] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. arXiv preprint arXiv: 2018.
[48] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barbbkka,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):3521-352 2017.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Details are in main paper and more the supplementary materials.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] More details are in the supplementary materials.
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
13

---

(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curatinn contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "zisilOKt83M", "reviewer": "Reviewer_3HqT", "summary": "The paper proposes to compress the critical information of a large dataset into compact addressable memories. This method achieves better results than the previous dataset distillation/condensation methods.", "questions": "1. Algorithm 1. Lines 8 - 13 are the detailed optimization algorithm for $\\theta$. In line 15, there is the optimization algorithm for $\\phi$. What are the details of this optimization algorithm?\n2. There is also a research direction that generative models are used to condense a dataset. This direction is closely related to this work and should be discussed in the paper.\n3. In previous data distillation works, the number of inner-loop optimization steps is small. The major reason is that the memory consumption and execution time are quite large with a large trajectory. Could the authors discuss how they address this issue?\n4. There is a trade-off between time and space. If we prefer a higher compression rate, we have to pay the cost of extra computation (e.g., the computation for decompression). Could the authors discuss this trade-off and make a comparison with previous work?\n5. Instead of reporting the performance given a specified compression rate, it is also important that what is the compression rate if we can fully recover the original performance. Namely, what is the size of the smallest dataset that can be trained to achieve the full dataset performance?\n6. The authors use the notation $M=${$b_0, ..., b_{K-1}$}. It can be simplified as $M \\in R^{K \\times d}$.\n7. Is it possible to compress the condensed dataset directly? For instance, the previous methods can generate a condensed dataset. Could we apply several lossless compression on them? It is better for the authors to discuss it.", "limitations": "As a derived method of the dataset distillation framework, the proposed method inherits the limitations of the original method on the large models and large datasets. The authors also mention it in the last section.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths.\n1. The results are much better than previous methods.\n2. The method is based on a reasonable assumption that there are redundancies in the previous condensed dataset. The method achieves a higher compression rate by reducing redundancies.\n3. The paper is well organized and developed.\n\nWeaknesses.\n1. The experiments are only on small datasets. The size of the dataset $N$ and the dimension of input $d$ are small. The authors should report results on large datasets. Specifically, in [15], Tiny ImageNet and ImageNet subsets are used in the experiments.\n2. The experiments on large models instead of small models are needed. Does the method depend on a specific neural architecture? The authors may investigate other neural networks.\n3. Other missing experiments are mentioned in the questions below.\n4. The writing is fair. Several typos and errors are listed below.\n\n* Line 45. significanly -> significantly\n* Line 50. lead to -> leads to\n* Line 63. There has been -> There have been\n* Line 64. criterions -> criteria\n* Line 66. It emphasize on -> It emphasizes\n* Line 70. gaussian processes -> Gaussian processes\n* Line 79. dillema -> dilemma\n* Lines 92, 100, 151. e.g. -> e.g.,\n* Line 99. the number of synthetic data sample -> the number of synthetic data samples\n* Line 115. seprately -> separately\n* Line 121. outperforms -> outperform\n* Line 146. defines -> define\n* Line 170. having to enumerating -> having to enumerate\n* Equation 3. J(\\phi) -> min J(\\phi)\n* Line 179. discuss in detail about the learning framework -> discuss in detail the learning framework\n* Line 212. there are strong evidence -> there is strong evidence\n* Line 213. there are information re-using -> there is information re-using\n* Line 223. with shape 28 × 28 -> with a shape of 28 × 28\n* Line 228. are color image 228 dataset -> are color image 228 datasets\n* Line 263. distinct to each other -> distinct from each other\n* Line 292. 200 sample -> 200 samples\n* Line 308. pervious -> previous", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "d5RzeBawjY", "reviewer": "Reviewer_AR9N", "summary": "Based on the observation that classes can share concepts and representations, the authors propose a new Dataset Distillation method that trains a learnable memory shared across classes. The objective is to learn basic concepts that can be shared and then, through a function A, learn to combine these concepts to generate examples given a class y_i. To learn the method, the authors use a bi-level optimization process. The inner loop verified the generalization properties of the current memory, and the outer loop updated the memory and attention function. Additional to the method, the authors propose a modification in the momentum factor during the inner loop that helps improve performance. This method achieves good results in various Dataset Distillation benchmarks and good results in Continual Learning scenarios.", "questions": "Q.1 In several works, the attention masks are learned independently of the model/memory weights, so that interference between the learning processes is avoided. Does it make sense to learn M and A separately? In some ways, this experiment seems helpful to ensure that the memory acquires basic concepts, and not that function A selects useful random vectors that can achieve a goal, like finding the lottery ticket from a memory.\n\nQ.1.2 Perhaps the previous question is related to what happens in Fig. 2. For some classes, there is an intuitive and testable relationship of similarity. However, this similarity is not seen in other intuitively similar classes (cats and dogs). Is there any explanation for what is happening?\n\nQ.3 One of the problems with using long paths in the inner loop is the amount of memory used to store the entire associated computation graph. Although there are techniques to mitigate this, it is not mentioned. Can this proposal be explained in more detail? Has this problem been encountered?\n\nQ.4 It is unclear how the method is used in the Continual Learning environment. A single-pass context is mentioned. However, how can the proposed method be learned in just one data pass? Did the simple-pass apply only to the test adaptation? Or is the model capable of learning to compress information in a single epoch?", "limitations": "The authors mention a limitation that is very relevant and I share", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- S.1 The observation that concepts are shared across tasks is very intuitive. This observation, as the authors note, helps compress data so that it does not grow linearly as the number of classes increases, and may also lead to finding better concepts for classes that the model has not considered before.not seen.\n- S.2 The paper is well written, placing much emphasis on the different contributions that the authors propose. The different concepts and notations used are well explained.\n- S.3 Using the bi-level optimization strategy helps generate weights that can support the problem of continuous labels.\n- S.4 The results show that the proposed method achieves outstanding results. The experiments that were carried out demonstrate the contributions proposed by the authors.\n\nWeaknesses:\n- W.1 The reason for adding the momentum update is not clear. Nor how it is done. Is there any intuition behind this contribution?\n- W.2 Is there a reason why line 12 of Algorithm 1 does not use the opt function? It is used in equation 3, but not in the algorithm, which can cause confusion.\n- W.3 It is known that benchmarks of the dataset distillation problem are standard and normally use only small images. However, many of the previous methods have problems when the size of the images increases. It would be interesting to confirm the performance of this method on larger images. Alternatively, add it to the limitations.\n- W.4 There is no comparison of the algorithm's time or the number of iterations. Adding more iterations to the inner loop must come at a high cost.\n- W.5 Typo: Line 110, the abbreviation BPTT is used before defining", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RfasM-yXnCF", "reviewer": "Reviewer_fXia", "summary": "This paper formulates the dataset distillation as a memory addressing process where the synthetic datasets can be generated through an addressing matrix based on a learned memory representation. The proposed method improves the compression rate of distilled data by considering the class similarity and improving the BPTT framework. The proposed method is evaluated on five image classification datasets and applied to continual learning and few-shot learning.", "questions": "Table 1\n- The comparison to previous methods is not perfectly fair, and \"I/C\" is misleading. Due to the parameterization, the proposed method can generate more synthetic data than previous methods. Therefore, the proposed method is expected to achieve higher accuracy than previous methods. Thus, the number of addressing matrices (or the exact number of synthesis examples) needs to be shown somewhere in the table. I want to see whether more distilled data can fully explain the improvement or if the distilled images also get better quality. Appendix Figures 4 and 5 show that the recalled images lack diversity. I am surprised that the method achieves such a good performance.\n\nTable 2\n- It may be better to clarify the abbreviation (i.e., ds, Aug) in the caption.\n\nFigure 3\n- You mention that unrolling the trajectories long enough (e.g., 200 steps) can improve performance (Line 202). Could you show a complete picture in Figure 3 up to 200 steps? Due to the long unroll of the BPTT, I would also like to see a plot of test accuracy versus training time and the memory requirement of the proposed method. How does it compare to other baseline methods?\n\nFigure 4\n- I am curious about the trade-off between the number of bases and addressing matrices given a fixed memory budget. What is the general rule to scale these two numbers? I want some plot where the x-axis is the memory budge, and the y-axis is the ratio between these two numbers.\n\nResults 2\n- I can see that long unrolls will be very important, but it is unclear why momentum can be so significant? What's the insight behind that?\n\nLine 301\n- \"the buffer size keeps shrinking when more compressed representation of tasks is stored\": does it imply that the later distilled data tend to have a low quality?\n\nMisc\n- What would happen if you use a soft label for query (as a data augmentation) during test time? Will the performance get improved?\n- It will be interesting to see the visualization of the interpolation between two different classes. Does the \"half dog half ship\" image make sense?\n- I want to see some visualization of the images generated by BPTT. How does it compare to the previous method?", "limitations": "Due to the long unroll of the BPTT, the training efficiency and memory requirement can be the biggest concern when scaling this approach to complex datasets or larger models.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is well-motivated and focuses on a significant problem (the parameterization of the distilled data) in dataset distillation. The proposed method is novel, and the results are encouraging.\n- Originality: The proposed method proposes a new way to parameterize the distilled data and opens up a new application for few-shot learning.\n- Quality: This paper is technically sound, and most of the arguments are well supported. The evaluation is not perfectly fair, and additional study is needed to understand the method better.\n- Clarity: This paper is easy to follow and mostly well-written, with minor flaws.\n- Significance: The proposed method achieves state-of-the-art performance when a fixed memory budget is given. The addressable memory parameterization can also be applied to other dataset distillation methods. This paper also shows that dataset distillation can benefit continual learning and few-shot learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept