Dataset Distillation via Factorization
Authors: Songhua Liu
Kai Wang
Xingyi Yang
Jingwen Ye
Xinchao Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Dataset Distillation
Dataset Condensation
Dataset Factorization

Abstract: In this paper, we study dataset distillation (DD), from a novel perspective and introduce a \emph{dataset factorization} approach, termed \emph{HaBa}, which is a plug-and-play strategy portable to any existing DD baseline. Unlike conventional DD approaches that aim to produce distilled and representative samples, \emph{HaBa} explores decomposing a dataset into two components: data \emph{Ha}llucination networks and \emph{Ba}ses, where the latter is fed into the former to reconstruct image samples. The flexible combinations between bases and hallucination networks, therefore, equip the distilled data with exponential informativeness gain, which largely increase the representation capability of distilled datasets. To furthermore increase the data efficiency of compression results, we further introduce a pair of adversarial contrastive \xw{constraints} on the resultant hallucination networks and bases, which increase the diversity of generated images and inject more discriminant information into the factorization. Extensive comparisons and experiments demonstrate that our method can yield significant improvement on downstream classification tasks compared with previous state of the arts, while reducing the total number of compressed parameters by up to 65\%. Moreover, distilled datasets by our approach also achieve \textasciitilde10\% higher accuracy than baseline methods in cross-architecture generalization. Our code is available \href{https://github.com/Huage001/DatasetFactorization}{here}.

Dataset Distillation via Factorization
Songhua Liu
Kai Wang
Xingyi Yang Jingwen Ye
Xinchao Wang
National University of Singapore
{songhua. lil,,iluuiee.e e0823044, xyyr22334
xyang] Qu mus edu, {jingweny, xinchao) @nus .edi.eeduuss..
Abstract
In this paper, we study dataset distillation (DD), from a novel perspective and introduce a dataset factorization approach, termed HaBa, which is a plug-and- play strategy portable to any existing DD baseline. Unlike conventional DD approaches that aim to produce distilled and representative samples, HaBa explores decomposing dataset into two components: data Hallucination networks and Bases, where the latter is fed into the former to reconstruct image samples. The flexible combinations between bases and hallucination networks, therefore, equip the distilled data with exponential informativeness gain, which largely increase the representation capability of distilled datasets. To furthermore increase the data efficiency of compression results, we further introduce pair of adversarial contrastive constraints the resultant hallucination networks and bases, which increase the diversity of generated images and inject more discriminant information into the factorization. Extensive comparisons and experiments demonstrate that our method can yield significant improvement on downstream classification tasks compared with previous state of the arts, while reducing the total number of compressed parameters by up to 65%. Moreover, distilled datasets by our approach also achieve ~10% higher accuracy than baseline methods in cross noss-architecture generalization. Our code is available here.
Introduction
The success of deep models on variety of vision tasks, such as image classification [23, 9, 34], object detection [33, 32], and semantic segmentation [39, 49, 25], largely attributed to the huge amount of data used for training and various pre-trained models [50]. However, the sheer amount of data introduces significant obstacles for storage, transmission, and data pre-processing. Besides, publishing raw data inevitably brings about privacy copyright issue in practice [40, 8). To alleviate these problems, Wang al. [47] pioneer research of dataset distillation (DD), to distill large dataset into synthetic one with only limited number of samples, that the training efforts with the distilled dataset for downstream models on the original dataset can be largely reduced,
which facilitates series of applications like continual learning [37, 36, 48, 27] and black-bor optimization [6]. Due the significant practical value of DD, many endeavours have been made on this area [55, 53, 54, 46, 21, 5, 56] to design novel supervision signals to train the synthetic datasets and to further improve their performances.
Nevertheless, there is potential drawback in conventional settings of DD: it largely treats each synthetic sample independently and ignores the inter coherence and relationship between different instances. As such, the information embraced by each sample, despite distilled, is by nature limited. Using the synthetic samples for training downstream models, therefore, inevitably leads to the loss of dataset information. Moreover, the few distilled samples incompatible with the enormous number of parameters in a deep model and may yield the risk of overfitting.
To verify these potential issues, we conduct pre-experiment on CIFAR10 dataset with 10 synthetic images per class, using MTT [5], the current SOTA solution on DD, the baseline. In addition
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Real Dataset
Performance Matching
Dataset
U
Factorization
Bascs
Hallucinators
H
H2
H2
Synthetic Dataset
Figure I: Intuition of our hallucinator-basis factorization for dataset distillation.
to the baseline setting, we also incorporate all the checkpoint synthetic datasets after each 100 DD iterations in the convergent stage train the downstream model. Since the synthetic images are fine-tuned during this stage. multiple checkpoints can be viewed as related but different, which may somehow increase the diversity. As result, it yields overall lower test loss and hence better final results in downstream training, as shown in the blue and green curves in Fig. 2, which indicates that current DD solutions can be potentially improved by leveraging some sample-wiss relationships to diversify the distilled data. Nevertheless, simply involving more data samples may also increase the memory overhead. This fact motivates us ask: is possible encode some shared relationships in dataset implicitly, instead of storing samples directly, to avoid such additional storage costs?
We show in this paper that, it can indeed be made possible through reformulating the DD task as a factorization problem. As shown in Fig. 1, we propose novel perspective dubbed HaBa, to factorize dataset into two compositions: data Hallucination networks and Bases. data hallucination network, or hallucinator, can take any basis as input and output the corresponding hallucinated image. Supervised by the training objective of DD, a set of hallucinators can synthesize multiple samples from common basis and are optimized to extract effective relationships among different samples in original datasets explicitly. In this way, information of [H] [8] images can be included for a factorization result with [H] hallucinators and [8] bases via arbitrary pair-wise combination, which improves the data efficiency of traditional DD exponentially. As shown in the yellow curve in Fig. 2, with the same budget the storage, our strategy achieves better test performance compared with the MTT bascline.
1.3
MIT
Tost Loss
1.2
Ours
1.1
1.0
100 200 300 400 500 Epachs
Figure 2: Visualization of test loss using syn- thetic datasets generated by MTT, MTT with multi- ple checkpoints, and ours.
To further increase the informativeness of factorized results, we introduce a pair of adversarial contrastive constraints promote sample-wise diversity. The goal of HaBa is minimize the correlation among images composed of different hallucinators but a common basis, while an adversary tries to maximize it. Such an adversarial scheme, turn, enforces the hallucinators to produce diversified images and increases the amount of useful information.
Notably, HaBa is versatile strategy that can be built upon existing DD baselines, since is compatible with any training objective for measuring the similarity between downstream performances as shown Fig. 1. We conduct extensive experiments demonstrate the advantages of the proposed method over baseline ones. In all benchmarks and comparisons, HaBa produces significant and consistent improvement on training downstream models, while reducing the total number of compressed parameters by up to 65%. Furthermore, it demonstrates strong cross- noss-architecture generalization ability with accuracy improvement higher than 10%. Our contributions are summarized as follows:
We study dataset factorization, a novel perspective to explore dataset distillation, and propose novel approach termed HaBa for hallucinator-bar bbsiss factorization.
We present a pair of adversarial contrastive objectives further increase the data diversity and information capability.

---

HaBa is plug-and -un--lay scheme compatible with all existing training objectives of DD and can yield significant and consistent improvement over the state of the arts.
2 Related Works
The goal of dataset distillation (DD) is to optimize a smaller synthetic dataset such that it is capable to take place of original one for training downstream tasks, which is different from coreset selection [1, 7, 13, 38, 44], another branch for dataset compression, directly selecting samples from raw datasets. In this section, we provide a detailed review of previous methods in DD.
Motivated from knowledge distillation [15, 12, 52, 51] aiming at model compression, Wang et al. [47] introduce the concept of dataset distillation for dataset compression. The idea is to optimize the synthetic images so that they can minimize loss functions of downstream tasks, where a bilevel optimization algorithm [11] is involved. Following this routine, several works further consider learnable labels beyond samples [3, 42]. Subsequently, Zhao et al. [55] and several following approaches [53, 24] consider matching gradients of a downstream model produced by synthetic samples and real images, which improve the performance significantly. Most recently, Cazenavette er al. [5] argue that single-iteration gradient matching may lead to inferior performance due to error accumulation across multiple steps and thereby propose to match long-range training dynamics of an expert trained on the original dataset. As an alternative method to profile training effects produced by different sets, Nguyen et al. [29, 30] also introduce the kernel ridge-regression approach based on the Neural Tangent Kernel (NTK) in infinitely wide convolutional networks [17].
Apart from matching training effects, there are also methods matching data distributions between original and synthetic datasets. For instance, Zhao et al. [54] propose simple but effective Maximum Mean Discrepancy (MMD) constraint for DD, which does not involve the training of downstream models and enjoys superior training efficiency. Wang et al. [46] propose CAFE, explicitly attempting to align the synthetic and real distributions in the feature space of a downstream network.
Above mentioned methods are dedicated exploring suitable training objectives and pipelines for DD. However, there are few works concerning improving the data efficiency for distilled samples. Although Zhao et al. [53] propose differentiable siamese augmentation (DSA) to enrich the training data, the augmentation operations used, e.g., crop, flip, scale, and rotation, cannot encode any information about the target datasets. In this paper, we study the task in factorization perspective, to factorize a dataset into two different compositions: data hallucination networks and bases. Both parts carry important knowledge of the raw dataset. For downstream training. hallucinators and bases can perform arbitrary pair- wise combination, i.e., sending any basis to any hallucinator, to create a training sample. The idea of factorization can improve the diversity of distilled training datasets significantly, without introducing additional costs for storage. It is also a versatile strategy compatible with all aforementioned DD methods, which will be demonstrated the experiment part.
Concurrent Works on Efficient Distilled Dataset Parameterization: As concurrent work, Kim et al. [21] propose IDC for efficient synthetic data parameterization. It reveals that only storing down-sample version of synthetic images and conducting bilinear upsampling in downstream training would not hurt the performance much. Thus, given the same budget of storage, can store 4x number of 2x down-sampl synthetic images compared with the baseline. Both IDC and HaBa in this paper are dedicated to improving the data efficiency of synthetic parameters. Interestingly. according to the definition of our hallucinator- basis factorization, IDC can in fact be treated as a special case of HaBa, where the hallucinator is a parameter- has a smaller spatial size. Nevertheless, the main focuses for IDC and HaBa are different and they are in fact two orthogonal techniques, which can readily join force to enhance the baseline performance, as discussed in Sec. 4.2.
upsampling function and each basis
3 Methods
In this section, we elaborate our proposed method HaBa for dataset distillation (DD). Assume that there is an original dataset T {x;, 11 with [T] pairs of training sample x; and the corresponding label yt- DD targets a synthetic dataset S = [(t,, [1) with |S] << [T] and expects that model trained on S can have similar performance than that trained on T.
3

---

B
H
T
Sample
Stream
Down
For B and H
Lnn
Encoder
Sample
Sample
Leas.
B1
task
B2
R₂
L Loo.
Decoder
For Adv.
Figure 3: Left: Overall pipeline of the proposed hallucinator-basi, factorization. H. and denote sets of bases, hallucinators, and original data respectively. Adv. denotes an adversary model. We adopt batch size 2 here for elarity; Right: Architecture of a hallucinator in detail.
Traditional DD methods treat each synthetic sample independently and ignore the inner relationship between different samples within a dataset, which results in poor data/information efficiency, Focusing on such drawback, we study DD from a novel perspective and redefine it as hallucinator-bast factorization problem:
S [Ho 3 U ((t 9:))
where there are [7[ hallucination networks and [B] bases. The j-th hallucinator is parameterized by 6 and we denote it by Ho, for 1 < j < 17]. For downstream training, training data pair (øtj. Rig) created online via sending the i-th basis, with any 1 15 S [B], the j-th hallucinator, with any 1 J j < [H], Le., Fig = Ho, (R.). In this paper, the label Dij is simply taken as ys.
An overview of our method is shown in Fig. 3(Left). To go deeper into the technical details, first start with the introduction of our basis and data hallucination network in Sec. 3.1. Then, we propose an adversarial contrastive constraint to increase data diversity Sec. 3.2. Finally, we present the whole training pipeline of the hallucinator-bast factorization for DD in Sec. 3.3.
3.1 Basis and Hallucinator
Basis: Typically, for image classification dataset T ((, ((2,,))) 2; E RXxxxx and yi {0, 1, I.... C' 13 for each 1 I i S [T], where each D3 is e-channel image with a resolution x w, and C' is the total number of classes. In previous DD methods, the format/shape of synthetic data pairs (f: () has to be held the same as that real data, so as to make sure the consistency between input and output formats in the training and test time for downstream models. By contrast, since hallucinator networks are capable of spatial-wise and channel-wise transformation, the shape each ft 1 < < [B], denoted as h' w' c, is not necessarily the same that of original samples and thus more flexible. And for classification problem, we do not modify its label space in this paper for simplicity and maintain the categorical format.
Given basis à IR ggl'xw'xe xe data hallucination network, aims create new image & E R&x wxxx based on é. which can be viewed as a conditional image generation problem. Inspired by image style transfer [19, 16, 18, 26], typical conditional image generation problem, we devise an 3(ighht). Specifically, the encoder, denoted as enc, is composed of CNN blocks, which non-linearly maps input å to feature space RA" x"xxxxx" Then, affine transformation with scale and shift is conducted on the derived feature, where a and μ are treated as network parameters in this paper. At last, the decoder dec under symmetric CNN architecture with enc projects the transformed feature back to the image space. Formally, this process can be written as:
Hallucinator:
based architecture for hallucinators, as shown in Fig.
f= enc(f), f=oxf+l I = dec(f),
(2)
where the multiplication is element-wiss operation. There are H| hallucinators in the whole factor- ization pipeline and each would be trained to implicitly encode some sample-wise relations by its network parameters.
3.2 Adversarial Contrastive Constraint
Ideally, the knowledge encoded by different hallucinators should as different/orthogonal as possible to get the most benefits for each individual. To instantiate such regularization, let's consider two

---

composed images 2, and ã. from two different hallucinators Hø, and Høg but common basis T; The divergence between øiy and ā.z is expected be large. To measure the divergence, feature extractor is required to map an input image to feature space, and how to train such feature extractor to find an appropriate feature space of great importance.
In this paper, we formalize the training of hallucinators and the feature extractor as min-max game self-consistent manner, where the feature extractor desires to minimize the divergence between Tij and āıs while hallucinators, as well as bases, are optimized to maximize it so that the two players can reinforce each other. specific, the feature extractor, denoted as F and parameterized by y is typically a CNN structure for the downstream task and we adopt features at the last hidden layer before the output layer, denoted ( and F-1 F-1( is optimized to maximize the correlation between the two feature vectors, which can be quantified by the metric of mutual information (MI). Inspired by the lower bound of MI [45], the objective mínimize the divergence for given by the following contrastive form:
exp (F [F_ (T (o))))1))
Coo..
Σ log
(3)
|2
Σ B u== exp cxp (Bt)
KSiKK Hl, jæk
where is scalar temperature coefficient. For the classification problem, can alternatively adopt the supervised form of the contrastive loss Lem., where Tuk with the same class label as Tij are also taken into consideration as positive samples in Eq. 3. The supervised contrastive loss can benefit to increase the correlation of samples from the same class [20] for a more reasonable feature representation.
In addition, the feature space is expected to reflect the task-specific property for a meaningful representation. Thus, we also incorporate the task loss Ltask; e.g., cross-entropy loss in classification tasks, over the synthetic dataset as a supervision signal for F. In this way, the overall training objective for F is defined as:
min LF = Acon. Loon. + deask Crask:
(4)
where Aeon. and Atosk are hyper- paremeerrss controlling the weight for each term.
F acts as an adversary to minimize the divergence between žiy and Tak, while the synthetic dataset expected to maximize it to increase data diversity. To this ends, the similarity between F- (maz) and (#ax) becomes one loss term for hallucinator-bass factorization. In this paper, we adopt the cosine-similarity and the objective Leos. is given by:
FI FL F FL1
Cens.
Σ
Σ
(5)
1
B
ISJA HI, jfk
During training, the feature extractor and factorized components are updated alternately to play this min-max game.
3.3 Factorization Training Pipeline
Following previous paradigms [55, 54, 5, 46], the synthetic dataset S is updated in an iterative algorithm. In each iteration, we randomly sample batch of hallucinators and bases and conduct pair-wise combinations. The composed images are evaluated by the objective of dataset distillation LDD and the similarity metric in Eq. 5:
min Ls = ADDLoD + Nooe..Coo
(6)
where parer-pprraeters ADD and Acos. balance the loss.
Notably, the hallucinator-bas basis factorization is compatible with a variety of configurations of LDD by previous arts, which makes it a versatile and effective strategy for DD. this paper, we adopt the trajectories matching loss in Cazenavette et al. [5] LDD by default thanks to its superior performance. The basic idea to update a downstream model from a cached checkpoint oi at iteration using the synthetic dataset S for N times, and using the real dataset T for M times
5

---

respectively. The updated parameters by the two cases, Si+N and 6i+M are enforeed to be consistent:
@itntI + :ctn - aV VVorta
ditn (SacH(S), de t 0 snn<N,
Pitm+1 t pitm -BV Nitm Lrask (T), 5 m < M,
1ÔttN - Pi+M
LDD 1 - 17+M
where a and | are learning rates with S and T respectively. is learnable in the framework while is a hyper-parameter. In Sec. 4.2, we also experiment with other settings of LDD.
Based on the supervised signals in Eq. 6, the gradients are backward propagated to the composed images and finally to the sampled hallucinators and bases so as to be updated using a decent algorithm such SGD. Since all the operations are differentiable, the training can be completed end-to-ens
Experiments
4.1 Datasets and Implementing Details
We conduct evaluations of our method on three standard image classification benchmarks: SVHN [28]. CIFARIO, and CIFARION 100 [22]. There are 60,000 images for real-world digit recognition in SVHN. For CIFARIO and CIFARI00, there are 50,000 training images in total. The number of classes for the three datasets are 10, 10, and 100 respectively. All the images are under 32 x 32 resolution in 3-channel RGB format. Following previous works [5], we use ZCA for image preprocessing with Kornia implementation [35] before all the experiments. Experiments with more datasets, including images in larger spatial scales, can be found in the supplement.
In this paper, for convenience of comparisons with prior works, we maintain the same size with images in original datasets, i.e., h' h. w' = w. and c' 3 for bases. We also experiment with other sizes of bases in Sec. 4.3. For hallucinators, the encoder and decoder contain ReLV blocks. The number of feature channel c" is 3. We use 5 hallucinators by default. The learning rates of hallucinators and bases, UH and T/. are the same and for the feature extractor, the learning rate Aeon., Atask, ÀDD. and Acos. are set as 0.1, 1, 1, and 0.1 empirically.
oF is 0.001. Hyper-paramen Sensitivities of these hyper-paramen parameters are analyzed in Sec. 4.3. The adversary network has the same architecture as that for computing CpD. In experiments on SVHN and CIFARIO. we incorporate all the bases in each iteration, while in experiments on CIFARIOM 100, we adopt a batch size of 300 when the total number of bases is greater than 1,000. We only consider random hallucinators in one iteration for simplicity. The maximal configuration of computational resources is 4 24GB 3090 GPUs. The GPU memory consumption is dependent on that of the baseline method for CDD and is slightly higher than it due to the computation of Loos. and Loon. The baseline method for LDD is MTT [5] not specified. Other settings related to DD hold the same as the baseline. All the quantitative results are based on the mean and standard deviation over 5 repeated experiments. To make sure fair comparisons, the dataset size in our method is equal to the number of bases [ and the hallucinators treated as parameterized data augmentors working online in downstream training. just as general data augmentations, which means that the dataset size does not increase compared with the baselines.
4.2 Comparisons
Comparisons with State of the Arts: We compare HaBa with previous state of the arts for DD in standard settings, synthesize 10, and 50 images per class (IPC) respectively, In our setting, the number of parameters in a hallucinator is is approximately equal to that for 2 synthetic images, while the size of basis is equal to that of an image. Taking the storage cost of hallucinators into consideration, we set the number of bases per class (BPC) as IPC minus 1 in each IPC con- figuration when IPC is greater than 1, to make the comparisons as fair as possible. Candidates are coreset based methods including Random [7, 31], Herding [4, 21, K-Center [10, 38], and Forget- ting [43], meta learning based methods including DD [47] and LD [3], training matching based methods including DC [55], DSA [53]. and MTT [5], and distribution matching based methods including DM [54] and CAFE [46]. The comparisons follow the standard protocol adopting a 3-layer
Conv-Instancelorres AvgPool ConvNet with 128 channels in training and testing.

---

Dataset
SVHN
CIFARIO
CIFAR100
IPC
10
50
Ratio %
0.014
0.14
0.7
0.02
0.2
0.2
146.1.
38.114.1
70.9109
24
26.0.112
4210
14.610.5
30.0.044
Herding
20.9
50.5+3.3
72.6-08
11
31,6407
40.40.6
8.4
173
33.7400.5
Coresct
K-Center
21.0:1:.5
14.0+1.3
20.1=14
21.51.
14.7409
27.011
8.34
71+02
30.540.
Forgetting
12.1117
16.8112
27.2
13.5#12
23.3+1.0
23.3
4.5003
98202
DDf [47]
36.81112
[3]
25.7=0.77
38.3+04
42.5+04
11.5+04
DC
31.2+114
76.1+
8233-033
28.3=0.5
44,9205
53.920.5
12.820.3
25.2
Distillation
DSA [S3]
27.5.114
79.2.00.5
8..4.0.4
28.8.00.7
52,11005
6066..55
13.910.3
32.3.0.3 428.10.
DM
26.0=2..8
48.9106
63.0:14
11440.3
29.7
43.6404
CAFE [46]
42.6#33.3
75.9+066
8133
30
46.3#06
55.5
12.9+0.3
27 37.9
CAFEADSA [46]
42.9.1..0
77.91066
82.14-0.4
31.640.88
50.9.1.5
62.3404
31.540.2 42.9402
MTT [S]
58.5+14 1.88+118
85.7=0.1
46.3=0.88
65,3+0.7
71.6+0.2
24.3++0.3
39.0=0.1
46.1
9
49
Factorization
Ratio %6
0.028
0.14
0.7
0.04
0.2
0.22
1,82
9.82
6..8.13
83.2.004
88.3.0.1
48.3.0.88
74.0.10.2
33.4
40.2.10.2 47.0.10.
Whole Dataset
95.4
84.8
56.2.2.33
Table I:
The performance (test accuracy % comparison to state-of-the-art methods. LDf and DD use AlexNet for CIFARIO, while the rest use ConyNet for training and testing. IPC: Number of Images Per Class; BPC: Number of Bases Per Class; Ratio (%): the ratio of distilled images whole training set. Underline denotes results by our implementation.
(d) Images by Baseline Figure Visualization of factorized results by our HaBa (70.27% test acc.) and baseline MTT (65.92% test acc.). Zoom-in for better comparisons.
(a) Bases
(b) Images by Hı
(c) Images by H2
The comparison results are shown in Tab. I and we can observe that HaBa achieves tate-of-the-as art performance in all datasets and settings. Especially when the ratio of distilled images the whole training set is less than 1,, our method can yield significant improvement over all the candidate methods, which demonstrates that the scheme of hallucinator-baa basis factorization improves the data efficiency for the task dataset distillation.
Qualitative Comparisons: We visualize the factorized results by our method well as the baseline on CIFAR 10 dataset with 10 BPC Fig. 4. Due to the space limitation, we only provide images generated by 2 hallucinators here. More results can be found in the supplement. As shown in the figure, we can find that bases mainly store some main structures and contour information. Different hallucinators would render a basis with diverse styles and details. Thanks to the dataset factorization scheme, the diversity of distilled images by our method is higher than that by the baseline.
Building upon Different Baselines:
To reflect the versatility of the insight, we implement HaBa training pipelines DD, including DC, DM, and MTT. We evaluate
on multiple state-of-thed the performance of synthetic datasets CIFAR10 10 and maintain the IPC of baseline methods as BPC plus 1, which makes storage costs for synthetic datasets as close as possible for fairness. As shown in Tab. 2, when training and testing ConvNet, the strategy of HaBa can make a consistent improvement over all the basclines, which demonstrates that factorization is a general idea to improve the data efficiency in DD.
Pross-Architedte Performance: For DD, satisfactory distilled dataset should have similar training effects to the original one on downstream models with arbitrary architectures. Thus, cross- architecture generalization performance is an important metric for DD. We use the synthetic datasets trained on ConvNet to train models with different structures including ResNet [14], VGG [41], and AlexNet [23]. The results can found in Tab. 2. Benefiting from the increased data diversity, HaBa can improve the across- architttcch accuracy significantly with a performance gain up to 17.57%. The consistent and significant improvement validates the superior ability of our method capture the informative features and thus original datasets can be replaced by the synthetic ones better.
7

---

Method|
DC [55]
DM [54]
MTT
IPC
2
11
2
11
51
2
11
BPC
1
10
50
1
10
1
10
Baseline 311 11..6..10.16
29
10.30
54.24....6111
5.55
50.35
62.03
50.59
69.81 19.48
ConvNet w. HaBa
34.11=0447 49
40.23
07.32-0..15 13
83
1I
64.44-0..00
0.3
69.48
Gain +2.75
-4.59
+4.67
+2.75
-6.48
+2.41
+6.17
+5.58
+3.44
Baseline
18.10:::0.76
18.36:1:.36
22.14::::::22255
40.00=11449
351
45.05==1.46
54,47=00995
ResNet
w. 1aiBa
24.49.10.55
24.27:10.56
11.08.10.32.11
1.035
47.39.....
57.97.10.88
64.35.10.60
Gain
+6.39
+6.11
+8.94
+9.09
+7.57
+6.21
+12.24
+12.92
+9.88
Baseline
/2.002::0026
35
22.22
41.6440..4
55,17
50,49:+1.02
01.36=0.30
VGG
w. 141
26.93=:00.22
49.41=0.36
(00044348.22600
08.56:
60.47::0.56
67.47=0.43
Gain
+1.40
+1.15
+3.18
+4.65
+7.77
+12.30
+10.22
+9.98
+6.11
Baseline 20.02:1:131 23342
29.48
20.66:3344
37.04
355
1.22
AlexNet
HaBa
22.24
33.02:+0:91
33.42+
44.
44.14
03.09=
43.63=1146
36000 1.37
Gain
+2.22
+10.60
+3.94
+11.47
+7.10
+3.95
+17.37
+13.01
+10.87
Table 2: Cross-architecture performance (test accuracy %) comparison to different baseline methods DD HaBa built upon.
Comparisons under the Same Number of Final Images: In the default comparison protocol, we compare our method with the baselines using the same budget of storage, where our method can store information of exponentially more images than the baselines with the same number of parameters. In this part, we also examine the performance of HaBa under the condition that the number of final images, i.e., [H|]
x [B. is equal to that used by the baseline. Intuitively, given that the objective functions of our method and the baseline are the same exactly, the performance of the baseline can be viewed an upper bound of ours, since there are significantly less parameters in method to carry the information of final images in this case. Therefore, we first remove the term Leos. from the loss function of DD in Eq. 6 guarantee a consistent optimization objective with the baseline. Then, we compare the performance of HaBa and the baseline using 10, 20, 30, 40, and 50 final images respectively. Here, the number of hallucinators | is and the number of bases is thus half of the number of final images. As shown in the red and green curves in Fig. 7, performance of the baseline can be well approximated by ours with only half of the number of parameters, especially when the number of images is relatively large. Remarkably, with the proposed adversary contrastive constraint, our method can even outperform the baseline consistently, as shown in the blue curve, which further demonstrates the effectiveness of the proposed solution.
Comparisons with Concurrent Works Efficient Distilled Dataset Parameterization: As concurrent work on efficient distilled dataset parameterization, IDC [21] is proposed to store 4x number of x down-sample synthetic images compared with the baseline. The core is to reduce the spatial size for efficient parameterization. For HaBa of this paper, instead, we do not modify the spatial size of bases in the default setting for better qualitative explainablity and more intuitive comparisons with the baselines. In this sense, IDC and HaBa are in fact two orthogonal techniques and they can readily join force to enhance the baseline performance. Here, we try using the technique of IDC and adopting 2x down-sampl synthetic images on the baseline MTT, based on which we further consider adding our HaBa and involving 5 hallucinators. As shown in Tab. 3, with the efficient parameterization of IDC, the performance of baseline can be improved. With HaBa in this paper, the performance can even be further improved a lot: 5. 5.14%, 1.29%, and 4.30% in the three settings respectively, which demonstrates that IDC and HaBa work in different ways.
To further demonstrate the advantage of the proposed method for improving data efficiency, following the setting of DM [54], we conduct experiments on the setting of continual learning on CIFAR-100, with 20 random classes per stage. The average number of parameters per class is 20 x 32 x 32 x 3. The synthetic datasets are trained with a ConvNet with 3 blocks. We evaluate synthetic datasets by our method and the DM bascline on the same ConvNet architecture and ResNet18. The results in Fig. 5 demonstrate that the proposed method increases the informativeness of synthetic datasets and thus produce significantly better performance, especially in the architerce setting.
Applications in Continual Learning:
4.3 Ablation Studies
Loss Terms:
To validate the effectiveness of the proposed adversarial contrastive constraints, we design ablation studies on the CIFAR10 10 dataset over three loss terms: Loos. in Eq. 5, Loan. in Eq. 3, and the task-specifi loss Lresk. Through the results in Tab. 4, we can find that deleting any one of
8

---

20
Besdin
Buiefer
60-
ComoNed Auceeee
55
50
of Param, Class 2x32x32x3
x1x32x32x3 32x
8 8
45
40
Baseline
49.89=0.95
65.92+00.22
50.73+t0.52
IDC HaBa
W. IDC
6..27.0.34
56.13=0.38
72.14.10.22
70.85:00.3
75.31:10.27
71.01:0441
&
30
20
10 60
30 100
20
40
60
80
100
Number of Classes
Namber of Classes
Table 3:
Comparisons with concurrent work
Figure 5: Comparisons on the setting of con- tinual setting. Results on the ConvNet3 (Left) and ResNet18 (Right) architectures are shown.
IDC [21] on efficient synthetic parameterization.
Baseline
Test Acc. of HaBo
BPC
50
70
70.2
70.2
70.2
70
70.2 70.2 702 70.2
69.6
HaBa wio Ceor.
54.56+:0.61
70.16+0.44
73.93=0.21
TatAAcurac
69
HaBa wio Lcon.
54.91=0.49
70,07=0.48
72.50=0.39
Hala w/o Ctask
54.62.10.22
70.07 1.0.6
0.27
HaBa Full
$5.66:10.29
70.27:10.63
74.04=0.16
IaBaa Coon, Downstream
56.7811.22 70.44.0..15 7..00
66-
0 0.01 0.1
0
0.5
Acos.
0.01 0.1 0.5 12244 Acon.
Table 4: Results of ablation study on loss terms
Figure 6: Impacts of different Aços. and Acon. on the test accuracy.
in HaBa: Lcos.. Loon. and Leask-
them would hurt the performance. We also experiment with involving Lcon. for downstream training. to enforce the similarity among images composed of different hallucinators and common basis. Observed from the last row of Tab. 4, the performance can be further improved, since Leon. helps the representation learning of related samples [20]. Note that we do not use this loss term for downstream training in other experiments for fair and standard comparison.
We examine the sensitivities of hyper-parameters Aeos. and Aeon. used to balance the weights of loss terms Lens. and Lem. respectively in Fig. 6. The results are evaluated on the CIFAR10 10 dataset with 10 BPC. We can observe that the overall performance is not sensitive to the selection of these hyper-parameters and our method makes a consistent improvement over the baseline with 11 IPC.
Class- Class-nndependent Hallucinators v.s. Shared Hallucinators: In the default setting of HaBa, each class maintains a certain number of bases independently and all the classes share the same set of hallucinators. But what about the case that hallucinators are also made fass-independent? We study this problem experimentally in Tab. 5. Given the same BPC, class- daass-independent hallucinators can indeed somehow improve the performance when there are fewer synthetic samples, e.g., 1 BPC. However, when BPC is higher, equipping each class with an independent set of hallucinators would not benefit the performance. There are probably two reasons: (1) shared hallucinators across all the classes extract global information of the whole dataset, which encodes more representative and universal knowledge; and (2) the class-independent case would make the number of hallucinators 10 times for the CIFARIO dataset, which leaves a heavy burden for the optimization process. Thus, as indicated in Tab. 5, a better solution is to make room for more bases using the memory allocated to store class-independent hallucinators initially, which would result in more satisfactory data efficiency.
Number of Channels Used by Basis: By default, the shape of basis is the same as that of real image, which is generally in RGB 3-channel format. In fact, in Fig. 8(Left), we also verify that is also possible to single-channel basis, which reduce the memory cost by nearly 2/3 without hurting the performance too much. Interestingly, if the memory cost is held the same, we can choose to use 3 times BPC to store single-channel bases, rather than 3-channel ones. This would yield impressive improvement on the test accuracy when BPC is small. Note that for baseline results, IPC set as the corresponding BPC plus
Number of Hallucinators: We study the impact of the number of hallucinators, i.e., [H]. in Fig. 8(Right). We can observe that when BPC is small, including more hallucinators is helpful for the performance. Nevertheless, when BPC is 10 or 50, the performance would not improve with more hallucinators when [76] 10. One reason is that when [7(] is large, the sampling of hallucinators in each iteration is sparse, which makes the joint optimization of all the hallucinators more difficult.
The similarity between our hallucinator set and data augmentation lies that both of them can contribute to generating more samples and increasing the diversity. However, the essential difference is that our hallucinators optimized to encode sample-wise relationships in
Data Augmentation:

---

72
1 Chamel
Clannel (3x BPC)
Chamel
70
75
Test Accurucy
Text Accuracy (%)
70
68
65
Accurscy
70
BPC-
BFC-10
66
Baseline
55
Ours wio
60
50
Ours
64
45
35
10
20
30
40
50
Number of Images
BPC
[2d
Figure 7: Comparisons with the baseline under the same number of final images.
Figure 8: Study on the number of channels used by bases and the number of hallucinators.
50
ConvNet
ResNet
VGG
AlexNet
w/o Shure
55.96.10.51 69.00.10.20
69.81 0,556
Share
39.66:10.29 70.27:10.33
74.04-1.116
w/o aug.
60.63-10.21
43.24-10.83
48.02.1.0.53
30.58
Baseline (IPC=BPC)
45 0..86 62.77:40.56
71.09:0.34
Baseline 63. 0.200
45 13.05:11. 146
50.49: 1.02
35.95 1.52
Share (Same Memory)
70.27:10.63
72.17:10.30
74.89.10.15
w/o aug.
68.08 1...3
56, 56.37:10.11
59.04:10.50
48.27:13.04
Baseline (Same Memory)
65.92:1:..2 68.58:10.49
73.55:0.48
Ours 69.48:10.26 57.97:10.88
60.47:100.6
48.96::3.00
Table 5:
Study on whether all the classes should share the same set of hallucinators.
Table 6: Impact of data augmentation.
a dataset, while data augmentation is based on some prior and heuristic knowledge of images. By default, both our method and the baseline adopt the data augmentation strategy DSA [53]. To study the relationship between the two schemes experimentally, we attempt to remove DSA from bascline and our method and report the corresponding results in Tab. 6. The evaluation is CIFARIO with 11 IPC for baseline and 10 BPC for ours. Through the results, we can find that (1) our method without data augmentation can also outperform the baseline method with augmentation significantly, which means that the mechanism of HaBa can benefit the dataset distillation task more with the learning of global information of a dataset in hallucinators; and with data augmentation, our performance can be further improved, which indicates that HaBa and DSA work in different manners.
5 Conclusions, Limitations, and Future Works
This paper proposes novel hallucinator basis factorization method dubbed HaBa for dataset dis- tillation (DD). It uses hallucinators encode inner relations between different samples in original datasets, which can largely improve the data efficiency of distilled results. To diversify the knowledge captured by different hallucinators, a pair of adversarial contrastive constraints is further introduced. Extensive evaluations and comparisons on multiple benchmark datasets demonstrate that HaBa is capable of significantly improving the performance of downstream models trained the synthetic dataset, using only 35% cost of memory for storage. Moreover, it is a versatile strategy that is compatible with different configurations of DD frameworks and yields consistent improvement.
Despite the superior performance of the proposed hallucinator basis factorization (HaBa) scheme, there also some potential limitations. On the hand. compared with the baseline method HaBa built upon, the process of online pairwise combination between hallucinators and bases in training adopted.
increases the cost of time and GPU memory slightly, although light-weight hallucinators On the other hand, may inherited the limitations of baseline methods. For example, when the number of images is large, further increasing the number would produce limited performance gain.
For future works, beyond the training efficiency of HaBa, introducing class-wise relationship may also be potential research direction. For example, it is probably optimal that one class shares hallucinators with some specific classes but does not share with others. also promising to explore more advance factorization for dataset to further improve the performance.
Acknowledgement
This research is supported by the National Research Foundation, Singapore under its Medium Sized Centre for Advanced Roboties Technology Innovation (WBS: 1-0009942-00-000, Xinchao Wang the corresponding author.
10

---

References
[1] Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint ToXXv:/703.067777 2017.
[2] Eden Belouadah and Adrian Popescu. Scail: Classifier weights scaling for class incremental learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1266-1275. 2020.
[3] Ondrej Bohdal, Yongxin Yang. and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint arXiv:2006. 08572, 2020.
[4] Francisco M Castro, Manuel J Marín-Jiméned" Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference computer vision (ECCV), pages 233-248, 2018.
[5] George Cazenavette, Tongzhou Wang. Antonio Torralba, Alexei A Efros, and Jun-Yau Zhu. Dataset distillation by matching training trajectories. arXiv preprint arXiv:2203.11923 2022.
[6] Can Chen, Yingxue Zhang, Jie Fu, Xue Liu, and Mark Coates. Bidirectional learning for offiine infinite- width model-based optimization. In Thirty-Sixth Conference Neural Information Processing Systems, 2022.
[7] Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. arXiv preprint urXiv:100.33772 2012.
[8] Tian Dong. Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? arXiv preprint arXiv:2206. 00240, 2022.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. / 19929, 2020.
[10] Reza Zanjirani Farahani and Masoud Hekkmatfrr. Facility location: concepts, models, algorithms and case studies. Springer Science & Business Media, 2009.
[11] Chelsea Finn, Pieter Abbcel, and Sergey Levine. Model-agnostic meta-learinn for fast adaptation of deep networks. International conference on machine learning, pages 1126 1135. PMLR, 2017.
[12] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1 2021.
[13] Sariel Pared and Akash Kushal. Smaller coresets for k-median and k-means clustering, Discrete & Computational Geometry, 37(1):3-19, 2007.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[15] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv: 3oXXx:5003.000 02531, 2(7), 2015.
[16] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization.
In Proceedings of the IEEE international conference on computer vision, pages 1501-1510, 2017.
[17] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnl: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[18] Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, and Shilci Wen. Dynamic instance normalization for arbitrary style transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4369-4376, 2020.
[19] Yongcheng Jing, Yang Liu, Yezhou Yang. Zunlei Feng, Yizhou Yu, Dacheng Tao, and Mingli Song. Stroke controllable fast style transfer with adaptive receptive fields. In Proceedings of the Europeon Conference Computer Vision (ECCV). September 2018.
[20] Prannay Khosla, Piotr Teterwak. Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola. Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661 2020.
[21] Jang-Hyun Kim, Jinuk Kim, Scong Joon Oh, Sangdoo Yun, Hwanjun Song. Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. arXiv preprint arXiv:2205. 14959, 2022.
[22] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[23] Alex Krizhevsky, 1lya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
11

---

[24] Sachyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. arXiv preprini arXiv:2202. 02220, 2022.
[25] Huihui Liu, Yiding Yang, and Xinchao Wang. Overcoming catastrophic forgetting in graph neural networks. In AAAI Conference on Artificial Intelligence, 2021.
[26] Songhua Liu, Jingwen Ye, Sucheng Ren, and Xinchao Wang. Dynast: Dynamic sparse transformer for exemplar-guidee image generation. In Proceedings of the European Conference on Computer Vision, 2022.
[27] Wojciech Masarczyk and Ivona Tautkute. Reducing catastrophic forgetting with learning synthetic data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). Workshop. 2020.
[28] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[29] Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kemel ridge-regression. XrXiv preprint arXiv:2011.000 1.00050, 2020.
[30] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jachoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34, 2021.
[31] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[32] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint 1rXiv:1804.02767, 2018.
[33] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[34] Sucheng Ren, Daquan Zhou, Shengfeng He, Jiashi Feng, and Xinchao Wang. Shunted self attentittnion via multi-scale token aggregation. In JEEE Conference Computer Vision and Pattern Recognition, 2022.
[35] Edgar Riha, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski, Kornia: an open source diffcrentiable computer vision library for pytorch. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3674-3683, 2020.
[36] Andrea Rosasco, Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, and Davide Bacciu. Distilled replay: Overcoming forgetting through synthetic samples. arXiv preprint arXiv:2103. 15851, 2021.
[37] Mattia Sangermano, Antonio Carta, Andrea Cossu, and Davide Bacciu. Sample condensation in online continual learning. In Proceedings of the International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2022.
[38] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint 7708.0089,099 2017.
[39] Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. IEEE transactions on pattern analysis and machine intelligence, (944)6600655,, 2017.
[40] Reza Shokri and Vitaly Shmatikov.
preaaccpreeerviigg deep learning. In Proceedings the 22nd ACM SIGSAC conference on computer and communications security, pages 1310-1321, 2015.
[41] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1oXi::1409.1535 2014.
[42] lla Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. In 202/ International Joint Conference on Neural Networks (ICCN), pages 1-8. IREE, 2021.
[43] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning, arXiv preprint 11v: 159, 2018.
[44] Tvor W Tsang, James T Kwok, Pak-Ming Cheung, and Nello Cristianini. Core vector machines: Fast svm training on very large data sets. Journal of Machine Learning Research, 6(4), 2005.
[45] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv e-prints, pages arXiv-1807, 2018
[46] Kai Wang, Bo Zhao, Xiangyu Peng. Zheng Zhu, Shuo Yang, Shuo Wang. Guan Huang. Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. arXiv preprint arXiv:2203.0/533 2022.
[47] Tongzhou Wang. Jun-Yan Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811. 2018
[48] Felix Wiewel and Bin Yang. Condensed composite memory continual learning. In Proceedings of the International Joint Conference Neural Networks (IJCNN, pages 1-8, 2021.
12

---

[49] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers, arXiv preprint arXiv:2105.15203, 2021.
[50] Xingyi Yang, Jingwen Ye, and Xinchao Wang. Factorizing knowledge in neural networks. In Proceedings of the European Conference Computer Vision, 2022.
[51] Yiding Yang. Zunlei Feng. Mingli Song. and Xinchao Wang. Factorizable graph convolutional networks. In Conference on Neural Information Processing Systems, 2020.
[52] Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[53] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages 12674-12685. PMI.R, 2021.
[54] Bo Zhao and Hakan Bilen.
Dataset condensation with distribution matching. arXiv preprint
arXiv: 2 10.0418), 2021.
[55] Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006. 05929, 2020.
[56] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. arXiv preprint arXiv:2206.000 00719, 2022.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please refer to the supplement.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] Please refer to the supplement.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please refer to Sec. 4.1.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We run all the experiments 5 times and report the mean and standard deviation of the performance.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please refer to Sec. 4.1.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "quiqq_LX7V1", "reviewer": "Reviewer_jG3P", "summary": "This paper proposes a novel algorithm for dataset condensation and introduces a dataset factorization approach - HABA which frame the factorization as a hallucinator-basis problem. This paper also introduces a pair of adversarial contrastive constrains to increase the diversity of generated images and inject more discriminant information into the factorization.", "questions": "See Weaknesses.", "limitations": "adequately addressed", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. This paper is well written and easy to follow, and the presentation is clear.\n\n2. The overall idea is novel and the experimental results are convincing.\n\n### Weaknesses\nI have to be honest that I'm not an expert in this area, but I have a few questions about this paper.\n\n1. Since the author has mentioned that the early dataset compression methods are inspired by the knowledge distillation, I'm wondering if these compressed data can be used for KD methods.\n\n2. Although this paper has performed the experiments across different architectures, however, I'm wondering about the effect of the model size on the final performance. (i.e. the performance gap between the compressed data with the whole data on ResNet18 will be greater or smaller than ResNet-101 ?)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WRkI5WaxMWF", "reviewer": "Reviewer_fSqe", "summary": "The paper investigates different ways to parametrize the synthetic learned/distilled dataset in the Dataset Distillation [1]. In particular, they propose to reparametrize the dataset as $\\{G_j(z_i)\\}_{ij}$, where $z_i$'s are \"basis\" (i.e., latents in usual generative modelling terminology), and $G_i$'s are the \"hallucinators\" (i.e., generators). (I took liberty to use variable $z$ rather than the $\\hat{x}$ in paper, for better clarity). \n\nThis reparametrization can be essentially applied with any Dataset Distillation method. In particular, the authors show that combining it with MTT (the SOTA Datasett Distillation work), can learn better distilled datasets **with same total number of parameters but with a much larger dataset size**. In fact, the performance gain mostly come from the increased dataset size (see below).\n\n\n[1] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018.", "questions": "1. **Why should we care about lower parameter count? Comparison with standard image compression techniques?**\n\n   Dataset Distillation (DD)'s scientific inquiry and practical applications both rely on the smaller distilled dataset size. Are there any applications that strongly relies on parameter counts? Indeed, smaller storage cost is always better, but I'm not sure if it matters that much when the distilled dataset often just contain tens or hundreds of images. Suppose that we care, then the authors should compare with regular compression techniques, including lossless and lossy ones, image-specific and generic ones.\n\n   As an example, a simple lossless GIF compression of [a publicly available image from the distilled dataset of CIFAR-10 trained using MTT](https://georgecazenavette.github.io/mtt-distillation/images/cifar10_zca_10/airplane_4.png) is just 1923 bytes (with meta info!), versus the raw 3072 bytes.  It should be easy to compress much further, and essentially get much smaller parameter count.\n\n2. **Given the importance of distilled dataset size in Dataset Distillation research, the paper should also compare methods at the same dataset size.**\n\n   E.g., the proposed method at **BPC(bases-per-class)=10** has **IPC(images-per-class)=50**, but is only compared with other methods run at **IPC=11**. In fact, when fixing the dataset size, the proposed method is often slightly inferior (e.g., BPC=10 vs IPC=50).  \n\n   The current comparison is done at the same parameter count, which is only meaningful if the emphasis on parameter count cannot be sufficiently justified (1st question). In either case, same dataset size (IPC) comparison should be done.\n\n3. **The stated motivation is not justified or verified.** \n\n   The motivation for the proposed bias-hallucinator parameterization  is that the original pixel parametrization don't learn relations among samples. (lines 29-33)\n\n   1. What exactly is \"relations\" referring to here? \n\n   2. The pixel parametrization is the most flexible and can represent any images. Admittedly, different parametrizations lead to different inductive biases / implicit regularizations. Is it correct to understand the claim as a hypothesis that the bias-hallucinator parameterization has better inductive bias to learn better distilled datasets due to sharing the same hallucinator across biases?  \n\n   If so, the hypothesis seems false from the experiment results (see 2nd question above).  In other words, the stated performance gain mostly come from the increased distilled dataset size, rather than from the proposed parametrization.\n\n3. **Motivating example in Section 1 is misleading.**\n\n   It is a comparison among **(1)** proposed parameterization **(2)** original pixel parametrization w/ same #parameters and thus much less \\#images **(3)** same as (2), but combining images from different checkpoints to form a larger distilled dataset.\n\n   (3) is a really bizarre and misleading choice because the original parametrization is perfectly capable of learning a larger distilled dataset, and (3) is not really reducing any \\#parameter or storage cost either. In fact, the later experiment results show that simply using original parametrization to distill a larger set have similar performance as (1). This makes the choice of (3) quite suspicious.\n\n3. **This is essentially generative modelling with a discrete set of latents. Why use image-shaped \"basis\" rather than generic vectors?**\n\n   It is not really wrong, but a somewhat weird choice given the common practice of using generic latent vectors. An explanation would be better to have.\n\n4. **Please emphasize somewhere early in the paper that the increased dataset size is needed to achieve good performance.**\n\n   A reader should not have to reach experiment section and to do the arithmetic to realize that the proposed technique comes at a cost.\n\n5. **Please reconsider the choice of using DC to refer to the task and writing the paper as if DC introduced the task (see first section).**\n\n6. The paper have some quite confusing places. Here are a few:\n\n\n     1. Fig. 3 mentions a $\\mathcal{L}_{cts.}$ that is not mentioned anywhere else.\n\n     3.  \" Constrain\" => \" Constraint\" in multiple places.", "limitations": "I do not think that the limitation is sufficiently discussed. As I emphasized above, the authors should make it clear that the claimed improvements come with the cost of much larger distilled dataset size, the one metric that Dataset Distillation research cares about the most.  Furthermore, the added generator components should increase both training time and the training GPU memory cost. The training time effect is not really discussed other than a sentence in the checklist, which is not part of the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I will first discuss what I believe are two important issues of the current paper, and then provide a list of strengths and weaknesses.\n\n### **Parameter count and dataset size**\n\nIt is important to note that\n+ The method essentially is a **compression** of the distilled dataset in terms of #parameters, but **not in** dataset size, the usual quantity of interest in Dataset Distillation. The authors unfortunately do not make this clear in most cases when claiming improvements, which in reality comes at a cost of larger distilled dataset size.\n+ It is unclear why #parameters should be a meaningful metric for Dataset Distillation. The paper does not provide arguments for it, or compare with any image compression methods.\n+ When comparing with same **distilled dataset size**, the proposed approach often obtains slightly inferior performance (than the original pixel parameterization). This runs directly in contrary of the authors' claim that optimizing pixels directly is difficult for learning relations between samples (lines 29-33,43-44,121-124,329-331).\n\n### **Improper credit assignment in writing**\n\nI believe that it is greatly damages the integrity and openness of the academic community to \n+ Refer to this task as Dataset Condensation (DC) [2] rather than Dataset Distillation (DD) [1] \n+ Write as if the DC paper [2] introduces this task (entire paper, esp. introduction and abstract; DD is only mentioned once in related work and as a row in results table), \n\nwhen\n+ Both DD [1] and DC [2] talk about the same task, with DD paper predating DC paper for **2 years**, DC paper citing DD paper and acknowledging that they investigate the same task,\n+ Neither DD paper nor DC paper explicitly gives a name to the task, but calls their proposed method DD or DC, and seems to refer to the task as DD or DC. \n\nIt is **extremely misleading** to people not familiar with this area, and essentially assigns credit in an obviously wrong way. I sincerely hope the authors did not do this purposefully. Admittedly, some other papers do the same thing, but it is no excuse to keep doing the harm. I strongly urge the authors to revise the paper in this aspect.\n\n### **List of Strengths and Weaknesses**\n\n**Strengths:**\n+ The authors show that, by reparametrization with deep generators in Dataset Distillation training, it is possible to further compress a distilled dataset into fewer parameters, without losing much performance. \n+ Therefore, with the same parameter count, the parametrization achieves better Dataset Distillation performance (at the cost of larger distilled datasets). If parameter count turns out a useful metric in future, this can potentially have better use cases, after validating its benefits against other image compression techniques.\n\n**Weaknesses:**\n+ ~The stated motivation for proposed reparametrization is that (original) pixel parametrization don't learn relations. This claim is not immediately clear why it is true, and is not verified. In fact, results in paper show that parametrization does not help when the dataset size is kept the same.~\n+ ~The motivating example in Section 1 is misleading.~\n+ The paper mentions reduced storage cost, but it is unclear why it should matter when the distilled dataset often only contains tens or hundreds of images.\n+ ~Writing is somewhat misleading in that whenever performance improvement is mentioned, the increased dataset size is almost never mentioned.~\n+ ~No comparison with standard image compression techniques.~\n+ Missing comparison with other DD parametrization work.\n\nPlease see my questions and suggestions below.\n\n[1] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018.\n\n[2] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929, 2020.\n\n---- \nThe author's comment addressed most of my concerns (striked through text above). I have adjusted my score accordingly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Bj8AZEglkxx", "reviewer": "Reviewer_9m6U", "summary": "This paper proposes a new parameterization (HaBa) of image data for dataset condensation. The proposed method decomposes a dataset into two components: data hallucination networks and bases, and considers the relationship between different condensed data points. Experiments show that HaBa achieves a better compression rate and better cross-architecture generalization performance.", "questions": "3.1 Basis and Hallucinator:\n- I am curious about the functionality of the encoder of the Hallucinators. Is it necessary? What will happen if you remove the encoder part and only train the decoder and corresponding style vector?\n- I wonder if it would be more parameter-efficient to learn multiple style vector pairs ( $\\sigma$ , $\\mu$ ) for a particular encoder and decoder pair.\n- I guess the weights can learn more information than the fixed basis. Why do you only use 1 Conv-ReLu block for Hallucinators? What will happen if you consider more layers in the encoder and decoder structure? Why not consider increasing the capacity of the hallucinator by expanding its width?\n\n3.2 Adversarial Contrastive Constrain\n- Why do you want a contrastive objective for the feature extractor and a cosine similarity for the hallucinator? Could you have just one objective (e.g., cosine objective) for both the feature extractor and the hallucinator (the feature extractor maximizes it, and the hallucinator minimizes it)?\n\n4.1 Datasets and Implementing Details\n- How long does the training take with the new parameterization compared to baselines? Is the optimization easier or harder with the new parameterization?\n- I can imagine that it is possible that some combination of hallucinator and basis may not be appropriate and introduce bias into the downstream classifier. For example, the dog's style may not be suitable for a ship. Suppose you train the downstream classifier on a ship instance with dog style. Would the classifier become confused at test time? What is your opinion on that?\n- Because the proposed methods have so many hyperparameters, I suggest adding a table summary of the hyperparameters used for each experiment in the appendix.\n\n4.3 Ablation Studies\n- Cross-Architecture Performance: Lack of explanation. Why does it improve the cross-architecture performance? What is the insight behind it? What design choice gives this nice property?\n- Cross-Domain Performance: Maybe change the title to \"robustness to corruption\"? I am unclear why \"diverse training data\" can give this nice property.\n- I would like to see some ablation studies on what will happen if we increase the hallucinator capacity (increase depth or width).\n\nMisc\n- Line 63: \"Such an adversarial scheme, in turn, enforces the hallucinators to produce diversified images and recover original data distribution thoroughly.\" The argument that \"recover original data distribution\" appears many times in the paper. I don't think dataset condensation methods aim to recover the data distribution. They are mining for informative examples instead. How come the condensed data recovers the original data distribution? Visually, the condensed data looks very different from the real data. What is your opinion on this? I would be more comfortable with an argument like \"capture the discriminative feature better.\"\n- Line 89: \"supervision signals from downstream tasks, e.g., cross-entropy loss for classification, are not rigorous enough for synthetic samples.\" What do you mean by \"rigorous enough\"? What do you consider a rigorous supervision signal?\n- Line 211: Typo. 10, 100, and 200 -> 10, 100, and 100\n- Figure 4 and Table 1 are far away from their text, which disrupts the reading flow.\n- Table 4 is confusing since it includes two different comparisons. Splitting into two tables or moving one to the appendix seems better.\n\nI would be happy to increase my score if my questions can be addressed appropriately.", "limitations": "The authors may want to discuss the limitations of their methods. Here are some of my incomplete thoughts.\n- There are so many hyperparameters in the proposed method that may impede its practical usage.\n- The parameterization only works for image data but may want to consider other modalities.\n- When scaling the method to many bases and hallucinators, there may be some optimization issues.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is well-motivated and focuses on an essential problem (the parameterization of the condensed data) for current dataset condensation methods. The proposed method is novel, and the results are encouraging.\n\n- Originality: Decomposing the condensed data to bases and hallucinators is novel and interesting. Related work focusing on the parameterization of the condensed data: https://arxiv.org/abs/2205.14959, https://arxiv.org/abs/2206.02916.\n- Quality: Most arguments are well-supported, and the experiments demonstrate the effectiveness of the proposed method. Some additional ablation studies and experiments are needed to make the paper more convincing.\n- Clarity: This paper is well-written and easy to follow, though the position of some figures and tables can disrupt the reading experience. Some training details can be added to the appendix.\n- Significance: The experiments demonstrate the effectiveness of the proposed method and can inspire some future work on the parameterization of the condensed data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept