Multisize Dataset Condensation
Authors: Yang He
Lingao Xiao
Joey Tianyi Zhou
Ivor Tsang
Venue: ICLR 2024 oral (2024)
Keywords: Dataset Condensation
Dataset Distillation
Image Classification
Primary area: general machine learning (i.e., none of the above)
TL;DR: Compress N condensation processes into one single condensation process to generate condensed datasets with various sizes.

Abstract: While dataset condensation effectively enhances training efficiency, its application in on-device scenarios brings unique challenges. 1) Due to the fluctuating computational resources of these devices, there's a demand for a flexible dataset size that diverges from a predefined size. 2) The limited computational power on devices often prevents additional condensation operations. These two challenges connect to the "subset degradation problem" in traditional dataset condensation: a subset from a larger condensed dataset is often unrepresentative compared to directly condensing the whole dataset to that smaller size. In this paper, we propose Multisize Dataset Condensation (MDC) by **compressing $N$ condensation processes into a single condensation process to obtain datasets with multiple sizes.** Specifically, we introduce an "adaptive subset loss" on top of the basic condensation loss to mitigate the "subset degradation problem". Our MDC method offers several benefits: 1) No additional condensation process is required; 2) reduced storage requirement by reusing condensed images. Experiments validate our findings on networks including ConvNet, ResNet and DenseNet, and datasets including SVHN,  CIFAR-10, CIFAR-100 and ImageNet. For example, we achieved 5.22%-6.40% average accuracy gains on condensing CIFAR-10 to ten images per class. Code is available at: [https://github.com/he-y/Multisize-Dataset-Condensation](https://github.com/he-y/Multisize-Dataset-Condensation).

Published as a conference paper at ICLR 2024
MULTISIZE DATASET CONDENSATION
1:2 Lingao Xiao!?? Joey Tianyi Zhou 122 Ivor W. Tsang 1,2,3 CCAR,, Agency for Science. Technology and Research, Singapore IHPC, Agency for Science, Technology and Research, Singapore School of Computer Science and Engineering, Nanyang Technological University Yang, Joey_Zhou, Ivor pror_Tsangl @cfar.aa.c...oo a-star. edu.sg.. sg
Yang Hel:2,
ABSTRACT
While dataset condensation effectively enhances training efficiency, its application in on-device scenarios brings unique challenges. 1) Due to the fluctuating compu- tational resources of these devices, there's s a demand for a flexible dataset size that diverges from predefined size. 2) The limited computational power devices often prevents additional condensation operations. These two challenges connect to the "subset degradation problem" in traditional dataset condensation: a subset from a larger condensed dataset is often unrepresentative compared to directly con- densing the whole dataset to that smaller size. In this paper, we propose Multisize Dataset Condensation (MDC) by compressing N condensation processes into single condensation process to obtaín datasets with multiple sizes. Specifi- cally, we introduce an "adaptive subset loss" on top of the basic condensation loss mitigate the "subset degradation problem". Our MDC method offers several benefits: 1) No additional condensation process is required; 2) reduced storage requirement by reusing condensed images. Experiments validate our findings on networks including ConvNet, ResNet and DenseNet, and datasets including SVHN, CIFAR-10, CIFAR- 100 and ImageNet. For example, we achieved 5.22% 6.40% average accuracy gains on condensing CIFAR- 10 to ten images per class. Code is available at: http:://gthbb.oommeem/Mbbbbbiiie Daciiiiizze-aaatat-CC-CCoe Connesssttnnnnnninnnnnnnn..
INTRODUCTION
With the explosive growth in data volume, dataset condensation has emerged as a crucial tool in deep learning, allowing models to train more efficiently by focusing on reduced set of informative data points. However, data processing faces challenges as more applications transition to on-device process-
Traditional Condensation
MDC Condensation
car oos
JFC
IPC
Dotaset
Full
Traditional
Ours
ing (Cai et al.) 2020: Lin al.] 2022, Yang et al.
Computation
2002
2023:
Qiu et al. 2022] Lee & Yoo 2 Dhar
Image Storage
1+2+_.+N
al. 2021 whether due security concerns, real-
Image
time demands, or connectivity issues. Such devices' inherently fluctuating computational resources require flexible dataset sizes, deviating from the conventional condensed datasets. However, this request for flexi- bility surfaces critical concern since the additional condensation process is unfeasible on these resource-restricte devices.
Figure 1: Condense datasets to multiple sizes requires N separate traditional condensation processes (left) but just single MDC pro- cesses (right).
Why not select subset from condensed dataset for on-device scenarios? We find the "subset degradation problem" in traditional dataset condensation: we select a subset from a condensed dataset, the performance of the subset is much lower than directly condensing the full dataset to the target small size. An intuitive solution would be to conduct the condensation process N times. However, since each process requires 200K epochs and these processes cumulatively generate 1 1 2 + N images (left figure of Fig. []. it is not practical for on-device applications.
"Corresponding Author.
Lingao Xiao is an intern student under the supervision of Yang He.

---

Published as a conference paper at ICLR 2024
To address these issues, present the Multisize Dataset Condensation (MDC) method to compress N condensation processes into just one condensation process, resulting in just one dataset (right figure of Fig. T- We propose the novel "adaptive subset loss" on top of the "base loss" in the condensation process to alleviate the "subset degradation problem" for all subsets. "Adaptive" means we adaptively select the Most Learnable Subset (MI.S) from N - 1 subsets for different condensation iterations. The "subset loss" refers to the loss computed from the chosen MLS, which then utilized update the corresponding subset.
How to select the Most Learnable Subset (MLS)? We integrate this selection process into the traditional condensation process with two loops: an outer loop for weight initialization and an inner loop for network training. MLS selection has three components. (i) Feature Distance Calculation: Evaluating distances between all subsets and the real dataset, where smaller distances suggest better representation. For each outer loop iteration, we calculate the average Feature Distance over all the inner training epochs. (ii) Feature Distance Comparison: We compare the average feature distances at two outer loops. A large rate of change in distances denotes the current subset has high learning potential and should be treated as MLS. (iii) MLS Freezing Judgement: To further mitigate the "subset degradation problem", our updating strategy depends the MLS' size relative to its predecessor. If the recent MLS exceeds its predecessor in size, we freeze the older MLS and only update its non-overlapping elements in the newer MLS. Otherwise, update the entire newer ML.S.
The key contributions of our work are: 1) To the best of our knowledge, it's the first work to condense the N condensation processes into single condensation process. 2) We firstly point out the "subset degradation problem" and propose "adaptive subset loss" to mitigate the problem. 3) Our method is validated with extensive experiments on networks including ConvNet, ResNet and DenseNet, and datasets including SVHN, CIFAR-10, CIFAR-100 and ImageNet.
2 RELATED WORKS
Matching Objectives. The concept of dataset condensation, or distillation, is brought up by Wang al. (018). The aim is to learn synthetic dataset that is equally effective but much smal ler in size. 1) Gradient Matching (Zhao et al.
2021 Jiang et al., 20222] Lee et al., 2022b Loo et al. 2023
methods propose to match the network gradients computed by the real dataset and the synthetic
dataset. 2) Other matching objectives include performance matching (Wang et al.
2018 Nguyen
ct al. 2021a]6] 66. /Zhou et al. |022] Loo et al. 2022), distribution or feature matching Zhao & Bilen
2023 Wang et al.) 2022 Zhao al. 2202), trajectory matching (Cazenavette et al. 2022 Du et al.
2023 Cui et al. (202)), representative matching (Liu et al. 20236 Tukan et al. (2023) ) loss-curvature
matching (Shin et al. (002), and BN matching (Yin et al. 2023] Yin & Shen 2023) However, all the
aforementioned methods suffer from the "subset degradation problem," failing to provide a solution.
Better Optimization. Various methods proposed to improve the condensation process, including data augmentation (Zhao & Bilen (021), data parameterization (Deng & Russakovsky [0022] Liu
al. 2022 Kim et | Nooralinejad et al.) 2022 Kim al. [002aa] (Sun al. 2023), model
augmentation (Zhang et al. (023b). and model pruning (Li et al.
2023)
Our method can combine
with these methods to achieve better performance.
Lee et al. 2022a Cazenavette al. 2023) leverage Generative Adversarial Networks (GANs) to enhance the condensation process. For instance, Wang et al. (2023) generate images by feeding noise into a trained generator, whereas Zhang et al. (2023a) employ learned codebooks for synthesis. The drawback that they require substantially more storage to save the model and demand 20% more computational power during deployment (Wang et al. 2023). In contrast, our solution delivers immediately usable condensed images, ensuring efficiency in both storage and computation.
Condensation with GANs. Several works (Zhao & Bilen 2022
Comparison with Slimmable Dataset Condensation (SDC; Liu et al. (2023a)) SDC aims to extract a smaller synthetic dataset given the previous condensation results. The differences include: 1) SDC needs two separate condensation processes, while our method just needs one; 2) SDC relies on the condensed dataset, but method does not; 3) SDC requires campuuationallintnsiii singular value decomposition (SVD) on condensed images, while our condensed images can be directly used for application.
2

---

Published as a conference paper at ICLR 2024
Dataset
Synthetic Dataset
Symbol
Condense
Storage
cinl (1, 2, N)
N 1+ +
Condensation process
Images indices Which condensation
(1,
C (1,2,
N)
N N
process?
IPC,
Images Per Class for on-devices scenarios
C|M]
N
Baseline-A A: c
Baseline-B: O%
Baseline-C: M
Ours
c|M
(b) Resources required for different baselines.
[m] represents the set {1, (1,2,....?) n).
IPG
60
DP
IPC
55
IPC
IPG:
50
IPC,
1x3
x3
35-
cendersaioes ingge
(a) "Multi-size condensation" with three baselines obtain condensed datasets size 1.2.3. Left: Baseline conducts 3 separate condensation processes and stores 1++2 + 3 images. Middle: Baseline B performs 3 times IPC; condensa- tion with different image indices as initialization and in total stores images. Right: Baseline condenses once stores 3 images. Multiple sizes are achieved with subset selection.
30
condorestice inaees
images
2
10
Images Per Class (PC)
(c) On CIFAR-10, the accuracy of three pro- posed baselines for "multi-size condensation" to get IPC size from to 10.
Figure 2: Three different baselines for multi-size condensation.
METHOD
3.1 PRELIMINARIES
Given a big original dataset B E Rxdx with M number of d-dimensional data, the objective is to obtain a small synthetic dataset E IR where & M. Leveraging the gradient- technique (Zhao et al. 202T, minimize the gradient distance between big dataset B and synthetic dataset S:
((Vol(S:0) VoE(B:)) = D(S,B:), SERNxa
min
(1)
where the function D(-) defined as a distance metric such as MSE, represents the model parameters, and Vol(.) denotes the gradient, utilizing either the big dataset B or its synthetic version S. During condensation, the synthetic dataset S and model 0 are updated alternatively,
S S (S,
t- 7 Vol (0;S),
where À and y are learning rates designated for S and 0, respectively.
3.2 SUBSET DEGRADATION PROBLEM
To explain the "subset degradation problem", we name the condensation process Eq. and Eq. 2 as "basic condensation". This can be symbolized by c.M] where [N] = {1, 2, ......N, The subscript of CB indicates the index of the condensation process, while the superscript of CH represents the index of original images that are used as the initialization for condensation.
For on-device applications, we need condensed datasets with multiple sizes, namely, "multi-size condensation". Inspired by He al. (2024); Yin al. (2023). introduce three distinct baselines for "multi-size condensation": : Baseline-A A, Baseline-B B, and Baseline-C. denoted as Clrl CH and c[M], respectively. Fig. 2a illustrates how to conduct "multi-size condensation" to obtain the con- densed dataset with sizes I. 2, and with our proposed baselines. Bascline-A employs three basic condensations of varying sizes, yielding six images. Baseline-B uses three size-1 basic condensations
3

---

Published as a conference paper at ICLR 2024
Image Forward
Synhele
(i) Feature Distance Calculatiin
Image Ujpdale (fardowan))
Gradent Matshhing
Real
Feature Distance
Iitl Lecl....)
by (SQ
Redf
Sprebatk
(Hi) Freesing Judgement Sea !!l Sg C Rail 20
Imager
Rec
Se
[baae losa subsut loan)
Syntheti
f
$
Sua
Updata images one
Updata Images losses
Feature Distance Comparison
bose
Basic Condensation
Our MDC
Max
(bose loue wulhent iona)
(a) Illustration of the "basic condensation" and our MDC method. The black box is the outer loop, and the orange box the inner loop. The modifications made by our MDC are marked with blue.
arg max
3 base
(b) Illustration of MLS selection process.
Figure 3: Explanation of Our MDC.
but varies by image index, resulting in three unique images. Baseline-C adopts single basic conden- sation to get three images, subsequently selecting image subsets for flexibility. Fig. 2b).:2eenet the count of required basic condensation processes and the storage demands in terms of image numbers.
In Fig. 2c we highlight the "subset degradation problem" using Baseline-A's orange line and Baseline-C C's black dashed line. Baseline-A requires ten condensation processes, while Baseline-C just condenses once and selects subsets of the condensed dataset sized at 10. The shaded orange region indicates a notable accuracy drop for the subset when compared to the basic-condensed dataset. critical observation is that the accuracy discrepancy grows as the subset's ' size becomes smaller.
3.3
MULTISIZE DATASET CONDENSATION
3.3.1 SUBSET Loss COMPRESS CONDENSATION PROCESSES
To address the "subset degradation problem". the objective function now becomes:
min D oV (uj.Spp ... 0) v (B:
(3)
SER of
where Spx -1.2...... C = S(N] represents n& subset of the synthetic dataset S E T Nxd We want each subset Spx] to have small distance from the big dataset B. S(N] contributes the "base loss", and S1...21.....IIN -1 contribute to the "subset loss". Note that the subsets also have relationship with each other. For instance, the 2nd subset Stzj = S(1.2} is also a subset of the 4 subset St4] = S(1.2.3.4).
We aim to incorporate the information of subsets without requiring additional condensation processes or extra images. To achieve this, we need to compress the information from the N - different condensation processes of Baseline-A, including c| .c!? I (N-1) into the process [! We
C
propose the "subset loss" on top of the "base loss" to achieve this purpose in a single condensation
[[N] while the "subset
process. The "base loss" is used to maintain the basic condensation process CN
loss" is used to enhance the learning process of the subsets via _ c| ...... N-1 new updating strategy:
-NN We have
(4) where S SIN represents the condensed dataset of N images and associated with the "base loss". S|m] is subset and contributes to the "subset loss". A comparison between Eq. and basic condensation is shown in Fig, 3a As depicted in Fig. 2bb our technique aligns with Baseline-C: in terms of the counts of both condensation processes and images.
S+s
B:0), n RE - 1].
3.3.2 SELECTING MLS FOR ADAPTIVE SUBSET LOsS
Among the N - 1 subsets from {Si, Sy...... - we identify particularly representative subset, S[n*], where n E [1, - 1]. We term this the Most Learnable Subset (MLS). In each condensation iteration, the ML.S is selected adaptively fit that particular iteration. Our approach
4

---

Published as a conference paper at ICLR 2024
relies on three components to determine the MLS. Each component is illustrated in Fig. 3b The algorithm of the proposed method is shown in Appendix
the gradient distance between subsets (i)). Eq. represents the traditional approach for computing and the big dataset B, This method requires gradient calculations to be performed N - times across the N 1 subsets, leading to considerable computational overhead. To alleviate this, we introduce the concept of "feature distance" substitute for the "gradient distance" to reduce computation while capturing essential characteristics among subsets. The feature distance at a specific condensation iteration i for subset (n) can be represented as:
Feature Distance Calculation (Fig. (b-()))
F. ()) = D (Je (5)) Jc(B))
(5)
where fr(-) is the feature extraction function for teh. condensation iteration, and D(-) is a distance metric like MSE. For subsets, the gradient distance mandates N - forward passes and an equal number of backward passes for total of N subsets. In contrast, the feature distance requires only a single forward pass and no backward pass. This is because the features are hierarchically arranged, and the feature set derived from subset of size n can be straightforwardly extracted from the features of the larger dataset of size N.
Feature Distance Comparison (Fig. (i)). Generally, as the size of the subset increases, the feature distance diminishes. This is because the larger subset is more similar to the big dataset B. Let's s consider two subsets Stp) and Sty) such that < < q < N. This implies that the size of subset Spp) is less than the size of subset S(a)- Their feature distances at iteration & can be represented as:
Fy (S(p): B)) F; () if
(6)
Initially, is intuitive that Siu,, being the smallest subset, would manifest the greatest distance or disparity when compared to B. As such, Stn should be the ML. at the beginning of the condensation process. As the condensation process progresses, we have:
F At (Spp) B) > F. ())
F. P. (Stg) B) F. (S B);
where t At and t are two different time points for the condensation process. The reason for Fi-Ae >> Fj is that the subsets get more representative the condensation progresses, causing their feature distances to shrink. So, the most learnable subset would be the one whose feature distance reduction rate is the highest. The feature distance reduction rate is:
AFS Fi (5m,), B) FI-A (5)m). B)]
(n): 1)
(8)
At where R(S(m). t) represents the rate of change of feature distance for subset S(n) at the time point t, and Ass, denotes the change in feature distance of subset Sin) from time At to t. An example for feature distance calculation can be found in Appendix B.2 The MLS for the time t can be described as:
At
StLs(t) = [1 arg max (R (S) where n [1,N - 1].
S(m)
Eq. 9 seeks the subset that has the steepest incline or decline in its feature distance from B over the time interval At. This indicates the subset is "learning" at the fastest rate, thus deeming it the most learnable subset (MLS).
MLS Freezing Judgement (Fig. 3bl (iii)). To further reduce the impact of the "subset degradation
problem", we modify the updating strategy in Eq. The judgement will be modified if the current MLS differs in size from its predecessor; otherwise, it remains unchanged:
Upddtte S
if SMIS (1) C SMI (1 - AA)
Using Eq. to
(10)
Upddate S SMIL(t- At) if SMLS (f) 3 St At)
where \\ is the symbol for set minus. If the size of the current MI.S SMLS (t) is smaller than its predecessor SMLs(t - At). we update the entire synthetic data S with Eq. 4 However, when the size of the current MLS is larger than its predecessor, updating the entire S would cause the optimized predecessor to be negatively affected by new gradients. Therefore, we freeze the preceding MLS StLs(t (t- At) to preserve already learned information. as shown the red shadowed S in Fig. (ii). As result, only the non-overlapping elements, i.e., S SMLLS(t At) updated.
5

---

Published as a conference paper at ICLR 2024
Dataset
5
8
9
Avg.
Dif:
A
68.50
15.27
79.55
81.85
83.33
84.53
85.66
86.05
86.25
87.501
81.85
SVHN
68.501
71.65
71.27
71.92
73.28
70.74
71.83
71.08
71.97
71.55
35.48
67.97
74.38
87.501
70.65
Ours
63.26
67.91
72.15
74.09
77.54
78.17
80,92
82.82
84.27
86.38
76.75
+6.10
50.80
61.84
65.53
66.33
67.50
CIFAR-10
B
50.80
53,17
55.09
56.17
55.80
56,98
57.60
57.78
58.22
58,38
56,00
C
27.49
38,50
45.29
50.85
53,60
57.98
60,99
63.60
65.71
67.501
53.15
Ours
49.66
54.55
58.18
61.51
63.36
66.72
A
28.90!
34,28
37.35
39.13
41.15
42.65
43.62
44,48
45.07
45,40
40.20
CIFAR-100 100
B
28.90
30.63
31.64
31.76
32.61
32.85
33.03
33.04
33.32
33.39
32.12
C
14.38
21.76
28.01
32.21
35.27
39.09
40.92
42.69
44.28
45,40
34.40
Ours
27.58
31.83
33.59
35.42
36.93
38.95
40.70
42.05
43.86
44.34
37.53
+3.13
(a) Results of SVHIN, CIFAR-10, CIFAR-100 targeting 1PC
Dalaset
1
3
4
7
10
20
30
40
50
Avg.
Difl.
68.55
79.55
84.53
86.25
87.50
89.54
91.09
91.38
SVHN
C 34.90
46.52
52.23
56
30
62.25
65.
68.84
69.57
71
74.69
83.73
87.83
89. 73
91.38
68.23
Ours
58.77
67.72
69.33
72.26
75.02
73.71
74.50
74.63
76.21
76.87
83.67
87.08
89.
39.46
91.39
76.47 +8.24
A 50.80
54.85
59.79
61.84
62.49
64.59
65.53
66.33
66.8
67.50"
70.82
72.86
74.30
75.07
65.26
CIFAR-10 10
C
27.87
35.69
41.93
45.29
47.54
51.96
53.51
55.59
56.62
58.26
66.77
70.50
72.98
74.50
54.21
Ours
47.85
52.18
56.29
58.52
58.75
60.67
61.90
62.74
62.32
62.64
88
70.02
72.91
74.56
62.01
+7.80
A
28.90
34.28
37.35
39.13
41.15
42.6
43.62
44.48
45.07
45.40
49.55
50
52, 54
53.47
43.56
100
C
12.66
18.35
23.76
26.92
29.12
32.23
34.21
35.71
37.18
38.25
45.67
49.60
52.36
53.47
34.96
Ours
26.34
29.71
31.74
32.95
34.49
36.36
38.49
39.59
40.43
41.35
46.06
49.40
51.72
53.67
39.45
+4.49
(b) Results of SVHN, CIFAR-IO, CIFAR- targeting sPC
Avg.
Diff.
A
60.40
63.87
67.40
68
71.33
70.60
70.47
71.93
72.87
72,80
75.50
76.601
70.21
ImageNet-10
60.40
62.07
62.80
63.40
64.67
63.13
62.67
63.60
64.13
63.60
62.73
64.13
63.11
44,00
57.27
62.80
66.13
64.33
69.47
69.53
70.53
71.33
73.00
74.47
75.73
66.58
Ours
55.87
61.60
63.40
64.40
63.80
67.73
67.13
70.07
71.07
71.13
76.00
79.20
67.62
+1.04
Results of ImageNet-10 10 targeting IPC20-
Table 1: Comparisons with three different baselines built with the IDC (Kim et al.
2022b
denotes
directly cited from original papers. Numbers with standard deviation can be found in Appendix B.4
EXPERIMENTS
4.1 EXPERIMENT SETTINGS.
Terms. IPC,, represents n Images Per Class for the condensed dataset.
Basic Condensation Training. We use IDC (Kim et al., 2022b to condense the CIFAR-IO, 10, CIFAR-
100 (Krizhessyy al.) 2009) and SVHN (Netzer et al. 2011 with ConvNet-D3 (Gidaris & Ko- modakis 2018). Ne (Deng et al. 2009) is condensed via ResNet10-AP (He et al. (0166). For CIFAR-I0, 10, CIFAR- 100 and SVHN, we use a batch size 128 for IPC < 30, and a batch size of 256 for IPC > 30. For ImageNet- 10 IPC20. we use batch size of 256. The network is randomly initialized 2000 times for CIFAR- 10, CIFAR- and SVHN, and 500 times for ImageNet- 10; for each initialization, the network is trained for 100 epochs. More details are provided in Appendix [.]
Basic Condensation Evaluations. We also follow IDC (Kim et al. 2022b). For both CotvNet-D3 and ResNet10-AT the learning rate is 0.01 with 0.9 momentum and 0.0005 weight decay. The SGD optimizer and a multi-step learning rate scheduler are used. The network is trained for 1000 epochs.
MDC Settings. i) Feature Distance Calculation. The last layer feature is used for the feature distance calculation. The computed feature distance is averaged across 100 inner loop training epochs for specific outer loop. ii) Feature Distance Comparison. For CIFAR-10, CIFAR-100 and

---

Published as a conference paper at ICLR 2024
DC
MTT
DREAM
DC
DSA
MTT
IDC
DREAM
Ours
15.35
16.76
18.80
27.49
32.52
49.66
16.32
12.50
15.13
27.87
27.57
47.83
19:75
21:22
24.90
38.50
39:57
54.58
2
18.77
15.19
23.92
35.69
36.57
52.18
22.54
26.78
31,90
45.29
48.21
53.92
3
21.24
19.69
26.53
41.93
43,50
56.29
26.28
30.18 18
38.10
50.85
53.84
54.55
4
21,42
22.02
30.30
45.29
47.35
58.52
30.37
33.43
43.20
53.60
55.25
55.18
23.32
23.28
32.71
47.54
49.81
58.75
33.99
38.15
49.20
57.98
60.46
58.80
6
23.63
24.79
35.54
51.96
53.38
60.67
36.36
41.18
51.60
60.99
63.27
61.51
7
25.35
25.62
34.12
53.51
54.58
61.90
39.83
45.37
56.30
63,60
8
27:40
27.84
40.60
55.59
56.78
62.74
42.68
49.21
58.50
65,71
67.40
65.41
29.57
56.62
58.91
62.32
10
44.90"
52.10'
62.80
67.50
69.40
66.72
10
28.00
32.51
45.99
58.26
60.10
62.64
Avg.
31.21
43.53
20
36.53
40.94
60.41
66.77
68.07
66.88
Diff.
-27.16
-22.93
-14.84
-5.22
-2.87
30
42.82
48.05
67.68
70.50
70.48
70.02
54.24
72.98
72.79
72.91
(a) CIFAR-10. 10, IPC
50
53.90
60.60
71.60
74.507
74.80
74.56
-
2
10
20
Avg.
Avg.
29.68
31,20
42.69
54.21
55.33
62.01
1.FS
(1,20)
20.34
23.69
28.58
35.39
42.47
30.09
Diff.
-32.33
-30.81
-19.32
-7.80
-6.68
LBS
(20.210) 210)
26.04
29.27
33.49
36.23
42.47
33.50
Ours
(1.20) 20)
27.66
31.09
35.50
41.56
49.30
37.02
(b) CIFAR-10, IPCso
(c) Comparing results with LFS and LBS 2023a CIFAR-100, IPC2o.
(Liu ct al.
Table 2: Comparison with SOTA condensation methods. f denotes directly cited from original papers.
SVHN, the feature distance is calculated at intervals of every At = 100 outer loop. For ImageNet-Ile 10, At 50. iii) MIL Freezing Judgement. We follow Eq. 10 for MLS freezing.
4.2 PRIMARY RESULTS
Comparison with Baseline-A, B, C. Three baselines defined in Sec. 3.2 including Baseline-A,B,C. are created with IDC (Kim et al. 20226). Tab. and Tab. Tb provide the comparisons on three datasets: SVHN CIFAR-10, and CIFAR- 100 targeting IPC and IPC50: Tab. prrovides the results on the ImageNet-10 dataset of IPC2.. As detailed in Fig. 2b] Baseline-C aligns with our condensation and storage requirements, so we mainly compare with Baseline-C. The results of Baseline-C C are shaded in grey, while our method's accuracy is shaded in blue. Evidently, our approach consistently outperforms Baseline-C. For instance, on CIFAR- targeting IPC our method improves 5.22% in average accuracy. The proposed method effectively addresses the "subset degradation problem" at small subsets. For Snjj of IPC we improve accuracy by +27.78% on SVHN, +22.17% on CIFAR-10, and -13.20% CIFAR-100. Even though Baseline-B requires much more image storage (N v.s. 1), our method beats Baseline-B for iPC by +5.37% on SVHN, +2.37% on CIFAR-10, and +5.41% CIFAR- 100. The visualization of accuracies is presented in Fig. 2c
Comparison with State-of-the-art Methods. In Tab, 2 we evaluate our approach against state-of- the-art (SOTA) condensation techniques including DC Zhao et al. 2021), DSA (Zhao & Bilen 2021). ) MTT (Cazenavette et al. 2022). IDC (Kim et al., (022b)). and DREAM (Liu et al. 2023b)) From the table, it becomes evident that not only IDC (Kim al. 2022b) but all condensation methods face the "subset degradation problem"'. Our MDC shows a clear advantage over other methods with a single condensation process and IPCy storage. Importantly, the accuracy of S is improved by 17.14% for IPC10 and 20.26% for IPC50 compared to DREAM (Liu et al. 2023b). Tab. 2 illlustaaees that our approach outperforms Slimmable DC (Liu et al. 2023a), another method for dataset flexibility. Notably, our method excels by 3.52% even against the resource- inturce-intensvve LBS.
R (1, 20) means requiring one condensation process and storing 20 images.
7

---

Published as a conference paper at ICLR 2024
Calculate
Compare
Freeze
2
4
10
27.49
45.29
53.60
60.99
65.71
53.15
49.35
48.27
50.00
52.30
54,20
58.29
60.90
63.63
65,90
67.63
57.08
40.12
54.91
56.02
56.12
56.18
59.74
61.68
63.41
65.56
67.01
58.08
49.66
54.58
53,92
54.55
55.18
58.80
61.51
63.36
65.41
66.72
58.37
Table 3: Ablation study of the proposed components. Calculate denotes whether compute the feature distance and include subset loss. Compare denotes whether to compare the feature distance. Freeze means whether to freeze preceding SMLS. CIFAR-IO, IPC
6
Avg.
Diff.
DenaeNer RexNet
B
41.2
41.6
47.0
49.5
52.5
51.4
54.8
57.9
53.3
52.8
59.0
61.0
54.5
54.4
55.5
62.1
55.8
51.5
564
Awee Asuna
550
requree
25.5
34.0
39.1
45.9
53.0
54.6
57.3
59.7
61.7
63.7
49.5
39.3
48.6
52.7
55.3
57.2
61.0
62.7
53.8
MDC he
39.8
51.5
58.5
59.1
60.7
55.3
rsedt ante
B
41.0
48.1
48.3
52.3
54.8
53.0
55.3
53.5
54,1
55.3
51.6
26.4
35.4
40.3
47.3
53.1
54.6
57.9
59.0
60.6
62.5
49.7
38,5
49.6
53.4
563
57.5
58.4
59.7
53.6
+3.9
(1ra)
Table 4: Cross-architecture performance of the proposed method. CIFAR-10, IPC
Figure 4: Accuracy train- ing time. CIFAR-10. IPC
4.3 MORE ANALYSIS
Ablation Study. Tab. Bl provides the ablation study of the components for MLS selection. The first row is exactly the Baseline-C C, which condenses once but does not include the subset loss. We can clearly observe the "subset degradation problem" this case. Row 2 includes the subset loss but does not consider "rate of change" for feature distance. In such a case, we find SMIs==[1] for all outer loops. We observe large improvement for Sty-- However, the accuracy Sig) does not exceed Snj even though the dataset size is larger. Row 3 does not consider the freezing strategy. It improves the average accuracy from 57.08% to 58.08% but makes the accuracy of S/1 drop from 49.35% to 40.12% Row 4 is the proposed method's complete version, enjoying all components' benefits.
Performance on Different Architectures. Tab, shows that the "subset degradation problem" not unique to the ConvNet (Gidaris & Komodakis 2018) model but also exists when using ResNet (He et al. 2016) and DenseNet (Huang et al.) 2017) Not surprisingly, the proposed method generalizes other models with improvement of +4.3% and +3.9% the average accuracy for ResNet and DenseNet compared to Baseline-C, respectvelly
Performance on Different Condensation Method. Apart from IDC, our MDC method can be applied to other basic condensation methods such as DREAM (Liu et al.) 2023b). The average accuracy increases from 58.37% 60.19%. Appendix B.3 shows the detailed accuracy numbers.
Reduced Training Time Needed. As shown in Fig. the introduction of "Adaptive Subset Loss" increases our total training time from 11.5 hours (IDC) to 15.1 hours. However, our MDC method does not need such long training time. As depicted by the red vertical line, our MDC method only needs 0.6 hours to match IDC's average accuracy. In other words, we reduce the training time by 94.8% for the same performance. This might be because our "adaptive subset loss" provides extra supervision for the learning process. More details can be found in Appendix [B.5
Subset Evaluation Metrics. To evaluate the subsets, we have more metrics apart from the feature distance, which is the first component Sec. 3.3.2] As shown in Tab. we can also consider gradient distances evaluate subsets based model accuracy when trained them. Our "Feature Distance" metric just needs two forward processes to calculate the feature of the subset SIN- and real images. Compared to "Feature Distance". "Gradient Distance" demands more computational processes (ten forward and ten backward processes) but yields a similar accuracy. While using "Accuracy Difference" outperforms "Feature Distance", it is computationally intensive and impractical. Here, the required number of forward process F and backward process B is E x (N 1) 1000 x 9 9, 000.
Visualization of MLS.. Selection of SMLS shown in Fig. See Appendix B.6 for class-wiss wise ML.S.
8

---

Published as a conference paper at ICLR 2024
Avg. Acc. of the run: 59.55%
airplane
4
Most Leamable Subset (MLL5
car
Frozen Subset
bird
Selected Subset
3
cat
deer
2-
dog
frog
1-
horse
ship
0
truck
12155411151 16
Selection Interval AI
(a)
(b) IDC
(c) MDC
Figure 5: MLS and frozen subsets
Figure (a) IDC condensed IPC1; (b) IDC condensed IPC
visualization. CIFAR-IN 10, IPC
(c) our MDC condensed 1PC
Evaluation
Avg.
Acc.
Dift.
Metric
+5.22
Feature Distance
49.66 54.58 53.92
54.55
55.18
58.80
61.51 63.36
65.41 66.72
58.37
2.
58.49
+5.34
58.73
+5.58
Runs
Gradient
Distance
49.20 53.64
56.48
56.37
55.82 59.53
61.05 63.31 65.00
66.90
58.73
N
N
58.90
59.20
+5.75
+6.05
Difference
Accuracy
48.18
52.66 $7.10
58.62
59.75
62.11 63.17 63.99 65.48 66.57 59.76
Ex(N-1) Ex(N-1)
Avg. 58.87
59.55
46.40
45.72
Table Different subset evaluation metrics for MLS. and B denote the number of Table 6: Effects forward and backward propagation, respectively. E and N denote the number of of condensation training epochs (i.e., 1000) and target IPC (i.e., 10), respectively. CIFAR-10, IPC runs.
Effects of Condensation Runs. The variation in our results arises from two aspects: the condensation process and the model training using the condensed dataset. Tab. shows that different condensation runs lead to improvements ranging from 5.22% to 6.40% over Baseline-C 15).15%) This variation is more substantial than the variation in model training (see Appendix B.4 Therefore, we report lower result (58.37%) Tab. too illustrate the effectiveness of our method conservatively.
Visualization of Condensed Dataset. As shown in Fig. 6 we visualize and compare the dataset condensed with IDC (Kim et al. 2022b) and our MDC. We utilize three horse images from different settings to articulate our findings. The horse image in IDC condensed IPC, is highlighted with yellow border, in the IDC condensed IPC10 with red border, and in our MDC condensed IPC; with green border. For clarity, we 'll refer to these as hor Storange horsered and horsegreen- 1) Upon comparing serrgag with horsered, it's evident that horseorange exhibits more distortion than horsered save other images' information. 2) Furthermore, when aligning hor horsGoragge with actual images (see Fig. 10 in Appendix C fhorseorange is almost the same as the real counterpart image, suggesting it doesn't include information from other images. This highlights the "subset degradation problem" when a small subset, such as horsered lacks guidance during condensation, it fails to adequately represent the complete dataset. 3) Upon evaluating horseorange against horse, sEgeenn we can observe that images condensed using our MDC approach display more pronounced distortion than those from IDC IPC, This increased distortion in hor hegeen arises because it also serves as subset for IPC₂, IPCs, ... IPCN- This increased distortion demonstrates our method effectively addresses the "subsct degradation problem". More visualization can be found in Appendix C
CONCLUSION AND FUTURE WORK
To achieve multisize dataset condensation, our MDC method is the first to compress multiple condensation processes into a single condensation process. We adaptively select the most learnable subset (MLS) to build "adaptive subset loss" to mitigate the "subset degradation problem". Extensive experiments show that our method achieves state-of-the-art performance on the various models and datasets. Future works can include three directions. First, our subset loss has an impact on the accuracy of the full synthetic dataset, so we plan to find way to maintain the accuracy better. Second, we aim to explore why our MDC learns much faster than previous methods. Third, better subset selection metrics are worth investigating.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
This work was supported in part by S*STAR Career Development Fund (CDF) under C233312004, in part by the National Research Foundation, Singapore, and the Maritime and Port Authority of Singapore Singapore Maritime Institute under the Maritime Transformation Programme (Maritime Al Research Programme Grant number 111-22222-MPP000
REFERENCES
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. Proc. Adv. Neural Inform. Process. Syst., 33: 1285-11199,, 2020.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2022.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jan Zhu. Gen- eralizing dataset distillation via deep generative prior. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pp. 3739-3748, 2023.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation imagenet- with constant memory. In Proc. Int. Conf. Mach. Learn., pp. 6565 6590. PMLR, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In Proc. IEEE Conf. Comput. Vis. Pattern Recog. pp. 248-255, 2009.
Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. In Proc. Adv. Neural Inform. Process. Syst., 2022.
Sauptik Dhar, Junyao Guo, Jiayi Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah. A survey of on-device machine learning: An algorithms and learning theory perspective. ACM Transactions on Internet of Things, 2(3):1 (03):11991110 2021.
Jiawei Du, Yidi Jiang, Vincent TF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accu- mulated trajectory error to improve dataset distillation. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2023.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pp. 4367-4375, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pp. 770-778, 2016.
Yang He, Lingao Xiao, and Joey Tianyi Zhou. You only condense once: Two rules for pruning condensed datasets. Proc. Adv. Neural Inform. Process. Syst., 2024.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proc. IEEE Conf. Comput. Vis. Pattern Recog, pp. 4700-4708, 2017.
Zixuan Jiang, Jiaqi Gu, Mingjie Liu, and David Z. Pan. Delving into effective gradient matching for dataset condensation. arXiv preprint wxXiv:2088.00//1 2022.
Balhae Kim, Jungwon Choi, Seanie Lee, Yoonho Lee, Jung-Wor Ha, and Juho Lee. On divergence measures for bayesian pseudocorescts. In Proc. Adv. Neural Inform. Process. Syst., volume 35, pp. 757-767, 2022a.
lang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Wes Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameteriza- tion. In Proc. Int. Conf. Mach. Learn., 2022b. UtL/htsss://gge http::::////ttt.b...../ com/ saro/ssu--ae mm ab/ SeFFiiiettDDtassee
Condensation
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
10

---

Published as a conference paper at ICLR 2024
Hae Beom Lee. Dong Bok Lee, and Sung Ju Hwang. Dataset condensation with latent space knowledge factorization and sharing. arXiv preprint arXiv:2208. 10494, 2022a.
Jinsu Lee and Hoi-Jun Yoo. An overview of energy-efficient hardware accelerators for on-device deep-neural-netwonn training. IEEE Open Journal of the Solid-State Circuits Society, 1:115-128. 2021.
Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensa- tion with contrastive signals. In Proc. Int. Conf. Mach. Learn., pp. 12352-12364, 2022b.
Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Dataset distillation using parameter pruning. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 2023.
Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, and Song Han. On-device training under 256kb memory. Proc. Adv. Neural Inform. Process. Syst., 35:22941-22954, 2022.
Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In Proc. Adv. Neural Inform. Process. Syst., 2022.
Songhua Liu, Jingwen Ye, Runpeng Yu, and Xinchao Wang. Slimmable dataset condensation. In Proc. IEEE Conf. Comput. Vis. Pattern Recog,, pp. 3759-3768, 2023a.
Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. DREAM: Efficient dataset distillation by representative matching. arXiv preprint arXiv:2302. /44 16, 2023b.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. In Proc. Adv. Neural Inform. Process. Syst. 2022.
Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. Proc. Int. Conf. Mach. Learn., 2023.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. The street view house numbers (svhn) dataset. 2011.
dl htt..tttffrr.dde/oooueeuuueeeeee
Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge- regression. In Proc. Int. Conf. Learn. Represent., 2021a.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In Proc. Adv. Neural Inform. Process. Syst., pp. 5186-5198, 2021b.
Parsa Nooralinejad, Ali Abbasi, Soheil Kolouri, and Hamed Pirsiavash. Pranc: Pseudo random networks for compacting deep models. arXiv preprint arXiv:2206.08445 2022.
Xinchi Qiu, Javier Fernandez- Fernandez-Maqques, Pedro PB Gusmao, Yan Gao. Titouan Parcollet, and Nicholas Donald Lane. ZeroFL: Efficient on-device training for federated learning with local sparsity. In Proc. Int Conf. Learn. Represent. 2022.
Seungjae Shin, Heesun Bae, Donghyeok Shin, Weonyoung Joo, and Il-Chul Moon. Loss-curvature matching for dataset selection and condensation. In International Conference Artificial Intelli- gence and Statistics, pp. 8606-8628, 2023.
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. arXiv preprint arXiv:2312.03526, 2023.
Murad Tukan, Alaa Maalouf, and Margarita Osadchy. Dataset distillation meets provable subset selection. arXiv preprint orXiv:2307.08055 2023.
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pp. 12196-12205. 2022.
Kai Wang, Jianyang Gu, Daquan Zhou, Zheng Zhu, Wei Jiang, and Yang You. Dim: Distilling dataset into generative model. arXiv preprint arXiv:2303.04707, 2023.

---

Published as a conference paper at ICLR 2024
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 2018.
Li Yang, Adnan Siraj Rakin, and Deliang Fan. Rep-net: Efficient on-device learning via feature reprogramming. In Proc. IEEE Conf. Comput. Vis. Pattern Recog,, pp. 12277-12286, 2022.
Yuedong Yang. Guihong Li, and Radu Marculescu. Efficient on-device training via gradient filtering. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pp- 3811-3820, 2023.
Zeyuan Yin and Zhiqiang Shen. Dataset distillation in large data era. arXiv preprint arXtv:2311.18838. 2023.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from perspective. In Proc. Adv. Neural Inform. Process. Syst. 2023.
David Junhao Zhang, Heng Wang, Chuhui Xue, Rui Yan, Wenqing Zhang, Song Bai, and Mike Zheng Shou. Dataset condensation via generative model. arXiv preprint 2rXiv:330900669. 2023a.
Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In Proc. IEEE Conf. Compur. Vis. Pattern Recog., 2023b.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In Proc. Int. Conf. Mach. Learn., pp. 12674-12685, 2021.
Bo Zhao and Hakan Bilen. Synthesizing informative training samples with GAN. In NeurIPS 2022 Workshop on Synthetic Data for Empowering MI. Research, 2022.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proc. IEEE Winter Conf. Appl. Comput. Vis., pp.
2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching, In Proc. Ini. Conf. Learn. Represent., 2021.
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pp. 7856-7865, 2023.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. In Proc, Ady. Neural Inform. Process. Syst., 2022.
12

---

Published as a conference paper at ICLR 2024
A
ALGORITHM
Algo. provides the algorithm of the proposed MDC method.
Algorithm Multisize Dataset Condensation
Input: Full dataset B, model @ MLS selection period A,, learning rate of the synthetic dataset X, learning rate of the model 1/, outer loop iterations T, inner loop epochs F,, and class loop iterations C.
Output: Synthetic dataset S
1: Initialize synthetic dataset S
2: Initialize the most learnable subset (MLS) SMLS
3: for = to T do
> Outer loop
4: Randomly initialize model weight 0,
5: for
to E do
Inner loop >> Class loop
6: for c= 1 to do
7:
Sample class-wise mini-batches Be B, Se ~ Update S. with subset loss according to Eq. 10
8:
9:
end for Oz.0411 < Or,e vol (Or,ei end for
10:
> Update model with real image mini-batch B ~ B
12: if L. %
A then
b Every Al iterations
Select SMLS according Eq.
14:
end if
end for
16:
return Synthetic dataset S
B EXPERIMENT
B.
EXPERIMENT SETTINGS
Datasets:
SVHN (Netzer et al. (011) contains street digits shape 32 32 The dataset contains 10 classes including digits from 0 to 9. The training set has 73257 images, and the test set has 26032 images.
CIFAR-10 (Krizhevsky al. 2009 contains images of shape x 32 3 and has classes in total: airplane, automobile, bird, cat, deer, dog, frog, horse, ship. and truck. The training set has 5,000 images per class and the test set has 1,000 images per class, containing total 50,000 training images and 10,000 testing images.
CIFAR-100 (Krizhevsky et al. 2009) contains images of shape 32 x 32 x 3 and has 100 classes in total. Each class contains 500 images for training and 100 images for testing. leading total of 50,000 training images and 10,000 testing images.
ImageNet-10 10 (Deng et al. 2009) a subset of ImageNet-1K K (Deng et al., 2009) containing images with average 469 x 387 3 pixels but reshaped resolution of 224 x 224 3. contains 1,280 training images per class on average and total of 50,000 images for testing (validation set). Following Kim et al. (2022b), the ImageNet-10 10 contains 10 classes: 1) poke bonnet, 2) green mamba, 3) langur, 4) Doberman pinscher, 5) gyromitra, 6) gazelle hound, 7) vacuum cleaner, 8) window screen, 9) cocktail shaker, and 10) garden spider.
Following IDC (Km al. 2022b), perform augmentation during training networks in condensation and evaluation, and we use coloring, cropping, flipping, scaling, rotating, and mixup. When updating network parameters, image augmentations different for each image in batch; when updating synthetic images, the same augmentations are utilized for the synthetic images and corresponding real images in batch.
Augmentation:
Color which adjusts the brightness, saturation, and contrast of images.
13

---

Published as a conference paper at ICLR 2024
Crop which pads the image and then randomly crops back to the original size.
Flip which flips the images horizontally with probability of 0.5.
Scale which randomly scales the images by factor according to a ratio.
Rotate which rotates the image by random angle according to a ratio.
Cutout which randomly removes square parts of the image, replacing the removed parts with black squares.
Mixup which randomly selects a square region within the image and replaces this region with the corresponding section from another randomly chosen image. It happens a probability of 0.5.
Multi-formation Settings. For all results we use IDC (Kim et al. 2022b) as the "basic condensation method" otherwise stated. Following its setup, CIFAR-10, CIFAR-100 100 datasets and a factor of 3 for ImageNet- 10.
formatio factor of 2 for SVHN,
Reason for Using Large Batch Size for IPC 32. For CIFAR-10. 10, CIFAR-100 and SVHN, use the default batch size (128) when IPC < 32 and a larger batch size (256) when 32 < IPC Σ 64. The reason is that our method is based on IDC (Kim al.) 2022b which uses multi-formatioon factor of f = 2 for CIFAR-10, CIFAR-100, and SVHN datasets. The multi-formation function splits synthetic image into f 2
images during the condensation process. To ensure all samples in a subset can be sampled during condensation, we increase the subsets when the number of images exceeds the default batch size, which 128. With a
factor 2, the maximum IPC of each sampling process is IPC 32 (i.e., IPC x 2 < 128). For ImageNet-10 1PC multi-formation factor of f is used. Hence, we use batch size of 256 (i.e., 128 < 20 x 3 < 256).
MTT Settings. The reported numbers of MTT (Cazenavette et al. 2022) are obtained without ZCA normalization to keep all methods using the standard normalization technique.
B.2 FEATURE DISTANCE CALCULATION
Tab. presents the feature distance computed at a specific outer loop without imposing the subset loss. The table conveys two pieces of information. First, the feature loss of smaller subset is always greater than that of a larger subset. That is reason why we need to find the rate of change. Otherwise, Stn will always be selected. Second, the feature distance of the smallest subset changes the most, and this contributes to why we select S(1) as the subset initialization.
7
3012
1678
1249
1013
896
807
738
701
675
2596
661
DHI..
416
384
358
352
382
378
365
369
377
Rate of change
8.32
7.68
7.16
7.04
7.64
7.56
7.30
7,38
7.54
Table 7: Feature distance of subsets summed over inner loop training. CIFAR-10, IPC10-
B.3
THE INFLUENCE OF BASIC CONDENSATION METHOOD
Tab. shows our method works on other basic condensation methods such as DREAM (Liu et al.
2023b
2
7
8
9
10
Avg.
IDC
49.66
54.58
53.92
54.55
55.18
58.80
61.51
63.36
65.41
66.72
58.37
DREAM
49.70
55.12
55.84
$7.59
58.72
62.92
63.61
64.71
66.26
67.44
60.19
Table 8: Comparison between the proposed method applied to IDC Kim al. DREAM Liu et al. (2023b) on CIFAR- 10 IPC
(2022b) and to
14

---

Published as a conference paper at ICLR 2024
B.4 PRIMARY RESULTS WITH STANDARD DEVIATION
In Tab, 2 we list the primary results with standard deviation for synthetic datasets with IPC and IPCso. including SVHN, CIFAR-10, and CIFAR-I00 datasets. The standard deviation is computed from three randomly initialized networks since the same subset is selected for each run. Even by taking into account these standard deviations, our method shows a consistent improvement in the average accuracy.
Dataset
7
10
Avg.
Diff.
68,50 +0.9
75.27+4o.a
79.55
81.85++o.2
o.. 84.53
85.66 86.05 0.1 86.25 +0.2
87.501
+o.a 81.85
SVHN
68.50 =0.9
70.65
=1.27=0.o
71.92.0.3
03.28
70.74
o 71.08 71.08 =.. 11.97 20.5 71.38
45.4840.4
51.55
60.42.+1.0
67.97.+..5
74.38+0.6 77.65
81.70+
83.86 ..s 85.96 87.501
+0.8 70.65
Ours
6..26::1.0
67.91
72.15 74.09
77.54
78. 78.17
80.92+0.
12.82=0.. $4.27.0.3
86.38
76.75 +6.10
20.80
24.85:20.4
59.79
61.84_2o.1
62.49:2..
24.5
65.53
66.33_
66.82.10.5
67.50 20.5 62.05
B
50.80 53.17.0.0
55 g9
56,
17.0.3
55. 80. 3.3 56.98 20.3 57.60
57.78 (1778_m
58.22.1..
58.38
56.00
CIFAR-10
27.49
38.50
45.29
- 50.85
53.60
57.98
00.2 60.99
63.60
65.71
+o.1
67.50
+0.b 53.15
Ours
$4.58 49.66
20.2
25.1822..
61.51 10.
10.3
66.72.0.1
A
28.90
20.2
34.2810
37.35 20.0
39.1
41. 1510 0.4 42.65 1 43.62
44,48
45.07
10.1
45.40
40.20
CIFAR-100
B
28,90'
20.2
30.63++o.1
31 0..
31.764002
32.61 a. 32.85 0.. 33.03
33.04
133.04.0..2
0.2
33.32
+0.2
33.39
20.2
32,12
20.2
21.76
28.01
20.2
20.3
27.27..
20.2 40.92
42.69_0.2 44.28
10.2
45.40.0.4
34.40
Durs
27.58
31.83
a 33.59
35.42 36.93 38.95
40.70 42.05
43.86
44.34
37.53 43.13
(a) Results SVHN, CIFAR-10, CIFAR- targeting IPC10-
Dataset
2
5
7
8
9
10
A
68.50' 40.9
75.27 +0.3
81.8
83.33
84.53
86.05 85.66
86.25
87,50' 20.3
SVHN
C
34.90
46.52.1.1
52.2310.9
36.3010.1
62.25
65.34
68.84
69.5721.7
71.93
74.69
58.77
67.7240.3
69.33
72.2610.4
75.02.0.3
73.7120.7
24.50
74.63
76.21
76.87
A
50.80
44.85
59.79
61.84...1
62.49.0.3
64.59.0.1
65.53
66.33
66.82.0.3
67.50" 10.5
CIFAR-10
C
27.87-0.4
35
35.69+0.5
41.93+0.2
45.29+0.2
47.54+0.4
51
51.9440.4
53.51+0.3
65.59
56.62+0.2
58.26+0.11
Ours
47.83=0.6
52.18
56.29:0.1
58.5240.2
58.7540.4
60.6720.3
61.0020.1
62.74.0.2
62.32.0.2
62.64:0.2
A
28.90* 10.2
34.28.1022
37.35g0.2
39.13.0.1
41.15
42.65.0.4
43.62
44.48.0.2
45.07
45.40.0.4
CIFAR-100 100
c
12.66-0.1
18.35
23.76+0.4
26.9+o..4
29.12+0.2
32.23+0.1
34.21+0.4
35.71
37.18+0.3
38.25+0.a
Ours
20.34_0.2
29.71
31.440.4
32.95
34.49.0.3 36.36.0.2
38.49
39.59.0.1
40.43
41.35
Dataset
20
40
Avg.
Diff.
A
89.54+0.2
90.27+0.1
91.09+..1
91 91.38+0.1
84.34
SVHN
C
83.73 87.83
89.73
91.38
68.23
Ours
83.67.0.a 87.08
89.4610.3
91.39
76.47 +8.24
70.82.0.3
72.86.0.0
74.30.0.0
75.07
CIFAR-10
c
66.77+0.1
70.50+0.2
72.98+0.8
74.50
54,21
Ours
66.88 70.02.0..2
72.91
74.56.
62.01
+7.80
A
49.50+0.6 22.28+0..a
52.54+0.3
53.47+0.6
43.56
CIFAR-100 100
45.67 49.60.0.2
52.36.0.11
$3.47.0.5
Ours
46.06
49.40+0.1
51.72+0.1
53.67
39.45
+4.49
(b) Results of SVHN, CIFAR-10, 10, CIFAR- 100 targeting IPCso-
Table 9: Comparisons between the proposed method and three different baselines built with the IDC Kim et al. (2022)). represents the numbers reported in the original paper. Results from sub-table (b) divided into two parts due to limited space.
15

---

Published as a conference paper at ICLR 2024
B.5 ACCURACY OF SUBSETS DURING CONDENSATION PROCESS
Using different condensation runs in Tab. 6 we analyze the effect of subset selection during conden- sation. As shown in Fig. 7 the with accuracy of 58.37% has only [PC-(1,2) selected as SMLS. Tab. shows the accuracies for each subset during the condensation process. As shown in Fig. 8 the run with an accuracy 59.55% has IPC- are selected as SMLS- Tab, shows the accuracies for each subset during the condensation process. We hypothesize that optimizing a subset with larger IPC during the condensation process will lead to higher accuracy as it provides more supervision signals to guide the subset optimization.
4
Most Learnalle Subset (MLS)
Frozen Subset
Selected Subset
3
2
1
0
15111114111117788888
Selection Interval At
100
Figure 7: Visualization for the run with SMLS including IPC; and IPC,
100
200
300
400
500
600
700
800
900
1000
48.79
48.62
48.16
48.81
49.58
49.03
49.22
49.37
49.19
49.75
47.24
46.45
46.56
47.02
51.94
53.76
53.93
3
47.95
48.14
48.04
47.93
54.02
53.73
53.91
53.74
53.81
53.82
4
51.07
50.54
50.98
50.82
54.31
54.17
53.98
54.16
54.55
54.51
52.92
52.05
52.18
52.08
54.84
54.53
54.77
54.38
55.17
55.26
56.03
55.58
56.01
55.74
57.63
57.99
57.78
58.14
58.73
58.10
7
58.09
58.27
57.92
57.81
60.46
60.56
60.97
60.14
60.97
60.90
8
59.72
60.31
60.03
59.83
62.43
62.21
62.17
62.27
62.82
62.68
61.88
61.90
61.60
62.13
63.79
63.95
64.31
63.97
64.64
64.54
63.21
63.19
63.52
63.00
65.79
65.59
65.70
65.38
66.35
Avg.
54.69
54.51
54.50
54.52
57.68
57.55
57.67
57.57
58.07
57.98
1100
1200
1300
1400
1500
1600
1700
1800
1900
2000
49.75
49.76
49.25
49.18
49.76
49.64
49.61
49.63
49.44
49.66
54.38
53.87
54.47
53.33
53.77
53.87
54.72
54.82
54.55
54.58
53.52
53.46
53.51
53.39
53.94
53.56
54.69
54.00
54.27
54.67
54.80
54.72
54.71
54.34
54.51
54.34
54.66
54.73
54.55
54.94
55:04
54,58
54,60
55.04
54.63
55.25
54.57
55.20
55.18
58.40
58.68
58.16
58.06
58.83
58.45
58.97
58.65
58.92
$8.80
60.83
60.77
61.08
61.14
61.06
61.13
60.99
61.45
62.78
62.76
62.68
62.95
63.04
62.88
63.23
63.65
63.90
63.36
64.91
64.91
64.99
64,52
64.87
64.86
64.98
65.29
65.05
65.41
10
66.04
65.90
66.37
66.69
66.56
66.41
66.76
67.10
66.40
66.72
Avg.
58.02
$7.99
57.98
57.86
58.12
57.99
58.35
58.38
$8.35
58.37
Table 10: Accuracy of subsets evaluated different outer loops. Selected SMLS includes IPC, and IPCz. CIFAR- 10, IPC
16

---

Published as a conference paper at ICLR 2024
Avg. Acc. of the run: 59.55%
4
Most Learnable Subset (MLS)
Frozen Subset
Selected Subset
3
2
1
0
3
6
7
91 10111123141111777118822
Selection Interval
At 100
Figure 8: Visualization for the run with SMLS including IPC₁, IPC,, IPC3, and IPC,
100
200
300
400
500
600
700
800
900
1000
49.40
50.20
50.10
51.00
50.00
50.60
49.80
48.70
48.90
49.74
47.30
47.60
50.30
48.20
49.50
3
48.40
48.60
$7.80
57.90
55.30
57.20
55.80
55.70
56.00
57.46
4 52,30
51.80
57,40
57.60
55.00
57.60
56.60
55.60
55.90
56.93
5
52.80
53.60
56.30
56.50
56.20
57.10
56.30
55.80
56.20
56.59
57.00
57.90
59.60
58.50
59.20
60.00
59.00
7
58.10
59.50
61.10
60.00
61.50
61.50
61.30
60.70
60.50
61.24
8
60.50
61.90
62.30
62.20
62.50
63.30
63.10
63.10
62.50
63.13
62.00
63.10
64.10
64.20
63.90
64.40
65.30
64.20
64.30
64.72
63.40
65.
64.50
64.90
66.50
Avg.
55.12
55.93
58.36
58.25
57.77
58.62
58:42
58.17
58.28
58.88
1100
1200
1300
1400
1500
1600
1700
1800
1900
2000
49.19
49.58
49.36
49.38
49.41
49.68
49.68
49.14
49.41
49.55
53.40
53.05
54.14
53.78
53.93
53.11
53.48
53.79
53.86
53.75
57.00
57.19
57.03
57.51
56.61
56. 74
56.58
56.54
57.12
57.38
56.81
57.17
57.17
56.75
56.88
56.51
58.88
59.33
5
56.88
56.49
56.84
56.80
56.72
56.33
56.33
56.64
58.05
58.13
59.58
59.37
59.29
59.62
59.62
59.38
59.59
59.22
60.55
60.62
61.35
61.51
61.41
61.20
8
62.78
63.93
63.38
63.55
63.20
63.64
63.41
63.18
63.41
63.59
64.75
64.68
65.12
65.36
65.14
65.32
64.88
65.52
63.40
65.25
10
66.15
66.56
66.22
66.22
66.
66.80
66.36
66.64
66.88
66.91
66.79
Avg.
58.82
58.97
$8.96
59.06
59.03
58.88
$8.88
58.86
Table 11: Accuracy of subsets evaluated at different outer loops. Selected SMLS includes IPC1, IPC2, IPC;, and IPC,. CIFAR-10 10, IPC10-
17

---

Published as a conference paper at ICLR 2024
B.6 CLASS-WISS WISE MLS SELECTION
Stable to Class-wise and Non-class-wise. By default, we employ uniform MLS size across all image classes for simplicity. However, our approach be easily extended maintain class-specific MLS sizes. As indicated in Tab. 12 our approach performs consistently in both class-wise and non-class-wise settings.
Visualization of Class-wise MLS Selection. Fig. 6 presents the choice of MLS of each class at every selection round. Compared the non-class-wise manner (Fig. 3b). the class-wise manner selection tends to select relatively larger subsets.
Wass-wiiee
7
10
20
40
50
Avg.
49.22
$2,90
56.13
56.98
57.55
61.05
62.22
63.57
65.44
66.90
59.19
IPC10
49.66
54.58
53.92
54.55
55.18
58.80
61.51
63.36
65.41
66.72
58.37
48.17
53.35
55.68
57.11
56.75
59.57
60.02
60.31
60.76
61.55
66.79
70.29
72.77
74.57
61.26
JPCsa
47.83
58.52
61.90
62.74
62.64
66.88
74.56
Table 12: Class-wiss v.s. non-class wise SMLS CIFAR-10.
Geweted Imagges
insiSinnnciii Outer Loop
0ia528 Fananciannii Outtr LoOp
oiS5ea 25 Cinaanncinn Outer Loop
Gilbane Finneiinnii Outer Loop
Outer Loop
Imagen Ohoo (PC)
Outer
Outer
Duser LOOD
Duter Loop
Outer Loop
Figure 9: Visualization of selected subsets using a class-wise approach.
18

---

Published as a conference paper at ICLR 2024
C
VISUALIZATION OF CONDENSED IMAGES
C.I CIFAR-10
Fig. 10 show the effectiveness of the proposed method. Note that the two figures using multi- formation factor of I are for the purpose of better visualization. All experimental results shown in this visualization use the same settings the main results reported in Tab. Fig. 12 presents the visualizations of MDC on the CIFAR-10 dataset using a factor of 2 (Kim et al. 2022b).
Figure 10: Visualization of the initialization of CIFAR-10, IPC
airplane
car
bird
cat
deer
dog
frog
horse
ship
truck
(a)
(b) IDC Kim et al. 2022b)
(c) MDC
Figure 11: Visualization of the proposed condensation method. (a) and (b) IDC (Kim al. 2022b) condensed to IPC, and IPC respectively. (c) is the proposed method, MDC, CIFAR-10.
(a) Initialization
(b) IDC Kim et al.
2022b
(c) MDC
Figure 12: Visualization of the proposed method. The number of the meaning each condensed image is a composite of four original images. CIFAR-10 10, IPC10-
factor is
19

---

Published as a conference paper at ICLR 2024
C.2
CIFAR-100
Fig. 13
visualizes the effects of MDC on the CIFAR-100 dataset using a factor of (Kim et al.
2022b).
(a) Initialization
(b) IDC Kim et al. 2022b
(c) MDC
Figure 13: Visualization of the proposed condensation method. The number of the multi-formation factor is 2, meaning each condensed image is a composite of four original images. CIFAR-100, 10. class: apple.
C.3 SVHN
Fig. 14 2022b)
presents the visualizations of MDC on the SVHN dataset using factor of (Kim al.
(a) Initialization
(b) IDC Kim et al. 2022b
(c) MDC
Figure 14: Visualization of the proposed condensation method. The number of the multi-formation
factor is 2, meaning each condensed image is composite of four original images. SVHN IPC10.
20

---

Published as a conference paper at ICLR 2024
C.4 IMAGENET
Fig. 15 uses factor of 3 for ImageNet. Through comparing the images (class: gazelle hound)
highlighted by orange, red and green boxes in Fig. 15 observe the similar pattern shown in Fig. that our MDC has large distortion.
(a) Initialization
(b) IDC Kim al. 2022b
(c) MDC
Figure 15: Visualization of ImageNet targeting IPC20. The number of the factor meaning each condensed image is a composite of nine original images. class: gazelle hound.
21


Reviews:
[{"review_id": "ry6BmDb0kU", "reviewer": "Reviewer_CWAb", "summary": "This paper introduces the Multisize Dataset Condensation problem that can derive multiple subsets from the condensed images for supporting on-device scenarios. The authors identify “subset degradation problem” where the performance of a subset from condensed images is lower than directly condensing the full dataset to the target size. Subsequently, the authors propose “adaptive subset loss” where the most learnable subset is selected to update the subset, to alleviate the “subset degradation problem” for all subsets. Experimental results demonstrate that MDC works well for various datasets.", "questions": "I have several questions:\n1. In Fig 2c, for baseline C, how to select subsets to calculate accuracy? Is it random? Let’s assume we have a subset of 2 images. Do we select 2 images from the condensed data randomly?\n2. In basic condensation training (Sec 4.1), for each initialization the network is trained for 100 epochs. Is it the inner loop E?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a solution for DC named Multisize Dataset Condensation which is crucial for on-device scenarios. The proposed method outperforms baseline C significantly.", "weaknesses": "1. The synthetic samples within the subset seem to be fixed, which may not reflect “Multisize Dataset Condensation” correctly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j5GOEM11ZJ", "reviewer": "Reviewer_ZxGV", "summary": "This paper proposes a method called Multisize Dataset Condensation (MDC) to compress multiple-size dataset condensation processes into a single process. The goal is to obtain a small synthetic dataset that is equally effective but much smaller in size. The authors introduce the concept of the Most Learnable Subset (MLS) and propose an adaptive subset loss to mitigate the \"subset degradation problem\" in traditional dataset condensation. The MDC method can reduce the condensing process and lower the storage consumption. The MDC achieves state-of-the-art performance on various models and datasets.", "questions": "1. Can you provide the computational resource consumption and algorithmic complexity compared to Baseline-A, B, C, and other SOTA methods? It can help authors better understand the effects of algorithms in devices with limited computational resources.\n2. Can you provide the values of hyperparameters such as λ and η in Formula 2?\n3. The section on Visualization of MLS is currently difficult to understand. It would be helpful to provide more detailed and accessible explanations to ensure a clear understanding for readers.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed Multisize Dataset Condensation (MDC) method can effectively condense the N condensation processes into a single condensation process with lower storage and addresses the “subset degradation problem”.\n2. The adaptive subset loss in the MDC method helps mitigate the “subset degradation problem” and improves the accuracy of the condensed dataset compared to the Baseline-C.\n3. The concept of the rate of change of feature distance as a substitute for the computationally expensive “gradient distance” reduces computational overhead while capturing essential characteristics among subsets.", "weaknesses": "1. When the IPC (Inter-Process Communication) is small, there still exists a large accuracy gap between the proposed model and Baseline-A as shown in Figure 2 and Table 1.\n2. The impact of the calculation interval (∆t) on the performance of the MDC method needs to be further analyzed to determine the optimal interval size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OauQ9CLUkg", "reviewer": "Reviewer_6Dsd", "summary": "This paper introduces the Multisize Dataset Condensation (MDC) method, aiming to address challenges associated with dataset condensation in on-device processing scenarios. The main innovation lies in the compression of multiple condensation processes into a single process to produce datasets of varying sizes. The authors combat the \"subset degradation problem\" with an \"adaptive subset loss,\" ultimately enhancing the representation quality of condensed subsets. Experiments spanning various networks and datasets showcase the method's effectiveness.", "questions": "Please see weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Originality: This paper offers a unique approach to dataset condensation, aiming to cater to the specific needs of on-device scenarios. The proposal to compress N condensation processes into one is innovative.\nQuality: The \"adaptive subset loss\" is a novel concept, targeting the \"subset degradation problem.\" The method to select the Most Learnable Subset (MLS) is well-thought-out and complex.\nClarity: The paper is organized logically, and concepts are explained clearly. The use of terms like \"adaptive subset loss\" and \"subset degradation problem\" helps the reader understand the core issues being addressed.\nSignificance: The problem space being tackled (on-device training with dynamic computational resources) is relevant. Solving this issue can have substantial implications for real-world applications.", "weaknesses": "The paper explains three baselines for comparison. Compared to baseline A, the accuracy is not higher. Please explain the reason.\nIs it possible to reach Baseline A's accuracies? \nEquation 7 is not that clear. How to calculate the distance between the full dataset and subset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "njdOms4iMc", "reviewer": "Reviewer_DoMB", "summary": "This paper proposes a novel method to compress the condensation process into one process. It is different from the model compression or dataset compression, and this topic sound new to me. The definition of “subset degradation problem” is important in this domain. It will help the researchers to consider the problem. The experiments validate the effectiveness of the propose method.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The figures are beautiful and easy to understand.\n2. The idea is novel. The process compression sound new to me since it is different from the model compression or dataset compression. \n3. The “subset degradation problem” is practical. Although I have find similar pattern in experiments, it is good to see it is officially and properly presented.\n4. The experiment results are promising. It save the computational cost by N times.", "weaknesses": "1. It's not clear what's the purpose of baseline B. It looks like the results are only compared to baseline A and C.\n2. It's not clear why the freezing is used in MLS selection. If adaptive is good, why not just use adaptive method to choose the subset?\n3. Will the additional loss bring extra computational cost?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)