Color-Oriented Redundancy Reduction in Dataset Distillation
Authors: Bowen Yuan
Zijian Wang
Mahsa Baktashmotlagh
Yadan Luo
Zi Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Computer Vision
Data Distillation
Parameterization
Primary area: machine_vision
TL;DR: A novel parameterization method for data distillation by exploring the redundancy in color space.

Abstract: Dataset Distillation (DD) is designed to generate condensed representations of extensive image datasets, enhancing training efficiency. Despite recent advances, there remains considerable potential for improvement, particularly in addressing the notable redundancy within the color space of distilled images. In this paper, we propose a two-fold optimization strategy to minimize color redundancy at the individual image and overall dataset levels, respectively. At the image level, we employ a palette network, a specialized neural network, to dynamically allocate colors from a reduced color space to each pixel. The palette network identifies essential areas in synthetic images for model training, and consequently assigns more unique colors to them. At the dataset level, we develop a color-guided initialization strategy to minimize redundancy among images. Representative images with the least replicated color patterns are selected based on the information gain. A comprehensive performance study involving various datasets and evaluation scenarios is conducted, demonstrating the superior performance of our proposed color-aware DD compared to existing DD methods.

Color-Oriented Redundancy Reduction in Dataset Distillation
Bowen Yuan Zijian Wang
Mahsa Baktashmotlagh
Yadan Luo Zi Huang
[bowen. yuan, zijian. sajian.wang, m. baktashmotlaghh y y.luo, helen huang]Oup edu, au
The University Queensland
Abstract
Dataset Distillation (DD) is designed to generate condensed representations of extensive image datasets, enhancing training efficiency. Despite recent advances, there remains considerable potential for improvement, particularly in addressing the notable redundancy within the color space of distilled images. In this paper, we propose AutoPalette, a framework that minimizes color redundancy the individual image and overall dataset levels, respectively. At the image level, we employ palette network, a specialized neural network, to dynamically allocate colors from a reduced color space to each pixel. The palette network identifies essential areas in synthetic images for model training and consequently assigns more unique colors them. At the dataset level, we develop color-guided initialization strategy to minimize redundancy among images. Representative images with the least replicated color patterns selected based on the information gain. A comprehensive performance study involving various datasets and evaluation scenarios is conducted, demonstrating the superior performance of our proposed color-aware DD compared to existing DD methods. The code is available at htttp://ggituu: ceetttuu.coom//// wwt.////....NNOn///////tteaa htw.:i/111///uooaatttaeta
Introduction
Large-scale training data is essential for achieving high model performance. However, the sheer volume of the data poses significant challenges, including computational inefficiency, prolonged training times, and substantial storage overhead. Data Distillation (DD) 40) offers a promising solution this problem. By synthesizing smaller dataset from the original dataset, DD allows models trained on the distilled dataset to attain comparable performance to those trained on the full dataset, thereby reducing the resources needed for training.
Existing DD primarily minimizes the difference between the network trained on the full dataset and the network trained the synthetic dataset. Different surrogate functions have been implemented to quantify such differences, including performance matching 40 29]] feature distribution matching 39 46 and model gradient matching [47 3 Generally speaking, DD considers the synthetic images as parameters and directly optimizes them. Building on this concept, parameterization-be based dataset distillation (PDD) extends DD by enhancing the storage utility and reducing redundancy in the image space. Parameterization-basee DD methods represent the synthetic dataset in lower-dimensional space and then, reconstruct synthetic images for model training. Current parameterization based DD includes: learning in spatially down- sampled space 200 factorizing distilled images [9] (4).
optimizing latent embeddings and generators [5 and selecting informative frequency bands 35
While the existing PDD methods have shown promising results, most of the methods overlook the redundancy in the color space, thereby falling short of achieving optimal parameterization performance. We argue that reducing the number of unique colors within one image can have minimal impact on the low-level discriminative features (e.g., shapes, edges) required for model
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Quantization
Ca
Lb, Cm
Lrask
T
Select
S
@ palette
Colors
B
Initialization
Training
Quantized Images
The overview of the proposed AutoPalette framework. Initialization: We compare the information gain of quantized images to select the images used in the initialization stage. Training: We forward the synthetic data the palette network obtain the color-reduced images. The objective functions of palette network include La, Lin L.m and Lrask. The synthetic dataset is updated by solely optimizes Ltask
Figure 1:
training. Moreover, images within the same class typically share similar color distributions; therefore, dedicating storage to unique class patterns rather than storing the replicated color information would be more cost-effective.
To address the limitations of existing PDD approaches, we propose a color-oriented redundancy reduction framework, namely AutoPalette. Specifically, AutoPalette contains efficient plug- and-play palette network to tackle the issue color space redundancy within one image. This palette network transforms 8-bit color images into representations with fewer colors (e.g., 4-bit) by aggregating pixel-level color information from input images. To enhance the color utility, we design two additional losses top of the dataset distillation loss: the maximum color loss and the palette balance loss. The maximum color loss ensures that each color in the reduced color space is allocated at least one pixel, while the palette balance loss balances the number of pixels allocated each color. The palette network synthesizes image datasets with reduced color space while preserving essential features of the original images. Furthermore, we equip AutoPalette with a color-guided initialization module to suppress the redundancy in- bebeeween synthetic images. The module selects the samples with low replication after color condensation as the synthetic initialization, whereas information gain is adopted to quantify replication.
Contributions. We propose AutoPalette, color-oriented redundancy reduction framework for data distillation tasks, enhancing storage efficiency by reducing the number of colors in the images while preserving essential features; We seamlessly equip the distillation framework with guided initialization strategy that selects images with diverse structures in the reduced color space for initialization; Extensive experimental results three benchmark datasets show that the model trained on the 4-bit images synthesized by our framework achieve competitive results compared to the models trained 8-bit images synthesized by other DD methods. With the same storage budget, our method outperforms others by 1.7%, 4.2% on CIFARIO and CIFAR100.
2 Related Work
2.1 Dataset Distillation
Dataset distillation aims to synthesize small but informative dataset, enabling models trained on these synthetic data achieve comparable performance to those trained on the complete dataset. Wang el al. [40] firstly proposed meta- meta-model learning approach to optimize a synthetic dataset matching the model performance of large scale dataset. Early works adopt performance matching
frameworks 28 30
49 37 and optimize synthetic data using model performance rolling over the
training process on original dataset. Distribution matching methods [39
46 4 32] address high
complexity issues in bi-level optimization by matching step feature distributions between synthetic

---

(a)
(b)
(c)
Figure 2: The visualization of (a) images under 8, 3, 1-bit color depths (b-c) color condensed synthetic images and their color palette. (b) our full model (c) our full model without palette loss. The larger difference among rows of a color palette indicates better color utilization.
data and original real data. Gradient matching (47 approaches aim to match model parameters' gradients for single or multiple training steps, leading networks trained on synthetic data and original data to follow similar gradient descent trajectories.
and trajectory matching 3
12 10
2.2 Parameterizzatonnnoe
Dataset Distillation
Apart from finding matching objectives between the synthetic dataset and the original full dataset, another aspect of data distillation involves appropriately parameterizing synthetic data in different yet more efficient representatives in memory space. Without storing synthetic data as individual spatial representations, parameterization comprehends mutual characteristics between data instances and regenerates more data instances of the original input representations. IDC [20] stores images in low-resolution manner to conserve storage resources, and upsamples to the original scale for usage. Factorization methods conjecture inter-class data share mutual and independent information and generate synthetic data based on combinations of bases. Bases can be either spatial representations
[9], frequency domains [1], embeddings decoded by networks 23
38
2.3 Color Quantization
Color quantization [31 43 intends to aggregate similar colors and transform them using one representative color. Accordingly, images with a reduced color palette require less storage as the pixel values can be encoded in fewer bits. To uphold optimal image authenticity, traditional color quantization methods such as Median Cut (15), dithering
[13] and OCTree [II) typically employ color clustering problem. They commonly devise strategies to identify similar or neighboring colors for quantization purposes. On the other hand, parameter-bas based methods 36 [27] 16 not only rely on predefined heuristics but also leverage neural networks learn patterns and relationships to compress images to lower bits.
color quantization
3 Methodology
3.1 Notations and Preliminary
Dataset distillation aims to learn a small but representative synthetic dataset S = {(0', ,. {') from
a given large scale dataset T = ((f,,'")) ) Here, SS] and [T] denote the number of samples in the synthetic dataset and original large dataset, where |S| [T]. By training the synthetic dataset S, a model o(:; 0) is aimed to achieve performance comparable to that model trained on the original dataset T. The objective of dataset distillation can be formulated bi-level optimization problem:
min E, (C(TT 0s)], where 0s = arg minC(S, 0),
where -(..)) and 0 represent loss function and parameters of the networks, respectively. The inner- loop optimizes the network on the synthetic dataset and the outer loop evaluates the trained network on the real dataset.

---

The bi-level meta learning in Eq. Tequires inner-loop training during every training steps, and thus suffers from inevitable computational cost. Therefore, some of the existing methods [47 Bl [6 try to avoid unrolled hack-propagation in the inner loop using various surrogate objectives, thereby the optimal synthetic dataset S* can be obtained by optimizing:
S = arg min [E) (()))))))))))))))) ((;;)))
(2)
$
In parameterization methods for dataset distillation, the synthetic dataset S is stored in more efficient representations consisting of bases B E xxxxXxx and a set of transformation functions F; B - S, which generate the synthetic dataset. Here, N denotes the number of bases, C' represents the channel, and H and W are the height and width of the bases, respectively. a color transformation that reduces the number of unique colors in the image bases while preserving the essential details after color reduction for model training.
this paper, we propose
3.2 Overview
In this paper, we explore new dimension of parameterization-bene based dataset distillation, concentrating on optimizing storage efficiency by minimizing color space redundancy. We argue that complex color representation within storage-sensitive distilled images is not crucial for training networks. Instead, the limited storage budget for distilled images should be allocated novel samples that exhibit diverse object structures. The overall framework of AutoPalette is illustrated in Figure
The proposed AutoPalette framework for dataset distillation consists of two components, including a palette network and a volor-guided initialization strategy. The palette network is designed to enhance the color utility of the synthetic images in reduced color space by generating pixel-leve color mappings. Accordingly, the original images ā E Cx E S are transformed into color- condensed images b RCXHXW B with reduced color spectrum, where we denote the process as Ppalette S - B. RGB image dataset generally contains 256 distinct colors for each channel, and the palette network aims reduce the number of colors by generating color palette containing only K colors. As such, the synthetic dataset can be stored in low-bit format, rather than the conventional 8-bit format. To better leverage the diverse color feature information within the original dataset, we equip the proposed framework with a novel initialization strategy, which employs the generalized graph cut function to select representative images for initialization. Our method dynamically evaluates the impact of real images based on their color structures and initializes the synthetic dataset with the images that yield the highest information gains of graph cut functions. The subsequent sections will illustrate each module in detail.
3.3 Color Reduction via Palette Network.
The core of our proposed parameterization method for dataset distillation is to condense the number of unique colors in an image so that the image can be stored with a more efficient manner. One of the key challenges of reducing the unique color number in dataset distillation lies in local discriminative feature preservation. When an image is represented with smaller range of color, it is inevitable for some pixels to be merged to their neighbour color blocks. In this case, some of the local discriminative features (e.g., edge, shape, etc.) can be erased or distorted in the color-reduced images, hindering the network trained on them to achieve the optimal performance.
To alleviate this issue, we design a simple yet effective network, namely palette network, which learns the pixel-level color allocation in the pruned color space with the discriminative feature maximally preserved. Particularly, the color palette network predicts the probability map m E RC xW x which indicates the probability of a pixel being allocated to color of the K-dimensional reduced color space by forwarding an image ã to the palette network:
Dcstor (T; 0c),
where 0g is the parameters of the palette network.
Given base image b and its corresponding probability map m, we formulate the palette mn e Cx as the average pixel values of all pixels assigned the same color buckets index:
Eesy #ea.s (k)
m
(4)
Eeso om,
4

---

where e and & denote the e-th channel for the k-th quantized color, and j denote the vertical and horizontal pixel position in an image, and 8t is the Kronecker delta function, which equals arg max M,,.jj equals k.
Once the color palette in and a probability map are generated for image, its color condensed image b, with the number of unique colors per channel reduced to K, can be generated by an index searching process:
Pe = mlce hJ, where h = arg max Me.i.
&
To learn the palette network, straightforward solution is optimize the distillation task loss during the training stage. However, we can observe from Fig. that solely relying on the task loss for training the palette network leads to most of the palette being inactivated. The network tends assign pixels to a limited number of color buckets, which strongly limits the capacity and expressiveness of the distilled images. Therefore, two additional losses, namely maximum color loss and palette balance loss, are incorporated to enhance the utility of color buckets in synthetic images. In particular, maximum color loss, Lm encourages the palette network generate color allocation such that each color bucket is at least filled with one pixel within color palette. By aggregating the maximum confidences from probability index maps across the spatial dimensions, we define the maximum color loss as:
1 CK
max hw,kk, (h,w)
(6)
While the maximum color loss ensures the activation of each color bucket in the palette, the distribu- tion of pixel numbers in color buckets still be extremely imbalanced. Therefore, framework leverages a palette balance loss La, which encourages a more balanced usage of the buckets within the color palette by promoting color-wise entropy. We formulate the palette balance loss the entropy of m over the spatial dimensions:
K ΣE
W ΣΣ
H W ΣΣ
-
L, - CK
mo.jj.) log P(
mcij&k)
HW
HW
j=1
where P(.) represents the softmax function of m over the spatial dimensions.
By integrating the palette network with the complementary losses, we obtain the color condensed images while preserving the informative features.
3.4 Color Guided Initialization Module
The empirical results of previous studies have shown a strong correlation 3 |6 between the original images selected during initialization and the resulting distilled images in terms of visual appearance. In light of this finding, we propose an initialization method aimed at solving the redundancy problem in color-condensed synthetic images. However, since we do not have access the optimized palette network, it is prohibitive to directly measure the information overlap within a class after color condensation. To mitigate this issue, we propose leverage the traditional color quantization approach [5] to approximate the output of the palette network. Here, we denote the quantized full dataset as T Our proposed initialization strategy leverages conditional gain within submodular information theoretics to identify the most diverse images of each class after color condensation. Specifically, the conditional gain G(AlC) implies the gain of information by adding set C to set A,
where A,C C T and AnC 0. Formally, we have:
- G(A9))=G(TT) - G(C), G
(8)
where G(.) denotes a submodular function. Submodular information functions (18 describe set of combinatorial functions that satisfy the Shannon inequality [34] 26 and can effectively model the diversity of a subset. In our implementation, generalized graph cut (2), which maximizes the similarities between samples in AA and and minimizes and dissimilarities among the samples in A. The generalized graph cut function (+(()) is defined as follows:
adopt a monotone submodular function, namely
G (*(ACC)= ΣΣ Sim(i.j) - Σ Sim(jnnja),
(9)
IEC jEA
j.JOEA
5

---

where i and j denote data samples from the sets C and A, respectively. Sim(::-) -) is a similarity function between two samples. Instead of directly measuring the feature level similarity, we propose to measure the similarity between the last layer gradients V0 as follows:
Sim(i.j) = cos (VoCc(Q(()" 0) Volce(Q(E")))))
(10)
where cos(:.) -) is the cosine similarity function, Q denotes the Median Cut quantization method, j² and 7 are the ith and jth samples of set T, and Ve is the gradient of cross-entropy loss between the prediction and the ground truth label on the last layer of the network. By substituting Eq. into Eq.
(8) we select a representative sample for inclusion A by:
arg max G (AA - EE Sim(i,c).
(11)
SEA eEC
The proof for the graph cut conditional gain is provided in Appendix A. From Eq.
we can see that the data sample obtaining the highest conditional gain may be selected. Intuitively, we select the most representative sample from the unselected set C, whilst ensuring is dissimilar to the already selected samples in A. The entire color diversity selection process is provided in Algorithm !|
By far, our initialization method can select diverse and representative samples of each class in the approximated quantization set. To minimize the difference between the approximation set and the output of the palette network, we put forward a regularization term La- The regularization term not only constrains the color allocation shifting of palette network, but also enhances allocation consistency, so similar colors are grouped together with higher fidelity. The regularization term La is defined as:
1 2 -
(12) where h denotes the the arg max of the color mapping indices by Median Cut over the color space, and @ denotes the element-wise multiplication resulting in self correlation matrix for palette bucket allocations of the palette network and Median Cut. When creating index mappings for pixels, different methods might cluster the same pixels into the same group, but the indices may not match. By utilizing fa, we emphasize clustering resemblance, disregarding the order of clustering indices.
La La= = N 1--yhhon
3.5
Overall Dataset Distillation Objective
Our framework aims to create a color-condensed synthetic version of the original dataset while maximally preserving task-related information. In line with other parameterizationn-aase methods, we incorporate the dataset distillation loss Ltask from existing works into our framework.
As such, update the palette network, we have the overall loss function defined as:
arg arg min Spalette = Lrask + aL + 3Lo + tfua,
(13)
6,
where a, 7 are the coefficients as the weights of loss components. The synthetic set S is optimized as follows:
0)). where B = Ópalette (S: (2)) arg min Crask = (()
(14)
where @palette (:: 0.) denotes the color quantization process using the palette network.
3.6 Storage Analysis
In our experiments, the images follow the 256-color storage convention, where pixel values occupy 8-bit storage space. Given the storage budget of images per class (IPC), the maximum storage budget for one class capped at 8 IPC CHHW, where C, Hl and W represent the channel, height and width of the images, respectively. When representing colorful image pixel value with bits, where 1 S n < 8, there can be at most 2d distinct colors per image. This must satisfy the condition 2"-" N; < 29, where N; is the number of colors for the i-th color reduced image and each N, < 2". Therefore, for images with n-bit format, up to 28-n colors can be represented in the storage budget using a bitmap index with small bytes. The bitmap index indicates the image number associated with the current lower bit color value.

---

Table Test accuracy (%) of previous works and our method on ConvNet D3. Our synthetic images are reduced from 256 colors to 64 colors. Our method outperforms previous methods and achieves state-of-the-art performance.
Dataset
CIFAR1O
CIFARI00
IPC
10
50
1
10
50
Random
14.41002
26.0111.2
43.41..0
4.2.10.3
14.610.5
30.0=0.4
Coreset
Herding 42
21.5+1:3
31.6110.7
40.410.6
8,410.3
17.310.3
33.7=0.5
K-Center
23.3:1.9
36,410066
48.7:1..3
8.640.3
20.7:10.2
33.6=0.4
DD 40
36.8+1122
DM 46
26.0=10.8
48.9:40.6
63.0+0.4
11.400.3
29.7:0.3
43.6=0.4
Distillation
DC 47
28.3=0.5
44.9=
53.9+0.5
12,8+033
25.2++0.3
TM
46.3++0.8
65.3+0..7
71.6+0.2
24.3+0.3
40.1+0.4
47.7-0.2
DATM
46.9.10.5
66.8:10.2
76.1
27.940.2
47.240..4
55.0=0.2
IDC 20
50.0:10.4
67.5
74.5:10.1
HaBa X4
48.3:10.8
48.31108
48.3110.8
33,410.4
40.210.2
4..0=0.2
RTP 9
66.4110.4
71.2:0.44
73.610.55
34,410.4
42,910.7
Parameterization
SPEED
41
63.2+-0.1
73.5+0.2
77.7+0.4
40.0+0.4
45.9+0.3
49.1-0.2
FReD 35
60.6:10.8
70.3+0.3
75.8++0.1
34,6+0044
42.740.2
47.8=0.1
AutoPalette
58.611.1
74.310.2
79.410.2
38.010.1
52.6-0.3
53.310.8
Experiments
In this section, we first evaluate the effectiveness of our method in comparison with other parameteri- zation methods various datasets. Afterwards, perform experiments on the relations between synthetic image color number and model performance. We also conduct ablation studies and assess the efficacy of each proposed component to distillation performance.
4.1 Experimental Setting
We conduct experiments of our model various benchmark datasets, including CIFAR-10 21], CIFAR-100 [21] and ImageNet 77 We compare our parameterization method with core-set methods and other existing DD works containing baselines such DD 40 DM 446, DC 47).
TM B]
and parameterization techniques including IDC [20]. HaBa [24]
RTP (9) SPEED 411, FReD [35
Experiments performed different distillation memory budget settings for 1/10/50 images per class (IPC). We follow the previous works to use a ConvNetD3 for the CIFAR family and ConvNetDS
for ImageNet the training and evaluation network. We follow the DATM [22] implementation based on trajectory matching, without soft label initialization using correctly predicted samples. Each experiment evaluated on randomly initialized networks, and the mean and standard deviation of the evaluation accuracy are recorded. We set loss coefficients a=1, B=1, 3=3 for all experiments if specified. All experiments can be conducted on Nvidia H100 GPUs that have 80GB RAM for each or 4xNvidia V100 GPUs that have 32GB RAM for each.
4.2 Experimental Results
Results on CIFAR1O and CIFAR100. We perform experiments under paramterization settings on CIFARIO 21), CIFAR 1O0 (21] We set color palette network to condense the number of colors of single image from 256 to 64, such that despite huge color space reduction quantized images still preserve much fidelity of images. As shown in Table our method achieves superior performance than other parameterization works in various tasks. Notably, in the experiments when IPC equals 10 and 50, our method significantly outperforms other methods. model achieves 52.6% and 53.3% classification accuracy when IPC is respectively 10 and 50, which increases 6.7% and 4.2% higher than previous state-of-the-at parameterization methods. These outstanding performances highlight that reducing the color redundancy within the synthetic dataset can improve the storage utility and thereby improve the distillation result.
CIFARIO0 experiments, our
Results on ImageNet. Following (3] we conduct experiments on six subsets of ImageNet, where each subset consists of classes and the images are of resolution 128x 128. We conduct experiments with the storage budget of I0C=10. ConvNetD5 is employed the backbone model for training

---

Table Test accuracy (%) on ImageNet-Subset ImageNette, ImageWoof, ImageFruit, ImageMeow, ImageSquawk, imageYellow All experiments are conducted on CIFAR10 with IPC=10 storage budget for parameterization methods.
Dataset
ImageNette
ImageWoof
ImageFruit
ImageMeow
ImageSquawk
Image Yellow
TM
3
1..0+1..
35.8=1.8
10,3+1..
40.4+2.2 42.910.9
52.3+1.0
6..0
HaBa 24
64.7#16
38.6=1.3
42.5#:..
5..8+100
63.0:1.6
FrePo
85
66,5+0.8
42.2-0.9
SPEED
41
72.9+1.5
44.1-1.4
50.0+-0.8
52.0+1.3
71.8+113
70.5+1.5
AutoPalette
73.210..
44.310.9
48.4111.8
53.610.7
68.011.4
72.011.6
and evaluation. From Table 2, we can see our method outperforms other PDD methods on most of ImageNet subsets, including ImageNette, Image Woof, ImageMeow and Image Yellow, while results of the other subsets still achieve comparable performance with previous state-of- srate-of-the-an ththhe-rt art results. Specifically, our method achieves 44.3% and 72.0% on hard datasets ImageWoof and Image Yellow, increasing 0.2% and 1.5% than previous best PDD methods. We also observe that for subsets with distinet classes, effectively preserves the key semantics necessary for accurate classification. On the other hand, for fine-grained subsets where the classes are similar, observe inferior performance. This is likely because fine-grained details are blurred in images represented by fewer colors, thereby making it challenging differentiate between classes.
method achieves promising results, which is because color condensation
Compatibility of Distillation Frameworks. While we take trajectory matching as our primary distillation method, we demonstrate that our framework effortlessly be equipped to improve other dataset distillation methods. As illustrated in Ta our method shows a significant performance boost across all IPC settings and datasets when adopted the distribution matching method. Espe- cially, our method increases the test accuracy up to 15% when IPC=10, and 15.7% for IPC=1 on CIFÀR 100. This observation aligns with our objectives that color-oriented redundancy man- agement framework should be adapted across different standard DD frameworks The performance improvement underscores the high compatibility of our methods with diverse DD frameworks.
4.3 Ablation Study
Here, we focus on comparing variants of our proposed framework. Therefore, we fix the number of synthetic images to 10 per class, rather than fully utilizing the available storage capacity.
Effectiveness of Loss Components. To validate the contribution of each loss term to the overall framework, conduct experiments on CIFARIO with IPC=10. Specifically, we construct three variants of our model by removing L.. Ls, and L., correspondingly. We show the experimental result in T Under the same experimental conditions, eliminating specific loss functions will suffer from performance decline. The results demonstrate the essential role played by each loss function in optimizing the palette network.
Effectiveness of Selection Criteria in the Color-guided Initialization. We compare our proposed initialization with two baseline sample selection criteria, including Random Real, and Graph Cut Real. Random real is widely adopted by DD methods, which randomly select images from the full dataset as the initialization of the synthetic dataset. In Graph Cut Real, we apply graph cut on 8-bit images
Table 3: Test accuracy (%) when certain loss component removed during training.
Table 4: Evaluation on the effectivenees of sub- modular selection using quantized images, in comparison with random initialization and sub- modular selection using full color images.
Lm
Cb
La
Accuracy
X
64.00
Initialization Method
Accuracy
X
61.40
Random Real
60.84
X
60.14
Graph Cut Real
61.41
66.20
Graph Cut on Quantized Image
62.13
8

---

Table 5: Test accuracy (%) on different DD frameworks including DM and TM across various IPC settings. We adopt ConvNetD3 as the backbone network to distil the synthetic dataset on CIFARIO and CIFARIOO.
Framework
DM 46
TM B
Dataset
IPC
Vanilla
AutoPalette
Increase
Vanilla
AutoPalette
Increase
1
26.0:
35.5++0.4
9.5t
46.3+.0.8
58.6+1.1
12.31
CIFARIO
10
48.9+0.66
60.9+t.11
12.0
65.3+0.7
74.3++02
9.01
50
63.0+0.4
71.6+0.4
8.6
71.6+002
79.4+0.2
7.81
1
11.440.3
20.9:0.11
9.5t
24.32:0.3
38.010.1
15.71
CIFARI00
10
29.7+0.33
44.74:0.1
15.01
40.1+0.44
52.640.3
12.51
50
43.6+#0.4
50.1+0.11
6.5t
47.7+0.2
54.1+0.88
5.61
and select the most representative samples with high information gain. Compared with Graph Cut Real, the selection criteria used in our framework computes the information gain over the quantized images. From table 4 we can see that our methods using quantized images exhibit better performance than comparison approaches. The graph cut with original full-color images also outperforms the baseline model using randomly selected real images as initialization, confirming the effectiveness of computing information gain over color-reduced images
Effectiveness of Color-guided Initialization under Dif- ferent Color Depth. Our experiments compare the perfor- mance of two ways initialize the hase images: em- ploying our method of color-guided initialization (denotes GraphCut), and randomly selecting real images the base images (denotes Baseline). We contrast two approaches on CIFARIO 10 with IPC=10, spanning from synthesizing images with low-bit quantization to those with higher-bit quantization. From Figure 3
66-
64-
Accuracy
60
58
GraphCut
we observe that our method brings better performance when quantized images are rep- resented in lower bits. Starting from 16 colors per pixel, we observe a gradual convergence in performance as we move towards utilizing full-color space. Meanwhile, as can be seen, when K=32 and 64, it achieves the best trade- off, as it still attains competitive performance comparable to that of full-color budget images, while substantially reducing the storage space required for quantized images.
56
Baseline
10
20
30
40
50
60
Num of Colors
Figure 3: Comparison between the per- formance of submodular color diversity initialization and random real images ini- tialization.
Conclusions, Limitations, and Future Work
In this paper, we aim to solve the color-redundancy issue within data distillation from both the image level and the dataset level. For condensing the colors within images, we utilize a palette network to capture the color redundancy between pixels and represent images using fewer colors. Beyond this, to reduce repetitive patterns in between synthetic images design guided image initialization module that selects samples by maximising information gain. Extensive experimental results demonstrate that our AutoPalette framework can effectively reduce color redundancy and simultancously preserve the essential low-level feature for model training.
Limitations. For instance, images from different classes may have bias towards color usage. Images of one class may be sufficient to be represented by fewer colors than those from other classes, in which case an imbalanced color budget arrangement may be a better option. In future, it is also promising explore the dynamic color depth allocation, which allocates more budgets to difficult classes, thereby improving the distillation performance.
Acknowledgments and Disclosure of Funding
This research partially supported by the Australian Research Council (DE240100105. DP240101814, DP23010116)

---

References
[1] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence, 34(11)2274-2282, 2012.
[2] Yuri Boykov and Olga Veksler. Graph cuts in vision and graphics: Theories and applications. In Handbook of mathematical models in computer vision, pages 79-96. Springer, 2006.
[3] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Yan Zhu. Dataset distillation by matching training trajectories. Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 4750-4759, 2022.
[4] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Generalizing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3739-3748 2023.
[5] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Generalizing dataset distillation via deep generative prior. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3739-377 2023.
[6] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- Ik with constant memory. In International Conference on Machine Learning, pages 6565-6590 PMLR, 2023.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages
Icee, 2009.
[8] Yining Deng, Charles Kenney, Michael S Moore, and BS Manjunath. Peer group filtering and perceptual color image quantization. In 1999 IEEE International Symposium on Circuits and Systems (ISCAS), volume 4, pages 21-24 IEEE, 1999.
[9] Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. Advances Neural Information Processing Systems, 35:34391- 34404, 2022.
[10] Jiawei Du, Yidi Jiang, Vincent YF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error improve dataset distillation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3749-3758, 2023.
[11] Michael Gervautz and Werner Purgathofer. A simple method for color quantization: Octree quantization. In New Trendy in Computer Graphics: Proceedings of CG International 888, pages 219-231. Springer, 1988.
[12] Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. To- wards lossless dataset distillation via fffficulty-aligneed trajectory matching. arXiv preprint arXiv:2310.05773, 2023.
[13] Jialing Han. An adaptive grayscale watermarking method in spatial domain. Journal of Information and Computational Science, 12:4759 08 2015.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[15] Paul Heckbert. Color image quantization for frame buffer display. ACM Siggraph Computer Graphics, 16(3):297-307, 1982.
[16] Yunzhong Hou, Liang Zheng, and Stephen Gould. Learning to structure an image with few colors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 101 10116- 2020.
[17] Yunzhong Hou, Liang Zheng, and Stephen Gould. Learning to structure an image with few colors and beyond, 2022.
10

---

[18] Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combina- torial information measures with upplications in machine learning. Algorithmic Learning Theory, pages 722-754. PMLR, 2021.
[19] Simonyan Karen. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
[20] Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data param- eterization. In International Conference on Machine Learning, pages 11102- 102-11111 11118. PMLR. 2022.
[21] A. Krizhevsky and G. Hinton. Learning multiple layers features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[23] Hae Beom Lee, Dong Bok Lee, and Sung Ju Hwang. Dataset condensation with latent space knowledge factorization and sharing. arXiv preprint arXiv:2208. 10494, 2022.
[24] Songhua Liu, Kai Wang, Xingyi Yang. Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. Advances in neural information processing systems, 35:1 15:1100-111 100-1111, 2022.
[25] Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17314-17324. 2023.
[26] William McGill. Multivariate information transmission. Transactions of the IRE Professional Group on Information Theory, 4(4):93-111, 1954.
[27] Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool. Practical full resolution learned lossless image compression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10629-10638, 2019.
[28] Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. arXiv preprint arXiv:201 1.00050, 2020.
[29] Timothy Nguyen. Roman Novak. Lechao Xiao, and Jachoon Lec. Dataset distillation with infinitely wide convolutional networks. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 5186-5198, 2021.
[30] Timothy Nguyen. Roman Novak, Lechao Xiao, and Jachoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Informtion Processing Systems, 34:5186-5198, 2021.
[31] Michael Orchard, Charles A Bouman, et al. Color quantization of images. IEEE transactions on signal processing. 39(12)2277--269, 1991.
[32] Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z Liu, Yuri A Lawryshyn, and Konstanti- nos N Plataniotis. Datadam: Efficient dataset distillation with attention matching. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 17097-17107, 2023.
[33] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: core-set approach. arXiv preprini arXiv:1708.00440 2017.
[34] Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3-55, 2001.
[35] Donghyeok Shin, Seungjae Shin, and II-Chul Moon. Frequency domain-based dataset distilla- tion. Advances in Neural Information Processing Systems, 36, 2024.
[36] Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pages 1747-1756. PMLR, 2016.

---

[37] Paul Vicol, Jonathan P Loorainee Fabian Pedregosa, David Duvenaud, and Roger B Grosse. On implicit bias in overparameterizze bilevel optimization. International Conference on Machine Learning, pages
PMLR, 2022.
[38] Kai Wang, Jianyang Gu, Daquan Zhou, Zheng Zhu, Wei Jiang, and Yang You. Dim: Distilling dataset into generative model. arXiv preprint arXiv:2303.04707 2023.
[39] Kai Wang. Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang. Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12196-12205, 2022.
[40] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 10955, 2018.
[41] Xing Wei, Anjia Cao, Funing Yang, and Zhiheng Ma. Sparse parameterization for epitomic dataset distillation. Advances Neural Information Processing Systems, 36, 2024.
[42] Max Welling. Herding dynamical weights to learn. Proceedings of the 26th annual interna- tional conference machine learning, pages 1121-1128, 2009.
[43] Xiaolin Wu. Color quantization by dynamic programming and principal analysis. ACM Transactions on Graphics (TOG), 11(4)3348-72, 1992.
[44] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages
PMLR, 2021.
[45] Bo Zhao and Hakan Bilen, Synthesizing informative training samples with gan. arXiv preprint arXiv:2204.0755 2022.
[46] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching, In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6514-6523, 2023.
[47] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprini urXiv:2006.0553 055929 2020.
[48] Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7856-7865, 7865, 2023.
[49] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. Advances in Neural Information Processing Systems, 11-98137987,, 2022.
12

---

A Appendix
A.1 Proof for Conditional Information Gain of Graph Cut
The generalized graph cut set function is defined as:
= E
Sim(i,a)
Σ
Sim(a1,a2),
(15)
iET aEAA
21,02EA
where C C. and Sim a similarity function. The submodular conditional gain is defined as:
f(AIB) & f(AUB) f(B),
(16)
which indicates the gain by adding samples in set B A. By substituting Eq. (16) with (15). can obtain:
S(AI)) = Σ E Sim(i,
Σ
(im e2)
NET cEAUB
c1,02EAUB
(17)
Sim(i,
+
Σ
Sim(b1, b2).
ET DEB
b1.d2EB
In the sample selection cases, AA and B are disjoint, and thus ECEAUB be rewritten ECEA-B Therefore, reformulate f(A U B):
S(AUB) = Σ
Σ
Sim(i, c) + A
ΣΣ
Sim(i, b)
iT cEAV
NET BEB
Σ
Sim(bi, b2)
Σ
Sim(ci, 02)
(18)
BB
CL,CEEA88
2
Σ
Sim(a', 6).
BEBBBBB
We then formulate Eq. 16
f(AlB) = f(A -2 E
Σ
Sim(a' 6).
(19)
BEEAYBBEB
When A and B are disjoint, A is independent of B and we then simplify the conditional gain of graph cut function to:
S(A = ((A) 2 Σ Σ
Sim((', ,b),
(20)
d'EA\BEEB
A.2 Experimental Details
Datasets. We conduct experiments on multiple datasets:
an image dataset consists of 50,000 32x32 RGB images for training, and 10,000 images for testing. CIFARIO contains 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck.
CIFAR10:
CIFAR 100: an image dataset that similar to CIFARI00, but has 100 classes containing 600 images each.
ImageNet subsets: high resolution image subsets of ImageNet 7 that contain all 128 x 128 RGB images and each contains 10 classes. ImageNet subsets include include ImageNette, ImageWook ImageFruit, ImageMeow, ImageSquawk, and Image Yellow.
Networks. For experiments of low resolution datasets including CIFARIO and CIFAR 100, 3-layer convolutional neural networks (ConvNet) are employed and we follow the identical network structures to the previous works. Each convolution layer contains 128 3x3 filters, followed by an instance normalization layer, a ReLU, and average pooling layer with 2x 2 kernel and stride 2.
13

---

Algorithm 1: Algorithm for guided image selection with maximum information gain.
Input: Original Dataset T; Number of classes Nelass: Images per class IPC; Network 0 Output: Set of selected data samples A for each class
for f- to Neloss do
X + all images belong to class from T;
A' <- {rand(X") ) // Initialize the selected set with a random sample
for j - IPC do C T AA:
c* - Eq. [1
// Select the data sample with the highest gain
A - A' U'c)'
For the high resolution datasets such as ImageNet subsets, use 5-layer ConvNets to perform the experiments. For the cross architecture experiments, we employ VGG11 [19]. AlexNet [22], and RestNet18 [14] and follow the implementations of the previous DD works. The palette network consists of two convolution layers and one ReLU between them. Both two convolution layers have 1x1 kernels and the second layer has no bias.
Implementation details. While we primarily use Trajectory matching (TM) as the distillation objectives, our method can be seamlessly adapted into other DD frameworks for various downstream tasks and datasets. In Table and T0, we provide the hyper-parameters settings in our work for both TM and DM different datasets. Specifically, although method is insensitive for most of the hyper-parameters, certain parameters including synthetic steps, the maximum starting epoch and the synthetic batch size should be carefully examined.
A.3
Algorithm for Sample Selection in Initialization
A.4
Cross Architecture Performance
One of the main concerns in data distillation arises from synthetic dataset overfitting to the training models, resulting in limited generalizability to be used by the other network architectures. Therefore, cross-architecture performance is crucial for evaluating the effectiveness of data distillation methods. To assess the generalized performance of our method, we employ CIFARIO synthetic datasets trained on ConvNet to train various network structures including VGG11 [19] AlexNet [22) and ReNnet 8 14]. The results of cross-network architecture are presented in Table We noticed that when IPC is relatively higher, our method outperforms the other baseline methods.
Table Cross architecture performance of synthetic dataset that is optimized by ConvNet. Experi- ments are performed on CIFARIO, 10, and VGG11, AlexNet and ResNet 8 are used for evaluating the cross architecture performance.
VGGII
AlexNet
ResNet18
IPC
2
11
51
2
11
51
11
$1
Method
TM
38.0
1.2
50.5
1.0
6144 0.3
26.1
+ 1.0
36.0
49.2 + 1.3
35.2 + 1.0
45.1
54.5 + 1.0
IDC
20
48.2
1 1.2
52.7
0.7
0.6
32.5
22
43.7 1:00
54.9 1.1
46.7 1 09
50.2
0.6
64.5 1
HaBa
24
48.3
0.5
60.5
0/6
6.
43.6
1.5
49.0
60.1 1.4
47.4 2
58.0
0,9
64.4 1 0.6
AutoPalette
41.3 - 111
57.6
176210
L1
70.3
1 0.2
36.7
25
44.5 1.1
72.5 0.2
46.7
122
66.0 1 1.3
75.8 0.2
A.5 Number of colors vs Performance
Our method still demonstrates competitive performance, even when synthetic images are quantized into extremely low bits, as shown in Table 7 The test results for CIFAR10 IPC=10 is provided, from which we can observe slight performance gap when number of colors used for quantized images decreases to merely 8 colors per pixel (3 bits) from the original 256 colors pixel (8 bits). The results demonstrate that our color quantization method captures key features for synthetic data and appropriately condenses images into fewer colors.
14

---

Table Model performance when evaluated on the synthetic dataset using at most different number of colors.
#colors
256 (full)
64
32
16
4
2
Accuracy
66.8
66.2
66.1
65.9
64.6
62.1
55,3
Table 8: Test accuracy (%) in comparison with the traditional color quantization methods. Quantiza- tion methods are applied to synthetic dataset for CIFAR1O, when IPC=10 and 50.
IPC
10
Scale
2
16
32
64
Median Cut
42.5
46.4
52.4
55.6
57.6
60.7
OCTree
23.7
30.7
53.1
59.6
60.8
Palette Network
55.3
62.1
64.6
65.9
66.1
66.2
Table 9: Hyperparameters for our method based on Distribution matching (DM) framework.
Dataset
IPC
Synthetic batch size
Palette network LR Synthetic Image LR
ZCA
0.05
True
CIFARIO
10
0.05
True
0.05
10
True
0.05
True
CIFAR100
10
0.05
True
50
50
0.05
10
True
A.6
Traditional color Quantization Methods
Our palette network aims to identify the essential color characteristics of the synthetic dataset and represent it with fewer colors. To test the efficacy of the palette network, compare with other color quantization methods on DD tasks, including Median Cut [15] and OCTree [I]]. We conduct the experiments on CIFAR 10, and Table shows that the palette network achieves the superior performances under different IPC settings.
Table 10: Hyperparameters for method based Trajectory matching (TM) framework.
Synthetic Synthtic stps
Expent spocls
spach
Palete Syutheti Image Step
CIFAR10
t0m5
10-2
10m5
10-2
1000
Tree
CHARR10
35
0.05
1030
0.66
1030
10-2
True
ImugeNette
80
20
20
0.01
10000
10
30
Bibe
15

---

A.7 Visualization of Distilled Images
Figure 4: CIFARIO color condensed synthetic images with ZCA whitening.
Figure 5: CIFARIO color condensed synthetic images without ZCA whitening.
Figure 6: CIFAR10 synthetic images in 3-bit color depth
16

---

Figure 7: Color condensed synthetic images for ImageNette
Figure 8: Color condensed synthetic images for Image Woof
Figure 9: Color condensed synthetic images for ImageFruit
17

---

Figure 10: Color condensed synthetic images for ImageMeow
Figure 11: Color condensed synthetic images for ImageSquawk
Figure 12: Color condensed synthetic images for Image
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main contributions of the work clearly stated in the abstract and introductions, which is about exploring data distillation from color perspectives.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the last section of the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification: All assumptions and proof are clarified in the paper. For example, the proof of the conditional submodular gain function.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The experiment environments and hyper-parameters are included in the experi- ment section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All the datasets open sourced, and the code is provided through the link in abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc
aublic//uuides//odeetubiii wtt.eeeeuuuiiisioooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppppccggua.eeaeaeccpuee Cttelee//ooSSSSSeSSS.eeaa www.ssssiicggyooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experiment details are specified, and the choices of hyper-paramters are discussed.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are clearly reported. All experiments are conducted for multiple times and the mean accuracy and standard deviation are given.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computation resources are included in the experiment sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The work follows NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work has no societal impacts, since it is not related to any social activities.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper has no such risks, since the data and models are all
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the existing works are properly referenced.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New framework is well presented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
No crowdsourcing is involved in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: No human participant is involved in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "q72mxAnJWf", "reviewer": "Reviewer_5XJK", "summary": "This paper introduces ColorPalette, a framework that minimizes color redundancy at the individual image and overall dataset levels. At the image level, the palette networks generate condensed images in reduced color bit-width while at the dataset level, a color-guided initialization strategy is proposed. The experiments are done using various datasets and IPCs.", "questions": "1. How is the performance of AutoPalette on ImageNet-1K?\n2. Since the method falls into the parameterization category, given an IPC storage size, how many samples does AutoPalette generate?\n3. In Table 1, why AutoPalette inferior to DATM on CIFAR-100 at 50 IPC?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. A new direction for exploring DC is proposed. \n2. AutoPalette explores the possibility of performing DC in a reduced color space.\nThe paper is easy to understand.", "weaknesses": "1. AutoPalette seems like it is built on top of [1] with DC loss.\n2. Lack of experiment on large-scale dataset ImageNet-1K.\n\n[1] Learning to Structure an Image with Few Colors, Yunzhong Hou et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1**: *AutoPalette seems like it is built on top of [1] with DC loss*:\n\nThank you for bringing up this important question! While color reduction plays a significant role in our methodology, our work primarily focuses on addressing two unique challenges inherent in dataset distillation with low IPC (limited synthetic samples). These challenges are fundamentally different from training networks with complete datasets as discussed in [1]. Specifically, the challenges include:\n1. The optimization process in dataset distillation is extremely unstable, especially when employing a color reduction network. This instability can lead to the optimization process becoming trapped in local optima without capturing global information.\n2. The color reduction model in [1] tends to select certain colors, potentially resulting in biases towards these colors and capturing spurious features. This bias can hurt the generalization of the trained model.\n\nIn contrast, our method addresses these limitations by introducing two losses **tailored** for the dataset distillation process: (1) a color regularization term $\\mathcal{L}\\_{a}$ to enhance color consistency to reach global optima, and (2) a color balance loss $\\mathcal{L}\\_{b}$ to avoid biased color assignment.\n\nOur ablation studies, as presented in **Table 3** of our paper, demonstrate the significance of these loss functions. They are **critical** for guiding the network towards a more balanced and effective representation of the color space. Without these loss functions ($\\mathcal{L}\\_{a}$ and $\\mathcal{L}\\_{b}$), the test performance drops by **5.06%**. The inclusion of these two unique loss functions facilitates faster and more stable convergence, highlighting the main differences with [1].\n\n**Weakness 2 & Question 1**: *Lack of experiment on large-scale dataset ImageNet-1K*:\n\nThank you for your feedback regarding the importance of conducting experiments on large-scale datasets like ImageNet-1k. We have expanded our analysis beyond CIFAR10 and CIFAR100 by applying our method to higher-resolution subsets of the ImageNet dataset, such as **ImageNette** and **ImageWoof**, as shown in **Table 2** of our paper. \n\nConducting comprehensive experiments on the full ImageNet-1K datasets poses significant computational challenges. As highlighted in [6], distilling Imagenet-1k requires 4 NVIDIA A100 GPUs, each with 80 GB of memory. Unfortunately, these requirements don't allow for running all requested experiments in the rebuttal period. Therefore, to demonstrate the scalibility and efficacy of our proposed method on large-scale datasets, we conducted **addtional experiments** during the rebuttal period on the *Tiny ImageNet* dataset, which contains 200 classes with ($64\\times64$) images. We adapted AutoPalette to the distribution matching (DM) [2] approach with Image Per Class (IPC) values of 1 and 10. Our findings show significant improvements in test performance compared to baselines. Specifically, our method improved test performance from **3.9\\% to 7.02\\%** for $IPC=1$ and from **12.9\\% to 29.52\\% for $IPC=10$.** The table below illustrates the test performance results.\n\n| Method/IPC | 1         | 10        |\n|------------|-----------|-----------|\n| DM         | 3.9       | 12.9      |\n| Ours       | **7.02** | **29.52** |\n\nWe apologize for not able to conduct experiments on the full ImageNet-1K dataset at this time. We are eager to explore this for the camera-ready version.\n\n**Question 2**: *Given an IPC storage size, how many samples does AutoPalette generate?*\n\nThank you for insightful question. With a fixed IPC storage budget, our AutoPalette method achieves a fourfold $4\\times$ increase in the number of generated instances compared to the baseline. In constract, methods such as IDC [3] and HaBa [4] typically generate instances with a fivefold $5\\times$ increase, while FReD [5] achieves increases ranging from fourfold $4\\times$ to sixtheenfold $16\\times$. This demonstrates the efficiency of AutoPalette in optimizing sample generation within the constraints of a given storage size.\n\n**Question 3**: *In Table 1, why AutoPalette inferior to DATM on CIFAR-100 at 50 IPC?*\n\nWe appreciate your question regarding the performance comparison. Our method incorparates the trajectory mathcing strategy from DATM [6], where the matching steps are gradually increased during the distillation process. However, unlike DATM, we do not utilize the soft-labelling method and instead focus on exploring color features. This difference in approach may contribute to the slight performance discrepancy observed on CIFAR100 at 50 IPC. We believe that integrating soft-labelling could potentially enhance our method's performance in future work.\n\n\n\n[1] Learning to Structure an Image with Few Colors, Yunzhong Hou et al.\n\n[2] Dataset Condensation with Distribution Matching, Bo Zhao et al.\n\n[3] Dataset Condensation via Efficient Synthetic-Data Parameterization, Jang-Hyun Kim et al.\n\n[4] Dataset Distillation via Factorization, Songhua Liu et al.\n\n[5] Frequency Domain-based Dataset Distillation, Donghyeok Shin & Seungjae Shin et al.\n\n[6] Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching, Ziyao Guo & Kai Wang et al.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oIQYxFgdks", "reviewer": "Reviewer_uhQ2", "summary": "The paper titled introduces AutoPalette, a novel framework for dataset distillation (DD) that focuses on minimizing color redundancy at both the individual image and overall dataset levels. Authors propose a palette network to dynamically allocate colors from a reduced color space to each pixel, ensuring essential features are preserved. Additionally, a color-guided initialization strategy is developed to minimize redundancy among images, selecting representative images based on information gain. Comprehensive experiments on various datasets demonstrate the superior performance of the proposed color-aware DD compared to existing methods.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Color quantization is an interesting way for dataset distillation, the motivation of this paper is interesting.\n2. The methodology is well-defined, with clear explanations of the palette network and the color-guided initialization strategy.\n3. The framework is shown to be compatible with other DD methods, indicating its potential for broad application.", "weaknesses": "1. The paper does not discuss the potential impact of the method on the performance of larger dataset beyond the CIFAR-10 and CIFAR-100. These 2 datasets are two small and could not show the effectiveness of the proposed method.\n\n2. There is limited exploration of how the method handles imbalanced datasets or classes with unique color distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1**: *The paper does not discuss the potential impact of the method on the performance of larger dataset beyond the CIFAR-10 and CIFAR-100. These 2 datasets are two small and could not show the effectiveness of the proposed method.*\n\nWe appreciate the concern regarding the need for experiments on large-scale datasets. In addition to the experiments conducted on CIFAR10 and CIFAR100, we have also applied our method to higher-resolution subsets (**$128\\times128$**) of the ImageNet dataset, such as **ImageNette** and **ImageWoof**, as demonstrated in Table 2 of our paper. To further address the need for experiments on datasets with more classes, we conducted addtional experiments during the rebuttal period on the **Tiny ImageNet** dataset, which contains 200 classes with ($64\\times64$) images. Please kindly refer to our response to R1W2 (Reviewer 5XJK).\n\n**Weakness 2**: *There is limited exploration of how the method handles imbalanced datasets or classes with unique color distributions.*\n\nTo evaluate the effectiveness of AutoPalette on  imbalanced datasets, we created an imbalanced CIFAR10 dataset following the protocol described in [1]. Specifically, we resampled the CIFAR10 dataset so that the number of samples per class is determined by $N_{i} = N * \\alpha^{\\frac{i}{N_c}}$, where $N$ is the original number of images per class, $\\alpha$ is the scaling factor, $i$ indicates the i-th class, and $N_c$ is the number of classes. We used Distribution matching (DM) [2] as our baseline distillation method. The performance for IPC values of 1 and 10 is shown below. \n\n| Ratio α | Method/IPC | 1     | 10    |\n|---------|------------|-------|-------|\n| 0.01    | DM         | 25.91 | 48.01 |\n|         | Ours       | **35.60** | **59.53** |\n| 0.005   | DM         | 25.54 | 46.71 |\n|         | Ours       | **34.58** | **58.11** |\n\n\nFrom the results, we observed that when the scaling factor ($\\alpha$) is 0.01 (with the minimum number of images per class being 50 and the maximum 5000), the distillation performance remains relatively stable. However, when the factor ($\\alpha$) is further decreased to 0.005 (with the minimum number of images per class being 25), the performance slightly drops. These results demonstrate that our proposed method can consistently improves upon the baseline method across a broad range of imbalance factors and dataset imbalance settings.\n\nWe agree that addressing imbalanced datasets is an important challenge. We believe that dynamically allocating limited storage resources across classes, rather than using a fixed number of images for each class, could potentially address this issue.  We are excited to explore this direction in future work.\n\n[1] Dataset Card for CIFAR-10-LT (Long Tail), Huggingface. \n\n[2] Dataset Condensation with Distribution Matching, Bo Zhao et al.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4deAD6tAl8", "reviewer": "Reviewer_Jnju", "summary": "This paper introduces a straightforward yet effective dataset distillation method called AutoPalette. The method minimizes color redundancy at both the individual image level and the entire dataset level. At the image level, it trains the palette network by maximizing color loss and palette balance loss, thereby reducing color redundancy in images. At the dataset level, a color-guided initialization strategy is proposed to minimize color redundancy across the entire dataset. Extensive comparative and ablation experiments convincingly demonstrate the approach's effectiveness.", "questions": "- How does the efficiency of this method compare to other methods?\n- Why does directly optimizing the task loss lead to assigning pixels to a limited number of color buckets in lines 156-158?", "limitations": "The authors have discussed the limitations of their work in Section 5. However, they could provide more detailed descriptions of how these limitations might impact the results.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method outperforms other dataset distillation methods in most tasks, providing a new perspective on dataset distillation.\n- The experiments and ablation study seem well done. The paper's experiments are comprehensive, and the results of the ablation studies are convincing.", "weaknesses": "-  The paper could benefit from a more detailed explanation of the color loss and palette balance loss. It would be helpful to include an explanation of why the palette balance loss might achieve a more balanced color palette.\n-   The paper does not seem to explain why the similarity between the last layer gradients is measured instead of directly measuring the feature level similarity in the Color Guided Initialization Module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1**: *The paper could benefit from a more detailed explanation of the color loss and palette balance loss. It would be helpful to include an explanation of why the palette balance loss might achieve a more balanced color palette.*\n\nThank you for your insightful comment. The palette balance loss is designed to encourage the network to distribute pixels **uniformly** across all colors, maximizing the representational capacity of each color. Specifically, the palette balance loss is calculated as the **average entropy** of $m$ over the spatial dimensions, colors, and channels, where $m$ represents the probability distribution for assigning pixels to color buckets. When the palette balance loss is minimized, it results in an even distribution of pixels across buckets, implying that the pixel count for each color bucket is **approximately equal**. This encourages the network to generate a palette with a more balanced color distribution preventing any particular color from dominating the images.\n\nAdditionally, the maximum color loss is computed as the negative sum of the maximum confidences for each color bucket. This encourages each color to be selected by at least one pixel, thereby aiming to diversify the colors within each image. We will provide a more detailed explanation of these concepts in the revised version.\n\n**Weakness 2**: *The paper does not seem to explain why the similarity between the last layer gradients is measured instead of directly measuring the feature level similarity in the Color Guided Initialization Module.*\n\nThank you for the insightful question. Measuring the similarity among samples by the last layer gradient and feature similarity can be **largely correlated**. The major difference is that the last layer gradient also captures the joint interaction beteen feature space and label space, while feature similarity mainly focus on the feature space.\n\nTo explore this further, we conducted **additional experiments** using feature similarity in the graph cut initialization method. We applied color-reduced images through the models to obtain their feature representations and then computed the cosine similarities among these features. Due to time constraints, we applied our method to distribution matching (DM) [1] with IPC values of 1 and 10. As shown in Table below, the results indicate that using feature similarity respectively achieve 35.36\\% and 60.9\\% test performance on 1 IPC and 10 IPC experiments. \n\nThe results **align** with our assumption -- Measuring by the last layer gradient and feature similarity can be largely correlated. We hypothesize that both methods for computing similarities can lead to good performance, suggesting flexibility in choosing similarity approach. \n\n| IPC        | With Gradient  | With Feature        |\n|------------|-----------|-----------|\n| 1          | 35.5  | 35.36     |\n| 10         | 60.9      | 60.9      |\n\n**Question 1**: *How does the efficiency of this method compare to other methods?*\n\nThe additional overhead introduced by our color palette network during the forward and backward process is minimal. Our palette network consists of 3 convolutional layers designed to generate color-reduced synthetic images efficiently. Consequently, the increase in wall-time is marginal compared to vanilla baseline methods such as trajectory matching [1] and distribution matching [2]. We would like to further highlight that AutoPalette could be more efficient than other parameterization methods [3-5], which require reconstructing the distilled image at the test time. This efficiency makes AutoPalette a practical choice for various applications where computational resources are limited.\n\n**Question 2**: *Why does directly optimizing the task loss lead to assigning pixels to a limited number of color buckets in lines 156-158?*\n\nEmpirically, we observe that during the early stages of optimization, the palette network tends to assign pixels to only a few color buckets. As the optimization progresses, if it just focuses on minimizing the task loss (distillation loss), it often neglects the need for constraining the palette network and the need for a diverse color representation. This leads to the underutilization of the available color buckets, resulting in poor generalization capacity. \n\nTo address this, we introduce the color maximum loss and palette balance loss as additional constraints on the optimization of the palette network. These losses encourage the network to utilize the full range of color buckets more effectively, leading to a richer and more balanced color representation in the color-reduced images.\n\n[1] Dataset Condensation with Distribution Matching, Bo Zhao et al.\n\n[2] Dataset Distillation by Matching Training Trajectories, George Cazenavette et al.\n\n[3] Dataset Distillation via Factorization, Sonhua Liu et al.\n\n[4] Sparse Parameterization for Epitomic Dataset Distillation, Xing Wei & Anjia Cao et al.\n\n[5] Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks, Zhiwei Deng et al.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MNb021o0Pg", "reviewer": "Reviewer_VepF", "summary": "The authors propose AutoPalette, which reduces color redundancy in dataset distillation. They use a palette network and color-guided initialization to enhance training efficiency and performance by minimizing redundant color information in synthetic images and datasets.", "questions": "- While watching the condensed images in the Appendices, the CIFAR images are hard to perceptually recognize categories, but easy for Figures 7-9. I’m wondering why this perceptual difference emerges.\n\n- How did you decide the parameters, alpha, beta, and gamma in the experiments?", "limitations": "- The evaluation was mainly based on a relatively small number of image datasets. I’m not sure to what extent the condensed images change when applying recent large-scale image datasets.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Color redundancy is a fundamental aspect of natural scene images but is often overlooked in large-scale image analysis. This study focuses on the missing part, and the proposed method is effective.", "weaknesses": "- In the abstract, the authors summarize their framework as the one that minimizes color redundancy at the individual image and overall dataset levels. I think that’s a good summary. However, the description is not utilized when they introduce their framework in the main text. Although they describe it in the last section, it would be better to include the summary in the middle of the main, e.g., when introducing an overview or Figure 1.\n\n- I am confused a little about the definition of the color bit in this manuscript. The authors often describe the 8-bits for the original image (e.g., Figure 2). However, if the color bit is based on the number of color palettes, the original image should have 24 bits. \n\n- Typo: \"> can encoded in fewer bits” should be \"can be encoded”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1**: *In the abstract, the authors summarize their framework as the one that minimizes color redundancy at the individual image and overall dataset levels. I think that’s a good summary. However, the description is not utilized when they introduce their framework in the main text. Although they describe it in the last section, it would be better to include the summary in the middle of the main, e.g., when introducing an overview or Figure 1.*\n\nThank you for your valuable suggestion. We acknowledge that providing a summary of our framework earlier in the main text would improve the clarity and flow of our paper. In the revised version, we will include a concise summary of the framework before introducing the components to give readers a better understanding upfront. In addition, we appreciate your attention to detail and have noted the typo you pointed out. We will also correct this in the revised version.\n\n**Weakness 2**: *I am confused a little about the definition of the color bit in this manuscript. The authors often describe the 8-bits for the original image (e.g., Figure 2). However, if the color bit is based on the number of color palettes, the original image should have 24 bits.*\n\nThanks for highlighting this point of confusion. The 8 bits we refer to represent the storage space for each channel (Red, Green, Blue) of each pixel in the image. When considering all three RGB channels together, the total storage space indeed amounts to 24 bits per pixel. Thus, both descriptions refer to the same concept of image representation, just from slightly different perspectives. We will clarify this in the manuscript to ensure that readers understand the distinction between the per-channel and total bit depth for image representation.\n\n**Weakness 3**: *Typo: \"> can encoded in fewer bits” should be \"can be encoded”*\n\nThank you for pointing out the typo. We will correct this in the revised version of our paper.\n\n**Question 1**: *While watching the condensed images in the Appendices, the CIFAR images are hard to perceptually recognize categories, but easy for Figures 7-9. I’m wondering why this perceptual difference emerges.*\n\nThank you for your obsesrvation and opinion. We hypothesize that this perceptual difference arises due to the resolution difference between CIFAR images ($32\\times32$) and ImageNet images ($128\\times128$). CIFAR images have a lower resolution, which makes the resulting synthetic images appear blurrier and less distinct, similar to an aliasing effect in low-resolution images. Another factor contributing to this observation could be the reduction in the number of colors. With fewer colors, it can be harder for humans to perceptually recognize objects, as colors palys a significant role in distinguishing and identifying visual elements. On the other hand, ImageNet images with higher resolution, can capture more details, allowing the synthetic images to represent objects more clearly and making their categories easier to recognize. We empirically find that visualizing results from other baseline distillation methods, such as DM and TM, shows a similar observation, that image resolution impacts the perceptual clarity and recognizability of synthetic images.\n\n**Question 2**: *How did you decide the parameters, alpha, beta, and gamma in the experiments?*\n\nThank you for your question regarding the hyper-parameter settings. We conducted experiments to evaluate the sensitivity of our method to various values of these hyper-parameters. Specifically, we applied our method to distribution matching [1] with 10 IPC to test parameter sensitivity. In these experiments, we varied one parameter while keeping the other two fixed. The results shown in the tables below, indicate relatively stable performance within a range of $\\alpha$, $\\beta$ and $\\gamma$ values. We observe that $\\mathcal{L}_a$ has a slightly higher effect than the other two parameters. For instance, when gamma is set to 0.5 and 1.25, the test performance is respectively 58.08 \\% and 58.06 \\%. As $\\gamma$ increases, we see an improvement in performance, which converges to around 60.9 \\%. For the other two parameters, $\\alpha$ and $\\beta$, the sensitivity is lower, with optimal performance observed when both are around 3. This indicates that these parameters can be set within a reasonable range without significantly impacting the results.\n\n| $\\alpha$    | 0.3 | 0.75 | 1 | 1.5 | 3.0 | 6.0 |\n|:----------:|:----:|:------:|:---:|:-----:|:-----:|:-----:|\n| performance | 60.63 | 60.8 | 60.9 | 60.77 | 60.91 | 60.94 |\n\n| $\\beta$    | 0.3 | 0.75 | 1 | 1.5 | 3.0 | 6.0 |\n|:-----------:|:-----:|:------:|:---:|:-----:|:-----:|:-----:|\n| performance | 60.94 | 61.42 | 60.9 | 60.78 | 60.9 | 60.95 |\n\n\n| $\\gamma$    | 0.5 | 1.25 | 2.5 | 3.0 | 5.0 | 10.0 |\n|:-----------:|:-----:|:------:|:---:|:-----:|:-----:|:---:|\n| performance | 58.08 | 58.56 | 60.19 | 60.9 | 60.6 | 60.9 |\n\n\n**Limitation**: *The evaluation was mainly based on a relatively small number of image datasets. I’m not sure to what extent the condensed images change when applying recent large-scale image datasets.*\n\nWe appreciate the concern regarding the need for experiments on large-scale datasets. In addition to the experiments conducted on CIFAR10 and CIFAR100, we have applied our method to higher-resolution subsets ($128\\times128$) of the ImageNet dataset, such as **ImageNette** and **ImageWoof**, as demonstrated in **Table 2** of our paper. To further address the need for experiments on datasets with more classes, we conducted addtional experiments during the rebuttal period on the **Tiny ImageNet** dataset, which contains 200 classes with ($64\\times64$) images. Please kindly refer to our response to R1W2 (Reviewer 5XJK).\n\n\n[1] Dataset Condensation with Distribution Matching, Bo Zhao et al.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to extend our sincere gratitude to all the reviewers for their time and effort in reviewing our work. We deeply appreciate the insightful suggestions and feedback provided. We also thank for the acknowledgement of 1）our color-oriented redundancy reduction provides **a new perspective** in dataset distillation (*5XJK, uhQ2, Jnju, VepF*) 2) the proposed method is **effective** (*Jnju, VepF*) 3) our paper is **easy to understand** (*5XJK*) and methodology is **well defined** (*uhQ2*) 

Based on the common suggestions, we have conducted additional experiments during the rebuttal period, summarized as below:

- **Additional experiments on large-scale dataset Tiny ImageNet.** Please refer to our rebuttals to Reviewer 5XJK Weakness 1, Reviewer uhQ2 Weakness 1, Reviewer VepF Limitations.

- **Impact of hyperparameters $\alpha$, $\beta$ and $\gamma$?** Please refer to the rebuttal for Reviewer VepF Question 2.

- **Performance on the imbalanced dataset.** Please refer to the rebuttal for Reviewer uhQ2 Weakness 2.


Please find the point-to-point response in each individual reply. Thank you once again for your valuable feedback and support.

Meta-review: The paper introduces AutoPalette, a dataset distillation method that reduces color redundancy at both the image and dataset levels. Reviewers acknowledge that this approach offers a new perspective on dataset distillation and demonstrates effectiveness in experiments (5XJK, uhQ2, Jnju, VepF). However, concerns are raised about the lack of experiments on larger datasets like ImageNet-1K, which limits the demonstration of the method's scalability and effectiveness (5XJK, uhQ2, VepF). Suggestions include providing more detailed explanations of the color loss and palette balance loss (Jnju), addressing how the method handles imbalanced datasets (uhQ2), and clarifying definitions such as color bits (VepF). Questions about the method's efficiency compared to others are also noted (Jnju).

In the rebuttal the authors address concerns by explaining how their method differs from prior work, emphasizing their unique loss functions that improve optimization and generalization. They acknowledge computational constraints preventing ImageNet-1K experiments but provide additional results on Tiny ImageNet to demonstrate scalability. They clarify that AutoPalette generates four times more instances within a given storage budget and explain performance differences compared to other methods. Overall the reviewers' concerns were addressed and the AC recommends accept.
Decision: Accept (poster)