Title: Color-Oriented Redundancy Reduction in Dataset Distillation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Computer Vision

Data Distillation

Parameterization
Abstract: Dataset Distillation (DD) is designed to generate condensed representations of extensive image datasets, enhancing training efficiency. Despite recent advances, there remains considerable potential for improvement, particularly in addressing the notable redundancy within the color space of distilled images. In this paper, we propose a two-fold optimization strategy to minimize color redundancy at the individual image and overall dataset levels, respectively. At the image level, we employ a palette network, a specialized neural network, to dynamically allocate colors from a reduced color space to each pixel. The palette network identifies essential areas in synthetic images for model training, and consequently assigns more unique colors to them. At the dataset level, we develop a color-guided initialization strategy to minimize redundancy among images. Representative images with the least replicated color patterns are selected based on the information gain. A comprehensive performance study involving various datasets and evaluation scenarios is conducted, demonstrating the superior performance of our proposed color-aware DD compared to existing DD methods.


Peer reviews:
comments: SUMMARY:
The authors propose AutoPalette, which reduces color redundancy in dataset distillation. They use a palette network and color-guided initialization to enhance training efficiency and performance by minimizing redundant color information in synthetic images and datasets.

STRENGTHS:
Color redundancy is a fundamental aspect of natural scene images but is often overlooked in large-scale image analysis. This study focuses on the missing part, and the proposed method is effective.

WEAKNESSES:
- In the abstract, the authors summarize their framework as the one that minimizes color redundancy at the individual image and overall dataset levels. I think that’s a good summary. However, the description is not utilized when they introduce their framework in the main text. Although they describe it in the last section, it would be better to include the summary in the middle of the main, e.g., when introducing an overview or Figure 1.

- I am confused a little about the definition of the color bit in this manuscript. The authors often describe the 8-bits for the original image (e.g., Figure 2). However, if the color bit is based on the number of color palettes, the original image should have 24 bits. 

- Typo: "> can encoded in fewer bits” should be "can be encoded”

QUESTIONS:
- While watching the condensed images in the Appendices, the CIFAR images are hard to perceptually recognize categories, but easy for Figures 7-9. I’m wondering why this perceptual difference emerges.

- How did you decide the parameters, alpha, beta, and gamma in the experiments?

LIMITATIONS:
- The evaluation was mainly based on a relatively small number of image datasets. I’m not sure to what extent the condensed images change when applying recent large-scale image datasets.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
This paper introduces a straightforward yet effective dataset distillation method called AutoPalette. The method minimizes color redundancy at both the individual image level and the entire dataset level. At the image level, it trains the palette network by maximizing color loss and palette balance loss, thereby reducing color redundancy in images. At the dataset level, a color-guided initialization strategy is proposed to minimize color redundancy across the entire dataset. Extensive comparative and ablation experiments convincingly demonstrate the approach's effectiveness.

STRENGTHS:
- The proposed method outperforms other dataset distillation methods in most tasks, providing a new perspective on dataset distillation.
- The experiments and ablation study seem well done. The paper's experiments are comprehensive, and the results of the ablation studies are convincing.

WEAKNESSES:
-  The paper could benefit from a more detailed explanation of the color loss and palette balance loss. It would be helpful to include an explanation of why the palette balance loss might achieve a more balanced color palette.
-   The paper does not seem to explain why the similarity between the last layer gradients is measured instead of directly measuring the feature level similarity in the Color Guided Initialization Module.

QUESTIONS:
- How does the efficiency of this method compare to other methods?
- Why does directly optimizing the task loss lead to assigning pixels to a limited number of color buckets in lines 156-158?

LIMITATIONS:
The authors have discussed the limitations of their work in Section 5. However, they could provide more detailed descriptions of how these limitations might impact the results.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
The paper titled introduces AutoPalette, a novel framework for dataset distillation (DD) that focuses on minimizing color redundancy at both the individual image and overall dataset levels. Authors propose a palette network to dynamically allocate colors from a reduced color space to each pixel, ensuring essential features are preserved. Additionally, a color-guided initialization strategy is developed to minimize redundancy among images, selecting representative images based on information gain. Comprehensive experiments on various datasets demonstrate the superior performance of the proposed color-aware DD compared to existing methods.

STRENGTHS:
1. Color quantization is an interesting way for dataset distillation, the motivation of this paper is interesting.
2. The methodology is well-defined, with clear explanations of the palette network and the color-guided initialization strategy.
3. The framework is shown to be compatible with other DD methods, indicating its potential for broad application.

WEAKNESSES:
1. The paper does not discuss the potential impact of the method on the performance of larger dataset beyond the CIFAR-10 and CIFAR-100. These 2 datasets are two small and could not show the effectiveness of the proposed method.

2. There is limited exploration of how the method handles imbalanced datasets or classes with unique color distributions.

QUESTIONS:
See weakness.

LIMITATIONS:
See weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

comments: SUMMARY:
This paper introduces ColorPalette, a framework that minimizes color redundancy at the individual image and overall dataset levels. At the image level, the palette networks generate condensed images in reduced color bit-width while at the dataset level, a color-guided initialization strategy is proposed. The experiments are done using various datasets and IPCs.

STRENGTHS:
1. A new direction for exploring DC is proposed. 
2. AutoPalette explores the possibility of performing DC in a reduced color space.
The paper is easy to understand.

WEAKNESSES:
1. AutoPalette seems like it is built on top of [1] with DC loss.
2. Lack of experiment on large-scale dataset ImageNet-1K.

[1] Learning to Structure an Image with Few Colors, Yunzhong Hou et al.

QUESTIONS:
1. How is the performance of AutoPalette on ImageNet-1K?
2. Since the method falls into the parameterization category, given an IPC storage size, how many samples does AutoPalette generate?
3. In Table 1, why AutoPalette inferior to DATM on CIFAR-100 at 50 IPC?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-17