Title: Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Models
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Synthetic data generation, Image augmentation, Diffusion models

Abstract: Synthetically augmenting training datasets with diffusion models has been an
effective strategy for improving generalization of image classifiers. However,
existing techniques struggle to ensure the diversity of generation and increase the
size of the data by up to 10-30x to improve the in-distribution performance. In this
work, we show that synthetically augmenting part of the data that is not learned
early in training with faithful images—containing same features but different
noise—outperforms augmenting the entire dataset. By analyzing a two-layer CNN,
we prove that this strategy improves generalization by promoting homogeneity in
feature learning speed without amplifying noise. Our extensive experiments show
that by augmenting only 30%-40% of the data, our method boosts generalization
by up to 2.8% in a variety of scenarios, including training ResNet, ViT, ConvNeXt,
and Swin Transformer on CIFAR-10/100, and TinyImageNet, with various
optimizers including SGD and SAM. Notably, our method applied with SGD
outperforms the SOTA optimizer, SAM, on CIFAR-100 and TinyImageNet.

TL;DR: We propose a novel targeted image augmentation method using synthetic images from diffusion models