Title: SLED: Self-Supervised Dataset Distillation for Lightweight Experience Replay
Primary area: reinforcement learning
Keywords: Dataset Distillation, Reinforcement learning

Abstract: Experience Replay (ER) is central to off-policy reinforcement learning, but its reliance on massive buffers creates prohibitive storage and sampling costs. We introduce SLED, a self-supervised dataset distillation framework that replaces conventional replay with a compact, learnable synthetic dataset. SLED progressively shifts the agent’s training from real interactions to a small, self-evolving knowledge base by decoupling data writing from training sampling. On the writing side, a temporal schedule gradually substitutes real trajectories with optimized synthetic samples, leaving the buffer composed solely of distilled data. On the sampling side, a quota-based strategy shapes the training distribution, enabling a seamless transition from real-dominated to synthetic-dominated updates without altering the base algorithm. To preserve the long-term utility of synthetic data, SLED adopts an online-validated evolutionary optimization scheme: candidate synthetic datasets undergo brief parallel training trials, followed by real-environment evaluation, yielding a dataset-level fitness signal that guides their continual refinement. The framework is plug-and-play with mainstream off-policy methods. Overall, SLED systematically extends the idea of dataset distillation to the non-stationary regime of reinforcement learning, providing a practical alternative to large-scale replay buffers. Extensive experiments on DMControl, Habitat, and Atari confirm that SLED delivers superior efficiency and scalability over existing ER approaches, demonstrating its broad effectiveness across diverse domains.