ICML 2025

Sliding Puzzles Gym: A Scalable Benchmark for
State Representation in Visual Reinforcement Learning
Bryan L. M. de Oliveira 1 2 Luana G. B. Martins 1 Bruno Brand˜ao 1 2 Murilo L. da Luz 1 2
Telma W. de L. Soares 1 2 Luckeciano C. Melo 1 3
Abstract
Effective visual representation learning is crucial
for reinforcement learning (RL) agents to extract
task-relevant information from raw sensory in-
puts and generalize across diverse environments.
However, existing RL benchmarks lack the ability
to systematically evaluate representation learning
capabilities in isolation from other learning chal-
lenges. To address this gap, we introduce the Slid-
ing Puzzles Gym (SPGym), a novel benchmark
that transforms the classic 8-tile puzzle into a vi-
sual RL task with images drawn from arbitrarily
large datasets. SPGym’s key innovation lies in its
ability to precisely control representation learning
complexity through adjustable grid sizes and im-
age pools, while maintaining fixed environment
dynamics, observation, and action spaces. This
design enables researchers to isolate and scale the
visual representation challenge independently of
other learning components. Through extensive ex-
periments with model-free and model-based RL
algorithms, we uncover fundamental limitations
in current methods’ ability to handle visual diver-
sity. As we increase the pool of possible images,
all algorithms exhibit in- and out-of-distribution
performance degradation, with sophisticated rep-
resentation learning techniques often underper-
forming simpler approaches like data augmen-
tation. These findings highlight critical gaps in
visual representation learning for RL and estab-
lish SPGym as a valuable tool for driving progress
in robust, generalizable decision-making systems.
1Advanced Knowledge Center for Immersive Technologies
– AKCIT, Brazil 2Institute of Informatics, Federal University of
Goi´as, Goiˆania, Brazil 3OATML, University of Oxford, United
Kingdom. Correspondence to: Bryan L. M. d...