Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
Authors: Bryan Lincoln Marques de Oliveira
Luana Guedes Barros Martins
Bruno Brandão
Murilo Lopes da Luz
telma woerle de lima soares
Luckeciano Carvalho Melo
Venue: ICML 2025 poster (2025)
Keywords: reinforcement learning
representation learning
benchmark
Primary area: reinforcement_learning->deep_rl
TL;DR: SPGym extends the 8-tile puzzle to evaluate RL agents by scaling representation learning complexity while keeping environment dynamics fixed, revealing opportunities for advancing representation learning for decision-making research.

Abstract: Effective visual representation learning is crucial for reinforcement learning (RL) agents to extract task-relevant information from raw sensory inputs and generalize across diverse environments. However, existing RL benchmarks lack the ability to systematically evaluate representation learning capabilities in isolation from other learning challenges. To address this gap, we introduce the Sliding Puzzles Gym (SPGym), a novel benchmark that transforms the classic 8-tile puzzle into a visual RL task with images drawn from arbitrarily large datasets. SPGym's key innovation lies in its ability to precisely control representation learning complexity through adjustable grid sizes and image pools, while maintaining fixed environment dynamics, observation, and action spaces. This design enables researchers to isolate and scale the visual representation challenge independently of other learning components. Through extensive experiments with model-free and model-based RL algorithms, we uncover fundamental limitations in current methods' ability to handle visual diversity. As we increase the pool of possible images, all algorithms exhibit in- and out-of-distribution performance degradation, with sophisticated representation learning techniques often underperforming simpler approaches like data augmentation. These findings highlight critical gaps in visual representation learning for RL and establish SPGym as a valuable tool for driving progress in robust, generalizable decision-making systems.

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
Bryan L. M. de Oliveira 12 Luana G. B. Martins' Bruno Brandão 12 Murilo L. da Luz" 12
Telma W. de L. Soares
Luckeciano C. Melo
Abstract
1. Introduction
Learning meaningful representations from raw sensory in- puts, such as visual data, fundamental to reinforcement learning (RL) agents' ability to generalize across different tasks in complex, open- open-world environments (Bengio et al., 2013; Lesort et al., 2018). In visual RL, agents must process high high-dimensional pixel data, extract useful features, and uti- lize these features for decision-making (Mnih et al., 2015; Yarats et al., 2021a). This becomes especially crucial as real-world applications demand adaptability to unstructured and diverse observations. However, measuring an agent's representation learning capabilities independently of other learning tasks, such as policy optimization dynamics modeling, remains key challenge in RL benchmarks.
Effective visual representation learning is crucial for reinforcement learning (RL. agents to extract task-rclevant information from raw sensory in- puts and generalize across diverse environments. However, existing RL benchmarks lack the ability to systematically evaluate representation learning capabilittis in isolation from other learning chal- lenges. To address this gap, we introduce the Slid- ing Puzzles Gym (SPGym), novel benchmark that transforms the classic 8-tile puzzle into a vi- sual RL task with images drawn from arbitrarily large datasets. SPGym's 's key innovation lies in its ability to precisely control representation learning complexity through adjustable grid sizes and im- age pools, while maintaining fixed environment dynamies, observation, and action spaces. This design enables researchers isolate and scale the visual representation challenge independently of other learning components. Through extensive ex- periments with model-free and model-based RL algorithms, we uncover fundamental limitations in current methods' ability to handle visual diver- sity. As we increase the pool of possible images, all algorithms exhibit in- and out-of--istribune distributiion performance degradation, with sophisticated rep- resentation learning techniques often underper- forming simpler approaches like data augmen- tation. These findings highlight critical gaps in visual representation learning for RL and estab- lish SPGym as valuable tool for driving progress in robust, generalizable decision-making systems.
While traditional RL benchmarks such as Atari (Bellemare et al., 2013) and DeepMind Control Suite (Tassa et al., 2018) offer valuable evaluation platforms for overall agent effec- tiveness, their performance metries inherently intertwine representation learning with policy optimization and envi- ronment dynamics. More recent dedicated benchmarks for visual learning, though promising. fall short in their capac- ity to precisely modulate visual complexity. For instance, ProcGen (Cobbe et al., 2020) simultaneously modifies both visual and task difficulty, obscuring the specific impact of representation learning, while the Distracting Control Suite (Stone et al., 2021) introduces visual distractors that are unrelated to the main task and can be safely disregarded by the model. Consequently, current benchmarks lack the necessary precision to systematically assess an agent's 's abil- ity to acquire task-relevant visual representations.
To address this gap, we introduce the Sliding Puzzles Gym (SPGym)', an open-source benchmark created evaluate how agents handle increasingly diverse visual observations across training runs. As depicted in Figure 1, SPGym trans- forms the classic 8-tile puzzle into visual RL challenge through three central design principles: (1) preserving con- sistent environment dynamics regardless of difficulty, ensur- ing that the underlying task remains unchanged; (2) enabling
'Avvnceed Knowledge Center for Immersive Technologies AKCIT, Brazil "Institute of Informatics, Federal University of Goiás, Goiània, Brazil 'OTMLL, University of Oxford, United Kingdom. Correspondence to: Bryan L. M. de Oliveira <bryanlin- coin @ discentc.ufg.br>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Available bryanoliveiras wwwiiiinnnuzzzllee
at https: / /ttpt/tttttu./////// com/

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
Dataset
Agent
Observation <IMAGE>
Action LEFT
Reward
Observation
Reward
Observation
Sample Run Pool
-0.6
<IMGE
<IMAGE>
images
Wrappers
Environment Dynamics
5
2
5
2
2
3
Seed
7
3
4
7
4
4
6
SPGym
8 6 1
8
6
1
7 8
Figure Overview of SPGym. The framework extends 8-tile puzzle by replacing numbered tiles with image patches. each images and, at each episode, it randomly selects of those images to form the observations.
training run, SPGym samples pool of While we scale visual diversity by adjusting the pool size, the task and environment dynamics remain fixed.
precise control over visual complexity through adjustable grid dimensions and image pool sizes; and (3) establishing clear success metric based on puzzle completion. This carefully engineered framework provides researchers with controlled environment systematically examine how agent performance degrades as visual diversity increases, offering insights into the fundamental limitations of their representational abilities while learning from experience.
Contributions. We make three key contributions to visual RL research: (1) we introduce SPGym, a novel benchmark that systematically evaluates representation learning by seal- ing visual complexity while keeping environment dynamics constant; (2) we conduct extensive empirical analysis of oftte-of-thh-art methods, uncovering critical limitations in their ability to process diverse visual inputs; and (3) we provide fundamental insights into the challenges of scaling visual RL, offering directions for advancing representation learning in decision making systems.
Our experiments show that SPGym effectively distinguishes agents by their out-of-the-box representation learning abil- ities, with strong correlations between linear probing curacy of learned encoders and task performance. While pretraining and data augmentation are beneficial, many ad- vanced methods with standard configurations for discrete visual RL underperform, suggesting either a need for task- specific tuning or fundamental mismatches with SPGym's sisuul-structural dynamics. More concerningly, our tiered generalization analysis exposes fundamental limitations: on SPGym, agents that master the training images fail trans- fer to unseen ones, even when trained on larger and more diverse image pools. In fact, performance often degrades as visual diversity increases, indicating that current meth- ods struggle to learn truly generalizable representations. further demonstrate SPGym's extensibility through ex- periments with procedurally generated images and larger puzzles, maintaining fixed observation/action spaces while expanding state space and visual diversity. These findings expose critical gaps in current visual RL methods and estab- lish SPGym as a valuable tool for ad advancing robust, gener- alizable artificial agents.
2. Related Work
Traditional RL benchmarks. Visual RL environments such as the Arcade Learning Environment (Bellemare et al., 2013), DeepMind Control Suite (Tassa et al., 2018), Deep- Mind Lab (Beattic et al., 2016), and CARLA (Dosovitskiy et al., 2017) are widely used for pixel-based agent evalua- tion. These platforms have driven progress visual repre- sentation learning for ing (Higgins et al., 2017: Stooke et al., 2021; Schwarzer et al., 2021), contrastive methods (Laskin et al., 2020b), self- supervised prediction (Schwarzer et al., 2020), and world models (Hafner et al., 2025). However, by measuring overall agent effectiveness, which relies on learning representations, policies, and dynamics, the specific impact of advances in representation learning for RL may be obseured. SPGym ad- dresses this drawback by regulating visual complexity while maintaining constant environment dynamics, allowing for targeted assessment of representation learning approaches for decision- makinng.
including pretrain-
2

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
Specialized benchmarks for visual RL. Recent bench- marks have sought to better assess visual generalization and robustness of RL agents. ProcGen (Cobbe et al., 2020) employs procedurally generated levels. but its difficulty scaling simultaneously affects both visual and task com- plexity. The Distracting Control Suite (Stone et al., 2021) evaluates robustness to visual variations, but these serve as task-irrelevant distractions that agents can safely learn to ignore while solving core tasks. COOM (Tomilin al., (022) provides continual learning benchmark for embod- ied pixel-based RI. in 3D environments, focusing on catas- trophic forgetting and knowledge transfer across tasks rather than isolating representation learning in a single task. How- ever, these approaches do not make visual understanding essential for task success. SPGym overeomes this limita- tion by making visual coherency fundamental to puzzle solving while maintaining fixed task complexity. Through controlled scaling of visual diversity in puzzle tiles, SPGym ensures that the representation learning challenges are tied to task completion, enabling a more precise evaluation of RL agents' visual learning capabilities.
3. The Sliding Puzzles Gym
SPGym extends the classic sliding tile puzzle, a game where players rearrange shuffled tiles on grid by sliding a tile into an adjacent empty space, with the goal of restoring ordered configuration (Figure 1). Our framework gen- eralizes this setup by supporting configurable H x W grid dimensions (from 2 x 2 upwards) and diverse observation modalities. In our experiments, we primarily use 3 x 3 grid where tiles are image patches, and the agent receives the composite image of the grid as input. Our implemen- tation builds on the Gym (Brockman, 2016) interface for modularity between environment and agent.
Formalization. We formulate SPGym as a partially ob- servable Markov decision process (POMDP) defined by the tuple AS R, S2 O), where S is the finite state space representing all solvable puzzle configurations, A is the discrete action space (tile movements), P: x A + S defines the deterministic transition dynamics, R: S R is the reward function, So is the initial state distribution (e.g.. uniform over S), Q is the observation space, and O S x I + Ω is the observation function parameter- ized by data pool I, which in this work consists of images.
Puzzle-based benchmarks. Estermann et al. (2024) demonstrated puzzle-based environments' value for evaluat- ing neural algorithmic reasoning, focusing on discrete state spaces. Though they tested pixel observations with sparse rewards, agents struggled with basic visual inputs. This work revealed puzzles' potential and challenges for visual learning, showing the need for systematic representation learning evaluation in these controlled settings. SPGym advances this research direction by incorporating rich vi- sual observations while maintaining the controlled nature of puzzle environments, facilitating systematic assessment of representation learning in the visual domain.
State space and observations. Crucially, agents do not have direct access to states s E S and must learn policies based solely on observations in Q. For visual observations, each training run begins by sampling p images from a pre- defined dataset to form an image pool I. At the start of each episode, we select a random image E I, partition it into H x W distinct patches corresponding to the puzzle tiles, and assign each patch to a tile position according to the current state s. The observation function ((s, i) then renders the composite image by arranging these patches according to the puzzle configuration. The agent's task is to reconstruct the original image, testing its ability to form compositional representations from visual input.
Methods for solving the sliding tile puzzle. Classical approaches to sliding puzzles employ search algorithms such as A and IDA#, leveraging domain-specific heuristics like the Manhattan distance (Korf, 1985; Burns et al., 2012; Lee & See, 2022). These methods guarantee optimal solu- tions but require direct access to the internal puzzle state and are often computationally intensive. Deep RL offers scalable alternative by learning effective strategies without hand-crafte hcuristics (Agostinelli et al., 2019; Moon & Cho, 2024; Estermann et al., 2024), though prior work has primarily focused on discrete state representations rather than learning directly from visual observations (Agostinelli et al., 2019; Moon & Cho, 2024). We build on these meth- ods by using the Manhattan distance as the basis for our reward function, providing well-shaped learning signal for RL agents. In SPGym, we evaluate standard RL algorithms where agents must learn solely from pixel observations, without access to internal states, to assess their out-of-the- box performance in this challenging setting.
This formulation provides two independent mechanisms for controlling complexity: (1) varying the pool size p adjusts the diversity of observations by changing the number of available images in I. directly affecting [S2], and (2) mod- ifying grid dimensions H x W alters both the state space size [S] and the visual complexity of individual observations. Both mechanisms operate while keeping the underlying tran- sition dynamics P, action space A, and reward function R invariant. SPGym is mataset-agnostte supporting any image dataset, including procedurally generated ones.
Action space and dynamics. The action space A con- tains four possible actions: UP, DOWN, LEFT, or RIGHT, which move the corresponding tile to the empty space in a discrete manner. Once the agent selects a tile to move, the dynamics P define the next puzzle state in a predictable and
3

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
deterministic way, as illustrated in Figure 1.
0
5
2
0
0
Reward function. The reward function R provides dense feedback to guide agent learning while accommodating the non-monotonic nature of optimal puzzle-solving paths. Fol- lowing established approaches in the sliding puzzle liter- ature (Korf, 1985; Burns et al., 2012; Lee & See, 2022; Moon & Cho, 2024), we base our reward on the Manhattan distance between each tile's current position and its target position, normalized across all tiles. This distance metric is widely adopted computational solutions to sliding puzzles due to its efficiency and effectiveness in capturing progress toward the goal state. Specifically, at each time step, the reward is computed as follows:
7 3
4
1
0
8
6
1
0
0
State
One-hot
Image Overlay
Figure 2. Different observation modalities in SPGym. Each modality presents a unique challenge for representation learning. The three presented observations represent the same puzzle state. We facus our experiments on image-hased observations.
While primary experiments use 3 x 3 image-based puz- zles, SPGym supports larger grids and other observation modalities, such as one-hot encodings (Figure 2), to isolate specific research questions.
D, if action is valid
R(s)= =
1, if action is invalid with (1)
+1, puzzle solved
The primary scaling mechanism in SPGym isolates the chal- lenge of representation learning by increasing visual diver- sity, illustrated in Figure 3, while keeping all other task components fixed across training runs. As the image pool size grows, agents require more samples to solve the task (Table 1), even though the state space, action space, and transition dynamics remain unchanged. By design, this increased difficulty stems directly from the representation learning challenge. We further confirm this link through lin- ear probe analysis (Appendix C.1.2), which shows a strong correlation between representation quality and task perfor- mance. This controlled scaling provides a stress test for vi- sual learning that is not possible in settings with fixed visual inputs or state- based observations (see Appendix C.I.1).
E Σ/= |us - -W,] + (uij - vizl
D
(2)
H Σ W j= max(i, H i) max(j W -))
Here, (us_j, )ig)) represents the current position of the tile at index (i, (uf v(,)) is its target position. D is the normalized Manhattan distance between current and target positions. Invalid actions, such as attempting to move tile outside the grid boundaries, result in penalty of -1 and don't alter the puzzle state. For example, in Figure 1, the DOWN and RIGHT actions would be invalid for the first state. Successfully solving the puzzle rewards the agent with- +1 and terminates the episode. This formulation pro- vides a well-behaved learning signal between [-1,+]] and encourages solving the puzzle in minimal steps.
The secondary scaling mechanism modulates the search challenge by adjusting grid dimensions. Increasing the grid size from 3 x 3 to 4 x 4 dramatically expands the state space complexity and, as result, the number of steps required to solve the task (see Table 3). For example, 3 x 3 puzzles have approximately 3-1 111 1.81 10 possible states, whereas x 4 puzzles pose a much greater challenge with roughly - 1013 possible states. This increase in grid size also makes representation learning more difficult, since the agent must correctly map and position greater number of image patches, modeling larger observation space. However, the size and shape of each rendered observation (i.e., the obser- vation dimensions) remain unchanged, as does the action space (still four directional moves), and the task dynamics continue be deterministic and fully predictable. While this scaling mechanism enables evaluation of exploration and policy-learning capabilities, our experiments show that 3 x 3 grids already provide sufficient discriminative power for assessing visual representation learning.
Initial state distribution. SPGym's 's initial state distribu- tion Sy can be defined through two methods. The primary method generates a uniformly random x W array where all tiles (including the blank tile) are placed randomly, and ensures solvability by adjusting the puzzle's parity through swapping the first two tiles when needed (Johnson & Story, 1879). The second method, designed support curriculum learning, starts from solved state and applies sequence of random valid moves to reach the initial configuration. Although this approach guarantees solvability due to the reversibility of moves, it becomes computationally expen- sive for larger grid sizes due to the sequential nature of move application. Given these efficiency considerations, we exclusively employ the first method in experiments.
Diversity scalability and extensibility. As previously mentioned, SPGym provides two orthogonal mechanisms for scaling task difficulty: visual diversity and grid size.
By maintaining consistent environment dynamics across these scaling mechanisms, SPGym creates controlled ex-

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
100% success rate for 100 consecutive episodes, indicating task completion. This early termination serves two purposes: it enables out-of-distribution evaluation before extreme en- coder overfitting occurs, and it reduces computational costs for running the comprehensive set of experiments. Fach ex- periment is conducted 5 times with different random seeds, and we report the mean + 1.96 standard errors (95% confi- dence interval).
Image pool size
We evaluate sample efficiency using pools of 1, 5, and 10 images across all agent variants. For standard agents, we additionally test with progressively larger pools up to 100 images or until final performance is less than 80% success rate after the full 10M training steps. This protocol allows us to analyze both the effectiveness and scalability of different algorithms and representation learning methods.
10
1
3
Algorithms and variants. We explore three distinct al- gorithmic approaches: Soft Actor-Critic (SAC) (Haarnoja et al., 201 1a), Proximal Policy Optimization (PPO) (Schul- man et al., 2017), and DreamerV3 (Hafner et al.. 2025), each representing different strategies for learning from vi- sual observations. For each algorithm, we evaluate several representation learning variants to assess their effectiveness in handling visual diversity.
Episode
Figure The visual diversity scalability of SPGym. Each row presents the first observation of 3 different episodes of the same training run. The visual diversity scales with the size of the image pool. Crucially, we keep the grid size fixed, ensuring that the increased difficulty comes solely from the agent's need to handle larger variety of visual observations.
For SAC, we begin with the standard implementation for discrete action spaces proposed by Christodoulou (2019). We then examine several representation learning variants: RAD (Laskin et al., 2020a), which employs data augmenta- tion; CURL (Laskin et al., 2020b), which uses contrastive learning: SPR (Schwarzer et al., 2020), which incorporates self-supervised prediction; DBC (Zhang al., 2021), which focuses state metric learning: SAC-AE and SAC-VAE (Yarats et al., 2021b), which utilize reconstruction-base learning: and a simple baseline (SB) proposed by Tomar et al. (2023), which implements a simplified approach with reward and transition prediction.
perimental setting where performance variations can be more directly attributed an agent's representation learn- ing and generalization capabilities.
4. Methods
In this section, we detail the evaluation protocol for various RL agents within SPGym, focusing on their ability to handle increasing visual diversity across training runs.
Experimental setup. We sample images from ImageNet- Ik's s validation split (Russakovsky et al., 2015) to construct visual observations, resizing each image to 84 x 84 pixels and normalizing values [0, 1]. To isolate visual diversity, we fix the puzzle size to 3 x 3 and vary only the image pool size p. For each training run, we randomly sample p distinct images to create a fixed image pool I. At episode start, we randomly select one image from this pool to generate the puzzle observations. We cap the number of environment steps to 10M and limit episodes 1,000 steps.
For PPO, we evaluate three encoder configurations: the standard version with random initialization, a variant pre- trained on the same image distribution (in-distribution, ID), and variant pretrained on different image distribution (out-of-diste distribbtiion, OOD). These pretrained variants re- spectively provide an upper bound on expected pretraining performance and help establish the potential benefits of generally pretrained encoders, though we acknowledge this may make direct sample efficiency comparisons with other methods less fair.
Our analysis focuses sample efficiency, measured by the number of environment steps required to solve the puzzle (lower is better). We calculate this metric by averaging the steps needed to reach 80% success rate across all parallel en- vironments in a run, then average this number across seeds. We terminate training runs early when an agent maintains
For DreamerV3, we compare the standard version against a variant without decoder gradients to evaluate the impact of the reconstruction objective on performance. Detailed descriptions for all agents and their learning objectives are provided in Appendix B.
5

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
Hyperpaaaameter. RAD, CURL, and SPR require data augmentation. We apply these augmentations to observa- tions after sampling them from the replay buffer. For each algorithm, we conducted individual augmentation searches with the objective of maximizing sample efficiency (detailed in Appendix A.4.2). These experiments consistently con- verged to a simple two-step pipeline, which we use through- out all evaluations: first converting to grayscale with 20% probability, then randomly shuffling the color channels.
Table 1. Million steps to reach 80% % success rate across pool sizes during training. Lower is better. Best performing variant for each algorithm and pool size is highlighted in bold.
Agent
Pool
Pool 5
Pool 10
PPO
1.75=0.4
7.80=1008
9.7320.36
PPO + PT (ID)
0.95.0.21
5,55
9.17=1.10
PPO + PT (OOD)
1.3420.42
7.03=1.0
9.70=0.41
SAC
0.33=0.07
=0.12
2.03=0.38
SAC + RAD
0.24
0.4220.06
0.8220.18
We evaluate out-of-the-box performance of existing ap- proaches in SPGym by adopting neural architectures and hyperparameters from established visual discrete control im- plementations. Our base architecture uses three-layer CNN encoders with mirrored deconvolutional decoders, while actor, critic, and auxiliary components employ multi-layer perceptrons (MLPs).
SAC + CURL,
0.46=0.10
1.56=0.31
5.24+1.92
SAC + SPR
2.0920.81
3.68=1.6 1.6
10.0000.00
SAC DBC
0.99+0.2
1. 1.12:0.22
2. 3:0.41
SAC + AE
1.0420.24
1.0220.19
2.01
SAC + VAE
1.13a0.14
5.30
10.00s0.00
SAC + SB
0.98=0888
2.08=0.30
10.00+0.00
DreamerV3
0.42+0.06
1.23:0.20
2.44
For SAC-hased agents, we follow Yarats et al. (2021b) and Tomar et al. (2023) by blocking actor gradients through the encoder to prevent representation collapse, while allowing critic and auxiliary gradients. PPO and DreamerV3 main- tain their original gradient flow patterns. While PPO and DreamerV3 worked robustly with default configurations, SAC-based agents required tuning of the temperature pa- rameter a (Appendix A.4.1). We found a fixed value of 0.05 to work best across all SAC variants, as the automatic tuning from Haarnoja et al. (2018b) proved ineffective here.
Dreamer DreamerV3wlor V3 dec.
1.13=0.12
1.79=0.61
=.57
encoders in this setup, we evaluate PPO with two types of pretrained encoders. It is important to acknowledge that di- rect comparisons of sample efficiency with encoders trained from scratch can be nuanced, as pretrained encoders have inherently been exposed to significantly more data during their pretraining phase. The in-distribution pretrained en- coder (PT (ID)) for PPO, intended as a proxy to estimate an approximate upper bound on performance achievable with pretraining, significantly boosts sample efficiency across all tested pool sizes. The out-of-distribution pretrained encoder (PT (OOD)), designed to represent the average performance one might expect from generally pretrained encoder, offers more modest gains which also decrease with larger pools. Notably, however, the rate at which performance degrades as the image pool size increases appears to be similar re- gardless of whether pretraining is used or not.
Where applicable, we preserve uniform hyperparameters while drawing algorithm-specife configurations from their respective source papers. The publicly available code repos- itory² extends both CleanRL (Huang et al., 2022) and the official DreamerV3 codebase. Additional methodological details are provided in Appendix A.
5. Results
For SAC, data augmentation with RAD consistently im- proves efficiency, especially as visual diversity increases. In contrast, auxiliary methods like CURL, DBC, VAE, SPR, and SB require more samples than standard SAC, partic- ularly with larger pools. The underperformance of these methods appears to stem from different factors. CURL may struggle because enforcing similarity between augmented observations could be ineffective when images from the pool lack shared structure. DBC might perform poorly when observations with similar visual features correspond to dif- ferent dynamics and rewards. SPR, SB, and VAE may also be less effective because enforcing latent space smoothness could be unsuited to SPGym's 's highly discontinuous obser- vations, as suggested by the relatively better performance of the unconstrained AE variant.
Our analysis reveals three fundamental tensions in visual RL: between method assumptions and environment struc- ture, sample efficiency and solution optimality, and training diversity versus generalization capability. We organize find- ings through seven research questions.
5.1. Can SPGym distinguish agents on their approaches to representation learning?
Table I demonstrates SPGym's 's ability to differentiate agents based on their representation learning approaches across varying levels of visual diversity. Our primary goal in this analysis is to compare the sample efficiency with which different agents learn useful visual representations.
To explore the potential benefits of leveraging pretrained
Finally, DreamerV3 demonstrates particularly strong per- formance, consistently outperforming both PPO and SAC variants across all pool sizes. The full Dreamer model
AAccessible at https:/ /ttpt/ggtttttt/////// com/ styamooiveiie//sgnmmeeeeeee
6

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
PPO
SAC
DreamerV3
Success Rate (%)
100
5
5
20
5
10
20
50
80
10
60
40
100
20
20
50
0
2M
4M
8M
10M
2M
6M
8M
10M
4M
6M
8M
10M
Steps
Steps
Steps
Pool Size
10
20
30
50
100
Figure Success rate as a function of environment steps. The gradual increase visual diversity affects the sample efficiency of standard PPO, SAC, and DreamerV3 agents at different rates, Each line represents different pool size, from 1 to 100 images.
shows remarkably stable performance across different pool sizes, significantly better than most other methods. The vari- ant without decoder gradients shows reduced performance across all configurations, highlighting the importance of the world model's discrete reconstruction objective. This sug- gests that Preameer33ss approach of learning predictive model of the environment provides a particularly effective foundation for handling visual diversity in SPGym.
ever, it also faces fundamental limitations, potentially as memorization becomes unsustainable. DreamerV3's world model architecture appears to foster more compressed repre- sentations, leading to more graceful degradation, but it too struggles with very large pools.
Preliminary DreamerV3 experiments with extremely large image pools (10,000-20,000 images) show that agents fail to learn useful policies, each observation becomes vir- tually unique. This suggests that simply increasing dataset scale is not enough if the RL signal becomes too sparse to effectively guide encoder training in the presence of high di- versity. These methods appear to memorize rather than gen- eralize, struggling when network capacity is overwhelmed by constantly novel inputs. This highlights fundamental limitations of current approaches when faced with diverse visuals where assumptions break and the learning signal is too weak to support meaningful learning.
These results highlight SPGym's value diagnostic tool, effectively distinguishing which approaches can handle its distinct combination of visual diversity and structured dy- namics. The benchmark exposes important limitations in methods that have proven successful other visual RL do- mains, suggesting that their underlying assumptions may not align with SPGym's 's unique characteristics. For detailed performance analysis and proposed explanations for agent behavior, see Appendix C.2.1.
5.2. How does visual diversity affect agents?
5.3. Can agents generalize to unseen visual inputs?
Figure 4 exposes critical limitations in current methods. While all agents eventually degrade with larger image pools, their failure modes differ. PPO degrades significantly at pool size 10 and fails at 20. SAC performs well up to pool size 20, degrades at 30, and fails at 50. DreamerV3 shows the most robust scaling, learning effectively at pool size 50 and exhibiting learning even at pool size 100.
Our analysis of generalization capabilities on PPO and SAC agents reveals a stark contrast between in-distribution and out-of- distributiion test performance. We evaluate two levels of generalization difficulty: 'Easy' OOD, using augmented versions of training images, and 'Hard' OOD, using com- pletely unseen pictures. Despite most agents achieving high success rates on their training distribution, they exhibit vary- ing degrees of failure on both generalization challenges. Since meaningful generalization evaluation requires agents that first achieve competent in-distribution performance, we primarily focus our OOD analysis on smaller pool sizes (i.c., 1, 5, and 10) where most methods successfully learn the task. Full OOD performance data for both Easy and Hard settings across all configurations can be found in Appendix C.2.
These distinct degradation patterns suggest differences in handling increasing visual diversity, potentially due to agents memorizing features instead of learning general- izable representations, thus exhausting network capacity. PPO's s early failure indicates difficulty forming robust rep- resentations and rapid capacity exhaustion. SAC's better performance on medium pools, possibly aided by its replay buffer and encoder training independent of actor gradients, suggests stronger representation learning initially. How-
For Easy OOD evaluation, we measure the success rate of PPO and SAC agents after applying each augmenta-
7

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
Table 2. Success rate of PPO and SAC agents on Easy OOD
The rare non-zero success rates in Hard OOD likely stem from chance encounters with nearly-solved initial states rather than genuine generalization.
across different training image pool sizes. Higher is better.
Algorithm
Pool 1
Pool 5
Pool 10
PPO
0.49
0.53+0.14
0.3420.08
These findings reveal fundamental limitations in current visual RI. methods and point toward specific research di- rections. The universal failure on Hard OOD, combined with degradation on Easy OOD as pool diversity increases, indicates that memorization-base learning is insufficient for visual generalization. Future research should focus on developing techniques that not only achieve good sample efficiency but also explicitly promote generalization to truly novel visual contexts. This might involve architectures that better separate visual representation learning from policy learning, incorporate stronger inductive biases for visual reasoning, or leverage self-supervised objectives that en- courage learning of more fundamental visual features, along with regularization methods that explicilly discourage mem- orization. Simply increasing the diversity of training images, as SPGym allows, is insufficient with current algorithms, highlighting the need for new approaches.
PPO + PT (ID)
0.33
0.53
0.27a0.07
PPO + PT (OOD)
0.49+0.12 12
0.52+0.14
0.34+0.08
SAC
0.45+0.12
0.58
0.46
SAC + RAD
0.62a0.15
0.42a0.13
0.30a0.11
SAC + CURL
0.76+0.09
0.44++.10 10
0.37+0.11
SAC + SPR
0.65+0.13
0.2 20.09
0,07=0.04
SAC + DBC
0.44
0.34
0. 0.1340.04
SAC + AE
0.78+0.11
0.64+0.16 16
0.55+0. 12
SAC+ + VAE
0.64+0.15
0.30s0.08
0. 12
SAC+ + SB
0.89+0.08
0.65+0.12
0.06+0.02
tion described in Appendix A.3, averaging results over 100 episodes per augmentation. Table 2 reports these results for training pool sizes 1, 5, and 10. As the diversity of the train- ing pool increases, we observe general decline in OOD performance. Some SAC variants show greater robustness to augmentations than standard SAC, especially when trained with a pool size of 1, but this advantage diminishes as the pool size grows. Overall, Easy OOD performance tends to decrease for all methods as training pool size increases. Notably, there is a strong correlation (Pearson -0.81, p 2.3 10-13) 13) between Easy OOD success and sample efficiency across methods and pool sizes, suggesting that agents which scale well with increasing visual diversity also tend to generalize better to augmented inputs.
5.4. Is representation quality linked with performance?
As a proxy for the quality of learned representations, we performed linear probing on frozen encoders from trained PPO and SAC agents, using a single-layer MI.P to predict one-hot puzzle states. We find a statistically significant correlation between probe accuracy and sample efficiency (Pearson r -0.81, p = 1.1 x 10 13), indicating that encoders capturing more task-relevant spatial information are strongly linked with faster learning. As image pool size increases, both probe accuracy and task performance sys- tematically degrade, with standard SAC maintaining high accuracy (100% at pool size 1, 97.63% at pool size 5) mir- roring its strong efficiency, while less efficient methods like SAC+VAE (78.21% 1% at pool size 5) and SPR (dropping from 94.31% to 75.48% from pool size 5 to 10) show reduced probe performance. These consistent trends across algo- rithms demonstrate SPGym's ability to identify learning procedures that develop representations better aligned with the task's spatial reasoning needs. Detailed probing results are provided in Appendix 1.1.2.
For Hard OOD evaluation, we test all seeds of PPO, SAC, and Dreamer agents on 100 episodes using images inde- pendently sampled from the ImageNet- Ik validation split. Unsurprisingly, agents almost universally fail, achieving near 0% success rates across all methods and training con- figurations. This stark failure on completely unseen images is a crucial diagnostic finding, exposing fundamental limi- tations of current end-to-end RL methods when faced with visual inputs that differ substantially from their training distribution. This finding reinforces our interpretation that agents may be memorizing specific visual patterns instead of learning generalizable representations.
Intuitively, one might expect training on larger, more di- verse image pools to improve generalization. However, our results demonstrate that even agents trained on pools of up to 100 images completely fail to transfer skills to novel vi- suals in the Hard OOD setting. Furthermore, performance on augmented training images often decreases training pool size increases. This perhaps counter-intuitive result may suggest that, in SPGym, agents trained on smaller, less diverse pools, learn representations more attuned to the spe- cific structural invariances of the task, making them robust to simple perturbations but still not fundamentally general.
5.5. Does performance generalize across image sources?
To validate that our findings generalize beyond ImageNet, we evaluated agents on DiffusionDB (Wang et al., 2023), dataset of procedurally generated images. As shown in Figure 8 in Appendix A.5, performance scaling patterns on DiffusionDB closely mirror those observed on ImageNet across PPO, SAC, and DreamerV3. This consistency across fundamentally different image sources (real photographs versus synthetic generations) demonstrates that visual diver- sity rather than semantic content drives the representation
8

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
learning challenge in SPGym. The similarity in degradation patterns pool size increases indicates that our algorithmic insights reflect fundamental properties of the tested methods rather than dataset-specifte artifacts. Procedurally generated images also offer practical advantages for future research: eliminating storage requirements through on-demand gen- eration, enabling fine-grained control over visual similarity, and providing unlimited training diversity.
training approaches the theoretical 22-step average optimal solution (Reinefeld, 1993). This confirms that early termina- tion, while needed for our experimental objectives, prevents agents from discovering more optimal solutions through continued training.
6. Conclusion
We introduce the Sliding Puzzles Gym (SPGym), novel benchmark designed to systematically evaluate representa- tion learning in RI. algorithms by isolating visual complex- ity controls from the environment dynamics. Our analy- sis reveals fundamental tensions challenging current visual RL methods. We show that sophisticated representation learning techniques struggle with SPGym's 's combination of visual diversity and structured spatial reasoning require- ments, with auxiliary objectives often underperforming sim- pler approaches like data augmentation. We also uncover a complex relationship between sample efficiency and gener- alization: agents trained on smaller pools learn task-specific invariances that paradoxically make them more robust to simple perturbations, with larger pools often degrading gen- eralization performance. Most critically, we expose severe limitations in end-to-end RL methods through universal fail- ure novel visual contexts, achieving near-zero success despite strong in di-ssirribuuion performance revealing that learned representations rely on memorization rather than genuine visual understanding. These findings highlight the need for algorithmic advances that move beyond memoriza- tion toward true visual understanding.
5.6. How does puzzle size affect learning performance?
Beyond visual diversity, we also explore the impact of in- creasing puzzle size. As shown in Table 3, the complexity increase from 3 3 to 4 x 4 grids significantly impacts learning. On the simpler 3 x 3 puzzle, PPO solved the puzzle in 1.75M steps, while SAC and DreamerV3 were more efficient, requiring 0.33M and 0.42M steps, respec- tively. For the 4 x 4 puzzle, PPO's sample requirements surged to 24, 44.46M steps, far exceeding the 10M step training budget, while SAC and DreamerV3 still solved the puzzle within budget at 8.14M and 2.26M steps, respectively. This demonstrates that while larger state spaces pose major challenge by requiring more exploration and more complex visual representations, more sample-efficient algorithms can still scale to harder tasks.
Table 3. Million steps to reach 80% success rate across grid sizes, with pool size 1. Lower is better.
Grid Size
PPO
SAC
DreamerV3
3x3
1.75+0.44
0.33+0.77
0.42+0.06
4x4
24.46+7.58
1443.66
2.26+0.29
Limitations. We identify two key limitations in this work. First, our objective was to evaluate out-of-the-box perfor- mance of all methods with minimal tuning, which means we may not have seen the true peak capabilities of each approach. Second, computational constraints limited to 5 independent runs per configuration. Given the high stochas- ticity in sampled image pools, more seeds would provide better statistical robustness for comparing methods.
5.7. How optimal are the learned solutions?
While our primary focus is on sample efficiency for task completion, we also analyze solution quality by examining the average number of steps agents take to solve puzzles. Our experimental design uses early termination when agents achieve 100% success rate to enable outttf-distributtion eval- uation before extreme encoder overfitting and save com- putational resources. However, this approach may prevent agents from discovering more optimal solutions through continued training. To investigate this trade-off, we trained PPO, SAC, and DreamerV3 on pool size I for the full 10M steps without early termination across 5 seeds. Comparing episode lengths between when these agents first achieve 100% success rate (where early termination would occur) and after completing the full training reveals substantial improvements in solution efficiency. For PPO, the first 100 successful episodes average 214.30 + 16.52 steps, while the last 100 episodes average 31.35 + 6.59 steps. SAC shows improvement from 64.16 + 9.81 to 57.27 + 12.29 steps, and DreamerV3 improves from 126.02 + 17.25 to 23.48 + 0.71 steps. Notably, D3eamerV3 with continued
Future work. SPGym opens several promising research directions for developing representation learning methods that can bridge the generalization gap while maintaining sample efficiency. The benchmark's 's controlled nature en- ables systematic investigation of the trade-offs between training diversity and generalization capability, while its easy integration with external image generation systems al- lows controlled experiments with varying degrees of visual similarity. Beyond images, SPGym supports incorporating other modalities as puzzle observations, enabling research on leveraging powerful pretrained models as representation encoders. We believe these directions would help develop more robust learning approaches that transfer effectively across different domains and modalities.
9

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
Acknowledgements
References
The authors gratefully acknowledge the valuable insights and constructive discussions provided by Professors Marcos R. O. A. Maximo and Flávio H. T. Vieira. We also thank the reviewers for their thoughtful feedback and constructive suggestions that helped improve this paper.
Agostinelli, F., McAleer, S., Shmakov, A., and Baldi, P. Solving the rubik's 's cube with deep reinforcement learning and search. Nature Machine Intelligence, 1(8):356-663 2019. doi: 10. 10000///42256-019-0007-----
Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wain- wright, M., Küttler, H., Lefrancq. A., Green, S., Valdés, V., Sadik, A., et al. Deepmind lab. arXiv preprint arXiv: (rXiv:1612.03801, 2016.
This work has been partially funded by the project Research and Development of Digital Agents Capable of Planning, Acting, Cooperating and Learning supported by Advanced Knowledge Center in Immersive Technologies (AKCIT), with financial resources from the PPI IoT/Manufatura 4.0 / PPI HardwareBR of the MCTI grant number 057/2023, signed with EMBRAPII
Bellemare, M. G., Naddaf. Y., Veness, J.. and Bowling, M.
The arcade learning environment: An evaluation plat- form for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Luckeciano C. Melo acknowledges funding from the Air Force Office of Scientific Research (AFOSR) European Office of Aerospace Research & Development (EOARD) under grant number FA8655-21-1-7017.
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE transac- tions on pattern analysis and machine intelligence, 35(8): 1798-1828, 2013.
Impact Statement
Brockman, G. Openai gym. arXiv: 1oeo.10060..444 2016.
arXiv preprint
This work advances our understanding of how visual rep- resentation learning affects reinforcement learning perfor- mance and generalization. The insights gained could in- fluence the development of more robust and efficient RL systems for real-world applications. However, several po- tential impacts warrant consideration:
Burns, E., Hatem, M., Leighton, M., and Ruml, W. Imple- menting fast heuristic search code. Proceedings of the International Symposium on Combinatorial Search, 3(1): 25-32, 2012.
Christodoulou, P. Soft actor-critic for discrete action settings. arXiv preprint (xXiv:/910.07200 072007 2019.
Positive impacts: By exposing limitations in current meth- ods, our benchmark may help direct research toward more reliable and generalizable RL systems. This could acceler- ate progress in areas like robotics and autonomous systems where visual understanding is crucial. The benchmark's s controlled nature also promotes more rigorous evaluation practices in RL research.
Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Lever- aging procedural generation to benchmark reinforcement learning. In III, D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 2048-2056. PMLR, 13-18 Jul 2020.
Negative impacts: As with any machine learning bench- mark, there is risk of overfitting to our specific evaluation criteria rather than addressing fundamental challenges. Ad- ditionally, improved visual RL capabilities could enable automation that displaces human workers enable surveil- lance applications if misused.
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. CARLA: An open urban driving simulator. In Levine, S., Vanhoucke, V., and Goldberg, K. (eds.), Pro- ceedings of the Ist Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pp. 1-16. PMLR, 13-15 Nov 2017.
Estermann, B., Lanzendörfer, L. A., Niedermayr, Y., and Wattenhofer, R. Puzzles: A benchmark for neural algo- rithmic reasoning. In Advances Neural Information Processing Systems (NeurIPS), 2024.
Mitigations: We have open-sourced our benchmark and evaluation protocols to promote transparency and repro- ducibility. We encourage future work to consider both per- formance metrics and broader societal implications when building on our findings. Researchers should carefully con- sider potential dual-use applications and implement appro- priate safeguards when deploying systems based on these techniques.
Ghosh, P., Sajjadi, M. S. M., Vergari, A., Black, M. J., and Schölkopf, B. From variational to deterministic autoen- coders. In 8th International Conference on Learning Representations (ICLR), April 2020.
10

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with stochastic actor. In 35th Inter- national Conference on Machine Learning (ICML), pp. 1861-1870. PMLR, 2a.
Li, L.,, Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(185)11-52, 2018.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Ve- ness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nature, 518 (7540):529-53) February 2015. ISSN 1476-4687 doi: 10. 1..1038/natrre14253
Haarnoja. T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. Soft actor-critic algorithms and applications. ArXiv, abs//abssee/e/ 182.05905, 2018b.
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap. T. Mastering diverse control tasks through world models. Nature, 640 (8059))647-653, April 2025. ISSN 1476-4687. doi: 10.1038/541586-025-008444
Moon, S. U. and Cho, Y. Improving the training perfor- mance of dqn model on 8-puzzle environment through pre-training. International Journal of Applied Engineer- ing Research, 6(1):63-67, 2024.
Higgins, I., Pal, A., Rusu, A., Matthey, L., Burgess, C.,, Pritzel, A., Botvinick, M., Blundell, C., and Lerchner, A. Darla: Improving zero-shot transfer in reinforcement learning. In 34th International Conference on Machine Learning (ICML), pp. 1480-1490. PMLR, 2017.
Reinefeld, A. Complete solution of the eight-puzzle and the bene t of node ordering in ida*. In International Joint Conference on Artificial Intelligence, pp. 248-253. Citeseer, 1993.
Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and Araújo, J. G. Cleanrl: High-quality single-file file implementations of deep reinforcement learn- ing algorithms. Journal of Machine Learning Research, 23(274): 2022.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV,, 115(3):21-252,, 2015. doi: 10.1007/s1123-015-000
Johnson, W. W. and Story, E. Notes on the '15' puzzle. American Journal of Mathematics, 2(4):397-404, 1879. doi: 10.2307/2369492.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. 2017.
Korf, R. E. Depth-first iterative-deepening: An optimal admissible tree search. Artificial intelligence, 27(1):97- 109, 1985.
arXiv preprint
Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D.. Courville, A. C., and Bachman, P. Data-efficient reinforcement learning with self- predicteee representations. In 8th Inter- national Conference on Learning Representations (ICLR), 2020.
Laskin, M., Lec, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems, 33: 19884-19895, 2020a.
I Laskin, M., Srinivas, A., and Abbeel, P. Curl: Contrastive unsupervised representations for reinforcement learning. In 37th International Conference Machine Learning (ICML), pp. 5639-5650. PMLR, 2020b.
Schwarzer, M., Rajkumar, N., Noukhovitch, M., Anand, A., Charlin, L., Hjelm, D.,, Bachman, P., and Courville, A. C. Pretraining representations for data-efficient rein- forcement learning. In Advances in Neural Information Processing Systems, 2021.
Lee, S. C. and See, T. H. Comparing the hamming and manhattan heuristics in solving the 8-puzzle by a* al- gorithm. In Mustapha, A. B., Shamsuddin, S., Zuhaib Haider Rizvi, S., Asman, S. B., and Jamaian, S. S. (eds.), Proceedings of the 7th International Conference on the Applications of Science and Mathematics 2021, pp. 189- 195, Singapore. 2022. Springer Nature Singapore. ISBN 978- 16-8-1699-31..
Stone, A., Ramirez, O., Konolige, K., and Jonschkowski, R. The distracting control suite-a challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.027223 2021.
Stooke, A., Lee, K., Abbeel, P., and Laskin, M. Decoupling representation learning from reinforcement learning. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Re- search, pp. 9870-9879, PMLR, 18-24 Jul 2021.
Lesort, T., Díaz Rodrígue,, N., Goudou, J.-F,, and Filliat, D. State representation learning for control: An overview. Neural Networks, 108:379-399 2018

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. I.., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq. A., al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.
Tomar, M., Mishra, U. A., Zhang, A., and Taylor, M. E. Learning representations for pixel-based control: What matters and why? Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Tomilin, T., Fang, M., Zhang, Y., and Pechenizkiy, M. Coom: A game benchmark for continual reinforcement learning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 67794- 67832. Curran Associates, Inc., 2023.
Wang, Z. J., Montoya, E., Munechika, D., Yang, H., Hoover, B., and Chau, D. H. DiffusionDB: A large- scale prompt gallery dataset for text-to-image generative models. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.). Proceedings of the 6lst Anmual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 893-911, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10665//v1/222.acllooorg
Yarats, D., Kostrikov, I., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In 9th International Conference on Learning Representations (ICLR), 2021a.
Yarats, D., Zhang, A., Kostrikov, I., Amos, B., Pineau, J., and Fergus, Improving sample efficiency in model- free reinforcement learning from images. Proceedings of the AAAI Conference Artificial Intelligence, 35(12): 10681, 2021b.
Zhang, A., McAllister, T., Calandra, R., Gal, Y., and Levine, S. Learning invariant representations for rein- forcement learning without reconstruction. In 9th Interna- tional Conference Learning Representations (ICLR), 2021.
12

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
A. Experimental Setup and Configuration
A.I. Model Architectures
We base our implementations on CleanRL's Atari agents for both PPO and SAC, with minor architectural modifications including additional normalization layers and increased network depth to approximate our SAC implementation to the one used by Tomar al. (2023). The architectures are detailed below.
For PPO agents:
SharedEncoder
(encoder) : Sequential
(0) (3,v2d 32, kernel_size- (8, 8), stride- (4, 4))
(1) ReLU()
(2) (3tchNorm2d eps-le-05, momentum=0.1 affine-True, stats=True)
(3) : (32, 64, kernel_size= (4, 4), stride= (2, 2))
(4) ReLU()
:5) PatchNorm2d aps=le-05, momentum=0.1, affine=True, stats=True)
(6): C64v2d(64, 64, kernel_size= 3), stride= (1, 1))
(7): BatchNorm2d eps=1c-05, um=0., affine=Irue, state=True)
:8): (staten(start_ddmm:) end dim=-1)
(projectionl Sequential
(0): (ineer
qut_features-512, bias-True) (1) LayerNorm (5112)) (512,), eps=le-05, ere
(2) Tanh()
Actor
(encoder): SharedEncoder
(mlp): Linear (in featuress,,,5,, cut_features-4, bias-True)
Critic(
(encoder): : SharedEncoder
(mlp): : Linear (in features=ss, cut_features=1, bias=True)
For SAC agents:
Sharedkncoder
(encoder) : Sequential
(0): (3, 32, kernel size= 8), stride= (4, 4))
(1): : ReLU()
(2) P3tchNorm2dd32, eps=le-05, momentum=0.1, affine=True, stats=True)
(3) C32v2d 64, kernel_size-(4, 4), stride= (2, 2))
(4): ReLU()
(5): BatchNorm22 (64, ops-1c-05, momentum=0.11 affine-True, stats=Tzue)
(6): (64, 64, kernel_size=(1 3), stride= (1, 1))
(7): (atchNorm22d64, eps-le-05, momentum=0.1, affine=True, stats=True)
(8): Flatten Flatten(start_dimn)) and_dim=-1)
Actor(
(encoder): Encoder
(shared_encoder): Shared#nccder
(projection): Sequential
(0): Linear(in faa___ffatrr=33666 out_ fet_tutrrre::,,, bias=True)
(1) LayerNorm (512,), eps=le-05, elementwise el affine=True)
(2): (anh
(mlp): Scquential(
(0) (in_faatin_ out_features-$12, bias=True)
(1): LayerNorm (512,), eps-le- 05, semeentwis__fffineeee
(2) ()
13

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
: Linear (i__feaurres- cut_features=4, bias-True)
Critic(
(encoder) Encoder
(shared (shared_encoder): SharedEncoder
encoder)
(projcction): Sequential
(0): Linear (in_features=3136, out features=s5 bias=Truc)
(1) LayerNorm (5112) eps=le-05, elementwiee
(2) (anh
pm Sequential
(0) Linear feature==511, features=512, bias=True)
(1) : (ppoNNorm(5512))) eps=1e-05, swwmenttise_affieeeeie
(2) ReLU()
(3): Linear (in_features=512, out_features=4, bias-True)
(mlp): : Sequential
( Linear (infeerurree---11, out fut_feetuure-512, bias-True)
(1): (0yerNorm((512))) eps-le- e deementrise_aafilee-nee
(2): ()LU()
(3): Linear (in features=512, out features4,, bias=True)
For the representation learning methods, we maintain the same base architecture and use a consistent MLP structure (2 layers with ReLU activation) for the projector, predictor, transition and reward models. The decoder architecture as follows:
ImageDecoder
(decoder): Sequential
(0): Linear (i__featuure- fut_featur,---12, blas-True)
(1) : MayerNorm((51)) (512,) eps=1c-05, elemettwise_affine--ie
(2) : ()LU()
(3) : Linear (in_eearrr11 out pot_features=3136, bias-True)
(4) Unflatten (dim=1, unflattened_size=113 (3, 7, 7))
(5): ConvTranspose2d 64, sernel_size= (3, 3), stride=(1, 1)
(6): ReLU
(7): (64 eps=le-05, momentum=U.1. aftine=True, stats=True)
(8): ConvTranspose2d (64, 32, kernel_size- (4, 4), stride- 2))
(9): ReLU()
(10) BatchNorm22 eps-1e-05, momentum=0.1, affine-True, stats-True)
(11) ConvTranspose2d (32, 3, kernel size= (8, 8), stride= 4))
(12) Sigmoid
For DreamerV3 agents, the same base 12M architecture the used by Hafner et al. (2025). Appendix A.2 contains the specific hyperparameters used in our experiments.
A.2. Hyperparameters
Table 5, Table 6, and Table list hyperparameters used across all experiments, unless noted otherwise. For DreamerV3, adopted hyperparameters from (Hafner et al., 2025), modifying only the decoder loss scale (set to 0) for the version without decoder. Table lists hyperparameters for representation learning methods and components. We use separate optimizer for the representation learning gradient flow, and we adopt a higher learning rate. When using representation learning methods, we update the target network's 's encoder faster, with an EMA 0.025. When using a crop augmentation, we set the image size 100 and crop it back to 84.
14

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
Table Benchmark settings
Parameter
Value
Max steps
10M
Puzzle size
3x3
Action space
discrete
Variation
image
Render size
100x100 (for crop augmentation)
84x84 (otherwise)
Dataset
ImageNet-1k validation split
Table 5. Hyperparameters for PPO
PPO Parameter
Value
Input image size
84x84
Env instances
64
Optimizer
Adam
Learning Rate (LR)
2.5e-4
LR annealing
yes
Adam e
le-5
Num. steps
16
Num. epochs
4
Batch size
64
Num. minibatches
4
0.99
GAE À
0.95
Advantage normalization
yes
Clip coef.
0.1
Clip value loss
yes
Value function coef.
0.5
Table 6. Hyperparameters for SAC
SAC Parameter
Value
Input image size
84x84
Env instances
64
Optimizer
Adam
Learning rate
3e-4
Replay buffer capacity
3e5
Batch size
4096
Warmup steps
2e4
0.99
Policy update frequency
Fixed temperature
0.05
Target network update frequency
Target Q functions EMA 7
0.005
Target encoder EMA 7
0.005 (standard)
0.025 (otherwise)
15

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
Table Hyperparameters for DreamerV3
DreamerV3 Parameter
Value
Input image size
80x80
Env instances
16
Model size
12M
RSSM deterministic size
2048
RSSM hidden size
256
RSSM classes
16
Network depth
16
Network units
256
Replay buffer capacity
5e5
Replay ratio
32
Action repeat
Learning rate
4e-5
Batch size
16
Batch length
64
Imagination horizon
15
Discount horizon
333
Decoder loss scale
(standard)
(no decoder)
Table Hyperparameters for Representation Learning Methods and Components
Method
Hyperparameter
Value
Learning rate
le-3
General
Loss coefficient
1.0
Min sigma
le-4
Transition and reward models
Max sigma
Probabilistic
no
Latent space decay weight
lc-6
SAC-AE
Decoder decay weight
le-7
SAC-VAE
Variational KL weight
le-7
Temperature
0.1
CURL
Positive samples
temporal/augmentted
Augmentations
crop
Augmentations
crop
RAD
channel
color jitter
Horizon H
SPR
Augmentations
crop
A.3. Augmentation Strategies
We evaluated several image augmentations in our preliminary experiments, as described in Appendix A.4.2. These augmentations are illustrated in Figure 5, and are as follows:
No augmentation: The image is fed as is to the agent.
Crop: Randomly crops a portion the image and resizes it back to the original dimensions. This helps learn translation invariance by forcing the agent to recognize patterns regardless of their position the frame.
16

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
No augmentation
Crop
Grayscale
Channel Shuffle
Shift
Inversion
Color Jitter
Figure Example image augmentations. The top left image shows the original observation, and the subsequent images show the observation after each augmentation procedure independently applied.
Grayscale: Converts the RGB image to grayscale by averaging color channels. This reduces the visual complexity and helps the agent focus on structural features rather than color information.
Channel Shuffle: Randomly permutes the RGB color channels. This encourages the agent to invariant to color transformations while preserving the image structure.
Shift: Translates the image by small random amount in both horizontal and vertical directions. Similar to crop, this promotes translation invariance in the learned representations.
Inversion: Inverts the pixel values by subtracting them from the maximum possible value (255 for 8-bit images). This teaches the agent to recognize patterns independent of absolute intensity values.
Color Jitter: Applies random color variations to the image, including brightness, contrast, saturation, and hue. This helps the agent to be invariant to color transformations while preserving the image structure.
After extensive experimentation, we found that the combination of grayscale and channel shuffle consistently produced the best results. This combination effectively reduces visual complexity while maintaining important structural information. adopted this augmentation pair as the standard for all our agents that use augmentation techniques.
A.4. Preliminary Experiments and Design Rationale
100
1. HYPERPARAMETER SELECTION PROCESS
Success Rate (96)
80
For SAC agents, the entropy coefficient a significantly impacts performance (Haarnoja al., 2018a). While Haarnoja al. (2018b) proposed automatic tuning (au- totune) based on policy entropy, found this approach ineffective for SPGym, even with various learning rates (LRs). Through systematic Hyperband (Li et al.. 2018) sweeps over pools of size 1, we identified a = 0.05 as optimal (Figure 6). This value provides a good balance be- tween exploration and exploitation, allowing the agent to efficiently learn the puzzle mechanics while maintaining enough randomness discover solutions.
60
40
20
0
100k
200k
300k
400k
500k
600k
700k
800k
Steps
Autotune LR 1.29e-4 ... Autotune LR 4,40e-4
Autotune LR 1.71e-3 -.. Autotune LR 2.20e-3
Fixed q=0.05
Fixed q=0.10
Fixed a=0.14
Fixed a=0.28
PPO and DreamerV3 agents proved more robust to hy- perparameter choices, performing well with their default configurations. This robustness is particularly valuable in our benchmark setting, as suggests these algorithms can adapt to new tasks without extensive tuning. Complete hyperparameter settings are provided in Appendix A.2.
Figure 6. Performance comparison of different a values for SAC agents pool size 1. Fixed 0.05 outperforms automatic tuning approaches across different learning rates.
17

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
A.4.2. DATA ACGMENTATION ANALYSIS AND CHOICE
Building on insights from Laskin et al. (2020a) that environment--pecific invariances influence optimal augmentation strategies, we systematically evaluated augmentation pipelines for RAD, CURL, and SPR in SPGym. Our analysis focused on sample efficiency to maintain consistency with core experimental objectives. We tested the five augmentation techniques detailed on Appendix A.3 on pools of 5 images, which offered good balance between task complexity and convergence speed. For SPR, specifically, we also experimented with shift + color jitter, as suggested by Schwarzer et al. (2020). Across all algorithmss, our experiments consistently converged to a simple two-stage augmentation process: probabilistic grayscale conversion (with 20% chance) followed by channel shuffling (Figure 7). This pipeline's effectiveness likely stems from its ability simultaneously reduce visual complexity through grayscale conversion while introducing beneficial stochasticity via channel shuffling, with both transformations preserving critical structural information while preventing overfitting to specific color patterns. We adopted this augmentation combination for all subsequent experiments. While our current evaluation focused on sample efficiency, investigating how different augmentation strategies affect generalization remains an important direction for future work.
A.5. Dataset Analysis and Choice
Our choice of ImageNet-1k Ik as the primary dataset for SPGym was motivated by several key considerations. First, ImageNet provides a diverse set of real-world images that challenge agents to learn generalizable visual representations. We hypothesized that real-world images would provide unique insights related to representation learning that should be applicable to other domains beyond the puzzle proposed in SPGym, as they contain the complex visual patterns and structures that agents encounter in practical applications. However, as shown in Figure 8 and in comparison to Figure the performance scaling patterns we observe ImageNet closely mirror those DiffusionDB, suggesting that our findings not specific to particular dataset but rather reflect fundamental properties of the algorithms being tested.
The similarity in scaling behavior between ImageNet and DiffusionDB is particularly noteworthy because these datasets differ substantially in their composition and generation process. While ImageNet consists of real photographs, DiffusionDB contains synthetic images generated by text-to-image models. The consistent performance patterns across these datasets suggest that our results capture fundamental algorithmic behaviors rather than dataset-specific artifacts.
Our demonstration with DiffusionDB reveals promising directions for future work with procedurally generated datasets. As shown in analysis, agents perform similarly on ImageNet and DiffusionDBB, suggesting that visual diversity rather than semantic content drives difficulty. Procedurally generated images offer several compelling advantages worth further investigation: they eliminate the need for large image storage by generating unique images on demand for each episode, enable fine-grained control over the generalization challenge by gradually increasing visual differences between generated images, and provide virtually unlimited training data diversity. These capabilities could enable more systematic studies of
RAD
CURL
SPR
100
Success Rate (%))
80
60
40
20
0
0.2M 0.4M 0.6M 0.8M 1M 1.2M 0.2M 0.4M 0.6M 0.8M
1.2M 1.4M
0.2M 0.6M 0.8M 1M 1.2M 1.4M
Steps
Steps
Steps
crop ... shift grayscale - inversion -- channel shuffle
color jitter grayscale,channel shuffle - shift,color_jittee
Figure 7. Comparative analysis of data augmentation strategies. Results show performance of SAC with RAD, CURL and SPR on 5-image pools. Grayscale conversion and channel shuffling emerge as the most effective combination, significantly outperforming other augmentation strategics.
18

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
PPO
SAC
DreamerV3
100
10
10
Success Rate (%6
80
Marden 10
60
40
20
2M
8M
10M
0.5M
1.5M
1M
4M
Steps
Steps
Steps
Pool Size
10
Figure Performance scaling with DiffusionDB images. Success rates for PPO, SAC, and DreamerV3 agents across different pool sizes 5, and 10) using DiffusionDB images. The performance patterns closely mirror those seen with ImageNet (Figure 4), Shaded regions represent 95% % confidence intervals across independent seeds.
visual generalization in reinforcement learning.
This cross-dataset consistency strengthens our confidence in the generalizability of our findings. It indicates that the relative performance of different representation learning methods is driven more by their core assumptions and architectural choices than by the specific characteristics of the training data. This is particularly important for our goal of understanding how different approaches handle increasing visual diversity, as it suggests our conclusions may extend to other domains beyond the specific datasets used in our experiments.
A.6. Hardware Setup and Runtime
Our hardware setup consists of an AMD Ryzen 7 3700X CPU, an NVIDIA RTX 3090 GPU, 64GB of RAM, and 128GB of swap space. Using this configuration, DreamerV3 experiments take approximately 20 hours per run, primarily because of the heavy use of swap space for replay buffers, which must store hundreds of thousands of images memory. SAC takes between 2 (e.g. standard, RAD) to 11 hours (e.g. SPR, SB) depending on the representation learning components used, For SAC, the longer runtimes are due to the number of sequential inference steps required to train the agent and auxiliary networks. PPO experiments are significantly faster, with the longest runs completing in about hour and 30 minutes.
B. Algorithms and Variations
We describe the algorithms and their variations used in this work in detail. Our experiments employ three main algorithms PPO, SAC, and DreamerV3 each with different representation learning approaches. We focus particularly on how these methods process and learn from visual observations, as this crucial for performance in our benchmark.
B.1. Pretraining and PPO
Drawing inspiration from previous work on pretraining methods in RL (Higgins et al., 2017: Stooke al., 2021; Schwarzer al., 2021). we implement pretraining approach for PPO that focuses on learning resk-relevant visual representations. The pretraining process involves training PPO agent to completion on a single environment instance, then extracting its CNN weights. These pretrained weights then used initialize new PPO agents, while all other network components (policy and value networks) start from random initialization. We evaluate two scenarios: in-distribution (ID), where new agents are trained on the same pool of images used during pretraining, and out-of-distribution (OOD), where a different image pool is sampled. The ID setting represents an upper bound on what pretraining can achieve with perfect visual alignment, while the OOD setting reflects the more realistic scenario of deploying pretrained encoders on novel visual inputs, similar to how general-purpose pretrained models would be used practice.
19

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
B.2. Data Augmentation and RAD
Reinforcement Learning from Augmented Data (RAD) (Laskin et al., 2020a) represents a simple yet effective approach to visual representation learning in RL. The key insight is that applying data augmentation to observations improve sample efficiency by exposing the agent transformed versions of experienced states. This creates an implicit regularization effect that helps learn more robust representations.
In our implementation, we combine RAD with SAC and apply data augmentation consistently during both policy updates and value function learning. These augmentations are applied to observations sampled from the replay buffer before being processed by the encoder network. The augmentation pipeline consists of two key transformations identified through our preliminary experiments (Appendix A.4.2): grayscale conversion and channel shuffling. Following the SPR approach (Schwarzer et al., 2020), we apply augmentations independently to each transition after sampling batches from the replay buffer, meaning that samples from the same episode may be augmented differently. We detail how each augmentation procedure is implemented in Appendix A.3.
B.3. Contrastive Learning and CURL
Contrastive learning methods learn representations by maximizing similarity between different views of the same observation while minimizing similarity to other observations. In the context of RL, CURL (Laskin et al., 2020b) applies this principle by using data augmentation to create positive pairs, enabling agents to learn invariant representations. The method achieves this by applying random crops to observations and treating differently augmented views of the same observation as positive pairs.
The contrastive loss for a positive pair of observations (I,x+t) and set of negative examples {t7] is formulated as:
RCURL. logg exp(fo(x)" +
where or is temperature parameter and ʃo is the encoder function.
In our implementation, we combine CURL with SAC and use the same augmentation strategy identified in Appendix A.3 (grayscale conversion and channel shuffling) rather than the random crops from the original CURL paper. The encoder is trained jointly with the RL objective, allowing the representations to adapt to both the contrastive learning task and the control problem. Negative examples are drawn from other observations within the same batch, providing computationally efficient way to obtain contrastive pairs without requiring additional memory storage.
B.4. State Metries and DBC
Deep Bisimulation for Control (DBC) (Zhang et al., 2021) takes a different approach to representation learning by focusing on behavioral similarity between states rather than visual similarity. The key idea is to learn an encoder that maps states to a representation space where distances reflect how similarly states behave in terms of rewards and transitions, rather than how visually similar they appear.
Given pairs of observations 2z), DBC trains an encoder fo to minimize:
J( (1 -1yl1 - |r; (P as), ((1)),
(4)
where & = fxx) represents the encoded state, E; = $g( (x:)) is the stop-gradient version the encoding, and P is a probabilistic transition model that predicts the next state distribution. The W2 term represents the 2-Wasserstein distance between predicted transition distributions, which for Gaussian distributions has a closed-form solution (Zhang et al., 2021).
In our implementation, we combine DBC with SAC, SAC, training the encoder with both the bisimulation objective and the RL objective. The transition model operates in latent space, predicting Gaussian distributions over next states. This approach helps the agent learn representations that capture behaviorally meaningful features while ignoring visual distractors that don't affect the game dynamics. Unlike methods that rely on data augmentation or reconstruction, DBC's focus on behavioral similarity makes it particularly suited for environments where visually different states might require similar actions.
20

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
B.5. reconstruction-Bassee Methods and SAC-AE/VAE
Reconstruction-based methods learn representations by training an encoder-decoder architecture compress and reconstruct observations. We evaluate two variants combined with SAC: SAC-AE using deterministic autoencoder and SAC-VAR using a variational autoencoder (Yarats et al., 2021b).
For SAC-AE, given an observation x from the replay buffer, we train an encoder fo and decoder to minimize:
CRAE = EgeD || g+(fo(z)) + (= A
(5)
where A and A are regularization coefficients that help prevent representation collapse and overfitting respectively (Ghosh et al., 2020).
For SAC-VAE, we replace deterministic encoder with a probabilistic encoder qo that outputs a distribution over latent states. The training objective becomes:
AV
1)),
(6)
where 3 controls the trade-off between reconstruction quality and latent space regularization.
In both variants, train the encoder jointly with the SAC objective, allowing the representations to adapt to both reconstruction and control tasks. The encoded states are used as inputs to the policy and value networks. Unlike methods that rely on data augmentation or behavioral similarity, these approaches learn representations by explicitly modeling the visual structure of observations through reconstruction.
B.6. World Models and D3eamerV3
World models learn to predict future states and outcomes by learning a compact latent representation of the environment. V3 (Hafner et al., 2025) represents the state-of-the-an in world model-based reinforcement learning, employing an online encoder fe that maps observations 2, into latent states &, recurrent dynamics model Aы operates in this latent space to predict future states conditioned on actions, while reward predictor estimates immediate rewards.
The model is trained using multiple objectives that create multi-task learning pressure. The encoder and corresponding decoder trained reconstruct observations, ensuring the latent space captures relevant visual features. The dynamies model is trained to predict future latent states that lead to accurate reconstructions of future observations. This temporal consistency objective forces the representations to be predictive of future states while supporting reconstruction and control.
DreamerV3 introduces several innovations for stable representation learning, including KL balancing to maintain informative latent states and symmetric cross-entropy loss for better gradients. Unlike methods focused solely on visual similarity, world models must learn representations that serve multiple purposes capturing visual features, encoding dynamics, and providing suitable space for policy learning. We refer readers to Hafner al. (2025) for implementation details.
B.7. Temporal Consistency Methods
Several methods leverage temporal consistency in the environment to learn better representations. These approaches based the principle that good representation should not only capture the current state but also be predictive of future states and outcomes.
B.7.1. SELF-PREDICTIVE REPRESENTATIONS (SPR)
SPR (Schwarzer et al., 2020) represents a non-contrastive approach that learns by predicting future latent states. Given a sequence states and actions (It:+K, (..+k) from the replay buffer, where K is the prediction horizon, SPR employs:
An online encoder fa that maps observations latent states: 2, fo(
A target encoder for providing stable training targets, updated via exponential moving average
An action-conditioned transition model h, that predicts future latent states: 4 hw a++k)
Projection networks Pç, Pe: and prediction head wy that transform representations for the prediction task
21

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
The model generates predictions UL+* (c((p(((+4)) and compares them target projections UI+ - -Ptr(E using cosine similarity loss:
K
180+Rl12 )
{
D++t [18e+All2 (11betbll))
T
CSPR Σ
B.7.2. SIMPLE BASELINE METHOD
Tomar et al. (2023) take minimalist approach to temporal consistency by combining two predictive objectives: reward prediction and transition prediction. We refer this approach as the Simple Baseline (SB) method. While originally intended baseline, it demonstrates the effectiveness of basic temporal prediction for representation learning.
The method augments standard RL algorithms with two predictive components in latent space:
A transition model h. that predicts the next encoded state
A reward predictor hreward that estimates immediate rewards
The transition loss Layn measures the mean squared error between predicted and actual next encoded states, while the reward loss Lreward measures the error in reward predictions. These predictive losses combined with the standard RL objective:
Listal
LRL. + freward + Ldyn
(8)
In our implementations, combine both SPR and SB with SAC. For SPR, apply the augmentation strategy identified in Appendix A.3 before encoding observations. Both methods operate entirely in latent space, avoiding the computational cost of pixel-space reconstruction while leveraging temporal structure to learn meaningful representations.
C. Supplementary Analyses and Results
C.1. Representation Learning Analysis
We further evaluate SPGym's representation learning assessment capabilities through two analyses: (1) comparing raw pixel ground-truth state learning, and (2) linear probes of learned representations.
C.1.1. STATE-BASER VS. IMAGE-BASE OBSERVATIONS
To establish baseline and understand the impact of the visual representation challenge, we trained PPO, SAC, and DreamerV3 agents using SPGym's one-hot encoding variation. These one-hot vectors represent the ground-truth puzzle state, identical to the targets used for our linear probes (see Appendix C.1.2). We compared their sample efficiency (steps to 80% success, averaged over 5 seeds) against their image-based counterparts. For PPO and SAC agents processing one-hot vectors, CNN encoders were replaced with 2-layer MLPs. DreamerV3 utilized its default non non-image encoder, a 3-layer MLP. Hyperparameters were kept consistent with image-based experiments, without specific tuning for the one-hot setting.
Table Steps to 80% success on one-hot vs. image-based observations. Lower better. indicates experiments were not run for that specific configuration due to computational constraints.
Algorithm
Grid Size
One-hot
Image (Pool 1)
Image (Pool 5)
PPO
3x3
661.69k481.44 444
1.75M
7.80M+100M
4x4
12.29M1467.Bo. 88kk
24.46M+7.58M
SAC
3x3
672.51k#63.100
334.26k+67.477
907.21k#116.208
4x4
5.09M:4463.144
8.14M+3.64M
DreamerV3
3x3
834.86k+61.10k
417.09k=55.03k
1.23M=199.499
4x4
3.68M+436.97k
2.26M+287.23k
5.81M 2.17M
The results Table offer several insights. For PPO (both grid sizes) and SAC (4x4 grid), learning directly from ground- truth one-hot states is more sample efficient than learning from images. This expected, the one-hot encoding removes
22

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
the burden of representation learning from pixels. The instances where image-based agents (SAC and DreamerV3 on 3x3 grid, pool 1) converged faster than their one-hot counterparts might be attributed to differences in network architectures (MLP vs. CNN-Transformer backbones) and the absence of specific tuning for the one-hot setting.
Crucially, the one-hot encoding setting presents a fixed, minimal representation learning challenge. In contrast, SPGym's 's image-based variations allow for a systematic sealing of the visual diversity challenge by increasing the image pool size (e.g.. Pool 1 vs. Pool 5 vs. Pool 10, etc., see Appendix C.2). Across all agents and grid sizes, increasing visual diversity from pool size 1 to pool size 5 (and beyond) consistently increases sample complexity. This demonstrates SPGym's 's ability to isolate and stress the visual representation learning component, as the underlying task dynamics remain constant. This controlled evaluation reveals limitations in how effectively different RL agents learn representations under scalable visual diversity, insights not apparent from the one-hot setting alone. While perfect disentanglement of representation learning from policy learning is challenging in end-to-end training, SPGym provides a valuable framework for structured, comparative evaluation of visual representation learning capabilities in RL.
C.1.2. LINEAR PROBE ANALYSIS
To directly assess the quality of learned visual representations, we conducted linear probing experiments. For each trained PPO and SAC agent, we froze its encoder and trained single-layer MIP classifier on top of the features extracted by it until convergence. The classifier's task was to predict the one-hot encoded ground-truth puzzle state corresponding to the input image. This setup allows to quantify how much task-relevant spatial information is captured by the agent's 's encoder. High probe accuracy indicates the encoder has learned features that are linearly separable with respect to the underlying game state.
Our analysis reveals several key insights. First, we find a strong, statistically significant correlation between linear probe accuracy and agent sample efficiency (Pearson 0=-0.81, 1e-11,),), with higher probe accuracy being highly predictive of fewer environment steps required to reach 80% success. This suggests that agents whose encoders capture more task-relevant spatial information tend to learn the task more efficiently.
Second, examining probe accuracies across pool sizes and algorithms further clarifies this relationship. As image pool size increases, both probe accuracy and task performance systematically degrade, isolating the effect of visual diversity on representation learning. For example, standard SAC maintains high probe accuracy (100% at pool size 1, 97.63% at pool size 5), mirroring its strong sample efficiency. In contrast, methods with lower sample efficiency show reduced probe performance: SAC+VAE achieves only 78.21% % probe accuracy at pool size 5, while SAC+RAD reaches 98 66%. Other representation learning methods that underperform standard SAC, such as SPR and DBC, also exhibit declining probe accuracy as pool size increases (c.g., SPR drops from 94.31% % at pool size 5 to 75.48% at pool size 10).
These trends indicate that different algorithms develop representations with varying alignment to the spatial reasoning demands of the task. The consistent link between probe accuracy and sample efficiency across diverse methods suggests that SPGym can help identify which learning procedures lead to representations that better support task performance.
Full linear probe accuracy data for all agents and pool sizes are presented in Table 10 below. Comprehensive performance metrics, including the sample efficiency data used in the correlation analysis, can be found in Appendix C.2.
C.2. Detailed Performance Analysis
This section provides comprehensive analysis of algorithmic performance in SPGym, examining how different representation learning approaches handle increasing visual diversity. We present detailed learning curves showing the training dynamies of each method, along with quantitative performance metrics across varying pool sizes.
The analysis includes detailed algorithmic variant analysis explaining why certain methods succeed or fail, and com- prehensive performance curves for PPO (Figure 10), SAC (Figure 11), and DreamerV3 (Figure 12). Tables 11 to 13 provide quantitative metrics per pool size (Pool) including sample efficiency (Steps), and episode length (Length), across algorithms and variants. Tables 11 and 12 also include in-distribution and out-of-disttibution success rates (ID Success, OOD Easy, OOD Hard), which were not computed for DreamerV3 due complexities in the original codebase. The rare non-zero success rates in Hard OOD likely stem from chance encounters with nearly-solved initial states rather than genuine generalization.
23

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
Table 10. Linear probe accuracy (%) ) for each agent and pool size. Accuracy of the linear probe indicates linearly separable features capturing task-relevan spatial information. There is statistically significant correlation between probe accuracy and agent sample efficiency (steps to 80% success).
Agent
Pool Size
Steps (M)
Linear Probe Accuracy (%)
1
1.7520.44
99.81=0.14
5
7.80+1.08
96.42:+1.00
PPO
10
9.73a0.36
87.84+1.440
20
10.00+0.00
66.97:15.28
0.95+0.21
99.81+00112
PPO+PT(ID)
5
5.55+1.22
96.83=0.61
10
9.17a1.10
89.54=0.95
1.3440.42
99.59+0.33
PRO+PT(OOD)
5
7.03+1.07
95.68110.7
10
9.7020.41
88.90=1.111
0.33+0.07
100.00=0.00
5
0.91
97.63110.68
10
1.6520.31
93.34+0.48
SAC
20
4.52+1.43
80.74++6.31
30
9.2320.96
66.6918.41
50
10.00+0.00
55.52:10.09
1
0.24+0.03
99.99--0.01
SAC+RAD
5
0.4220.06
98.66:10.20
10
0.82a0.18
89.7410.73
0.46+0.10
99.98+00.03
SAC+CURL
5
1.5620.31
97.1410.17
10
5.24+1.92
89.47=1.23
2.0920881
99.99-1001
SAC+SPR
5
3.6821.68
94.3111.24
10
10.00a0.00
75.48+1.82
0.4420.04
100.00=0.00
SAC+DBC
5
0.99+0.25
94.26+11119
10
10.00+0.00
76.59
0.4220.09
100.00=0.00
SAC+AE
5
1. 1.04+0.24
95.52+4.56
10
2.03+0.38
88.66 1.88
1.1320.14
99.6610.00
SAC+VAE
5
5.30a0.68
78.21+2.35
10
10.00+0.00
64.76-10.11
0. 0.98+0.88
99.90+00.03
SAC+SB
5
2. 2.08
96.69
10
10.00+0.00
81.93+6.06
24

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
C.2.1. ANALYSIS OF ALGORITHMIC VARIANTS
Our preliminary analysis of algorithmic variants suggests how method assumptions may influence effectiveness in SPGym. PPO with in in-distribution pretraining (PT (ID)) significantly boosts sample efficiency across all pool sizes, though this advantage diminishes at pool size 10. Out-of-distritution pretraining (PT (OOD)) offers more modest gains that also decrease with larger pools, suggesting limited transfer from ceneral-purpose pretrained encoders. For SAC, data augmentation via RAD consistently improves efficiency across all pool sizes, with particularly pronounced benefits for larger pools. Conversely, many sophisticated auxiliary methods struggle: CURL, SPR, and VAE variants consistently require more samples than standard SAC, with particularly poor performance on larger pools. DBC and AE generally underperform or offer marginal improvements. DreamerV3 demonstrates particularly strong performance, consistently outperforming both PPO and SAC variants across all pool sizes with remarkably stable performance. The variant without decoder gradients shows reduced performance, highlighting the importance of the reconstruction objective and suggesting that learning predictive environment model provides an effective foundation for handling visual diversity. We now provide more detailed hypotheses for these behaviors.
Pretraining. As shown in Figure 10, pretraining provides clear benefits for PPO, especially with larger pools. While in-distribution pretraining provides strong gains, almost matching the performance of PPO with one-hot-based observations (see Appendix C.1.1), this represents an optimistic upper bound since real-world scenarios rarely permit task-specific pretraining. Interestingly, 86% success at pool size 5), suggesting some transfer of useful visual features from the pretrained encoder. This indicates that even general- purpose pretrained encoders can provide helpful initialization for RL tasks, though not matching the performance of task-specific pretraining.
dittr--dittiibutinn pretraining also shows benefits compared to random initialization (90% vs
Data augmentation. RAD succeeds by enforcing spatial invariances through grayscale+channel shuffling, preserving structural relationships critical for puzzle solving while adding beneficial stochasticity. Its weak assumptions make it robust across diversity levels, maintaining strong and consistent performance across pool sizes through this simple augmentation-based approach.
Contrastive learning. CURL underperforms as instance discrimination may prioritize whole-image features over tile-level details needed for puzzle solving. This suggests contrastive learning's focus on global image similarity may not align well with the local spatial reasoning required for puzzle solving.
State similarity learning. DBC fails possibly because its core assumption - that states with similar dynamics should have similar representations breaks down in two ways: identical puzzle states appear radically distinct between episodes with different sampled images, while different states can share visual patterns due to being from the same episode or having the same base image.
Temporal consistency. While the environment's underlying dynamics are deterministic, temporal consistency methods such as SPR and SB face three key challenges: (1) The visual manifestation of state transitions varies dramatically between episodes due to different base images, forcing the encoder to learn position-invariant invariant representations that capture tile relationships rather than visual content a difficult disentanglement problem. (2) The assumption of smooth latent space transitions is violated by the discrete nature of tile movements, where single actions induce significant changes in both visual appearance and puzzle state. (3) Most crucially, these methods must simultaneously learn two competing objectives: temporal predictability in latent space (for transition modeling) and visual discriminability (for representation learning). This creates a conflict where features useful for predicting latent transitions (tile positions) are obscured by visually salient but dynamically irrelevant image content. SPR's prediction horizon mechanism exacerbates this by compounding representation errors through multiple latent transition steps. Similarly, SB's transition/reward prediction suffers because the latent space conflates visual features with positional information while rewards depend solely on tile positions, the visual diversity in observations provides no direct positional cues. The same absolute position (e.g., top- corne) shows completely different visual content each episode, making aose latent representations particularly difficult to learn.
Reconstruction-uee based learning. The success of DreamerV3's decoder highlights the value of the discrete reconstruction loss in learning useful representations for SPGym. In contrast, simple autoencoders (AE) offer little benefit for SAC, and variational autoencoders (VAE) hurt performance possibly because their continuous latent space assumptions conflict with SPGym's 's discrete state transitions.
25

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
These findings align with observations from Tomar et al. (2023), who noted that many representation learning methods underperform fail completely when tested outside their original domain. We note that evaluation focused using each algorithm's s suggested hyperparameters for visual RL with discrete actions, aiming to assess their out-of-the-box performance. The results may not reflect the best possible performance achievable through extensive hyperparameter tuning.
Table 11. Performance metrics for PPO agents across image pool sizes, 'Steps' indicates the number of environment steps (in millions) required reach 80% success during training. 'ID Success' shows the final success rate the end of 10M training steps when evaluating on the same pool used during training. 'OOD Easy' shows the success rate when evaluated on 100 unseen images from the "Easy' distribution. 'OOD Hard" shows the
when evaluated 100 unseen images from the 'Hard' distribution. 'Lengthh
indicates the average number of steps required to solve an instance of the puzzle at the last 100 episodes.
Algorithm Variation
Pool
Steps (M)
ID Success (%)
OOD Easy (%)
OOD Hard (%)
Length
1.7520.44
10000
0.4920.13
0.020.0
72.9244
Standard
1.80
86*16
0.53+0.14
0.3+0.3
212.0=190
10
9.7320.36
47a18
0.3420.08
0.6 0.620.4
599.6x25.6
20
10.00+0.00
12+7
0. 0.12+0.03
0.0a0.0
903.4=23.9
PPO
0,95+0.21
100g0
0.3320.09
0. 1.0.
76.224.8
PT (ID)
5.5521.22
100g0
0.5320.16
0.3a0.4
83.247.2
10
9.1721.10
38a20
0.27z0.07
0.920.7
658.4
1.34s0.42
100a0
0.49+0.12
0.240.3
72.7+40
PT (OOD)
7. 1.03
90+16
0.52+0.14
0.3+0.3
156.0=16.0
10
9.7020.41
46119
0.34
0.821.2
572.0226.0
26

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforeement Learning
Table 12. Performance metries for SAC agents across image pool sizes. 'Steps' indicates the number environment steps (in millions) required to reach 80% success rate during training. 'ID Success' shows the final success rate at the end of 10M training steps when evaluating on the same pool used during training. 'OOD Easy" shows success rate when evaluated on 100 unseen images from the 'Easy" distribution. 'OOD Hard" shows the success rate when evaluated on 100 unseen images from the 'Hard" distribution. 'Length'
indicates the average number steps required to solve an instance the puzzle at the last 100 episodes.
Algorithm Variation
Pool
Steps (M)
ID Success (%)
OOD Easy (%)
OOD Hard (%)
Length
1
0.33a0.07
100s0
0.45
0.0a0.0
86.7+195
5
1.91
100k0
0,5820.12
0.020.0
76.8216.2
Standard
10
1.65=0.31
100s0
0.46#0.12
0.0+0.0
78.7+16.5
20
4.52+1.43
98*2
0.35
0.0z0.0
63.2+18.4
30
9.2 9.2320.96
47238
0.1920.04
0.0a0.0
525.0263.
10.00+0.00 50
7+5
0.06+0.02
0.0+0.0
917.0x35.5
a.24 1
10040
0.62a0.15
0.0a0.0
50.9
RAD
0.42+0.06 5
100s0
0.42+0.13
0,0+0.0
76.9+12.0
0.82.0.18 10
100k0
0.3020.11
0.020.0
144. 1
0.46#0.10 1
100s0
0.76x0.09
0.0=0.0
77.2+13.1
CURL
1.56+0.31 5
100e0
0,44
0.2+0.4
84.9+17.7
5.24+1.92 10
100+0
0.37
0.0+0.0
88.0+20.3
2.0920881 1
10020
0.65
0.621.2
206.4223.8
SAC
SPR
3.68 5
69a13
0..09
0.0a0.0
523.9255.6
10.00=0.00 10
9+3
0.07+0.04
0.0+0.0
912.1+36.0
0.4420.04
10020
0.4420.13
0.020.0
65. L213.8
DBC
0.99=0.25 5
100go
0.34
0.0z0.0
111.2+22.7
10
10.00.0.00
2a1
0.1320.04
0.020.0
588.5163.3
1
0. 0.42+0.09
100+0
0.78+0.11
0.0+0.0
85.4+22.1
AE
1.0420.24 5
100z0
0.6420.16
0.0z0.0
102.6x226
2.03+0.38 10
100s0
0.55+0.12
1.3+26
78.0x17.1
1. 1.1320.14
100kg
0.6420.15
0.020.0
12116
VAE
5.30#0.68 5
100s0
0.30a0.08
0.0+0.0
81.2+18.2
10.0000.00 10
25a17
12.003
0.420.5
834.3+47.5
1
0.98
100a0
0.8920.08
0.0g0.0
130.
SB
5
2.08
91=17
0.65a0.12
0,0z0.0
117.8+20.5
10
10.0000.00
3a2
0.06a0.02
0.2+0.4
980.3+18.0
Table 13. Performance metrics for DreamerV3 agents across image pool sizes. 'Steps' indicates the number of environment steps (in millions) required reach 80% success rate during training. 'ID Success' shows the final success rate the of 10M training steps when evaluating on the same pool used during training. 'Length' indicates the average number of steps required to solve an instance of the puzzle at the last 100 episodes.
Algorithm
Variation
Pool
Steps (M)
ID Success (%)
Length
1
0.42#0.6
100s0
83.5
5
1.23a0.00
100g0
31.0a3.9
10
1.44+0.58
100+0
32.9+4.0
Standard
20
3,96#0.61
100+0
27.8+3.5
DreamerV V3
30
5,84a0.71
99a1
38.0a6.
50
6.62+2.67
87114
177.82248
100
8.18+2.33 1
29=14
676. 1432.
1.13+0.12
100+0
36,0+2.5
w/o decoder
5
1.7920.61
100ke
42, 1332
10
2.574n.1
100+0
46.7+3.9
27

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
4M
12M
DreamerV3
DreamerV3 w/o reconstruction
PPO
3.5M
PPO pretrained (OOD)
10M
Samples to solve
3M
Samples to solve
pretrained (ID)
8M
6M
1.5M
4M
1M
2M
0.5M
0
10
5
10
Image pool size
Image pool size
12M
SAC
SAC RAD
SAC CURL
SAC
SAC
SAC SPR
SAC
10M
Samples solve
6M
2M
Image pool size
Figure 9. Sample efficiency of different methods across pool sizes (lower is better). SPGym differentiates agents based their representation lcarning capabilities. Top left: DreamerV3 variants demonstrate the value reconstruction learning. Top right: PPO results show benefits of pretraining. Bottom: Comprehensive comparison of SAC variants reveals trade-off' between different representation learning approaches.
28

---

Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
Pool Size
Pool Size 5
Pool Size 10
100
Success Rate
1M
2M
3M
4M
2M
4M
6M
8M
2M
4M
6M
8M
Steps
Steps
Steps
PPO PPO pretrained (OOD) PPO pretrained (ID)
Figure Learning curves for PPO variants. Success rate during training baseline PPO versions with pretrained encoders across different pool sizes. Shaded regions represent 95% confidence intervals across independent runs.
Size
Size
Pool Size 10
Success Rate
2M
4M
6M
8M
2M
4M
6M
8M
2M
4M
6M
8M
Steps
Steps
Steps
SAC SAC + RAD = SAC + AE SAC CURL SAC + DBC SAC VAE = SAC + SPR
SAC + SB
Figure Learning curves SAC variants. Success rate during training baseline SAC versions with different representation
learning components across different pool sizes. Shaded regions represent 95% confidence intervals across independent runs.
Pool Size
Size
Size 10
100
Success Rate (%6)
80
60
40
20
MI AWW A
0
0.2M 0.4M 0.6M 0.8M
1M 1.2M 1.4M
0.5M
1M
1.5M
2M
2.5M
0.5M
1M
1.5M
2M
2.5M
3M
3.5M
Steps
Steps
Steps
DreamerV3
DreamerV3 w/o reconstruction
Figure 12. Learning curves for DreamerV3 variants. Success rate during training for Dreamer V3 with and without decoder across different pool sizes. Shaded regions represent 95% confidence intervals across independent runs.
29


Reviews:
[{"review_id": "ycKAgx02ru", "reviewer": "Reviewer_o6ZL", "summary": "The paper introduces SPGym, a novel benchmark for visual RL that extends the classic sliding puzzle by replacing numbered tiles with image patches. This enables scaling visual diversity while keeping the puzzle dynamics fixed, with the aim of isolating representation learning from policy learning. The authors evaluate a range of RL baselines: on-policy PPO (with ID and OOD pretrained encoders), off-policy SAC with data augmentation and representation learning techniques, and the model-based DreamerV3, to assess sample efficiency and generalization. Their experiments show that pretraining and data augmentation enhance sample efficiency, baselines react differently to increasing the images in the training pool, nearly all approaches achieve high in-distribution performance yet completely fail to generalize to unseen images, and PPO continues to learn more efficient solutions after reaching 100% average success.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thorough and constructive feedback. We appreciate the recognition of SPGym's potential and the detailed suggestions, which will significantly improve the paper.\n\n## 1. Answers to Direct Questions\n\n1.  **Missing Tile:** The missing tile's starting position is randomized in each episode as part of the initial shuffling.\n2.  **Training Termination (\"Early Stopping\"):** Our primary motivation for terminating runs based on sustained success rate was computational efficiency, given the number of experiments. A secondary reason was to evaluate OOD generalization before potential extreme overfitting. We agree \"early stopping\" is imprecise terminology, as agents can still improve solution efficiency, and will rephrase this in the revision.\n3.  **PPO 4x4 Convergence:** PPO does not converge within 10M steps on the 4x4 grid. Extended runs (100M steps) show it requires approximately 24.5M ± 7.6M steps (avg. 5 seeds) to reach 80% success, indicating feasibility but requiring much more data. We will add this finding.\n\n## 2. Addressing Key Concerns\n\n1.  **Disentanglement of Representation and Policy Learning:** We acknowledge the reviewer's point on end-to-end learning. However, SPGym's design isolates the visual representation challenge as the key variable by keeping all underlying MDP components fixed and only varying the visual observation function. Performance differences thus reflect agents' visual processing capabilities. For a detailed explanation of the POMDP formulation and supporting empirical evidence from linear probing, please see our response to **Reviewer R6Sn**. We will add these clarifications and results to the revision.\n\n2.  **Optimal Solution Claim:** You are correct. Given our termination criterion and that only PPO was tested for continued training, the claim that best-performing agents fall short might be too strong. We will revise this to reflect that agents under our protocol didn't reach the optimum, acknowledging the potential for improvement with longer training.\n\n3.  **Reward Function:** The Manhattan distance reward is standard in puzzle literature (Korf et al., 1985; Burns et al., 2012; Lee et al., 2022; Moon et al., 2024) and encourages minimizing steps by accumulating negative rewards until the goal (+1) is reached. While local optima are possible, this aligns the RL objective with finding efficient solutions. A perfect dense reward is non-trivial, and sparse rewards are infeasible here.\n\n4.  **Evaluation Criteria Clarity (80% Threshold):** We apologize for the omission. We calculate this by finding the first environment step where the average success rate across all parallel environments reaches 80% for each run, then averaging these step counts across seeds. We will clarify this.\n\n5.  **Data Augmentation & Hyperparameter Selection:** We acknowledge the limitation of optimizing augmentations only for RAD and applying that strategy universally. This was a trade-off for controlled comparison vs. method-specific tuning. Similarly, hyperparameters were tuned for training efficiency. Evaluating based on generalization is important future work. We will clarify these limitations.\n\n6.  **Fairness of Pretraining Comparison (PPO):** We agree it's not a direct 'from scratch' comparison. The intent was to assess the impact of leveraging pretrained features (ID/OOD) vs. learning purely from the RL signal in SPGym. We will clarify this motivation.\n\n7.  **Generalization Evaluation (Table 3 & Pool Size Choice):** The consistent OOD failure across methods and pool sizes is a key diagnostic finding about current end-to-end visual RL limitations. Our rationale for focusing Table 3 on pool size 5, preliminary findings on larger pools, and the interpretation of the OOD results are discussed in detail in our response to **Reviewer ukCk** (due to space limitations).\n\n8.  **Procedural Generation:** We agree the current results don't show a demonstrated advantage over ImageNet. We will moderate the claim, framing it as a feature with potential benefits (diversity, control, storage) for future investigation.\n\n## 3. Other Points\n\n*   **Missing References:** Thank you. We will add CARLA and COOM to the related work discussion.\n*   **Code:** Code is in the supplementary material, omitted from the paper for anonymity. We will add links in the camera-ready version.\n*   **Minor Points:** We will refine terminology (model-based definition, visual diversity) and acknowledge unevaluated features (other modalities) and future analysis directions (image difficulty).\n\nWe hope these responses and planned revisions address the reviewer's concerns. We value the feedback and are committed to improving the paper. We appreciate the reviewer's openness to reconsidering their evaluation.", "claims_and_evidence": "1. The authors repeatedly claim that SPGym disentangles representation learning from policy learning, using this as a core motivation for their work and distinguishing it from prior benchmarks like The Distracting Control Suite and ProcGen. However, I don’t see how their experimental setup fully supports this claim. While PPO is tested with pretrained encoders (both in-distribution and out-of-distribution), SAC and DreamerV3 still learn representations end-to-end, meaning representation learning is not truly isolated across all agents. SPGym does not incorporate explicit representation learning evaluations, such as frozen encoder tests or linear probes, which would help validate the claim of disentanglement. Since policy updates inherently shape learned representations, the benchmark does not provide a clear separation between the two processes.\n2. The authors claim that the best-performing agents fall short of the theoretical 22-step optimal solution, basing this assertion on the 5-pool training results in Table 2. However, agents like SAC and DreamerV3 benefit from larger training pools, with DreamerV3 achieving the lowest episode length of 27.8 steps using a 20-image pool (Table 5). Moreover, since early stopping is applied, preventing further improvements, there is evidence from PPO that continued training can significantly reduce episode lengths (from 214.30 to 31.35 steps). Given that the best version of DreamerV3 is trained using only about 40% of the allowed data budget, it has the potential to approach the theoretical optimal solution with extended training. Therefore, the claim that the best-performing agents fall short of the optimal solution is not fully substantiated.", "methods_and_evaluation_criteria": "1. SPGym has two clearly defined training evaluation metrics: 1) **task completion**, measuring whether an agent successfully solves the puzzle within the maximum episode length of 1,000 steps, and 2) **completion efficiency**, measuring how quickly an agent solves the puzzle in terms of the number of steps taken. These well-defined metrics make evaluation and comparison straightforward.\n2. SPGym measures sample efficiency by tracking the number of environment steps an agent requires to achieve an 80% success rate.\n3. Increasing the training image pool size introduces greater visual diversity while keeping the underlying dynamics unchanged, allowing for an analysis of how different algorithms perform across varying levels of visual complexity (Figure 4 & Table 5).\n4. SPGym is effective for evaluating generalization to unseen data, as all baseline methods fail to transfer their learned representations to unseen images, regardless of the training pool size. \n5. One of my main concerns is the **reward function**. Using the Manhattan distance between each tile’s current and target positions as a reward signal discourages moves that might be necessary for reaching the optimal solution. There are likely scenarios where a tile that is already in place, or close to it, must be temporarily moved further away to solve the puzzle efficiently. This reward function risks biasing the agent toward locally optimal but globally suboptimal strategies, which could explain the poor baseline performance. While I acknowledge that a purely sparse +1 reward upon completion would make learning infeasible due to the extremely low probability of solving even a 3×3 puzzle through random actions, the current approach may still be counterproductive. The authors should design a dense reward signal that does not penalize necessary intermediate steps or provide empirical evidence that the current formulation does not hinder learning.", "theoretical_claims": "This paper does not contain theoretical claims or formal proofs; its primary contributions are experimental and methodological.", "experimental_designs_or_analyses": "1. The authors evaluate a diverse set of SOTA baselines, including on-policy PPO with pretrained encoders, off-policy SAC with various data augmentation strategies, and multiple recent variants from the literature, along with the model-based DreamerV3. This strengthens the paper’s contributions.\n2. The baseline comparison of increasing the training pool size is insightful, as it reveals how well the algorithms adapt to greater visual diversity. \n3. In Table 2, the authors assess sample efficiency by measuring the number of samples required for baseline methods to reach an 80% success rate with a small image pool. Establishing a clear evaluation criterion for sample efficiency is beneficial for the benchmark. However, while they mention that early stopping occurs when the agent achieves a 100% success rate for 100 consecutive episodes, they do not specify the window size used for averaging the 80% success rate threshold. This omission makes it unclear how success rates are computed and whether short-term fluctuations could affect the reported results.\n4. The authors determine the best data augmentation strategy by evaluating only **RAD**, then apply the selected approach (grayscale + channel shuffle) to all augmentation-based methods, including **CURL** and **SPR**. This is problematic because different methods have distinct augmentation requirements. **CURL** [1] has been shown to benefit more from cropping rather than random color shuffling, while **SPR** [2] relies on temporal consistency and may be particularly sensitive to spatial distortions. Optimizing augmentations solely for RAD does not guarantee optimal performance for other methods, potentially skewing the results and disadvantaging CURL and SPR. This could explain why RAD outperforms CURL and SPR in sample efficiency.\n5. The Hyperparameter Selection and Data Augmentation Analysis focus solely on optimizing training performance. However, since the benchmark also assesses evaluation, the selection process should consider generalization performance, not just faster learning. The chosen hyperparameters may improve training efficiency but contribute to overfitting, making it crucial to evaluate augmentation methods based on their impact on generalization.\n6. Table 2 shows that pretrained encoders improve PPO’s sample efficiency. However, this comparison is not entirely fair, as the pretrained encoders have already seen a large number of samples during pretraining, even if from OOD data. This gives them a significant advantage over agents learning representations from scratch, making direct comparisons of sample efficiency misleading.\n7. Table 3 presents the ID and OOD evaluation results, highlighting SPGym’s effectiveness as a generalization evaluation tool. While a training pool of only 5 images is understandably insufficient for meaningful generalization, the authors note that even models trained on pools of up to 100 images fail to transfer to unseen images. In my view, this shows the benchmark’s core challenge, demonstrating that overfitting remains a problem regardless of pool size. However, since SAC and DreamerV3 achieve strong in-distribution performance with larger training pools over 10M steps, it would be more informative to display their main results in the table on a larger pool size to better assess their generalization potential. Additionally, the authors only evaluate OOD performance for the base PPO, SAC, and DreamerV3 models. Extending this analysis to other baselines included in the study, such as data-augmented or contrastive learning variants, would provide a more comprehensive understanding of how different representation learning methods handle distribution shifts.\n8. The authors apply early stopping once agents achieve a 100% success rate for 100 consecutive episodes, yet their own results show that agents continue to improve solution efficiency well beyond this point. The first 100 successful episodes average 214.30 steps, whereas the final 100 episodes average just 31.35 steps, indicating that further training yields significantly more optimal solutions. This raises concerns about the motivation behind early stopping in this setting. Typically, early stopping is used when a model has converged and is no longer learning anything meaningful. Here, it appears to halt training prematurely, limiting the evaluation of efficiency improvements. \n9. The use of procedurally generated images, such as those from DiffusionDB, offers advantages like reduced storage overhead and access to near-infinite diversity, potentially aiding controlled generalization. However, Figure 5 shows that performance trends closely mirror those on ImageNet, suggesting that this diversity does not significantly impact learning outcomes under the tested conditions. While procedural generation allows fine-grained control over visual complexity, the results do not demonstrate a clear advantage over large pre-existing datasets. To strengthen this contribution, the authors could analyze its benefits in terms of computational efficiency, storage, or controlled generalization.\n\n[1] Laskin, Michael, Aravind Srinivas, and Pieter Abbeel. \"Curl: Contrastive unsupervised representations for reinforcement learning.\" *International conference on machine learning*. PMLR, 2020.\n\n[2] Schwarzer, Max, et al. \"Data-efficient reinforcement learning with self-predictive representations.\" *arXiv preprint arXiv:2007.05929* (2020).", "relation_to_broader_scientific_literature": "The authors position their work among benchmarks that introduce visual diversity, such as ProcGen and DM Control Suite, and those that add visual noise, like the Distracting Control Suite. They claim that SPGym uniquely disentangles representation learning from policy optimization by extending the 8-Puzzle task, previously used in RL research, to support visual observations.", "essential_references_not_discussed": "There are other benchmarks tailored for visual RL that have a setting which keeps the environment dynamics the same while increasing visual diversity [1, 2], similar to SPGym.\n\n[1] Dosovitskiy, Alexey, et al. \"CARLA: An open urban driving simulator.\" *Conference on robot learning*. PMLR, 2017.\n\n[2] Tomilin, Tristan, et al. \"Coom: A game benchmark for continual reinforcement learning.\" *Advances in Neural Information Processing Systems* 36 (2023): 67794-67832.", "other_strengths_and_weaknesses": "1. Since the authors introduce a new benchmark, access to the code would be beneficial for properly assessing their work. However, they have not uploaded their code or provided a link to an anonymous repository.\n2. This benchmark has strong potential to be valuable to the community. The 8-Puzzle problem with visual observations is a commendable proposal, particularly since it aligns with how humans approach the task. Benchmarks introducing novel problems are always beneficial. However, I am not fully convinced by the strong emphasis on disentangling feature and policy learning, as the current evidence does not fully support this claim. Additionally, some aspects of the experimental evaluation need improvement, as outlined above. I am very open to raising my score if the authors adequately address my concerns.", "other_comments_or_suggestions": "1. The authors provide wrappers that allow for different observation modalities beyond image-based tiles, including text and one-hot encodings. While this adds flexibility to SPGym, it is not a fully realized contribution since there is no empirical analysis or baseline evaluation using these alternative modalities. Without an evaluation of how different modalities impact learning performance, this aspect remains more of a hypothetical feature rather than a demonstrated advantage.\n2. The authors state that they explore three algorithmic paradigms: off-policy, on-policy, and model-based. However, model-based methods can themselves be either off-policy or on-policy, depending on how data is used. The distinction could be better clarified for conceptual accuracy.\n3. The evaluation of data augmentation methods is solely based on their impact on training performance, without assessing their effect on generalization during evaluation.\n4. It would be interesting to analyze whether some images make it more difficult to solve the puzzle than others.\n5. The authors repeatedly state that increasing the training pool size increases visual complexity. However, I believe it only increases visual diversity rather than complexity, unless certain images make it more difficult for the agent to solve the puzzle.", "questions_for_authors": "1. Is the missing tile always in the bottom-right corner, or is it random across episodes?\n2. Why do the authors use early stopping if the model still has the potential to learn more efficient solutions?\n3. Does PPO converge when trained for 10M steps on the 4x4 grid, or could longer training improve performance further?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w0moDOYysZ", "reviewer": "Reviewer_ukCk", "summary": "This paper presents SPGym, a new benchmark for visual reinforcement learning (RL) based on the classic 8-tile puzzle. SPGym uses a visual observation space derived from large datasets and allows researchers to manipulate representation complexity by adjusting visual diversity. Experiments using model-free and model-based RL algorithms on SPGym reveal that current methods struggle to generalize across varying visual inputs, with performance degrading as visual diversity increases.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback on SPGym's potential and the detailed, constructive comments regarding the evaluation settings and experimental design. We address the concerns below:\n\n## 1. In-Distribution Evaluation\n\nWe understand the concern that Table 2 results might seem like overfitting. However, our primary goal with the in-distribution (ID) setting is not to evaluate generalization across images but to measure the sample efficiency with which different RL agents learn useful visual representations under controlled visual diversity. We chose pool size 5 because it offers a balance: it introduces enough visual diversity to discriminate between different representation learning approaches (as seen by the varying steps-to-convergence) while remaining learnable for most agents within our computational budget.\n\nAs detailed in our response to Reviewer R6Sn, SPGym is formulated as a POMDP where only the visual observation function changes between settings; the underlying task dynamics remain fixed. Therefore, differences in sample efficiency (Table 2) directly reflect how effectively each agent's representation learning component handles the visual aspect of the task.\n\n## 2. Out-of-Distribution Evaluation\n\nWe acknowledge the reviewer's point that zero OOD success might seem like a limitation. However, we view this consistent failure not as a flaw of the benchmark, but as a crucial diagnostic finding about the limitations of current end-to-end visual RL methods when faced with generalizing visual features learned purely through a task-specific RL objective. Standard RL objectives, like the one used here, do not explicitly optimize for generalization to unseen visual appearances of the same underlying state. SPGym effectively highlights this gap.\n\nOur choice to present the main OOD results (Table 3) using agents trained on pool size 5 stems from this being the setting where most methods achieved high ID success. Evaluating OOD generalization is most meaningful when agents have demonstrated competence on their training distribution. While some base agents achieved ID success on larger pools, they still failed completely OOD, reinforcing the finding that simply increasing training diversity within this range wasn't sufficient for generalization with current methods. Showing zero OOD success across methods aims to motivate research into algorithms with better visual generalization properties.\n\n## 3. Training Set Size\n\nWe appreciate the suggestion to explore significantly larger training pools. In preliminary experiments, we attempted training with pools of thousands of images. However, we found that with such high visual diversity, the RL training became unstable. Because each observation was essentially unique (rarely seen more than once), the gradients from the RL objective were insufficient to train a useful visual encoder from scratch. Without a stable encoder, the policy failed to learn.\n\nTherefore, our approach was to identify the approximate limit of visual diversity that current standard algorithms (PPO, SAC, DreamerV3) could handle within a reasonable budget (10M steps), which led to testing pools up to 20, 50, and 100, respectively. This reveals the scaling limitations of these methods rather than aiming for OOD generalization via massive datasets, which might require different training paradigms (e.g., pretraining on external data, different objectives).\n\n## 4. Tiered Generalization Evaluation\n\nWe thank the reviewer for this excellent suggestion. We agree that evaluating generalization across different levels of visual similarity would provide deeper insights. As a preliminary step in this direction, we evaluated the trained PPO and SAC agents on augmented versions of their training images. We observed a strong correlation (Pearson r=-0.81, p=2.5e-12) between success rates on these augmented images and the agents' sample efficiency, suggesting a link between robustness to simple transformations and learning speed. We plan to include these results and discuss the tiered approach as a key direction for future work in the revised manuscript:\n\n|Agent/Pool|1|5|10|20|30|50|\n|-|-|-|-|-|-|-|\n|PPO|0.49±0.13|0.53±0.14|0.34±0.08|0.12±0.03|-|-|\n|PPO+PT(ID)|0.33±0.09|0.53±0.16|0.27±0.07|-|-|-|\n|PPO+PT(OOD)|0.49±0.12|0.52±0.14|0.34±0.08|-|-|-|\n|SAC|0.45±0.12|0.58±0.12|0.46±0.12|0.35±0.11|0.19±0.04|0.06±0.02|\n|SAC+AE|0.78±0.11|0.64±0.16|0.55±0.12|-|-|-|\n|SAC+VAE|0.64±0.15|0.30±0.08|0.12±0.03|-|-|-|\n|SAC+SPR|0.65±0.13|0.21±0.09|0.07±0.04|-|-|-|\n|SAC+DBC|0.44±0.13|0.34±0.13|0.13±0.04|-|-|-|\n|SAC+CURL|0.76±0.09|0.44±0.10|0.37±0.11|-|-|-|\n|SAC+RAD|0.62±0.15|0.42±0.13|0.30±0.11|-|-|-|\n|SAC+SB|0.89±0.08|0.65±0.12|0.06±0.02|-|-|-|\n\nWe will revise the paper to clarify the evaluation rationale, incorporate the results on augmented images, and explicitly frame the tiered generalization as important future work, addressing the points raised. Thank you again for the constructive feedback.", "claims_and_evidence": "The reviewer agrees that the ability to \"scale the representation learning complexity while maintaining consistent environment dynamics\" is a crucial aspect of research in visual reinforcement learning. The proposed SPGym environment, despite its simplicity, demonstrates potential in fulfilling this objective. By merely utilizing different images within the puzzle, one can effectively control the visual complexity of the task.", "methods_and_evaluation_criteria": "The reviewer has concerns regarding the evaluation settings.\n\n1. In the 'in distribution' setting, there appears to be no separate hold-out validation set for evaluating the learned policy. Given the deterministic nature of the game, the reported numbers in Table 2 seem to only reflect how well each method overfits to the five training game instances. The reviewer found that the results did not provide clear conclusions or insights about the tested algorithms.  \n\n2. Table 3 presents results that suggest none of the existing algorithms were able to solve the puzzle when the testing image was not included in the training pool. This indicates a significant limitation of the benchmark. A benchmark that results in a zero success rate for all RL algorithms does not provide a useful basis for comparison or meaningful insights into the relative strengths and weaknesses of different algorithms.\n\nThe reviewer believes that the SPGym environment still holds potential despite the aforementioned issues. \n\nFor instance, one could use the similarity between training and testing images as a parameter to control the difficulty of the visual task. This could involve a tiered approach:  \n\n- Easy Level: The testing images could be augmented versions of the training images.\n- Intermediate Level: The testing images could be drawn from the same classes as the training images.\n- Hard Level: The testing images could be sourced from entirely different datasets than the training images.\n\nBy evaluating different RL algorithms across these varying levels of visual difficulty, researchers could gain deeper insights into the visual capabilities and generalization abilities of these algorithms.", "theoretical_claims": "No formal theoretical claim is presented.", "experimental_designs_or_analyses": "In addition to the issues discussed in 'Methods And Evaluation Criteria' section, the reviewer has concerns regarding the size of the training sets. Utilizing only five images in the training set appears insufficient, and the reviewer finds it unsurprising that existing algorithms fail to generalize to unseen images under these conditions. While the authors mention experimenting with training sets containing up to 100 images, this quantity is still considered limited by the reviewer. The reviewer suggests exploring the use of significantly larger training sets, on the order of 100,000 or even 1 million images, to assess the impact on generalization performance.", "relation_to_broader_scientific_literature": "This paper has close connections to visual reinforcement learning, such as [a, b]. The goal of the paper is to create an environment that can separate visual complexity from dynamics, which is indeed an important topics in the field. \n\n[a] Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages, Ma, 2024. \n\n[b] DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization, Xu, 2024.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The paper is generally easy to follow and well organized.", "other_comments_or_suggestions": "None", "questions_for_authors": "Please see the above discussions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q8uDjGO1IO", "reviewer": "Reviewer_R6Sn", "summary": "The paper introduces a sliding puzzle based environment for evaluating visual RL. It provides a number of baselines on the environment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive assessment of our experimental design and literature review, and for the opportunity to clarify the core motivation behind SPGym regarding the evaluation of visual representation capabilities.\n\n## Why SPGym Tests Visual Representation Capabilities\n\nThe central idea behind SPGym is to create a controlled environment where the primary challenge being scaled is the agent's ability to process and understand diverse visual inputs, even though the agent is trained end-to-end on a standard RL task. We achieve this through specific design choices:\n\n1.  **POMDP Formulation:** As detailed in Section 3, SPGym is formulated as a Partially Observable Markov Decision Process (POMDP) defined by $(\\mathcal{S}, \\mathcal{X}, \\mathcal{A}, \\mathcal{P}, \\mathcal{R}, \\mathcal{S}_0)$. Critically, the agent never observes the true underlying puzzle state $s \\in \\mathcal{S}$. Instead, it only receives visual observations $x \\in \\mathcal{X}$. The agent must infer the relevant state information solely from these high-dimensional visual inputs.\n\n2.  **Isolating the Visual Challenge:** The key design element is that across all experiments and all compared agents, the core components of the MDP remain fixed: the state space $\\mathcal{S}$ (tile permutations), the action space $\\mathcal{A}$ (up, down, left, right), the deterministic transition dynamics $\\mathcal{P}$ (how tiles move), the reward function $\\mathcal{R}$ (based on Manhattan distance), and the initial state distribution $\\mathcal{S}_0$. The only thing that changes between experimental conditions (e.g., different pool sizes) or agent comparisons (e.g., different representation learning modules) is the emission function that maps the underlying state $s$ to the visual observation $x$. We vary this by overlaying the puzzle state with different pools of images.\n\n3.  **Performance Reflects Representation Quality:** Because all core task elements (dynamics, rewards, etc.) are constant, any difference in performance (e.g., sample efficiency) between agents or across different visual diversity levels must be attributed to how effectively the agent's visual encoder maps the observation $x$ to a useful internal representation. An agent cannot solve the puzzle without implicitly or explicitly understanding the tile configuration from the image. Better visual representations enable the policy to make better decisions, leading to faster learning and higher success rates. While policy learning is intrinsically linked, the bottleneck being systematically stressed and evaluated is the visual representation learning component.\n\n### Empirical Evidence\n\nTo provide further empirical support for this link between representation quality and task performance, we conducted linear probe evaluations (inspired by feedback from reviewer o6ZL) on the frozen encoders learned by PPO and SAC agents. We trained linear classifiers to predict tile positions from the learned features. Our results show a statistically significant negative correlation (Pearson r=-0.81, p=1.1e-13) between the probe's test accuracy and the number of environment steps the RL agent needed to reach 80% success. This demonstrates that encoders capable of producing more informative features (higher probe accuracy) enable faster learning on the downstream RL task. The probe accuracies for tested agents follows:\n\n|Agent/Pool|1|5|10|20|30|50|\n|-|-|-|-|-|-|-|\n|PPO|99.81±0.14|96.42±1.06|87.84±1.40|66.97±5.28|-||\n|PPO+PT(ID)|99.81±0.12|96.83±0.61|89.54±0.95|-||-|\n|PPO+PT(OOD)|99.59±0.33|95.68±0.77|88.90±1.11|-||-|\n|SAC|100.00±0.00|97.63±0.68|93.34±0.48|80.74±6.31|66.69±8.41|55.52±0.09|\n|SAC+RAD|99.99±0.01|98.66±0.20|89.74±0.73|-||-|\n|SAC+CURL|99.98±0.03|97.14±0.17|89.47±1.23|-||-|\n|SAC+SPR|99.99±0.01|94.31±0.24|75.48±1.82|-||-|\n|SAC+DBC|100.00±0.00|94.26±1.19|76.59±5.82|-||-|\n|SAC+AE|100.00±0.00|95.52±4.56|88.66±1.88|-||-|\n|SAC+VAE|99.66±0.06|78.21±2.35|64.76±0.11|-||-|\n|SAC+SB|99.90±0.03|96.69±1.08|81.93±6.06|-||-|\n\n## Conclusion\n\nIn essence, while the agent learns end-to-end, SPGym isolates the difficulty scaling to the visual domain. An agent succeeding in SPGym, especially across varying image pools, demonstrates robustness in its visual processing specific to the task structure. We believe this setup effectively probes an agent's ability to form useful representations from pixels under varying visual conditions, which is a critical aspect of visual RL.\n\nWe will revise the Introduction and Methodology sections to incorporate this detailed explanation and motivation, ensuring it is clear how our design choices allow for the evaluation of visual representation learning capabilities in a controlled manner. We hope this addresses the reviewer's concern and clarifies the value proposition of SPGym. We appreciate the reviewer's willingness to reconsider their evaluation based on this clarification.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes. It is sound.", "relation_to_broader_scientific_literature": "The authors do a good job discussing the previous works such as distracting DMC, vanilla DMC, ProcGen, etc.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weakness related to writing: it is still not clear to me why this method would test an RL agent's visual representation capability? Clearly, an RL agent needs to put together _what_ is the goal image. It only understands so when the goal is an image that _makes sense_. From that point of view, it is not convincing to if we are _only_ evaluating the visual aspect of RL.\n\nI would encourage authors to provide a better motivation to this question in the paper. Based on the response, I am open to changing my evaluation (both positive and negative).", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a new RL benchmark along with experiments on existing model-free and model-based RL algorithms. All reviewers appreciate the quality of the contribution, even though a few elements could be improved and clarified. Reviewer o6ZL and ukCk mention the the levels of evaluation difficulty is limited and that it is unclear why the evaluation performance decreases as the image pool increases (it is likely not testing generalization in this paper but the capacity to overfit). Reviewer o6ZL mentions some experimental designs elements that could be improved.
Decision: Accept (poster)