A Survey of State Representation Learning for Deep Reinforcement Learning
Authors: Ayoub Echchahed
Pablo Samuel Castro
Venue: Accepted by TMLR ()

Abstract: Representation learning methods are an important tool for addressing the challenges posed by complex observations spaces in sequential decision making problems. Recently, many methods have used a wide variety of types of approaches for learning meaningful state representations in reinforcement learning, allowing better sample efficiency, generalization, and performance. This survey aims to provide a broad categorization of these methods within a model-free online setting, exploring how they tackle the learning of state representations differently. We categorize the methods into six main classes, detailing their mechanisms, benefits, and limitations. Through this taxonomy, our aim is to enhance the understanding of this field and provide a guide for new researchers. We also discuss techniques for assessing the quality of representations, and detail relevant future directions.

Published in Transactions on Machine Learning Research (06/2025)
A Survey of State Representation Learning for Deep Reinforcement Learning
Ayoub Echchahed
ayoub. wwwwhahedddilll..oooooo quebec
Mila Québec AT Instibute, Université de Montréal
Pablo Samuel Castro
psc com
Mila Québec AI Instibule, Université de Montréal
Google DeepMind
Reviewed on OpenReview: https: // optpr/veenee.e.... pet/forunt id- 90k34v0Htz
Abstract
Representation learning methods are an important tool for addressing the challenges posed by complex observations spaces in sequential decision making problems. Recently, many methods have used wide variety of types of approaches for learning meaningful state rep- resentations in reinforcement learning, allowing better sample efficiency, generalization, and performance. This survey aims to provide a broad categorization of these methods within a model-free free online setting, exploring how they tackle the learning of state representations differently. We categorize the methods into six main classes, detailing their mechanisms, benefits, and limitations. Through this laxonomy, our aim to euhance the understanding of this field and provide guide for new researchers. We also discuss techniques for assessing the quality of representations, and detail relevant future directions.
Contents
Introduction
4
Problem Definition
5
Formalism
2.2
Partial Observability
5
2.3
Deep Reinforcement Learning
5
2.4
4
State Representation Learning
6
2.5
Defining Optimal Representations
6
3
Taxonomy of Methods
8
3.1
Overview of the Taxonomy
8
Matric-based Methods
8
3.3
Auxiliary Tasks Methods
Data Augmentation Methods
3.5
Contrastive Learning Methods
16

---

Published in Transactions on Machine Learning Research (06/2025)
3.6
Con-Contrastive Learning Methods
18
7
Attention Attention-based Methods
20
3.8
Alternative Approaches
21
3.9
Comparison and Practical Guidelines
22
Benchmarking Evaluation
23
Common Evaluation Aspects
23
4.2
Assessing the Quality of Representations
23
5
Looking Beyond
25
5.1
Multi-Tass Representation Learning
25
5.2
Offline Pre- Training of Representations
................................
26
Pre-trained Visual Representations
26
5.4
Representations for Zero-Shot RL
27
5.5
Leveraging Language Models' Prior Knowledge
28
5.6
Multi-Modal Representation Learning
28
Other Directions
28
Conclusion
29
2

---

Published in Transactions on Machine Learning Research (06/2025)
Prologue
What is State Representation Learning? Why is it useful?
In soquential decision-making systems, state representation learning (SRL) is the process of learning to extract meaningful, lask-relevant information from raw observations. In other words, these algorithms aim to distill complex sensory inputs processed by a decision-maker into compact, structured representations, prioritizing critical features while filtering out irrelevant ones. For example, consider a simulated autonomous vehicle navigating busy urban environment, encountering diverse stimuli like traflic conditions, pedestrians, and changing weather (as shown in Fig. 1).
Complex Input Space
Simplified Representation Space
+ encodar
00
Figure 1: Simplified intuition for SRL: Raw semsory inputs from a busy environment are distilled into compressed, task-relevant representations, enabling improved cedision-making,
While the color of buildings or the type of roadside trees might be detectable, these details are irrelevant to effective navigation. Instead, elements such the positions and velocities of vehicles, traffic light statuses, road signs, and pedestrian movements are essential for effective decision-making. SRL could ensure that these crucial variables are emphasized in the learned representation, enabling policies to focus on what matters.
By reducing the complexity of the input space, SRL can enhance learning efficiency, generalization, and ro- bustness to environmental variations (e.g., altered street layouts or weather conditions), enabling autonomous driving agents trained on cities A and B transfer more easily their learned control policies to cities C and D either zero-shot with minimal fine-tuning. However, extracting the appropriate features from high- dimensional observations remains challenging problem, often addressed manually in applications such robotics and autonomous driving. SRL methods seek to automate this process, producing decision-making systems that are scalable, efficient, and adaptive.
3

---

Published in Transactions on Machine Learning Research (06/2025)
1 Introduction
The use of deep reinforcement learning (DRL) for complex control environments has several challenges, in- cluding the processing of large high-dimensional observation spaces. This problem, commonly referred to as the "state-space explosion", imposes severe limitations on the efficacy of traditional end-to-end RT, ap- proaches, which learn actions or value functions directly from raw sensory imputs, such as pixel observations. As environments grow increasingly complex, these end-to-end methods demonstrate progressively worse data efficiency and generalization, even in response to minor environmental changes. Addressing these limitations is therefore essential for scaling RL to real problems that inherently involve complex and noisy input spaces.
In response to these challenges, recent research has focused on decoupling representation learning from policy learning, treating them as two distinct problems. This strategy has proven effective for managing complex observations by transforming raw inputs into structured representations that retain essential information for decision-making while discarding irrelevant details. By improving data efficiency, state representation learning (SRL) accelerates training, enhances generalization across tasks, and strengthens DRL robustness.
End-to- End RL
SRL + RL
Actions
Camplex
Pollcy
Complex
Encoder
Input
SRL
RL
Overfitted policy, alow learning, generaltzation.
lffictency, geseralization, performanoe,
Comparison of End-to-End RL (left) and SRL+RL (right). End-to-end directly maps high-dimensional inputs to actions, while SRI separates representation learning and policy learning.
Figure 2:
Motivation. In recent years, there has been a growth in methods that integrate improved representation learning into DRT, using various approaches. However, many works present inconsistent structuring and categorization of those approaches in their related work sections, making it challenging obtain clcar and comprehensive understanding of the field. To our knowledge, past surveys provide valuable information but either do not cover the latest developments or focus exclusively on specific SRL classes. Many also group together approaches that roly on fundamentally different principles, blurring key distinctions between them.
[Lsorr et al. 2018 Ni et al. 2024 Böhmer el al. 2015 de Bruin el al. 2018 Botteghi et al. 2022
This survey therefore builds on those preceding works by providing an updated and structured analysis of the different approaches in state representation learning for DRL, organizing them based on their principles and effectiveness in different scenarios. Through detailed taxonomy, analyze the inner-wond working of these classes, highlighting their potential to improve the performance, generalization, and sample efficiency of deep-RL agents. We also explore different ways of evaluating the quality of learned state representations, and discuss promising directions for the field. Overall, this survey can serve as good resource for researchers and practitioners looking familiarize themselves with this field.
Organization. This work is structured as follows: Section introduces the foundational concepts of state representation learning (SRL) within the deep reinforcement learning (DRL) framework. It defines the problem, objectives, and the characteristics of effective state representations, providing formal basis for understanding subsequent sections. Section 3 presents the core taxonomy of SRL methods, categorizing them into six primary classes, while elaborating on their mechanisms and highlighting notable work from the literature. Section 4 addresses the critical aspect of evaluation, discussing benchmarks and metrics used assess the quality and effectiveness of state representations, including their impact on sample efficiency, generalization, and robustness. Lastly, Section 5 explores promising directions for advancing SRL in DRL, such multi-task learning, leveraging pre-trained visual models, and integrating multi-modal inputs.
4

---

Published in Transactions on Machine Learning Research (06/2025)
2
Problem Definition
2.1 Formalism
Reinforcement Learning (RL) is typically modeled as a Markov Decision Process (MDP), characterized by the tuple (S, P. R, 7). Here, S denotes the state space, and A denotes the action space. The transition probability function P: 8 x A + A(S), where ((S) denotes the space of distributions over S, defines the probability P(s'|s, (ss'ss,oo a) of transitioning from state s to state s' given action a, representing the environment dynamics. The function R: S x A -> I specifies the immediate reward R(s,a) received after taking action from state s, providing feedback on the action taken.
The objective of an RL agent is to learn a policy a : S ((A) that maximizes the expected cumulative discounted reward. Using n, the agent progressively generates experiences (s, a, r,)),, which can be organized into a trajectory т. For each trajectory т, the return G, represents the total accumulated reward from time step 1 onwards. It. is expressed immediate rewards over future ones. To evaluate how good particular state state-action pair is, we define value functions. The state value function (s) under policy л is the expected return starting from state and following policy x, given by (*) B[Ga|s; = s]. Similarly, the action-value function represents the expected return starting from state 8, taking action a, and subsequently following policy n, defined as (=, a) = E[Gr|s; = 8, a, = a]. Therefore, the objective of the agent can now be expressed as finding an optimal policy x² that maximizes (s.
G2 = Lk= * where 7 E [0, 1) is the discount factor that prioritizes
a)
2.2 Partial Observability
In many RL settings, full observability is rare. For example, in robotics, sensors might not capture all relevant state factors for optimal decision making in one time-step of data. POMDP, or partially observable MDP, generalizes the notion of a MDP by accounting for situations where the agent does not have direct access to the full state E S of the environment, hence needing to rely on past observations to infer the current state. Recurrent neural networks (RNNs) are commonly employed to address this partial observability issue, leveraging their hidden state to retain and process information from previous time steps. Another way to handle this is by concatenating the last n observations (or, 0t-+1)) to approximate a sufficient statistic for decision-making, thus mitigating the effects of partial observability. For example, agents trained on the ALE benchmark (Bellemare et al. 2013) often employ this technique, known as 'frame stacking'
In this survey, we adopt POMDP framework defined as M (S,O,A,P,R, where S is the state space and O is the observation space. An observation function, 0: O, maps underlying true state to ions, indicating that agents receives only partial, potentially noisy summary of states. This function establishes the connection between the complete state information and the limited data available to the agent. For an MDP, this function becomes the identity mapping, mcaning that cach state is fully observable. The transition function P and the reward function R are still defined over states, as in an MDP. In this setting, an end-to-end policy æ : O - A(A) maps observations directly to a distribution over actions. This framework is chosen because agents typically do not have full access to the states and must often rely partial high-dimensional inputs. Other relevant but unaddressed frameworks include Contextual Decision
Processes (Krishnamurthy et al. 2016 Jiang et al. 2017) and Block MDPs (Du et al. 2019
2.3 Deep Reinforcement Learning
Deep reinforcement learning (DRL) differs from traditional RL by utilizing deep neural networks to approx- imate value functions (or policies), either from digh-dimensional inputs, or from encoded latent states. This becomes desirable when the state space is large or continuous, which is not well-suited for methods that rely on representing state-action pairs individually in a lookup table, known as tabular reinforcement learning. DRL methods can be broadly categorized into three kind of approaches: actor-critic methods. Value-based methods, such as Deep Q-Networks (DQN) (Mnih ct al. (2013). use a neu- ral network approximate the action-value function Q(s, a). Policy-based methods, such as REINFORCE (Williams] 1992), directly parameterize the policy [a]ss; 0) and optimize it using gradient. ascent the
based, policy-based, and
5

---

Published in Transactions on Machine Learning Research (06/2025)
expected cummlative reward. Actor-Crithe methods combine both value-based and policy-based approaches as they maintain two networks: the actor, which updates policy =[a|s;0) 0), and the critic, which evaluates a value function Q(s, a) or (s)). This combination enhances stability and efficiency, making it widely used.
2.4 State Representation Learning
The traditional end-to-end approach, which directly maps observations to actions, led to impressive results (Mnih et al., 2013 However, this approach becomes increasingly challenging as the complexity of the environment increases, which is why more efforts are directed towards learning better representations. Rep- resentation learning by itself can defined as the process of automatically discovering features from raw data that are most useful for a task (Bengio et al. 2012). Although representation learning for DRL can be divided into state and action representation learning, the former will be the focus of this survey.
Problem: We define the objective of state representation learning (SRL) for reinforcement learning (RL) learning representation function dt Oo x O ... O₂ - X, parameterized by 0g. which maps k-step observation sequences to representation space X, thereby allowing us to define policies ILx over this reduced space. This encoder enables either a policy network 1/ to compute actions a; = th (2z), or a value network 4hv to compute values v, = thv (2+), based on the representation H = ((x)), instead of directly using high-dimensional observations. The representation I₁ is a vector in Rd where is the dimensionality of X.
Policy-based SRL
Value- hased
Poltey
Aciuss
Value
encsser
ercsdor
O
Ieprosantatinn
Representation
Figure 3: Illustration of State Representation Learning (SRL) for RL, where parametrized transformation ø is learned, mapping sequences observations representations. Two configurations are presented depending value-based RL approach is used (right) or policy-based one (left).
However, not all representation functions are useful to obtain; the goal is to learn an encoder that captures the essential characteristics of effective state representations, good encoder simplifies the input space into a compact and relevant representation sample efficiency and performance by facilitating the function approximation process performed by the policy/ /value nelwork; (2) enhancing generalization as learning policy /value network from representations avoids the overfitting issues seen with high dimensional, unstructured, and noisy observation spaces.
reviewed in the next section. Learning a
thereby (1) improving
In the presented taxonomy, the focus will be mostly methods that learn state representations within model-free online setting, where agents learn representations and policies in real-time through interactions with the environment without using an explicit model of the environment for taking actions. This differs from model-based RL, which involves learning a model of the environment's dynamics that is used for planning, enabling higher sample-effciency at the cost of higher complexity. Section 5 also explores the offline pre-training of representations. Finally, Table provides a structured overview of key SRI. settings.
2.5 Defining Optimal Representations
An optimal representation can be defined by its ability to efficiently support policy learning for a set down- stream tasks. The learned space should be constrained to a low dimensionality, while remaining sufficiently informative enable the learning of an optimal policy (or value function) with limited capeciiyy function approximators. If X has too much information, it slow down the learning process and hinder convergence to the optimal policy. Alternatively, a space with insufficient information will prevent convergence to the optimal policy (Abel 2022). Hence X' ideally balances well information capacity and simplicity.

---

Published in Transactions on Machine Learning Research (06/2025)
Setting
Description
Pre-trained
Joint-training
Representations are learned either before begins or multaneously with the RL training objective.
Online
Offline
Learning occurs in real-time through interactions with the euviromment or from pre-collected datasets.
Coupled
Decoupled
optimized jointly with policy and/or value objectives or independently of them.
Encoder parameters
Reward-based
Reward-free
Representations are influenced by task rewards focus euvironment dynamics and visual features.
Single-task
Multi-task
Representations learned for specific task shared across multiple tasks to capture common structures.
Model-free
Model-based
Representations directly used for decision-making making or integrated into a world model for planning.
Table 1: Overview of key settings when learning state representations for DRL.
Structure: The learned representation space should be structured to encode task-relevant information while remaining invariant to noise and distractions. This means that points within a neighborhood around a rep- resentation 2; should exhibit high degree of lask-relevant similarity, which gradually diminishes as the distance from this point increases. These similarities can be encoded in the representation space using in- formation distances derived from observation features, environment dynamics, rowards, etc. Additionally, the encoder should remain invariant to noise, distractions, or geometric transformations that do not alter the true underlying state of the agent.
Continuity: Good latent structure should ensure strong Lipschitz continuity of the value function across nearby representations (Le Lan et al. 2021). In other words, points that are close the latent space should produce similar value distribution predictions, even if they are distant in the input space. This continuity simplifies the function approximation process performed by the value network v'y and promotes better generalization to unseen but nearby stales. This similarly applies to the policy network 1h,; ensuring that changes in the action distribution occur smoothly within X.
Sparsity: Enforcing sparsity constraints on the representations can allow the identification of the most relevant aspects of high-dimensional observations as encourages the inputs to be well-described by a small subset features at any given time. This enhances computational efficiency by reducing the number of active features, lcading to simpler reprosentations.
also helps avoid overfitting by focusing on the most relevant features, promoting better generalization. Finally, improves interpretability by making it easier to understand which features drive the decision-making process of agents.
Structurally Informative and Invarlant
Disentanglement
dieegen
Lpm
Lerge change
cuange
Represssutke Space
Entangled
Diorscoggee
Figure 4: Illustration of some optimal state representation properties. The left section demoustrates invariance noise, distractions, and masking in the representation space, preserving task-relevant information. The right section illustrates disentanglement, where changes individual factors ideally lead localized latent impacts.

---

Published in Transactions on Machine Learning Research (06/2025)
Previous works have sought to define characteristics of effective representations and abstractions for RL. According to Wang et al. (2024b), optimal representations should exhibit high capacity, efficiency, and ro- bustness, A (2022) identifies three essential criteria for state abstractions in RL: efficient decision-making, solution quality preservation, and ease of construction. These criteria stress the importance of balancing compression with performance to facilitate effective learning and planning complex environments. Other works that discuss characteristics of good representations for RI. include (Böhmer al. 2015), (Lesort al. 2018), and (Botteghi et al., 2022). Moreover, definitions of optimal representations the broader context of Self-Supervised Learning (SSL) can often overlap with those needed for control, making them applicable to reinforcement learning as well.
3 Taxonomy of Methods
3.1 Overview of the Taxonomy
We categorize the representation learning methods into six distinct classes, which are presented in table 2. For each class, we provide definition, details, benefits, limitations, and some examples of methods. While there are likely other methods in each class, the goal is not to be exhaustive, but rather to focus on the classes themselves. Additionally, some methods may be hybrid, combining techniques from multiple classes.
Class
Description
Netric-based
Shape the representation space through task-relevant distance metric be- tween embeddings. They enhance generalization and efficiency by abstracting states with similar information, reducing complexity.
Auxiliary Tasks
Enhance the primary RL task with other simultaneous predictions that indi- rectly shape representations. These require additional parameters, but can provide accelerated learning the main task.
Augmentation
Leverage data augmentation for learning invariances to geometric and pho- tometric transformations observations. They do not directly learn repre- sentations, but enhance efficiency and generalization.
Contrastive
Shape the representation space by learning separate representations for dif- ferent observations, and similar ones for related inputs. Temporal proximity and/or visual transformations can define similarities.
Non-Contrastive
Construct their representation space by only minimizing the distance be- tween the representations of similar observations. Unlike related contrastive approaches, no negative pairs are used during training.
Attention-based
Learn attention masks (Bahdanau et al. 2015) for computing scores that highlight important features of the input, helping agents disregard irrelevant details and increase the interpretability of decision-making
Table 2: Overview of the classes presented in this taxonomy.
3.2 Metric-based Methods
Metric-based methods aim to structure the embedding space by using a metric that captures task-relevant similarities between state representations. By mapping functionally equivalent states to similar points in the latent space, these methods can enhance sample efficiency and improve policy learning. For instance, two different visual observations in a game lead to the same downstream behavior and similar rewards, they can be mapped to the same latent region.
Definition:
8

---

Published in Transactions on Machine Learning Research (06/2025)
The observation encoder, denoted as Φ0 : - R" with parameters 8, maps observations to embedding space X where distances d((v(o;;, oo(;)) reflect some task-relevant similarities. For example. the distance metric d could correspond to the L2 norm, while the metric could be bisimulation (Ferns al, 2012), which is introduced below. The representation learning objective can then be formalized minimizing the expected squared difference between the latent distance ((0o(o.), (o(0;)) and metric d' (I,, 2yy defined over representations (Chen & Pan 2022). as illustrated in the following equation:
Details:
L(óa) = (d(ooloo)) (00))))))00)))) - d' (I,) (2)))25)
(1)
Different functional state
Similar functional
dg
Representation Space
Task: Scoring Basketball points
Figure 5: Metric-based methods shape the representation space to capture task-relevant information. Representa- tions with similar outcomes (e.g,, good basketball trajectory) have minimal distance di, while those with different outcomes (e.g., wrong basketball trajectory) have larger distance d2, hence d1 < d2.
Benefits: Metric-based mots can offer strong theoretical guarantees by bounding the differences value function outputs for pairs embedded states, ensuring that states close in the metric space exhibit similar optimal behaviors. This formalized as [V*(x;) - V* (x;;)] S d(z;, which key to improving sample efficiency and generalization, as it allows the agent to treat behaviorally equivalent states similarly. Addi- tionally, these methods leverage task-relevant. MDP information, such rewards and transition dynamics, to shape the latent state space, making them particularly effective in abstracting away irrelevant visual distractions in more complex environments (Zhang et al. 2021). Additionally, some metric- based methods avoid training extra parameters, providing computational efficiency (Castro et al. 2021).
The operations involved cortain motrics, such as the Wasscrstein distance used in bisimula- tion, are known to be compulationally challenging (Castro 2022).. This can lead to the need for approxima- tions or relaxations, which can weaken the original theoretical guarantees (Chen & Pan 2022). Furthermore, these methods typically require access task-speeific MDP information, which may not always be readily available easy lo obtain in real-world settings. In fact, even if rewards are available, real-world seltings are often characterized by sparse reward structures, which can create latent instability collapses in metric-based methods. Embedding explosion is another issue that can affect these methods Kemertas & Homentado-AAmsteen nature of the policy during training, which causes continuous updates to the embedding space and metrics, therefore favoring latent instabilities and hindering consistent performance compared to some other classes.
Limitations:
even embedding
20021). Finally, these methods are impacted by the non-stationary
Categorization: Various metrics can be defined to quantify the similarity between states, each influencing how state representations are learned and aggregated.
a) Bisimulation Metrics
Bisimulation metrics, originally introduced for MDPs by Ferns et al. (2012). offer a way to quantify behavioral similarity between states. By measuring distances between states based differences in both their rewards

---

Published in Transactions on Machine Learning Research (06/2025)
and transition dynamics, it allows state aggregation while preserving crucial information needed for effective policy learning. Formally, the bisimulation metric d(): between latent states H2 and zy is updated using the following recursive rule:
(e(,) = max [[1 - c) [(x,, a) a)) c (a aEA
(2)
In this formulation, (x(d) represents an operator that updates the distance function d(zi, zz), where e E [0, 1] is a parameter controlling the balance between the importance of reward differences and transition dynamics. In practice, it is common to set c c= 7, which corresponds to the discount factor RI, without using (1 - c). The term Wa(P(-)z,, a). P(-Ja,, P() a)) represents the Wasserstein distance (or Kantorovich distance) between the next-state distributions induced by the transitions from states x; and æi under action
Intuitively, Wa can be seen as quantifying the distance between two probability distributions, which cor- responds in this case to the next-state distributions of (z), More precisely, the Wasserstein distance is known to measure the cost of transporting one probability distribution to another, and is formalized as finding an optimal coupling between two probability distributions that minimises a notion of transport cost associated with the base metric (Villani 2008). By itcratively applying the operator (c(d), the bisimula- tion distance dy) converges a fixed point d", yielding the final metric between states that minimizes the loss. This iterative process that progressively shapes the representation space ensures that states with similar rewards and transition dynamics are mapped closer in the representation space, while dissimilar states are mapped further apart. Convergence details and formalism can be found in Castro et al. (0021).
Methods: Several methods integrate the bisimulation metric for learning more compact and generalizable representations in reinforcement learning. DBC (Zhang et al. 2021 uses the bisimulation metric to map behaviorally similar states closer in latent space, improving robustness to distractions, but is susceptible embedding poplosions/collapseses and relies on the assumption of Gaussian transitions for metric computation. Kemertas & AaimmenntddoAAmssrooon (2021) address these issues by (1) adding norm constraint to prevent embedding explosion and (2) using intrinsic rewards plus latent space regularization through the learning of an Inverse Dynamics Model (IDM) as an auxiliary task to prevent embedding collapse. The second point is particularly relevant in sparse or near-constant reward settings, where carly similar trajectorics can incorrectly lead the encoder to assume bisimilarity. A more recent method tackling the sparse-reward challenge in bisimulation-based approaches was introduced by Chen al. (2024b). [Cattro al. (2021) resolves some computational limitations of traditional bisimulation metrics with a scalable, sample- based approach that removes the need for assumptions like Gaussian deterministic transitions (Zhang et al., 2021
Castro 2020), and explicitly learns state similarity without requiring additional network parameters.
b) Lax Bisimulation Metric
The lax bisimulation metric (Taylor et al. 2008 extends this concept to statc-action equivalence by relaxing the requirement for exact action matching when comparing states, allowing both MDPs to have different action sets, thus providing greater flexibility. For example, Rezaci-Shoshtari et al. (2022) demonstrated the use of this metric for representation learning, which led to improved performance when learning from
pixel observations. Le Lan et al. (2021)'s work also highlights why the lax bisimulation metric can provide continuity advantages over the original bisimulation metric.
c) Related Metrics
Several alternative metrics have been proposed to shape the representation space of RL agents. For instance, temporal distance metric was used in Florensa et al. (001 and Park et al. (2024b), which captures the minimum number of time steps required to transition between states in a goal-conditioned valuo-based setting. In Rudolph et al.] (2024), their action-bisimmlation metric replaces the reward-based similarity term of traditional bisimulation with a control-relevant term obtained by training an IDM model, making the approach reward-free. Agarwal al. (2021a) introduced the Policy Similarity Metric (PSM), which replaces the absolute reward difference in bisimulation with a probability pseudometric between policies and has been shown to improve multi-task generalization.
10

---

Published in Transactions on Machine Learning Research (06/2025)
d) Impact of distance d Representations
The choice how distances between representations are measured often influences the actual nature of the learned representations. For example, the L1 distance, based on absolute differences, promoles sparsity by applying a constant penalty that drives smaller values toward zero, emphasizing distinct features. This can be useful when only few key features matter in distinguishing states. In contrast, the L2 distance, which uses squared differences, promotes smoother representations by spreading the error across all components, reducing large individual components while retaining contributions from smaller ones. This is more effective when information from all features is relevant, even if some contributions are minor. Some methods instead use orientation- based metrics, such as cosine similarity or angular distance, which can be advantageous higl-dimensional spaces where direction more significant than magnitude, or where specific properties, such non-zero self-distances, are desirable (Castro et al. 2021). They can however come with drawbacks, such as embedding norm growth and convergence slowdowns when optimizing cosine similarity, limiting effectiveness without normalization
Draganov et al. 2024 See Table 3 for precise distances formalism.
Distance
Formula
Description
Sum of absolute differences between corre- sponding vector components.
L1 (Manhattan)
Square root the sum aquared differ- ences between corresponding vector compo- nents, emphasixing larger deviations.
(Euclidean)
Di=
x
2 Ii=e 1
Normalized angle between vectors, ranging from (identical direction) (opposite di- rection), capturing rotational differences.
Angular distance
arocos
7
Cosine similarity of vectors, measuring orien- tation aligument, with values ranging from -1 (oppased) (aligned).
Cosine distance
Table 3: Distance measures used to structure the representation space by quantifying similarities between state embeddings. Each metric defines how distances measured, but also influences key latent properties.
3.3 Auxiliary Tasks Methods
This category composed of methods that enhance the primary learning task (RL) by having agents simultancously predict additional cuviromment-velated outputs. This done by splitting the repre- sentation part of an agent into different heads, each with their own set of weights and dedicated task. During training, the errors from these heads propagated back to the shared encoder é, guiding the learning in conjunction with the main objective. The role of these predictions is to help agents enrich their reprosentations with additional auxiliary signals.
Definition:
Observations
x/
Main RL task
π
encoder
Vouxt
Auxiliary
k
task(s)
Vourz
Figure 6: The representation 3e of RI agent used make additional predictions on auxiliary task function(s). These predictions are used to improve the representation itself.
11

---

Published in Transactions on Machine Learning Research (06/2025)
Details: Let Eprimary (0) denote the loss associated with the primary RL objective. Auxiliary tasks de- fined as additional functions Vauz, (2:) that, with their own set of parameters Aurr {0auz aanx: ...y. fnxx, ): process the representation to output a set of real values with task-dependent dimensions. The loss for each auxiliary task represented as Li(Baux,,, and the overall auxiliary task loss is the sum of all task losses. The combined objective shown below, with weights X balancing primary and auxiliary tasks.
Emethod Auxx) Lprimary (0) Σ (o
(3)
Precision: We define auxiliary tasks as something different than what is called auxiliary losses in the RL literature. Auxiliary losses refers to any loss optimized jointly with the main RL objective, which is somcthing done in methods belonging to most classes here. However, we specifically define auxiliary tasks additional predictions made during training, using the representation 24 as input, which indirectly enhance the quality of d4- By definition, those supplementary tasks require additional parameters for each task-head, unlike auxiliary losses.
Benefits: Auxiliary tasks for RL can enhance the learning process by utilizing additional supervised signals from the same experiences. When faced with environments with sparse rewards, auxiliary tasks can still provide some degree of learning signals for shaping the representation, which increases the learning efficiency of an agent. They can also serve as regularizers, enhancing generalization and reducing overfitting during learning. Finally, they can promote better exploration by guiding the agent toward states that provide more informative signals for the auxiliary tasks.
However, a downside of using auxiliary tasks to improve representations is the lack of the- oretical guarantees when it comes to whether it is actually benefiting the learning process of the main RL objective or not (Du ct al. 2020). Defining preciscly what makes a good auxiliary task also in itsolf a hard problem (Lyle el al. 2021 (Ralice el al. 2022). Finally, choosing the auxiliary weight(s) that balance(s) the importance of the auxiliary task(s) compared to the main task requires the right tuning of typer-parameters.
Limitations:
Categorization: In the next sections, explore the inner mechanisms of some class of auxiliary tasks that are commonly employed to learn good state representations in reinforcement learning.
a) Reconstruction-basee Methods
These methods aim improve state representations by learning to reconstruct original ob- servations Or using decoder ô, v'recon (2t2) that takes input the encoded representations 2 d(o.). This reconstruction process be performed using simple autoencoders (AF), where the objective is to minimize the reconstruction error between the original observation Og and its predicted reconstruction â Additionally, it can be achieved using variational autoencoders (VAEs) (Kingma & Welling, 2014), where additional regularization encourages latent variables to follow a predefined distribution (e.g- Gaussian), improving generalization and disentanglement of representations.
Definition:
Observations
4(A)
Main
Φ encoder
X
6,
Reconstruction
noomttraatio
Auxiitary Task
Figure 7: Reconstruction auxiliary lask: The encoder learns compact latent representations by ensuring that the original observation can reconstructed from the representation.
12

---

Published in Transactions on Machine Learning Research (06/2025)
These methods enforce the latent space to capture the essential features needed for accurate reconstruction, promoting the learning of compact, denoised representations. This helps the agent to fo- cus on key task-relevant information, improving generalization across euvironments and enhancing sample efficiency in high-dimensional spaces. Mask-basod latent reconstruction avoids the need to reconstruct full observations by locusing the reconstruction only on latent variables, thereby discarding irrelevant features from observational space.
Purpose:
Failure Cases: A common failure arises when reconstructing observations that contain significant irrelevant noise distractions. In such cases, especially when task-relevant features occupy only small portion of the observation, the model may learn to preserve unnecessary details, leading to poor state representations that degrade learning performance.
methods can also used to achieve disentangled reprosentations, where the latent space X' idcally structured into independent subspaces X,, cach capturing distinct factor of variation v; from the observation space. Methods like 8-VAE (Higgins et al.) 2016) enforce stronger constraints the latent space than regular VAEs in order to promote disentanglement, ensuring
Disentangled Representations:
Reconstruction
that changes in one factor (e.g., object color) do not affect others (e.g., position), thus enhancing the
robustness and adaptability of learned representations in complex environments. More related methods
include (Higgins et al.] 2017) Thhmas et. al. 2018) (Kabra et al. 2021 (Dunion et al. 2023 (Dunion et al. 2024
Dunion Albrecht 2024
b) Dynamics Modeling Methods
Definition: Dynamics modeling methods use latent forward and inverse models auxiliary lasks to im- plicitly improve representations. A latent forward dynamic model (FDM) predicts the next representation 6+41 = f(1,, aei (twa) from the current reprosentation Ia and action a,, while latent inverse dynamic model (IDM) predicts the action àg = g( T441; (uv)) that caused transition.
Forward Dynamics
Inverse Dynamles
V/=
AGA)
Vi+
4cA)
Peenter
Gaxzerr
Viuu
FDM loss
ACAo
Vi=
eacoder
Opt
Worue
s
JDM
Jas
Joss
Figure Dynamics modeling as an auxiliary task: Forward Dynamics Models predict future representation(s) based on the current representation and action, capturing environment dynamics. Inverse Dynamics Models predict action that caused transitions between representations, emphasizing controllable features.
Purpose: FDMs help the agent learn representation that captures environment dynamics, ensuring that the latent space encodes the essential transition information needed to predict future states. IDMs ensure that the representations encode information to recover the action that led to the state change, focusing on controllable aspects of the environment.
13

---

Published in Transactions on Machine Learning Research (06/2025)
A failure case of using FDMs occurs when the transition model lacks a grounding objec- tive, such as reward prediction (Tomar et al. 2021). In such cases, the model can collapse by mapping all observations to the same representation, minimizing the loss trivially, and failing to learn meaningful representations, especially if the critic's signal becomes noisy due to distractions.
Failure Case:
Using k-step predictions, where the model predicts multiple future representations instead of just one at each step, can further enhance the representation by capturing longer-term dependencies and improving performance across time (Schwarzer et al. 2020). For IDMs, predicting initial actions from trajectories 0r to Ot+* can also ensures positive control properties (Lamb et al., 2023 Islam et al. 2023a).
k-step predictions:
Hierarchical Models: Mclnroe et al. recently introduced a hierarchical approach utilizing multiple la- tent. forward models (FDMs) to capture environment dynamics at varying temporal scales. Each level in the hierarchy learns a distinct FDM that predicts the representation T++& k-steps ahead based on previ- ous representations and actions. Additionally, a learned communication module facilitates the sharing of higher-level information with lower-level modules. When compared on suite of popular control tasks, it achieves noticeable performance and efficiency gains over baseline approaches. Importantly, this differs from predicting all k next representations T4+1 to Zz4k.
c) More Auxiliary Tasks
A wide variely of additional predictions can be used to support representation learning in RL. Here, we highlight a few additional examples. (i) Reward Prediction (Yang et al. 2022) (Zhou et al. 2023 involves predicting the immediate reward Tr based on the current state Ht and action at, guiding the agent to encode task-relevant features essential for value estimation. This task is especially useful in non-sparse reward settings, where it serves a discriminator of critical information and benefits from being combined with latent modeling to capture relevant dynamics. (ii) Random General Value Functions (GVFs) (Zhnng ct al.) 2021) predict random features of observations based on random actions, generating varied signals that enhance state representations, even when the main RL task is detached through a stop-gradient. (iii) Termination Prediction (Kartal et al. 2019) anticipates whether a state will lead to the end of an episode, helping the agent recognize conditions for task completion and improving decisions around critical states. (iv) Multi-Horizon Value Prediction (Fedus et al.) (2019) involves predicting value functions over multiple future horizons, allowing the agent to account for both short and long-term consequences, supporting more balanced and informed docision-making. (v) Proto- Value Networks (Farebrother et al. 2023 help agents learn structured representations by predicting future rewards under random conditions.
3.4 Data Augmentation Methods
Data augmentation (DA) methods represent a class of techniques that enhance sample-efficienc and generalization capabilities of RT agents through the manipulation of their observations. By applying geometric and photometric transformations their inputs, such as rotations, translations, and color shifts, these methods can enforce invariance to irrelevant visual changes.
Definition:
Details: These methods normally introduce an observation transformation function T that generates aug- mented observations ő based on the original observations o, where ő = T()). The transformation I' is chosen such that it preserves the essential task-relevant properties of o. A form of explicit and/or implicit regular- ization then used to enforce some degree of Q-invariance and/or r-invariance. Formally, the invariance of a Q-function with respect to a transformation fr is defined as ((, a) = Q(fr(s), a) for all s E S,a A. Similarly, a policy æ is considered invariant to fT if T(a s) = <(a (a fT())) for all S,a
Two type of strategies can be used to enforce invariance: (1) Implicit regularization applies transformations directly to the input data during the training process, using both original and transformed observations to train the network to generalize across these variations (Hu et al. 2024);; (2) Explicit regularization, on the other hand, achieves invariance by modifying the loss functions to ensure that both the policy (actor) and the value estimates (critic) remain unchangod by the transformations fr. This is done by adding terms in the loss functions that penalize discrepancies between outputs, such as Q-values or action distributions, for both original and transformed inputs.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Implicit Augmentation
Explicit Augmentation
Replay Buffer
Buffer
BL Policy
tw
(A
(6))
Rb Policy
RL Loss
Inverience Penaltiee
RL Pulicy
Figure 9: Implicit DA (left) augments observations directly used to train the policy and/or value network, promoting robustness through diversity without explicit constraints. Explicit DA (right) augments observations supplemented by regularization penalties that enforce Q/T invariances.
DA-based methods enhance sample efficiency by diversifying training samples, enabling robust policy learning with fewer interactions and reducing overfitting. They improve generalization by simulating visual variations, reducing sensitivity to distribution shifts and aiding adaplation to new settings. Crucially, while remaining simple and effective, the usage of DA was shown to preserve learning plasticity (Ma et al. 2024), an essential aspect with non-stationary objectives.
Benefits:
These approaches don't directly learn or structure the representation space for incorporating task-specific information, making them fundamentally limited. Additionally, strong augmentations can intro- duce noise that disrupts training (e.g., high variance in Q-value estimates), and augmentations non-adapted to task may affect the learning of critical features.
Limitations:
Common augmentations applied to observations in DRL include the following: (i) Ran- dom Cropping, which modifies the image borders without altering central objects; (ii) Color Jittering, which
Augmentations:
adjusts brightness, contrast, and saturation to mimic varying lighting conditions; (iti) Random Rotation, involving slight image rotations that do not affect task orientation; and (iv) Noise Injection, where stochastic noise is addod to images to simulate sensory disturbances or camera imperfoctions (Ma al. 2022). Some
augmentations, such as random cropping, have shown greater benefits (Laskin et al. 2020 although their effectiveness be highly task-dependent.
Methods: ragu-rogul Q (DrQ) (Kostrikov et al. 2020 enhances data efficiency and robustness by integrating image transformations and averaging the Q larget and function over multiple transformations, thereby reducing the variance in Q-function estimation. Building on DrQ, DrQ-v2 (Yarats et al. (202) intro- duces improvements like switching from SAC to DDPG and incorporating n-step returns, along with more sophisticated image augmentation techniques such as bilinear interpolation further enhance generalization and computational efficiency. RAD [Laskin et al., 2020), ; on the other hand, focuses training with multi- ple views of the same input through simple augmentations, improving efficiency and generalization without altering the underlying algorithn. DrAC (Raileanu et al., 2021] was introduced as explicit regularization method that automatically determines suitable augmentations for any RL task and uses regularization terms for both the policy and value functions, enabling DA for actor-critic methods. SVEA (Hanson et al.) 2021 proposes an augmentation framework for off-policy RL, which improves the stability of Q-value estimation. Addressing previous limitations, SADA (Almuzairee al. (024) enhances stability and generalization by augmenting actor and critic inputs, allowing a broader range of augmentations.
Some methods also rely more unique techniques: Li et al. 2024 propose normalization techniques for improved generalization, acting as latent data augmentations by altering feature maps instead of raw observations. To stabilize policy/ (Q-estimation outputs on augmented observations even further, Yuan et al. (2022a) proposed to identify task-relevant pixels with large Lipschitz constants (by measuring the effect of pixel perturbations on output. decisions), and then to augment only the task-irrelevant pixels, which preserve critical information while benefiting from data diversity. Inspired by Fourier analysis in computer vision, Huang al.] (2022) introduced frequency domain augmentations, which provide a task-agnostic plug-and- play alternative to traditional spatial domain data augmentation methods.
15

---

Published in Transactions on Machine Learning Research (06/2025)
4
Geometric Transformations
Cropping
Horizontal Flipping
Photometrrc Transformations
Color litter
Grayscale
Original Observation
Other Transformations
Masking
Notse Injection
Figure 10:
Common observation angmentations in RL. Geometric transformations alter spatial properties like cropping or flipping, while photometric transformations alter visual features such as lighting and color. Other augmentations also exist, such masking or noise injection.
Precision: Although DA methods are surveyed here, they do not always rely on an encoder for enhancing efficiency and generalization in RL. This class also focuses on observation augmentations only, although
other forms, like transition trajectory augmentations, can improve learning (Ma el. al. 2022 Yu el al.
2021
For more DA in RL, see Hu et al. 2024 and Ma et al. (0022
3.5
Contrastive Learning Methods
Contrastive learning methods aim to learn effective representations for deep RL agents by con- trasting positive pairs (similar data points) against negative pairs (dissimilar data points). This approach utilizes contrastive loss function that encourages the model to increase the similarity of representations de- rived from positive pairs while simultaneously decreasing the similarity of representations from negative ones. These methods can leverage different strategies to define positive pairs, such as using data augmentations or exploiting the temporal structures in the data.
Definition:
Details: The InfoNCE loss (van den Oord et al. 2018b) is a widely used contrastive loss for learning representations, both in vision-based SSL and RL specifically. It can be defined as:
exp(sim(ds(0). (e(o+)))
LNCE = (o,t (o,7)) log (p)(iim((o(0) +E exp(sim(@e((). (b(,; )))
(4)
In the objective above, o represents an observation (anchor), of is a positive sample-typicall typically similar obscrvation o, generated through data augmentations like cropping, rotation, or jittering -and for } are negative samples, which are dissimilar observations selected randomly or based on temporal differences. The encoder de maps observations to representations, while similarity between representations, sim(z, z'), can be assessed using cosine similarity or dot product.
Intuitively, optimizing this loss encourages the model to make the numerator (similarity between o and o+) as large as possible relative to the denominator (which sums the similarities between and each negative sample). This pushes representations of positive pairs closer together and separates representations of neg- ative pairs, ensuring that observations with similar underlying features cluster in the representation space, while dissimilar observations spread apart.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Instancediscrimnnnattv
Temporal
KL Jaas
encadar
t~
Aug
Gutrrative loss
dox
Cenmruativ Joss
C'-T
meeter
encoder
Two contrastive learning frameworks: (1) Iustauce-diseriminatiiv contrastive learning with data aug- mentation (left), and (2) Temporal contrastive learning (right).
Figure 11:
Contrastive methods can be categorized by how they generate positive and negative observation pairs. Instanie-iseriniinativ contrastive learning uses data augmentation to create variations of the same observation as positives, with different batch observations as negatives. Temporal contrastive lcarning leverages the sequential nature of inputs, trcating nearby time steps or as positives capture temporal consistency, while distant observations serve as negatives.
Categorization:
Contrastive methods structure the representation space informatively, either by creating invari- ance to non-task-relevant variations in obscrvations and/or by making the representation space temporally coherent and smooth. This invariance aspect is especially valuable in complex environments, where dif- ferent observations might not alter the fundamental true state St, thus aiding in maintaining consistent decision-making processes.
Benefits:
Scaling contrastive methods to high-dimensional spaces be challenging due to the expo- nential growth of contrastive samples required to learn representations the input space's 's dimension grows [LeCun] 2022). Finding appropriate negative pairs also be challenge: if negatives
Limitations:
too easy, learn- ing plateaus without gaining useful insights, while overly difficult negatives can hinder learning. Contrastive methods therefore require high batch sizes to avoid biased gradient estimates caused by limited negative samples within a batch (Chen et al.] 2022). Finally, because these methods do not leverage reward signals, they may be limited when such are available.
a) Distiine- Contrastive Learning
CURL (Srinivas et al. 2020) is popular approach that make uses of a contrastive loss to ensure that representations augmented versions of the same image are closer together than representations of differ- ent images, which enforces some beneficial invariance properties in the representation space that improve generalization and efficiency visual RI. tasks. To advance this direction further, future methods could integrate ideas similar to (Wang al. 2024ch, where their notion of augmentation consistency ensures that stronger angmentations push augmented sample's representation further from the original than weaker ones, structuring the latent space more informatively.
b) Temporal Contrastive Learning
CPC (van den Oord et al. 2018a) use autoregressive models predict future latent states by distinguishing between true and false future states, encouraging representations that capture essential predictive features.
Building on CPC, CDPC (Zheng et al. 2024a) introduces temporal difference estimator for the InfoN CE loss
used in CPC, improving efficiency and performances in stochastic environments. ATC (Stooke et al.) 2020 aligns temporally close on under augmentations, learning representations independently of policy updates, which has proven effective in complex settings.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Additional related approaches, such ST-DIM (Anand al. 2019 and DRIML (Mazoure et al. 2020 formulate their objectives based on mutual information maximization between global and local representa- tions [jjelm et al. 2018). Some methods combine contrastive learning with auxiliary tasks, such as Allen cct al. 20211 who combines contrastive learning with an Inverse Dynamics Model (IDM) to learn Markov stale abstractions. TACO (Zheng al., 2024b) takes a different approach and learns both state and action latents by maximizing mutual information between representations of current inputs & action sequenees, and the representations of corresponding future inputs.
Linking Contrastive and Metric-based
Both metric- basric and contrastive methods use some notion of similarity between embeddings to learn de but they define it differently. Contrastive methods binary approach, treating pairs of observations either positive or negative, aiming to minimize representation distances for positives and maximize them for negatives. Metric-basse methods, however, quantify similarity more precisely with a continuous distances, often given by a metrie that reflects task-relevant information (e.g. rewards, transition). While contrastive methods are task-agnostic, relying on data transformations or temporal proximity, metric-based approaches incorporate lask-specific information, enriching X with lask similarity when available.
3.6 Non-Contrastive Learning Methods
Non-contrastive methods in RL learn effective representations by minimizing the distance between similar observations, identified through temporal proximity data transformations, as in contrastive approaches. However, unlike contrastive methods, they do not explicitly push apart dissimilar observations, relying only on positive pairs during training.
Definition:
Details: Methods in this class rely heavily on techniques to prevent total dimensional collapse-a failure mode where the representation space collapses a single constant vector. This collapse occurs when em- beddings are only drawn together using positive pairs, leading a trivial solution where all embeddings converge constant vector, ((ox) = c, which minimizes error but retains no meaningful information. To mitigate this, non-contrastive approaches employ two kind of strategies:
(i) Regularization techniques, which modify the loss function to preserve embodding diversity, e.g., enforcing a covariance matrix of a batch of embeddings close to identity (Bardes et al.) (021);; (ii) Architectural
techniques, which introduce learning asymmetries via mechanisms such as latent predictors, momentum encodors, and stop-gradients, regulate learning updates and prevent collapse (Grill al. 2020).
Contrastive
Bepr Iepresentation space
Non-Contrastive
Representation
Roplay Butter
Replay Buffer
Min Distance
+ Max Distance
Min Dietance
Temporal Proxinity
Temporal
Prosimity
avoid withost neptivone
or
thne
time
O₂
Instance Discrimination
Discrimination
Instance
Archltectural Deeigo
Regularizatinn
Figure 12: Distinction between contrastive and non-contrastive approaches. Contrastive methods rely on both positive and negative pairs to structure the representation space, maximizing similarity within positive pairs and minimizing it for negatives. Non-contrastive methods, which avoid the of negative pairs, address the challenge of representation collapse through architectural designs or loss regularization. In both approaches, positive samples are generated either through instance discrimination using data augmentations of the o via temporal proximity.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Avoiding the need for negative pairs greatly simplifies the learning process, making these methods more computationally efficient and stable in high-dimensional spaces. Additionally, by focusing solely on positive pairs, non-contrastive methods are better suited for scaling to complex observation spaces, avoiding the pitialls of hard-to-balance negative sampling that can limit contrastive approaches. These approaches are also intultively aligned with biological representation learning, where positive associations are done without contrasting them with negative examples.
Benefits:
Non-contrastive methods are susceptible to informational collapse (also known as dimensional collapse), where the embedding vectors fail to span the full representation space, resulting in lower- dimensional subspace that limits the information encoded. This issue, affecting both contrastive and non- contrastive methods, leads to redundancy in the representation, as embedding components can become highly correlated rather than decorrelated, reducing the diversity and effectiveness of the learned features. Additionally, these methods often lack task-specific information, such as rewards, which could guide the formation of more meaningful state representations.
Limitations:
Categorization: Methods in this class can be classified based on whether they make use of a latent predictive component in their architecture not. In the self-supervised learning terminology, we refer to the former Joint Embedding Architectures (JEA) and the latter as Joint Embedding Predictive Architectures (JEPA) (Assran et al. 2023). Non-predictive methods aim to make representation ions invariant to transformations without using a predictor between representation backbones. In contrast, predictive methods incorporate a non-constant predictor, making the representations self-predictive by learning a latent dynamics model during training, which is discarded afterward.
a) Non-Predict Methods
BarlowRL (Cagatan & Akgun 2023 can be seen as an example of a non-contrastive mothod used for RL that does not use a predictive component. Based on the Barlow Twins framework (Zbontar el al. 2021 and the Data-Ffficiint Rainbow algorithm (DER) (Hessel et al. 2018), this regularization--aase method trains an encoder to map closely together embeddings of an observation or and its data-augmented version of. Tested the Atari 100k benchmark, it showed better results than the contrastive framework CURL (Srinivas et al. 2020), but didn't outperform non-contrastive predictive method presented in the next section called SPR (Schwarzer et al., 2020
b) Self-Predictive Methods
Self- Self-predictive methods can be further categorized by whether the predictor relies only on the representation 2y or is also conditioned on transformation parameters between observations, such the action a, (Garrido ct al. 2024). Without conditioning, methods like BYOL (Grill et al.] 2020) and SimSiam (Chen & He invvrnaaiaaiiiiiiiaanntt 2021 learn transformation-ing invariant representations. But conditioning on as enables the encoding of action effects on representations by the predictor, capturing the dynamics of the environment. Among approaches, temporal self-predictive methods predict future latent representations by using temporally close observations as positive pairs, encouraging the encoder capture compressed and predictive information about future states. Specifically, the encoder o(o,) is jointly learned with latent transition model P(x,, which be extended to predict multiple steps into the future. Augmenting raw observations during training can also euhance positively the robustness of representations and promote richer feature learning.
Methods: SPR [Schwarzer et al. 2020). inspired by BYOL (Grill et al. 2020), learns a latent transition model to predict representations from augmented inputs several steps into the future, achieving strong efficiency in pixel-based RL and outperforming expert human scores on several Atari games. PBI. (Guo et al. 2020), designed for multi-task generalization, predicts future latent embeddings that recursively predict agent states, creating a bootstrap effect to enhance dynamics learning. SPR and PBL both operate in the latent space, allowing richer and multi-moda learning.
19

---

Published in Transactions on Machine Learning Research (06/2025)
A comprehensive analysis of self-predictive learning in MDPs and POMDPs conducted by Ni et al. (202)), where they also introduced minimalist self-predictive approach validated across various control settings, including standard, distracting, and sparse-reward environments. [Tang et al. (2023) also explored self- celf-predictive learning in reinforcement learning, specifically highlighting its ability to learn meaningful latent representations by avoiding collapse through careful optimization dynamics. Building on their insights, they introduced bidirectional self-predictive learning, using forward and backward predictions to improve representation robustness. More works in self-predictive representation learning include (Zhang al. 2024b;
Fujimoto et al. 2024 2025 Khetarpal et al. 2024 Voelcker al. 2024
Yu et al. 2022
3.7 Attention-basel Methods
Attention-based methods in reinforcement learning involve mechanisms that enable agents to focus on relevant parts and features of their complex observations while ignoring less important information. This selective focus allows an agent to process inputs more efficiently, leading to better learning efficiency, performance, and interpretability of an agent's decision making.
Definition:
Details: Attention mechanisms visual RL agents are typically implemented using mask-basod or patch- based altention. Mask-based attention learns weights to highlight relevant regions in observations, while patch-based attention divides imputs into patches and learns relevance scores to focus on the most significant ones. Different types of attention such as regular or self-attention, soft or hard attention, temporal or spatial attention, single-head or multi-head attention, and top-down or bottom-up attention can be integrated within an RT. agent's 's architecture. For CNN-based encoders, agents extract feature maps h1, h2,... through convolutional layers C1, O2..., starting from the observation Ot, and map these to the representation ze using fully connected layers. Sclf-attention modules can be applicd diffcrent stages, targeting low-level or high- level features, even spanning multiple layers. Alternatively, attention can directly bottleneck inputs (i.e., by learning mask over input patches before any encoding layer to select only the most salient regions).
The computational steps of a self-attention module operating on feature maps (c.g. top of figure 13) are as follows. Starting with a feature map H, the module projects into query, key, and value matrices: Q = WW, K = H, and V = W,H, where Wg, WR, and W. are learned transformations. Attention weights are calculated as A = softmax(QK") /d) where d is the dimensionality of each key vector; A thus reflects the relevance each region in H. The resulting attention-weighted representation Y = AV aggregates information from V according to these relevance scores, focusing on parts of the input that enhance the relevance of It.
Baakk-hazee Attention
Poller
Loa
CNN Banwder
Patch Patch-hased based Attention
Petley
Lave
Nocil
Figure 13: Top: A self-attention module operates high-level feature maps extracted from observations, creating masks that reweight features via element-wiee multiplication. Bottom: An attention bottleneck is applied directly to observations, where an attention module selectively focuses patches of the input.
20

---

Published in Transactions on Machine Learning Research (06/2025)
Benefits Benefits/Limitatii LLissaaiiiittiiii These methods enhance efficiency by focusing task-critical regions both tempo- rally and ially, allowing agents to process relevant observations more evfec and reduce the complexity of the state space. They also improve interpretability, providing insights into the agent's 's focus area with the usage of saliency maps, making the decision- process casier to understand. On the other hand, those mechanisms can increase computational complexity due to the additional parameters required. They may also exhibit poor generalization new settings or with heavy distractions (Tang et al., 2020
Methods: Tang et al. (020) introduced self-attention bottleneck that learns to select the top K image patches for efficient processing of relevant visual information. Wu et al. (2021) proposed attention module that uses a self-supervieed approach to generate attention masks, enhancing CNN-based RL performance on Atari games. Chen et al. (2019) integrated temporal and spatial attention into a hierarchical DRL framework for improved lane changing in autonomous driving, achieving smoother and safer behaviors. Chen et al.) (2024a) propose Focus-Then-Deciie (FTD), a method that combines two auxiliary tasks with the RI. loss to train an attention mochanism. Their mechanism selects task-rclovant objects from outputs of a foundational segmentation model, leveraging prior knowledge to enhance performance in complex visual scenarios.
Bertoin et al. (0222) propose Saliency-Guidee Q-networks (SGQN) for improved generalization, framework that generate saliency maps highlighting the most relevant parts of an image for decision-making. The training procedure is supported by two additional objectives: a self-supervised feedback loop where the agent learns to predict its own saliency maps and a regularization term that ensures the value function depends more on the identified important regions. [Sodhani et al. (2021b) introduced an adtention-basee multi-task learning method that uses a mixture of k encoders, with task context from pre-trained language model determining soft-attention weights for combining encoder outputs, improving task performance. Mott et al. 2019 proposed a soft, top-down attention mechanism that enhances interpretability and performance by generating foskefocused attention maps, enabling better generalization and adaptability to unseen game configurations, surpassing bottom-up approaches.
3.8 Alternative Approaches
This section provides brief overview of two additional classes of methods for state representation learning in DRL. While less widely common than other classos, they still offer unique insights.
a) Spectral-based Methods
Spectral-based methods for stale representation learning employ the eigenvalues and eigenvec- tors of matrices derived from transition dynamics to capture structural and geometric information about the environment. These methods create embeddings that preserve the connectivity and global topology of the state space, enhancing the representation o, of the observation Oz.
Definition:
Details: In these methods, observations on can be represented as nodes in a graph G = (0, W), where W is a matrix that reflects transition probabilities between observations. The Laplacian matrix L D W, where D is matrix capturing how connected each observation is, provides spectral properties used to create embeddings. By using the smallest eigenvectors of L,, each observation is mapped to a vector &; = [ei (ot), ... eo.) that captures both local and global relationships in the environment. Laplacian representations were originally fixed after pretraining but recently enable online adaptation to policies.
Methods: Gomez et al. (0024) introduces a framework that approximate Laplacian eigenvectors and eigen- values effectively, while addressing challenges such as hyperparameter sensitivity and scalability. [aagg et al. (203)) improves traditional Laplacian representations by making sure the Euclidean representation distance between two observations also reflects a measure of reachability between them in the environment, allow- ing better reward shaping in goal-rcaching tasks. Wu et al. (2019) propose a scalable and sample-cfficient approach for computing Laplacian eigenvectors, enabling practical applications in high-dinensional or con- tinuous environments. Ren et al. (2023) improves spectral methods by learning state-action features that do not depend on the data collection policy, allowing a better generalization across policies. Finally, [Shribak] et al. (2024) improves spectral representations by using nonlinear diffusion- based approximations, ensuring sufficient expressiveness for any policy's 's value function.
21

---

Published in Transactions on Machine Learning Research (06/2025)
b) Information Bottleneck Approaches
The cettrnatioo--ootleenee (IB) principle (Tishby et al. 2000 provides a framework for learn- ing compact and task-relevant representations by optimizing the trade-off between compression and relevance. When used for SRL, IB aims to learn state encoder that minimizes the mutual information I(0; X) be- tween observations O and reprosentations X to compress irrelevant information while maximizing I(; where Y corresponds to lask-relevant largets.
Definition:
Details: The IB objective balances compression and relevance of state representations by minimizing X) - YY), where O, and I2 denote observations and representations, respectively. Regular IB requires estimating mutual information terms, which is computationally intractable for high-dimensional in- puts. Variational Information Bottleneck (VTB) (Alemi et al. 2017) addresses this by introducing parametric approximations with an encoder qs (X O) and decoder pi (Y (Y The VIB objective therefore combines task relevance and compactness, making it scalable for DRL.
Methods: REPDIB (Tslam et al. 2023b) leverages the IB principle by incorporating discrete latent represen- lations to enforce a structured and compact representation space. It maximizes the relevance of task-specific information while filtering out exogenous noise, leading to improved exploration and performances con- tinuous control tasks. DRIBO (Fan
2022) employs a multi-view framework to filter out irrelevant information via a contrastive Multi- View IB (MIB) objective, enhancing robustnoss to visual distractions. IBAC (Igl al.] 2019) integrates IB into an actor-critic framework, promoling compressed representalin and better feature extraction in low-data regimes. Additional methods include IBORM (Jin et al.) 2021). which leverage IB in multi-agent setting, and MIB (You & Liu 2024), which introduced multimodal IB approach for learning joint representations from egocentric images and proprioception data.
3.9 Comparison and Practical Guidelines
The table below provides brief glimpse of practical guidelines for selecting an SRL method that belongs to the main classes of methods presented before. Metric-based methods are most effective when dense rewards and known dynamics are available, offering provable abstraction guarantees but suffering under sparse or non-stationary rewards. Contrastive methods deliver strong invariances when large batches of positive and negative pairs can be formed, but depend on careful negative mining and high batch sizes. Auxiliary-task methods supply additional learning signals in sparse-reward or exploration-intensive settings, at the cost of extra hyperparameter tuning. Data augmentation methods improve robustness in visual domains with distractors but do not directly shape the latent space. Non- con-contrastive methods avoid the need for negative samples yet require architectural or boss-based measures to prevent collapse. Attention-based methods focus computation on task-relevant regions and enhance interpretability, though they introduce additional parameters and may overfit.
Class
Use Case
Pros
Cons
Metric-based
Known dynamics
Provable generalization
Sparse-reward failure
Auxiliary Tasks
Sparse rewards
Extra learning signal
Hyperparameter tuning
Data Augmentation
Visual distractors
Simple & robust
No latent shaping
Contrastive
Ample negatives
Strong invariance
Negative mining
Non-Contrastive
Low-batch regimes
Low memory footprint
Collapse risk
Attention- based
Selective focus
Interpretability
Parameter overhead
Table 4: Comparison of SRL Classes with Use Cases, Benefits and Trade-Offs.
22

---

Published in Transactions on Machine Learning Research (06/2025)
4 Benchmarking & Evaluation
Evaluating correctly state representation learning methods requires appropriate benchmarks and tools to assess the quality of the learned representations. Key properties like reward density, task horizon, or the presence of distractions constitute important aspects to consider when choosing benchmarks. Ideally, com- parisons between methods should primarily focus on enviromment-specffic properties, as suggested by Tomar et al. )2021). Therefore, instead of claiming universal superiority on a benchmark, it better to state that experiments show an approach is better suited for learning under visual distractions (e.g., noisy backgrounds).
The next sections review common SRL evaluation aspects and methods for assessing represcntation quality.
4.1 Common Evaluation Aspects
Evaluating state representation learning methods requires assessing their effectiveness in achieving key ob- jectives during and after RL training. Comparisons typically focus on the following aspects:
Performance: In DRL, better performance is defined by achieving higher expected cumulative reward. An improved representation o should enable policy Te that maximizes reward, such that ( J (Thaseline). This applies whether the representations are pre-trained via SRL and kept fixed during RL, or when perfor- mance is evaluated as the representations are learned.
Sample Efficiency: Data efficiency can be quantified by the number of samples N required to achieve a specific performance level. Let [e) be the number of samples needed to achieve a performance within e of the optimal performance J An improved representation & enhances sample efficiency if N₂(c) < NNaseeliee (c), meaning fewer samples are needed with the improved representation to achieve the same performance.
The generalization ability of representation quantifies its capacity to support policy per- formance in previously unseen environments. A representation generalizes well if the policy T achieves con- sistent expected return across different environments, measured by the condition [J(TT; train) - ( test)) ] < 6, where 5 is a small tolerance threshold. Additionally, fine-tuning with minimal environment interactions to achieve prior high performance further indicates the effectiveness of a representation in this context. Generalization may also be assessed by metrics such as transfer success rate or zero-shot adaptation score.
Generalization:
The robustness of state representation can be assessed by its stability across variations in underlying RL algorithms, hyperparameters, and training conditions within the SRL method. Formally, given a set of configurations C' (e.g., different RL algorithms, hyperparameter settings, or noise levels) and a tolerance 0, the representation is considered robust if maxegc [ c*)l S 6, where e* represents an optimal configuration. A robust reprosentation exhibits minimal performance variance across C, indicating reduced dependency on specific settings and greater applicability across various RL scenarios.
Robustness:
4.2 Assessing the Quality of Representations
Evaluating the quality of learned state representations in reinforcement learning (RL) is essential for un- derstanding how well representations capture task-relevant information. Therefore, having good metrics for quantifying the quality of those learned state representations is crucial. Various methods can be utilized for this purpose. We categorized those based on whether they require access to ground truth states 8t or not, which is not always a realistic assumption to make.
a) Evaluation Without True States
The most common approach for evaluating the quality of learned state representations is simply to let an RL agent use the learned states to perform the desired task and assess the final return obtained with specific representations methods. This verifies that the necessary information to solve the task is embedded in the representation It. However, this process is often time-intensive and computationally demanding, necessitating substantial data and multiple random soeds to account for the high variance in performance (Agarwal et al. 2021b). The performance can also vary depending on the base agent, adding further complexity to this evaluation approach.
Total Return:
23

---

Published in Transactions on Machine Learning Research (06/2025)
Visual Inspection: Another technique that does not require access to ground truth states involves ex- tracting the observations of the nearest neighbors in the learned state space and visually examining if those observations match approximately to the close neighbors in the ground truth state space (Sermanet et al. 2018). In other words, close points in the latent state space should correspond to observations with similar lask-relevant information.
More general metrics for assessing the quality of representations can be based on measuring properties of good SSL representations, such as the variance of individual dimensions across batch, the covariance between representation dimensions, the average entropy of representation vectors, spectral properties of a representation matrix, such as its effective rank or condition number (Garrido et al. 2023). Disentanglement could be measured by perturbing randomly small parts of an input observation and measuring the impact on the dimensions of representation It. as disentangled representations are expected to limit the effect of random small perturbations to only a few dimensions, reflecting independent and meaningful feature encoding.
Latent Information:
Metrics can also be designed to evaluate the contimuity and smoothness of Q-functions, action predictions, simply temporal coherence in the representation space (Le Lan et al. 2021 By examining multiple local areas in the representation space, along with the nearest neighbors and their corresponding Q-values, action distributions, or time-steps, such metrics can assess whether nearby points yield similar values or actions. Ensuring this continuity helps maintain stable decision-making and simplifies the function approximation of the policy dx (az) and/or value network (r,), enhancing efficiency.
Latent Continuity:
Linear Probing: Zhang et al. (0224)) proposed to use probing tasks for assessing the quality of learned reprosentations: (i) reward prediction in given state, and (ii) action prediction taken by an expert in a given state. The authors used linear probing specifically, where linear layer is trained on top of frozen representations for each prediction task, constraining the probe's performance to rely heavily on the quality of 24. Overall, their probing tasks were shown to strongly correlate with downstream control performance, offering an efficient method for assessing the quality of state representations. Probing on frozen representa- tions Te can also assess latent information by reconstructing observations. However, this technique may be less effective with noisy or distracting inputs.
Interpretability: Understanding the key information that RL agents focus within observations and encoded representations can provide better insights into their decision-making. A general scheme for deter- mining the attention levels at different parts of an observation consist of perturbing random areas of the input, then measuring the resulting policy or value changes (Greydanus et al.) 2018) (Yuan et al. 2022a) Regions causing higher variance under similar perturbation ions are likely more relevant Tor the agent. This perturbation-basee approach can also be extended to individual representation dimensions to evaluate their importance by analyzing induced changes in policy/ /value outputs. With stacked observations, pradient-based techniques [Weitkamp et al. 2019) can offer practical alternative via action-specific activation maps.
b) Evaluation with True States
Evaluating the quality of learned state representations can be done by training linear classifier on top frozen representations to predict ground-truth state variables (Jonschkowski et 2017). and reporting metrics such as the mean Fl score. This linear probing approach was applied by |Anand et al. (2019) to evaluate the performance of a representation learning technique in Atari. The underlying assumption is that successful regression indicates that meaningful fcatures are well-encoded in the learned state, and good generalization performance on the test set suggests a robust encoding of these features. This concept can also be extended to non-linear probes (Tupper & Neshatian 2020).
Probing:
Geometry: The KNN-MSE (K-Nearest Neighbors Mcan Squared Error) metric from Lesort et al. 2017bb can evaluate learned stale representations by first. identifying the k-nearest neighbors of each image / in the learned state space. It then calculates the mean squared error between the ground truth states of the original image and its nearest neighbors I' to assess the preservation of local structure in the learned representations. Manifold learning metries such as NIEQA (Zhang et al. 2012) can also evaluate how well the learned representations preserve the original state's 's local and global geometry (Lesort al., 2017a).
24

---

Published in Transactions on Machine Learning Research (06/2025)
Disentanglement: The metric from Higgins et al. 20166) evaluates how well representation separates factors of variation by fixing one factor in data pains, measuring representation differences, and predicting the fixed factor with linear classifier. Higher accuracy indicates better disentanglement.
5 Looking Beyond
While current state representation learning (SRL) methods for deep reinforcement learning (DRL) have made good progress in improving sample elficiency, generalization ion, and performance, there is still room for improvement. As environments become more complex and varied, is important to explore more ways of enhancing those techniques for more challenging settings. This section looks at several directions, each extending the learning of state representations broader domains.
Direction
Description
Multi-Task
Explore the sharing of representations across multiple tasks to capture common structures.
Offline Pre- Pre-Training
Leverage datasets of past interactions for pre-training state represen- tations, boosting efficiency and transfer.
Pre-trained Vision
Integrate representations from pre-trained visual models into agents for efficiency and generalization gains.
Zero-Shot RL
Produce representations that enable agents to perform new tasks with- additional training.
Leveraging Priors
Utilize large language models (T..Ms/VL.Ms) incorporate prior knowledge into representations.
Multi-Modal
Methods that integrate information from multiple sensory modalities for getting richer representations.
Table 4: Promising directions for enhancing state representation learning in DRL.
5.1 Multi-Task Representation Learning
Definition: Multi-task representation learning (MTRL) involves training an RL agent to extract a shared low-dimensional representation among a set of related tasks and use either one separate heads attached to this common representation to solve each task. This approach leverages the similarities and shared features among tasks to improve overall learning efficiency and performance. Although various settings of MTRL exist, they oflen share the common points presented here.
Benefits: MTRL reduces sample complexity by exploiting shared task structures, enhancing learning con- vergence, generalization, robustness to tasks, and enabling knowledge transfer, where learning one task improves performance related ones (Cheng et al. 2022 Efroni et al. 2022
Challenges: Negative transfer when shared representations are suboptimal for certain tasks represents an important issue, which can lead to interference and degraded performance (Sodhani et al. 2021a). Balancing task contributions to the shared encoder is also challenging when tasks vary in difficulty or nature. Addi- tionally, differences in data distributions among tasks can limit the effectiveness of representations, with benefits often relying on some assumptions (Lu et al. 2022).
Methods: To mitigate negative interference, CARE Sodhani et al. 2021a proposes to encode observations into multiple representations using a mixture of encoders, allowing the agent to dynamically attend to relevant representations based on context. Efroni et. al.] (202)) introduced a framework for efficient. representations ional transfer in reinforcement learning, showcasing sample complexity gains. Kalashnikov et al. (2022) presented MT-Opt, scalable multi-task robotic learning system leveraging shared experiences and representations. PBL (Guo al. 2020) trains representations by predicting latent embeddings of future observations, which are simultaneously trained to predict the original representations, enabling strong performances in multitask
25

---

Published in Transactions on Machine Learning Research (06/2025)
and partially observable RL settings. Hessel ct al. (019 introduced PopArt, a framework that automatically adjusts the contribution of each task to the learning dynamics of multi-task RT agents, hence becoming invariant to different reward densities and magnitude across tasks. [Ishfa et al. (0224) proposed MORL, an offline multitask reprosentation learning algorithm that enhances sample efficiency in downstream tasks. Cheng et al. (2022) introduced REFUEL, a representation learning algorithm for multitask RL under low- rank MDPs, with proven sample complexity benefits.
5.2 Offline Pre- Training of Representations
The offline pre-training of state representations refers to the learning of state representations from static datasets of trajectories ((o,,,,s,,: = 1,...,, N; ......,;: = I,... NV I I.... in order to accelerate learning on downstream RL tasks. This strategy is motivated by the necessity to enhance data efficiency downstream tasks and overcome the limitations of learning tabula rasa, which often leads to some degree of overfitting Akin to human decision-making, this aims to leverage prior knowledge contained in some already collected interactions.
Definition:
T 1.....,,
or demonstrations ((o,,,
Benefits: By leveraging large amounts of pre-collected data, offline pre-training of representations can enhance data efficiency by reducing the need for extensive online interactions to achieve high performance. This pretraining process can lead to better initializations for RL algorithms, resulting in faster convergence and superior final performance on downstream lasks. Moreover, representations learned from diverse oflline datasets can enhance RL agents' robustness, improving generalization across environments and tasks.
The quality and diversity of the offline dataset are crucial, as poorly curated or biased datasets can result in suboptimal representations. Ensuring that the learned representations are transferable and useful for a wide range of downstream tasks is also complex, as certain features may not generalize well beyond the pretraining context. Additionally, the pretraining of large models on extensive datasets demands substantial computational resources, making the process both time-consuming and expensive.
Challenges:
Methods: The study performed by Yang & Nachum (2021) demonstrate that offiine experience datascts can successfully be used to learn state representations of observations such that learning policies from these pre-trained representations improves performance on a downstream task. Through their investigation, they demonstrate performance gains across 3 downstream applications: online RI, imitation learning, and offline RL. Their results provide good insights on different representation learning objectives, and also suggests that the optimal objective depends on the downst.ream lask's nature and is not absolute. Kim et al. (2024) also investigated the efficacy of various pre-training objectives on trajectory and observation datasets, but focused specifically
evaluating the genoralization capabilities of visual RL agents compared to broader range of pre-training approaches. [Farebrother el al. (2023) introduced Proto-Value Networks (PVNs), a method that scales representation learning by using auxiliary predictions based on the successor measure to capture the structure of the environment, producing rich state features that enable competitive performance with fewer interactions. [Schwarzer et al. (2021) introduced SGI, a self-supervised method for representation learning that combines the latent dynamics modeling from SPR (Schwarzer et al. 2020), the unsupervised goal-conditioned RL from HER (Andrychowiez et al., 2017), and inverse dynamics modeling for capturing environment dynamics. It achieves strong performance on the Atari 100k benchmark with reduced data, and good scaling properties.
5.3 Pre-trained Visual Representations
Definition: Pre-trained visual representations (PVRs), also called visual foundation models for control, involve utilizing unlabeled pre-training data from images and/or videos to learn representations that can be used for downstream reinforcement learning tasks. These representations are trained to learn the spatial characteristics of observations {o; | I,. and the temporal dynamics from videos {or, i = 1,...,Ny N; 1. 1,..., T}, and can be seen as initializing an agent with some initial vision capabilities before learning a task. PVRs can be pre-trained on domain similar data or general data with transferable features.
Benefits: PVRs benefit from abundant and inexpensive image and video data compared to action-reward. labeled trajectory data, enabling scalable learning across domains. They improve sample efficiency by pro-
26

---

Published in Transactions on Machine Learning Research (06/2025)
viding pre-learned visual features, reducing task-specific relearning. PVRs also enhance generalization by transferring robust visual features across environments, even under variations or unseen conditions.
Downsides include the lack of temporal data leveraged in Image-based PVRs, while video- based PVRs often struggle with exogenous noise (e.g., background movements) that degrade performance. Without action labels, distinguishing relevant states from noise becomes significantly harder, and sample complexity for video data can grow exponentially (Misra et al.) 2024). Additionally, distribution shifts between pre-training and target tasks further complicate video representation learning (Zhao et al. 2022). Finally, while PVRs benefit model-free RL, Schneider et al. (2024) found they fail to enhance sample efficiency or generalization in model-based RL, especially for out-of-distributio cases.
Challenges:
Methods: Yuan el al. (202bb) demonstrated that frozen ImageNet. ResNet representations combined with DrQ-v2 Yarats et al.) 2021 as a base algorithm can significantly improve generalization in challenging settings, though fine-tuning degraded performance. MVP (Xiao et al. (022) showed that pre-training on diverse image and video data using masked image modeling (He et al.] 2022) while keeping the weights of the visual encoder frozen preserves the quality of representations and accelerates RL training on downstream motor tasks. Majundar et al. (2023) studied PVRs across tasks, finding: (1) no universal PVR method dominate despite overall better performance than learning from scratch; (2) scaling model size and data diversity improves average performance but not consistently across tasks; (3) adapting PVRs to reamstream tasks provides the best results and is more effective than training representations from scratch. The study made by Kim et al.] (2024) of different pre-training objectives suggest that image and video-based PVRs improve generalization across different target tasks, while reward-specific pre-training benefits similar do- mains but performs poorly in different. target environments. Some methods are also exclusively designed for video-based PVRs. Misra et al. (2024) analyzed different approaches for video-based representation learn- ing and found that forward modeling and temporal contrastive learning objectives can efficiently capture latent states under independent and identically distributed (iid) noise, but struggle with exogenous noise, which increases sample complexity. Their empirical results confirm strong performance in noiso-free sottings but a degradation under noise. Other relevant work includes VIP Maa el al.) 2023) and R3M (Nair el al., 2022), though the latter was initially limited to behavior cloning. Finally, approaches that learn latent rep- rosentations while recovering latent-action information solely from video dynamics (Schmidt & Jiang. 2024) represent an interesting avenue for video-based PVRs trained on large action-free dataset.
5.4 Representations for Zero-Shot RL
Zero-shot RL aims to enable agents to perform optimally on any reward function provided at test time without additional training, planning, or fine-tuning. The objective is to train agents that can understand and execute any task description immediately by utilizing a compact environment representation derived from reward-free transitions (St, at, 8t+1). When a reward function is specified, the agent should use the learned representations to generate an effective policy with minimal computation. More precisely, given reward-free free MDP P,,), the goal is to obtain learnable representation E such that, once reward function S x A- + R is provided, we can compute without planning, from E and r, policy a whose performance is close to optimal.
Definition:
Benefits/ CChaittnshhhll Zero-shot RL offers flexibility by enabling agents to adapt to various tasks without retraining, enhancing efficiency and scalability across numerous tasks without additional training or planning. However, challenges include developing comprchensive representations without reward information, ensuring transferability across complex tasks, consistently achieving near-optimal performances, or assuming access to good exploration policies during pre-training.
Methods: Forward-Backwand (FB) representations, introduced by Touati & Ollivier (2021). enable zero- shot RL by learning two functions: a forward function to capture future transitions and backward function to encode paths to states, trained in an unsupervised manner on state transitions without rewards. The intuition for these representations can be seen as aligning the future of a state with the past of another by maximizing F(s)T B(s') for states and s' that are closely connected through the environment's dynamics. This approach offers a simpler alternative to world models, enabling efficient computation of near-optimal
27

---

Published in Transactions on Machine Learning Research (06/2025)
policies for any reward function without additional training or planning, though it relies effective exploration strategy.
In a related study by Touati et al. (2023) FB representations have shown to deliver superior performances across a wider range of settings compared to methods based on successor features (SFs), which also aim to do zero-shot RI. based on successor representations (SRs) (Dayan) 1993). Recently, [een et al. (2024) proposed
version of FB representations, addressing the performance degradation issue of previous methods when trained small, low-diversity datasets. Other similar recent works include Proto
Successor Measure (Agarwal et al. 2024). Hilbert Foundation Policies (Park et al. 2024a), and Function
Encoder [Ingebrand et al. 2024
5.5 Leveraging Language Models' Prior Knowledge
Loveraging Vision-Language Models (VLMs) and Large Language Models (LLMs) for state representation learning involves using large pre-trained models lo transform visual observations into natural language descriptions. These descriptions serve as interpretable and semantically rich state representations, which can then be used to produce task-relevant text embeddings, allowing reinforcement learning (RL) agents to learn policies from embeddings rather than pixels.
Definition:
Benefits/ /Challengssl Leveraging VIMS/LLMS for this purpose can improve generalization by creating invariant representations that are less affected by disturbances in observations. Indeed, these models can successfully extract the presence of objects and filter out irrelevant details based on task-specific knowledge, focusing only what is relevant. This has the advantage of leveraging the vast prior knowledge embedded in those large models, and can also enhance interpretability by allowing for a more transparent understanding of the agent's 's meeision-makinn However, many challenges still exist for a successful integration of VLMs/ LLMs/LLMs LLMs within an RL framework, starting by the higher computational resources necessary to leverage the capabilities of those models.
Methods: An interesting work in this direction was done by Rahman Xue (2024) where they proposed to use VLM to generate image description, refined by LLM to remove irrelevant information, and used for producing state embedding given to the reinforcement learning agent. Their method showed generalization improvements compared to an end-to-end RL baseline. Other related works include (Chen
lt al. 2024c Wang et al. 2024a).
5.6 Multi-Modal Representation Learning
Definition: Multi-Modal State Representation Learning (MMSRI.) integrates multiple data types to create richer and more comprehensive state reprosentations for RL agents. By combining diverse information sources with different properties, MMSRL can enhance an agent's 's understanding of the environment, improving decision- decision-making and generalization. For example, a robot navigating a room with a camera and a microphone will be able to learn unified representations combining sight and sound with MMSRL. Hence if it hears glass shatter but doesn't 't see it, the robot will be able to infer danger in another room.
Benefits/ Challenges: MMSRL creates richer representations by capturing more comprehensive environ- mental fcatures, making it resilient to noise or occlusion in individual modalities. This robustness enhances generalization and adaptation, improving performance in complex environments.
Methods: The work done by Becker et al. 2024 introduces a framework that enables the selection of the most suitable loss for each modality, such as using a reconstruction losses for low-dimensional proprioception data and contrastive one for images with distractions.
5.7 Other Directions
Several directions fall outside the scope of this work but still deserve consideration: (i) Exploration strate- gics for enhanced state representation learning, rather than for rewards dircctly, will be essential for future open-ended applications, as they can ensure a relevant state space coverage and mitigate the risk of learning good representations for only a small part of the state space. In fact, the interplay between effective explo-
28

---

Published in Transactions on Machine Learning Research (06/2025)
ration and high-quality state representations is particularly important since effective exploration relies solid understanding of previously encountered states. (ii) State representation learning in continual learning settings, where representations are learned from continually evolving environments, aligns more closely with the dynamic nature of real-world problems and should be investigated further. (iii) Evaluating the scala- bility of SRL approaches remains an open challenge, with future methods needing to scale with increasing environment complexity, number tasks, and aspects like computational resources, data, and parameters.
Moreover, action spaces, like observation spaces, become high-dimensional in complex environments. Learning compact action representations has been shown to improve generalization over large action sets (Chandak et al. 2019) by allowing agents to infer the effects of novel actions from those of similar actions encountered earlier. An important future direction is to jointly learn state and action representations, producing dual informative latent spaces that capture both environmental dynamics and action semantics to further enhance policy efficiency and transfer.
6 Conclusion
This survey provides a comprehensive overview of techniques used for representation learning in deep re- inforcement learning, focusing on strategies to enhance learning efficiency, performance, and generalization across high-dimeusional observation spaces. By categorizing methods into distinct classes, we have high- lighted each approach's mechanisms, strengths, and limitations, clarifying the landscape of SRL methods and serving as practical guide for selecting suilable techniques. We also explored evaluation stralegies for assessing the quality of learned representations, especially as techniques are applied to increasingly challenging settings. Robust evaluation remains essential for real-world applications, supporting reliable Decision-making and generalization.
Looking forward, SRL methods must adapt to a broader set of settings, such as those outlined in Section 5. For each direction in that section, we reviewed related work that could serve as a foundation for further exploration, emphasizing the importance of continued research in these areas. Ultimately, advancing SRL will be crucial for developing robust, generalizable, and efficient DRL systems capable of tackling complex real-world tasks. We hope this survey serves as a resource for researchers and practitioners aiming to deepen their understanding of SRL techniques and offers a strong foundation for learning representations in DRL.
This survey primarily examines state representation learning methods within the model-free online RL sctting, without addressing model-based approaches and offline RL evaluation. The comparisons between classes are also largely theoretical rely on previous studies. Future work could include exper- imental evaluations to compare approaches multiple aspects. Lastly, while the taxonomy provides overview of the main classes for learning state representations in RL, it does not explore each class in detail, as each could be the focus of its own survey. Some isolated approaches may also be missing due to our focus on categorizing mostly the recent developments.
Limitations:
29

---

Published in Transactions on Machine Learning Research (06/2025)
References
David Abel. A theory of abstraction in reinforcement learning. ar preprint or 2022.
Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Marc G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In International Conference Learning Representations, 2021a.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, volume 34, pp. 29314-29327, 2021b.
Siddhant Agarwal, Harshit Sikchi, Peter Stone, and Amy Zhang. Proto successor mcasure: Representing the space of all possible solutions of reinforcement learning, 2024. URL https://aaxxi wwtp:oos///....1.11 19418.
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In Proceedings of the 5th International Conference Learning Representations (ICLR), 2017.
Cameron Allen, Neev Parikh, Omer Gottesman, and George Konidaris. Learning markov state abstractions for deep reinforcement learning. Advonces in Neural Information Processing Systems, 34:8229 -8241, 2021.
Abdulaziz Almuzairee, Nicklas Hansen, and Henrik T Christensen. A recipe for unbounded data augmentation in visual reinforcement learning. Reinforcement Learning Journal, 1:130 157, 2024.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R. Devon Hjelm. Unsupervised state representation learning in atari. CoRR, 1ttp//106000222 2019. URL http://arxiv. org/ 2tg/abb//190.08255
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAT Pieter Abbeel, and Wojeiech Zaremba. Hindsight. experience replay. Advonces in neural information processing systems, 30, 2017.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint- ceint-embedding predictive archi- tecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619 15629, 2023.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning lo align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URLattp://arddv.or//aa//// 0473.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-inariinee-cvvaaaaa regularization for self-supervised learning. arXiv preprint arXim:2105.0 04906, 2021.
Philipp Becker, Sebastian Mossburger, Fabian Otto, and Gerhard Neumann. Combining reconstruction and contrastive methods for multimodal representations in rl, 2024. URL https://axx..or//abbb/// a200a////..0///////22..
05342
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253 279, 2013.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Unsupervised feature learning and deep learning: 5538
A review and new perspectives. CoRR, abs/ abs/1206.5538, 2012. URL
David Bertoin, Adil Zouitine, Mchdi Zouitine, and Emmanuel Rachelson. Look where you look! saliency- guided q-networks for generalization in visual reinforcement learning. Advances in Neural Information Processing Systems, 35:30693 0330066333766,, 2022.
30

---

Published in Transactions on Machine Learning Research (06/2025)
Wendelin Böhmer, Jost Tobias Springenberg, Joschka Boedecker, Martin A. Riedmiller, and Klaus Ober- mayer. Autonomous learning of state representations for control: An emerging field aims to autonomously learn state representations for reinforcement learning agents from their real-world sensor observations. URL https://api scl.tccccoa wtt.cccchooaa..oroorropssslllili
KI - Künstliche Intelligenz, 29:353-366 362, 2015.
15176564
Nicolò Botteghi, Mannes Poel, and Christoph Brune. Unsupervised representation learning in deep rein- forcement learning: A review. arXiv preprint arXic:2208. 120088...222 1/226, 2022.
Omer Veysel Cagatan and Baris Akgun. Barlowrl: Barlow twins for data-efficient reinforcement learning,
2023.
Pablo Samuel Castro. Scalable mothods for computing state similarity in deterministic markov dccision processes. In Proceedings of the AAAT Conference on Arlificial Intelligence, volume 34, pp. 10069-10077 2020.
Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. MICo: Improved represen- tations via sampling-bases state similarity for markov decision processes. In A. Beygelzimer, Y. Dauphin, P. Liang and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL htttps///openteviee.neettotooore utte://rrrun11ds/PpppeeleLLLL
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip Thomas. Learning action representations for reinforcement learning. In International conference machine learning, pp. 941 950.
PMLR, 2019.
Changyou Chen, Jianyi Zhang, Yi Xu, Liqun Chen, Jiali Duan, Yiran Chen, Son Tran, Belinda Zeng, and Trishul Chilimbi. Why do we need large batchsizes contrastive learning? a gradient-bias perspective. Advances in Neural Information Processing Systems, 35:33860-3355 338755, 2022.
Chao Chen, Jiacheng Xu, Weijian Liao, Hao Ding, Zongzhang Zhang, Yang Yu, and Rui Zhao. Focus-then- decide: Segmentation- reinforcement learning. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 38, pp. 11200-11248, 2024a.
Jianda Chen and Sinno Pan. Learning representations via a robust behavioral metric for deep reinforcement learning. Advances in Neural Information Processing Systems, 35:36654-36666 2022.
Jianda Chen, Wen zheng terence Ng, Zichen Chen, Sinno Jialin Pan, and Tianwei Zhang. State chrono representation for enhancing generalization in reinforcement learning. In The Thirly-eighth Annual Con- ference on Newral Information Processing Systems, 2024b. URL https://openreete wwwpopeewewweeeeeeeeeeeeeeeeeeee net/forum?id J42SwBemEA]
William Chen, Oier Mees, Aviral Kumar, and Sorgey Levine. Vision-language models provide promptable representations for reinforcement learning, 2024c. URL https://arriv.orr//bb wttp://wwww...r///////..000. 02651
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Yilun Chen, Chiyu Dong, Praveen Palanisamy, Priyantha W. Mudalige, Katharina Muellin,, and John M. Dolan. Attention-basee hicrarchical deep reinforcement learning for lane change behaviors in autonomous driving. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3697 3703, 2019. URL. https: //ppi. www.sssiaaaaanllaaaaamam htwp:/cccccaaaarrooooooooeeeoo www.rorrg//o.aaoo...ooooo 1B 198119613
Yuan Cheng, Songtao Feng, Jing Yang, Hong Zhang, and Yingbin Liang. Provable benefit of multitask representation learning in reinforcement learning. Advances in Neural Information Processing Systems, 85:31741-31755 2022.
Peter Dayan. Improving generalisation for temporal difference learning: The successor representation. Neural Compulation, 5(4):613 624, 1993.
31

---

Published in Transactions on Machine Learning Research (06/2025)
Tim de Bruin, Jens Kober, Karl Tuyls, and Robert Babuška. Integrating state representation learning into deep reinforcement learning. IEEE Robolics and Aulomation Letters, 3(3):1394-1401, 2018. doi: 1w..1009/LAA.2018.2800000
Andrew Draganov, Sharvaree Vadgama, and Erik Bekkers. The hidden pitfalls of the cosine similarity loss.
arXiv preprint arXi::2406.1440 2024.
Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning, pp. 1665-1674. PMLRR 2019.
Yunshu Du, Wojeiech M. Czarnecki, Siddhant M. Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity, 2020. URL https: /oops://arree //rxiv..
Mhairi Dunion and Stefano V Albrecht. Multi-view disentanglement for reinforcement learning with multiple cameras. Reinforcement Learning Journal, 2:498-515, 2024.
Mhairi Dunion, Trevor McInroe, Kevin Sebastian Luck, Josiah P. Hanna, and Stefano V Albrecht. Temporal
disentanglement of representations for improved generalisation in reinforcement learning. In The Fleventh International Conference on Learning Representations, 2023. URL http::///ppperrreeeeett/e/oootot
sPgP6aISLTD
Mhairi Dunion, Trevor McInroe, Kevin Sebastian Luck, Josiah Hanna, and Stefano Albrecht. Conditional mutual information for disentangled representations in reinforcement learning. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Yonathan Efroni, Sham Kakade, Tengyang Xie Ma, and Lin F Yang. Provable benefits of representational transfer in reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pp. 29998 30010, 2022.
Jiameng Fan and Wenchao Li. Dribo: Robust deep reinforcement learning via multi-view information bottleneck. In Proceedings of the 39th International Conference Machine Learning (ICML). PMLR, 2022.
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Mare G. Bellemare. Proto-value networks: Scaling represcntation learning with auxiliary tasks, 2023. URL https://aarxxx.oorr//b/2/20.. 12567
William Fedus, Carles Gelada, Yoshua Bengio, Mare G Bellemare, and Hugo Larochelle. Hyperbolic dis- counting and learning over multiple horizons. arXiv preprint r=Xt::102.00665, 2019.
Norman Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. CoRR, abs/1207.4114, 2012. URL htRLLhttp///aatt http:///rxiv...a http:://aaxxi..rrg/bb///220000 4114
Carlos Florensa, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, and Martin A. Riedmiller. Self- supervisod learning of image embodding for continuous control. CoRR, http://arxiv.orgg aabs/199..00943
2019. URL
Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Scott Fujimoto, Pierluca D'Oro, Amy Zhang, Yuandong Tian, and Michael Rabbat. Towards general-purpost model-free reinforcement learning, 2025. URL https://arxiv http://aaxxvvvo..aaaa// htggabs//5001.11
Quentin Garrido, Randall Balestricro, Laurent Najman, and Yann Lecun Rankme: Assessing the down- stream performance of pretrained self-supervised representations by their rank. In International conference on machine learning, pp- 10974. PMT.R, 2023.
32

---

Published in Transactions on Machine Learning Research (06/2025)
Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun. Learning and leveraging world models in visual representation learning, or Xiv preprint or orXin:2433. 005004, 2024.
Diego Gomez, Michael Bowling, and Marlos C. Machado. Proper laplacian representation learning. In International Conference on Learning Representations (ICLR), 2024.
Samuel Greydamus, Amurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In International conference machine learning, pp. 1792-1801 PMLR, 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 0922127-212844 222244, 2020.
Zhaohan Daniel Guo, Bernardo Ávila Pires, Bilal Piot, Jean-Bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent-predictive representations iannataio for multitask reinforcement learning. CoRR, abs/ Abs/2004.11466 2020. URL hatpps:/aarxtw.orr/abb//// /
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing doep q-lcarning with convnets and vision trans- formers under data augmentation. Advances in neural information processing systems, 34:3680-3663 3693, 2021.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autocncoders are scalable vision learners. In Proceedings of the COEE/CVI conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado Van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3796 3803, 2019.
Trina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-vac: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Represenlations, 2016. URL https:
//api.semanticscholar ott.aa/wwii..ii.....oooooooooraa
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In International Conference on Machine Learning, pp. 1480 1490. PMLR, 2017.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon,, Karan Grewal, Phil Bachman, Adamn Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arxiv: 1 808..88.000 06670, 2018
Jianshu Hu, Yunpeng Jiang, and Paul Weng. Revisiting data augmentation in deep reinforcement learning, 2024.
Yangru Huang, Peixi Peng, Yifan Zhao, Guangyao Chen, and Yonghong Tian. Spectrum random masking for generalization in mase-based reinforcement learning. Advances in Neural Information Processing Systems, 35:20393 20406, 2022.
Maximilian Igl, Kamil Ciosck, Yingzhen Li, Scbastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottle- neck. In Advances in Neural Information Processing Systems (NeurIPS). 2019.
33

---

Published in Transactions on Machine Learning Research (06/2025)
Tyler Ingebrand, Amy Zhang, and Ufuk Topeu. Zero-shot reinforcement learning via function encoders. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 21007-21019. PMLR, 21-27 Jul 2024. URL html
ftttp:://proceedngg mlr mlr. paress//235//ngebraae
Haque Ishfaq, Thanh Nguyen-Taan Tang, Songtao Feng, Raman Arora, Mengdi Wang, Ming Yin, and Doina Precup. Offline multitask representation learning for reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://opperevieer wwt/ frrmmid=
72tRD2MEjd]
Riashat Islam, Manan Tomar, Alex Lamb, Yonathan Efroni, Hongyu Zang, Aniket Didolkar, Dipendra Misra, Xin Li, Harm Van Seijen, Remi Tachet Des Combes, and John Langford. Principled offline rl in the presence of rich exogenous information. In Proceedings of the 40th International Conference on Machine Learning, ICML '23. JMLR.org, 2023a.
Riashat Islam, Hongyu Zang, Manan Tomar, Aniket Didolkar, et al. Representation learning in deep rl via discrete information bottleneck. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2023b.
Scott Jeen, Tom Bewley, and Jonathan Cullen. Zero-shot reinforcement learning from low quality data. In The Thirly-eighth Annual Conference on Neural Informalion Processing Systems, 2024. URL https: //openreview.oo http:///eeoooom/m////e////.
Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Contextual decision processes with low bellman rank are pac-learnable In International Conference on Machine Learning, pp. 1704-1713. PMLR, 2017.
Yue Jin, Shuangqing Wei, Jian Yuan, and Xudong Zhang. Information-bottlecck-bbe behavior represen- tation learning for multi-agent reinforcement learning. arXiv preprint (cXfx:2109.14188 2021.
Rico Jonschkowski, Roland Hafner, Jonathan Scholz, and Martin A. Ricdmiller. Pves: Position-velocity encoders for unsupervised learning of structured stale representations. CoRR, abs/1705.09805, 2017.
URL httt:://atxxiv barg/bb///a 1705. 1etps/1150000005555
Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher P. Burgess. SIMONe: View-invariant, temporally-absttrated ob- ject representations via unsupervised video decomposition. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL wttp://rw.......oo/ooomoo//aaaaa
Dmitry Kalashnikov, Jake Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Scaling up multi-task robotic reinforcement learning. In Conference on Robot Learning, pp. 557 575. PMLR, 2022.
Bilal Kartal, Pablo Hernandez-Leal, and Matthew E Taylor. Terminal prediction auxiliary task for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 15, pp. 38 44, 2019.
Mete Kemertas and Tristan Aumentado-AAmmsiinn Towards robust bisimulation metric learning. CoRR, abs/2110.14099 2021. URL http::////xxivv..gabbbae/ 14096
Khimya Khetarpal, Zhaohan Daniel Guo, Bernardo Avila Pires, Yunhao Tang, Clare Lyle, Mark Rowland, Nicolas Heess, Diana Borsa, Arthur Guez, and Will Dabney. A unifying framework for action-conditional self-predictive reinforcement learning, 2024. URL
wwt.:/orrg/bbs//4060/ 02035
Donghu Kim, Hojoon Leo, Kyungmin Lee, Dongyoon Hwang, and Jacgul Choo. Investigating pro-training objectives for generalization in vision-based reinforcement learning, 2024. URL (RLLhttp:: //apxii///xxvv.../ 2406. 06037)
34

---

Published in Transactions on Machine Learning Research (06/2025)
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd Interna- tional Conference on Learning Regresenlations (ICI.R), 2014.
Nya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. CoRR, abs/2004.13649, 2020. URL https://arxii: /tt////xxxx..rraa//////////// 13649
Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Pac reinforcement learning with rich observa- tions. Advonces in Neural Information Processing Systems, 29, 2016.
Alex Lamb, Riashat Islam, Yonathan Efroni, Aniket Rajiv Didolkar, Dipendra Misra, Dylan J Foster, Lekan P Molu, Rajan Chari, Akshay Krishnamurthy, and John Langford. Guaranteed discovery of control- endogenous latent states with multi-step inverse models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL hntts:://openrevier http://ewwe..eeeooooooo///ooooao
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. CoRR, abs/2004.14999, 2020. URL
Charline Le Lan, Mare Bellemare, and Pablo Samuel Castro. Metrics and continuity in reinforcement learning- In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8261-82000 2021.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62 (1):1 62, 2022.
Timothée Lesort, Mathieu Seurin, Xinrui Li, Natalia Diaz-Rodrigued and David Filliat. Unsupervised state representation learning with robotic priors: a robustness benchmark. ar preprint (rXiw:170.00556 2017a.
Timothée Lesort, Mathieu Seurin, Xinrui Li, Natalia Díaz Rodríguez, and David Filliat. Unsupervised state representation learning with robotic priors: a robustness benchmark. CoRR, abs/ abs/1779.051885, 2017b. URL http://arxiv.. wwwp:/iii.orraaaaoee 11tps///19...00 05185
Timothée Lesort, Natalia Díaz Rodríguez, Jean-Frangois Goudou, and David Filliat. State representation learning for control: An overview. ColRi, abs/1802.04181, 2018. URL http://arxiv httpaar/ivv.or/a//////22...
04181
Lu Li, Jiafei Lyu, Guozheng Ma, Zilin Wang, Zhenjie Yang, Xiu Li, and Zhiheng Li. Normalization enhances generalization in visual reinforcement learning. In Proceedings of the 23rd International Conference on Au- tonomous Agents and Multiagent Systems, AAMAS '24, pp- 1137-1144 1146, Richland, SC, 2024. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9798400704864.
Rui Lu, Andrew Zhao, Simon S Du, and Gao Huang, Provable general function class representation learning in multitask bandits and mdp. Advances in Neural Information Processing Systems, 35:11507-11513 2022.
Clare Lyle, Mark Rowland, Georg Ostrovski, and Will Dabney. On the effect of auxiliary tasks on represon- tation dynamics. CoRR, abs/2102.13089, 2021. URL
Guozheng Ma, Zhen Wang, Zhecheng Yuan, Xueqian Wang, Bo Yuan, and Dacheng Tao. A comprehensive survey of data augmentation in visual reinforcement learning, 2022. URL https: http://///aariiooo///////// org/abs//210.. 04561
Guozheng Ma, Lu Li, Sen Zhang, Zixuan Liu, Zhen Wang, Yixin Chen, Li Shen, Xueqian Wang, and Dacheng Tao. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. Tn The Twelfth International Conference on Learning Representations, 2024. URL https://openreview..rt forumiiidFDRRssyYro.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang, Vip: Towards universal visual reward and representation via value-implicit pre-training, 2023. URL
35

---

Published in Transactions on Machine Learning Research (06/2025)
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Tingfan Wu, Jay Vakil, et al. Where are we in the search for an artificial vi- sual cortex for embodied intelligence? Advances in Neural Information Processing Systems, 36:655 677, 2023.
Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and Devon Hjelm. Deep reinforcement and infomax learning. Advances in Neural Information Processing Systems, 33:3686-3600 2020.
Trevor McInroe, Lukas Schäfer, and Stefano V Albrecht. Multi-horizon representations with hierarchical forward modols for reinforcement learning. Transactions on Machine Learning Research.
Dipendra Misra, Akanksha Saran, Tengyang Xie, Alex Lamb, and John Langford. Towards principled representation learning from videos for reinforcement. learning, 2024. URL https: ptttps://arxiv.oot /bbs/ 2403. 13765
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning, CoRR, abs/ 1b8/1332.5602, 2013. URL
attp://arxis org///////... hbtg//bb//312..... 5602
Alex Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo J. Rezende. Towards interpretable reinforcement learning using attention augmented agents. CoRR, abs/1996.02500, 2019.
URL http:
//axxiv.org/ass/196.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation, 2022.
Tianwei Ni, Benjamin Eysenbach, Erfan Seyedsalehi, Michel Ma, Clement Gehring, Aditya Mahajan, and Pierre-Luc Bacon. Bridging state and history representations: Understanding self-predictive rl, 2024.
Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with Hilbert representations. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. URL htttps://proceedinngs amatiinssssrr.....m wwt.pp/ww/////pppppe http:/arkk222ttt//
PMLR, 21-27 27 Jul 2024a.
Scohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations, 2024b. URL htttpp:/aaaxivv.or///a/////////e 15567
Banafsheh Rafiee, Jun Jin, Jun Luo, and Adam White. What makes useful auxiliary tasks in reinforcement
learning: investigating the effect of the target policy, 2022. URL hattps///araav..or///b///200/
Md Masudur Rahman and Yexiang Xue. Natural language-based based state representation in deep reinforcement learning. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1310-1319, 2024.
Roberta Raileanu, Maxwell Goldstein, Denis Yarals, Tlya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. In Newral Information Processing Systems, 2021. URL httts:://ppl:: www.sticscoooaaaaaeeoeoomamm httpo////o//////poo.eeeootccoota wrtpaarrrrr/rorrr:::200 221094227
Tongzheng Ren, Tianjun Zhang, Lisa Lee, Joseph E. Gonzalez, Dale Schuurmans, and Bo Dai Spectral decomposition representation for reinforcement learning. In The Eleventh International Conference Learning Representations, 2023. URL pttps://openrevies net/forum?id-FBMLeen.
Sahand Pezaei-Shoshtee Rosie Zhao, Prakash Panangaden, David Meger, and Doina Precup. Contimu- ous MDP homomorphisms and homomorphic policy gradient. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
https://openreview -et.foormmiii-Adl-------00
36

---

Published in Transactions on Machine Learning Research (06/2025)
Max Rudolph, Caleb Chuck, Kevin Black, Misha Lvovsky, Scott Niekum, and Amy Zhang. Learning action- based representations using invariance, 2024. URL https://arxiv.. org/ ha////bs/44400 /403. 16369
Dominik Schmidt and Minqi Jiang. Learning to act without actions. In The Twelfth International Conference on Learning Representations, 2024. URL
net http://www....oooor/eerrrrro////
Moritz Schneider, Robert Krug, Narunas Vaskevicius, Luigi Palmieri, and Joschka Boedecker. The surprising ineflectiveness of pre-trained visual representations for model-based reinforcement learning. In The Thirly- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https::/openrevveeu net/foruumiideLxxy07/oo
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data- efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.0592 2020.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R. Devon Hjelm, Philip Bachman, and Aaron C. Courville. Pretraining representations for data-efficient reinforce- ment learning. CoRR, abs/2106.04799, 2021.
URL https://arxiv wwtp:/ogg/abbaaeeeeeeee 04799
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. rase-ventrastive networks: Self-supervised learning from video. In 2018 IEEE internalional conference on robotics and automation (ICRA), pp. 1134 1141. IEEE, 2018.
Dmitry Shribak, Chen-Xiao Gao, Yitong Li, Chenjun Xiao, and Bo Dai. Diffusion spectral representation for reinforcement learning. In The Thirly-eighth Annual Conference on Newral Information Processing Systems, 2024. URL
rett/forun2id--3EXXGE.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In Proceedings of the 38th International Conference on Machine Leurning, pp. 9767 9779. PMLR, 2021a.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. CoRR, abs/2102.06177, 2021b. URL https://arrai..orrraa///22....0.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. CURL: contrastive unsupervised
repprpprrrrrirrepppeeeettttttooo representations for
reinforcement learning. CoRR, abs/ abs/2004.04116, 2020. URL http::://arrvv...r//bb//0000/000 04136
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling reproseetatinnnnnooon learning from reinforcement learning. CoRR, abs/2009.08319, 2020. CRL htalltttpe//arrrr.o///////000000
Yujin Tang, Duong Nguyen, and David Ha Neuroevolution of self-interpretable agents. CoRR, abs/2003.08165, 2020. URL httpss//arxxvvorr//aa//000. 00866
Yunhao Tang, Zhaohan Daniel Guo, Pierre Harvey Richemond, Bernardo Avila Pires, Yash Chandak, Rémi Munos, Mark Rowland, Mohammad Gheshlaghi Azar, Charline Le Lan, Clare Lyle, et al. Understanding self-predictive learning for reinforcement learning. In International Conference on Machine Learning, pp. 33632 33656. PMLR, 2023.
Jonathan Taylor, Doina Precup, and Prakash Panangaden. Bounding performance loss in approximate mdp homomorphisms. pp. 1649 1656, 01 2008.
Valentin Thomas, Emmanuel Bengio, William Fedus, Jules Pondard, Philippe Beaudoin, Hugo Larochelle, Joelle Pineau, Doina Precup, and Yoshua Bengio. Disentangling the independently controllable factors of variation by interacting with the world. arXiv preprint arXiv:1802.00) 094884 2018.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint wrXiephhysics/00040077, 2000.
37

---

Published in Transactions on Machine Learning Research (06/2025)
Manan Tomar, Utkarsh A. Mishra, Amy Zhang, and Matthew E. Taylor. Learning representations for pixel-
based control: What matters and why? CoRR, abs/2111.07775, 2021. URL httts://arxiv. abs/
2111.07775
Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. CoRR, abs/2103.07945,, 2021. URL
Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist?, 2023. URL
bttps://arxiv.orr//aa
14935
Adam Tupper and Kourosh Neshatian. Evaluating Iearned state representations for atari. In 2020 35th International Conference on Image and Vision Computing New Zealand (IVCNZ), pp. 6, 2020. doi:
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, htss1887.03744, 2018a. URL sttp:///arxv.orr//abb////// wwtp:////..aaa...oob///// 03748
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive 18074
coding. CoRR, abs/18012 1807.4387,, 2018b. URL
Cédric Villani. Optimal transport: Old and new. 2008. URL. http://app..mmatiiccccccoo.....
11133220
Claas Voelcker, Tyler Kastner, Igor Gilitschenski, and Amir-massoud Farahmand. When does self- prediction help? understanding auxiliary tasks in reinforcement learning. Reinforcement Learning Journal, 4:1567-1597, 2024.
Boyuan Wang, Yun Qu, Yuhang Jiang, Jianzhun Shao, Chang Liu, Wenming Yang, and Xiangyang Ji. LLM-empowered state representation for reinforcement learning. In Forty-first International Conference on Machine Learning, 2024a. URL
Han Wang, Erfan Miahi, Martha White, Marlos C Machado, Zaheer Abbas, Raksha Kumaraswamy, Vincent Lūu, and Adam White. Investigating the propertics of neural network representations in reinforcement learning. Artificial Intelligence, pp. 104100, 2024b.
Kaixin Wang, Kuangqi Zhou, Jiashi Feng, Bryan Hooi, and Xinchao Wang. Reachabilit--ware laplacian representation in reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning (ICML). PMLR, 2023.
Zihu Wang, Yu Wang, Zhuotong Chen, Hanbin Hu, and Peng Li. Contrastive learning with consistent representations. Transactions on Machine Learning Research, 2024c. ISSN 2835-8856. URL https:
Laurens Weitkamp, Elise van der Pol, and Zeynep Akata, Visual rationalizations in deep reinforcement learn- ing for atari games. In Artificiol Intelligence: 30th Benelux Conference, BNAIC 2018, 's- 's-Hertogenbosch, The Netherlands, November 8-9, 2018, Revised Selected Papers 30, pp. 151-165. Springer, 2019.
Ronald J. Williams. Simple statistical tradienn-following algorithms for connectionist reinforcement learning. Mach. Learn., 8(3 22: 256, may 1992. ISSN 0885-6125. doi: 10.1007/BF00992266. URL https: codoi.org/bl 10. 1007 /FO00///000996966
Haiping Wu, Khimya Khetarpal, and Doina Precup. Self-supervised attention-aware reinforcement learn- ing. In AAAI Conference on Artificial Intelligence, 2021. URL
org/
235349100
Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in rl: Learning representations with efficient approximations. In International Conference on Learning Representations (ICL.), 2019.
Tete Xiao, Iljja Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. Xiv preprint or xixn:22003006173 2022.
38

---

Published in Transactions on Machine Learning Research (06/2025)
Mengjiao Yang and Ofir Nachum. Representation matters: Offine pretraining for sequential decision making. CoRR, abs/2102.05815, 2021. URL
Rui Yang, Jie Wang, Zijie Geng, Mingxuan Ye, Shuiwang Ji, Bin Li, and Feng Wu. Learning task-relevant representations for generalization via characteristic functions of reward sequence distributions. In Pro- ceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2242-2252, 2022.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. CoRR, abs/2107.096445, 2021. URL hntpps//arxiv.. org/abs/ org/abs/2107.0965
Bang You and Huaping Liu. Multimodal information bottleneck for deep reinforcement learning with multiple sensors. Newral Networks, 2024. Accepted for publication.
Tao Yu, Cuiling Lan, Wenjun Zeng, Mingxiao Feng, Zhizheng Zhang, and Zhibo Chen. Playvirtual: Aug- menting consiscon virtual trajectories for reinforcement learning. Advonces in Neural Information Processing Systems, 34:5276-522 5289, 2021.
Tao Yu, Zhizheng Zhang, Cuiling Lan, Yan Lu,, and Zhibo Chen. Mask-based latent reconstruction for reinforcement learning. Advances in Neural Information Processing Systems, 35:25117 25131, 2022.
Zhecheng Yuan, Guozheng Ma, Yao Mu, Bo Xia, Bo Yuan, Xueqian Wang, Ping Luo, and Huazhe Xu. Don't touch what matters: Task-aware lipschitz data augmentation for visual reinforcement learning, In Lud De Raedt (ed.), Proceedings of the Thirty- First International Joint Conference on Artificial Intelli- gence, IJCAT-22, pp. 3702 3708. International Joint Conferences on Artificial Intelligence Organization, Main Track.
7 2022a. doi: 10. /1j96///jaai2202/ 2.14 URL httph:tttp/oooo
2ttp///002996/1jca... 2022/512 551
Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. Advances in Newral Information Processing Systems, 35:13022 15:13022-13087, 2022b.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pp. 12310-12320. PMLR, 2021.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2021. URL https:// www.:n/eoooommiidaoooeeeee
Peng Zhang, Yuanyuan Ren, and Bo Zhang. A new embedding quality assessment method for manifold learning. Newrocomputing, 97:251-2000 11611126,2222 2012.
Wancong Zhang, Anthony GX-Chen, Vlad Sobal, Yann LeCun, and Nicolas Carion. Light-weight, probing of unsupervised representations for reinforcement learning. Reinforcement Learning Journal, 4:1924- 1949, 2024a.
Wancong Zhang, Anthony GX-Chen, Vlad Sobal, Yann LeCun, and Nicolas Carion. weight probing of unsupervised representations for reinforcement learning, 2024b. URL https://aaxxiv... http:///aaxii..o/aa//////// www.oogg//abb////o/////////
12345
Tony Zhao, Siddharth Karamcheti, Kollar Thomas, and Chelsea Finn. What makes representation learning from videos hard for control. In RSS Workshop on Scaling Robot Learning, 2022.
Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive difference predictive coding, 2024a.
Ruijic Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, and Furong Huang. Taco: Temporal latent action-driven contrastive loss for visual reinforcement learning. Advances in Neural Information processing Systems, 36, 2024b.
39

---

Published in Transactions on Machine Learning Research (06/2025)
Zeyu Zheng, Vivek Veeriah, Risto Vuorio, Richard L. Lewis, and Satinder Singh. Learning state rep- resentations from random deep action-conditional predictions. CoRR, abs/2102.04897, 2021. URL
Qi Zhou, Jie Wang, Qiyuan Liu, Yufei Kuang, Wengang Zhou, and Houqiang Li. Learning robust repre- sentation for reinforcement learning with distractions by reward sequence prediction. In Uncertainty in Arlificial Intelligence, pp. 2551 2562. PMLR, 2023.
40


Reviews:
[{"review_id": "joONbCw7hH", "reviewer": "Reviewer_7dyv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As a survey paper, the work covers different representation learning techniques. To my knowledge, I did not see any parts that were missing.\n\nI do not have any particular weaknesses to point out.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors lay out the different classes of representation learning techniques in the context of RL. As someone who works with the techniques in RL, I found this work enjoyable to read.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fUtxGacA35", "reviewer": "Reviewer_G9du", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper is well written and well structured paper. The authors goes beyond simply listing the recent works, each section contains benefits/limitations takeaways which is exactly what I look for when reading survey papers.\n- I appreciated the many figures illustrating the different techniques and concepts throughout the paper. That helped me validate if my understanding of the text was indeed correct or not.\n- As someone that already has some knowledge in state representation learning, I feel I still got valuable insights and knowledge on that topic. To me, this survey is a good starting point to start digging deeper.\n\n### Weaknesses\n- Minor. The author clearly stated the focus of the survey was on the state representation learning of the observation space only. Though, I think state representation learning for the action space is going to play an big role in the future, e.g. AI agent using tools or function calls. There might be something to be said about this in the Looking Beyond section. Anyway food for thoughts.\n\n- Minor. For survey papers (especially for journal papers with less restrictions on pages count), it is better to spell out the acronyms of the methods mentioned as reference. That way readers can have a better idea (sometimes) of what each method does. Some acronyms are already spelled out but there are many that are not, it could be great to uniformize this.\n\n- On page 20, in the Attention-based Methods/Details subsection, what does it mean \"attention can directly bottleneck inputs\"?\n\n- On page 22, in the Benchmarking & Evaluation section, what is \"distraction-based learning\"? That term does not seem to be mentioned before.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper surveys recent state representation learning techniques (SRL) in the context of deep reinforcement learning (DRL), more specifically in the model-free setting. The paper covers the problem definition, the existing methods, the evaluations, and what's next for this research field. The authors propose a taxonomy for the methods found in the literature. They also make use of several figures to support and explain the different topics discussed in the paper. Overall, this survey paper aims at depicting the recent research in this field as accurately as possible and provides several future directions one can act upon to extend it.\n\nAs a researcher in an adjacent field, it is my impression that I got a great deal of information from this survey paper. I'm certain other in the TMLR community will get something out of reading it as well. I recommend this paper for acceptance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Typos\n- p.10: \"an IDM as an auxiliary\" but IDM is only defined at p.18\n- p.12: (Kingma & Welling, 2022) -> That paper was accepted at ICLR 2014.\n- fig.9: \"D.A\" -> DA\n- p.15: \"camera imperfections Ma et al. (2022)\" -> \"camera imperfections (Ma et al., 2022)\"\n- fig.10: \"such as cropping\" -> \"such as masking\"?\n- p.16: \"SSL\" is never defined.\n- p.20: \"softmax(QK^T|\\sqrt{d})\", should mention what $d$ stands for here.\n- p.21: \"focus area with the usage saliency maps\" -> \"...  usage of saliency maps\"?\n- p.26: \"from SPR Schwarzer et al. (2020)\" -> \"from SPR (Schwarzer et al., 2020)\"\n- p.26: \"Misra et al. (2024)\" -> \"(Misra et al., 2024)\"\n- p.26: \"Zhao et al. (2022)\" -> \"(Zhao et al., 2022)\"\n- p.27: \"Agarwal et al. (2024)\" -> \"(Agarwal et al., 2024)\"\n- p.27: \"Other related works include (Chen et al, 2024c) (Wang et al., 2024a).\" -> \"... (Chen et al, 2024c; Wang et al., 2024a).", "broader_impact_concerns": "I don't think there is any concern on ethical implications of this work, i.e. a survey paper on existing methods.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "e4q2x76Mzr", "reviewer": "Reviewer_sRW8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper covers a wide range of contemporary methods and evaluation techniques for representation learning in model-free RL. The topics are fairly comprehensive, far exceeding my knowledge of the field. I find the categorization particularly helpful for gaining a systematic understanding of the field. \n- The paper is well written and easy to read.\n\n**Weaknesses**\n- The paper does not go into much details for the specific methods in each category. It's more like a glossary than an instruction manual. It would help to have a section providing intuition for when to use each representation learning method / what methods perform well for a specific use case.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey paper provides a comprehensive overview of representation learning methods for model-free reinforcement learning. The paper starts off by formulating the reinforcement learning problem. It then dives into a taxonomy of representation learning methods for RL, broadly categorizing the methods into six categories: metric-based, auxiliary tasks, augmentation, contrastive, non-contrastive, and attention-based. For each category, the paper provides an overview of its definition, benefits and limitations, as well as a summary of prominent methods. In the next section, the paper reviews benchmarks and metrics for assessing the quality of the representations, ranging from task performance to similarity metrics. The following section summarizes the emerging directions of RL representations, such as pretrained visual representations and representations for zero-shot RL. Overall, the survey covers a broad array of contemporary methods in a methodological manner, providing a comprehensive guide for researchers in the field.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The paper can be strengthened by adding a section on when to use each representation learning method / what methods perform well for a specific use case. This will provide a more practical guide for researchers in the field.", "broader_impact_concerns": "The paper surveys relevant representation learning methods for model-free RL. While RL has many real-world implications, this paper's contribution is non-technical in nature and does not require a broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)