State Chrono Representation for Enhancing Generalization in Reinforcement Learning
Authors: Jianda Chen
Wen zheng terence Ng
Zichen Chen
Sinno Jialin Pan
Tianwei Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: bisimulation metric
deep reinforcement learning
representation learning
Primary area: reinforcement_learning

Abstract: In reinforcement learning with image-based inputs, it is crucial to establish a robust and generalizable state representation. Recent advancements in metric learning, such as deep bisimulation metric approaches, have shown promising results in learning structured low-dimensional representation space from pixel observations, where the distance between states is measured based on task-relevant features. However, these approaches face challenges in demanding generalization tasks and scenarios with non-informative rewards. This is because they fail to capture sufficient long-term information in the learned representations. To address these challenges, we propose a novel State Chrono Representation (SCR) approach. SCR augments state metric-based representations by incorporating extensive temporal information into the update step of bisimulation metric learning. It learns state distances within a temporal framework that considers both future dynamics and cumulative rewards over current and long-term future states. Our learning strategy effectively incorporates future behavioral information into the representation space without introducing a significant number of additional parameters for modeling dynamics. Extensive experiments conducted in DeepMind Control and Meta-World environments demonstrate that SCR achieves better performance comparing to other recent metric-based methods in demanding generalization tasks. The codes of SCR are available in https://github.com/jianda-chen/SCR.

State Chrono Representation for Enhancing Generalization in Reinforcement Learning
Jianda Chen'
Wen Zheng Terence Ng
Zichen Chen
Sinno Jialin Pan'
Tianwei Zhang!
Nanyang Technological University
*Continental Automotive Singapore "The Chinese University of Hong Kong
3 University of California, Santa Barbara
antuu {j 1iidda001, ngue0099) @ntu. edu. sg crcchen_chenuuche .ed
sinnopan&cuhk, edu. tianwei. zhang@ntu. edu.sg
Abstract
reinforcement learning with image-based inputs, it crucial to establish robust and generalizable state representation. Recent advancements in metric learning, such as deep bisimulation metric approaches, have shown promising results in learning structured low-dimensional representation space from pixel observations, where the distance between states measured based on task-relevant features. However, these approaches face challenges in demanding generalization tasks and scenarios with non-informative rewards. This is because they fail to capture sufficient long-term information in the learned representations. To address these challenges, we propose a novel State Chrono Representation (SCR) approach. SCR augments state metric-based representations by incorporating extensive temporal information into the update step of bisimulation metric learning. It learns state distances within a temporal framework that considers both future dynamics and cumulative rewards over current and long-term future states. Our learning strategy effectively incorporates future behavioral information into the representation space without introducing significant number of additional parameters for modeling dynamics. Extensive experiments conducted in DeepMind Control and Meta- World environments demonstrate that SCR achieves better performance comparing to other recent metric-based methods demanding generalization tasks. The codes of SCR are available in https//gtthbb.comjjjaaaachhheeht
Introduction
In deep reinforcement learning (Deep RI.), one of the key challenges is to derive an optimal policy from highly dimensional environmental observations, particularly images [5, 12, 33]. The stream of images received by an RL agent contains temporal relationships and significant spatial redun- dancy. This redundancy, along with potentially distracting visual inputs, poses difficulties for the agent in learning optimal policies. Numerous studies have highlighted the importance of building state representations that can effectively distinguish task-relevant information from task-irrelevant surroundings [45, 46, 8]. Such representations have the potential to facilitate the RL process and improve the generalizability of learned policies. As a result. representation learning has emerged as fundamental aspect in the advancement of Deep RI. algorithms, gaining increased attention within the RI. community [21].
The main objective of representation learning in RL is to develop a mapping function that transforms high-dimensional observations into low-dimensional embeddings. This process helps reduce the influence of redundant signals and simplify the policy learning process. Previous research has utilized autoencoder-like reconstruction losses [43, 18], which have shown
Related Works.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

impressive results in various visual RL tasks. However, due to the reconstruction of all visual input including noise, these approaches often lead to overfitting on irrelevant environmental signals. Data augmentation methods [42, 22, 23] have demonstrated promise in tasks involving noisy observations. These methods primarily enhance perception models without directly impacting the policy within the Markov Decision Process (MDP) framework. Other approaches involve learning auxiliary tasks [33], where the objective is to predict additional tasks related to the environment using the learned representation as input. However, these auxiliary tasks are often designed independently of the primary RL objective, which can potentially limit their effectiveness in improving the overall RL performance.
In recent advancements, behavioral metrics, such as the bisimulation metric [11, 10, 6, 4, 5, 47] and the MICo distance [7], have been developed to measure the dissimilarity between two states by considering differences in immediate reward signals and the divergence of next-state distributions. Behavioral metrics-based methods establish approximate metries within the representation space, pre- serving the behavioral similarities among states. This means that state representations are constrained within a structured metric space, wherein each state is positioned or clustered relative to others based on their behavioral distances. Moreover, behavioral metrics have been proven to set an upper bound on state-value discrepancies between corresponding states. By learning behavioral metrics within representations, these methods selectively retain task-relevant features essential for achieving the optimal policy, which involves maximizing the value function and shaping agent behaviors. At the same time, they filter out noise that is unrelated to state values and behavioral metrics, thereby improving robustness in noisy environments.
However, behavioral metrics face challenges when handling demanding generalizable RL tasks and scenarios with sparse rewards [20]. While they can capture long-term behavioral metrics to some extent through the temporal-difference update mechanism, their reliance on one-step transition data limits their learning efficiency, especially in the case of sparse rewards. This limitation may result in representations that encode non-informative signals [20] and can restrict their effectiveness in capturing long-term reward information. Some model-based approaches attempt to mitigate these issues by learning transition models [14, 16, 27]. However, learning a large transition model with teajecttreriice
long trajectories requires large number of parameters and significant computational resources. Alternatively, N N-step reinforcement learning methods can be used to address the problem [48]. Nevertheless, these methods introduce higher variance in value estimation compared to one-step methods, which may lead to unstable training.
Contributions. To overcome the above challenges, we introduce the State Chrono Representation (SCR) learning framework. This metric-based approach enables the learning of long-term behavioral representations and accumulated rewards that span from present to future states. Within the SCR framework, we propose the training of two distinct state encoders. One encoder focuses on crafting a state representation for individual states, while the other specializes in generating a Chronological Embedding that captures the relationship between a state and its future states. In addition to learning the conventional behavioral metric for state representations, we introduce novel behavioral metric specifically designed for temporal state pairs. This new metric is approximated within the chrono- logical embedding space. To efficiently approximate this behavioral metric in lower-dimensional vector space, we propose an alternative distance metric that differs from the typical Lp norm. To incorporate long-term rewards information into these representations, we introduce "measurement" that quantifies the sum of rewards between the current and future states. Instead of directly regressing this measurement, we impose two constraints to restrict its range and value. Note that SCR is a robust representation learning methodology that can be integrated into any existing RL algorithm.
In summary, contributions are threefold: 1) We introduce tion learning with a focus on behavioral metrics involving temporal state pairs. We also provide a practical method for approximating these metrics; 2) We develop novel measurement specifically adapted for temporal state pairs and propose learning algorithms that incorporate this measurement while enforcing two constraints; 3) Through experiments conducted on the Distracting DeepMind Control Suite [34], we demonstrate that our proposed representation exhibits enhanced generalization and efficiency in challenging generalization tasks.
framework, SCR, for representa-
2

---

2 Preliminary
Markov Decision Process: A Markov Decision Process (MDP) is mathematical framework defined by the tuple M (S, P.r, r.),y). Here 8 represents the state space, which encompasses all possible states. A denotes the action space, comprising all possible actions that an agent can take in each state. The state transition probability function, P, defines the probability of transitioning from the current state $t E S to any subsequent state $1+1 E at time t. Given the action ag E A taken, the probability is denoted as P(st+1|s; at). The reward function, S x A R, assigns an immediate reward r(s,, at) to the agent for taking action ar in state St. The discount factor, Y E [O, 1],
determines the present value of future rewards.
policy, π S A, a strategy that specifies the probability x ra)s)) of taking action state 8. The objective in an MDP is to find the optimal policy nt" that maximizes the expected discounted cumulative reward, π arg max; Ensx |2, 7"r((,, ae]].
Behavioral Metric: In DBC [47], the bisimulation metric defines a pseudometric d: S x S - IR to measure the distance between two states. Additionally, a variant of bisimulation metric, known as π r-bisimulation metric, is defined with respect to a specific policy æ.
Theorem 2.1 (r-bisimulation metric). The n-bisimulation metric update operator Fhisim M + M defined as
Frisimd(x,y) = 1r ry (W)) where ML is the space of d, rx = E(gAAT(aXX)) PK EoEA ((ax)) W is the Wasserstein distance, and rx is r(x, a) for short. Fussim has a unique least fixed point diteimm
MICo [7] defines an alternative metric that samples the next states instead of directly measuring the intractable Wasserstein distance.
Theorem 2.2 (MICo distance). The MICo distance update operator FMIC. M - M is defined as FMICod(x.y)) == 1rx - ry + Flxx' P2 y' ,(x,8').
FMIC. has a fixed point d dicco
3 State Chrono Representation
Although the bisimulation metric [47] and MICo [7] have their strengths, they do not adequately capture future information. This limitation can de- grade the effectiveness of state repre- sentations in policy learning. To ad- dress this issue and incorporate future details, we propose a novel approach, State Chrono Representation (SCR). The detailed architecture of SCR is il- lustrated Figure 1.
d(xy)
Pollcy Network
O Network
Figure 1: Overall architecture of SCR.
SCR consists of two key components: a state representation Φ(x) E R. for given state x, and chronological embedding (((,,, R" that captures the relationship between current state x; and its future state X, within the same trajectory. The state representation, d(x), is developed using a behavioral metric that quantifies the reward and dynamic divergence between two states. On the other hand, the chronological embedding, (xx,,,)) fuses these two state representations using deep learning, with a focus on capturing the long-term behavioral correlation between the current state x; and the future state Xj. To compute the distance between any two chronological embeddings, we propose a "chronological" behavioral metric, which is further improved through Bellman operator- like MSE loss [36]. Moreover, o(x) incorporates novel temporal measurement m to evaluate the transition from the current state to a future state, effectively capturing sequential reward data from the optimal behavior. Due to the complexity of the regression target, this temporal measurement operates within the defined lower and upper constraints, providing stable prediction target for both the measurement and state representation during the learning process.
3

---

3.1 Metric Learning for State Representation
The state representation encoder e is trained by approximating behavioral metric, such as the MICo distance. In our model, we utilize MICo-based metric transformation operator. Instead of using sampling- basee prediction MICo, we employ latent dynamics- based modeling to assess the divergence between two subsequent state distributions. This approach draws inspiration from the methodology SimSR [45]. The metric update operator for latent dynamics, denoted by F, defined as follows.
Theorem 3.1. Let d : RR x RR - lR be a metric in the latent state representation space, (s,x;,y') == d(á(xi), o(xx)) be a metric in the state domain. The metric update operator F defined as,
Fdo(x, = 1xx - -Ty,l Tyyl+ 7E o(x+1)) )(x4)))-((()xx))... ((xxx)))))
(I)
d(y 1) p(lddy )oyg))
are Ory
where Ox, and oy are the actions in states x; and yı, respectively, and P is the learned latent dynamics model. F has fixed point dg.
Proof. Refer to Appendix A.3.1 for the detailed proof.
To learn the approximation for dg in the embedding space, one needs to specifyy the form of distance d for latent vectors. Castro et al. [7] showed that a behavioral metric with sample-based next state distribution divergence is a diffuse metric, as the divergence of the next state distribution corresponds the Eukaszyk-Karmowsst Karmowski distance [49] that measures the expected distance between two samples drawn from two distributions, respectively (detailed definition is in Appendix A.4.1).
Definition 3.2 (Diffuse metric [7]). A function d x X + based on a vector space is a diffuse metric if the following axioms hold: 1) d(a. b) for any a, b X) d(a, b) d(b, a)
for any a, b E X: 3) d(a, b) + d(b, c) 2 d(a,c) c) for any a, b, c E X.
MICo provides an approximation of the behavioral metric using an angular distance: dMIco (a, b) = | bllă + 38(a, b), where a, b E R", 0(a, b) represents the angle between vectors a and b, and 8 = 0.1 is hyperparameter. This distance calculation includes non-zero self-distance, which makes compatible with expressing the Lukaszyk-Karmowski function ((a, b) only considers the angle between a and b, which requires computations involving cosine similarity and the arccos function. This can sometimes lead to numerical discrepancies. DBC [47] recommends using the L, norm with zero self-distance, which is only suitable for the Wasserstein distance. On the other hand, SimSR [45] utilizes the cosine distance, derived from cosine similarity, but it does satisfy the triangle inequality and does not have a non-zero self-distance.
Karmowski distance [49]. However, the angle
To address the challenges mentioned above, propose revised distance, d(a, b), in the embedding space. is characterized as a diffuse metric and formulated as follows:
Definition 3.3. We define
R x RR + R as distance function, where d(a, b) =
VIaaß + | b. for any a, a.be R..
Theorem 3.4. d is a diffuse metric.
Proof. Refer Appendix A.3.2 for the detailed proof.
Lemma 3.5 (Non-zero self-distance). The self-distance of d is not strict to zero, i.e., d(a, a) ||a||2 2 0. It becomes zero if and only if every element in vector is zero.
Theorem 3.4 validates that d is diffuse metric satisfying triangle inequality, and Lemma 3.5 shows d has non-zero self-distance capable of approximating dynamic divergence, which is Lukaszyk- Karmowski distance. Moreover, the structure of d closely resembles the L2 norm, with the only difference being that the factor of a b is -1 instead of -2. This construction, which involves only vector inner product and square root computations without divisions or trigonometric functions, helps avoid numerical computational issues and simplify the implementation.

---

To learn the representation function é, common approach is to minimize MSE loss between the two ends of (1). The loss, which incorporates d and is dependent on Φ, can be expressed as follows:
L = " (((( -Irx; - ryyl
(2)
where represents the replay buffer or the sampled rollouts used the RL learning process.
3.1.1 Long-term Temporal Information
m z.5))
The loss (Ø) in Eqn. (2) heavily relies on the temporal- difference update mechanism, utilizing only one-step tran- sition information. However, this approach fails ef- fectively capture and encode long-term information from trajectories. Incorporating temporal information into the representation while maintaining the structure and adher- ing behavioral metrics presents a complex challenge. To address this challenge, we propose two distinct methods: Chronological Embedding (Section 3.2) and Temporal Measurement (Section 3.3). Each technique is designed to capture the temporal essence of rollout denoted as r(x;; xj), which represents sequence starting from state x; and reaching future state Xj-
(2 2ty))
do do My)
(z ye)
dø (Trs, By vo, vy)
( y5)
T (
yy
y8
m myr, My)
Figure An example with two rollouts.
As illustrated in Figure the goal of the chronological embedding is create a novel paired state embedding, denoted as (x; X;), capturing the temporal representation of the rollout T (x;;,)) This is achieved by transforming a pair of state representations 6(x:) and ((x;)
vector:
(x;,, (,):=(((((x))) o(x,)) E IR". The objective of learning v/ is to develop a "chronological" behavioral metric dy that quantifies the distance between rollouts 7(x;; and yyr::yy) On the other hand, temporal measurement aims to compute a specific "distance" m between states x; and Xy, which provides insights into the cumulative rewards accumulated during the rollout (x)x)))
Note that learning the state representation solely based on either d, or m poses challenges. While dy improves the representation expressiveness, capturing both rewards and dynamics may be less efficient in the learning process. On the other hand, learning m is efficient but may not adequately capture the dynamics. Therefore, we propose a unified approach that leverages the strengths of both dy and m to overcome the challenges and improve the overall state representation. Empirical results in Section 4.3 support the effectiveness of our proposal.
3.2 Chronological Embedding
The chronological embedding, denoted by (x,,,, X₁) E R",, is designed to capture the relationship between a given state x; and any of its future states xj over the same trajectory. To capture the comprehensive behavioral knowledge, we introduce a distance function dæ (S x S) x (S x X) - R. incorporate behavioral
which is intended to reflect the behavioral metric and enable the encoder information. Building upon the MICo distance in Theorem 2.2, we specify the metric update operator FChrono for du in the following theorem. Proof detailed Appendix A.3.3.
Theorem 3.6. Let My be the space of dgg The metric update operator Fchrono M defined as follows,
My is
Fehrono dog (X,, Nv. 11x - Tyy, T. PI d. X,,
(3)
where time step i and it < < FChrono has a fixed point di.
Here d3 represents the "chronological" behavioral metric. Our objective is closely approximate di. which measures the distance between two sets of states (x)xxy) and (yr. taking into account the difference in immediate rewards and dynamics divergence. To co-learn the encoder & with di. we represent di as d (x;, y yv.yr) == d(((((x, xj)), ('yr,, yrr)), where d is defined in Definition 3.3. Similar to Eqn. (1), we construct di to compute the distance in the embedding space.
= Ixx Ty.. + 7xx in P Ne d( x)), ((3441133)). (4)
5

---

To ensure computational efficiency, the parameters of the encoders and d are shared. The encoder v extracts outputs from Φ, and the distance measure uppropriately adapted as d (x, X2, Nv. yy,) := d( ((x,)), ()((y))), ((yr))))))))))) 1)). The objective of learning the chronological embedding is to minimize the MSE loss between both sides of Eqn. (4) w.r.t. & and v.
L 6) = E XIX. 12
d(v((((x)) ((xx)) (((())) o(y,))) 1rx: -Tyll
- (x))), () o(y,')
(5)
Minimizing Eqn. (5) is to encourage the embeddings of similar state sequences to become closer in the embedded space, thereby strengthening the classification of similar behaviors.
3.3 Temporal Measurement
To enhance the ability of SCR to gain future insights, we introduce the concept of temporal mea- surement, which quantifies the disparities between a current state x; and future state Xy. This measurement, denoted by mxx, x,). aims to measure the differences state values or the cu- mulative rewards obtained between the current and future states. We construct approximate measurement based on the state representation Φ, denoted by m x,) := fx()(x;), Φ(x,)) )). Here, i R x R IR. is non-parametric asymmetric metric function (details are presented at the end of this section). This approach structures the representation space of Φ(x) around the "distance" m, ensuring that (xx) contains sufficient information for future state planning.
We propose using m to represent the expected discounted accumulative rewards obtained by an optimal policy X² from state x; to state Xj: x(x,,x)) E,-- D >'r,, So x;, Sy- Xj However, obtaining the exact value of m x,) is challenging because the optimal policy x is unknown and is, in fact, the primary goal of the RL task. Instead directly approximating m(x, x,). we learn the approximation ñxx;, X,) in an alternative manner that ensures it falls within feasible range covering the true m(x,,,)) x,).
To construct this range, we introduce two constraints. The first constraint, serving as a lower boundary. states that the expected discounted cumulative reward obtained by any policy æ, whether optimal or not, cannot exceed m:
Ex
Σ
7
Txx So
x,, S-- =x, xj S mmxx,,,):
(6)
This constraint is based on a fact that any sub-optimal policy is inferior to the optimal policy. Based on this constraint, we propose the following objective learn the approximation ma:
T- Llowo = Ey (x(xx))) ReLU Σ 0=
2
Y''x. - m((((;)()))))))))))))))(())))) (xxx))
where ReLU(x)= m). This objective is non-zero when the constraint in Eqn. (6) is not satisfied. Hence, it increases the value of m()(x;), Φ(x,)) until it exceeds the sampled reward sum.
The second constraint serving as the upper boundary is proposed based on Fig. 2. To ensure that the absolute value [m(x,, x,) remains within certain limits, we propose the following inequality:
xx)) I d(x;, + jmlyy.. d(xx.
(8)
where d is the behavioral metric introduced in Section 3.1. The right-hand side of the incquality represents the longer path from x; to xj. This inequality demonstrates that the absolute temporal measurement (m(x,,x)) should not exceed the sum of the behavioral metrics at the initial states (x; and ),), i.e., (x). ) the final states pair (x, and y) i.e. d(x,. yy), and the measurement [m I. This constraint leads to the following objective for training ma:
Lup = R ((elLu(((((x))) ((x))) (x))))-sge (g((((())) o(y + o(y,)) m(d(y)), ((y;))
where sg means "stop gradient". This objective aims to decrease the value of my when the constraint in Eqn. (8) is not satisfied. By simultaneously optimizing both constraints in unified manner,

---

we guarantee that the approximated temporal measurement. m, is confined within specific range, defined by the lower and upper constraints. The overall objective for rn is formulated as follows:
C Liow (6) Lup(
Asymmetric Metric Function for m. The measurement (x, Xy), designed to measure the distance based on the rewards, is intended to be asymmetric with respect to x, and x,. This is because we assume that state x, comes before Xy. resulting in a distinct relationship compared to the progression from xj to x;. Recent research has focused on studying quasimetrics in deep learning [28, 39, 38] and developed various methodologies to compute asymmetric distances. In our method, we choose utilize Interval Quasimetric Embedding (IQE) [38] to implement in (details are in Appendix A.5.1).
3.4 Overall Objective
As shown in Figure 1, the encoders are designed to predict state representations o(x) for individual states and chrono embedding (x,, x,) 1 to capture the relationship between states x; and x,. The measurement n. is then computed based on ((x;) and o(x;) ) to incorporate the accumulated rewards between these states. The components s and in work together to enhance the state representations and capture temporal information, thereby improving their predictive capabilities for future insights. The necessity & and n is demonstrated in the ablation study in Section 4.3. Therefore, comprehensive objective to minimize is formulated in unified manner follows,
[(o, v (Ø) + La Φ) + (o).
(10)
Our method, SCR, can be seamlessly integrated with a wide range of deep RL algorithms, which can effectively utilize the representation ((x) crucial input component. In our implementation, we specifically employ Soft Actor-Critic (SAC) [13] as our foundation RL algorithm. The state representation serves the input state for both the policy network and Q-value network in SAC. Other implementation details can be found in Appendix A.5.
Experiments
4.1
Configurations
Deep.Mind Control Suite. The primary objective of our proposed SCR is to develop a robust and generalizable representation for deep RL when dealing with nigh-dimensional observations. To evaluate its effectiveness, we conduct experiments using the DeepMind Control Suite Control) environment, which involves rendered pixel observations [37] and distraction setting called Distract- ing Control Suite [34]. This environment utilizes the MuJoCo engine, offering pixel observations for continuous control tasks. The
environment provides diverse testing scenarios, including background and camera pose distractions, simulating real-world complexities of camera inputs. (1) Default setting evaluates the effectiveness of each RL approach. Frames are rendered a resolution of 84 x 84 RGB pixels as shown in Figure 3(a). We stack three frames as states for the RL agents. (2) Distraction setting evaluates both the generalization and effectiveness of each RL approach. The distraction [34] consists of 3 components, as shown in Figure 3(b): 1) background video distraction, where the clean and simple background is replaced with a natural video; 2) object color distraction, which involves slight changes in the color of the robots; and 3) camera pose distraction, where the camera's position and angle are randomized during rendering from the simulator. We observe that tasks become significantly more challenging when the camera pose distraction is applied.
Baselines. We compare method against several prominent algorithms in the field, including: 1) SAC [13], baseline deep RL method for continuous control; 2) DrQ [42], a data augmentation
(a)
(b)
Figure 3: Examples of DML C with (a) default setting and (b) distraction setting.
7

---

BiC-Catch
C-SwingUp
C-SwinUUSSpare
Ch-Run
F-Spin
H-Stand
R-Easy
W-Walk
SAC
447.54914
133.
30. 101177
354,4=1544
518.6=29.3
139.8=15.6
321.1
363.7=154,
DrQ
962
840.148.2
760.9
487.6+13.9
962.0-11, 11.3
861.2-20.9
970.2+-2.77
927.3- 14.5
104.2133.3
281.3127.6
730.31136.9
330,3114.44
MICo
215,4+1224
803.2+12.0
0.0+0..0
4,9-1.8
2.0+-0.1
800.8+21.1
186.1+194
29.7+-3.3
217.9.307.5
255.31.27..5
SCR
944.2411.7
849.422.1
768.4115.1
778.4=29.7
964.9-25,5
851.3=45.6
946,82226.0
919.0=20.0
Table 1: Results (mean:Estd) DM_Control with the default setting at 500K environment steps.
BiC-Catch
Up
U:SwingUpSperse
F-Spon
H-Stand
R-Easy
W-Walk
SAC
82.61.0.2
2184149
0.7.1077
177.4.18.4
22.51.27.7
19.6.126.2
149.4-77.5
167.218
DrQ
124.0499.6
230.0+33.33
11.2-6.4
103.0-8449
579.8 579.8-282.88
16.8+11.8
70.5+40..1
33.6++6.3
DBC
48.81.2.4
127.7119.2
0.0.10.0
9.2.11.9
7.7.100.7
5.6.2.55
149.6 42.8
30.9.4.88
MICa
104.2+10
200.2-+6.6
0.040.1
7.4+-3.2
86.9476.1
11.8+10.5
132.3-37.8
27.5+7.77
SimSR
13.4441.5
148.4+1..3
0.0+0.0
28.24223.8
0.4+0.1
6.6+1.2
78.4+17
28.6+33.1
SCR
221.3-45544
565.1+-59.9
185.74-93.0
331,7+11.4
738.3--24.5
400.8+-19.2
666.1 14.5
855.1+31.2
Table 2: Results (mean=tstd) DM_ Control with distraction setting at 500K environment step. Distraction includes background, robot body color, and camera pose.
Median
IQM
Mean
Optimality Gap
SAC
DrQ
DBC
MICo
SimSR
SCR
0.15
0.30
0.1
narmalledd scores 0.1
0.2
0.3
0.4
0.7
0.8
0.9
Figure 4: Aggregate metrics on distract setting.
method using random crop; 3) DBC [47], representation learning with the bisimulation metric; 4) MICo [7], representation learning with MICo distance; and 5) SimSR [45], representation learning with the behavioral metric approximated by the cosine distance.
4.2 Main Results
Default Setting. To verify the sample efficiency of our method, we compare it with others on eight tasks in [M__Cotrol and report the experimental results in Table 1. We trained each method on each task for 500K environment steps. All results are averaged over 10 runs. We observe that SCR achieves comparable results to the augmentation method DrQ and the state-of-the-at behavioral metric approach SimSR. is worth noting that for a DM Control task, the maximum achievable scores 1000, and a policy that collects scores around 900 is considered nearly optimal. These findings highlight the effectiveness our SCR in mastering standard RL control tasks.
Distraction Setting. To further evaluate the generalization ability of our method, we perform comparison experiments on DM WM__Cntrol with Distraction using the same training configuration as in the default setting. We evaluate 10 different seeds, and conduct 100 episodes per run. The scores from each run are summarized Table The std computed across the mean scores of these 10 runs. The results show that method outperforms the others in all eight tasks, notably in the sparse reward task curtpole-swing 1at____p_Spp00 up_sparse, where other methods achieve nearly zero scores. The camera- pose distraction poses a significant challenge for metric-based methods like DBC, MICo, and SimSR, it leads to substantial distortion of the robot's 's shape and position in the image state. DrQ outperforms other behavioral metric methods, possibly due to its random cropping technique, which facilitates better aligument of the robot's 's position. The aggregate metrics [2] are shown in Figure 4, where scores normalized by dividing by 1000. Figure 5 presents the training curves, while additional curves can be found in Appendix B.3. These results demonstrate the superior performance of our method in handling distractions and highlight our strong generalization capabilities.
4.3 Ablation Study
Impact of Different Components. To evaluate the impact of each component in SCR, we perform an ablation study where certain components are selectively removed or substituted. Figure 6 shows the training curves on cheetah-rum and walker-walk in the distraction setting. SCR denotes the full
8

---

200
X21
200
100
100
2
Figure Training curves of SCR and baseline methods the distraction setting of DM Control. Mean scores on 10 runs with std (shadow shape). Training curves of all tasks are shown in Appendix B.3.
cheetah Tan
Cheeratt
EX0
330
5200
130
500
Figure 6: Ablation study on cheetah-run (left) and walker-walk (right) in the distraction setting. Mcan scores 10 runs with std (shadow shape).
Figure 7: Training curves with varying sampling steps. Left: cheetah-run. Right: walker-walk. Mcan scores on 10 runs with std (shadow shape).
model. SCR w/o & removes the chronological embedding vh. SCR w/o m refers to the exclusion of the approximation m. SCR w/ é only removes losses (6, v) and Ca (δ) but keep L SCR w/ cos replaces the distance function d for computing metrics on the representation space with cosine distance, akin to SimSR does. SCR w/ MICo replaces d with MICo's angular distance. SCR w/ L1 replaces d with the L1 distance adopted by DBC. The results demonstrate the superior performance of the full model and the importance of jb and n. The absence of these components can lead to worse performance and unstable training. The results also demonstrate that the proposed d surpasses existing distance functions.
Impact of Sampling Steps. In our previous experiments, we uniformly sample the number of steps between and in the range [1, 100). In this experiment, we investigate the impact of this hyper-parameter on SCR. We conduct experiments on the cheetah-run and walker-walk tasks in the distraction setting with various step sampling ranges: [1, 10), [1, 50]. [1, 100), and [1. 150). We also make comparisons with fixed step counts: 50 and 100. The results presented in Figure 7 show that sampling step counts in the range [1, 100) yields the optimal results and remains stable across tasks. Therefore, we set this hyper-parameter to uniformly sample in the range [1, 100] for all experiments.
4.4 Experiments on BertdWorld
We present experimental investigations conducted Meta-Worke [44], a comprehensive simulated benchmark that includes distinct robotic manipulation tasks. Our focus is six specific tasks: basketball-v2, coffee- butten--., door- open-o, drawer-open-v2,, pick-place-v2, and window-open-vv rendered as 84 84x84 RGB pixels
chosen for their diverse objects in environments. The observations with frame stack of 3, following the DM Control format. Table 3 displays the average success rates all tasks and five seeds, while Figure 8 shows the training curves for subset of these tasks. Additional training curves for all tasks can be found in Appendix B.5. Our proposed SCR consistently achieves optimal performance in all tasks, while other baseline methods, except for DrQ, fail to converge to such high levels of performance. Even though DrQ demonstrates proficiency in achieving optimal performance, shows less sampling efficiency than SCR, highlighting the effectiveness of SCR in the applied setting.
5 Conclusion
Our proposed State Chrono Representation learning framework (SCR) effectively captures information from complex, high-dimensional, and noisy inputs deep RI.. SCR improves upon previous metric- based approaches by incorporating long-term perspective and quantifying state distances across

---

SAC
DrQ
DBC
MICo
SimSR
SCR
Meta- World
0.495
0.475
0.886 1 0.125
0.479
0.453
U.495
0.482
0.258 + 0.365
0.969 0.032
Table 3: Average success rates for six tasks in Meta-Wail World.
tenkerbaling
(ivespane))
2
103
Training curves in Meta- World. Mean success rates 5 runs with std (shadow shape). Training curves of all tasks are shown in Appendix B.5.
Figure 8:
Meta-World.
temporal trajectories. Our extensive experiments demonstrate its effectiveness compared with several metric-based baselines in complex environments with distractions, making a promising avenue for future research on generalization in representation learning.
Limitations. This work does not address truly Partially Observable MDPs (POMDPs), which common in real-world applications. future work, we plan to integrate SCR with POMDP methods to solve real-world problems, to design unified solutions that better accommodate these complexities.
Acknowledgement
We thank the anonymous reviewers for the constructive feedback. This study is supported under RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in kind contribution from the industry partner(s). Sinno J. Pan thanks the support of the Hong Kong Jockey Club Charities Trust to the JC STEM Lab of Integration of Machine Learning and Symbolic Reasoning.
References
[1] The successor representation in human reinforcement learning. Nature human behaviour, 1(9): 680-68 180-692,200 2017.
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
[3] André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
[4] Pablo S. Castro and Doina Precup. Using bisimulation for policy transfer in mdps. In Proceed- ings of the 9th International Conference on Autonomous Agents and Multiagent Systems: Volume 1 Volume 1, AAMAS 10, Richland, SC, 2010. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9780982657119,
[5] Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. Proceedings of the AAAI Conference on Artificial Intelligence, 34(06): 10069-10076 Apr. 2020. doi: 10.1609/aaai.v34444 v34106.6564. URL https://ojs. aaai.
aaai org/
index. php/ AAAI/ httpa/ariceeeeeeeeeeeeeeeee
[6] Pablo Samuel Castro, Prakash Panangaden, and Doina Precup. Equivalence relations in fully and partially observable markov decision processes. In Proceedings of the 2 International Joint Conference on Artificial Intelligence, LICAT'09, San Francisco, CA, USA, 2009. Morgan Kaufmann Publishers Inc
10

---

[7] Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. MICo: Improved representations via sampling- based state similarity for markov decision processes. In A. Beygelz- imer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. htL.httpss:/oppenreee httparrwwwweeeeeeoroneeaeama
[8] Jianda Chen and Sinno Pan. Learning representations via robust behavioral metric for deep reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A, Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 36654-36666, Curran Associates, Inc., 2022. URL https: / //poce/dd/s//soooe/g/eoee neurips cc/ wtce/pppr_ffii_e http://--PPaee--ooofeeeeeeeeeee pdf.
[9] Maxime Chevalier-Boisver Bolun Dai, Mark Towers, Rodrigo Perez- VicentV Lucas Willems, Salem Lahlou, Suman Pal. Pablo Samuel Castro, and Jordan Terry Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems 36, New Orleans, LA, USA, December 2023.
[10] Norm Ferns and Doina Precup. Bisimulation metrics are optimal value functions. UAI' 14, pages 210-219, Arlington, Virginia, USA, 2014. AUAI Press. ISBN 9780974903910.
[11] Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. UAI '04, pages 162-169, Arlington, Virginia, USA, 2004. AUAI Press. ISBN 0974903906.
[12] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G. Bellemare. Deep- MDP:
Learning continuous latent space models for representation learning. In Kamalika Chaud- huri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2170-2179. PMLR, 09-15 Jun 2019. URL https: http///pppeee..cc//e////oo mlr press / /v7 wtvv///eeele/e htt///geaaa19.... html.
[13] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870. PMLR, 10-15 Jul 2018. URL https: httlhttss///ooeeeee http:////occeennsse//ee .ml.. ww..m..rreee.
htta////aaanoaaaaaaaaa html.
[14] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representa- tions, 2019.
[15] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages
2019.
[16] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04 2023.
[17] Philippe ulation makes analogies in goal-conditioned reinforcement learning. In Kamalika Chaud- huri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Pro- ceedings of Machine Learning Research, pages 8407-8426. PMLR, 17-23 Jul 2022. URL .mr.....mmm....mmm..m.......m..m
Amy Zhang, Ashvin Nair, Patrick Yin, and Sergey Levine. Bisim-
http::/:/pooceeiigg... mlr. www..pressvvvie wessss/vv62/ev wttp//11/////ae.ee/a hww.en-neeeeeeeeeeee eererennsssucchhh2eee html.
[18]
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. DARLA: Improving zero-shot transfer in reinforcement learning. Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference Machine Leurning, volume 70 of Proceedings of Machine Learning Research, pages 1480-1490. PMLR, 06-11 Aug 2017. URL hww.::ttt/////cctm http:///e/occeeee../ mlr. http///////iigiii/e/// htm..

---

htt.://rnn.n//oommmdddddeeeeelee [19] Michael Janner, Igor Mordatch, and Sergey Levine. Gamma-models, Generative temporal difference learning for horizon prediction. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1724-1735. Curran Associates, Inc., 2020. URL https::/ppooceeeeeeeeee http//////.......0006660666aa2a2
neurips ec/ .df.
paper_ paper_files files/
[20] Mete Kemertas and Tristan Ty Aumentado-Arsstrong Towards robust bisimulation metric learning. In Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openrevien wttpa/rreieeeeeeeeorreeeeeeeeeee net/forum? id=ySFGIFjgIfN.
[21] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. J. Artif. Int. Res., 76, feb 2023. ISSN 1076-9757. doi: 10.161 /1.1611///......./ 1. 14174. URL https :ttpss://dss htt:://dooooppee org/10..///. 1613/ jair. 14174.
[22] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised rep- resentations for reinforcement learning. In Hal Daumé III and Aarti Singh, editors, Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pages 5639-5650, PMLR, 13-18 Jul 2020. URL https:/ https://proceedings mlr prwss/vesivveee
v1 111991ssin/11 html..
[23] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learning with augmented data. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal- can, and H. Lin, editors. Advances in Neural Information Processing Systems, volume 33, pages 19884-19895. Curran Associates, Inc., 2020. URL. http:tttt///pprcceedinge neurip cc/ paper_ http:r//obo/iiieeeocccccaaaaaaa2
.df.
[24] Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balean, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 741-752. Curran Associates, Inc., 2020. URL https:: 9808855001552200011180000---
pdf.
[25] Kuang-Huei Lee, Ian Fischer, Anthony Liu. Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Infor- mation Processing Systems, volume 33, pages 11890-11901. Curran Associates, Inc., 2020. URL https: https://proceeding neurips.. www.ipp.cccpppprrea.e..... wwlls//ppper//20.... 3f Paper
[26] Qiyuan Liu, Qi Zhou, Rui Yang, and Jie Wang. Robust representation learning by clustering with bisimulation metrics for visual reinforcement learning with distractions. In Proceedings of the Seventh AAAI Conference on Innovative Applications of Artificial Intelligence and Thirleenth Symposium on Educational Advances in Artificial Intelligence, AIAT
Artificial Intelligence and Thirty-Fifth Conference
AAAI Press, 2023. ISBN org/ org/10.1609/
978-1-57735-880.1 doi: 10.1609/aaai v37.. 26063. URL aaai. 43717.260655
[27] Tung D Nguyen, Rui Shu, Tuan Pham, Hung Bui, and Stefano Ermon. Temporal predictive coding for model-based planning in latent space. In International Conference on Machine Learning, pages 8130-8139. PMLR, 2021.
[28] Silviu Pitis, Harris Chan, Kiarash Jamali, and Jimmy Ba. An inductive bias for distances: Neural nets that respect the triangle inequality. In International Conference on Learning Representations, 2020. URL
[29] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and Lue Van Gool. The 2017 davis challenge on video object segmentation. 17X 2017.
12

---

[30] Chris Reinke and Xavier Haameda-Pinedte Successor feature representations. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: net/
forum?id-KTFE 1rDDEI.
1rDDI...
[31] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint TrXiv:1007.0337, 2017.
[32] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self self-predietive representations. In Inter- national Conference on Learning Representations, 2021. URL https: https://opennerview net/ forum? FerrmiidduCOOFPZ/aau..
[33] Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In 6th Annual Conference on Robot Learning, 2022. URL https: // /ppppepeeeeiiieeeeeeeeeeeeeeeeee wwrrii.eee/ttt
[34] Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting con- trol suite challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722 2021.
[35] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9870-9879. PMLR, 18-24 Jul 2021. htthsts:::// http::/://////ee/eppppeeee mlr. w.t.press//ve /atesss/vv////e/ stookelia.....oo html.
[36] Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
[37] Saran funyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. dm_ con_toool:: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi: https:// dbo..rg//0..1016...imp...000 100022. URL httph::/t//// /ww. sciencedirect. cww/scoiccccccoccooo htc//ciien science/ atriclleppi1/2266533000000
[38] Tongzhou Wang and Phillip Isola. Improved representation of asymmetrical distances with interval quasimetric embeddings. In NeurlPS 2022 Workshop Symmetry and Geometry in Neural Representations, 2022. URL /ttps://cpenrrviee, http:/iiiw..eeeeooooo/rdd/ds.ss.
[39] Tongzhou Wang and Phillip Isola. On the learning and learnability of quasimetrics. In International Conference on Learning Representations, 2022. URL https://
[40] Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching Andreas Krause, Emma Brunskill,
reinforcement learning via quasimetric learning. Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceed- ings of the 40th International Conference on Machine Learning, volume 202 of Proceed- ings of Machine Learning Research, pages 36411 1 1-36430. PMLR, 23-29 Jul 2023. URL attps://proceedings mlr ppesss/202/wang2222 httll.
[41] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. 2021.
[42] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. URL https: ftttp:://operrevieee wttp:viwwwi..eeeeeeeeeeeeeeeeeee www/ www.oorrmmmmmiioooo mid=-ii-6sTGGGeeeee
[43] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency model-free reinforcement learning from images. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):1067441063 May 2021. doi: 10. 2a.11///aai..55512.112..... view/17276.
URL. https://ojjs aaai...s...a......aa.aaa..aa...a org/index. www./indxx.pp//AAA...... wttppppII/ataiiaaeecc
13

---

[44] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRI.), 2019. URL https: //txx//avvvvv///////////ea/aaava org/ abs/ 1910.10897
[45] Hongyu Zang, Xin Li, and Mingzhong Wang. Simsr: Simple distance- based state representations for deep reinforcement learning. Proceedings the AAAI Conference Artificial Intelligence, 36(8):8997-9005, Jun. 2022. doi: 10.1609) 10.1609/aaai.1 index httpp///AAttaattle//eeeeeeee////
wt..6/9//aai.vv666.000000 v36.8.2088333 URL attps://oj3 aaai org/
[46] Xianghua Zeng, Hao Peng, Angsheng Li, Chunyang Liu, Lifang He, and Philip S Yu. Hier- archical state abstraction based on structural information principles. In Proceedings of the Thirly-Second International Joint Conference Artificial Intelligence, pages
2023.
[47] Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Leaming invariant representations for reinforcement learning without reconstruction. In Inter- national Conference on Learning Representations, 2021. URL https:: h/tppp/ee////ereeee..... net/ forum?id=-20
Corumm:d==-FCoDXRREE.
[48] Junmin Zhong, Ruofan Wu, and Jennie Si. A long n-step surrogate stage reward for deep rein- forcement learning. In Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 12733-12745. Curran Associates, Inc., 2023. URL https::/ www.ies.eeiioeee cc/ ctpppper_ppppppleepcecee http://00001111//aaaccooo0011110 Caperr--r-reeeeeeeeeeeeeeee pdf.
Conf
[49] Szymon Lukaszyk. A new concept of probability metric and its applications in approximation of scattered data sets. Computational mechanics, 33:299-304, 2004.
14

---

A
Appendix
A.1
Broader Impact
This paper is a research on improving the representation of deep reinforcement learning. We think that there are potential societal consequences that need to be highlighted.
A.2 Additional Related Work
Recent studies have extensively explored representation learning in reinforcement learning (RL). Some previous works [18, 24, 43] employ autoencoders to encode image states into low-dimensional latent embeddings, resulting improved visual perception and accelerated policy learning. Other approaches [42, 22, 23, 41, 35] utilize data augmentation techniques such as random crop or noise injection, combined with contrastive loss, learn more generalizable state representations. Another set of approaches [25, 33, 15] focus on learning representation by predicting auxiliary tasks to extract more information from the environment. Successor representations [19, 30] tackles the state occupancy distribution to learn a more informative state representation, showing generalization capabilities between tasks.
In recent research, metric learning methods have been employed in RL to measure distances between state representations. Some approaches aim approximate bisimulation metries [47, 20] while others focus on learning sample-based distances [45, 7J. Bisimulation metrics have also been utilized in goal-based RL to enhance state representation [17]. A
recent work introduces quasimetrics learning as a novel RL objective for cost MDPs [40] but it is not for general MDPs.
Self-predictive representation (SPR) [32] learn representations regarding future state but it focuses on predicting future states, enforcing the representation of predicted future states to be close to the true future states. SPR's representation focuses dynamics learning, without directly considering the reward or value. contrast, SCR focuses on predicting a metric/distance between current state and future state and the metrics related to rewards values regarding policy learning.
Successor Representations (SRs) [1, 3, 30] are a class of methods designed learn representations that facilitate generalization. SRs achieve this by focusing on the occupancy of states, enabling generalization across various tasks and reward functions. In contrast, SCR measures the distance between states specifically to handle observable distractions.
A.3 Proofs
A.3.1 Proof of Theorem 3.1
Theorem 3. Let d: R
x TR - R be metric in the latent state representation space, dig(xi, d(p(xi), ((yx)) )) be a metric in the state domain. The metric update operator F is defined as,
(x).)) = |xx, Tyye
BB ((x4))>-((06x))))) d(0(x;+1); o(yv41))
(11)
((
where Ml is the space of d, with ox, and dy. being the actions at states x, and ye, respectively, and P is the learned latent dynamics model. F has fixed point d=
with Proof. We follow the proof techniques from [7] and [45]. By substituting d(()(x.1), ((y+4))))))))))))
do (X v1+1), we have
Fds(x. = |xx, - -Ty yE
o(x; ((x;-1)-f (()()))))))) do (x(xx+1)8444) oly oly._1)> (((o)))))(())))))...).)))
(12)
The operator Fdl. contraction mapping with respect to the Loo norm because.
[dd((x,)) F (x,
7E
(((x)))(((((0)))..) d(y.41)~P( (lx.))
(dg - de (xs+),
(13)
<n dor (X++1. S8411)8441ll
By Banach's 's fixed point theorem, operator JF has fixed point dz.
15

---

A.3.2 Proof of Theorem 3.4
To prove the distance d is diffuse metric, we need to prove that d satisfies all of three axioms in the Definition 3.2 (definition of diffuse metric).
Lemma A.I. ||a| |bll2 a b 2 for any a, b E R.
Proof. Because b < ||a [|]|| we have,
| | a 'b
> |al 2 |b|²² 2 ||a| b|
2 |al + |b|12 2||a||| bl
(14)
= (||all
[bb]]²
00
This lemma indicates that the term under the square root of d is always non-negative. d is able to measure any two vectors a, E IR". Lemma A.2 (Non-negative). d(a, b) 2 0 for any a. b.b. R.
Proof. By definition, the square root non-negative.
Lemma A.3 (Symmetric). d(a, b) = d(b,a) a)
Proof. b) =
b a -
b
a
b d(b.a) a)
Lemma A.4 (Triangle inequality). d(a, b) + å(b, c) 2 d(a, c). for any a, C R.
Proof. To prove this lemma, it is equivalent to prove the following inequality by definition of d,
v /|aa|/2 |bbl² -a--aa T
/
||b|| + ||c||2 - bTc / /|aa| ||c||2 aTc.
(15)
Because
x
x
Xx |ly Vx we have
|a 2
|b| TTb
[/|l2 + |||||2 bTc
(16)
||||
[b|
a
b
|
|||||
||||||||
and
v 1|aa + || + | ||||
2
||al + |||||2 a Tc.
(17)
If
/ ||a| + |b|2
|a||
bb| +
/|b
+ ||||2 - |||||||| 2
V|aa|2
||||| + ||a|||c||
(18)
true, then inequality (15) is true and Lemma A.4 is proven. To prove inequality (18), can take squares on both sides without sign changing because both sides are non-negative. Then we have,
(Vila|1++ |b|2 |||||||||||
/lbb||2
||c||2
[|b|l|||]]
||a| ||c|| + ||a]]] ||c|]- (19)
To prove inequality (18), it is equivalent to prove inequality (19). Expand and simplify inequality (19), have
2 a + 2/2 |a b bb|
|c|2
||||||||| 2 bl| + ||a|| ||b + |||||cc| + ||a|||||||
(20)
The left-hand side of inequality (20) is non-negative.
if right hand side -2|[b|[2 + ||a|| [b| + |[b||||||| + |||||||||| O. then inequality (20) proven and backtrace to Lemma A.4 is proven.
16

---

2) if right hand side 2/ + |||| [b] + |b] inequality (20) and have,
|||||||| + ||||||||| 2 O, we take square on both sides of
4(||a|| |b|l2 - |||||| [b]] [[][[||||[[ |||||| || cl]] 2 b|| | |b| |||||||||| + ||a|||]]]]@
(21)
To prove inequality (21), we let left-hand side subtract right-hand side,
4[[|a]| - |bb - |a || (1) bl2 + |e||² ||||||]|| - 2-2 +
||||||||| + || + +||||||||])*
| +
|| +
+ 3| b ||c/2 6||a| b|llcc 6/ ||||||||||| + ||||||||]] ||||||
=3(|||] ||| + |b|||||| - ||]]
20.
(22)
Therefore, inequality (21) is proven. Consequently, inequality (20) in the case of 2
[b|]] 2+
|b|||||| + |||||||| 2 0 is proven. Summarize with the case of b2 ||a]|]b]] |||||||| < O, inequality (20) is proven and Lemma A.4 is proven.
[b
Theorem A.S. d is a diffuse metric.
Proof. By summarizing Lemma A.2, A.3, and A.4, function d holds the three axioms of diffuse metric. Therefore, function d is diffuse metric.
A.3.3 Proof of Theorem 3.6
Theorem 5. Let My be the space of dg. The metric update operator FChrono M.g - Ms is defined as,
Fy (x,, Ne.yy) = Ixx, - -Tyll + Txx N. (X,41. Xy. (23)
FChrono has a fixed point.
Proof. In practice, we have assumed that the sampled x, and Yy are future states of x; and ye, respectively. To be precise, we extend the definition of FChrono to cover the cases that i j or > jr.
Fohronodhy (X; Xj.
i j>jor'' >J' (24)
[rx, Ty,e + 7Ex, PF Nv 11 Py dhe (Xi41, Nr+1,83), otherwise.
We follow the proof in Section A.3.1. F FChrono is contraction mapping because.
Fehrono (x;, X,,,4 Xj, Yv Vx. FChrono day (x, |E 1 (dg - dg (x Ne41.y,)) Xx
(25)
21/l(dg dyy) (x++1. X,, Yr N
By Banach's fixed point theorem, operator Fohrono has a fixed point dz.
A.4 Theoretical Details
A.4.1 Eukaszyk-Karmowse Distance
Definition A.6 (Eukaszyk- Karmowski Distance). Given metric space (X, d) where d is metric defined on x X, the EulkaszykKarmowwkk Karmowski distance [49] a distance between two probability measures μ and X using metric d, and is defined as follows:
dr. (d)(()) v) Ex=pa""sdd(rrr)
(26)
Intuitively, dr measures the expected distance between two samples drawn from y and respec- tively. If we measure the d (d) distance between two identical distributions, the distance is not restricted to zero, i.c., drK (d) 4) 2 0.
17

---

A.5
Implementation Details
A.5.1 Implementation of m
We adopt IQE [38] to implement m. Given two vectors a, b E IR,, reshaping to Rixi where
k x 1 = IQE first computes the union of the interval for each component:
(a b) = U a.y. max(a biy )I, Vi = 1,,,..., h
(27)
j=1
where [ is the interval on the real line. Then it computes the distance among components d; as
diqu(a, b) = a (ax (a, b), dz (a, b)) + (1 - a)) meanned (a, b), .. (a, b)).
(28)
where E R is adaptive weight to balance the "max" and "mean" terms. In the scope of our method, we adopt dIQE (a, b) to implement m..
A.5.2 Network Architecture
The encoder @ takes an input of the states and consists of 4 convolutional layers followed by fully- connected layer. The output dimension of Φ is 256. The encoder vi takes an input of 512-dimensional vector (concatenated with (xx;) and ((x,)), ), feed into two layer MIPs with 512 hidden units, and output 256 dim embedding. network and policy network are 3-layer MLPs with 1024 hidden units.
A.5.3 Hyperparameters
Hyperparameters
Values
Stack frames
3
Observation shape
(3 x 3, 84,84) 84)
Action repeat
2 for finger-spin, walker-walk
8 for cartpole-swing up, cartpole-swing wttpss____ssapuuaeeeeea up_sparse
4 for others in DeepMind Control Suite
1 for Meta-Worle
Convolutional layers
4
Convolutional kernal size
3 x
Convolutional strides
[2, 1,
Convolutional channels
32
dimension
256
? dimension
256
Learning rate
le-4
Q function EMA aq
0.01
Encoder & EMA as
0.05
Initial steps
1000
Replay buffer size
500K
Target update freq
2
Batch size
128
Discount factor ?
0.95 for hall_in_cup-cate Reacher-easy
0.99 for others
Table 4: Hyperparameters
18

---

A.5,4 Algorithm
Algorithm A learning step in jointly learning SCR and SAC.
Require:
Replay Buffer D, Q network Q, policy network т, target Q network Q, state encoder ø, target state encoder Φ, chronological encoder v/. Sample a batch trajectories with size B: {Tu [T&]R=) D
Sample state Xg, transition at x, and its future state x, from each trajectory Tki TT]R=
((x,, X Ta, ajjk x)kx
Compute loss (o) in (2)
Compute loss (1) ②) in (5)
Compute loss (6) in (9)
Compute loss [(o, (b) Lo(d) + (v) + (o( in (10)
Update & and 3b by minimizing loss ( 1)
Compute RL loss LRL according to SAC objectives
Update d, Q and π by minimizing loss LRL Soft update target network: Q aQQ (1 a
Soft update target state encoder: é asΦ + (1 aΦ)
def dxx, y): pow (x pou (2.). sum (dim=-1, krepdin-True) y. (2.) sum (dim=-1,
xeepdim=True))
x + y (x y) sum (dim=-1, keepdin=True) #
return (a b) sqrt() # exrt(x + y-2 xy)
def encoder (ocoder_losssselll (self, s_i, s_i_1, s-j, s-j-1, r_i, r-j, agg_rev_i_j): phi_i = self. encoder phi_i_1 self. encoder phi_j self encoeeee
10
phi_j_1 self. colf.enccderss_j.11)
11
concatenate i and j for compute loss (2) where j not required phi_t torch. cat ([phi_,, phi_jl)
12
phi_ti corch.cat[[phi... cath[pii_i_1 phi-j-1])
= torch. cat_(rii, r-j])
15
compute loss (2)
16
permute batch to create y perm = torch. (pn.permpphi.t.hheee d_phi_t dphi_t, phi_t(perm])
17
19
d_phi_ti d(phi_tl, phi_ti(perm]]
20
loss (2)
21
loss_phi = f.mse_lose (dss(d_ppi__t_ r_t self discount d_phi__t. ())
23
compute psi self pet..si..lllphi..1 phi_j)
(5)
24
25
pai_1 1 self wwt.:ppamppppppi....p.pppppppppp permute batch to create y perm rarch.rrandpem(pptt shape [o])
phi_j)
28
d_psi d(sii psilperm])
29
d_psi_1 dppsi_1, psi_1[perm])
30
# loss (5) loss_psi F.mes loss (d_psi, r_i + self discount * d_psi detach ())
32
33
compute m (phi_i, phi_j)
compute loss (8)
36
lou_bound agg_rev_i.
37
loss_low (e.mse_ll loss (m, 1ow_bound, reduction"'noned (m low_bound detach ()) mean
compute loss (10)
19

---

up_bound = dphii_i, phi_i (perm]) + dppii_j, phi_j [perm])
agg_rew_ [perm]
40
up_bound up_ detach
41
1pss_up up @s.mse_lose (m. abs (), up_bound, Ieduction='none" (m.
abs up_bound) detach ()) mean ()
return loss_phi phi _ss Loas_lop loss_up
44
Listing I: Pseudo code of learning of representation
A.6
Computational Resources
The experiment is done with I 128-cores CPU,512 GB RAM and NVIDIA A100 GPUs. Two instances are running in one GPU at the same time.
B
Experiments
B.1 Additional Information for Distraction Settings in DeepMind Control Suite
For the distraction setting in Section 4.2, we utilized the Distracting Control Suite [34] with the setting "dificullyzeasyy... This involves mixing the background with videos from the DAVIS2017 [29] dataset. Specifically, the training environment samples videos from the DAVIS2017 train set, while 9 the evaluation environment uses videos from the validation set. Each episode reset triggers the sampling of new video. Additionally, it introduces variability in each episode by applying 11 uniformly sampled RGB color shift to the robot's body color and randomly selecting the camera pose. The specifics of the RGB color shift range and camera pose variations in line with the Distracting Control Suite paper [34]. Different random seeds are used for the training and evaluating environments at the start of training to ensure diverse environments.
13
15
B.2 Name Tasks Deep.ind Control Suite
Due to limited space in main text, use short name for tasks in DeepMind Control Suite. The full name tasks are listed below Table
Short Name
Full Name
BiC-Catch
ball jn_ in_eup-catch
C-SwingUp
cartpole-swing up
C-SwingUpSparse
cartpole-swing saapollsswinguu__ppppttte up_sparse
Ch-Run
cheetah-run
F-Spin
finger-spin hopper-stand Reacher-easy
H-Stand
R-Fasy
W-Walk
walker-walk
Table 5: Full name of tasks in DeepMind Control Suite
17
B.3 Additional Results of Distracting Control Suite in Section 4.2
19 extend the experiments Section 4.2 with compared with RAP [8], showing results Table Figure 9 shows the training curves of SCR and baseline methods in distraction setting of DM Control.
20

---

BiC-Catch
C:SwingUpp
C-SwingUpSparee
Ch-Run
F-Spin
S-Stand
R-Fasy
W-Waik
SAC
82.6.120.2
21844149
0.7.1..7
177,4184
22,5122.7
19.612262
149,4177.5
167.218
DIQ
124.0=99.6
230.0433.3
11.246,4
103.0=84.9
579.84282.8
16.82118
70.5244011
33.6=6.3
DBC
48.8:144.4
127.7119.2
00100.0
9.21199
7.7111077
5.6.12.5
149.6112.8
3..9-4.8
MICo
104.2=10
200.246.6
0.0=0.11
7.4+322
86.9276.1
1188
132.3437.8
27.5=7.7
SimSR
106,4-13.5
148.1117.3
0.0:00.0
28.241/3.8
0.4.0.1
6.611.2
78.4117
28.6-:311
RAP
82,6
451.7474.1
0,1
243,2=26.4
571.7=140.4
6.1211.7
87.8186.7
183,7=108,7
SCR
221.3-55.4
565.11159.9
185.7.93
331.71:14
738.3:124.5
400.8119.2
666.111455
556,11131.2
Table 6: Results (ttan--std) on DDM_onntron with distraction setting at 500K environment step. Distraction includes background, robot body color, and camera pose.
ball cup.cuppce cwich
cattpoll
cattaole
250-
300
A02
110
220
100
200
500
ensttonmenn stars
105
atwtorment 2083S
3e5
ansfrommntt stape
105
420
000
200
800
R
202
220
unvicommnn atar
1d5
ursharment
easy
000
/00
500
500
430-
8 F5
200
100
anchonment
160
Figure 9: Training curves of SCR and baseline methods in distraction setting of MMnCContoo. Curves are evaluation scores average on 10 runs and shadow shapes are std.
21
B.4
Additional Comparison Experiments with Data Augmentation Methods
22
Data augmentation is a category of methods efficient on reinforcement learning, toward sample efficiency and noise invariance. Additional baseline CBM [26] is bisimulation metric method combing with data augmentation method DrQ-v2, and achieves significant performance in distracting DM Control. To compare with CBM, we embed data augmentation method DrQ into proposed method SCR. Table indicates that the performance of SCR combined with DrQ surpass baseline CBM.
23
24
26
BC--Cacch
C-SwingUp
L-Swingl--Sporse
Ch-Kun
P-Spin
H.Stand
K-Fasy
W-Walk
DiQ
124.0:99.6
23000:333
11.2464
103.0484.9
579.842888
16881118
70.3440.1
3366663
CRM
642.54666.
464 14-554
27.6+55.2
422.1482.0
817.6+-1177
212.9+24.6
319.5+-27466
688.54665.6
SCR
221.31155.4
565.1159.9
185.7-93.0
331,7114
738.312455
400.8119.2
666.1114.5
553.1
SCR
DrQ
936.0+97.0
7839++620
417.2-89.5
480.2+488.
838.3+-37.3
706.4=165.5
829.0-32..3
838.6+79.5
Table 7: Results (mean:Estd)
DMM_Conten with distraction setting at 500K environment step. Distraction includes background, robot body color, and camera pose.
28
B.5 Training Curves Morld
Figure 10 shows the training curves of SCR and baseline methods in Meta-World in Section 4.4.
21

---

besteethal
2.2
6.4
speniz
Figure 10: Training Curves of Metta-World Mean success rates on 5 runs with std (shadow shape).
B.6 Additional Experiments on MiniGrid
31 We provide additional experiments on SeniGridd-FurRRooee [9] environment. MiniGrid is under a sparse reward setting that the agent receives reward only when it successfully reaches specific 33 goal, without rewards at previous steps. This experiment verifies the motivation of SCR that is able to handle non-informative rewards. We compare various netric-based representation methods that
35 combine with backbone PPO [31] algorithms. We train each run for 5M environment steps. Table shows the results with PPO, PPO+DBC, PPO+MICo, PPO+SimSR, and PPO+SCR.
Methods
PPO
PPO+DBC
PPO+MICo
PPO+SimSR PPO+SCR
Scores
0.515 #0.029 0.515 #0.012 0.350 +0.064 0.321 #0.04 0.546 =0.012
Table 8: Results (mean=std) MiniGrid-FourRRomm [9] at 5M environment steps. Results average over seeds.
36
04
122
02
08
Figure 11: Training curves on [9]. Results average over 5 seeds.
22

---

NeurIPS Paper Checklist
37
38
1.
Claims
39
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
40
41
Answer: [Yes]
42
Justification:
We have included the claims of contributions and scope.
43
Guidelines:
44
The answer NA means that the abstract and introduction do not include the claims made in the paper.
48
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
47
48
NA answer to this question will not perceived well by the reviewers.
49
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
51
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
52
53
2.
Limitations
54
Question: Does the paper discuss the limitations of the work performed by the authors?
55
Answer: [Yes]
56
Justification:
We discuss in the Conclusion Section.
57
Guidelines:
58
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
80
The authors are encouraged to create separate "Limitations" section in their paper.
61
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
63
65
66
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
67
58
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
70
71
72
73
74
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
76
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
77
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
79
81
89
84
3. Theory Assumptions and Proofs
85
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

88
Justification:
We provide complete proof in Appendix.
89
Guidelines:
90
The answer NA means that the paper does not include theoretical results.
91
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
98
All assumptions should be clearly stated or referenced in the statement of any theorems.
94
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
95
96
97
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
98
99
Theorems and Lemmas that the proof relies upon should be properly referenced.
100
4. Experimental Result Reproducibility
101
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
102
103
104
Answer: [Yes]
105
Justification:
We provide implementation details in Appendix.
106
Guidelines:
107
The answer NA means that the paper does not include experiments.
108
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
109
110
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
112
113
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
114
115
116
117
118
119
120
122
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
123
125
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
126
127
129
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
130
132
133
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
134
135
136
137
138
5. Open access to data and code
139
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
140
24

---

142
Answer: [No]
143
Justification:
We provide pseudo codes in Appendix, We will release source code after
144
published.
145
Guidelines:
The answer NA means that paper does not include experiments requiring code.
147
Please see the NeurIPS code and data submission guidelines (attps://nips.. cc/
148
olicyyyyyy)l for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
150
152
153
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //n /nips. ccc/public/pee geguddde/CodeSuuiiiee wwissssoicccyyyyy for more details.
154
155
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
157
158
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
159
160
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
162
163
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
164
185
6. Experimental Setting/Details
168
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
187
188
169
Answer: [Yes]
Justification:
We provide details in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
173
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
174
The full details can be provided either with the code, in appendix, as supplemental material.
176
177
7.
Experiment Statistical Significance
178
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
179
180
Answer: [Yes]
181
Justification:
We report the standard deviation the experimental results.
182
Guidelines:
183
The answer NA means that the paper does not include experiments.
184
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
185
186
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
188
180
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
181
The assumptions made should be given (e.g.. Normally distributed errors).
25

---

193
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
194
195
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
197
198
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
199
200
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
202
203
8.
Experiments Compute Resources Question:
204
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
205
208
Answer: [Yes]
208
Justification:
We provide details in Appendix.
209
Guidelines:
210
The answer NA means that the paper does not include experiments.
211
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
213
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
216
9. Code Of Ethics
219
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://///eeuueeeeeeeeeeee
220
Answer: [Yes]
Justification:
We follow NeurlPS Code of Ethics.
223
Guidelines:
224
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
225
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
228
227
The authors should make sure to preserve anonymity (e.g., if there a special consid- eration due to laws or regulations in their jurisdiction).
229
10.
Broader Impacts
230
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
233
Justification: This paper is a research on improving the representation of deep reinforce- ment learning. We think that there are no potential societal consequences that need to be highlighted.
234
236
Guidelines:
237
The answer NA means that there is no societal impact of the work performed.
238
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
239
240
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
242
26

---

244
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
245
247
249
251
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
252
253
254
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
256
258
259
11. Safeguards
280
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
281
262
263
Answer: [NA]
264
Justification: Our paper poses no such risks.
285
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
288
289
270
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
272
273
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
274
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
278
Answer: [Yes]
Justification: We have cited their original paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
284
The authors should cite the original paper that produced the code package or dataset.
285
The authors should state which version of the asset is used and, if possible, include URL.
286
287
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
288
209
290
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
292
294
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

296
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
297
298
13.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
300
301
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
304
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
306
The paper should discuss whether and how consent was obtained from people whose asset is used.
309
310
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
311
14.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
314
318
Answer: [NA]
317
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
318
Guidelines:
319
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
320
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
322
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
325
328
15.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
328
329
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
331
333
Answer: [NA]
334
Justification: This paper does not involve crowdsourcing nor research with human subjects.
395
Guidelines:
336
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
337
338
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
339
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
342
344
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "AdLkDmfLwW", "reviewer": "Reviewer_R7iQ", "summary": "The paper presents the State Chrono Representation (SCR), a novel approach to enhancing generalization in reinforcement learning (RL) with image-based inputs. SCR introduces a temporal perspective to bisimulation metric learning. The authors propose a learning framework that includes two encoders to capture individual state representations and the relationship between current and future states. The method is evaluated extensively in DeepMind Control and Meta-World environments, demonstrating SOTA performance in generalization tasks. The paper is well-structured and provides a solid theoretical foundation for the proposed approach.", "questions": "1. How does SCR handle data augmentation, and what impact does this have on its performance relative to other methods? As far as I know, data augmentation is very effective to generalization.\n2. Could the authors elaborate on the role of future information prediction/regularization in SCR and how it compares to existing techniques in the literature?", "limitations": "1. The paper could benefit from a more thorough exploration of its limitations and a discussion of situations where SCR might not be the best-performing approach.\n2. There is a need for a more comprehensive discussion on the relationship between SCR and established practices in RL representation learning, particularly concerning the use of future information for regularization.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper offers a robust theoretical foundation and a well-reasoned motivation for the SCR framework.\n2. The incorporation of future behavioral information into the representation space is a logical and effective strategy for improving generalization in RL.\n3. The experimental results are compelling, showing SCR's superiority over existing methods, particularly in challenging generalization tasks.", "weaknesses": "1. The discussion on limitations lacks depth. The paper could benefit from an insightful analysis of scenarios where SCR might underperform, such as the noted inferior performance on the R-Easy task compared to DrQ and DBC.\n2. While the paper mentions the commonality of future information prediction/regularization in RL representation learning, it does not provide a discussion on how SCR's approach differs from prior works like SPR or PlayVirtual and what new insights it offers. This would be valuable for the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and insightful feedback. We address your concerns as follows:\n\n> 1. The discussion on limitations lacks depth. The paper could benefit from an insightful analysis of scenarios where SCR might underperform, such as the noted inferior performance on the R-Easy task compared to DrQ and DBC.\n\nThe Reacher-Easy task presents a challenge because it requires the agent to locate a randomly placed target on a 2D plane, a task that heavily relies on understanding spatial relationships within the environment. DrQ leverages random crop data augmentation to effectively randomize the target’s absolute position, thereby help the agent in capturing the relative positioning, which is important for task success. SCR lacks of such augmentation and have less efficiency in this scenario. However, DrQ and SCR are different kinds of methods, data augmentation vs metric-based representation, integrating data augmentation into SCR would potentially utilizes DrQ's advantages and improves performance in Reacher-Easy.\n\nAdditionally, in the distraction setting of Reacher-Easy, background variations further distract perception of target object, leading to high variance and lower mean scores as shown in Table 2. While DBC appears to perform better by learning to stretch the robot arm towards a location that commonly spawns target objects, this strategy does not help to learn a robust policy for exact target reaching. \nWe will to extend the limitations section to discuss these scenarios comprehensively in the revised manuscript.\n\n> 2. While the paper mentions the commonality of future information prediction/regularization in RL representation learning, it does not provide a discussion on how SCR's approach differs from prior works like SPR or PlayVirtual and what new insights it offers. This would be valuable for the community.\n\nWe acknowledge the necessity to discuss SPR and PlayVirtual. SCR focuses on predicting a metric or distance between the current and future states which is linked to **reward, value function and policy learning**. This is in contrast to SPR, which focuses on **future states predictions** to true future states, and thus centers more on dynamic learning without direct reward integration. PlayVirtual extends SPR’s approach by incorporating a backward dynamics model and consistency loss, but like SPR, remains focused on **dynamics model**. The revised manuscript will include a comparison to clarify these differences with SPR and PlayVirtual.\n\n> 1. How does SCR handle data augmentation, and what impact does this have on its performance relative to other methods? As far as I know, data augmentation is very effective to generalization.\n\nWe agree that data augmentation is effective for enhancing generalization in RL. Currently, SCR does not incorporate data augmentation techniques because it focuses on metric-based representation learning.  However the robust performance of SCR in the distraction setting of DM_Control demonstrates its generalization ability, even comparing to data augmentation methods DrQ.  Given the distinct mechanisms of SCR and DrQ, we believe integrating data augmentation with SCR is a promising way to further improve its effectiveness. \n\n> 2. Could the authors elaborate on the role of future information prediction/regularization in SCR and how it compares to existing techniques in the literature?\n\nSCR utilizes future information prediction/regularization by estimating a metric between current and future states rather than predicting exact future state dynamics, such as SPR,  PlayVirtual and model-based methods. SCR is parameter-efficient and related closely with the RL’s $n$-step Bellman operator that can introduce higher value variance with increased prediction steps. However, SCR mitigates the higher variance issue by ensuring the predicted metrics in a feasible range rather than exact approximation. \n\n> 1. The paper could benefit from a more thorough exploration of its limitations and a discussion of situations where SCR might not be the best-performing approach.\n\nwe will extend the limitations section in the revised manuscript to discuss scenarios where SCR may underperform, specifically emphasizing tasks like Reacher-Easy. \n\n> 2. There is a need for a more comprehensive discussion on the relationship between SCR and established practices in RL representation learning, particularly concerning the use of future information for regularization.\n\nThanks for your insightful suggestion.\nWe will improve the related works on how established practices such as SPR, PlayVirtual, and other model-based methods contrasting to SCR in integrating future information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m6bcTklgw1", "reviewer": "Reviewer_EHt5", "summary": "This paper argues that metric learning for RL from one-step reward signal faces challenges in non-informative reward / sparse reward settings. The authors thus propose SCR, attempting to incorporate multi-step information in metric learning. The key components are as follows:\n1. The authors provide a MICo-like metric with a new distance function, which is also a diffuse metric and more numerically stable.\n2. Then they delineate a MICo-like chronological behavioral metric, measuring distance between two pairs of states. States in each pair are taken from the same trajectory.\n3. Then they propose a temporal measurement which can be approximated by upper-bounding and lower-bounding the measure, and thus shape the representation.\n\nThe paper experiments on the DMC environment with default setting and distraction setting, and multiple settings in Meta-World domain and show positive results.", "questions": "- Eq. 2 and Eq. 5: is $\\phi$ in the target also optimized? \n- Line 83: Should Meta-World be better  to prove the generalizability of your proposed method, rather than distracting DMC?\n- Line 144: \"a_x and a_y are the actions in states x and y\" is not precise. Do the actions come from policy \\pi?\n- Line 267: it's better to succinctly talk about what IQE is and how you use it for completeness.\n- Appendix A.2: not finished writing\n- Can you talk more about the connection to GCB (Hansen-Estruch et al.)?\n- According to the SimSR and RAP paper, in some environments it requires more than 1e6 steps to converge. Is 5e5 steps enough for convergence?", "limitations": "It is unclear that the proposed method can be seamlessly applied to intrinsically stochastic environments (i.e., the underlying abstract MDP has stochastic dynamics), where the temporal distance between two latent states may not be fixed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The issue of learning metric from sparse rewards is crucial and reported by previous work such as robust DBC. \n\nThe idea incorporating multi-step information in metric learning is interesting and novel.\n\nResults on DMC benchmarks are clearly better than chosen baselines. \n\nThe ablation study shows the effectiveness of the moving components of this approach.", "weaknesses": "Methodology:\n1. Eq.3 is not well-defined. What is the base case (e.g., what if $i=j$ but $i^\\prime\\neq j^\\prime$, thus $i+1>j$ and the recursive step is not well defined?)\n2. Eq.6 may not be true, as the optimal policy might not achieve the optimal expected discounted reward in any segment. In line 239, do you mean an optimal policy only within a given segment $(x_i,x_j)$? \n3.  It is unclear that Eq. 8 will co-hold with the classic metric bound that d upper bounds value difference, especially given that m measures local property while d measures global one. \nThe overall method is complicated which adds two more temporal embeddings based on state representations.\n\nExperiments: \n- Since the motivation of the proposed method is to handle non-informative reward scenarios, I think the experiment section should stress more importance on the sparse reward settings.  However, most experiments are done in environments with dense rewards except carepole_sparse and ball_in_cup. Neither do the authors give a comprehensive discussion on the challenges and results of those environments. \n- To align with the main claim, it should be expected to mainly work on Minigrid/Crafter-like sparse-reward environments. \n- For ablation study, the tasks of cheetah_run and walker_walk are dense rewards which cannot effectively support the claim on non-informative rewards, either. \n- Some other strong baselines are missing and worth comparing with, e.g., RAP, robust DBC (DBC-normed). RAP seems to outperform the SCR in some DMC tasks. \n- For experiments on Meta-World, please also report std in Table 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "78xg0DIbER", "reviewer": "Reviewer_b4Fc", "summary": "This paper introduces SCR that extends state metric-based representations by embedding rich time-related information into the bisimulation metric learning process. SCR is designed to calculate state distances by contextually framing them in a temporal framework that considers both future dynamics and cumulative rewards in immediate and distant future scenarios. This technique fosters the development of long-term behavioral representations, capturing the progressively accumulated rewards from present to future states. Two distinct encoders are employed: one for generating representations of individual states and another dedicated to forming chronological embeddings that represent the interconnections between a state and its subsequent states. The framework also introduces a novel metric within the chronological embedding space, alongside an unconventional distance metric that diverges from the standard $L_p$ norm. Additionally, constraints are applied to manage the magnitude of the cumulative rewards. The methodology was tested within DeepMind Control and Meta-World settings, where SCR demonstrated state-of-the-art results in tasks requiring advanced generalization capabilities. Furthermore, an ablation study was conducted to assess the individual effects of the components comprising SCR.", "questions": "1) What factors contribute to SCR's occasional underperformance in the default setting?\n2) Is the distraction setting providing SCR with an advantage, and would the findings be consistent in an alternative setting demanding generalization abilities?", "limitations": "Yes. The limitations and societal impact are adequately addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper addresses a noteworthy challenge in RL: capturing long-term temporal state representations within a well-defined metric space. To this end, SCR is proposed for learning representations centered on behavioral metrics that consider pairs of states over time. Moreover, it presents an effective technique for approximating the behavioral metrics. It develops a novel metric tailored for assessing pairs of states across temporal sequences and introduces new algorithms for this learning process. The introduced representation significantly outperforms others, particularly in the distraction setting and Meta-World.", "weaknesses": "In Table 1, the outcomes for the default setting do not demonstrate a strong level of significance, with the possible exception of noteworthy results for Ch-Run. Sometimes, the performance appears to be worse than the compared methods. There is a section of the text discussing the distraction setting, and the results presented in Table 2 are quite impressive. These should be considered the primary findings of the study. Nonetheless, the paper does not sufficiently convey the rationale or motivation behind these findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful reviews. We address your concerns as follows:\n\n> In Table 1, the outcomes for the default setting do not demonstrate a strong level of significance, with the possible exception of noteworthy results for Ch-Run. Sometimes, the performance appears to be worse than the compared methods. There is a section of the text discussing the distraction setting, and the results presented in Table 2 are quite impressive. These should be considered the primary findings of the study. Nonetheless, the paper does not sufficiently convey the rationale or motivation behind these findings.\n\n\nWe appreciate the reviewer's observations. We would like to clarify the core aim of this paper is to investigate the **generalization capabilities** of SCR.\nThe major contributions of this paper are demonstrated in the results for the **distraction setting** (Table 2).  In the **default** setting (Table 1), our claim is not that SCR outperforms baseline methods significantly but rather that it performs **comparably** (see **lines 306-308**). These experiments (Table 1 and 2) serve to demonstrate that SCR facilitates **improved generalization** rather than only offering a trade-off between default and distraction settings.\n\nThe SOTA methods achieve high scores in the **default** setting of DM_Control (Table 1), this study does not aim to surpass these in the default configuration. Table 1 illustrates that SCR maintains competitive mean scores and similar variance level to baseline methods, with the exception of the Reacher-Easy task. This Reacher-Easy task involves stretching a robotic arm to reach a randomly placed target in 2D plane at the start of each episode. The random cropping data augmentation technique (DrQ) proves effective in Table 1, while data augmentation currently absent in SCR. Nonetheless, SCR could **incorporate such data augmentation** to enhance performance in tasks like Reacher-Easy.\n\nThe results in Table 2 and Figure 4 robustly show that SCR significantly outperforms other baselines in the **distraction** setting. This setting, characterized by random selection of background videos, body colors, and camera poses at the beginning of each episode, presents a more complex challenge. These results underscore SCR's efficiency in generalizing across varied distractions. Our ablation study in Section 4.3 further delineates the contributions of individual SCR components to this performance.\n\n> 1. What factors contribute to SCR's occasional underperformance in the default setting?\n\nAs detailed earlier, the lack of random cropping data augmentation in SCR compared to DrQ particularly affects its performance in the Reacher-Easy task. SCR focuses on measuring the distance between states, which is beneficial for generalization over distractions. **Integrating data augmentation** into SCR could potentially improve its performance in similar tasks.\n\n> 2. Is the distraction setting providing SCR with an advantage, and would the findings be consistent in an alternative setting demanding generalization abilities?\n\nThe distraction setting from [1], used as a benchmark in DM_Control to assess the generalization abilities of RL algorithms, is not uniquely providing SCR with an advantage. Unlike  implementations by others, e.g. DBC and SimSR, which modify only the background video, our approach also modify body color and camera pose. Such combined modifications increase the complexity of the setting, leading to more challenging environments. This rigorous setup ensures that our findings regarding SCR's generalization are consistent and reliable across various settings.\n\n-----\n[1] The Distracting Control Suite--A Challenging Benchmark for Reinforcement Learning from Pixels. Stone et al. 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zdBQV91oiI", "reviewer": "Reviewer_UVpA", "summary": "The paper proposes an algorithm to improve upon the recently proposed class of deep bisimulation-based methods by accounting for long-term consequences (in terms of future states) instead of relying one-step bootstrapping based on reward differences to be more robust in settings such as sparse reward cases.", "questions": "1. A common issue in self-predictive representation algorithms (see Weaknesses above) is that the training procedure may lead to representation collapse [1]. Is SCR susceptible to this as well? It appears like it may be so. If not, how is that avoided here? \n2. I am curious how much of a burden it is to track future states in a given trajectory? A benefit of the other deep bismulation methods is that they can operate on a dataset of the form $\\{(s,a’,s’)\\}$, which is very useful in off-policy learning. SCR seems to require trajectory knowledge which may be a memory burden. In general, this is one of the advantages of one-step bootstrapping over n-step returns. If this is a limitation, it should be mentioned in the limitations section.\n3. Related to the above, how far into the future does a state $y$ have to be from previous state $x$ when computing the loss function. That is, using the terminology of n-step returns, how many steps in the future does $y$ have to be when computing these loss functions?\n\n\n[1] Understanding Self-Predictive Learning for Reinforcement Learning. Tang et al. 2022", "limitations": "Yes, the paper addresses this in Section 5.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. I think the idea is valuable. The sparse reward setting is challenging for bisimulation-based methods and this type of idea of factoring future states is a reasonable attempt to make these algorithms better.\n2. The algorithm is evaluated on a variety of challenging pixel-based environments, which is good when evaluating representation learning algorithms.", "weaknesses": "1. It is a bit concerning that the main results are over only 5 seeds. Personally, it makes it hard for me to reach a conclusion on the performance because of this. I’d refer authors to [1].\n2. Related to above, the variance of the reported results are quite high. For example, in Table 1, 2 and Figure 3, the variance is so high that it is difficult to conclusively say which algorithm is better than another.\n3. Table 3 does not report the number of trials nor any measure of variance.\n4. While the approach proposed for bisimulation methods is indeed unique, I believe there are strong conceptual relations to self-predictive representation algorithms and successor representations [2, 3]. It would be good if these were discussed in the related works section and if similarities/differences can be drawn between SCR and these related works to better shape the work and help readers.\n5. It appears like Section 3.1 is basically prior work, but the paper includes it as though it is a contribution. I think it would be better to move this to the Appendix since the actual contributions are the sections after Section 3.1.\n\nIf the above issues are adequately addressed, I will consider increasing the score.\n\n[1] Empirical Design in Reinforcement Learning. Patterson et al. 2023.\n\n[2] Data-Efficient Reinforcement Learning with Self-Predictive Representations. Schwarzer et al. 2020.\n\n[3] Understanding Self-Predictive Learning for Reinforcement Learning. Tang et al. 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your effort and insightful comments. \n\n> Weakness 1.& 2.\n\nWe appreciate your concern regarding the number of seeds. Due to resource constraints, we opted for a balance between the number of seeds and computational cost, thus using 5 seeds. To enhance the robustness of our results, each seed was evaluated over 100 episodes, totaling **500(=5x100) episodes** across all seeds. The mean and std scores in **Table 1** and **2** are calculated over all 500 returns.\n\nThe high variance in Table 2 come from the internal randomness of the distraction settings in the DM_Control environment, where backgrounds, body colors, and camera poses vary per episode. The combination of these attributes leads to varying distraction effects, making some episodes more challenging and resulting in lower scores. This variance underscores the challenges of these settings and validates the necessity of our approach for such distraction setting.\n\nTo further ensure the reliability of the results, we present **aggregate metrics** in Figure 4, a common practice to compare different RL algorithms. Figure 4 demonstrates that our SCR significantly outperforms other baselines in the **distraction setting**.\n\nRegarding the results in the **default setting** shown in **Table 1**, our intention is not to claim that SCR outperforms other baselines but to demonstrate that it is **comparable** to them (**lines 206-308**). The default setting of DM_Control has been extensively studied, with SOTA methods achieving high scores. The purpose of including results in the default setting is to verify SCR's effectiveness in a standard scenario. Combined with the results in Table 2, this further confirms that SCR **enhances generalization** rather than merely **trading off** performance between distraction and default settings. Table 1 shows that the mean scores of SCR are **comparable to baseline methods**, with **variance at a similar level**, thereby validating our claims.\n\n>3.\n\nTable 3 is also evaluated over 5 seeds, with each seed evaluated over 100 episodes. We update the table to include standard deviations. \n|SAC |DrQ| DBC| MICo| SimSR |SCR|\n|-|-|-|-|-|-|\n|0.495 $\\pm$ 0.475|0.886 $\\pm$ 0.125|0.479 $\\pm$ 0.453|0.495 $\\pm$ 0.482|0.258 $\\pm$ 0.365|0.969 $\\pm$ 0.032|\n\n>4.\n\nSelf-predictive representations (SPR) shares similarities with SCR, as both methods learn representations regarding a future state. However, there are several key differences between them. SPR focuses on **predicting future states**, enforcing the representation of predicted future states to be close to the true future states. In contrast, SCR focuses on **predicting a metric/distance** between current state and future state. The metrics in SCR (in Section 3.1-3.3) are all related to rewards or values, which is correlated to policy learning. SPR's representation focuses on dynamics learning, without directly considering the reward or value.\n\nSuccessor Representations (SRs) are a class of methods designed to learn representations that facilitate generalization. SRs achieve this by focusing on the **occupancy of states**, enabling generalization across various tasks and reward functions. In contrast, SCR measures the distance between states specifically to handle observable distractions. \n\n>5. \n\nWe would like to clarify the contributions of Section 3.1. The major **technical contributions** are as follows:\n1. We identify the limitations of existing metric approximations.\n2. We propose a novel approximation metric, detailed in Definition 3.3.\n3. We provide theoretical support for this new metric in Theorem 3.4 and Lemma 3.5.\n\nThese points are original contributions of our paper. Empirical results in the ablation study (**Section 4.3 / Figure 5**) show that substituting our proposed metric with existing ones like MICo, SimSR, or DBC results in performance degradation, highlighting the effectiveness of our approach in Section 3.1.\n\nFrom a writing perspective, retaining Section 3.1.1 in the main text is important as it introduces the long-term temporal information issues that motivate the subsequent sections (3.2 and 3.3). Although Eq.1 may appear less novel due to its similar form to existing work, it sets the stage for the necessity of designing a new approximation metric and leads to Eq.2, which is integral to the final objective (Eq.10) of SCR. We will revise Section 3.1 to enhance clarity and explicitly highlight its contributions.\n\n> Questions 1.\n\nRepresentation collapse in bisimulation metrics has been discussed in [1]. This issue can arise in scenarios with sparse rewards, where the reward difference term in Eq.1 is always zero, leading to the collapse of the distance between states $x$ and $y$. Addressing sparse rewards is one of the objectives of our paper.\n\nTo mitigate the risk of collapse, we introduce a temporal measurement that extend to multiple states, thereby increasing the likelihood of encountering non-zero rewards. Additionally, the representation function in SCR is continuously updated through the RL objectives, which provide additional gradients and help avoiding the representation collapse.\n\n>2.\n\nWe store entire episodes in the replay buffer in a contiguous manner, with each step stored as $(s, a, s')$, similar to other one-step off-policy methods. The only additional memory burden is the storage of indices for the start and end of each episode. These indices are integers, therefore, the memory overhead is negligible.\n\nWhen sampling a batch of training data from the replay buffer, we first uniformly sample a batch of steps $i$. We then sample steps $j$ based on the indices of the start and end of the episode, ensuring that steps $i$ and $j$ are within the same episode. Finally, we swap $i$ and $j$ if $j > i$ to ensure that $j$ is a future step relative to $i$. This method keeps the memory overhead minimal while effectively tracking future states.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Nearly all reviewers recommend acceptance of this paper. The sole exception is reviewer EHt5, who provided valuable feedback and raised valid concerns. The authors have addressed some of these concerns, but additional experiments are necessary to fully resolve these issues. However, I believe the current set of experiments is sufficient to demonstrate the capabilities and some limitations of the model.

That being said, I agree with reviewer Eht5 that the current set of experiments does not fully support the claim of "SOTA performance in demanding generalization tasks." This claim should be rephrased to be less assertive. However, I don't think this is enough grounds for rejecting the paper. The paper provides valuable theoretical and empirical contributions to the field.  

I encourage the authors to incorporate the results from the rebuttal, make the SOTA claim more moderate, and overall recommend this paper for acceptance.
Decision: Accept (poster)