Learning Representations via a Robust Behavioral Metric for Deep Reinforcement Learning
Authors: Jianda Chen
Sinno Pan
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep reinforcement learning
representation learning
deep learning

Abstract: Learning an informative representation with behavioral metrics is able to accelerate the deep reinforcement learning process. There are two key research issues on behavioral metric-based representation learning: 1) how to relax the computation of a specific behavioral metric, which is difficult or even intractable to compute, and 2) how to approximate the relaxed metric by learning an embedding space for states. In this paper, we analyze the potential relaxation and/or approximation gaps for existing behavioral metric-based representation learning methods. Based on the analysis, we propose a new behavioral distance, the RAP distance, and develop a practical representation learning algorithm on top of it with a theoretical analysis. We conduct extensive experiments on DeepMind Control Suite with distraction, Robosuite, and autonomous driving simulator CARLA to demonstrate new state-of-the-art results.  

Learning Representations via a Robust Behavioral
Metric for Deep Reinforcement Learning
Jianda Chen
Sinno Jialin Pan
Nanyang Technological University, Singapore jianda0010e ntu. edu.
Nanyang Technological University, Singapore sinnopanOntu. edu.
Abstract
Learning informative representation with behavioral metrics is able accelerate the deep reinforcement learning process. There are two key research issues on behavioral metric-based representation learning: 1) how relax the computation of a specific behavioral metric, which is difficult or even intractable to compute, and 2) how approximate the relaxed metric by learning an embedding space for states. In this paper, we analyze the potential relaxation and/or approximation gaps for existing behavioral metric-based representation learning methods. Based on the analysis, we propose a new behavioral distance, the RAP distance, and develop practical representation learning algorithm on top of it with a theoretical analysis. We conduct extensive experiments on DeepMind Control Suite with distraction, Robosuite, and autonomous driving simulator CARLA to demonstrate new state-of-the--a results.
Introduction
Deep reinforcement learning (RL) aims to interactively learn optimal policy from high-dimensional environmental observations states in an end-to-end manner, In the literature, it has been demon- strated that robust representation of states, which are task-relevant and invariant to task-irrelevant background information, able significantly speed up the RL process and makes the learned policy more generalizable. Therefore, representation learning has played key role in Deep RL algorithms and attracts more and more attention in the RL community 4 23 17 20).
Prior work on representation learning is focused on learning embeddings to represent states based on reconstruction loss [26 25 13] Though promising results have been reported on some RL application domains, policies learned with such representations usually fail to generalize well a complex environment because minimizing a reconstruction loss may potentially introduce local (visual) features with task- moskirrelevaant information. Another research direction is to apply data augmentation techniques to learn robust state representations for an RL agent 199 28 27 While data augmentation-bast approaches may be able to learn more robust feature representations, they do not take the characteristics Markov Decision Processes (MDPs), e.g., reward signals and dynamics models, into consideration when learning representations. As result, the learned representations may not be informative for learning an optimal policy for the task interest. A third research direction to construct some auxiliary tasks in addition to the prime RL task and learn state representations by learning all the tasks simultancously [14 2 21]. These approaches can be considered as shaping the state representations implicitly by learning the auxiliary tasks, which do not have guarantee for learning a better policy, especially when the auxiliary tasks fail to benefit the learning of the RI. task.
Recently, behavioral metrics, such as the bisimulation metric and its variants 7 8 3. have been exploited in representation learning for deep RL. Behavioral metrics are originally proposed to measure the behavioral similarity between states in terms of rewards and dynamics models, e.g, state transition probabilities. The high-level idea is to learn state embeddings by preserving the behavioral
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

similarity between states based on a specific behavioral metric | 16, 4 As behavioral metrics provide a theoretical bound on the difference between the outputs of value function of a pair of states, the learned representation enjoys theoretical guarantee to capture behavioral structure in the environment for policy learning. However, as behavioral metrics are expensive or even intractable to compute, different approximation approaches and learning objectives have been proposed to make behavioral metric-based representation learning for RL agents more efficient [29 46.4
Though behavioral metric-based representation learning methods have achieved state-of-the-art results on some benchmark RL problems, they suffer from at least one of the following three issues: loss function mismatch, relaxation of dynamics model divergence and the L2/L2 norm limitation.
Loss function mismatch. In behavioral metric-based representation learning, a loss function is to measure the difference between the distance between states on the embedding space and their behavioral metric. As behavioral metrics are expensive even intractable to computers, approximation or relaxation are necessary. However, based on our analysis, state-of-the-an methods adopt improper approximations, which may introduce a bias and make the bound of the value function looser.
Relaxation of dynamics model divergence. The bisimulation metric or its on-policy invariant is one of the most widely used behavioral metrics, which requires estimating the 1- I-Wasserstein distance between dynamics models. As the I-Wasserstein distance is usually difficult intractable estimate, prior methods propose relaxations replace the estimation of the 1- I-Wasserstein, which may break some theoretical guarantees of the bisimulation metric.
L,/L2 norm limitation. The Ln and the L2 norms are commonly-used distances with zero self-distance. However, due to the two gaps mentioned above, the approximations or relaxations of behavior metrics in prior approaches Using the L1 or Le distance on embedding space is difficult to learn robust representation to preserve the behavioral similarity between states.
The
potentially non-zero self-distance.
To address the aforementioned issues or gaps, in this paper, we first introduce a new behavior metric namely the Reducing Approximation Gap (RAP) distance, and then develop a practical approximation algorithm with consistency to its theoretical prototype. this way, our algorithm is guaranteed to learn robust state representation capture the behavioral similarity between states. We conduct extensive experiments on DeepMind Control Suite (DMC) 24 with background distraction, Robosuite (30) and autonomous driving simulator CARLA 6 demonstrate new state-of-the- of-thh--rtt-ttt results compared with other behavior metric-based representation learning methods.
The contributions of this paper are two-fold: 1) we analyze the potential approximation gaps for existing behavioral metric-based representation learning approaches, and 2) we introduce new behavior distance RAP and develop its practical approximation algorithm with theoretical guarantees.
2 Related Work
In RL., early research work has focused on learning state representations by designing and optimizing some auxiliary objectives in addition to the RL task of interest. For instance, Hafner et al. [I]] propose learn dynamics model to predict future states with reconstruction loss. Gelada et al. aim to learn state representations by predicting dynamics model and reward function
embedding space. Laskin al. [17 apply contrastive learning with samples generated by the momentum encoder. Hansen et al. [12 predict an inverse model as an auxiliary task for representation learning. Another type of approaches aims to apply data augmentation techniques to improve representation learning. For instance, Laskin et al. [18] conduct an extensive study of data augmentation for deep RL with pixel-based input. Yarats et al. [28] adopt random crops on pixel-based input and add regularization terms the Q-function objectives. Lee et al. [19] introduce convolutional neural networks with randomized parameters. Stooke et al. (33 use augmented samples for representation contrastive learning. In contrast to these works, our proposed method aims to encode state representations into structural metric space based on behavioral metrics.
Recently, behavioral metric-based representation learning has attracted more and more attention in the RI. community. For instance, Zhang al. [29] aim to learn representations by approximating the bisimulation metric 7 3 on an embedding space. Agarwal et al. [ propose a behavioral

---

metric considering distance between action distributions given states for representation learning. Kemertas and A Armstrong [[6] propose to improve the robustness of the representation learning method proposed in [29 by adding norm constraints on the embedding space and intrinsic rewards. Castro
4 introduce new behavioral distance and develop a sampling-hasee approach to preserve the behavioral similarity between states on the embedding space. Chen and Pan [S] propose to learn neural networks to approximate components in the bisimulation metric on the state embedding space.
3 Preliminaries
Reinforcement Learning We consider Markov Decision Process (MDP) defined by a tuple (S, R, P. P.)). where the high-dimensionas state space, A is the action space, P(s'|s. [ss"|s, a) the transition distribution that captures the probability of entering next state s' S given current state s E and an action E A, R S x + is the reward function and y E [0, 1) is the discount factor. In the sequel, use Ps and r to denote P(-ls, a) and R(s, respectively. A policy <a]s) is a probability distribution over each action a conditioned on a state s. The value function V* S - I for given policy и at state defined as the expected sum of discounted future rewards,
(T)
E
Σ
Ett
SD
(RL)
The goal of reinforcement learning is to find optimal policy = max, V that maximizes the expected future rewards. the scope representation learning for deep RL, state encoder S + R" maps nigh-dimensional state low-dimensional vector, with which policy a (a (())) is learned.
Bisimulation Metrics The bisimulation metric 7 g defines a pseudometric d S R
to measure the behavioral distance between states. Recently, a variant of the bisimulation metric, on-policy bisimulation metric (or r-bisimulation metric) is proposed [3], which focuses on behaviors relative to particular policy и. The и r-bisimulation metric consists of reward difference term and Wasserstein distance in dynamics models between states.
Theorem 3.1 (r-bisimulation metric (3]. Let Mi be the set of all pseudometrics on space S. pseudometric transformation function FB M - MI is defined as,
Fi(d)(s,,s)) | Ea r2. Eajon Tsj + W PT,)
(1)
where Bajwa aa = (aaoe(l-)s.)) na, P = Eawn(-)))) Po and W1 is the 1-Wasserstein distance. F3 has least fixed point di and di a n-bisimulation metric.
The following theorem shows that the difference in value function is bounded by d' Theorem 3.2 (Value difference bound
3) Given states and Sj, and policy n, we have
(s:)s; - V" (s;) S (s;
4 a variant of the π r-bisimulation metric, which measures the distribution distance between dynamics models by computing the distance between sampled next states from the dynamics models order to avoid the computation the Wasserstein distance.
MICo distance The MICo distance
Let Mi be the space of distance function d - IR, if the MICo metric function F. M. ML is defined as,
Theorem 3.3 (MICo distance 41 ).
F (d) s3) |E n m.
Ea
1xj + BB d(s), 8,)s
(3)
then Fhr has unique fixed point di. Theorem 3.4 (Value difference bound 4)). Given states and 8j, and policy n, we have
(s:)s:) V (s,)) die (s.,
(4)
3

---

Approximation Gaps in Behavioral Metric-based Representation Learning
The high-level idea of behavioral metric-based representation learning is to learn an embedding space such that after mapping states onto the embedding space, the behavioral similarity can be preserved. denote the state encoder by ow :S + IR" with parameters w and the distance between states on the embedding space I" by (s,), (sy)), e.g., d can be the Lg norm distance. The problem of be cast a minimization problem of the following expected
representation learning in terms of squared difference or loss between the distance on the embedding space, (s:), ow (s,)), and the corresponding behavior metric, d' 85), between any pair of states S; and Sy²
[(ow) E ( (d(d - df (st,
(5)
To develop practical algorithm minimize the above objective, prior approaches adopt different approximation or relaxation strategies to make the resultant optimization problem computationally tractable. As discussed above, there three common issues underlying most prior approaches, which introduce gaps between the practical algorithms and their theoretical prototypes, i.e., (5). namely loss function mismatch, relaxation of dynamics model divergence, and the L₁ the L2 norm limitation. We discuss these 3 gaps in detail in the following sections.
4.1 Loss Function Mismatch
In general, learn the encoder dw by minimizing is computationally intractable or expensive
because of the computation of the behavior metric d We take MICo (MICo-based representation learning [4 as an example. By specifying the term dl (s,, sy) in (5) as the MICo distance defined in (3. the loss of MICo becomes,
((ow) = ( / d(o))s)). (w,)) - [Ban wa r3) Enjoan pm rayl y
B or d(os (s)) ef
(6)
where @ is copy of parameters for the target network. However, the reward expectations Ea,~x 7 and Eay- T's in the 2nd term in (6) are computationally intractable and also difficult estimate even based on sampling. Thus, Castro et al. 4 propose to approximate the loss in (6) with the following alternative loss,
[(Ow)
[ (1() E Ag Sommeea P. PS)
|r: - -,. (s) (s.).
(7)
Note that (7). the expectation operator on rewards is moved out of the absolute value operator of the difference between rewards to avoid the estimation of expectation
rewards, Ea, rh: and Ea,~x rsyy. and enable sampling more efficient. However, such a revision introduces gap between solutions of minimizing 7 and (6), because in (7) the reference behavioral metric is no longer the MICo distance but the "shift" MICo distance defined follows.
Definition 4.1 (Shift MICo distance). The shift MICo distance function Fig is defined
-Tz] F (d)(s) 8;) = Eaa-r |r:: - 8:1 + + H ay
B. d(s), 8t).
s
Lemma 4.2 (Fixed-point), The shift MICo distance function F7y has unique fixed-poiie di
Proof. (Sketch) This be proved by following the proof of Theorem 3.3 by using Banach's fixed-point theorem. A detailed proof can be found in Appendix A.1
The above lemma shows that the approximated distance on the embedding space d((w(s)). ow (s,)) still converges to the Shift MICo distance, di by minimizing (7). However, as Ea, -o in aj Nu Ea,oy 8. Eajna 1x] we have the following theorem, whose proof can be found in Appendix A.2
Ts3 2

---

Theorem 4.3 (Looser value difference bound). Given states s; and Sy, and policy n, we have
(s.) (s, Sy) dd (s;:
(8)
Based on the above theorem, as the Shift MICo distance has looser value difference bound, it may be less relevant the value function. As result, the learned representation may not be able to encode the behavioral similarity between states accurately, Apart from MICo, other behavioral metric-based methods, which consist of the on-policy reward difference term, such as DBC 29 16 and AMBS [5]. also suffer from similar approximation gap.
4.2 Relaxation of Dynamics Models Divergence
The 7-bisimulation metric needs compute the 1-Wasserstein distance W1 between dynamics models to measure the distribution distance. In r-bisimulation metric-based representation learning, one can learn the encoder &w by minimizing the following loss,
((ow) IE (dld
Ba. r
Egg muT Wi(d)) (L.))) P (s)))
However, the 1- I-Wasserstein distance is computationally expensive or intractable. In DBC [29 the 2- Wasserstein distance W2 is proposed replace W1. as W2 has convenient closed-form of Gaussian distribution with respect to the L2 distance. Specifically, the loss function of DBC with batched sampled transitions is defined as,
[(dw) - (ddo (w) - 1ra: - - an (s)) Po )
where || ||2 is the L2 norm, P a dynamics model on the representation space, and स is the expected policy output. The use of W2 almost breaks all theoretical guarantees for the bisimulation metric. The existence of unique fixed-point in the bisimulation metric requires the continuity and monotonicity of Wi with respect to d 77 The properties of continuity and monotonicity do not hold with W2- Therefore there is more guarantee about the fixed-point existence in DBC except that both the dynamics model and the policy π are deterministic, which case (2(d) degenerates to d and Banach's fixed- point exists [16]. However, this assumption may be too strong hold in practice.
Instead of using the family of Wasserstein distances, in MICo as shown distribution divergence, P...Pr, d(s),, s))), is used to measure the difference between dynamics models. This sample-based distribution divergence can be considered as a Bukkaszk-Karmoon metric. While Wasserstein distance has zero self-distance, a Lilkaszyk-Karmoww metric does not satisfy the identity of indiscernibles. As a result, the approximated distance on the learned embedding space based the MICo distance, which involves a Lukaszyk- Kaankks-KKKkkki metric to measure distance between dynamics models, may also suffer from the violation issue of the identity of indiscernibles.
the sample-based
4.3 Limitation Using the L1 or the L2 Norm on the Embedding Space
As mentioned in Section 4.1 to avoid the expensive or intractable computation of the expectation over rewards, prior approaches, such as MICo, and DBC, use an alternative term to measure the reward difference between states, Barosrrr2 rsj J. In the following, we discuss the case that state and sj are identical.
Lemma 4.4. Ifs, sy. then Earor/rg 1r
r 2 0. The equality holds only if the reward function
aynu constant w.r.. action a or the policy is deterministic.
Note that most RL tasks, a stochastic policy is widely used for exploration and reward function is rarely constant w.r.t. actions. Therefore, in practice, Eacor y.a.. r53 0 for most RL tasks.
a, R,~-x
Besides, as mentioned in Section 4.2 a dynamics models does not satisfy the identity of indiscernibles. Therefore, the behavioral metric, which is a sum of the reward difference term and the dynamics model distance term (no matter Wasserstein distance a Karmowkkk metric), can greater than zero on a pair of identical states.
Karmowski metric measuring distance between
5

---

Note that both the L1 and the L2 norms satisfy the identity of indiscemibles, i.c. when x; x,, ||x; Xy|1 =|x; x;112 = 0. If leverage the L the Lg norm the form of distance d on the embedding space to approximate the behavioral metric, then the identical states pairs has zero distance on embedding space. However, the regression target, i.e., the behavioral metric, greater than zero on identical states. As result, when minimizing regression loss between the L1 / L2 norm and the behavioral metric, the representations for similar states, especially identical ones, will be pushed apart in the embedding space.
Recently, AMBS [5] proposes to neural networks to measure distance state embedding space rather than using the L, or the L2 norm. However, in this case, the learned "behavior metric" does not have theoretical support such the fixed-point convergence guarantee. Besides, MICo proposes a new form of distance on the embedding space, which is a sum of angular distance between embeddings and the L2 norm of the embeddings. Note that the proposed distance has non-zero self-distance.
5 The Proposed RAP Distance
We firstly propose a new behavioral metric to measure the state similarity without computing the Wasserstein distance between dynamics models in Section 5.1 The proposed behavioral metric namely the RAP distance enjoys theoretical properties such as fixed-point existence and a value difference bound. We then present practical algorithm to learn the state encoder by approximating the RAP distance on the state embedding space in Section 5.2 Our algorithm uses the learned estimation of reward functions and dynamics models provide distance approximation which is consistent to the behavioral metric. It addresses all the aforementioned approximation gaps and preserves the theoretical guarantee about the value function difference bound. Particularly, the approximation gap of loss function mismatch is addressed in Section 5.2 the relaxation of dynamics model divergence is addressed in Section 5.1 and the limitation of the Lı the La norm on the embedding space is addressed in Section 5.3
5.1 Definition and Properties of the RAP Distance
In order avoid the high computational cost W or the approximation gap introduced by relaxation to W2 as described in Section 4.2 we consider a distance measure between dynamics models pr and Pa, with sampling. To be specific,
on-poliey behavioral distance is defined follows.
Definition 5.1 (the RAP distance). Let M be the space of distance function d S x + R, the RAP distance function Fg M + M is defined as,
(s) 8n)
Ea M
Eay ra;
+ + Eaa d(E E(s)D)
(9)
ay~*
where [[s[]] = P [s|] is the expectation value of next state over the dynamics model Pa:
We design the behavioral distance by measuring the expected states over dynamics models recursively, which removes the requirement of sampling on pa and Ps, but only performs sampling on the policy æ. In practice, the expected next states are generated by a learnable approximated dynamics model as described in Section 5.2
Theorem 5.2. F is a contraction mapping w.r.t the Los norm and has a unique
dz
Proof. Let d, d' E M. We have
(F(()(s,ss) FF(dd)
n (aa d'"()) < < d' d'lloo
Therefore, F& is contraction mapping w.n.t. the Tax norm and there exists unique fixed-point for Fg due to Banach's 's fixed-point theorem. This completes the proof.
provides a convergence guarantee for the RAP distance that by iterating Fg. distance Theorem 5.2
will converge to the fixed-point dz

---

Theorem 5.3 (Value function difference bound). Given states and Sj, and a policy π, we have V" (s,)) - (s,,)),
(s:)s)
(10)
The proof can be found in Appendix A.3 Theorem 5.3 demonstrates that the RAP distance between states upper-bounds the difference of their states values.
5.2 Approximation of RAP
straightforward way learn a representation encoder approximate the RAP distance on the embedding space is to minimize the loss in (5). However, the approximation gap of loss function mismatch as mentioned in Section 4.1 still exists if relax the first term as Ba ny~
Ir: rsj
This leads to the learned metric having looser value difference bound than the original behavioral metric as proved in Section 4.1 Here, we propose another alternative relaxation of |a " 22 Baysr rs; to address this issue. Let rs be a random variable over the action distribution defined
a, 2 by p(rss r Tals)) We first analyze the difference between EaasT Ira. Tsj
and
Bn, - 2
E. En, T'a Baaox |
raj
En, ~xx
Taa
Enjax -ix
ray
12
E [(r))
+
E aymis
[(r2)) }
[1a] r=;]
TE miima
T'
+2
IE
Lay
E [(ra))
IE
rs
+
[(r())]]
E Lay-un
Taj
2Ea. 8y
[1
1-2
E aa
a,-
Var[r.:] + Var[rs,] 2Coulle,
(11)
Since Tss and rsy are independent, Coufrass Ts,] = O. Therefore, have the reward difference term
Eart P.
Eajaw
Tsj ay-n
EanT
|ra rzj [3]
Var[ra.] Var[ra,]
and the revised RAP distance at fixed-poine as
(S;, =
V
Ear-r ai
8, [2] 183
Varlra.] - V Var[rs,]
+ (e 9,
[B -- (5))) (12)
In 122 we successfully move the expectation operator on rewards out of the absolute value operator without introducing any approximation gap caused by loss function mismatch. However, there are three issues that need sampling, 2) the variances Var[rs.] and Var(rs,) are intractable to compute, and 3) how to estimate the expected next states E P [s[]] and E P [s]]]
further addressed: 1) the square root introduces new bias under
In order to reduce the bias issue introduced by the square root, we try to remove the square root loss of learning the RAP distance. We move the dynamics term in (12) the left-hand side, then take square on both sides and get
(s:, ( Ea,- d% (E, p aj NTT
[s]], og re (sS])
)
Eaic a, mutT
||r::
ráy r3 ]
(13)
Let (sch, (s;)) be the approximated RAP distance between s; and sy parameterized by w. The loss for learning di (s:), (_.(s)) (s,)) is to minimize the mean squared error between the left-hand side and the right-hand side in (1)
L = (((o (w),)) 7Ba de ( [s1], 1B, 33 (s,1)) 2
(Ira: rxa m Var[ra.) - -Vrr(rs))) )² (14)
7

---

Such a loss consists of a pair of squared difference terms as the regression target, which approxi- mate the squared difference between distances of current states and distances of dynamies models. respectively.
The reward variance Var[rs] is computationally intractable, but approximator to estimate it by assuming that the reward i' on state is Gaussian distributed. Let Tú be the learned reward function approximation parameterized by v, which outputs Gaussian distribution, (s) = (ū(rs), (ra)), where (r(rs) and (ra) are the mean and the standard deviation, respectively.
learn neural network
Similarly, in order to estimate the expected next states Eg [s'] with a neural network ap- proximator on the embedding space, learn dynamics model P taking input of state bedding (s8) and action a and outputs a Gaussian distribution over the next state embedding, (()w(s) a) = ( (s)), ē Pa (s))), where T Pa (s))) and ã Pa (s) are the mean vector and the standard deviation vector for predictive embeddings of the next state, respectively.
The RAP distance between expected next states, dg ([())) can be approximated by a(p(Pa (x)))) ( (s,))) The variance Var[rs] can be replaced by the learned reward function
output as (rr)))) By replacing the dynamics term and reward variance terms in (1).. propose the RAP loss defined as
CRAP( ED (d(s. (w)))) yd(fi( pra. (.
sa, )o(t,)))))
- 135 (0(rz,))=))) (() (o(rx,) -
(15)
where @ is a copy of parameter with stop gradient and Φa is the encoder in the target network and D is the replay buffer or the set of data that RL algorithm, e.g. SAC [10), learns from. The loss [5 is trained over the transitions sampled from D.
5.3 Explicit Form of Distance on the Embedding Space
the behavioral metric is usually with non-zero self-distance. Besides, the distance measured between the next-state distributions in the RAP distance a Eukaszyk-Karman Karmowski metric [22], which also has non-zero self-distance. Here, we adopt the approximation form of distance on the embedding space as proposed in MICo [4.
As discussed in Section 4.3
Definition 5.4 (MICo approximation). Let dG R X IR be distance function representa- tion space RR, x, y E Ih" and l: >> O. de is defined as da y) ||x [| || + k0(x,, where angular function evaluating the absolute angle between vectors x and y, and i a hyperparameter. In practice, we set 10-3 for our method.
As the above form of distance produces non-zero self-distance, with de the approximation of the RAP distance will not push apart similar states on the embedding space.
Lemma 5.5 (Non-zero self-distance). The self-distance of de not restrict to zero: de(x, x)
2 2| | 2 0. The equality holds only if all the elements of x are zero.
5.4
Implementation
du()))
S4C
The network architecture of our method is shown Figure I Our method is built upon SAC [10] Actor and critic networks take input of state representation (s). The SAC objectives and the RAP re- gression loss CRAP( (tww are optimized jointly. More implementation details can be found in Appendix B The source code available at https:/ / /gi thub. com aidda- Chen/RAP Chen/RAP_distancee distance
Inar(é
du(ss)
Oflic
Axdor
Figure 1: Network architec- ture of our method.

---

Experiment
In this section, evaluate the efficiency, robustness and generalization ability of the representation learned by our method on three RL benchmarks: 1) Distracting Deep Control Suite (DMC) (24, 2) Robosuite [30], and 3) CARLA [6] These all control tasks in continuous action spaces with visual input. We compare our method with several baselines and state-of-the-an methods including metric-based representation learning and data augmentation: 1) SAC (TO), baseline RL method for continuous control, 2) MICo [], sample-based behavioral metric-based representation learning method for RL, 3) DBC [2)] representation learning method by approximating the bisimulation metric with the L1 norm, 4) RobustDBC [T6]. DBC-styled method with intrinsic rewards and inverse dynamics, and 5) DrQ [28]. an image augmentation method on pixel inputs RL.
(a)
(b)
Figure 2: Illustrations of observations in DMC: (a) original background setting; and (b) natural video background setting. From left to right: cartpole-swingup. theetah-run, finger-spin, and walker-walle
Distracting DeepMind Control Suite To evaluate the generalization ability and robustness of our method, we perform experiments on 2 settings, original background and natural video background settings, in DMC [24). We render 84 84 pixels observation at each time step and stack frames as state. In the original background setting, we use the default background provided by DMC (as shown in Figure 2ap. For the natural video background distracting setting, follow 29 to replace the background with natural video sampled from Kinetics dataset 15 (as shown in Figure We sample 1000 continuous frames from the video dataset background for training RI agents and evaluate agents on another 1000 continuous frames. The video background is considered as distraction RL algorithm. For each setting, we train and evaluate 4 tasks: cartpole-swingun cheetah-run, finger-spin, and walker-walk. Each task is trained using million environment steps.
2000
1200
2000
500
200
36 01 02 24 33 08
28
03 83
01 04
05 05 25
06 05 05 05 03 83 02 01 04 23 05
00 83 02
Figure 3: Experimental results on DMC with original background. Each curve is average on runs.
Cerppele setrgup
cheeta
Trnger cpdn
wolker 2o0k
620
$000
230
300
Pedoeeent Medi
40t www.eor Mixe
Figure 4: Results on natural video background settings in DMC. Each curve is average on 3 runs.
Figure shows the training curves in original background settings. It confirms that our method RAP can accelerate the training and has comparable results to data augmentation method DrQ. Figure shows the training curves in natural video background settings. Our method RAP outperforms DBC and MICo on all 4 tasks. It also converges to higher score than DrQ in I task and learns much faster in tasks. The new SOTA results verify that our method of reducing the gaps approximation of behavioral distance can improve the efficiency and robustness for deep RL and they also confirm that our method is able learn generalizable state representation complex environments.

---

Robosuite Robosuite [30] is robotic simu- lator with various manipulation tasks based on MuJoCo engine. To evaluate the robustness of methods, we use the distraction settings for Ro- bosuite by randomizing the colors of robot arms and table, the lighting source and luminance, and the camera position. The color, lighting and camera position are changed at the beginning of an episode but consistent within the episode. Our experiment is performed by controlling the Panda arms. The control rate 20Hz and the episode length is 500 steps. We evaluate on two tasks: Door Opening and Two Arm Peg-In-Hoke Figure Shows illustrations of observations of two tasks. We compare our method with behavioral metric methods MICo and DBC, and data augmentation method DrQ. Table shows the training scores in 500k environment steps. Our method RAP outper- forms the others. The converge scores are around 7x and 1.4x compared to the second- best method in Door Opening and Two Arm PoglIn-Hole, respectively. It shows the robustness of our RAP in environments with random distractions.
Figure 5: Illustrations in Robosuite. Left: Door Opening. right: Two Arm Peg-In-Hole.
Table 1: Experimental results Robosuite trained with 500K environment steps.
Task
RAP(Ours)
SAC
MICo
DBC
DrQ
Door Opening
I02.19 = 26.11
8.8419989
6.06:16.57
3.15+3.54
14.63
Two Arm Peg-In-Hole
207.27=25.70
191.294344.8
123.69+43.25
219.564336.7
156.86++33.57
CARLA In order to validate the generalization abil- ity and learning efficiency in natural scenarios, we construct experiments on an autonomous driving sim- ulator CARLA [6]. which provides 3D realistic on- world scenarios. The goal of this task is to control vehicle driving as far as possible on high-way map (Town 4) in 1000 time steps. The reward func- tion followed [29] is designed to encourage driving far and avoid collisions with other vehicles. The observation is formed as 420 x pixels a 300 de- grees ego-centric view, constructed by concatenating five cameras. In order to evaluate the generalization ability, randomly sample kind of weather in each episode starts. The weather (sunlight, rain, etc.), which affects the visual observation, ered as real-world distraction to RL agents. Figure 6 shows the comparison results of our RAP and other SOTA methods. Our method achieves comparable scores with data augmentation method DrQ. and its score is higher than behavioral metric DBC and MICo. shows that our RAP is able to generalize well in tasks with real-world scenarios.
carla
800
600
scores
400
RXPhoUnS
SAC
DBC
-200
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8 0.9 1.0
environment steps
1e5
Figure 6: Training curves on CARLA.
be consid-
7 Conclusion
Representation learning is one of the most critical problems in high-ddmensional deep RL. In this paper, we propose a new behavioral metric and a practical representation learning algorithm on top of the new behavioral metrie for deep RL. We provide theoretical analysis for our proposed metric as well as the representation learning algorithm. We conduct empirical studies on multiple RI. domains to verify the effectiveness of our proposed method.
Acknowledgement and Funding Disclosure
This work is supported by the 2020 Microsoft Research Asia collaborative research grant.
10

---

References
[1] Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Mare G Bellemare. Con- trastive behavioral similarity embeddings for generalization in reinforcement learning. In International Conference on Learning Representations, 2021. URL net/forum?id-qdee -Vg84
[2] Marc G. Bellemare, Will Dabney, and Rémi Munos. distributional perspective on rein- forcement learning. In Doina Precup and Yee Whye Tch, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learn- ing Research, pages 449 458. PMLR, 06-11 Aug 2017. URW htttps:/ mlr srrss/vv0//ellemave
html
[3] Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference Artificial Intelligence, pages 10069-10076, 2020.
[4] Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. MICo: Improved representations via sampling-based state similarity for markov decision processes. In A. Beygelz- imer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[5] Jianda Chen and Sinno Pan. Learning generalizable representations for reinforcement learning via adaptive meta-learner of behavioral similarities. In International Conference on Learning Representations, 2022. URL, http:///gpppaaeeeeeeeeeeerrrrrrn wtt.://ooorrrot100011111111111
[6] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference Robot Learning, pages 1-16, 2017.
[7] Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, UAI '04, page 162-169, Arlington, Virginia, USA, 2004. AUAI Press. ISBN 0974903906.
[8] Norm Ferns, Prakash Panangaden, and Doina Precup. Bisimulation metrics for continuous markov decision processes. SIAM J. Comput., 1066)1166-1111,, December 2011. ISSN 00097-3377. doi: 10. 1377/10080484X. URI. htt.httst///oo.... htt.://..////1111//1.1... 1137/10080484)
[9] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G. Bellemare. Deep- MDP: Learning continuous latent space models for representation learning. In Kamalika Chaud- huri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2170-2179. URL [tt.:/ttt:///rrceediini
PMLR, 09-15 Jun 2019.
mlr. wwt..mml.....ssseeee wwwpss/vre//// http://977geaaaa9...aaaaaaa html
[10] Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870. PMIR, 10-15 Jul 2018. UtL.attt:://proceeddii
mlr. Paress/v0//aanee a18b. html
[11] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representa- tions, 2020. URL httph::: //opprrv//peeeevies
[12] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyå, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang- Self-supervised policy adaptation during deployment. In International Conference on Learning Representations, 2021. URL https://openrevierr Ltte/forumiidoo_-MMMyyyy
[13] Irina Higgins, Arka Pal, Andrei Rusu, Loie Matthey Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. DARLA: Improving zero-shot transfer in reinforcement learning. Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine

---

Learning Research, pages 1480-1490. PMLR, 06-11 Aug 2017. URL http://proccedinns mlr. pptps//t7/oiiggnsslll... html
[14] Max Jaderberg, Volodymyr Mnih. Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary
tasks, 2017.
[15] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017.
[16] Mete Kemertas and Tristan Ty AumentadooArmsttrong Towards robust bisimulation metric learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. httpaaa///ppeerreeeen http///pereiiee.eeeeefforrreeree net/forum? id=ySFGiFjgIEN
[17] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised rep- resentations for reinforcement learning. In Hal Daumé III and Aarti Singh, editors, Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 Pro- ceedings of Machine Learning Research, pages 5639-5650. PMLR, 13-18 Jul 2020. URL http://proceedingg mlr wttppp////11//assiiia22aa html
[18] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 19884- 19895. Curran Associates, Inc., 2020. URI p
pdf
[19] Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. In International Conference Learning Representations, 2020. URL http:::/operreeeiiiiii httppparreveee.eetfouruuraaaaaaa
[20] Kuang-Hung Hueni Lee, Ian Fischer, Anthony Liu, Yijie Guo. Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11890-11901. Curran As- sociates, Inc., 2020. URL https::/prrocsddii neurips ctcppps.cccppeee cttpcc////////.. wttpppppe//22//22/.... pdff
[21] Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-But E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL ptattts://proceeeee mewpeeedd///nerrrrpeeecccc/ppp/p #iile/09900ad84f631118888 200.aa2886....822222220000006631 Paperr pdf
[22] Szymon Eukaszyk. A new concept of probability metric and its applications in approximation of scattered data sets. Computational Mechanics, 33(4):299- -044, 2004.
[23] Adam Stooke, Kimin Lee, Pieter Abbecl, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Marina Mcila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9870-9879, PMLR, 18 24 Jul 2021. URL /RRLhttts://pprceeddig mlr. wwt.press/vile wttps///v33/stttoooseoo htmm
[24] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite, 2018
[25] Niklas Wahlström, Thomas Schön, and Mare Peter Deisenroth. From pixels to torques: Policy learning with deep dynamical models, 2015.
12

---

[26] Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamies model for control from raw images. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. DRL.htttss:/proceedie Paper pdf
[27] Denis Yarats, Rob Fergus. Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved duga-nneeedted reinforcement learning. CoRR, abs/2107.09645, 2021. URL
https://arxis org/ abs/2107 09645
[28] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. URL htttpss:/oopereeeeeett www.:/eee/ooummmddtteemmm 6aTvaff
[29] Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Leaming invariant representations for reinforcement learning without reconstruction. In Inter- national Conference on Learning Representations, 2021. URL http:::/oppnnreviee..eee
Corum/id=-2FCaDXRR
[30] Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. 12293, 2020.
13


Reviews:
[{"review_id": "fN0MrjHKX5r", "reviewer": "Reviewer_EWpZ", "summary": "This paper proposes a representation learning approach based on behavioral metric. The core idea is to project states into an embedding space such that the distance between them w.r.t behavioral similar is preserved. Therefore any distractors or variability in the environments could be potentially soaked due to the flexibility in the loss function. This is contrary to L2 reconstruction based objectives as auxiliary tasks which will learn to capture task irrelevant details in each variable environmental condition. \n\nThe core technical idea is to formulate the behavioral metric based approach as a new loss function. This new loss function is added as an auxiliary loss within an actor critic framework. In order to make the loss function tractable, two core assumptions are made: (1) reward expectations in the loss function of the behavioral metric are simplified to absolute errors in the rewards between two sampled states and (2) a new distance metric (RAP) is proposed, which leverages expectations over future states under the policy instead of expensive wasserstein based metrics over the transition model. \n\nThe approach is validated on a variety of continuous control tasks. The main results is that the expected returns is higher with this method over baselines.", "questions": "- Figure 3 and 4 have mixed results. It might be helpful to run on a more diverse variety of tasks to learn about strengths and weaknesses\n\n- It might be helpful to analyze and show state similarities in a navigation or some 2D projected view of an environment to better understand behaviorally induced state similarities. Currently it is all hidden under a single empirical performance metric and it is very difficult to understand what is happening. \n\n- \"we can learn a neural network approximator to estimate it by assuming that the reward rs on state s is Gaussian distributed.\". This seems like a major assumption for many environments with discrete reward functions. What type of environments and problems would break under these assumptions? \n\n- \"Similarly, in order to estimate the expected next states... we learn a dynamics model Pb taking input of state ... outputs a Gaussian distribution over the next state embedding\". Is this assumption necessary? Can a neural network be used to estimate the successor representation trained via TD learning?\n", "limitations": "Discussed above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis method uses behavioral metrics for representation learning which can be more robust to environmental variations. It is an interesting and well grounded approach to handle all sorts of variabilities, especially visual distractors. Instead of hand crafting augmentations, this approach could in principle learn more robust policies under arbitrary pixel variations without apriori knowing the set of invariances and equivariances in the environment. \n\nSo it might be a better approach where practitioners don't have good intuitions about the domain and therefore making it hard to build good data augmentations.\n\nWeakness:\n1. \"Our method achieves comparable scores with data augmentation method DrQ, and its score is higher than behavioral metric DBC and MICo.\" -> This empirical finding suggests to me that the authors should try this method on domains where it is very difficult to design data augmentations manually. \n\n2. While the empirical performance is shown to be better on some environments, it is not clear why it works so well in some cases and not in others. There is no qualitative analysis so it is difficult to understand what this method is really learning.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Loj8muFprwf", "reviewer": "Reviewer_BWMd", "summary": "The paper proposes a novel state distance metric to regularize the latent space of model-free reinforcement learning agents. The proposed RAP metric accounts for the variance of the rewards received in a pair of states under a stochastic policy. This enables the metric to be an upper bound on the difference between the values of a pair of states. The metric is added as a latent space regularizer to the soft actor-critic and evaluated in multiple high-frequency continuous-control environments.", "questions": "1. Is it straightforward to implement the RAP metric compared to MICo? Could you include a code snippet to compare the implementation of the two metrics?\n2. I would like to see an ablation where the correction for the variance of the rewards in states s_i and s_j is removed. Would the ablation be similar / identical to MICo?", "limitations": "Limitations were not addressed. I would like to see a discussion of using the RAP metric with agents other than the soft actor-critic.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper contains a rigorous analysis of previously proposed behavioral metrics to motivate the proposed RAP metric.\n* Excellent evaluation in multiple continuous control environments. Strong results in favor of the proposed metric in all domains.\n\nWeaknesses:\n* The derivation and interpretation of some equations could be made clearer.\n* No analysis of the learned models. E.g. a visualization of the learned latent space should be included if the main contribution of this paper is a regularizer on the latent space.\n\nComments:\n\n– It is not completely clear how we went from equation 12 to 13. Equation 13 should be described in more than one sentence.\n\n– I believe the term \\hat{\\sigma(s_i)} is dependent on the policy \\pi. Equation 13 might be misleading if the policy is not included in the notation.\n\n– Equations 11 and 12 should be cleaned up. Using colors or other annotations to match the text and the terms in the equation would be helpful. \n\n– Line 193: bad spacing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AqENIcVIEJo", "reviewer": "Reviewer_nvyx", "summary": "This paper is about behavioral metrics and state representation. This paper proposes a behavioral distance, the RAP distance, and develops a practical representation learning algorithm on top of it. Experiments on DeepMind Control Suite with distraction, Robosuite, and autonomous driving simulator CARLA demonstrate promising results.", "questions": "In Equation 9, d is defined as a distance from a space of distance functions. \n- Can you confirm that some distance functions d wouldn't make a lot of sense? \n- Theorem 5.2 and theorem 5.3 uses the l_\\infty loss but then in Section 5.2, a different distance function is used. How are the theorems useful for the practical algorithms?\n", "limitations": "As acknowledged by the authors, the limitations of the work are not discussed: \"Did you describe the limitations of your work? [No]\".\n\nIn addition to the lack of connexion between the theorems and the actual algorithm due to the metric d, an important characteristic/potential limitation of the RAP distance is that it depends on the policy. The explicit dependance on the policy is removed in Equation 13, why? How do you train the algorithm in practice given that samples might not be all obtained with one given known policy?", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Post rebuttal: I increase my score to 6 (see discussion below)\n\n---\nStrengths:\n- The results seem to be promising\n- The paper is globally well written.\n\nWeaknesses:\n- Contributions: the method and the algorithm do not seem very clear because there are large discrepancies between the few theoretical results and the actual algorithm (see questions and limitations). The claim in the abstract \"We provide theoretical analysis on the proposed algorithm.\" does not seem very accurate.\n- Reproducibility: Concerning the code, is it proprietary or do you want to publish it? It seems that at least one of these two claims is wrong (1) \"Did you include the license to the code and datasets? [No] The code and the data are proprietary.\" and (2) \"Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] We will publish code once paper is accepted.\"\n- Presentation: The approach is only presented starting from page 6 and it is basically presented in two pages. It would be useful to go more directly and more in depth into the actual contributions of the paper.\n- (minor) Typos/grammar: \"we randomly sample a weather at each episode starts.\", \"Deep RL algorithms and attracts more and more attraction in the RL community\".instead of \"expensive\" experiments, I suppose you meant \"extensive\"? (even though both might be true).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0oArMb_jusH", "reviewer": "Reviewer_TAKM", "summary": "In this paper, the authors focus on the behavior metric design problem so as to learn an informative and robust representation. To this end, three theoretical issues of the current behavior metrics are analyzed, including loss function mismatch, relaxation of dynamics model divergence, and limitation of the L1/L2 norm. Then a new metric named Reducing Approximation Gap (RAP) distance is proposed, and a practical representation learning algorithm is proposed based on RAP. Empirical results on three benchmarks show the effectiveness of the proposed method.", "questions": "1. What is the tightness of RAP bound in Eq. 10 compared with $\\pi$-bisimulation metric [1] and MICo [2]? Can RAP bring a tighter bound?\n2. Why the performance curves of baselines like DrQ in Figure 3 are lower than their original papers'? For example, DrQ can get 800 score in the original DrQ paper (Figure 3).\n3. More visualization of the latent space property of RAP over $\\pi$-bisimulation and MICo is preferred.\n\n[1] Castro P S. Scalable methods for computing state similarity in deterministic markov decision processes[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(06): 10069-10076.\n\n[2] Castro P S, Kastner T, Panangaden P, et al. MICo: Improved representations via sampling-based state similarity for Markov decision processes[J]. Advances in Neural Information Processing Systems, 2021, 34: 30113-30126.", "limitations": "No.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**originality**\n\nThe main contributions of this paper are clear. The authors carefully analyze the theoretical issue of current behavior metrics. Motivated by the above issues, a novel behavior metric for reducing the approximation gap from the perspective of the dynamics model. A thorough theoretical proof is given for the value function difference bound.\n\n**quality**\n\nThe empirical results on three different domains, i.e. DM Control, Robot control, and autonomous driving are given. The performance curves validate the effectiveness of the proposed method. But For some environments, e.g. cheetah_run in DMC, the baseline curves are not consistent with the original paper.\n\n**clarity**\n\nThe paper is well-written and easy to follow. Although this is a theoretical paper, the authors give a very clear background and the main problems of the current behavior metrics. These details help to understand what is the main contribution of the proposed RAP and why it gives a better theoretical property. But for the experimental part, more visualization of the learned latent space property will help to understand what is the difference between RAP over $\\pi$-bisimulation metric and MICo.\n\n**significance**\n\nThe representation learning plays an essential way in reinforcement learning (RL) for achieving generalizable and robust capacity. The behavior metrics-based research has drawn large attention in the RL community. The proposed method not only gives the theoretical shortcomings of the previous metrics but also provides a new metric with the better theoretical property.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept