Towards A Unified Policy Abstraction Theory and Representation Learning Approach in Markov Decision Processes
Authors: Min Zhang
Hongyao Tang
Jianye HAO
YAN ZHENG
Venue: Submitted to ICLR 2023 (2023)

Abstract: Lying on the heart of intelligent decision-making systems, how policy is represented and optimized is a fundamental problem. The root challenge in this problem is the large scale and the high complexity of policy space, which exacerbates the difficulty of policy learning especially in real-world scenarios. Towards a desirable surrogate policy space, recently policy representation in a low-dimensional latent space has shown its potential in improving both the evaluation and optimization of policy. The key question involved in these studies is by what criterion we should abstract the policy space for desired compression and generalization. However, both the theory on policy abstraction and the methodology on policy representation learning are less studied in the literature. In this work, we make very first efforts to fill up the vacancy. First, we propose a unified policy abstraction theory, containing three types of policy abstraction associated to policy features at different levels. Then, we generalize them to three policy metrics that quantify the distance (i.e., similarity) of policies, for more convenient use in learning policy representation. Further, we propose a policy representation learning approach based on deep metric learning.  For the empirical study, we investigate the efficacy of the proposed policy metrics and representations, in characterizing policy difference and conveying policy generalization respectively. Our experiments are conducted in both policy optimization and evaluation problems, containing trust-region policy optimization (TRPO), diversity-guided evolution strategy (DGES) and off-policy evaluation (OPE). Somewhat naturally, the experimental results indicate that there is no a universally optimal abstraction for all downstream learning problems; while the influence-irrelevance policy abstraction can be a generally preferred choice.

Under review as a conference paper at ICLR 2023
TOWARDS A UNIFIED POLICY ABSTRACTION THE- ORY AND REPRESENTATION LEARNING APPROACH IN MARKOV DECISION PROCESSES
Anonymous authors Paper under double-blind review
ABSTRACT
Lying on the heart of intelligent decision- systems, how policy is represented and optimized is fundamental problem. The root challenge in this problem is the large scale and the high complexity of policy space, which exacerbates the difficulty of policy learning especially in real-world scenarios. Towards a desirable surrogate policy space, recently policy representation in low-dimensional latent space has shown its potential in improving both the evaluation and optimization policy. The key question involved in these studies is by what criterion we should abstract the policy space for desired compression and generalization. However, both the theory on policy abstraction and the methodology on policy representation learning are less studied in the literature. In this work, we make very first efforts fill up the vacancy. First, we propose a unified policy abstraction theory, containing three types of policy abstraction associated to policy features at different levels. Then, we generalize them to three policy metries that quantify the distance (i.e., similarity) of policies, for more convenient use in learning policy representation. Further, propose policy representation learning approach based on deep metric learning. For the empirical study, we investigate the efficacy of the proposed policy metrics and representations, in characterizing policy difference and conveying policy generalization respectively. Our experiments are conducted in both policy optimization and evaluation problems, containing trust-region policy optimization (TRPO), enversity-guidee evolution strategy (DGES) and off off-policy evaluation (OPE). Somewhat naturally, the experimental results indicate that there is no a universally optimal abstraction for all downstream learning problems; while the eeslueece-irrelevarr policy abstraction can be a generally preferred choice.
INTRODUCTION
How to obtain the optimal policy is the ultimate problem in many decision-makinng systems, such as Game Playing (Mnih et al. 2015), Robotics Manipulation (Smith al. 2019), Medicine Discovery (Schreck et al.) 2019). Policy, the central notion in the aforementioned problem, defines the agent's behavior under specific circumstances. Towards solving the problem, a lot of works carry out studies on policy with different focal points, how how policy can be well represented (Ma al., [0200] Urain jet al. 2020), how to optimize policy (Schulman et al. 2017a| Ho & Ermon (016) and how to analyze and understand agents' behaviors (Zheng et al. 2018 Hansen & Ostermeier (001).
The root challenge to the studies on policy is the large scale and the high complexity of policy space, especially in real-world scenarios. As a consequence, the difficulty of policy learning is escalated severely. Intuitively and naturally, such issues can be significantly alleviated if we have an ideal surrogate policy space, which are compact in scale while keep the key features of policy space. Related to this direction, low-dimensional latent representation of policy plays important role Reinforcement Learning (RL) (Tang et al.) 2020). Opponent Modeling (Grover et al. 2018).
Fast Adaptation (Raileanu et al.) 2020 Sang et al. (2022. Behavioral Characterization (Kanervisto al., (0200) and etc. In these domains, few preliminary attempts have been made in devising different policy representations. Most policy representations introduced in prior works resort to encapsulating the information of policy distribution under interest states (Harb et al. 2020) Pacchiano (et al. (202)). e.g.. learning policy embedding by encoding policy's state-action pairs (or trajectories) and optimizing policy recovery objective (Grover et al., 2018 Raileanu et al. 2020). Rather than policy distribution, some other works resort to the information of policy's 's influence the

---

Under review as a conference paper at ICLR 2023
environment, e.g., state(-action) visitation distribution induced by the policy (Kanervisto et al. 2020 Mutti et al. 2021). Recently, [Tang et al. (2020) offers several methods to learn policy representation through policy contrast or recovery from both policy network parameters and interaction experiences. Put shortly, the key question of policy representation learning is by what criterion we should abstract the policy space for desired compression and generalization. Unfortunately, both a unified theory on policy abstraction and systematic methodology on policy representation are currently missing.
In this paper, we make first efforts to fill up the plank in both the theory and methodology. In- spired by the state abstraction theory (Li et al.] (0066), first we introduce a unified theory of policy abstraction. We start from proposing three types of policy abstraction: Eistribution-irelleaann irreeeeancc, and value-irreleve irrtleenrreee.cce They follow different abstraction criteria, each of which concerns distinct features of policy. Concretely, we utilize the exact equívalence relations between policies and derive the corresponding policy abstractions. Further, we generalize the exact equivalence relations to policy metrics, allowing quantitatively measure the distance (i.e., similarity) between policies. Such policy metrics are more informative than the binary outcomes of policy equivalence and thus provide more usefulness in policy representation learning. Moreover, towards applying practical policy representation in downstream learning problems, we introduce policy representation learning approach based on deep metric learning (Kaya & Bilge 2019). We propose an alignment loss for a unified objective function of learning with different policy metries. The policy representation is learned to render the abstraction criterion through minimizing the difference between the distance of policy embeddings and the quantity measure by the policy metrics. In particular, we use Maximum Mean Discrepancy (Gretton et al. 20112] Nguyen-Tare et al. (02T) for efficient empirical estimation of the policy metries, and we adopt Layer-wise Permutation-inss invariiat Encoder (Tang et al. 2020) for structure-aware encoding of the parameters of policy network.
In addition to the theoretical understanding of policy abstraction, further investigate the empirical efficacy of different policy metries and representations in characterizing policy difference and convey- ing policy generalization respectively. We conduct experiments in both policy optimization and policy evaluation problems. For policy optimization, we adopt Trust-Region Policy Optimization (TRPO) and Diversity- Guided Evolution Strategy (DGES) as the problem settings from (Kanervisto et al. 2020), covering both gradient-based and gradient-free ffreentttteeeeeeee policy optimization. For policy evaluation, consider Off-policy Evaluation (OPE). In particular, we establish a series of OPE settings with differ- ent configurations of training data and generalization tasks. These settings reflect the circumstances often encountered in RL. Our experimental results indicate that, somewhat naturally, there is no a universally optimal abstraction for all downstream learning problems. Additionally, it turns out that the influence-ing iireeeeanceeeeeeeeeee abstraction can be preferred choice in general cases.
Our main contributions are summarized as follows: 1) problem and to our knowledge, we propose unified theory of policy abstraction along with several policy metrics for the first time. 2) We propose unified policy representation learning approach based on deep metric learning. 3) We empirically evaluate the efficacy of our proposed policy representations in multiple fundamental problems (i.e., TRPO, DGES and OPE).
We focus on the general policy abstraction
2 BACKGROUND
Reinforcement Learning We consider a Markov Decision Process (MDP) (Puterman 2014) typically defined by a five-tuple (S, A. R.).). with the state space S, the action space A, the transition probability P: S: - ASS), the reward function R : S + R and the discount factor y [0, 1). A(X) denotes the probability distribution over X. A stationary policy म S A( (A) is a mapping from states to action distributions, which defines how to behave under specific states. An agent interacts with the MDP at discrete timesteps by its policy n, generating trajectories with 80 ~ po(.), de ~ (T(1)), 8141- ~ P( 8t, at) and r. = R (st, at), where Po is the initial state distribution. We use pM (s'(ss))= = Ea--(-ls) P(s'|s,, a) to denote the distribution of next state s' when performing policy π at state s. For a policy π, the return G, = Σe r'r is the random variable for the sum of discounted rewards while following ग., whose distribution is denoted by ZT, The value function of policy т defines the expected return for state s, i.e., V V*((s) = E, G. Su = s]. The goal of an RL agent is to learn optimal policy x' that maximizes J (TT) = ( [V* (so)).
Metric Learning Here we recall the standard definition of metrics which is central to our work.
Definition (Metrics (Royden 1968 ). Let X be a non-empty set of data elements and a metric is a real-valueds function d: X x X +> [0. oo) such that for all xx J. E X: 1) d(xr, y) =0 = x. = y: 2) d(x,y) y) d(y, x); 3) d(x, v) S d(xx2) 2) + (z. y). A pseudo-metric d is a metric with the first condition replaced by x = y d(x,y) = 0. The combination (X, d) is called a metric space.
2

---

Under review as a conference paper at ICLR 2023
A metric d is often used to quantify the distance between two data elements in general sense. In this paper, we will also use metric to stand for pseudo-metric for brevity. Typically metric learning aims to reduce the distance between similar data and increase the distance between dissimilar data. With nonlinear transformation offered by deep neural networks, Deep Metric Learning allows to find such optimal metrics by optimizing a latent representation space of raw data.
POLICY ABSTRACTION THEORY
Inspired by the state abstraction theory (Li et al., 2006), in this section, we make the first effort in proposing a unified policy abstraction theory. First, we propose the formal definition of three types for policy abstraction; then, we generalize the abstractions to three types of policy metrics. Finally, we analyze the properties of policy abstraction and compare them in several Gridworld MDPs.
3.1 POLICY ABSTRACTION
First of all, following the classic definition of an abstraction (Giunchiglia & Walsh 1992), we propose a general definition of policy abstraction as follows:
Definition 2 (Policy Abstraction). A policy abstraction f ; II + X, is a mapping from ground policy space TI to an abstract space X. ((TT) E X is the abstract policy representation corresponding to ground policy
II, and the inverse image f fx)() with X E X, is the set ground policies that correspond to X under abstraction function! f.
It is apparent that there are many such abstractions since we may have many possible ways to partition the policy space. However, we are only interested in some useful ones among them that follow specific abstraction criteria to preserve the important features related to decision making. In this paper, we present three types of policy abstraction which are defined below:
Definition 3. Given an MDP and a ground policy space 11, for any two policies T&, TT, E 11, we define three types of policy abstraction follows:
1. A distribution-irrelevaa f (1²)) implies that {a s) TTy(a | s)-
abstraction (f=) is such that for all 8 E S, a E A, (x;) =
2. An influence-irrelevanee abstruction (fpx) is such that for all 8, E E. (mx) J (Tz) implies that p= (s' s) = =P (s' s).
3. A value-irrele irreeeerreeeeecc abstraction (fv-) is such that for all s E S. fv- (T.) = fv" (Ty) implies that V (s) V* (s).
These abstractions aggregate policies based on the corresponding equivalence relations with respective concerns on different features of policy. Intuitively, the Eestribution-irrelvaaee abstraction (fx) preserves the action distribution of the policy; the influence-ireelvance abstraction (fp=) ) preserves the state transition distribution induced by the policy, i.e., the influence caused by the policy on the environment; and irrelevncee abstraction (Jv-) preserves the value function of the policy. In addition to the policy abstractions introduced in Definition we provide some other ones in Appendix A.2 Moreover, we revisit the policy abstractions adopted in prior related works and summarize them from the angle our policy abstraction theory Table of Appendix B
3.2 POLICY METRICS
The policy abstractions allow us to aggregate policies according to equivalence relation. However, exact equivalence is rarely encountered in continuous policy space (e.g., the usual case with neural policies), thus useful abstraction can be seldom obtained. Moreover, the equivalence relation offers only qualitative (i.e., binary) outcomes and is incapable of measuring the similarity between policies, which is significant to policy representation learning. To this end, we generalize the policy abstractions to policy metrics which quantitatively measures the distance between two policies.
Corresponding to the three types of policy abstraction, we define the following three policy metrics: Definition& 4. Given an MDP, a ground policy space II, a state distribution p(s) and distribution joseric--mmtricc (. for any two policies ni, J II, we define three policy metrics as follows:
1. A distribution-iirelevaae metric: dz (T., nT) Eemp(s) [D (TT(a s),z,(( |s]]]
2. An inffuence-irrelevane metric: dp- (Te, TT) = E [D (P (5' 8), PT (s' s))]
3. A value-irrelevance metric: dys (T TT) Essp(s) 1D (Z (
3

---

Under review as a conference paper at ICLR 2023
Table 1: Properties of different policy abstraction.
Abstraction
Abstraction Criterion (for Tı, m2, Vs, 8', E 5 x A)
Fineness
Task Relevance
je
Policy Parameter Equivalence (8, (22)
Highest
None
Action Distribution Equivalence (T, (a 8) TTLa s))
High
Low
fpr
Dynamies Influence Equivalenee (P% (s' (5'(s
pd (s'(s)
Middle
Middlc
Syr Value Function Equivalence (V*) (8) (*))
Low
High
J0
Triviality (taking all policies as the same)
Lowest
None
These metrics follow the same abstraction criteria as in Definition B] i.e. the irrelevance regarding action distribution, influence and value, measuring the similarity of policies by the distance at respective levels. Compared the binary outcomes offered by the equivalence relations, the metrics defined here are continuous, thus are more informative in comparing and representing policies in finer views. Specially, one may see that the equivalence relations used in Definition 3 induce corresponding discrete pseudo metrics, e.g., dx (TT πT) = if (a;) fo (Tyy), and otherwise. Notice the metrics proposed above depends on the distribution metric D and state distribution p(s)- For D, typical choices can be Jeffreys Divergence [Jeffreys 1946) and Maximum Mean Discrepancy (MMD) [gguyen- -Tan et al.J 2021]. For p(s). intuitively, it should be the distribution of states we are interested in when comparing two policies. We defer the concrete choices for practical implementation of these metries in Section
3.3 PROPERTIES OF THE ABSTRACTIONS
Superficially, the three abstractions proposed preserve features that are progressively more relevant to decision making in the learning task, but essentially, what is the relationship between the three abstractions? To investigate the problem, we define the fineness of policy abstractions similar to the one for state abstractions used in (Li et al. (2066) to prove how the three abstractions are related.
Definition 5 (Abstraction Fineness). Let Fn denotes the set of abstractions on ground policy space II. Suppose S2 E FII We say S1 is finer than 12, denoted J1 Z Jz. ifʃ mi, Te E II, (((T)) = (TT) implies fe(mL) = =T) If, fi + f2. then fi is strictly finer than fz. denoted J f2. In contrast, we may also say f2 is (strictly) coarser than f1, denoted f22 I f1 (52 < f1)-
is casy to see the relation 2 satisfies self-reflexivity, antisymmetry and transitivity, thus it is a partial ordering. Consider the set of possible policy abstractions, while the coarsest abstraction (fo) the trivial representation where all policies are treated as the same; while the finest abstraction is the identity representation, e.g., fe( = 0 for a policy neural network parameterized with 0 e O. With the partial ordering Z, we further derive the following theory.
Theorem 3.1. Under the Definition 3 and 5, if the reward function R depends only on state E S, we have (fe =)f+ 2 fP= ž fv= (2=(z// fo).
The proof is provided in Appendix A.T along with more discussion on other cases of the reward function. The theorem declares how the three policy abstractions are related to each other in the sense of abstraction fineness with the two extreme cases (ʃe, fo) for reference. The coarser the abstraction is, the more the original policy space is abstracted.
In Table I we summarize the properties of different policy abstractions, regarding abstraction criteria, abstraction fineness and task relevance. The major conclusion is that there is inverse relation between abstraction fineness and task relevance. Except for the two extreme cases (fe. fo) that are totally task- indeiindeeendntt, the policy abstraction becomes more task-relevant as the abstraction criterion concerns more policy features related to the learning task. The cetrriuttion-irrlevaaa abstraction fr concerns the policy behavior in the learning task, defined by action distribution at interested states; meanwhile f= is coarser than fe since the same policy behavior can be realized by non-unique policy parameter. Taking one step closer to the task, the celluence-irreleevn frr cares about the state transition dynamics induced by policy behavior. Obviously, fr is coarser than fa different behaviors may induce the same transition distribution. The value-irrelevance abstraction fv- further involves the rewards of long-term dynamics, thus is the most rele-atl and coarsest among the three types of policy abstraction.
abstraction
To compare these policy ab- stractions in a quantitative view, we demonstrate how the distances of two policies measured by the corresponding policy metrics differ in several Gridworld MDPs We use Distinct Policies, Doorway from Kanervisto al. 2020) and design new task, Key Action for simple prototypes of tasks with
Empirical Comparison of Policy Metrics in Gridworld MDPs
4

---

Under review as a conference paper at ICLR 2023
Deject seldie
Docnway
Eeyaction
Inter-policy distance
0.20
Doerway
0.10
0.10
0.05
0.00
100
150
103
Prob. random action (%)
Figure I: Policy comparison with different policy metrics Gridworld. Top Panel: The illustration of three Gridworld MDPs and two deterministic policies (blue and green). Bottom Panel: The distance curves of the two policies measured by de.dpr.dl (y-axi), against the stochasticity the environment (r-axi). dp- is able to distinguish the two policies across all the settings.
increase the stochasticity of the environment for better evaluation. In particular, Pg--p(() a) :::)) :) is calculated by average the absolute differences over all states. The illustrations and results are shown in Fig. ! while more results can be found in Appendix
different features; morcover,
We observe that the distribution-iirelevaaar and outcome between the two policies, e.g., in Doorway. This is because d, measures the difference in the action distribution itself, i.e., independent of the dynamics as well as the increasing stochasticity of the environment. This issue may be resolved by using a designated distribution that concentrates over key states. Conversely, the value-irrelevance dy- measures the difference in the outcomes of the two policies regardless their differences in action distribution and dynamics, e.g.. in Distinet Policies. Another discovery that dy= quickly degenerates and turns to be not informative as the increase of stochasticity, showing its poor robustness. By contrast, the anfluence-irreleannee metric dp- is sweet intermediate point, consistently keeping the ability of distinguishing the two policies across all the environments and stochasticity configurations. In a summary, different policy abstractions and metrics may yield different outcomes for the same two policies and the optimality depends the specific downstream learning problem concerned. Later, we evaluate these options representative downstream RL problems in Section and for useful insights.
metric dx may fail to show the difference in dynamics
4 POLICY REPRESENTATION LEARNING APPROACH
The next question concerned in practice how can we learn the representation of RL policies (usually modeled by NNs) in general way? Based on the policy metrics introduced above, we propose a policy representation learning approach by following the principle of Deep Metric Learning.
4.1 LEARNING POLICY REPRESENTATION BY EMBEDDING ALIGNMENT
The policy metrics proposed in previous section measure the quantitative relationship between policies from difffrentt perspectives of the policy abstraction criteria. For unified objective function of learning from different policy metrics, we use the alignment loss, with which the difference between the distances of two policies in the representation space and in the policy metric space is minimized. Concretely, consider a policy representation function fø. and the alignment loss can be formalized as,
(1) where we consider d, E {dn, dp., dy- } and r is the weight for scaling. Similar forms of the alignment loss are also adopted in the studies on state representation learning (Zhang et al. 2020).
(AL(3) En.x'en (() fx
nd. (M, (m,
As we can see, CAL(U) (1) consists of two metrics, i.e., the L2 distance function (1I-112) of two inputs and the policy metric (d, (..-)) Intuitively, minimizing the alignment loss is to align the two metrics by optimizing the policy representation function fv- By this means, policy representation functions, which maps the ground policy æ E II to the latent embedding Xx (T) E X. The embedding preserves the policy features corresponding to the abstraction criteria reflected by the specific policy metric considered. For a practical implementation. the following problems are the estimation of policy metries d,, and the realization of the training for policy representation function fe. which detailed in the next two subsections respectively.
able learn different
5

---

Under review as a conference paper at ICLR 2023
In the literature of learning policy representation, representative methods follow the principle of behavior recovering (Grover et al., 2018) and policy contrast (Tang et al. 2020). To our knowledge. none of prior works take a systematic view of policy abstraction. In Table 4 we show that prior methods are specific instances of one of our proposed policy abstractions that differ in realization.
4.2 ESTIMATING POLICY METRICS VIA MAXIMUM MEAN DISCREPANCY
Given a tractable metric D and a state distribution p. the policy metrics (i.e., dz. dpr,, dy=) can be calculated exactly if the probability distributions (i.e., n, P". Z²) are available. However, this is usually infeasible in practice: instead, in more regular cases, only finite samples of policy interaction are available. Although the empirical distributions can be estimated in simple MDPs where the state-action space is finite (as in Appendix D), unfortunately, approximating the distributions and computing the metrics are non- arivillvial, especially with high-dimensional continuous state-action space.
Therefore, we estimate the policy metrics directly from the samples, bypassing estimating the empirical distributions (i.e., म. pT ) In particular, adopt MMD (Gretton et al. 2012 Nguyen Tang et al.) [ as the distribution metric, i.c., let D be DMMD. MMD measures the maximum value of the mean discrepancy of two distributions regarding all possible functions in a predefined
family. Conventionally, let the class of functions h X IR be a unit ball in a Reproducing Kernel Hilbert Space (RKHS) H associated with a continuous kernel k<(:.) on X, p, q be two distribution defined on X, x. x'' and y, be i.i.d. samples from p and g respectively, the MMD is defined as:
D DMMD (p. q; H) = sup
(Eayp [/a (x)] - Byng [h (y))) = 11pp - Hallx
NEH:||MII
(2)
= (E [R( x')] + Eyy [R(y. v/)] 2E y()²,
where /4p = Jx k(x,, (p(()) is the mean embedding of p into H/Smola et al. can be empirically estimated with samples {a: and {ui)-m ~ q:
2007). Thus, MMD
RAMO ((r:), (u:); = N2
Σ M(),,1))+ + M 1
Σ
k(yC U3) 1.J
2
Σ
k(r,,,5))
(3)
NM
According to Eq. 3] we can estimate the policy metrics dy, dp= , dy empirically from the samples {ai), (6(). {G.} of different policies respectively, under the sampled states {si) for the expectation Ep(s)- However, it is often impractical to obtain multiple samples under the same state. Thus, we
resort to estimating the surrogates, e.g., dp= (T;, rT)) D (s, s'), (s ss)))) for dps, where the joint distributions rather than the state-conditioned distributions are measured. We use Gaussian kernel by default, i.e., xx) = exp
||x 11 Consequently, the empirical estimates of the
policy metrics serve as the self-supervision in Eq.
4.3 REALIZING THE TRAINING OF POLICY REPRESENTATION FUNCTION
With the empirical policy metrics provided in previous section, the training of policy representation is straightforward with a differentiable function fo by optimizing the aligmment loss (Eq. [T. The realization of policy representation function concerns two aspects: 1) the choice of policy data (or original representation) and 2) the construction of f (i.e., how policy data encoded).
For the first aspect, we focus on parameterized policy no (typically by a neural network) and use policy parameter 0 as the policy data. One may recall that 0 itself can be viewed the finest representation obtained by policy abstraction fe in Table Such an original representation (i.e., 8) is high-dimensional and highly nonlinear, offering no help in the compression and generalization of policy space. In addition, we are aware that in some cases the policy parameters may be not available, and thus the interaction experiences generated by the policy can be alternative policy data, as used in (Grover et al. 20188 8 Tang et al.) 2020). Our policy representation learning approach is compatible with such alternatives with the need of possible slight modifications, For the second aspect, we adopt Layer-wise Permutation-in invariant Encoder (LPE) [Tag et al.) 2020) as the implementation choice of I (see Fig. 5. which has demonstrated the effectiveness in encoding conventional policy networks. To be specific, for the parameter {W,, be 1 of policy æ, i.e., the weights and biases of k-layer MLP the weight W; Rl and bias b, E IRIxx 1x14+ (1: is the unit number of the é-layer; and
'The activation function not considered since the structure is fixed for policies in convention RL setting. In principle, I.F. can be generalized tailor other advanced network structure.

---

Under review as a conference paper at ICLR 2023
l are for the input and output layers) are concatenated (e) and transposed, followed by MLP (Fe,s) and a mean-reduce operation (MR), resulting in layer embedding z;; Thereafter, the policy embedding is obtained by concatenating the embedding of each layer. Formally,
4+1 Σ
24 MR (IW @ b.l)))
fas ((IW, bs)))))) Xmo (v(
(4)
==
Each row of [W; @ by]T, indexing by the subscript j. describes a transformation of the i-layer into the next layer. All the rows are fed into f separately and are then averaged into 2i. In a consequence, the policy embedding serves as the compact representation of the policy network by summarizing the transformations made by the each layer of it. The significant difference between LPE and straightforward MLP encoder is that, LPE provides structure-aware representation, i.e., both the intra-layer and inter-layer structures are explicitly considered. Intuitively, this alleviates the difficulty of learning representation from the policy network parameters. Other advanced encoder structures are beyond the scope of this work and we leave them as future work.
Till now, we can update the parameters of LPE & (wh by optimizing Eq. with the policy samples from some given set of policies and the empirical policy metrics estimated accordingly. Depending on the specific choice of policy metric, the policy representation is learned to render the policy abstraction in Table D starting from fe and going downwards to the corresponding level.
APPLYING POLICY ABSTRACTION TO POLICY OPTIMIZATION
Despite the theoretical understanding of the policy abstraction, we have no idea about how the derived policy metrics behave in different downstream learning problems. To shed some light on this, we evaluate the efficacy of the policy metrics proposed in Sec. 3.2 in policy optimization, including Trust-Region Policy Optimization (TRPO) and Diversity-Guider Evolutionary Strategy (DGES).
Trust-Region Policy Optimization We adopt TRPO problem as the first test stone for our pol- icy abstractions. Specifically, the objective of TRPO problem is to maximize the policy return while constraining the difference between old and new policies: (f) = E.,,,, [R(r)),
s.t., d. (Tg. Tema) S σ. where a is a threshold. For our experiments, we consider the policy metrics d. E {dzz, dp=, dy=) ). In another word, the learning agent checks if the difference measured by the policy metries are larger than for each policy update. In this experiment, the original TRPO (Schulman et al.] 2013) is generalized to incorporate different alternative metrics for the trust-region constraint. Thus, we can evaluate the efficacy of the different trust regions provided by our proposed policy metrics, shedding some light on what policy features we care the most in TRPÓ.
We adopt Gridworld envi- ronment where the agent can move to one of N directions at each grid and only one direc-
Averagy Average Return (1e2)
Average return
-7
tion yields high reward (Kan-
4.
ervisto et al. 2020). The re-
-8-
VenilaPP TRPO-1
TRMDH- TRPO-6-
sults are shown in Fig. 2(a)
DGES-6
DGES-6-
We observe that all our TRPO variants (i.e.. TRPO-f. out- perform Pa (i.e., no
0.2
0.6
1.0
-9
0.2
0.4
0.6
Timesteps
Timeste ps
(a) TRPO
(b) DGES
trust-region constraint used), demonstrating the effective- ness of our policy abstractions. Moreover, TRPO-fa outper- forms the others. This is be- cause fo follows the abstraction criterion regarding action distribution, thus pertains to the essence of TRPO. By contrast, fp- and fv- utilize coarser abstraction which does not hold the features of action distribution. In addition, we demonstrate the superiority of our policy abstractions when compared with existing related methods in Appendix E.
Figure 2: Performance of different policy abstractions in: (a) Trust- Region Policy Optimization (TRPO) for a Gridworld environment; and (b) DiversityGGuider Evolution Strategy (DGES) for Point environment. Results are the mean and half standard deviation (shaded) over 10 and trials for TRPO and DGES respectively.
Next, we adopt DGES problem the second test stone for our policy abstractions. Formally, the objective of DGES problem is to maximize the policy return of the current policy Na and maximize its policy difference to the ancestor policy म: (DGES(0) = E...P., [R(T)] + 3 En d. d= 7): where 3 0 is the weight. Similarly, we consider the policy
Diversity-Guidel Evolution Strategy
7

---

Under review as a conference paper at ICLR 2023
metrics d. {dr; (dy)) in different ways. We aim at exploring the diversity concerning which policy feature is the most effective in DGES.
}. Here, the choices of policy metries realize the population diversity
To explore this, we leverage the Point environment with deceptive rewards (Pacchiano et al. 2020).
The results are reported in Fig. 2(b) In comparison to Vanilla-ES (i.c.. | = 0), optimizing policy diversity (i.e., 3 > 0) based on our policy metrics (i.e., DGES-S. ) does help exploration and thus leads to better performance. In particular, DGES- performs the best. Since ES optimizes policy in gradient-frre fashion. the evolution process concerns only policy return. Therefore, the distribution-irellvaace abstraction f (i.e., the winner in the TRPO experiment) can be redundant since multiple action distributions may have the same outcome (i.e., influence and value). For the value-irrelevance abstraction fv=, it turns to be too fine to contain the features of policy behavior (i.e., action distribution and influence). Therefore, the inffuence-irreeeannce abstraction f fpr serves as a sweet point. Furthermore, we provide additional comparative evaluation in Appendix E.2
6 APPLYING POLICY ABSTRACTION TO OFF-POLICY EVALUATION
After the investigation in policy optimization, now we move to policy evaluation. Typical OPE Fu iet al., [021] Harb et al., 2020) focuses on using offline data to evaluate unseen policies. Likewise, are interested in studying the value generalization performance on unseen policies of the rep- resentations learned regarding different policy abstractions. The appealing characteristic of policy representation in value generalization has been studied in (Tang et al. 2020), where Policy-extended Value Function Approximator (PeVFA, ((=)) takes as input the policy representation Xx approx- imates the values of multiple policies and offers implicit value generalization among the policy representation space.
For policy data collection, we run PPO (Schulman et al. 201 in OpenAl Gym continuous control tasks: InveriedDDubbePPendllum
(IDP-v2) and Lunarl LammLLanderoomoo ander[Cotinuous--22 v2 (LLC-v2) (Brockman al. 2016). By collecting the policies at intervals during the learning process, we build an offline policy set, based on which we train our policy representations and a PeVFA V(x=). For concrete problem settings, we establish both weak and strong generalization OPE scenarios which differs at the difficulty of evaluating the unseen policies. For the weak generalization scenario (easy), we sample training data uniformly from the whole band of the policy set. For the strong generalization scenario (hard), we separate the policy set and use the low-performance policies for the training data, with the rest taken as the unseen policies to evaluate. For both the settings, the ratio of sampling and separation is set to be 20%. 40%, or 80%. We report the results of the ratio 20% in Table and leave the results of other ratios in Appendix [ For evaluation protocols, we report the evaluation (testing) error of unseen policies ( (T-error) and the generalization gap (G-gap), i.e., the difference between training and testing error. We denote different policy representations by their underlying policy abstraction (e.g.. fm)) correspondingly. Complete details can be found in Appendix
Weak Generalization Scenario in OPE First, we study the empirical comparison in the weak gen- eralization scenario. Table reports the results of value generalization for the policy representations learned based on corresponding policy abstractions. To be specific, the fe denotes directly using policy parameters 0 as policy representations (i.e., no representation training). For our proposed pol- icy abstractions f= fr=, , we learn the representations for them according to Eq. based on the LPE and MMD estimation (Sec. 4.3). To further complete the comparison, we include two additional representations fIæ and fEL: JAE uses a randomly initialized LPE with no further training while fEL uses the LPE trained by the end-to-end OPE loss (see Appendix F.2 respectively. Note that fEL can be viewed as a variant of fv- since it also learns from values but does not optimize the alignment loss. Besides, we also include the representation (fCl) learned by unsupervised contrastive learning based on InfoNCE loss (van den Oord al. 2018). as proposed in (Tang et al. 2020).
From the Table 2 (Weak Generalization). we can observe that fn, fv= outperforms fe. fRE- and SEI in both IDP-v2 and LLC-v2. This demonstrates the effectiveness and superiority of our proposed representations in value function approximation and generalization. fEL is significantly better than the fe and fRE. indicating the advantages of LPE structure and training. We can observe that contrastive policy representation fer performs poorly. We postulate that with less training data available at the 20% sampling ratio, the fCL with emphasis on policy instance-level comparison suffers from higher evaluation error and generalization gap. The superiority of fv- compared to SEI. from the Table dabeonddemssttttess the effectiveness of alignment loss. This is because although both fv=, fEL learn policy representation from the information of policy value, naive end-to-end training is less effective than alignment optimization which establishes the representation space based on the
8

---

Under review as a conference paper at ICLR 2023
Table 2:
Performance of different policy abstractions in Off-policy Evaluation (OPE). The minimum value for each task is highlighted. Results are the mean + a standard deviation over 10 and 5 trials (for weak and strong respectively). The fv- has lower T-error and G-gap both the generalization tasks.
Env
Absraction
Terrur
Week Generalization
G
Tetror
Strong Generalization
G Ggpap
fa
(10059) # 0.008
9,0059 : 0.0006
0,1592 = 0.01077
0.9778 = 0JN437
0,0056 1: 0.0009
0.0088 1 0.0010
0.1676 1: 0.0088
0.16:4 =: 0.0087
0.048 + 000033
0.0027
0.000
0.1783 0.00K0
0.1712 + 0.0145
IDP-22
0.0067 + 0.0010
0,046
0.0008
0.1567
0.0081
0.1491 = 0.0007
0.0044 1 0.0003
0.0025
0.0006
0.1812 1: 0,0013
0,1803 1: 0.0011
fyr
0.0044
0.0003
0.0044
0.0006
0.1789 0.0045
011778 0.0049
fy
01046
010003
0.0023
0.0095
0.1329 0.0093
0,1295
0.0114
Ja
UD006
0.1395 030257
01992
0.0028
0000077
0.0007
D.0729
0.0197
0,718
0.0196
1nt. 0.0017
0.0004
0.001 0.0004
0.0656
0.0088
0.066 1 0.0092
LLC-2
Jca 0.0035
0.0005
0.0033
0000
0:0589
0.0766
0.672 0.0188
0.0015
1: 0.0005
0.0013
1 0.0005
0.1365
0.0367
0.1318
= 0.0332
Fe"
0.0015
1. 0.0004
0.0013
0.0004
0,0905
0.0802
0.0900
1 0.0404
0.0014 0.0003
0.0011 0.0003
0,0473 0.0045
0.0470 0.0042
policy metrics. In general, the value generalization results among our abstractions fr f fv do not differ much. This is mainly because in the weak generalization setting, the unseen policies obey the same distribution the training policies, thus posing less difficulty of value generalization.
Strong Generalization Scenario in OPE Now we move to the study in the strong generalization scenario and similarly the results are reported in Table (Strong Generalization). Compared to the weak generalization scenario, the overall T-error and G-gap are significantly higher in the strong generalization scenario. This is reasonable because there is larger performance difference between the training and unseen policies. In other words, the unseen policies belong to out-of-distribtiion data. The fv- obtains the lowest evaluation error on the two environments, which indicates the value- irrelevance abstraction with higher task relevance may be best suited for the strong generalization setting. For the explanation, since the objective of OPE lies at the value function approximation and generalization, we consider that the irtelevceee principle of fv" is consistent to the objective and thus fits naturally.
With only low-performance policies for the training data at the 20% sampling ratio (hardest), the results of other abstractions including our proposed fx and fp- on the task are poor. The main reason may be that under the strong generalization setting, there a large data-shift between the training and unseen policies. fa and fp- fails to learn a policy abstraction with generalization ability in the absence of diversity policies. Nevertheless, from the Table9 TO as the sampling ratio increase and training policies become more diverse, the advantage of fx and fP- over other policy abstractions gradually emerge. Moreover, in the hardest case, fp= is better than fx. which shows that the iireleevaeeeeeeeaeeee policy abstraction may be general policy abstraction option.
For the other baselines. fe still shows few competition. For fRE, JEL, fCL, they falls behind fvv while slightly outperforms fr- and fn in Table 2 Such slight advantages no long holds in the settings of higher sampling ratios (i.e., 40% in Table 9 and 80% in Table (0). Unlike weak generalization scenario, fer is not so bad in strong generalization scenario. The main reason is that encountering hard policy evaluation tasks (Strong Generalization), other methods suffer from performance degradation and are no longer superior to contrastive learning. In contrast, contrastive learning based policy instance-level comparison maintains a relatively good result.
Other Experiments In addition to Table amount in Appendix G for both the weak and strong generalization scenarios. Other comprchensive
Z. we provide more results under different settings of data
studies (e.g., extrapolation behaviors, visualization) can found in Appendix G.2 G.3
CONCLUSION & LIMITATIONS
In this work, we introduce a unified policy abstraction theory, including three major types of policy abstraction, and corresponding policy metrics derived from the abstraction, as well as the analysis of their properties. We further propose a policy representation learning approach based on deep metric learning. We empirically evaluate the efficacy of different policy abstraction in both policy optimization (i.c., TRPO, DGES) and off ofolyy evaluation (OPE). For limitations and future work, we only provide the theory on the fineness of policy abstraction, while provide no theory on the optimality, although the optimality ought to depend on the downstream problem considered. For policy representation learning, the alignment loss and MMD metric are not the only choices; besides, other representation learning principles (Bardes et al. 2021) are potential.

---

Under review as a conference paper at ICLR 2023
REFERENCES
A. Bardes, J. Ponce, and Y. LeCun. Vicreg: Variince-invaranne-ccvvvaa regularization for self-supervised learning. arXiv preprint (xXi::2/05504900 2021.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.00 015400 2016.
K. Choromanski, M. Rowland, V. Sindhwani, R. Turner, and A. Weller. Structured evolution with compact architectures scalable policy optimization. ICML, pp. 970-978. PMLR, 2018.
Faccio, L. Kirsch, and J. Schmidhuber. Parameter-based value functions. arXiv preprint (xXi::0006.09226, 2020.
Fu, M. Norouzi, O. Nachum, G. Tucker, Z. Wang, A. Novikov, M. Yang, M. R. Zhang, Y. Chen, A. Kumar, et al. Benchmarks for deep off off-policy evaluation. arXiv preprint arXiv:2103.16596, 2021.
Giunchiglia and T. Walsh. theory of abstraction. Artificial intelligence, 57(2-3):323-389) 1992.
A. Gretton, K. M. Borgwardt, M. Rasch, B. Schölkopf, and A. J. Smola. kernel two-sample test. JMLR, 13:723-773, 2012.
A. Grover, M. Al-Shedivat, J. K. Gupta, Y. Burda, and H. Edwards. Learning policy representations in multiagent systems. In ICML, volume 80 of Proceedings of Machine Learning Research, pp. 1797-1806, 2018.
N. Hansen and A. Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2):159-195, 2001.
J. Harb, T. Schaul, D. Precup, and P. Bacon. arXiv:2002. 1833, 2020.
Policy evaluation networks. arXiv preprint
J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 11:4565-45733 2016.
H. Jeffreys. invariant form for the prior probability estimation problems. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 186(1007):453-661, 1946.
A. Kanervisto, T. Kinnunen, and V. Hautamäki. General characterization of agents by states they visit. arXiv preprint arXiv:2012.01244, 2020.
M. Kaya and H. S. Bilge. Deep metric learning: A survey, Symmetry, 11(9):1066 2019.
L. Li, J. Walsh, and M. L. Littman. Towards unified theory of state abstraction for mdps. ISAIM, 4:5, 2006.
X. Ma, J. K. Gupta, and M. J. Kochenderfer. Normalizing flow model for policy representation in continuous action multi multi-agent systems. In AAMAS, pp. 1916-1915 2020.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare. A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, Wierstra, Legg, and D. Hassabis. Human-level control through deep reinforce- ment learning. Nature, 518(7540):529-533
2015.
M. Mutti, S. D. Col, and M. Restelli. Reward- policy space compression for reinforcement learning. Unsupervised Reinforcement Learning Workshop at ICML, 2021.
T. Nguyen-Tann -aane, S. Gupta, and S. Venkatesh. Distributional reinforcement learning via moment matching. In AAAI, pp. 9144-9152 2021.
A. Pacchiano, J. Parker Holder, Tang, K. Choromanski, A. Choromanska, and M. I. Jordan. Learning to score behaviors for guided policy optimization. In ICML, volume 119, pp. 7445-7454, 2020.
10

---

Under review as a conference paper at ICLR 2023
M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
R. Raileanu, M. Goldstein, Szlam, and Fergus. Fast adaptation new environments via policy-dynamics value fuunction.. In ICML, volume 119, pp. 7920-7931, 2020.
H. L. Royden. Real analysis / h. 1. roiden. 1968.
T. Sang, H. Tang, Yi Ma, J. Hao, Y. Zheng, Z. Meng, B. Li, and Z. Wang. Pandr: Fast adaptation to new environments from offline experiences via decoupling policy and environment representations. arXiv preprint arXiv:2204.02877 2022.
S Schreck, C. Coley, and K. JM Bishop. Learning retrosynthetic planning through simulated experience. ACS central science, 5(6):970 981, 2019.
J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In ICML, volume 37, pp. 1889-199,, 2015.
Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 177.063777, 2017a.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization 2017b.
algorithms. arXiv preprint arXiv:1707.0666
L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine. AVID: learning multi-stage tasks via pixel-Ievel translation of human videos. arXiv preprint arXiv:1912.04443. 2019.
A. Smola, Gretton, Song, and Schölkopf. hilbert space embedding for distributions. In ALT, volume 4754 of Lecture Notes in Computer Science, pp. 13-31, 2007.
H. Tang, Z. Meng, J. Hao, C. Chen, D. Graves, Li, C. Yu, H. Mao, W. Liu, Y. Yang, W. Tao, and L. Wang. What about inputing policy in value function: Policy representation and policy-extended value function approximator. arXiv preprint arXiv:2010.09536 2020.
Urain, D. Tateo, T. Ren, and Peters. Structured policy representation: Imposing stability in arbitrarily conditioned dynamic systems. arXiv preprint arXiv:20/2.06224 2020.
I. V. d. Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, (11), 2008
A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 18X7.00307 2018.
A. Zhang, R. McAllister, R. Calandra, Y. Gal, and S. Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020.
Y. Zheng, Z. Meng, J. Hao, Z. Zhang, T. Yang, and C. Fan. A deep bayesian policy reuse approach against non-stationary agents. NeurlPS 2018, pp. 962-972, 2018.

---

Under review as a conference paper at ICLR 2023
A MORE CONTENT ON POLICY ABSTRACTION THEORY
A.1 PROOF OF THEOREM 3.1
Proof. We prove the partial ordering (z) of the Theorem 3.1 one by one in the following.
f = frx. Given an MDP M. two policies иs, TTy E 11. We define P the transition probability and use (s))ss = (-- P(s'ls, s, a) to denote the distribution of next state s' when performing policy ı at state 8, respectively. Then, we have
p= Eawn (-16) P())(s,),
pd (5'1s) Ea (-16) (())s,,))))
If fx [T;) = fTT)) with the Definition we have Vs S, Va E A, m((as) (alss) Combined with Eq. we derive,
Vs, s' S, (s)(ss)) pas (s (s' Recall the definition of fp=,
(6)
obtain fx = fp..
@ fr- = fvv.. Given an MDP M, two policies ns, TT E IT, a reward function R. Start with the definition of the value function (+(.), we consider the two cases:
Case 1. The reward function R depends only on state s S, we derive the value function:
Ys E S, (s) = Rs)) + Σ PM
(8) (s').
If (T)) = for (Ty),, with the Definition we have Vs, s E ES P (s'ls) = (s'lss When the
Eq. 7 holds, we have:
Vs E ( V*>
Recall the definition of fv., we can obtain fr= ž fvv.
Case 2. The reward function R depends on both state E S and action E A, we derive the value function:
v= (6))
V (8) - Σ π (a s) [((s,)) + Ep (8' IP(d' (s,
s,a)
(9)
If p (T:)) fp=((T)) (Ty), with the Definition 3 we have Vs, s' S, p 8)
s). Unlike
the 7 πJ may not be equivalent to x, regarding the abstraction criterion of value irrelevance. Therefore, the partial ordering (fp- > fv~) is not obtained under Case 2. Nevertheless, Case is fairly standard across broad set of real world RL problems.
Remark A.I (More discussions on R(s, a) and R(s)). For Case 2 discussed above. i.c., the reward function R depends on both state 8 E S and action a E A, we can further separate into two categories according to our knowledge on real-world decision decision-making making problems:
In our first category, the dependence of on state and action due the consequence of s, a in leading the decision system into the specific new state s', i.c., R(s, a) E. P(s'lss (x') where U(s') is the utility of s' (we adopt the expectation form for the convenience of discussion). In the cases that fall into this category, it can be casy to re-define the reward function by the utility function, i.e., R(s, a) = Us)) for any E A. With such conversion, we can also obtain f Z fv in these cases.
Our second category covers the exclusive cases of the first category. For example, consider an environment, where two actions 41 a2 lead to the same new state s' from state but gain different rewards. In such cases, the reward is independent on the dynamics caused by s, We consider that such cases are minority the ones of interest.
A.2 OTHER POLICY ABSTRACTIONS
In this paper, we also propose three other policy abstractions in Definition 6 Before introducing the definitions of additional policy abstraction, we make some necessary notations. We use * to denote the distribution of state when policy performs & steps from initial states regarding the initial state distribution po. The discounted state visitation distribution from initial states regarding
Pn is defined d= (5') (1 7) Ee= dddm.(s') for any s' E S. Additionally, we use dz. to
12

---

Under review as a conference paper at ICLR 2023
denote the distribution of state when policy performs & steps from any states s. The discounted state visitation distribution from any state a defined as dā (5')
Ei=o " for any
E S.
Definition 6. Given an MDP A, P.P.R. R, and ground policy space II, for any two policies Ta Rj E 11, we define four more policy abstractions as follows:
1. An influence-irrelevance abstraction (faz) is such that for all 8, & E S, [ay(m;) = Say (TTz) implies that d% (s') d=² (s').
2. An implies that dd (s) d (s).
abstraction (fa-) is such that for all S, far (T.)) = fa (Tz))
3. A value-irrelevance abstraction (f)=) is such that fI=( f (Tz) implies that E Expppo [V (so)] Epppp [V*) (so))-
4. A value-irreled irrelevance abstraction (fz=) such that for all E (m,) fz" (Tyy) implies that 7 (8) 77:(8))
Similarly, we prove how the newly proposed policy abstractions are related to other abstractions we introduce in the main body of the paper.
Theorem A.2 (Partial Ordering (z)). Under the Definition 3 5 and O we have f = S z
faz - fa - fax; 2 fx = fo ? fax = fv= 2 fJz. An illustration is provided below:
FT
z
fp
faz fan
IV
IY
fym
fym
Proof. The partial ordering (2) satisfies transitivity, thus, let us prove the Theorem A.2 one by one in the following.
1 frr- = fay. Given an MDP M, two policies ny, nT II. If (T.) fr- (m₂)) with the Definition we have Ys, s' E S, (s' ()) = (s' s). Since the initial state distribution are the same, and ne. Tÿ follow two identical Markov chains, we derive fp= ł Say.
fa; = far Given an MDP, two policies Ti, T, E II. If fay (TT)) = fay( (my), with the Definition 6 we have Vs, s' E S, d (s') = (s'). Furthermore, we derive,
Ysu E Po. s' E S, d (s') = (s').
(10)
Thus, faz ž far. Ola fr. Given an MDP, two policies ni, Ny II. If fa- (m.) = fa (TT)), with the Definition 6 we have Vs E S, (s) = d (s). When the reward function R depends only on state s E S, we have,
J(x,) = (1--)-11))) - E. [R(s)]
(11)
-(xx)= = (1- - [R(s)].
Thus, far 2 fr.. ffr 2 fv-. Given an MDP, two policies n. ny 11. If faz (π.) = faz (TT)) with the Definition 6 we have Vs, E S, d, (5)) (6'). When the reward function depends only on state E S.
we have,
Vs E S, V (s) = (1--)) (1-7)--BBrr) Reaa [R(s')]
(12)
V (8) = (1 (1-7)-1 R red [R(s')]
Thus, Jaz 2 fv=. ① fv= - frr. Given an MDP, two policies n. v, 11. If (m;) fv-(TT)) with the Definition we have V E V (s) = a (s). Furthermore, we derive,
Vsp E Po: V= V/3 (80); Esgxpo [V (8o) Eso Tv (so)
(13)
Thus, fv= 2 fzr.
In particular, when the reward function R depends on both state and action a E A, n; may not be equivalent to πj. The fa= ż fo= and faz > fy= are not hold. Connecting Definition 6 and TTherrm/A.. A.2 we summarize the properties of different policy abstraction in Table
13

---

Under review as a conference paper at ICLR 2023
Table 3: Properties of different policy abstraction, including the additional ones introduced in A.2
We use (+) and (-) to denote the higher or lower degree in the same level.
Abstraction
Abstraction Criterion (for xi W2, Vs, s", 5 x A)
Fineness
Task Relevance
Je Dynamics Influence Equivalence (P*((s">))
Policy Parameter Equivalence (0, = 0/)
Highest
None
Action Distrihution Equivalence (Tzia 5) = [1(a x))
High
Low
(s
Middle (+)
Middle ( (-)
fag
Dynamics Influence Equivalence (d"+(a) (e')
(s'))
Middle
Middle
fa
Dynamics Influence Equivalence (d" (a) d== (s))
Middle (-)
Middle (+)
Value Function Equivalence (s) (s])
Low
High
f--
Value Function Equivalence (Eagr [V* [so]] Eagrono [V%s (sol)
Low (
High ( i)
f0
Triviality (taking all policies as the same)
Lowest
None
Table 4: A taxonomy of prior policy abstractions under our policy abstraction theory.
Prior Policy Representation
Abstracbon Chleriin
Related Pobey Abstraction
Voctorized Network Parameters Faccio stal.
2020
Polacy Parameter Equivaleace
Contrastive OPR/SPK
Lang arain 20200
Policy Instance Contrast
Policy Recovery OPR/S
Pid Lain al. 2069
2020
Action Distribution Prediction
[- (Det
[ind OPR/SPR
Value Function Prediction
fy. (Deha
Network Fingerprint
Harh ml 2020
Action Distribation Similarity
fr (Def (
Behavior Embedding (State)
paceiisaoo 01 3020
Final State Similarity
Je- (Det
Behavior Embedling (Action
Distribution Similarity
f.
Behavior Embhekling (Reward hiccchlan et
2020
Return Similarity
f- (Debio
Generative Representation
Grover er mi 2015
Action Distribution Prediction
(Del
Discriminative Representatisnff
Policy Instance Contrast
e-compression (Mutii Di a. 2023
Action Distribution & Dynamics Influence Similarity
f- (Det
& Je (ee
Policy Supervector Kamerveio 2020
Dynamics Influence Similarity
(Del
B TAXONOMY OF PRIOR POLICY ABSTRACTIONS UNDER OUR THEORY
Closely related our work, (Harb et al. 2020) adopts policy fingerprints differentiable policy representation obtained by concatenating the distribution of actions of policy in a set of key states. Obviously, it's more concerned with policy distribution information. (Kanervisto et al. 2020)
make use of Gaussian mixture models to learn policy supervectors (policy representation) which characterizes agents' behaviour by the distribution of states they visit. Pacchiano et al. (2020) define Behavioral Embedding Map (BEM) with different implementation options and propose using Wasserstain distance in the latent space induced by BEM to measure the similarity among policies. Like us. (Tang al., 2020) also employs neural network-basse based policy parameters policy data, but it utilize policy distribution reconstruction and contrast learning principles to learn policy representation. The principle of contrastive learning itself is a general learning principle irrelevant to the learning task. Therefore, the contrast of instantiation makes its abstraction level low. Compared with these two works, (Faccio et al. 2020) is a simpler and more direct way of constructing policy representation, because it directly compress the policy parameters based neural network into vectors and regards it as a form of policy representation. In fact, the policy space may be redundant due to the existence of some policies that induce similar behaviors. It provides feasibility for policy abstraction and policy space compression (Mutti al. (02T). For instance, Mutti et al. (202T) study policy space compression by formulating a Set Covering problem with Rényi Divergence of discounted state-action distribution of policies converts opponent modeling into an opponent policy representation learning problem and attempts to distinguish opponents by their policy representations. In Table we categorize these works under our abstractions.
metric. In addition, in multi-agent system, (Grover et al. 2018 8) also
C CASE ILLUSTRATION OF POLICY ABSTRACTION
To compare these policy abstractions in a quantitative view, we demonstrate how the distances of two policies (blue and greed in Fig. 3) measured by the corresponding policy metrics differ in several Gridworld MDPs. We borrow Distinet Policies, Doorway from (Kanervisto et al.) (2220) and design a new environment, Key Action for simple prototypes of environments with different features. All three environments are 5x5 Gridworld, with the starting state and goal in the lower left and upper right grid, respectively. The discrete action space is {up, down, left, right). We obtain the estimated value v(-) of each state by estimating 1 [expected number of step to goal when start ing from a given state]. About the Key Action, the agent obtain positive reward only if chooses right at the key state (i.e., marked by the red box in Fig. 3). Every agent rollouts 10k episodes for comparing different policy metrics.
14

---

Under review as a conference paper at ICLR 2023
Perspolly actance
9.35
0.30
0.2
0.20
0.3
0.2
Distinct
Dosrwayy
0.15
0.5
0:5
Key
0.4
010
0.00
0.2
0.00
Prob. of random action (%)
Figure 3: Policy comparison with different policy metrics in Gridworld. Top Panel: The illustration of three Gridworld MDPs and deterministic policies (blue and green). Bottom Panel: The distance curves of the two policies measured by dxy dpr, dy daz, d (y-axi), against the stochasticity of the environment (r-axi).
To be specific, policy is represented by tensor т E 5x5xx4 and the distribution-irelleaace metric is defined as (Tt, ft ET. [[ k.kj #tj,,l] Similarly, the state transition dynamics induced by policy is represented by tensor pa E Rox5x5 and the influence-ireleeancee metrie is defined as drw (T, ft = EP P...
P i.j.k
P. j.j. Different from the d sampling-
based estimation, we leverage the dynamic programming to learn the value function 1/ E TR5x5 Thus, the value-irrelevance metric is defined as dy- (x, f) = Ev evr,evv,y .V,, eV+
[lV V In
addition, we also compare the two other policy abstractions proposed in Appendix A.2 Among them, we estimate the discounted state visitation distribution d (s) via dividing visits state
by total number of interactions (i.e., da- ( Ear, Edd,,,,e ed [ df,,)). Simply and
naturally, based on the definition of f-== in Definition 6 we have d. (TT, +) |Eso~po [V* (so)
Exgxpo [V+ (so)] J. Fig. 3 shows the illustrations and the results of the five policy metries. The dæ- and d are newly added results compared the original paper. We observe that the dæ cannot indicate the difference in dynamics between the two policies in Key Action. Like the dy=, the d
measures the difference in the outcomes of the two policies but shows the poor robustness as the increase of stochasticity.
D ADDITIONAL DISCUSSIONS
D. ESTIMATING POLICY METRICS VIA JEFFREYS DIVERGENCE
In simple MDPs where the state-action space is finite, we calculate the frequency distribution (i.e.. म. P 2d) using sufficient samples as an estimate of the exact probability distribution (i.e., m, P*
)".. Then we use the Jeffreys Divergence (Jeffreys 1946) between empirical distributions policy
metrics (i.e., dx (.).,, d (-,-), and dy (+))))
dz (Ti, TT) DKL (T [ + DKL (T p
dpr (T.s ~ DKL
P
+ DKL
| 1
(14)
dy- (T,, ~ DEL
7 + DEL
E DETAILS ON APPLYING POLICY ABSTRACTIONS TO POLICY OPTIMIZATION
The policy optimization experiments are run on single NVIDIA GeForce GTX 2080Ti GPU, Our codes implemented with Python 3.7.13 and Torch 1...
15

---

Under review as a conference paper at ICLR 2023
Table 5: Trust-region constraint value choices of different Methods, We use "_' to denote the 'not applicable' situation.
Methods
Trust-region threshold (o)
Vanilla-PO
Max TV
{0.001, 0.005, 0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5}
Gaussian
(0.5, 1.0, 3.0, 5.0, 10.0, 15.0, 20.0)
Supervector
{0.01, 0.05, 0.15, 0.2, 0.3, 0.4, 0.5}
TRPO-
(0.05, 0.3, 0.5} 0.055 0.1, 0.4, 0.5 0.05, 2.0
TRPO-
TRPO-fv
E.1 TRUST-REGION POLICY OPTIMIZATION
In this experiment, adopt a N-dimensional Gridworld MDP provided by (Kanervisto et al. 2020). where the position coordinates of the agent form the state space. At each state, the agent choose one of N actions corresponding N directions. In the experiments, we set N=5 and only one action of N action move the agent forward. The agent is rewarded with +1 reward for taking the correct action and the maximum length of each episode is 25. The agent uses a two-layer network with 16 units and lanh-activations each.
For the policy optimization, we rewrite the implementation of the policy metries in the code of (Kanervisto et al. 2020). The learning agent checks if the difference between old and new policies measured by the policy metrics are larger than trust-region threshold If the threshold is exceeded, we stop updating the policy with current samples and move on to collect new samples for the next policy update. The sample size is 4096 and the policy is updated for 100 mini Mini-batches of 64 items over the collected samples, or until constraint prevents updates.
We compare with existing related methods, i.e., Vanilla-POO PanillaaPO..aaa Total Variation Divergence(Max TV), Gaussian and Supervector, and follow hyperparameters in (Kanervisto et al. (2200) Among them, Vanilla-PO means updating policy 100 mini-batches with no trust-region constraint; the Total Variation Divergence measures the maximum amount of how much probability of taking any single action (in any state) can change and is defined as d (T, RT) = max, Σn [a, s) Ty (a s)]; the Gaussian refers to fitting multivariate, diagonal Gaussian collected states from 5 trajectories for old and new policy, respectively and measures Jeffreys Divergence between old and new policy; the Supervector calculates the upper bound of KL-divergence between old and new policy supervectors, which obtained via fitting four- four-component UBM collected states from 5 trajectories for corresponding policy. To ensure a fair comparison, we estimate the proposed policy metrics de dp dy= collected 5 trajectories from old and new policy. All approaches repeat ten times for same environment and the search space of trust-region threshold for them are shown Table S
Fig. 4(a) reports the empirical results for different methods using the corresponding optimal thresholds which are selected based on the largest AUC (i.e., Area Under The Curve). Overall, the TRPO-fn and Max TV are better than others, mainly because they concern the action distribution at interested states. The Gaussian and Supervector are concerned with the state visitation distribution induced by the policy, which are specific instances our proposed dynamics influence abstraction. Thus, as with f' and fiv,, they ignore the differences of policies in action distribution.
E.2 DIVERSITY-GUIDED EVOLUTION STRATEGY
In the experiments, adopt the Point environment with deceptive rewards using MuJoCo simulator: The deception comes from a barrier, which misleads the agent to move directly forward leading to a suboptimal policy. For the Point environment, the state and action are represented by a dimensional vector and 2 dimensional vector, respectively. At each timestep. the agent is penalized for its distance from a given goal, and we limit episode length to 50 steps. To reduce the number of policy parameters and thus the time cost, we use Toeplitz policy (Choromanski et al. 2018) often used for ES algorithms. For DGES, we set the population size to 50 (i.c., 50), and estimate the proposed policy metrics dy. dp=, dv= on one episode from the currently policy and the ancestor policy.
16

---

Under review as a conference paper at ICLR 2023
Average Return (1e2)
Average return
-7
4-
Vanilba PO
Max Action TV
TRFO-
8
VanillaEES BGES
DGES-fp
2-
Gaussian
DGES-fy-
Supervector
D
0.4 Timesteps
0.6
0.8
1.0
0.2
0.4
0.6
Timesteps
(a) TRPO
(b) DGES
Figure 4: Performance of different methods in: (a) Trust-Region Policy Optimization (TRPO); and (b) Diversity- Guided Evolution Strategy (DGES). Results the and half standard deviation (shaded) over 10 and 5 trials for TRPO and DGES respectively.
We compare with state-of-th the art method on the Point environment for exploration: BGES (Pacchiano (et al.] (020), which uses the terminal state as a policy representation and compare two policies by Wasserstein distances. The hyperparameters associated with the BGES remain consistent with the original paper. From the Fig. 4(()) we observe our approach DGES-fp- comparable to the BGES (Pacchiano et al.) 2020) and performs better than the DGES- f+ and DGES- It illustrates the crfluence-irellvancee metric dp- sweet intermediate point, keeping the ability of distinguishing the two policies.
F DETAILS ON APPLYING POLICY ABSTRACTIONS TO OFF-POLICY EVALUATION
The off- off-policy evaluation experiments are run on single NVIDIA GeForce GTX 2080Ti GPU. Our codes are implemented with Python 3.6.13 and tensorflow 2.2.0.
F.1 COLLECT POLICY DATA FOR OFF-POLICY EVALUATION
Benchmark Environments. We conduct our experiments OpenAl Gym continuous control tasks and detailed description is below.
PaiieeDDDobbeeeedduuuuee (IDP-v2): Balance pole on pole on cart. The agent gets reward for every timestep that the pendulum has not fallen off the
(LLC-v2): Navigate a lander to its landing pad. Landing pad is always coordinates (0,0). Reward for moving from the top of the screen to landing pad and zero speed is about 140 points. If lander moves away from landing pad loses reward back. Episode finishes if the lander crashes or comes to rest, receiving additional -100 or +100 points.
Each leg ground contact +10. Firing main engine is -0.3 points each frame. Solved is 200 points.
Policy Data Collection. We collect policies in two stages. The first stage, we train the PPO (Schulman et al. 2017a) agent for 1M steps and store checkpoint each 10 updates of the policy. Then, evaluate each checkpoint by 10 rollouts. To be specific, for the ppo agent, the update frequency of critic is 5 per epoch on two environments, and the update frequency of actor is 2,5,10 10 per epoch on IDP-v2 and 10 per epoch on LLC-v2, respectively. The ppo agent with each update frequency trained using 20 random seeds. The second stage. we divide 50 intervals I,(r 1, 1.... 50) depending on the performance range of collected policy set I. Naturally, each interval contains certain number of policies. Then, we randomly select K K(K 40) policies from each interval. Finally, we collect B(B 200) trajectories of data per selected policy. Further, for each policy, we calculate the average return Go over the 200 trajectorics, and randomly sample state-action pairs {ss, 1=m sexte-next state pairs {(s 8/) state-value pairs {(s,, Gy) The pseudo-code for policy data collection presented in Algorithn[]
In the experiment, we construct policy dataset for IDP-v2 and LLC-v2, respectively, and each policy dataset contains 2000 policies. Fig. shows the histograms of the two policy datasets, where the
17

---

Under review as a conference paper at ICLR 2023
Algorithm I Collect Policy Data for Off-poolccy Evaluation
Input: Policy set = {11, .z.... } collected based on PPO algorithm; the number T of policy performance intervals; the number K of policies selected per interval; the rollouts B and policy dataset D -0
for 7 = 1.2,-.. do
2: Randomly select K policies in J for k to K do
3:
4:
R 0 +
5:
for rollout b= to B do Sample Monte-Carlo return Go using policy Th Collect data with Ty in real environments: {(8, a, 8 R. RU (G)). T U {(s,a,T,s))
6:
7:
8:
9:
end for
10: TD < D U ( R, T, Oz represents the parameters of policy
end for
12: end for
Output: IB
Algorithm 2 Off-policy Evaluation
Input:
Training policy dataset Darain; the data of the policy consisting of policy parameters
0,, state-action pairs {(>),,;)){=T, state-next state pairs {(s,, [(*),,])} " and state-value pairs
{18,, G;)) the expected return Go
Initialize: The policy evaluation function Va with parameters B
Initialize: The policy representation function fo with parameters /
1: Calculate d.. dpr, and dy- for all (T) E Mrrain
see Section 4.2
2: for iteration = 0,1,2,... do
3: Sample mini-batch N of policy data B from Irtran
4: Calculate the alignment loss LAL
5: (W) = E,,, E [(I
(0x;) - (0,,)) 11222222 (T) (dr,
6:
Calculate the evaluation loss LEL
7: (3, +) En (f
Go)
8: Update parameters 1) B to minimize CAL and LEL
9: end for
Output: Parameters & of the policy representation function fs: parameters 3 the policy evaluation function V
x-axis is the average return of policies and the y-axis is the number of policies. As shown in the Fig. 6 the policy datasets basically satisfy the diversity and balance of policies, which allows us to design a series of Off- Off-Policy Evaluation (OPE) scenarios.
F.2 OFF-POLICY EVALUATION
With regard off-policy evaluation tasks, in this work, adopt policy evaluation function (Harb jt al., (2020) representing the expected return of policy. Naturally, the policy evaluation function can be trained based on sampled training policies and then generalize to unseen policies. The policy evaluation loss LEL can be formalized as,
LEL (3, x) = Exen ((V -Go)])
(15)
where V.(+) parameterized by B denotes the policy evaluation function. (-) parameterized by v is the policy representation function which takes the policy parameters 0 as input. Detroin denotes the training policy dataset sampled from the collected policy dataset D. Below, provide the implementation details and pseudo-code (Algorithm 2 for off-policy evaluation.
Network Structure. The Fig. illustrates the policy encoder (LPE) we used. The Table 6 shows
the structure of the policy network and the policy evaluation network. As shown in Table 6 we use a two-layer feed-forward neural network of 32 and 32 hidden units with ReLU activation (except for the output layer) for the policy network. In this paper, we learn policy representation (pr) for
18

---

Under review as a conference paper at ICLR 2023
W
[o. 4]
[1. [2]
[22 [3]
1,
[1. 4
[1, 12]
[1. [g]
[W, + b,]
[1, lo 1]
[1, 4 + 1]
[4., l2 + 1]
MR (WW
[1, z1
[1, Ø2l
[1, Ø3
[1, 2 Z2 Zz]
Figure An illustration for Layer-wise Permutation of hidden units for the hidden layers. The main idea is to perform permutationninvariaa transformation for inner-layer weights and biases for each layer first and then concatenate encoding of layers.
-iviaaaiionnnnniiantt invariant Encoder (LPE). 4 denotes the numbers
Table 6: Structure of policy network and policy evaluation network.
Layer
Policy Network (r(a)s) Policy Evaluation Network (V(-))
Fully Connected
(state dim, 32)
(pr dim. 128)
Activation
ReLU
ReLU
Fully Connected
(32, 32) ReLU
(128, 128) ReL.
Activation
Fully Connected Activation
(32, action dim) tanh
(128, 1)
None
RL policy from its network parameters. Like the policy network, the policy evaluation network also uses two-layer feed-forward neural network, where the input the policy evaluation network is the policy representation and the output the predicted value of the expected return the policy.
Hyperparameter. In the Table 7 we discuss the policy representation dimension, the number of hidden units of the policy evaluation network, and the batch size for training policy evaluation network. For fair comparison, we perform a 10-fold cross-val tross-validation weak generalization experiment with 20% sampling ratio based on the fnæ and IDP-v2 to search for best hyperparameters. For all methods and environments, the batch size, the number of hidden units and the policy representation dimension are 256, 128 and 256, respectively. Additionally, we report the common hyperparamters of off-policy evaluation experiments in Table
G OTHER EXPERIMENTAL RESULTS
G.1 COMPLETE EXPERIMENTAL RESULTS OF DIFFERENT GENERALIZATION SCENARIOS
In the paper, we design two OPE generalization settings, namely weak generalization and strong generalization. For each of the generalization settings, we further construct generalization scenarios with different sampling ratios. As with the Table the results of the generalization experiments on the 40% and 80% sampling ratio are presented in the Table and the Table I0, respectively. Not surprisingly, the performance of all methods improves the amount of training data increases. In
19

---

Under review as a conference paper at ICLR 2023
Table 7: Experimental results of fRE for IDP-v2 with different hyperparameters. The minimum value is highlighted. Results(xIe-2) are the mean # a standard deviation over 10 trials (for weak generalization with 20% sampling ratio).
representation dimension
batch size
hidden units
64
128
256
T-error
G-gap
T-crror
G-gap
T-error
G-gap
64
64
0.64
0,42#0.10
0,5920.05
0.3920.06
0.53+0.05
0.3440.05
64
128
0,59=0.07
0,40=0.08
0.56=0.06
0.36#0.09
0.59+0.07
0.39:0.09
64
256
0.61
0.42=0.10
0.57=0.06
0.37=0.07
0.59+0.09
0.37+0.09
128
64
0.62=0.06
0,40=0.07
0.58=0.09
0.37#0.09
0.57+0.06
0.37+0.07
128
128
0.64=0.06
0.43=0.07
0.55=0.06
0.36+0.09
0.57=0.06
0.39+0.08
128
256
0.58=0.07
0.38=0.07
0.56#0.06
0.37=0.08
0.55+0.05
0.37=0.08
256
64
0.69=0.10
0.38+0.10
0.64+0.09
0.38=0.09
0.60#0.06
0.36+0.08
256
128
0.65+0.09
0.38=0.07
0.6420.07
0.41
0.54
0.34+0.05
256
256
0.65=0.09
0.41#0.10
0.59=0.07
0.38+0.09
0.55+0.07
0.35+0.10
Table 8: A comparison of common hyperparameter choices of algorithms. We use 'not applicable' situation.
to denote the
Hyperparameter
fe 1.10
JRE
JEL 1-10-1
JCL 1-10-3 3
Jm, Jps, fv 1.10
Evaluation Model Learning Rate
1-10-3
Representation Model Learning Rate
1 1-10-3
1 1-10-3
1-10- 3
1-10-3
1.10-3
Optimizer Batch Size
Adam
Adam
Adam
Adam
Adam
256
256
256
256
256
Policy Representation dim
256
256
256
256
Evaluation Model Update Epoch (20%)
10k
10k
10k
10k
10k
Evaluation Model Update Epoch (40%)
50k
50k
50k
50k
50k
Evaluation Model Update Epoch (80%)
100k
100k
100k
100k
100k
Kernel mu
2.0
Kernel number
Sample Size m
1000
the weak generalization experimental scenarios, our methods are better than or comparable to other methods. In the strong generalization experimental scenarios, the proposed method, especially ʃv=, is significantly more robust than the other methods.
G.2 How DIFFERENT POLICY ABSTRACTIONS INTRAPOLATE AND EXTRAPOLATE IN OPE?
To present the results of weak and strong generalization more intuitively, we attach scatter plots of policy evaluation results for two generalization settings with 20% sampling ratio in Fig. 7 7 I0 To be specific, the horizontal axis of scatter plots represents the true value and the vertical axis the predicted value. The evaluation results of the training policies (red dots) and target policies (blue dots) are unified in scatter plot for each method. Experimental results are the best of 10 and 5 trials (for weak and strong respectively). Obviously, the results of the weak generalization setting basically remained near the diagonal, while the strong generalization experiments show an overall underestimation. This is due to the fact that in the strong generalization setting only sample now-performance policies as training data and the rest (i.c., high seghhpeeformance policies) are used as test policies.
G.3 VISUALIZATION LEARNED POLICY REPRESENTATION
In this work, we further analyze different policy abstractions (i.e.. fe. fRE. fEL. fu. fpr, fy*) by visualizing policy representations. Among them, the So denotes directly compressing policy parameters as policy representations. The fRE and fEL refer to learning policy representations by random initial abstraction function and optimized abstraction function by the evaluation loss, respectively. The unsupervised contrastive learning approach (fcr) learns policy representations based on the InforNCE (Tang et al. 2020). fx. fp-, fv" correspond to the three policy abstractions propose, and the policy representations are learned via the alignment loss.
20

---

Under review as a conference paper at ICLR 2023
Palicy Set Histogram
Policy Histogram
70
50
Frequency 40
Fraquency
40
20
Average Recum
Average Return
(a) InverteIDoublccenndttuu-
(b) LunatLanderConimnouswyy
Figure 6: Histograms of policy dataset collected from the (a) InverceDDubbleennulluw and (b) y-axis is the number of policies.
environments, where x-axis is the average return of policies and
Visualization Details. We perform experiments under two generalization settings with 20% sampling ratio. For a fair comparison, different methods use the same training dataset to optimize the policy representation function. The hyperparameters about training phrase can be found in Tablg8 We store the last policy representation model for all methods, and compute the representations of training policies and target(unseen) policies. Finally, we plot 2D embedding of each policy representation by the T-SNE (V. d. Maaten & Hinton 2008). In the visualisation results, each colored point represents a policy and the label of the color bar for all results is the expected return of the policy.
Visualization Results. Overall, the policy representations in Fig. II are more clearly distinguishable and all of our proposed methods show some degree of discriminability. In contrast, in the strong generalization experiments, the better performing policies are less discriminative as only 20% of the poor performing policies used training data. Nevertheless, our proposed iellee-irelevaace policy abstraction still learns a more compact and discriminative representation. The same results can also be found in the LLC-v2 environment (Fig. 13 14). Differently, JRE has already obtained better discriminability on that environment. We consider two possible reasons for this result, one being due to the variability of the different environments, and the other possibly being the superiority of the policy encoder network (LPE) adopt. To verify the latter, show the visualization of the unabstracted policy in fig. I5] From the results, clear that our policy encoder network is indeed able to perform an effective compression of the policy information and obtain compact policy representation space.
21

---

Under review as a conference paper at ICLR 2023
Table 9: Performance of different policy abstractions in Off-policy Evaluation (OPE). The minimum value for each task highlighted. Results are the mean a standard deviation over and 5 trials (for weak and strong with 40% sampling ratio respectively).
Env
Ahstraction
Weak Cinerralization
Strong Generalization
T-error
G-gap
T-crror
G-gap
0.0041 = 0.0002
0.0018 = 0.0005
0,0684 = 0.0056
0,0679 = 0.0057
fRE
0.0042 + 0.0004
0.0021 = 0.0005
0.0752 + 0.0123
0.0728 + 0.0124
fRI.
0.0039 + 0.0002
0.0015 0.0005
0.0725 0.0106
0.0717 0.0098
IDP-v2
fot
0.0046 1 0.0002
0.0025 0.0004
0.0762 0.0059
0.0676 0.0091
fe
0.0037 + 0.0002
0.0014 0.0005
0.0843 0.0091
0.0816 0.0087
fp-
0.0037 1 0.0002
0.0014
0.0005
0.0874
0.0063
0.0845
0.0077
fve
0.0037 1 0.0003
0.0011
0.0005
0.0584
0.0072
0.0548
0.0087
Je
0.0008 1 0.00001
0.0007 1 0.0001
0.1061
0.0137
0.1046
0.0138
fRE 0.0013 0.0003
0.0011
0.0003
0.0339
0.0142
0.0295
0.0124
0.0008 1 0.0001
0.0007
0.0001
0.0322
0.0035
0.0320 1: 0.0036
LLCxx2
JeL
0.0015 + 0.0002
0.0013 = 0.0002
0.0434 + 0.0019
0.0365 + 0.0040
0,0008 1 0.0001
0.0006 0.0001
0.0193 0.0022
0.0163 0.0023
fp- 0.0008 1 0.0001
0.0006 0.0000
0.0215 1 0.0041
0.0169 1 0.0055
fv-
0.0008 + 0.0001
0.0006 0.0001
0.0301 0.0024
0.0278 0.0043
Table 10: Performance of different policy abstractions Off-policy Evaluation (OPE). The minimum value for each task is highlighted. Results the mean + standard deviation over 10 and 5 trials (for weak and strong with 80% sampling ratio respectively).
Weak Ceneralization
Strong Generalization
Env
Abstraction
T-error
G-gap
T-error
G-gap
fe
0.0029 1 0.0002
0.0004
0.0004
0.018I 1 0.0000
0.0140
0.000S
JRE
0.0029
0.0002
0.0005
0.0004
0.0117 + 0.0002
0.0063
0.0011
JEL
0.0028
0.0002
0.0004
0.0004
0.0135 + 0.0017
0.0067
0.0026
for.
0.0028
0.0003
0.0011 = 0.0005
0.0100 1 0.0009
-0.0072 + 0.0004
IDP-v2
fn
0,0028 +:: 0.0001
0.0003
0.0003
0.0127 1 0.0006
-0.0063 = 0.0065
fr-
0.0002
0.0003 = 0.0004
0.0158 1 0.0010
0.0004
fv=
0.0027 + 0.0002
0.0004
0.0005
0.0082 + 0.0027
-0.0020 + 0.0069
Jo
0.0005 1 0.0001
0,0004
0.0001
0.0102 1 0.0012
0.0078 = 0.000S
fRE
0.0007 + 0.0001
0.0005
0.0001
0.0127 + 0.0036
0.0122 + 0.0039
JEL
0.0005
0.0000
0.0004
0.0001
0.0029 1 0.0005
0.0026 0.0005
fer
0.0007
0.0001
0.0006
0.0001
0.0061 + 0.0022
0.0037 0.0046
LLC--2
fn 0.0005
0.0000
0.0003
0.0000
0.0035 0.0001
0.0022 0.0002
fp- 0.0005 1 0.0000
0.0004
0.0000
0. 0.0043 1 0.0003
0.0037 1 0.0003
fv1 0.000S
+ 0.0001
0.0001 0.0004
0.0032 + 0.0012
0.0028 + 0.0013
22

---

Under review as a conference paper at ICLR 2023
10
Prodectd
62
04 03
11
62
34
06
88
12
03
82
64
35
60
10
60
14
35
12
(a) fe
(b) fRe
(c) fEL
(d) for
1.0
Rae Velu
Real vik
Beal Ak
(c) fx
(f) fr
(g) fy
Figure 7: Results of policy evaluation with 20% sampling ratio (Weak Generalization, IDP-v2). The results show the true and predicted values of the train policies (red dots) and target policies (blue dots) at the minimum testing error for policy abstraction methods.
Emst set
Test sit
1eat
med sas
0.
00
0.2
04
05
0.5
00
00
0.2
08
0.5
15
1.0
02
04
0.0
3.
6.0
82
04
06
3.0
(a) fe
(b) fRE
(c) fEL
(d) fer
0.
Real 1er
Roat vake
Rool kih
(e) fx
(1) fp
(g) fy
Figure 8: Results of policy evaluation with 20% sampling ratio (Strong Generalization, IDP-v2). The results show the true and predicted values of the train policies (red dots) and target policies (blue dots) at the minimum testing error for policy abstraction methods.
23

---

Under review as a conference paper at ICLR 2023
1.0
40
02
30
(a) fe
(b) fRE
(c) SEL
(d) fcr.
train
sot
tra set
(e) f+
(1) f
(g) fyr
Figure 9: Results policy evaluation with 20% sampling ratio (Weak Generalization, LLC-v2). The results show the true and predicted values of the train policies (red dots) and target policies (blue dots) at the minimum testing error for policy abstraction methods.
1.25
1.3
120
50
22
(a) fe
(b) fRe
(c) fel
fol
38 02 04 25 03 12 34
60 02 64 20
03
33
15
25
(0) fr
(f) fpM
fv
Figure 10: Results of policy evaluation with 20% sampling ratio (Strong Generalization, LLC-v2). The results show the and predicted values of the train policies (red dots) and target policies (blue dots) at the minimum testing error for policy abstraction methods.
24

---

Under review as a conference paper at ICLR 2023
taro
8000
tisne
8000
bone
8000
6000
6000
6000
4000
4000
4000
2000
2000
(a) JAK
(b) Scr.
(c) SEL.
8000
t-sne
8000
8000
6000
6000
6000
4000
2000
2000
2000
(d) fr
(c) f
(f) fv"
Figure 11: T-SNE visualization results of policy representation (Weak Generalization with 20% sampling ratio, IDP-v2). Each colored point represents policy and the label of the color bar for all results the expected return the policy.
T-sne
1:500
8000
tone
8000
6000
6000
6000
4000
4000
2000
2000
2000
(a) fRE
(b) fer
(c) fEL
8000
8000
8000
6000
4000
4000
4000
2000
2000
2000
(d) fr
(e) fp
(f) fv
Figure 12: T-SNE visualization results of policy representation (Strong Generalization with 20% sampling ratio, IDP-v2). Each colored point represents policy and the label of the color bar for all results is the expected return of the policy.
25

---

Under review as a conference paper at ICLR 2023
200
200
200
-200
-200
-200
400
(a) SRE
(b) Scr.
(c) JEL.
tone
tone
Lane
200
200
200
-200
-200
-400
-400
-400
(d) fx
(e) fp
( fvz
Figure 13: T-SNE visualization results of policy representation (Weak Generalization with 20% sampling ratio, LLC-v2). Each colored point represents policy and the label of the color bar for all results is the expected return the policy.
200
200
200
-200
-200
-200
-400
-400
(a) fRE
(b) for
fEL
200
200
-20
-200
-200
-40
-400
400
(d) fr
(c) f
(f fv-
Figure 14: T-SNE visualization results of policy representation (Strong Generalization with 20% sampling ratio, LLC-v2), Each colored point represents a policy and the label of the color bar for all results is the expected return of the policy.
26

---

Under review as a conference paper at ICLR 2023
tisne
200
200
-200
-200
4400
-400
(a) fe
(b) fns
Figure 15: T-SNE visualization results of policy representation (Weak Generalization with 20% sampling ratio, LLC-v2). Each colored point represents policy and the label of the color bar for all results is the expected return of the policy.
27


Reviews:
[{"review_id": "txf1lXhwG7", "reviewer": "Reviewer_5gmd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a theoretical framework to categorize policy abstractions based on what properties are preserved. The main contribution is the proposal of irrelevance-based definitions based on policy action distribution, state transition distribution and state values. The authors then propose corresponding pseudo-metrics that permit for more approximate notions of irrelevance. The theoretical contributions conclude by showing that proposed categories of abstraction can be ordered using a concept of abstraction fineness. The authors show some illustrative empirical results in the case where the proposed metrics and their relevant distributions can be computed and compared exactly.\n\nFor the purpose of learning policy representations, the authors then propose a scalable approximate method for optimizing these metrics based on optimizing an alignment loss where the metrics are approximated using maximum mean discrepancy, a RKHS based approach. Empirical results are provided for a policy optimization task using TRPO and a diversity-guided evolution approach comparing the behavior of when simultaneously learning and constraining with, in the case of TRPO, the policy representation. The final empirical contributions compare the metrics and learned representation's ability to generalize state-values in the context of off-line policy evaluation.", "summary_of_the_review": "The base idea of this work is very interesting but isn't carefully examined. The theoretical results plus the illustrative experiment might not be sufficient on it's own, as is. The other contributions, the approximate policy representation learning approach and accompanying experimental results, don't seem particularly insightful and, in some cases, significant. For this reason, I'm on the fence as to whether this is ready for publication but I'm open to discussion.", "strength_and_weaknesses": "High-level comments\n===============\n\nThe propose categorization of policy abstractions are interesting. Seeing as they are inspired by similar efforts for state abstraction and, more generally, by existing formalism of abstraction, I would expect these definitions to be provide useful categorizations future work can build off of. The accompanying theorem further strengths the theoretical contributions and my only complaint is that I which there was more.\n\nMy biggest issue is that many of the experiments don't seem to provide much insight on the properties of these abstractions. The empirical contributions are probably the weakest aspect of this paper. Overall, these contributions mostly involve retroactively interpreting and attributing meaning to the results of the experiment through the lens of the proposed abstractions, rather than focusing on more targeted experiments which provide insight on the benefits and limitations of these abstractions, e.g., with domains designed to highlight stark differences. When scaling to complex settings with approximations and many design decisions, it becomes difficult to know if some of the patterns we see are due to interpretation the author's provide or if it is just a result of the particular combination of choices and domain.\n\nDetailed comments and questions\n==============\n\nWhy is definition 4.3 based on distributions over returns while definition 3.3 is defined on expected returns, i.e., state values?\n\nWhat discount factors are used? Notably, in Figure 1?\n\nSec 4.3 was hard to follow and took me a fair bit of effort/time to understand what the authors were doing. It feels like this can be explained a bit better. Maybe making equations more explicit or a diagram would help?\n\nIn addition to targeted experiments, experiments that vary fewer variables, e.g., small methods but different environment, can also provide some evidence if trends are robust. However, the empirical results found in Figure 2 vary both domain and the optimization method possibly conflating any trends.\n\nAm I mistaken in saying that there is no original TRPO (which uses max KL-divergence as proxy for total variation if I recall correctly) as baseline?\n\nWhy does the performance of several of the TRPO variants decreasing?\n\nFigure 2, why plot only 1/2 standard deviation?\n\nSec. 6, many of the comparisons seem to make strong conclusion from what seems like insignificant differences in results. In some instances, differences are well below the standard deviation. I don't think much can be said from these results, especially in the \"strong generalization\" case which uses only 5 seeds.\n\n\nMinor nitpicks\n===========\n\nParagraph 2, sec 4.2, should probably use Gretton et al. 2012 as main reference.\n\nEq. 3, might be worth mentioning that the biased estimate was used instead of the unbiased one.\n\nlast para, sec 4.3, \"policy buffer\" doesn't tell me where they came from. I would recommend saying something like \"past observed policies\" or abstract it away more explicitly, e.g., \"policy samples from some given dataset of policies\".\n\nFigure 2 (and several other places), \"std\" what? Standard error or standard deviation? This should be stated fully.\n\nFigure 2, caption probably should mention that the two subplots show results for different environments (i.e., shouldn't be directly compared).\n\nTable 2, it might be worth discussing training error when commenting on generalization. Do cases with the best test errors also have the best train error? The current table makes this tedious to deduce. The test and train error could be reported instead, using highlighting to show better gap. Alternatively, just discussing the train error in the text would do.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "p9d3wUURI9P", "reviewer": "Reviewer_DwZy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper first highlights the observation that the policy space size can be prohibitive for policy search and off-policy evaluation when the  state-action space is large (e.g., high-dimensional continuous). Then, the paper aims to propose a unified theory of policy abstractions for determining equivalence between given pairs of policies analogously to the unifying state abstraction theory of Li et al. (2006). Three distinct policy abstractions are identified based on equality criteria for (i) action distributions, (ii) next-state distributions and (iii) expected value when the agent's behavior is conditioned on the respective policies. A theorem that attempts to establish a coarseness-based partial ordering between said abstractions (as well as two trivial extremal abstractions) is presented. The abstraction equivalence relations are relaxed to similarity metrics via an arbitrary statistical distance. Three GridWorld MDPs are considered to provide intuition on the differences between these similarity metrics. A policy representation learning approach is proposed: the approach minimizes an MSE error between policy embedding distance and an MMD-based realization of the target policy similarity distance. Empirical evaluations investigate the use of these metrics in the context of policy search on trust-region policy optimization (1st order) and diversity-guided evolutionary strategies (0th order) for GridWorld environments. Further empirical evaluations on Off-Policy Evaluation (OPE) performance are provided. These measure the ability to interpolate and extrapolate policy performance from policy parameters without interaction with the environment given prior policy-value pairs that are used to learn policy representations that respect policy distances in latent space.", "summary_of_the_review": "My current recommendation for the paper is rejection. I am not convinced of the significance and validity of neither empirical nor theoretical results to a satisfactory extent, and further believe that a major revision is required to improve the writing and organization. ", "strength_and_weaknesses": "Weaknesses:\n\n1. [Major] I think that the writing of this paper could be significantly improved and it needs copy editing (frequent grammatical mistakes, awkward wording, etc.).\n\n2. [Major] Simply put, Thm. 3.1, which is the main theoretical result of the paper, is factually incorrect and therefore misleading as it stands. As discussed in the Appendix, as soon as the expected immediate reward depends on the action selection, $f_{P^{\\pi}}(\\pi_i) = f_{P^{\\pi}}(\\pi_j)$ does not imply $f_{V^{\\pi}}(\\pi_i) = f_{V^{\\pi}}(\\pi_j)$. The discussion in the Appendix downplays the importance of this case as a minority among interesting environments, but one can easily imagine, for instance, an Atari game (hardly a minority) where taking different actions in a given state yields different rewards despite transitioning the environment to the same next state. For a theorem to be mathematically correct, its surrounding conditions and assumptions must be listed explicitly. These cannot be deferred to the Appendix. The appeal of a simpler expression cannot take precedence over facts.\n\n3. [Major] The formulations of the *-irrelevance abstractions and metrics are fairly trivial and should be readily apparent to a savvy RL audience, as are the coarseness relations among them that do hold true. For instance, it's quite obvious that $f_\\pi \\succeq f_{P_\\pi}$ since $\\pi_1(a|s) = \\pi_2(a|s), \\forall a \\in A \\Rightarrow P_{\\pi_1}(s'|s) = P_{\\pi_2}(s'|s)$.\n\n4. [Major] I would have expected significantly more discussion on the choice of the state distribution $p$ over which the expectation of the distance is taken to compare policies. In Fig. 1, the doorway environment is used to dismiss $d_\\pi$ as a metric that fails to show the difference, but as noted in the paper itself, this is only the case because a uniform $p$ is used. Then, isn't this a failure to choose the right $p$ for computing $d_\\pi$ rather than a failure of $d_\\pi$ itself?\n\n5. [Major] It's not clear to me whether increased stochasticity in Fig. 1 indeed makes \"for a better evaluation\" and what \"better\" means. For example, in the case of $d_{V^\\pi}$ the evaluation no longer makes sense as the value of both policies decay rapidly with increasing stochasticity since the probability of reaching the goal decays. Given both value functions $V^{\\pi_1}$ and $V^{\\pi_2}$ decay, it's not at all surprising that their difference would as well. Note that this can also be remedied by choosing $p$ to be concentrated around states adjacent to the goal.\n\n6. [Major] The empirical evaluations on policy optimization only cover GridWorld environments. This would have been acceptable for a more theoretical paper, but the theory here is fairly simple so I would have expected evaluation on a larger number of more serious benchmarks (e.g., 3D continuous control). Furthermore, the shaded confidence areas in Fig. 2, especially 2a, show significant overlap between various approaches, which make it difficult to draw meaningful conclusions or be convinced of the claims made in the paper. Even if we take the mean curves, none of the policy metric learning approaches seem to significantly improve over the vanilla TRPO approach (which uses the maximum TV distance to measure policy distance). This result is once again deferred to the Appendix Fig. 4, but must be included in Fig. 2 in the main text in my opinion.\n\n7. [Minor] Both, IDP and LLC are listed as MuJoCo environments, but I believe only IDP is a MuJoCo environment. The LunarLanderContinuous-v2 environment should be a simpler 2D environment from Brockman et al. (2016) to the best of my knowledge.\n\n8. [Minor] It would have been nice if the paper engaged more with the literature on state abstractions and state similarity metrics.\n\n9. [Minor] Near the end of Sec. 3.2, it is noted that a common choice for $D$ could be the KL divergence, but this is inconsistent with Def. 4, which requires that $D$ is a (pseudo-)metric. KL divergence is neither a metric nor a pseudo-metric, but a _divergence_. \n\nStrengths:\n\n- I appreciated the importance of studying policy abstractions and the attempt at providing a unified theory. \n- I think the paper includes some interesting ideas and has a good set of initial experiments, which would make for a good publication after a major revision with better writing, more extensive evaluation and more precise claims & mathematical statements.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ozi69UgSphy", "reviewer": "Reviewer_LtQT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on providing a theory and methodology on policy abstraction and representation to reduce the high complexity of policy space in the Markov Decision Process. To achieve this, they first proposed a unified policy abstraction theory and discussed three types of policy abstraction. Then they generalize these policy abstractions to three policy metrics that can quantify the distance of policies instead of only binary signals. Further, they proposed a policy representation learning method based on the policy metrics. Empirically, they conducted experiments in both policy optimization and evaluation problems, which demonstrates the effectiveness of the proposed representation method. ", "summary_of_the_review": "This paper is a very complete work about policy abstraction and may inspire future related works. Although the optimality of the policy abstraction is not guaranteed or discussed, this paper still fill up the plank in both the theory and methodology of this sub-field. ", "strength_and_weaknesses": "**Strengths**\n\n1. The problem this paper considers is very important, and this paper provides a new method to solve it.\n2. The literature review is sufficient. This paper has offered a detailed table of a taxonomy of prior policy abstractions under the policy abstraction theory in the Appendix. \n3. This work is well-motivated. To solve the challenge of large scale and high complexity of policy space, policy abstraction representation might be a very promising way. And this work provides a unified theory for this line of works. \n4. The empirical evaluation and theoretical results are of high quality. (1) Figure 1 is especially interesting and it clearly showcases the differences between the three policy metrics. (2) The definition of abstraction fineness is also interesting and might be a key property of policies. \n\n**Weaknesses**\n\n1. The empirical results are significant in some easy tasks and some specific algorithms. These results might be enough for this paper but the reviewer is still wondering whether there is a unified way to use policy abstraction to other algorithms that do not need policy metrics explicitly, such as PPO, for policy learning not policy evaluation. In other words, can $f_\\psi$ in Equation 1 be directly used for policy learning?\n\n2. This paper does not provide enough discussions of limitations. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "66rV76OmprE", "reviewer": "Reviewer_M8fJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the metrics (or abstractions) on the policy space for a given MDP. The paper defines three pseudo-metrics on the policy space: $d_\\pi(\\pi,\\pi’)$ that measures the distance between the outputs of $\\pi$ and $\\pi’$ given a state, $d_{P^\\pi}(\\pi,\\pi’)$ that measures the distance between the distributions of next state given a state and policy $\\pi$ (and policy $\\pi’$), and $d_{V^\\pi}(\\pi,\\pi’)$ that measures the difference between the value functions of $\\pi,\\pi’$. This paper also provides an algorithm that learns a representation of policies such that the l2 distance of the representations is approximately equal to a given metric. Empirically, the metrics on the policy space can be used to improve algorithms such as TRPO and DGES on grid world environments, and can help value generalization on off-policy evaluation tasks.", "summary_of_the_review": "My main concern is that the contribution of this paper is limited both theoretically and empirically. As a result, I recommend a rejection for the current version of this paper. \n\n==== after rebuttal ====\n\nI thank the authors for addressing my concerns regarding the relevance of policy abstraction in prior works. However, I am still not convinced that the contribution of this paper is significant enough. Therefore, I raised my score slightly, and encourage the authors to evaluate the policy abstraction algorithms systematically on benchmarking environments.", "strength_and_weaknesses": "### Strengths:\n\n- The question of studied in this paper is interesting and relevant to many deep RL algorithms. For example, TRPO iteratively optimize its policy in a small neighborhood, whose definition requires a metric on the policy space. Different metrics induces different algorithms and it’s interesting to see how the choice of the metrics affects the performance.\n- The results for value generalization on off-policy evaluation tasks are interesting, and raises several questions to ask in future works: does the same conclusion hold for more complex environments? Is it possible to use this method to off-policy optimization tasks? Does the method improve over other baselines?\n\n### Weaknesses:\n\n- The contribution of this paper is somewhat limited. The theoretical results are rather  straight-forward, and the empirical results are limited to grid-world environments (for policy optimization) and simple Mujoco tasks (off-policy evaluation). For the policy optimization experiments, the results only demonstrate that, unsurprisingly, the performance of TRPO and DGES depends on the choice of metrics. For off-policy evaluation, this paper only tests two tasks from the Mujoco environment.\n\n- It is unclear to me whether a policy abstraction is necessary for online RL. Many algorithms for online policy optimization do not require a metric on the policy space, and whether TRPO with a good metric can outperform those algorithms remains a question.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject