SPP-RL: State Planning Policy Reinforcement Learning
Authors: Jacek Cyranka
Zuzanna Opała
Jacek Płocharczyk
Mikhail Zanka
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
off-policy
constrained optimization
robotics
state planning
state-state
mujoco
safety-gym
antpush

Abstract: We introduce an algorithm for reinforcement learning, in which the actor plans for the next state provided the current state. To communicate the actor output to the environment we incorporate an inverse dynamics control model and train it using supervised learning. 
We train the RL agent using off-policy state-of-the-art reinforcement learning algorithms: DDPG, TD3, and SAC. To guarantee that the target states are physically relevant, the overall learning procedure is formulated as a constrained optimization problem, solved via the classical Lagrangian optimization method. We benchmark the state planning RL approach using a varied set of continuous environments, including standard MuJoCo tasks,  safety-gym level 0 environments, and AntPush. In SPP approach the optimal policy is being searched for in the space of state-state mappings, a considerably larger space than the traditional space of state-action mappings. We report that quite surprisingly SPP implementations attain superior performance to vanilla state-of-the-art off-policy RL algorithms in the tested environments.

Under review as a conference paper at ICLR 2022
SPP-RL: STATE PLANNING POLICY REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce an algorithm for reinforcement learning, in which the actor plans for the next state provided the current state. To communicate the actor output the environment we incorporate an inverse dynamics control model and train it using supervised learning. We train the RL agent using off-policy state-of-the-art reinforcement learning algorithms: DDPG, TD3, and SAC. To guarantee that the target states are physically relevant, the overall learning procedure formulated as constrained optimization problem, solved via the classical Lagrangian opti- mization method. We benchmark the state planning RI. approach using a varied set of continuous environments, including standard MuJoCo tasks, affetygymm level 0 environments, and AntPush. SPP approach the optimal policy is be- ing searched for in the space of state-state mappings, considerably larger space than the traditional space of state-action mappings. We report that quite surpris- ingly SPP implementations attain superior performance to vanilla tate-of-tthearr off-policy RL algorithms in a wide class of robotic locomotion environments.
INTRODUCTION
Research on reinforcement learning (RL) has brought tremendous number of successful appli- cations in diverse fields of science and technology. Application areas of RL can be split into two classes: discrete and continuous. Here, we are interested in continuous simulation environments, mostly in the robotics domain. Despite the magnitude of successful applications of RI. in this do- main, two of the main issues: sample efficiency and interpretability of RI. trained agents, persist. These problems have utmost practical importance, especially for mission-critical applications. The current methods often require a vast amount experience for training. The decision-makin pro- cess of trained agents is not interpretable, sometimes resulting in finding proxy solutions. Thus, it is vital to research new RL algorithms that may partially solve the mentioned problems.
Traditionally, RL is based on the principle of searching for the optimal policy within the space of state-action mappings. We propose a new algorithm based on the principle of training an actor (a policy) operating entirely in the state space (state-state mappings). We call such policies the state planning policies (SPP), whose actions determine desired trajectories in the state-space. The task of training SPP may initially seem infeasible due significantly larger dimension of states than of actions. Nonetheless, quite surprisingly, we show that the approach is feasible and often leads to significant improvements in average performance for a class of robotic locomotion tasks.
We call our approach State Plamning Policy Reinforcement Learning (SPP-RL). It is a generic ap- proach for problems specified using continuous environments. The main building block of SPP-RL the RL agent can be implemented using virtually any RL algorithm. We chose to develop our approach using the Mfate-of-th--an off- off-policy DDPG (Lillierap et al. 2016), TD3 (Fujimoto et al.) 2018), and SAC (Haarnoja et al. 2018a) algorithms. Note that, in SPP-RL we need another train- able model to communicate the policy output to the environment; as such, we incorporate a learnable inverse dynamics control model (IDM), see Fig. [ The overall algorithm optimizes the policy si- multaneously with CM. To ensure that the policy target states satisfy physical and under-actuation constraints, formulate constrained optimization objective for policy training. Our work lies within the category of RL methods that already implemented state-state policies, including work on hierarchical RI. like Nachum et al. 2018), and the D3G algorithm from Edwards et al. (2020).

---

Under review as a conference paper at ICLR 2022
Summary of results. Although the SPP-RL algorithm searches for the optimal policy within much larger space, our performance benchmarks revealed that SPP-RL implementations often out- perform their vanilla RL counterparts. Experiments performed in Ant & Humanoid tasks from Mu- JoCo suite (Todorov et al. 2012 Brockman et al. 2016) reveal that SPP-DDPG outperforms DDPG. Experiments in -afety-Gyym Level 0 environnmnnss (Ray et al. 2019) demonstrate that SPP-TI3 and SPP-SAC outperform by a great margin TD3 and SAC, respectively. Experiments in AntPush task (Nachum et al. 2018) show that SPP-TD3 outperforms hierarchical RL method HIRO (Nachum et al. 2018) and provides some interpretability of the agent behavior.
We hypothesize that the superior performance of SPP-RL in the tested continuous environments originates in more efficient state-space exploration by state-state policies than traditional state-action policies; here noise being added to target states rather than actions. To argue this, we performed series of experiments, including evaluation of shadow agent utilizing experience from SPP and vanilla replay buffers (Sec. (S4c. and a study of the distributions of states gathered different replay buffers (App. E.2 To analyze which features of the algorithm are crucial for superior performance, we report on a thorough ablation study SPP-TD3 (App. E.T.
We implemented SPP-RL methods modular PyTorch library shared as open-source. SPP-RL algorithms are derived from their vanilla RL counterparts, making extending the library with new RL algorithms straightforward. We also share videos with test episodes of the trained agents to accompany benchmark plots (web) 2021)
Last but not least, demonstrate theoretical convergence of clipped Double Q-learning imple- mented in SPP-TD3 algorithm within the classical stochastic processes, finite-state/action spaces setting and in environments based rigid body dynamics model, refer to App. B.3
1.1 RELATED WORK
We present a (non-exhaustive) list of related works; refer to Tab. for a perspective on related work. The closest approach to ours is the D3G algorithm introduced by Edwards et al.] (2020), which includes state planning policies, and introduces a novel form of the value function defined on state- next state pairs. There are two main ways our method is distinct. First, SPP employs the classical formulation of the value function. Also, we do not include a forward dynamics model nor the cycle loss. Instead, to guarantee consistency of the policy target-states in SPP, we formulate a constrained optimization problem (compare Fig. solved via Lagrangian optimization.
Our work builds the classical RL algorithms going back to REINFORCE (Williams) 1992). Asyn- chronous Actor-Crine Crioi- (Mnih et al. 2016), and especially the off-policy actor-critic algorithms in- cluding Q-Prop (Gu et al. 2017). DDPG (Lillicrap et al. 2016), SAC (Haarnoja et al. 2018a b). and TD3 (Fujimoto et al.] 22118) State planning policies have been used in hierarchical RL (HRL) meth- ods like HIRO (Nachum et al. 2018) and FuN (Vezhnevets et al. 2017). Contrary to HRL SPP-RL approach does not employ a hierarchy of multiple policies nor state conditioned value functions.
Training predictive models (like IDMs) is fundamental for the model-based RL approach including algorithms: a locally linear latent dynamics model (Watter et al.) (201)), model- planning for discrete and continuous actions (Henaff et al., (017). model-predictive control (Chua et al.) 2018). and model based policy optimization (Janner et al. 2019). We deployed IDMs for mapping current- target states to actions: other applications of IDMs RL include the context of planning: search on
replay buffer (Eysenbach et al. (2019). episodic memory graph (Yang et al. 2020), and topological memory for navigation (Savinov et al., 2018). Existing many other applications of IDM in context of RL including: policy adaptation during deployment (Hansen et al. (021). sim to real transfer (Christiano et al. (2066), adversarial exploration (Hong et 20200). and curiosity-drived exipeeeooiieeeellottatoooo (Pathak et al. 2017).
1.2 BACKGROUND
Following the standard setting used in RL literature, we work with infinite horizon Markov decision process (MDP) formalism (S, P. r. Po: 7), where S is a state space, A is a action space, P: S x A [0, 1] is a transition probability distribution, S A R is reward function, Pa is an initial state distribution, and y E (0, 1) is a discount factor. From now on we assume that the MDP is fixed. In RI. the agent interacts with E. in discrete steps by selecting action a, for the state 8t
2

---

Under review as a conference paper at ICLR 2022
Table 1: A Perspective on Related Work
Technique
State-state policy
Inverseforwardd model
State cond. Q funet.
Policy Hierarchy
Planning horizon
SPP (ours)
yes
inverse
single policy
single
D3G
yes
inverse
yes yestupper level)
single policy
single slep
HRL.
yes(upper level)
inverse
multiple policies N/A
multiple steps
Planning
yes
inverse
yes
multiple steps
Model based
Do
forward
single policy
single stop
time t, causing the state transition St+1 (st, at), as result the agent collects scalar reward r T (St; dz), the return is defined as the sum discounted future reward Rt = E (s;,, az). The goal in RL is to learn a policy that maximizes the expected return from the start distribution.
2 STATE PLANNING POLICY REINFORCEMENT LEARNING APPROACH
Our SPP approach is rooted in state-state reinforcement learning, by which we mean setting in which RL agent trained to plan goals in the space, the approach already employed c.g., in HRL, planning. D3G RI. algorithms (see Tab. T.. In SPP state planning policy æ given the current state 8, outputs 2 the desired target state to be reached by the environment in the next step. Forcing the environment reach the desired state requires translating the target state to suitable action ar. Hence, we employ an additional model capable of mapping the current state-target state pair (s,, ze) to the action 0g (trainable) IDM model. Ideally, we like to have consistency 2 (st) = 81+1- The consistency cannot be guaranteed a-priori, is rather achieved in SPP setting by employing constrained optimization approach. A diagram illustrating SPP approach is presented in Fig.
We have freedom of choice of the particular RL algorithm (RL agent) and IDMs implementations. Currently, we use feed-forwand neural networks, and RI. Agent using implementations of the state- of-the-art off-policy RI. algorithms: DDPG (Lillicrap et al.) 2016). TD3 (Fujimoto et al., (018) and SAC (Haarnoja et al.) 2018a). We present details SPP-RL implementation in Sec. using as the example SPP-DDPG. The encountered experiences during the execution of an off-policy RL algorithm are stored in replay buffer D.
The main building block of SPP-RL are the state planning policies, intuitively state planning policy selects a desired trajectory in the state-space of the environment.
Definition 1. We call a state planning policy map my: S - P(S) parametrized using vector of parameters 0 E (R=((), and we denote no (2|s) a probability of the desired target state z E S for the given current state E S.
We call deterministic state planning policy a parametrized map ne > S, and we denote Ts(s)= z.
We assume that has continuous and bounded derivatives with respect to We will call state planning policy whenever it is clear from the context determmiissiccssochastie and omit the parameter subscript π = mp.
Besides the state planning policy (Def. D the second main building block of the overall SPP agent is model for mapping the current state-target pair (s), to suitable action az. Following the existing literature, we call such model the inverse dynamics control model (IDM), or simply the control model.
Definition 2. For a given MDP (S, Por,po, 7). Let E. S. We define the control model:
CM: x S CM(s,2) = o,
i.e. for the given two states CM computes the action a. We call s, the initial state and the target state respectively, where informally satisfies arg maxtes P(s, a, b) ~ for stochastic E, or 2 RJ E(s, a) for deterministic E.
Obviously, in order to work, SPP requires consistency of the target states generated by the policy with the actual next-states the environment, We call this property the state consistency property (or simply consistency) of refer to Fig. 2 As it may be intractable to verify SPP for all possible
3

---

Under review as a conference paper at ICLR 2022
interactions in continuous environments, the experiences stored in the replay buffer.
interested in guaranteeing the state consistency for
Property 1. Let D be a replay buffer, CM be an IDM and x be a (SPP) policy. We say that т has the state consistency property with threshold d 0f holds that
E
||0+1 - 2x S d,
for deterministic x. we have -T(+)). Given (Z,, $t-1), we call distance || st+1ll2 the state-
consistency distance (refer Fig. wwit threshold d >> 0'.
We will often assume that d is known from context and omit
actor (spp)
planned 2₁
RL Agent
Control model
control model +at
state consistency distance
St.
Environment
actual
81+1
Figure 2: Diagram presenting the idea of state consistency property, ultimately we want to achieve 2 8t+1.
Figure I: Diagram presenting SPP method
3. ENSURING STATE CONSISTENCY PROPERTY BY CONSTRAINED OPTIMIZATION
Our SPP algorithm is utilizing three parametrized models: IDM CM policy model ne, and Q- function model(s) Qs. In our current implementation, all of the models are feed-forward neural networks. One way of ensuring the state consistency property (Prop. [T] is to modify the policy training loss, such that the expected values are maximized under fixed state consistency distance penalty. However, such approach has many disadvantages, e.g., choosing appropriate learning lemperature (A) for the state consistency penalty is a very delicate issue, see the ablation study in Appendix E. It is easy notice that setting its value too low would result in π biased towards nonphysical larget-states. On the other hand, setting its value too high would make overly con- servative for off-policy states. Hence, we find a solution relying on constrained optimization more appealing for the studied problem. Namely, the objective for policy training is to maximize the sum of discounted rewards assuming a fixed threshold for the state consistency distance.
Definition 3. Let π be a state planning policy (Def. D CM be a control model (Def. 2}. 'D be the replay buffer with experience generated by executing off-policy RL algorithm. In particular 2 'ss are target states evaluated on-the-fly fly by ı (susceptible to be changed during the course of algorithm), and 1 ' are E next-states. Let d > be fixed hyperparameter.
We define the constrained objective for state planning policy as follows
max
E TWE,CM
Σ 2 ai)
(la)
s.t.
E (ED (
||0c+1- 10011-zell =2112 S <d,
(lb)
where d is hyperparameter for determining the allowed threshold for the expected divergence of predictions from actual next-states, equation Ia is an expectation over the policy trajectories gener- ated by both of the state planning policy and IDM (trajectory composed out of tuples (St, Rt, az)). The whole optimization process of equation Ta being performed off-policy, see Sec.
The constrained objective in equation I is being solved using the standard Lagrange multiplier method. The max-min Lagrangian objective C(T, for the constrained optimization problem takes
the form max, minx>0 L(T, A) = E [R((T)]
E
d For more ||tt1 2x12 (.)(())))
details refer App. C.1
4

---

Under review as a conference paper at ICLR 2022
4 ALGORITHM IMPLEMENTATION
We briefly present here details of the SPP Algorithm implementation, more detailed discussion can be found Appendix [ We implemented SPP-DDPG, SPP-TD3, and SPP-SAC as a modular Python library within PyTorch framework (Paszke et al.]| 2019). All gradient optimization steps were performed using the Adam optimizer by Kingma & Ba (2014). Many the algorithmic choices motivated by the Spinning Up RL on-line resource (Achiam, 2018). We publish the modular SPP-RI. software package as open-source (web) 2021). For illustrative purposes, we present the pseudo-code of the full SPP-DDPG algorithm in Algorithm [ SPP-SAC and SPP-TD3 algorithms are presented in Appendix C We provide theoretical derivations concerning SPP-RL algorithms, including convergence in the classical finite setting and policy gradient analogues to vanilla RL in Appendix B We emphasize that SPP algorithms are not using any extra samples, i.e. the samples utilized for ICM training are added the buffer and then reutilized for RL training, and if the buffer is full new samples are not being added anymore. An important caveat of our policy implemen- tation in SPP-DDPG, not present in the vanilla DDPG, is that the output ı is being normalized in order to reflect the physical bounds. Contrary to vanilla DDPG where outputs actions within well-defined uniform bounds, SPP a suitable normalization of target state æ output is being com- puted online- depends network (refer to Appendix foror details) with tangential outputs bounded within [-1, 1]. Hence, we normalize the output of neural network by utilizing the current mean and min/max values of the past observations in replay buffer D. We recompute mean and min/max values after each episode of the algorithm. Our base implementation of DDPG algorithm is parametrized by the usual hyper-
the past E' observations. Implementation of i is a feed-forward neural
Algorithm 1: SPP-DDPG Algorithm
input :nvironment E; initial model parameters 0, Φ, t/; state planning distance threshold d; empty replay buffer D; the DDPG algorithm hyperparameters
output: trained model parameters 0, vi; total return repeat
Sample random action Ut;
Store experience (St, At, 2t St+1, Tr+1, S(+1) 1) in replay butfer D; (use next-state as initial actor actions)
until random exploration is done:
repeat
if buffer 'D not full then Compute actor prediction 2, (sx)) where A;
Compute action a, = (M 2,) and observe reward re+1 next state 8t+1; Store experience (St, zt, de, $141, Fc41) in DD end
if it's time to update CM then
Sample {b. {((t, 1041), )) b batches of samples from replay buffer D;
SGD train CM using the batches and MSE loss;
end
it's 's time to update actor and critic then for update steps do Randomly sample B = {(Ss,, 2z, az, (241)) set of batches from 2>;
Compute ās+1= CM(st+1, xt (8++1));
Compute targets F141 + QQu (S141, ux41) (using target parameters dtarg):
to Update e = -- isi
V Ee
Q (st, at)
Update policy parameters (ascent w.n.t 0 of max-min max-min Lagrangian obj.) 0
0 1
VoΣB Q Q.. (st, ax)
Vo Es ||sc+1 TO mo()))l
[9]
Update (descent w.r.t. À of max max-min Lagrangian obj.)
A A + L
[ Es || To To
Update actor & critic dearg = (1 T)starg Th: Ororg = (1 - 7)0rar + T0;
end
end
until convergence:
5

---

Under review as a conference paper at ICLR 2022
parameters including episode length, update batch size, Polyak averaging parameter (7), actor and critic learning rate maximal episode length, number of test episodes, 7. To ensure that we perform minimization equation 6 within the domain of positive À values, optimize the parameter of the softplus function. All relevant hyper-parameters are provided in Appendix [
We performed a thorough ablation study of the different features implemented in SPP-TD3 algo- rithm including the Lagrangian multipliers method (and different values of fixed X), target state normalization, state-state critie (Q(sss')) we report results in Appendix E.1
5 EXPERIMENTAL EVALUATION
To show the feasibility of our method, we performed experiments on set of benchmarks using continuous environments, most of them having large space dimensions. We performed all of the reported experiments using the default vector state input. We show that SPP-RL implementations compare favorably to their vanilla RL counterparts. As our SPP approach differs considerably from the vanilla off- off-policy RL, performed thorough values from the actual SPP implementations in Appendix For the sake
sweep from scratch. We pro-
vide the of presentation we share videos with example test episodes rendered using the trained actors and high-resolution benchmark plots. All of our experiments are reproducible, we share the sources, training. evaluation, and trained models online (web) 202I). All of the reported experiments were run using CPU only, and single experiment was always run on single CPU core, i.c., we have not performed collecting experience in parallel. The experiments were performed on an example ma- chine: AMD Ryzen Tr. 1920X, 64 Gb RAM, Ubuntu OS 1 18.04. Example average time of execution of 10 steps stands at SPP-DDPG 5hrs 58', DDPG 3hrs, 7', (SPP-)SAC 19hrs 20', SAC 11hrs.
5.1 CLASSICAL MUJoCo
First, to measure the performance of SPP-RL we run benchmarks in the MuJoCo control tasks (Todorov al., 2012) from OpenAI gym (Brockman et al. 2016): Ant and Humanoid. The ex- periments were performed using solely the vector state input. We present the results in Fig. 3 We compare the performance of SPP implementations with vanilla RI, implementations in terms of av- erage test episodes return. We emphasize that SPP algorithms do not increase the overall sample efficiency of the RL procedure. The samples utilized for ICM training are added to the buffer and then reutilized for RL training, and if the buffer is full new samples are not added anymore (refer to Algorithm D The algorithm does not take advantage of a pre-trained ICM before the actual RL training. rather optimizes and trains ICM simultaneously. However, as reported in Fig. B guite sur- prisingly, SPP based RL algorithms perform on par (TD3 & SAC case) or significantly better (DDPG case) than corresponding vanilla RL. implementations in the MuJoCo environments. Analyzing the case of (SPP-)DDPG Figs Ba 3d reveals that SPP significantly outperforms the vanilla implementa- tion, as the vanilla implementation does not converge on those envs (compare with aathaamhiam| (2018). ). Analyzing the case of (SPP)-TD3 reveals that for Ant env (Fig, 3b) SPP-TD3 performs on par with vanilla TD3 and outperforms by far D3G the most related method. For Humanoid env (Fig. Bel we see that SPP-TD3 outperformed vanilla TD3, the visible high variance of vanilla TD3 caused by a single run, which did not converge, D3G do not converge in this env. There is visible divergence of the results we obtained for D3G in Ant from those reported by Edwards et al. (2020). We used MuJoCo v. 1.5 library version that is known to be compatible with OpenAl gym; is known that using v.2.0 results in zeroing out part of observation space (in Ant & Humanoid) (gym) 2019) which may alter the results.
5.2 SAFETY-GYM (LOCOMOTION TASKS)
In the next set of benchmarks, we use environments from the safety-gym gym suite by Ray et al. (2019). Currently, we employed only Level 0 environments (which does not involve the cost function for vi- olating the safety). We find Level 0 tasks from the safety-gym suite as the perfect ground to study the performance of SPP-RL in robotic locomotion environments, the goal being to steer agents (robots) to solve planar goal-reaching tasks. Moreover, dimensionality of the state (and actions) space rather than maximizing returns under safety con- straints. The experiments were performed using solely the vector state input. We leave investigating
concentrate difficulties arising from higher

---

Under review as a conference paper at ICLR 2022
(a) Ant, (SPP-)DDPG
(b) Ant, (SPP-YTD3
(c) Ant, (SPP-)SAC
(d) Humanoid, (SPP-)DDPG
(c) Humanoid, (SPP-)TD3
(f) Humanoid, (SPP-)SAC
Figure 3: Experimental comparison of SPP implementations with corresponding vanilla off-policy RL on the two most complex MuJoCo benchmarks Ant (upper row) & Humanoid (bottom row). Figures show test return computed every 5k frames averaged over 10 different seeds. The continuous curve the mean, whereas the faded color regions represent std. deviation. SPP RL algorithms in blue, corresponding vanilla RL in orange, only case of TD3 Fig. 3b] [e include results obtained using D3G implementation in green. Refer to Appendix for exact hyperparameters. Dimensions: Ant: dim(S) = 111, (AA) = 8; Humanoid: dim(S) 376, dim 17.
the higher-level environments considering the cost function as a topic of future research. We chose subset of the most challenging Level 0 tasks, including Car-Push, Doggo-Goal, Doggo-Button en- vironments. We also create a custom environment (termed Doggo-Colummn based on Doggo-Goal with additional 10 fixed pillars placed in the arena, obscuring the paths toward the goal.
We evaluate our SPP-TD3 implementation against state-of -ftt---rrttttt off off-policy algorithms like TD3 and SAC. We do not include on-policy algorithms like PPO (Schulman et al. (017) or TRPO (Schul- man et al. 2015), as their performance in the studied environments is inferior to off-policy, refer to Ray al.
2019) and the project web-page.
The results presented in Fig. Fig-A clearly show that SPP-TD3 is superior to vanilla off-policy algorithms within the studied safety-gym environments. Also, there is noticeable difference the learned behavior of the trained agents, The agents trained using the SPP-RL approach show smarter and more efficient behavior; for instance, the trained using SPP doggo robot learned an efficient gait of moving backward to mark goal or press button. For comparison, we publish videos of the trained agents online (web) 202T).
We argue that the performance boost exhibited by SPP-RI. algorithms over vanilla counterparts is due improved exploration in some sense. Sec. 5.4 we show results from evaluating TD3 shadow agent, i.e. vanilla TD3 agent utilizing for training some portion of experience from SPP- TD3 replay buffer. In Appendix E.2 we investigate differences distribution states collected by both of the methods.
5.3 HARDER EXPLORATORY TASK ANTPUSH
In order to demonstrate interpretability aspects the SPP approach, we describe an experiment in AntPush environment from Nachum et al. (2018). The experiments were performed using solely the vector state input. The task to control the ant such that reaches the goal. The goal is hidden within chamber behind block. Therefore, Ant needs to learn to walk around the block and push to the right first before eventually reaching the goal. Success defined as finishing the episode within radius 5 from the goal. We benchmark SPP-TI33 against the state-of-the-art
7

---

Under review as a conference paper at ICLR 2022
Doggo-Columne (custom), (SPP)ID3 SAC
(a) Car-Push, (SPP)TD3 & SAC
(b) Doggo-Goal, (SPPYTD3 & SAC
(c) Doggo- Button, (SPPYTD3 & SAC
Figure 4: Experimental comparison SPP-TD3 with corresponding vanilla oof-cy RL on set of safety-gym level 0 environments. Figures show test return computed every 5k frames averaged over 10 different seeds. The continuous curve is the mean, whereas the faded color regions std. deviation. SPP-TD3 algorithm blue, vanilla TD3 orange, SPP-SAC red, vanilla SAC green, D3G purple. D3G did not converge (return oscillated around zero, or it diverged in CarPush to large negative score removed from the plot for clarity). Refer Appendix D for exact hyperparameters that used to perform those experiments.
hierarchical RL HIRO method by Nachum et al. 2018). Specifically, we used the implementation by Qin 2021). Instead of reporting the achieved success rate of single training run like in Nachum et al. (0188). which may be spurious if a lucky seed is chosen, we report the mean and std. of the AntPush success rate using 10 random seeded training runs. Our experiments revealed that HIRO is highly susceptible to the random seed used. Only single HIRO agent out of 10 trained using random seeds in total achieved a positive success rate, comparing to 7 out of 10 SPP-TD3 agents successfully learned to solve the task. The performance reported in Fig. Sa shows SPP-TD3 eventually superior to HIRO. Example two solution paths are marked on Fig. 56 (blue curves). The right path is suboptimal Ant blocks the entrance to the chamber where the goal is. The left path is optimal, Ant traverses to the left to push the red block away and open the passage towards the goal (green arrow).
Finally, Figs 5c 5d show the obtained paths in the state space (blue dashed), and the policy target states 2, s (orange solid), only the first two coordinates corresponding to the position of the Ant body in x, y coordinates are illustrated. Observe that the case of the suboptimal path in Fig. 5e the planned path diverts to the left from the actual path (blue dashed), which indicates that the agent learned and attempted the correct behavior of pushing the red brick away and successfully open the entrance the goal. In this case, however, the block is not movable; it is stuck as it was pushed forward before, hence as we see, the actual path the state-space diverts in the middle. Figs 5c 5d show that apparently, the policy target states path being more erratic than the actual path in the state space. Erratic behavior can be mitigated by adjusting the hyperparameter d in equation lb (the smaller d, the closer the paths will be). Nonetheless, the policy target paths (orange) in Figs 5c 5d could be potentially used to cluster agent behavior, qualitatively differentiating two example agents executing (sub)optimal path. This information could then used pick appropriate agents for deployment.
5.4
MORE EFFICIENT EXPLORATION IN SPP-RL
Our experimental evaluation using the safety-ga gym environments show that SPP-RL implementations outperform by a great margin their vanilla ofl-py RL counterparts (TD3 and SAC) in terms of the average returns (See Fig.4). The performance boost is also visible in case of DDPG in MuJoCo
Ant (Figs. 3a 3d
In this section, we argue the performance boost of SPP-RL compared to the vanilla RL counter- parts. Our intuition is that exploration performed in the target-state space rather than in the action space may be more efficient in some cases. Exploration using SPP policies results in more viable experience being collected in the replay buffer, leading to more efficient Actor & Critic training. is also possible that the constrained optimization induces some kind of curriculum. To confirm the mentioned intuition, we evaluated the performance of a TD3 shadow agent i.e., a vanilla TD3 Ac- Critic trained using (partially) experience collected by the SPP-TD3 agent. Both of the agents
8

---

Under review as a conference paper at ICLR 2022
(b) Goal in AntPush env (green arrow), and two possible paths (blue curves)
suboptimal Ant path example the state space (blue dashed) and target states (orange solid)
(d) Example of an optimal path in the state space (bluc dashed) and target states (orange solid)
(a) AntPush, SPPTD3 & HIRO
shows success rate for SPP-TD3 & HIRO computed every 5k frames averaged over 10 different seeds (10 indepen- dent training runs were used). The continuous curve is the mean, whereas the faded color regions std. dev. Fig. [5b] 5b) show (blue dashed) and target states (orange solid), the coordinates describing the position of the Ant body (z, 3) are used.
Figure 5: Experiment in AntPush environment from Nachum et al. (2018) Fig. 5a
possible solution Ant paths. Figs [c| show show the paths in the state space
were trained in parallel. The TD3 shadow agent updates were performed using samples drawn from two the replay buffers. The replay buffers of the SPP-TD3 and TD3 agent were used according to 50/50 ratio. The results are presented in Fig. 6 Such TD3 shadow agent outperforms vanilla TD3, and eventually, its performance matches SPP-TD3 agent's in all of the studied safety-gyn environments, excluding Doggo-Button.
Additionally, to address the question of how the distribution of experiences gathered by vanilla RL/SPP-RL compares each other, performed additional experiment in Appendix E.2 We present plots the discretized distributions of states encoded using a random encoder and cross- entropy of two distributions w.r.t. the quantity of gathered experience.
(d) Doggo-Colume (custom), (SPP)TD3 SAC
(a) Car-Push, (SPP)TD3 & SAC
(b) Doggo-Goal, (SPP)TD3 & SAC
(c) Doggo-Button, (SPP)TD3 & SAC
Figure 6: Experimental evaluation TD3 shadow agent, i.e. vanilla TD3 trained utilizing experi- ence collected by SPP-TD3 agent, on set of safety-gym level 0 environments. Presented metrics are same as in Fig. Shadow agent marked with the darkest color, vanilla TD3 and SPP-TD3 using lighter colors respectively.
CONCLUSIONS
We presented the State Planning Policy Reinforcement Learning (SPP-RL) approach for rein- forcement learning, where the policy is selecting target states. Experiments performed on continuous benchmark environments often show the superior performance of SPP-RL compared to state-of-the- art off off-policy RL algorithms. There are various avenues for future work pertaining to this research. One path is to include in approach physically informed control models. Another, important path of work is to implement a long-term policy planning method scheme and application the safety RL. setting of SPP approach.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
To ensure reproducibility of our results we published a supplementary material web page available https://sitts. woaittess.ooole..ooo wooglle.com/view/spee. there link to public github repository. The repository contains the sources, training & evaluation scripts, and trained models. The webpage also contains benchmark plots higher resolution, and also include videos with agents trained using different methods in the considered environments.
REFERENCES
Openai gym issue related to newer mujoco lib versions. gym/ issues/ 1541 2019.
http::://.thub..com/////
Spp-rl supplementary material webpage. 2021.
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Dimitri P. Bertsekas. Dynamic Programming and Optimal Control, volume I. Athena Scientific, Belmont, MA, USA, 3rd edition, 2005.
V.S. Borkar. An actor-critic algorithm for constrained markov decision processes. Systems Con- trol Letters, 54(3)2207-213, 2005, ISSN 0167-6911. doi: hutp:///o..og///110066/.....ooo/ 2004.08.007 URL 5016769110400122
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.015400, 2016. URL /RR./ttt/////xiirrrr
abs/1606. 01540
Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojeiech Zaremba. Transfer from simulation to real world through learning deep inverse dynamics model, 2016.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine Deep rein- forcement learning in handful of trials using probabilistic dynamics models. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL httppss/pproceeding...ee.e http::///ppoodddno.....ooecceaas (ale51100111199900000000
htt.:pp..cc/pppr//////EEE pdf
Ashley Edwards, Himanshu Sahni, Rosanne Liu, Jane Hung, Ankit Jain, Rui Wang, Adrien Ecoffet, Thomas Miconi. Charles Isbell, and Jason Yosinski. Estimating q(s.s)) with deep deterministic dy- namics gradients. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2825-2835. PMLR, 13-18 Jul 2020. URL http::tpp://ooo/o edwards20a. html
ngs. mlm. hgs.mlr.prees//1119 ptwpssv1191
Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search the replay buffer: Bridging planning and reinforcement learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL
www.neuuii.... cc/ Paw.re..pppppp..............
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. arXiv e-prints, art. arXiv: 802.09947, February 2018.
Shixiang Gu, Timothy Lillicrap. Zoubin Ghahramani, Richard E. Turner, and Sergey Levine. Q- prop: Sample-efficient policy gradient with an off-policy critic. 2017. URL https: //axxii..// brgg/bbs/1611 02247
10

---

Under review as a conference paper at ICLR 2022
Tuomas Haarnoja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1352-1361, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
wwt.ppppsssseaaaeepeeepp httpee///w////aaaaoaaaaaaaa html
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and An- dreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870, Stockholmsmässan. Stockholm Sweden, 10-15 Jul 2018a. PMLR. URL 188C/haarnn ja 9b. html]
mlr. press/
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint (rXiv:/812.05900) 2018b.
Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyå, Picter Abbeel, Alexci Efros, Lerrel Pinto. and Xiaolong Wang. Self-supervised policy adaptation during deployment. In International Conference on Learning Representations, 2021. URL/bttps://opeereivi.e net /forum?r
id=o_ V-MjyyGV
Mikael Henaff, William Whitney, and Yann LeCun. Model-based planning with discrete and continuous actions, 2017.
Zhang-Wei Hong, Tsu-Jui Fu. Tzu- Shann, and Chun-Yy Yi Lee. Adversarial active explo- ration for inverse dynamics model learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the Conference on Robot Learning. volume 100 of Pro- ceedings of Machine Learning Research, pp. 552-565. PMLR, 30 Oct-01 Nov 2020. URL http:///roceedi.ng mwr www.sess...ppr.pp.om..........ee 2ttee///100/hoog20s..ooo
Tommi Jaakkola, Michael Jordan, and Satinder Singh. Convergence of stochastic iter- ative dynamic programming algorithms. In J. Cowan, G. Tesauro, and J. Alspec- tor (eds.), Advances in Neural Information Processing Systems, volume Morgan- pttpppppp/////////
Kaufmann, 1994. URL
hwwp:eeeeedddin....ei.seee.eeeee pdf
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché- Auuc,-BBu,, Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL 679967778777 add 01099966111cc3363334334401
wwwpins.neuriss... cttpiipsscccppppe//1..... .pff
fi Glle/ffaf46ee
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL /att:://arxia orq/ abs /abs/147 11122, 6980 cite 1rxiv:1412.66 (rxxv:1412.69800ommmee 6980Commett Published as a con- ference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2016. URL hhtt:::///blppp uni teritrirrrccc de/ibrrdde/dbb... /b/
conf/iclr/ic. ctt.ellriiirlllil1o1e 6..0066.htte htw.htmlll........ HLI lil crapHPHETS15
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. /RL/
5602
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference Research, pp. 1928-1993 1937, New York, New York, USA, 20-22 Jun 2016. PMLR. URL http: 7/proceed ngs. ml www.ll.rrrrssss.. wrw.s mn hawlhhhhaa 16..666663 html
Machine Learning, volume 48 of Proceedings of Machine Learning
11

---

Under review as a conference paper at ICLR 2022
Ofir Nachum, Shixiang (Shane) Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018 URL
inforcement learning. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N.
neuri pdf
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems 32, pp. J cctpppppprcccppc
8024 8035. Curran Associates, Inc., 2019. URL httt:://pappr....tt hww.ppp...........psp........... style ott.:://eeeaaa.ooooooooooeeeeeea
9015 pdf
library..
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. by self-supervised prediction. In ICML, 2017.
Curiosity-driven exploration
Friedrich Pfeiffer and Christoph Glocker. Multibody Dynamics with Unilateral Contacts. Wiley.
Ziang Qin. Repository implementing hiro algorithm.
htttps://itthb.co com/
eianggin-stu/zz hiro 2021.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deepreinforremment learning, 2019.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. International Conference on Learning Representations, 2018. URL https:
7/ //openreview.o
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897, Lille, France, 07-09 Jul 2015. PMLR. URL hittp://ppt http:://rroeeedings .ml.. html
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, 0ads/1777.06 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on International Conference on Machine Learning Volume 32, ICML pp. 1-387-1-395. JMLR.org, 2014.
Satinder P. Singh, Tommi Jaakkola, Michael L. Littman, and Csaba Szepesvári. Conver- gence results for single-step on on-policy ceinforeement-lermiig algorithms. Mach. Learn., 38(3): 287-308, 2000. UtLhttp: hntp:::://olpp.niieeee wnreerier.........ie.e de/ wtw.d////////oaa///mm//l hattr/aaammmmmmmmmmm htt///13....mmmm..mmm SinghJLSO0
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based
control. In IROS, pp. 5026-5033, IEEE, 2012. ISBN 978-1-4673-1737-5. URL http:
de/ http:/oooooor//roo000/.....//// iit/iiios.i 1rc.2012.2222111
Lung-Wer Wen Tsai. Robot Analysis: the Mechanics of serial and parallel manipulators. John Wiley & Sons, New York, 1999. ISBN 0-471-32593-7.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 3549. Jor,. 2017.
12

---

Under review as a conference paper at ICLR 2022
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Em- bed to control: A locally linear latent dynamics model for control from raw im- ages. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems 28, pp. 2746 2754. Curran Associates, Inc., 2015. URL http::/ ni nips cc/paper/ locally linear latent dynamics mode for control - from raw
5964 to- pdf
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Mach. Learn., 8(3 8(3-4):229-256 May 1992. ISSN 0885-6125. doi: 10.1007/ BF00992696. URL httthttt///ooot wttp://oo///....//0000000.00
Ge Yang. Amy Zhang, Ari S Morcos, Joelle Pineau, Pieter Abbeel, and Roberto Calandra. Think, act and learn episodic memory graph. In ICLR 2020 workshop: Beyond tabuda in RL. (BeTR-RL.), 2020.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
Videos visualizing the trained agents, and link to the git software repository with SPP-RL algorithms implementations are available online
B THEORY OF STATE PLANNING POLICY RL ALGORITTMM
In this section we establish foundations of SPP-RL theory.
B. STATE-VALUE AND ACTION-VALIA VALUE FUNCTIONS
In order to employ in SPP RL agents based on the state-of-the-art RL actor-critic algorithms, start with defining the classical state-value and action-value functions SPP setting.
Recall Rg = E' 7 O) (8is a;) is the sum of discounted future rewards. The classical Q-function
("{,,a))) = Bri acse>v [ ozl,
however, in our setting we use state planning policies (Def. D and we do not directly sample an action using the policy a, л. Instead we sample target state z; æ. Then the actual action ag is being computed a; = (M(s;, Z:).
Hence, the Q-function in SPP setting can take the form
Q z4) er sisen a;=CM(+.,))) (Re at];
i.c the expected value of following the trajectory planned by the policy æ in the state-space, and using IDM CM.
As we use deterministicc IDM map CM: (St, 21) + de, we further reformulate equation 2 into
Qu. at) Brosiss A CC ar].
(3)
Analogously define the state-value function V*,CM
V=,CM(s.))
[QT,CM CM(ss, Z1:-T
E.
(4)
Observe that if the parameters vector v of CM is fixed, both of the formulations equation and equation 3 are equivalent as CM maps ($t, 21) + at. However, in SPP CM is being trained, and its parameters are being changed during the execution of the full SPP algorithm. Therefore, we find the formulation equation 3 more suitable, as the established theory proves convergence of the Bellman iterates applied equation 3 towards the optimal Q
We evaluated the first variant of Q-function equation 2 in the ablation study presented in Ap- pendix E. and does not perform well. For the Q-function equation we show theoretical conver- gence within the established framework in App. B.3
B.2 POLICY GRADIENT THEOREM
this section, for the sake of completeness we include the fundamental Policy Gradient Theorem in the setting of SPP policies. We provide formula for the gradient of the standard reward function in the deterministic policy gradient setting (Silver et al. 2014). which is applied in DDPG off-policy
RL algorithm (Lillicrap al. 2016), and also generalized in SAC off-policy algorithm (Haarnoja 2018a) for stochastic policies.
et al.
Observe that the standard reward function in the deterministic policy gradient setting (I and CM are deterministic) is given by
J(O)=
E 8
[QT. CM(s, (s))))
(5)
where d ) is the stationary discounted state distribution.
14

---

Under review as a conference paper at ICLR 2022
Let the reward in the deterministic setting be given by equation Assume CM is deterministic and
differentiable, rest of assumptions like in (Silver et al. 2014 Thm. 1). Then, the formula for the standard policy gradient follows from (Silver al. 2014 Thm.
VoJ(O) = E E Vomo(s) V,CM(8,
VaQ(s, (0((8,))==20)) (0)0)))))==(()00==)))))))
srida
It is straightforward extension of the formula given in Siiver et al. 2014 Thm. 1), applying one additional step the chain rule.
Observe that in algorithm we compute the policy gradient of the Lagrangian objective L(T,, A) the deterministic setting, i.c.,
[( ((0) -
E ||ss
mo())
d
(a,e))D
where an experience replay buffer (given), and the standard reward function in equation The gradient w.r.t. 0 of the second term satisfies
| 1o5'= -d
AVg
(ane(s) ||s - -2112 (S,""JED
(ED
by applying the Leibniz integral rule to exchange order of derivative and integration, and continuity of Ty and its derivative. particular, note that the next state sampled from the buffer and thus does not depend on the policy.
Therefore, holds that
VoL(To, = E andd
VeTo(s) V.CM(s,2) - VoQ(s,a))===ddd a c
E (a.e))ED
VoTo(S) Vells'-21 21
B.3 CONVERGENCE Q-LEARNING IN SPP SETTING
In this section we argue convergence of Q-learning in SPP-TD3 algorithm within finite setting using the existing framework of convergent stochastic processes based on the classic lemma by Bertsekas 2005). Used in Singh subsequently in Fujimoto et al. (2018) 8) to prove convergence of TD3 algorithm Fujimoto et al. 2018). We provide the lemma here for completeness.
al. (2000) to prove convergence of SARSA algorithm, and
Lemma B.1. Consider a stochastic process (a,, Se,,,,F) Ft), t 2 O, where at, A,, Fr: X + R, which satisfies the equations
A++1()) (z) =
(1 =1
([-aa(())), + +a, IEX, t = 0,1,2
Let P. be a sequence of increasing ofillds such that on, Ao Po-mesuurable and at A, and Fr are Pr-meauurable,
1, 2,.. Assume that the following hold
1. the set of possible states is finite.
2. S (1(x) 1, E, (1() = oc, of) < <0 w.p.
[[E(F(()) w S x AxIlAllww + et, where E [0, 1) and e, converges zero w.p.
4. (a)) < K(1 ||A.]/w)² where K is some constant.
Then, AS converges to zero with probability one.
The proof can be found e.g. in Jaakkola al. (1994); Singh al. (2000).
Based on Lemma B. Fujimoto et al. (2018) show the convergence of Clipped Double Q-learning to the optimal value function Q", as defined by the Bellman optimality equation w.p.
15

---

Under review as a conference paper at ICLR 2022
Theorem B.2 (Thm. 1 in Fujimoto al. (2018). Given the following conditions:
Each state action pair sampled an infinite number of times.
2. The MDP is finite.
3. Y E [0,
4. Q values are stored in a lookup table.
5. Both QA and Q receive an infinite number of updates.
6. The learning rates satisfy (8 a) E [0, E, (x(5, xo, E, (ar(s, < xo with probability 1 and an (s a) O, V(s a) o (s).
7. Var[r(s, a)] < oc, As,
Then Clipped Double Q-learning will converge to the optimal value function Q', as defined by the Bellman optimality equation with probability 1.
We can apply Thm. B.2 to show convergence of Q learning in SPP-TD3 algorithm. First, observe that SPP-TD3 utilizes the same Q-functions, hence, the assumptions ....., can be transferred to SPP-TD3 setting. However, the policy part is different.
Let us focus our attention to the assumption of Thm. B.2 Observe that the classical setting the assumption is easily realizable by employing policy performing either fully random exploration or s-greedy exploration. the studied SPP setting the situation much more subtle. We do not have a-priori guarantees that the such an assumption is satisfied. Observe that even randomly exploring policy may not realize such assumption. will not satisfied if for example the IDM is degenerate and maps each pair of states to the same action.
However, we show
for each pair of reachable states there is unique action that results in the transition of the previous state to the next state,
the SPP policy outputs only realizable target states.
then assumption 1 of Thm. B. can also be satisfied in SPP-RL setting by employing a fully random or e-greedy SPP policy. The first condition above may not true for general environments, as we can easily imagine an environment, where particular state may be reached from another state by applying different actions. However, in the case rigid body environments we can demonstrate that the desired property is true. We formalize this in the following lemma.
Lemma B.3. Let E' be a (deterministic) environment based on the rigid body dynamics model, i.e. the state transitions are computed using time- discretization of continuous and differentiable dynamics model. Let s, E S be a pair of reachable states environment E.
If there exists a, a' A, such that E(s, a) E(s,a') s', i.c. actions and a' result in the same state transition, then it holds that a d'.
Proof. is known that the dynamics of rigid bodies is governed by the laws of kinematics and by the application of Newton's 's second law (Tsal 1999). Namely, the equations governing the dynamics of rigid bodies in (time-dependent) generalized position-velocity coordinates (g. v), q: R." w: R - IR" take the following form
M(g) du dt
k(g, + p(g. +
where M: 1n - Ruxn is the positive definite, symmetric inertia matrix, k: RT x 1 + R" is the vector of external and contact forces, p: I x R R² is the of friction components, T: + RD is the applied external force (Pfeiffer & Glocker Todorov et al. 2012).
du Hence, there is one-to-one mapping of state tuple (g, v, P) and the external force (torque). The one-to-one mapping still holds for realizable environment E' based on simulation, where in- finitesimal acceleration du is approximated by the finite difference et+1 Uz di is a fixed time-step,
16

---

Under review as a conference paper at ICLR 2022
is calculated from the Cartesian body position and the fixed domain constraints, external force calculated from imput action a by a linear transformation.
We conclude with the following corollary demonstrating the convergence of Q learning in SPP-TD3 algorithm within the finite setting.
Corollary B.3.1. Let
E be a (deterministic) environment based on the rigid body dynamics model, i.e. the state transitions are being generated by a time-wise discretization of a continuous and differen- tiable dynamics model.
the experiences utilized for Q updates be gathered using random/e-greedy SPP (deter- ministic) policy n
the target states form a be realizable, i.e. if = (s) for E S, then it holds that = E S,
IDM CM be implemented lookup table and receive infinite updates.
Moreover, assume 2, ...... 7 Thm.
Then, clipped Double Q-learning in SPP-TD3 will converge to the optimal value function Q", as defined by the Bellman optimality equation, with probability
follows that for any pair of reachable states s' of environment E there exists a unique action a resulting in transition E(s, a) = Hence, as CM receives infinite updates and is implemented as a lookup table it will learn to map any of the policy target states (realizable states) to the unique action. follows that each state action will be sampled infinite number of times in the limit. As the rest of Thm. B.2 assumptions are assumed obtain the claim.
Proof. From Lemma B.3
C SPP-RL ALGORITHM DETAILS
C.I LAGRANGIAN OPTIMIZATION
To solve the constrained optimization problem we use the idea of Lagrange multipliers going back to the classical calculus. Lagrange multipliers were already applied in the RL context in Ray et al. 2019), where authors developed on-policy RL algorithms for safety RL. This method uses an adap- tive penalty coefficient to enforce constraints. We find this method plausible, it translates a con- strained objective into solving a classical max-min problem that can be incorporated within any RL algorithm. The max-min Lagrangian objective L(T, A) for the constrained optimization problem equation I takes the form
max min L(T, A) E [Ro(T)] - A
||st11 24 2007((
d).
(6)
Intuitively, when the constraint equation Tb is satisfied, then the min w.r.t À is attained for - 0. and causes À to decrease. On the other hand, if the constraint is not satisfied, then the min is attained for À oc, and hence causing À increase, the constraint satisfaction receives more weight in the overall max-min objective, eventually forcing the constraint to be fulfilled.
We solve the max-min Lagrangian objective for SPP policy using simultaneous gradient ascent w.r.t the policy parameters 0, descent w.r.t. the Lagrange multiplier X, updates. Such an approach can viewed as an instance of primal-dual algorithm for constrained MDP. It is known in the literature that time-steps of Actor, Critic & dual variable gradient updates should be adjusted sepa- rately to guarantee convergence, refer to Borkar (2005) and references therein. The SPP algorithms three learning rates: actor (lo), critic (l)), and dual variable (lx), which were adjusted using hyper-paramett optimization. The actor and critic learning rates were equal in all of the studied algorithms, whereas the dual variable was fixed in all algorithms I 0.0001.
17

---

Under review as a conference paper at ICLR 2022
C.2 SPP-SAC ALGORITHM
The original DDPG algorithm. which is the base of SPP-DDPG presented Algorithm T aims training a deterministic policy, contrary for example SAC algorithm by Haarnoja al. (2018a b) that trains a stochastic policy. Its main principle is rooted in the very successful deep Q-learning method, originally applied for solving the Atari benchmark problem (Mnih al. 2013). However, DDPG is suitable for continuous action spaces like in the MuJoCo tasks. SAC is an algorithm that combines ideas of DDPG and soft Q-learning Haarnoja et al. (2017) utilizing the advantages of both approaches, SPP-SAC is presented in Algorithm
Algorithm 2: SPP-SAC Algorithm
repeat
Sample random action a ~ U;;
Store experience (St, 2t, 04; 8t41, r2) D; (use next-state the initial actor actions)
until random exploration is done;
repeat
if buffer D is not full then
Sample actor prediction 2 ~ (8));
Compute action a, ());;;
Observe reward r; and next state 8t+1;
Store experience (8t, zt, de, ; rz)) in D; 8141 is terminal, reset environment state;
end
if it's time to update CM then
Randomly sample {b; {(s,, Se4 o.) batches of samples from replay buffer
D:
SGD train CM using the batches and MSE loss; end
it's time to update actor and critic then for update steps do
Randomly sample = { 24, ae, s141, set of batches from D; Compute actions åe+1 CM(+++1, 2,1),, using samples 4411 7(2-118641);
Compute targets y = тı + y min Qutar. ($1+1,
1=1.2
Compute Φ = 40 V
ΣB
Qa CM
(st, oz)
Compute 8 =
V B
Es min Q CM
(se, (S)) log (o))),
1=1,2
where (st) T
st) is differentiable wIt 0:
Compute 0
lad
V, EB ||5 20.e (s)))
[
Update À A + lA
Es |1s111 - -0, d):
Update target networks Dearg, = (1 - T) + Thi, for i= 1,2;
end end until convergence;
C.3 SPP-TD3 ALGORITIMM
Twin Delayed Policy Gradient (TD3) FFujimoto et al. 2018 is improved DDPG algorithm using several tricks, which can be summarized as Clipped Double-Q Learning, Delayed Policy Updates, Target Policy Smoothing. SPP-TD3 algorithm is presented in Algorithm
18

---

Under review as a conference paper at ICLR 2022
Algorithm 3: SPP-TD3 Algorithm
input environment initial model parameters Φ1, d2, 1/; state planning distance threshold empty replay buffer D; algorithm hyperparameters
output: trained model parameters 0, σ1, Φ2, di; total return repeat
Sample random action a ~ U:
Store experience (St, dt, 2t, (1+1,80+1 ) in replay buffer D; (use next-state the initial actor actions)
until random exploration done:
repeat
if buffer D is not full then
Compute actor prediction ž, = clip (sx) (ss))=) alamr Zhigh)). where .V; Compute action
(M(,s, 24) and observe reward r2411 and next state $1411 Store experience (St, Zt, de, St+1, rt+1) D; end
if it's time to update CM then
Sample (b. {((Ee, St+1) a))))) batches of samples from replay buffer D;
SGD train CM using the batches and MSE loss;
end
if time to update actor and critic then
for update steps
Randomly sample B = {($t, Se, ae, St41 (141) set of batches from D: Compute target states 2++1($++1) = clip (Teaary (st+1) ) + clip(s, -c, c), Row, Zhigh). where (N0, r);
Compute target actions (++1(8 CM(5++1 1:t11
Compute targets y(r, 8t+1) = 11n Q7 (st+1, (++1( (using target parameters 0targ.1, (harg.2);
Update Φi Φ1
Vei
Es (v(1,881) - 2 TM (st, as)
BT
Update Φb Φ2 if current-sepp mod ppllicyddeaay 0 then Update & = 0
l V
Es (r(r, (111))811)) - T.. (si, as)
B1
to
VoEB C.M ar) -M=CM(e
Vo Vo 1151114111 (o(
Update = x l (10) EB |15111 To( d)
Update actor & critics Oearg.i = (1 T)@targ.i + TØs; Otarg (1 - 7)aarrg 70;
end end
until convergence;
D HYPERPARAMETERS USED IN EXPERIMENTS
We present hyper-parameters used in our experiments in Table 2 for the SPP-DDPG, in Table
the SPP-SAC. SPP-TD3 hyper-parameters were dependent the set of benchmarks, and hyper- parameters specific for MuJoCo environments are in Tab. for SafetyGym Tab, and for Ant- Push in Tab.
In SPP-DDPG implementation we used the following architectures:
Actor: dSm(S) 256 ReLU 256 ReLU dim(AA) tanh;
Critic: dim(S) + (dim(A) - 256 ReLU - 256 ReLU + 1;
CM: 2 dSm - 100 tanh - + 50 tanh - dim(AA tanh;
In SPP-TD3 implementation we used the following architectures:
Actor: dSm(S) - 256 ReLU - 256 ReLU - dim(AA) tanh;
Critie: dSm(S) + dim(.A) - 256 ReLU + 256 ReLU 1; CM: 2 dim(S) 100 tanh - 50 tanh dim(A) tanh;
19

---

Under review as a conference paper at ICLR 2022
Hyperparameter
Value
?
0.99
0.005
actor/critic learning rate Lo lo
0.0005
À learning rate ly episode length batch size episodes update freq. & grad.steps.
0.0001
5000
100
10
50
exploration noise param.
0.05
replay buffer size
same number of env. interacts
parameters specific for SPP-DDPG
d state consistency
0.2
CM hyper-parameters
init. rand. samples learning rate
20000
0.005
batch size
128
update frequency
500 steps
update epochs
update batches
100
Table 2: SPP-DDPG Algorithm hyperparameters, DDPG used the same hyperparameters, except exploration noise (set to 0.1).
Hyperparameter
Value
0.99
0.005
actor/critic learning rate La lg
0.0001
À learning rate la episode length batch size test episodes update freq. & grad.steps.
0.0001
5000
100
10
50
exploration noise param. policy noise noise clip.
0.2
0.2
0.5
replay buffer size
same number of env. interacts
parameters specific for SPP-TD3
d state consistency
0.2
CM hyper-parameters
init, rand. samples
25000
learning rate
0.001
batch size
128
update frequency
500 steps
update epochs
1
update batches
200
Table 3: SPP-TD3 Algorithm hyperparameters for MuJoCo environments, TD3 used the same (com- mon) hyperparameters.
In SPP-SAC implementation we used the following architectures:
Actor: dim(S) - 256 ReLU - 256 ReLU - dim(AA tanh;
Critic: dSm + dim(A) - 256 ReLU - 256 ReLU 1;
CM: dim(S) 64 tanh
32 tanh
dim(A) tanh;
20

---

Under review as a conference paper at ICLR 2022
Hyperparameter
Value
?
0.99
0.005
actor/critic learning rate Lo lo
0.0002
À learning rate ly episode length batch size episodes update freq. & grad.steps.
0.0001
5000
100
10
50
exploration noise param.
0.1
policy noise noise clip.
0.2
0.5
replay buffer size
same number of env. interacts
parameters specific for SPP-TD3
d state consistency
0.2
CM hyper-
init, rand. samples
400000
learning rate
0.001
batch size
128
update frequency
500 steps
update epochs update batches
1
200
Table SPP-TD3 Algorithm hyperparameters for SafetyGym environments. Observe that for this environment specifically used larger number of CM pretrain samples
Hyperparameter
Value
0.99
0.005
actor/critic learning rate la = lg
0.0001
À learning rate la
0.0001
episode length batch size test episodes update freq. & grad.steps.
5000
100
10
50
exploration noise param.
1
policy noise
0.2
noise clip.
0.5
replay buffer size
250000
parameters specific for SPP-TD3
state consistency
0.2
CM thyper-parameter
init. rand. samples
100000
learning rate
0.001
batch size
128
update frequency
500 steps
update epochs
1
update batches
200
Table 5: SPP-TD3 Algorithm hyperparameters for AntPush environment. Observe that for this vironment specifically we used fixed buffer size which is significantly smaller than the total number of env interacts, and much larger exploration noise param equal 1.
21

---

Under review as a conference paper at ICLR 2022
Hyperparameter
Value
?
0.99
T
0.005
actor/critic learning rate (p ig
0.001
A learning rate la batch size test episodes
0.0001
100
10 each 1000 frames
update freq. & grad.steps. a learning rate
50
0.2
0.001
replay buffer size
1e6
parameters specific for SPP-SAC
d state consistency
0.2
CM hyper-parameters
init. rand. samples
10000
learning rate
0.001
batch size
100
update frequency update batches
1000 steps
100
Table 6: SPP-SAC Algorithm hyperparameters
E ADDITIONAL EXPERIMENTS
E. ABLATION STUDY
We performed thorough ablation study of the features that we implemented in the SPP-TD3 algo- rithm presented in Sec.([ Ablation study was performed in Ant (Fig. 7a and Doggo-Goal (Fig. 7b We explain the meaning of particular ablations visible in Fig.
SPP-TD3 less init. sampl. means that less randomly generated experience the algorithm added to the replay buffer at the beginning of executing Alg. 3 (the first repeat until block). In some cases, like SPP-TD3 for Doggo-Goal, see Table we choose to include a large number of random samples in the buffer (400000). However, this has no considerable effect on performance. In fact, the SPP-RL agent utilizing the number of random samples the same as vanilla RL performs better initially. Only in the long term, slightly losing to the agent utilizing more experience. This shows that the performance boost of SPP-RI. still persists when feeded the same number of initial random samples in the corresponding vanilla RL method.
SPP-TD3 w/o state consistency the state consistency defined in Property is not introduced in the algorithm; therefore, there is no incentive on the policy to choose as targets realiz- able states. Therefore it demonstrates that the SPP approach is more than just policy reparametrization.
SPP-TD3 Q(s, s') critic (state-state critic) replaced the traditional target Q function com- putation using state-action pairs with state-next state pairs. Hence the step of computing target actions is omitted. As a result, the algorithm did not converge in several cases and exhibited significant variation of the obtained returns between different runs. This is con- sistent with the observed performance of D3G algorithm in Ant (Fig. 3bb.. We do not show this ablation in Doggo-Goal as the algorithm did not converge in this environment.
SPP-TD3 w/o state normalization π not being normalized using the current mean and min/max values of the past observations in replay buffer D.
SPP-TD3 const. X = the Lagrangian multipliers method is not used. The algorithm uses the constant value À = when performing policy updates. We chose this particular À value, as it tends work well in Ant, whereas in Doggo-Goal results in lack of conver- gence; this shows how delicate the matter of choosing appropriate À (when fixed) per given environment is.
22

---

Under review as a conference paper at ICLR 2022
SPP-TD3 const. À = 0.1 or À 0.5 the Lagrangian multipliers method is not used, the algorithm uses the constant value = 0.1 or = 0.5 respectively when performing policy updates. The particular À values resulted in much better performance in Doggo-Godd than À = 1 (which works in Ant). Observe that À = 0. results in even better performance than the base SPP-TD3 implementation utilizing Lagrangian multipliers. However, employing the Lagrange multipliers provides natural way of solving the constrained objective opti- mization, and the pepee--rraeeter d is adjusted once for all the environments.
121 und
2003
100
Ablation study using Ant
(b) Ablation study using Doggo-Goal
Figure 7: Ablation study of various SPP-TD3 algorithm features, performed using Ant and Doggo- Goal environments. For the exact features of the algorithm that were tested refer to the discussion in Appendix E. Figures show test return computed every 5k frames averaged over 5 different seeds. The continuous curve is the mean, whereas the faded color regions std. deviation.
E.2 SPP-RL vs VANILLA RL REPLAY BUFFER
As argued in Sec. 5.44 the performance boost visible in SPP-RL vs. vanilla RL approaches is presum- ably due more efficient exploration performed by SPP-RL algorithms vanilla RL approaches. One empirical argument given concerns the performance of the so-called TD3 shadow agent, i.c., vanilla TD3 agent implementation utilizing (partially) the experience gathered by SPP-TD3 agent to perform its Actor&Critic updates. Here we provide empirical evidence that the distribution of observations in replay buffers gathered by SPP-RL and vanilla RL implementations differs consid- erably. In Fig- g we present an empirical study of distributions of states gathered in SPP-SAC and vanilla SAC replay buffers for the Doggo-Goal task. The state-space in this task has dimensions. Hence to make it amenable to visual investigation and entropy computation, we encode the state vectors using random encoder. The encoder architecture that we used for this task is just initial- ized (default PyTorch ini) and not trained architecture: 72 > 20 tanh - 10 tanh - 2. Fig. g show plots of discretized state distributions gathered by example run of vanilla SAC and SPP-SAC respectively using the Doggo-Goz environment and encoded using the random encoder. We also compute cross-entropy to quantify
difference between those distributions as the training both algorithms progresses and the replay buffer is being filled up. Observe that the cross-entropy is clearly increasing as the replay buffer is being filled up. which suggests that vanilla RL and SPP-RL algorithms gather different observation distributions in the replay buffer.
23

---

Under review as a conference paper at ICLR 2022
-ca
(c) Cross-cntropy the two encoded states distributions (vanilla SAC SPP-SAC)
(a) Encoded state density from vanilla SAC buffer
(b) Encoded state density from SPP-SAC buffer
-0
(f) Cross-entropy the encoded states distributions (vanilla
(d) Encoded state density from vanilla TD3 buffer
(e) Encoded state density from SPP-TD3 buffer
TD3 vs SPP-TD3)
Figure 8: Visualizations of states distribution (density histogram plot) in the replay buffer the end of training taken from single vanilla SAC, TD3, SPP-SAC, and SPP-TD3 runs. States are encoded in 2D using the random encoder. Figs. [c is the cross-entropy of the discrete state distributions (vanilla SPP) with respect to the quantity of experience in the replay buffer (excluding the initial random initialization).
24


Reviews:
[{"review_id": "nP3aX0c4rKb", "reviewer": "Reviewer_oVKQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors mention that, during learning, if the buffer of the\nsamples is full new samples are not added anymore. Why is that? It is\njust for solving the ICM or also for learning Q and \\pi? Please\nclarify. \n\nApparently there were some implementation issues with D3G, otherwise it\nis not clear why there was a big difference in its performance, given\nthat D3G follows a similar approach. Why the authors did not try to\nre-implement D3G in their environment?  \n\nIt is not clear why SAC-vanilla behaves better than SPP-SAC in the\ncar-push domain?\n\nIf would be interesting to discuss how to include other constraints,\ne.g., a cost function for violating safety, within the proposed\napproach. \n\nThe authors argue, and experimentally show, that a state-state mapping\ncan perform better exploration than a state-action mapping, but it is\nnot clear why this is the case. Why there is a better exploration?\n\nThe parameter \"d\", which regulates the state consistency, is quite\nrelevant to the proposed approach, however, the authors did not show\nhow different values of \"d\" affect the results. \n\nIt seems that the transition function is assumed to be\ndeterministic. How much this affects the applicability of the proposed\napproach?\n\nThe paper in the introduction talks about data efficiency and\ninterpretability, however, it is not clear how is interpretability\nimproved by the proposed approach?\n\nTypos:\n- build on The => build on the\n- caption Fig 5 c): ... bule ... => ... blue ...\n- however, we if we show \n", "summary_of_the_paper": "The paper describes an approach where a state-state mapping is\nlearned (called state-planning policy or SPP) coupled with an\noff-policy RL system (the authors included experiments with DDPG, TD3, \nand SAC).\n\nLearning a state-state mapping, Q(s,s'), needs also to learn\nan inverse dynamic model to estimate which action (a) will take the\ncurrent state (s) to the next state (s').\n\nThe authors framed the problem as a constrained optimization approach,\nwhere the objective is to maximize the sum of discounted rewards such\nthat the differences between the target states (produced by the\npolicy) and the actual states (produced by the environment with the\naction from the control model) are within a certain threshold\ndistance. In particular, the authors solved the optimization problem\nvia the Lagrange multiplier method.\n\nExperiments were performed in different domains and with different\nstate-of-the-art DRL systems.\n", "summary_of_the_review": "The paper describes a new approach for learning a state-state mapping.\nPros:\n- It is a well presented and clearer approach for learning a state planning\npolicy, than previous work (in particular, D3G)\n- It opens a new research avenue in terms of exploration comparisons between\nstate-state and state-action policies\nCons:\n- An alternative formulation of a previously published idea (D3G) which can be\nseen as an incremental contribution\n- Not completely clear why and under which circumstances the state-state\nmapping is preferred over the state-action mapping\n- Lack of assessment on the performance of the approach with different\nstate consistency threshold values (b)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "P9ZgdR3--lr", "reviewer": "Reviewer_B8gx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Since this is simply a \"new algorithm\" paper, it is evaluated along three axes: 1) Empirical results, 2) Theoretical results 3) Insights into why it works and when to use it. For 1) the gains for the doggo environment seem strong. However, that is about it. I do not think it is sufficient to propose a new RL algorithm and solely have results for one environment from a proprioceptive state as evidence that it works. There are no pixel based environments, which the community is currently focused on, and the MuJoCo experiments are weak for both TD3 and SAC, showing little gains despite increased wall clock time. 2) There is no theory, which is fine, but adds emphasis to 1). Finally for 3) I am not convinced at all this method makes sense based on the analysis provided. I don't read this and feel like I really understand why it should work and where I should expect it to work. Given this, my score is a reject. I would be willing to increase my score if there are improvements along these three axes, most likely 1) and 3). For example, an additional set of experiments + intuition about why it works/doesn't work in those particular experiments.\n\nMore detailed comments:\n* The opening paragraph makes claims about poor sample efficiency and interpretability of RL algorithms, but does not cite any papers for this. Then the paper itself does not really address this. \n* The performance does not seem particularly impressive in Figure 3 - SPP basically adds nothing to TD3 and SAC, aside from one bad seed in TD3 for Humanoid. However, the actual numbers reported are lower than usually seen for Humanoid. It is common to see ~5k after 1M steps and ~6k after 2M steps for both TD3 and SAC. Given this, the results seem quite weak, and given that this is an empirical paper, it is a significant negative.\n* There are no pixel based tasks. A few years ago this criticism would have been unfair, but nowadays the community is very focused on this setting, and there are many RL algorithms that could be built on top of with this type of approach, for example RAD uses a similar off policy algorithm. \n\n* After reading the paper I don't really get *why* I would use this method. It takes a longer time to run, and doesn't outperform in many cases. What is it about the doggo environment that makes it work better? The comment \"due to improved exploration in some sense\" shows that even the authors do not know why their method did well on the environments where it did. I see the plots in Section E.3 and it does seem to collect more different states, but this would be expected with a stronger policy anyway, I still don't know why this is the case. For example, the ablation study has very different ordering for the different components for Ant and Doggo, why? It isn't clear to me at all.\n* How were the hyperparameters chosen? I see them listed, but there is no discussion of how they were tuned.  \n* It is great to see the method run for ten seeds, and with transparency about the wall clock time. I will say though that the wall clock time is definitely a negative of the method which may limit applicability. How does the wall clock time scale for higher dimensional problems? What is the bottleneck that causes this doubling in time for the environments considered?\n\nMinor comments/typos:\n\n* In the first line of the paper it says RL is a \"machine learning technique\". This may be picky, but I am not sure it is a great way to describe RL. RL is both a problem (maximizing reward in an MDP) and a class of methods.\n* In the \"summary of results\" section change \"overperform\" → \"outperform\".\n* \"Related Works\" → \"Related Work\".\n* Bottom of p3, \"Obviously, SPP approach to work requires\" → \"in order to work, SPP requires\".\n* Sec 4, \"details of SPP algorithm\" → \"details of *the* SPP algorithm\". \"*the* Spinning Up RL online resource\". \"*the* DDPG algorithm\".\n* Sec 5, \"we performed a set of benchmarks\" → \"we performed experiments on a set of benchmarks\". \"overall sample efficiency of RL procedure\" → \"overall sample efficiency of *the* RL procedure\".\n* Sec 5.4, first sentence doesn't make sense.\n* The use of apostrophes/speech marks for \"shadow agent\" is incorrect.", "summary_of_the_paper": "This paper proposes a new RL algorithm whereby the policy selects a new state rather than action, with constraints to ensure the next state selected is a valid one. The contribution is the new algorithm \"SPP-RL\" which can be applied to off policy algorithms such as TD3/SAC. There is experimental evidence this may be an effective approach in some settings.", "summary_of_the_review": "This is an empirical RL paper introducing a new algorithm, but the results are not convincing either based on their strength or intuition. At present the paper does not convince the reader to use this method or build upon it, thus it does not meet the bar for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JZg4TesDaZ", "reviewer": "Reviewer_F9WZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the problem is of interest, I have several concerns about this work.\n\n- The treatment of the constrain in the problem is lacking formality, which makes it difficult to evaluate some of the authors contributions.\n\na) The dual variable \\lambda needs to be projected to the non-negative orthant in Algorithm 1, 2 and 3.\n\nb) The author's algorithm takes the form of a primal-dual algorithm for CMDPs. In this regard and due to the non-convexity of the Lagrangian, the choice of both the primal and dual step sizes plays a critical role in guaranteeing convergence. The authors completely ignore this issue (there is no dual step size in their algorithm). See e.g., [A].\n\nc) Some of the analysis in Appendix B does not make much sense. For example, section B.2 presents a policy gradient theorem, but this is not the gradient that the authors take in Algorithm 1 (the gradient is taken with respect to the Lagrangian). In general the analysis disregards the effect of the dual variables.\n\n- The hyperparameter d in the optimization problem (1) might be difficult to design. How sensitive are the results to its value? How to handle infeasibility in the problem?\n\nMinor comments:\n\t- Several references are going out of margins.\n\n[A] Borkar, Vivek S. \"An actor-critic algorithm for constrained Markov decision processes.\" Systems & control letters 54, no. 3 (2005): 207-213.\n", "summary_of_the_paper": "This work presents a a new reinforcement learning algorithm in which planning is performed purely on the state space instead of the more common state-action space. A constrained optimization problem is formulated which is solved exploiting duality.", "summary_of_the_review": "In its current form, I cannot recommend this paper for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a RL planning algorithms where a policy selects a reachable state. The empirical evaluation shows promising results in some environments. While all the reviewers agreed that the state planning RL is a relevant and promising direction, the reviewers expressed concerns with the rigor, significance of the results, and incremental novelty.

To improve them paper the authors should:

- Bring the theoretical foundation in the main text, and add more rigorous analysis, including the limitations of the method.
- The readability of the figures needs to be improved. The legend on the figures is too small and colors are too similar that renders the figures unreadable and confusing.
- If the authors' goal is to develop a method for interpretable RL, then some results and analysis need to address the interpretability of method.
Decision: Reject