Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work presents a a new reinforcement learning algorithm in which planning is performed purely on the state space instead of the more common state-action space. A constrained optimization problem is formulated which is solved exploiting duality.
main_review: While the problem is of interest, I have several concerns about this work.

- The treatment of the constrain in the problem is lacking formality, which makes it difficult to evaluate some of the authors contributions.

a) The dual variable \lambda needs to be projected to the non-negative orthant in Algorithm 1, 2 and 3.

b) The author's algorithm takes the form of a primal-dual algorithm for CMDPs. In this regard and due to the non-convexity of the Lagrangian, the choice of both the primal and dual step sizes plays a critical role in guaranteeing convergence. The authors completely ignore this issue (there is no dual step size in their algorithm). See e.g., [A].

c) Some of the analysis in Appendix B does not make much sense. For example, section B.2 presents a policy gradient theorem, but this is not the gradient that the authors take in Algorithm 1 (the gradient is taken with respect to the Lagrangian). In general the analysis disregards the effect of the dual variables.

- The hyperparameter d in the optimization problem (1) might be difficult to design. How sensitive are the results to its value? How to handle infeasibility in the problem?

Minor comments:
	- Several references are going out of margins.

[A] Borkar, Vivek S. "An actor-critic algorithm for constrained Markov decision processes." Systems & control letters 54, no. 3 (2005): 207-213.

summary_of_the_review: In its current form, I cannot recommend this paper for publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new RL algorithm whereby the policy selects a new state rather than action, with constraints to ensure the next state selected is a valid one. The contribution is the new algorithm "SPP-RL" which can be applied to off policy algorithms such as TD3/SAC. There is experimental evidence this may be an effective approach in some settings.
main_review: Since this is simply a "new algorithm" paper, it is evaluated along three axes: 1) Empirical results, 2) Theoretical results 3) Insights into why it works and when to use it. For 1) the gains for the doggo environment seem strong. However, that is about it. I do not think it is sufficient to propose a new RL algorithm and solely have results for one environment from a proprioceptive state as evidence that it works. There are no pixel based environments, which the community is currently focused on, and the MuJoCo experiments are weak for both TD3 and SAC, showing little gains despite increased wall clock time. 2) There is no theory, which is fine, but adds emphasis to 1). Finally for 3) I am not convinced at all this method makes sense based on the analysis provided. I don't read this and feel like I really understand why it should work and where I should expect it to work. Given this, my score is a reject. I would be willing to increase my score if there are improvements along these three axes, most likely 1) and 3). For example, an additional set of experiments + intuition about why it works/doesn't work in those particular experiments.

More detailed comments:
* The opening paragraph makes claims about poor sample efficiency and interpretability of RL algorithms, but does not cite any papers for this. Then the paper itself does not really address this. 
* The performance does not seem particularly impressive in Figure 3 - SPP basically adds nothing to TD3 and SAC, aside from one bad seed in TD3 for Humanoid. However, the actual numbers reported are lower than usually seen for Humanoid. It is common to see ~5k after 1M steps and ~6k after 2M steps for both TD3 and SAC. Given this, the results seem quite weak, and given that this is an empirical paper, it is a significant negative.
* There are no pixel based tasks. A few years ago this criticism would have been unfair, but nowadays the community is very focused on this setting, and there are many RL algorithms that could be built on top of with this type of approach, for example RAD uses a similar off policy algorithm. 

* After reading the paper I don't really get *why* I would use this method. It takes a longer time to run, and doesn't outperform in many cases. What is it about the doggo environment that makes it work better? The comment "due to improved exploration in some sense" shows that even the authors do not know why their method did well on the environments where it did. I see the plots in Section E.3 and it does seem to collect more different states, but this would be expected with a stronger policy anyway, I still don't know why this is the case. For example, the ablation study has very different ordering for the different components for Ant and Doggo, why? It isn't clear to me at all.
* How were the hyperparameters chosen? I see them listed, but there is no discussion of how they were tuned.  
* It is great to see the method run for ten seeds, and with transparency about the wall clock time. I will say though that the wall clock time is definitely a negative of the method which may limit applicability. How does the wall clock time scale for higher dimensional problems? What is the bottleneck that causes this doubling in time for the environments considered?

Minor comments/typos:

* In the first line of the paper it says RL is a "machine learning technique". This may be picky, but I am not sure it is a great way to describe RL. RL is both a problem (maximizing reward in an MDP) and a class of methods.
* In the "summary of results" section change "overperform" → "outperform".
* "Related Works" → "Related Work".
* Bottom of p3, "Obviously, SPP approach to work requires" → "in order to work, SPP requires".
* Sec 4, "details of SPP algorithm" → "details of *the* SPP algorithm". "*the* Spinning Up RL online resource". "*the* DDPG algorithm".
* Sec 5, "we performed a set of benchmarks" → "we performed experiments on a set of benchmarks". "overall sample efficiency of RL procedure" → "overall sample efficiency of *the* RL procedure".
* Sec 5.4, first sentence doesn't make sense.
* The use of apostrophes/speech marks for "shadow agent" is incorrect.
summary_of_the_review: This is an empirical RL paper introducing a new algorithm, but the results are not convincing either based on their strength or intuition. At present the paper does not convince the reader to use this method or build upon it, thus it does not meet the bar for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper describes an approach where a state-state mapping is
learned (called state-planning policy or SPP) coupled with an
off-policy RL system (the authors included experiments with DDPG, TD3, 
and SAC).

Learning a state-state mapping, Q(s,s'), needs also to learn
an inverse dynamic model to estimate which action (a) will take the
current state (s) to the next state (s').

The authors framed the problem as a constrained optimization approach,
where the objective is to maximize the sum of discounted rewards such
that the differences between the target states (produced by the
policy) and the actual states (produced by the environment with the
action from the control model) are within a certain threshold
distance. In particular, the authors solved the optimization problem
via the Lagrange multiplier method.

Experiments were performed in different domains and with different
state-of-the-art DRL systems.

main_review: The authors mention that, during learning, if the buffer of the
samples is full new samples are not added anymore. Why is that? It is
just for solving the ICM or also for learning Q and \pi? Please
clarify. 

Apparently there were some implementation issues with D3G, otherwise it
is not clear why there was a big difference in its performance, given
that D3G follows a similar approach. Why the authors did not try to
re-implement D3G in their environment?  

It is not clear why SAC-vanilla behaves better than SPP-SAC in the
car-push domain?

If would be interesting to discuss how to include other constraints,
e.g., a cost function for violating safety, within the proposed
approach. 

The authors argue, and experimentally show, that a state-state mapping
can perform better exploration than a state-action mapping, but it is
not clear why this is the case. Why there is a better exploration?

The parameter "d", which regulates the state consistency, is quite
relevant to the proposed approach, however, the authors did not show
how different values of "d" affect the results. 

It seems that the transition function is assumed to be
deterministic. How much this affects the applicability of the proposed
approach?

The paper in the introduction talks about data efficiency and
interpretability, however, it is not clear how is interpretability
improved by the proposed approach?

Typos:
- build on The => build on the
- caption Fig 5 c): ... bule ... => ... blue ...
- however, we if we show 

summary_of_the_review: The paper describes a new approach for learning a state-state mapping.
Pros:
- It is a well presented and clearer approach for learning a state planning
policy, than previous work (in particular, D3G)
- It opens a new research avenue in terms of exploration comparisons between
state-state and state-action policies
Cons:
- An alternative formulation of a previously published idea (D3G) which can be
seen as an incremental contribution
- Not completely clear why and under which circumstances the state-state
mapping is preferred over the state-action mapping
- Lack of assessment on the performance of the approach with different
state consistency threshold values (b)
review_title: null

Meta-review: The paper presents a RL planning algorithms where a policy selects a reachable state. The empirical evaluation shows promising results in some environments. While all the reviewers agreed that the state planning RL is a relevant and promising direction, the reviewers expressed concerns with the rigor, significance of the results, and incremental novelty.

To improve them paper the authors should:

- Bring the theoretical foundation in the main text, and add more rigorous analysis, including the limitations of the method.
- The readability of the figures needs to be improved. The legend on the figures is too small and colors are too similar that renders the figures unreadable and confusing.
- If the authors' goal is to develop a method for interpretable RL, then some results and analysis need to address the interpretability of method.
Decision: Reject