Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a solution to the planning as inference problem in domains where it is easy to generate samples of actions in the current state but it is expensive to simulate samples of next states. The proposed solution involves learning a parametric approximation of the soft value of each state-action pair by using gradient descent. The paper also shows some examples on a self-driving car datasets and demonstrate better results. There are also some ablation studies.

The main contribution of the work is the observation that in some domains it is hard to simulate the next state, but very cheap to sample an action in the current state. This implies that the existing methods in the literature of learning a soft value of a state is not as helpful as learning the soft-value of a state-action pair (the so called critic function). The authors demonstrate this point by devising a particle filter-based algorithm that uses hypothetical particles in the training episodes to learn the critic function quickly.
strength_and_weaknesses: The paper is written quite clearly and the main point is very easy to grasp. This is a very logical extension of the previous work by Piché et al 2019 which learns the soft-state-value function. The idea of using putative particles to learn the soft-state-action-value function aka the critic is quite well motivated.

What I have would have liked to see is some tradeoffs between using a large number of putative particles versus using a few putative states. In the work of Piché only one putative state is used, but one could consider a simple extension of that approach where multiple putative states are used. I understand the paper's contention that putative actions are cheaper than putative states, but putative actions are not entirely free either.

Nevertheless, there are clear benefits of doing model-free learning unlike the previous approach of Piché.

clarity,_quality,_novelty_and_reproducibility: As stated, the main point of the paper comes out very clearly but some of the auxiliary points are not as clear. For example, I didn't follow the extension to human-like driving behavior. The paper discusses using a baseline model ITRA, but it doesn't go into enough detail of how this model was merged/improved with the Critic SMC algorithm in the current paper.

The main idea of critic SMC, on the other hand, appears easy to reproduce.

As far as novelty this paper is a logical extension of prior work, as explained earlier. However it is still distinct from previous work.

The quality of the writing and experiments is quite good. In particular the state, action visualization in Figure 3 very clearly demonstrates the main contribution of this work.

summary_of_the_review: Slightly novel contribution in the area of planning as inference using deep learning with some parts unclear, but in general a reasonable contribution.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposes the Critic Sequential Monte-Carlo approach, a simple extension of the Sequential Monte-Carlo method, in the framework of "planning as inference".
The approach combines the sequential Monte-Carlo technique with heuristic factors computed using a soft q-function.
The authors then test the framework in two safety-related tasks, a multi-agent point-mass environment, and a driving simulator.

strength_and_weaknesses: - The paper is sufficiently well written
- The paper is too incremental
- The paper lacks a proper experimental analysis on standard and complex benchmarks: the toy example is too simple, and the simulator seems too complicated and convoluted to understand the algorithm's behavior.
- it is missing a strong theoretical contribution to the method, right now it seems to be mostly a heuristics
- The paper builds on pretty strong assumptions: known model and deterministic dynamics
- The paper tries to tackle a typical problem of safe reinforcement learning, without citing any of the approaches and papers on the areas.
clarity,_quality,_novelty_and_reproducibility: The paper is reasonably well written.
The novelty of the work seems extremely limited. Every modification presented in this approach seems to be already present in the previous literature.
I cannot evaluate the reproducibility of this approach at this stage.
summary_of_the_review: While this paper seems to be overall a decent contribution, it is too incremental. To accept incremental work, I would expect clear performance improvements, a strong experimental campaign, vast ablation studies, or interesting insights from the theoretical standpoint.
This paper misses all the points above.

Furthermore, the authors are using safety scenarios as a test bed. There is an entire literature on safety in Markov decision processes:
1) The Constrained MDP framework (CMDP)
2) The Control barrier function approach
3) The shielding approach
4) Reachability analysis

Many of these methods are also model-based and are able to ensure safety while optimizing a policy. It is easy to perform slight modifications to these methods and could fit perfectly the target domains (I suspect these methods will outperform the proposed approach in the given tasks)

I suggest the reviewer either expand the set of tasks to show more generality of their approach (they claim it, but is not shown) or add safe RL baselines to the comparison. For sure, if the focus of the paper is safety, at least a discussion on safe RL methods should be fundamental, explaining why a "planning as inference" approach should be better than methods explicitly designed to enforce safety.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an approach for planning-as-inference in reinforcement learning problems with sparse rewards. A method based on sequential Monte Carlo is derived, similar to Piché et al. (2019), to sample from the optimal state-action trajectories distribution by applying filtering and smoothing techniques. The algorithm uses estimates of a soft Q-function as a critic to provide heuristic factors when resampling actions according to their optimality within the SMC loop. In contrast to prior work on SMC for planning in RL, CriticSMC uses a fixed prior policy, instead of learned one, and uses the learned Q-function to guide action sampling, which simplifies the training process.
strength_and_weaknesses: # Strengths
* The idea of using Q-functions to guide sampling in SMC when observations are sparse is interesting and might be useful beyond the RL and planning contexts.
* The proposed algorithm shows significant performance improvements in a toy collision-avoidance experiment and a more realistic one using a benchmarking dataset when compared to a baseline human-like driving policies and soft-actor critic.

# Weaknesses
* **Background.** The background on SMC for planning-as-inference is quite confusing for someone familiar with SMC and control, but unfamiliar with this line of work applying SMC to planning. Questions I had at a first pass: *Is it only simulating states in parallel and "observing" the optimality indicator*? *How do you use the sampler with a (physical) agent*? There is no clear distinction between **what's done in simulation and what's applied in the actual environment** by the agent. I had to go and read Piché et al.'s (2019) work on SMC for planning to understand that SMC is playing the role of a sampler from the optimal state-action trajectories distribution. The sampled optimal action trajectories can then either be executed blindly by an agent in the target environment after sampling or be used in a model predictive control (MPC) fashion where only the first action is applied to the environment, and from the resulting next state a new SMC planning procedure unrolls.

* **Sec. 3.1: Deterministic vs. stochastic state transitions.** The soft Q-function approximation in Equation 3 relies on the assumption of deterministic state transitions, but the algorithm descriptions in both Figure 2 and Algorithm 1 use stochastic state transitions. It also gives the impression to me that deterministic transitions could lead to some degenaracy in the SMC scheme. However, this mixed presentation in the methodology is confusing and does not explain how deterministic transitions may affect sampling performance in practice.

* **Sec. 3.2: Data for training.** It is not clear to me what is the used to learn the parametric Q function estimators in Eq. 4. No reference for what *state-action occupancy* (SAO) means is provided. I am not sure if it is based on the experience data (from the target environment) observed up to a given point in time, when executing online, or purely on simulation data using the world model that is applied within SMC.

* **Related work.** It might be worth contrasting the paper with some earlier work on SMC for RL:

    Lazaric, A., Restelli, M., & Bonarini, A. (2007). Reinforcement Learning in Continuous Action Spaces through Sequential Monte Carlo Methods. In J. Platt, D. Koller, Y. Singer, & S. Roweis (Eds.), Advances in Neural Information Processing Systems (Vol. 20).

* **Experiments.** The method is proposed as a general framework for planning in environments with sparse rewards, but it is only tested on two autonomous driving environments. It would be beneficial to compare its improvements over state-of-the-art on RL methods for sparse rewards in other classic environments used in this line of work, such as Atari games. Otherwise, the claims of the paper should at least be adjusted to reflect the focus on autonomous driving.
clarity,_quality,_novelty_and_reproducibility: * The paper is mostly well written and the ideas seem novel.
summary_of_the_review: The paper's contribution seems novel, but aspects of the methodology are confusing and the experimental comparisons should be expanded to better support the claims.

#### Rebuttal update
I have raised my score (5 to 6) after reading the authors' response.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a sequential monte carlo (SMC) algorithm for planning. The main contribution is to define the particle weights using the Q(s, a)-function, which allows using a large number of actions for the same state without having to evaluate the transition model for each.
strength_and_weaknesses: **strengths**

* The paper is proposing a simple modification of regular SMC (in the context of planning) which brings two benefits: a computational one (that we can use a larger number of particles for the same number of transition evaluations) and a conceptual one (that the resulting algorithm can be used in a model-free fashion).
* The paper contains results on refining a prior policy, which is an interesting scenario that is somewhat underrepresented in current research.

**weaknesses**

* Given the method's proximity to [1], it is unfortunate that the paper doesn't contain a comparison to [1] in the same experimental suite or an overlapping one. That makes it harder to compare the two methods.
* The baselines considered in the experiments are: ITRA, SAC, SMC and rejection sampling. The only baseline which has previous published results on these problems is ITRA, which the proposed method does outperform, but that is also to be expected since the authors are building on top of ITRA. In other words, the paper is missing a baseline which has previous published results on this experiment suite. In my opinion this is the paper's biggest issue at the moment.

---

[1] Probabilistic Planning with Sequential Monte Carlo Methods, https://openreview.net/forum?id=ByetGn0cYX
clarity,_quality,_novelty_and_reproducibility: The algorithm described in the paper (see: Algorithm 1) suffers from the well-known optimism bias of the control-as-inference framework [2]. This is when conditioning on optimality causes the planner to assume that any stochasticity in the system to turn out in the agent's favor. The authors are aware of this problem, and yet the only discussion of this happens in the final paragraph of the paper. Here, the authors point out that this issue does not affect the experiments in the paper because the systems used are deterministic. I have two issues here:

* The way the optimism bias is discussed is not ideal, because a reader that is familiar with the control-as-inference framework might notice the bias issue as soon as reading the algorithm box for Algorithm 1 and will be confused about why this problem is not addressed. To be clear, it is fine to assume deterministic systems, but if an algorithm has an obvious pitfall, it should be discussed more prominently.
* I'm not sure that the systems in the paper are actually deterministic. The toy experiment features additional agents which "[...] move adversarially towards the ego agent with some added Gaussian noise." (Section 4.1). This sounds like a stochastic system to me, unless the movement of the adversaries is pre-computed and then kept fixed throughout.

---

[2] Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review, https://arxiv.org/abs/1805.00909
summary_of_the_review: The paper proposes a simple method which has tangible benefits. The presentation could be improved to make limitations clearer (i.e. the optimism bias) and the experimental evaluation could be strengthened (e.g. by comparing to [1] using parts of their experiments section) but the paper still makes contributions that are worthy of acceptance.

---

[1] Probabilistic Planning with Sequential Monte Carlo Methods, https://openreview.net/forum?id=ByetGn0cYX
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:

This paper presents CriticSMC, a sequential Monte Carlo (SMC) method for reinforcement learning under the planning as inference view.  Concretely, this work extends the work of Piche et al (2019), by using a soft Q function to construct a proposal distribution for re-weighting particles associated to an afterstate (after the action has been selected but before the next state is observed).  Using the soft Q function to modify particle-selection enables a more efficient use of available compute in several domains (where evaluating an action value can be substantially cheaper than propagation through a transition dynamics function).  The method was demonstrated on a toy domain and a driving simulator.

Strengths:

The overall idea is simple.  After resolving questions about the problem setup and goals, three reviewers were in agreement that this work provides a useful method.

Weaknesses:

The reviewers also noted several weaknesses in the paper.  One reviewer was concerned about the limited treatment of safety constrained RL, which was the domain used to showcase the method, but where other techniques from that literature were ignored.  Multiple reviewers were concerned by the fact that the overall approach suffers from an overoptimism bias coming from the use of deterministic models, which is well known and was not given adequate treatment in the original paper.  A final concern lies in the limited experimental reproducibility and empirical evidence from the one complex driving simulator and the one toy domain.  
justification_for_why_not_higher_score: The paper provides a contribution with the proposed method, but all reviewers agree the paper could provide better evidence for the method and discussion. 
justification_for_why_not_lower_score: The reviewers were split, three for accept and one strongly against.  The detailed rebuttal by the authors and the final agreement of most reviewers indicate that the core contribution of the paper is sufficiently convincing for most readers.  The objection of one reviewer is noted.  The objection stems mostly from the choice of demonstration domain as a safe RL autonomous driving situation, which was complex and not the best experiment to showcase this method as there is a rich literature of specific methods for such domains.  

For researchers engaged with SMC methods in general simulation domains, this paper provides a useful contribution. The area chair is in favor of accepting the paper.  The AC recommends that the authors use suggestions from the reviews to improve the final paper.
Decision: Accept: poster