Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: The paper proposes modelling the score function of the perturbed empirical data distribution in order to capture "guide" the optimization closer to the data, which is a way of tackling uncertainty in planning methods. Further as a theoreticall result i is shown that the NLL of a perturbed empirical distribution is equivalent to a smoothed distance to the data up to a factor. Results are provided in simulated domains and on real hardware.
review: ## clarity

the paper is mostly written clearly.

## strengths

the work is well motivated

theoretical result connecting the smooth distance to data and the proposed score estimation

robotics experiments

## weaknesses

the way that uncertainty is treated here, it doesn;'t extrapolate.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: there are two reasons for this rating:
1) I doubt that this method is practical, since there is no extrapolation beyond data (could be convinced otherwise)
2) unclarities (see review)
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: Offline RL needs careful consideration of uncertainty estimation. This paper examines the use of smoothed distance to data as a metric, demonstrating its ability to stably converge to data and facilitate analysis of model bias through Lipschitz constants. Additionally, the authors establish an equivalence between smoothed distance to data and data likelihood, enabling the application of score-matching techniques to learn distance gradients to data. The authors highlight that offline model-based policy search problems that aim to maximize data likelihood do not require likelihood values but solely rely on the log-likelihood gradient (the score function). To tackle high-dimensional problems where zeroth-order methods struggle to scale and ensembles fail to overcome local minima, the authors propose Score-Guided Planning (SGP). This offline RL planning algorithm leverages score-matching to enable first-order planning. Empirical results show comparable performance with baselines.
review: The paper is well-written. Technical novelty is sound, with highly relevant results to robotics. The work is interesting.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is carefully written with robotics experiments. Source code provided. I choose to accept the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents a method for offline model-based RL (MBRL) called score-guided planning (SGP). The method is introduced by suggesting that the distance to data is good uncertainty metric and that a smoothed version of this distance is just the data likelihood. The method then uses the score function of the data generating distribution as a regularizer to prevent a gradient-based (single shooting) planner from choosing OOD actions. Experiments are presented in simple simulated control problems, a single integrator with a pixel-based action space, and a real robot pushing task.
review: ## Strengths

1. The core idea of using the score function as a regularizer for offline model based RL is interesting and provides a nice connection between generative modeling and offline RL.

2. The experiments demonstrate qualitatively that the method is able to work effectively with high dimensional observation and action spaces.

3. The examples about the potential weaknesses of ensemble-based uncertainty provide a nice illustration of why something like distance to data may be preferred in practical scenarios.

## Weaknesses

1. It would be good to present some more quantitative results in the experiments section. The qualitative results for the non-D4RL experiments are useful, but do not make clear quantitative comparisons between methods. 

2. The performance on the D4RL benchmark suite is not particularly strong. SGP gets some small wins on some D4RL datasets, but does so poorly on others that the average performance is worse than the baselines. It could also be interesting to see comparisons on some of the higher dimensional robotic manipulation tasks in the D4RL suite instead of just the mujoco locomotion tasks. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, this paper presented an interesting idea and backed it up with a diverse set of experiments, including on robotic tasks. There is room for improvement, but I think the paper is a good contribution and should be accepted.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: In this paper the authors (i) propose an epistemic uncertainty metric for learned dynamics models, namely smoothed distance to data, (ii) identify a connection to score-based generative modeling that enables tractable gradient-based optimization with objectives incorporating this metric, and (iii) apply this methodology in the context of offline model-based reinforcement learning in a novel algorithm called Score-Guided Planning (SGP). Because the uncertainty metric is amenable to gradient-based optimization, SGP scales to problems with high dimensional state/observation spaces (e.g., images) where zeroth-order optimization methods struggle. Experiments demonstrate that SGP outperforms other baseline uncertainty estimators and optimizers (e.g., ensembles, CEM) on standard benchmarks, and moreover succeeds in high-dimensional and data-sparse regimes where it also makes more accurate policy rollout predictions than vanilla (unregularized) MBRL.
review: The reinterpretation of score-matching for a perturbed empirical distribution as learning gradients of smoothed distance to data is a clever idea, and the paper is well-written with convincing experiments supporting the authors' main claims. I am pleased to recommend its acceptance, however I believe that there are two areas of improvement which, if addressed, would significantly improve the impact of this paper.

1. *Relevance of the theory:* Section 3 argues that smoothed distance to data is a good choice for gradient-based optimization as it can be guaranteed that, when solely optimizing for this metric, gradient descent will converge to a point in the dataset as the smoothing parameter is taken to 0. This fact is not particularly surprising (e.g., an alternate/more constructive proof could reason about Voronoi cells, minus a shrinking boundary as $\sigma \rightarrow 0$, as regions of attraction), but perhaps critically it overlooks the possibility that selecting a spot between data points may be desirable. A common rationale to use neural networks to fit dynamics is, if not to generalize/extrapolate (e.g., through meaningful learned representations as mentioned in the discussion of limitations), at least to interpolate. In light of this, $\sigma$ may be interpreted not only as a penalty for straying outside of the training dataset, but also as a factor allowing for interpolation. That is, $\sigma$ seems like one of the more critical tuning parameters of this method (similar to how the choice of a kernel bandwidth parameter is regarded as a key tuning parameter for Gaussian process-based methods), and may deserve more empirical study than the theory might suggest.

2. *Experimental evaluation:* As noted above, the experiments in this paper do convincingly demonstrate the applicability of smoothed distance to data to gradient-based trajectory optimization. Beyond that, however, the experiments do not yield much insight on how SGP's performance depends on the choice of its tuning parameters (i.e., $\sigma$ and $\beta$). For example, in line with the previous point, studying sparser data (i.e., lower $N$) or finer time discretization may be a way to study the role of $\sigma$ in allowing for interpolation. Also, arguably all of the experiments are conducted on effectively low-dimensional state/action spaces (by construction in the case of the Pixel-Based Single Integrator). What happens if you try the real-world box-pushing task with image observations instead of mocap marker positions? Though undoubtedly mechanically SGP can perform gradient-based optimization in multi-thousand-dimensional spaces, if the method doesn't scale to tasks where you cannot ensure that a solution trajectory can be assembled as a sequence of states you've seen in the training dataset (i.e., *data stability* is impractical, which would at face value seem to be the case for increasingly high dimensional systems), it may be worth mentioning in the limitations section (e.g., this may be less of an issue for uncertainty quantification methods that attempt to infer model confidence irrespective of distance to data in the input space).

Minor comments:
- In the problem definition of offline model-based RL which mentions an empirical distribution over state transitions (line 93), it initially appears that the dynamics themselves may be stochastic (i.e., contain aleatoric uncertainty). It is mentioned only later in the last line of the limitations that aleatoric uncertainty is uninvestigated; for reader clarity it may be worth mentioning earlier (i.e., in Sections 1 and 2) that the dynamics are assumed deterministic and that learning $f$ can justifiably be treated as a (uncertainty-regularized) regression problem.
- How do local Lipschitz constants (computable perhaps for the cart-pole system) of the learned dynamics $f$ vary as uncertainty is increasingly penalized?
- Is some degree of data preprocessing (e.g., whitening) necessary to make squared (Euclidean) distance the right distance to penalize?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I recommend this paper's acceptance at CORL 2023 as the offline RL problems it addresses (particularly those with high-dimensional perceptual input) as well as the methodology it develops are highly topical in the robot learning community today. As noted above, I believe that some of its claimed insights could be better explained/better probed by additional experiments, but such additions would only be supplementary to the existing sound core work.
review_title: null

Meta-review: This paper does model-based offline RL with a score matching regularization term that encourages the resulting trajectories to go towards the offline data. All reviewers have voted for acceptance after the authors' rebuttal. While the method is interesting, one of the main questions that has remained unanswered is why are the D4RL results not always competitive with existing imitation learning or offline RL methods? Also, one of the main advantages of using offline RL is to leverage the "stitching" behavior (available data from A -> B, B->C means that through offline RL the policy can figure out A->C even if it is not in the data). The loss function here forces the trajectories to stay close to the data, so it is possible that that hinders stitching behavior, which makes me wonder why is offline RL the right RL setting to showcase this method, i.e. why not online RL + the regularization term? Finally, I would definitely have expected that the paper cite and compare against COMBO https://arxiv.org/abs/2102.08363, which as far as I can tell it doesn't.

So, while I think I would personally argue for a rejection, as I don't think offline RL is the right setting for this contribution, I will respect the reviewers evaluations and recommend acceptance.
Decision: Accept (Poster)