Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method for accounting for the geometry of the data in devising methods for enforcing pessimism in offline RL. To do so, the paper ends up training a state-conditioned distance function between actions, and enforces a policy constraint using the learned distance function. The hope is that this is useful when the distance function can generalize (though I have questions about this below).
strength_and_weaknesses: Strength: The idea seems interesting, and new, and certainly improves over policy constraints in their experiments (TD3+BC vs their method). 

Weaknesses:

- I think I don't understand why the approach should perform well in general -- doesn't it depend on what kind of a function we train for g(a, s)? What if g(a, s) were a table, would the method not be just TD3+BC? But the paper doesn't seem to discuss the connection between the complexity of g(s, a) and the efficacy of the method.

- Taking the above reasoning, shouldn't a value function conservatism method like CQL already take into account the geometry of the data? Why should the proposed method be better than that, if it is not just for tuning issues?

- The theory doesn't seem to discuss the role of g and the complexity of g. It builds on the Bellman-consistent pessimism paper, yet no comparisons to the approach from that paper and their theoretical results exists in the paper.

- Empirical results: besides benchmark results, more clear ablations and analysis are needed to understand why the method actually works. Can we do ablations on the network architectures for both TD3+BC and CQL, and the proposed approach to see how all of these behave with smoothing? 
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow, and the idea is novel.
summary_of_the_review: Overall, I am not convinced that the method is robust and reliable unless the concerns of g(s, a) are discussed. The theoretical analysis doesn't explain why g(s, a) should be good in practice or why alternatives such as value conservatism are not doing the same job. More empirical evidence and improved theoretical analysis is needed.

-------

## After Rebuttal

Thanks for the responses! I revisited Theorem 2, but I still don't see why $d_1$ and $d_2$ alone control the the Bellman concentrability coefficient tightly. There are many more terms in there that depend on the Q-function class in Theorem 2 ($Q(s_0, a_0)$, $\epsilon_\mu$), and in general, it cannot be said that the sample to centroid distance is alone what it takes to control this distributional shift. This just makes me think further that the approach would better benefit from a "geometry constraint" on the Q-function class (as it would more tightly control the Bellman concentrability coefficient) as opposed to the continuous action space distance constraint.

Regarding comparison with TD3+BC conceptually -- Sorry, I am still not sure. In equation 4, it seems like $g(s, \hat{a}) = ||\hat{a} - a_\text{data}||_2$ at the optimum. This means that the learned function when Equation 4 is solved exactly is measuring the L-2 distance to the dataset action (assume for a second that there is only one action at a given state in the dataset). Then isn't it identical to TD3+BC? When multiple actions are observed for a given state in the dataset, one could argue that even TD3+BC would have a similar global effect, as you would train $\pi(s)$ to be close to $a_1$, $a_2$, ... for all actions observed in the data, which would achieve the same kind of a smoothing effect as in Figure 3 (a).

It seems like the author's intuition goes beyond a state-conditioned function to some smoothing over the dataset more generally. But this is not clearly explained, and it is not clear why other methods and TD3+BC, most directly, will also not enjoy some kind of global smoothing effect more generally with a small function class. 

So I don't understand how is this true  _"In TD3+BC, the BC term simply minimizes the absolute sample-to-sample distance between actions from the policy and the dataset, which essentially avoids the policy from producing OOD actions. Whereas in DOGE, the state-conditioned distance function  characterizes a sample-to-dataset distance that captures the overall data geometry. "

Overall, I am inclined to keep my original score. Since my score is at a disagreement from other reviewers, I will reduce my confidence by 1.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Motivated by the over conservativity of existing offline RL algorithms, the paper proposes a Distance-sensitive Offline RL with better GEneralization (DOGE) method. In fact, DOGE describes a constraint set of possible policies, relying on a state-conditioned distance function. Such a distance function accounts for the data geometry. Theoretically, a connection between DOGE and bellman-consistent coefficient is established. Empirically, DOGE is demonstrated to have comparable or better performance versus the state-of-the-art methods.
strength_and_weaknesses: ================= Strength =================

The paper focuses on a very important problem in RL: when offline data do not have strong data coverage, how to best exploit the logged data set. The proposed method is novel to me.

The paper is clearly written with many graphical illustrations. Most of the sections are easy to follow.

Empirical results are convincing, although I am not totally familiar with the experimental environments.

================= Weakness =================

The claim "Theoretical analysis demonstrates the superiority of our approach to existing methods that are solely based on data distribution or support constraints" may not be well supported, as in Section 3.3, the idea is to transform the policy constraint to Bellman-consistent coefficient. Moreover, Theorem 3 is a bit hard to interpret as no concrete rate is given. It would be helpful to elaborate the claim.

Theoretical results in Theorem 2 have a relatively vague connection to DOGE. At a first glance, the constraint threshold $G$ in equation (6) does not appear in Theorem 2 nor Theorem 3, making it unclear how to evaluate the advantage of DOGE. At the same time, Theorem 2 is still hard to interpret because of the complicated math expressions without intuitive explanations.
clarity,_quality,_novelty_and_reproducibility: My major clarity concerns are on Theorem 2 and Theorem 3. Theorem 2 is stated under the assumption of NTK, what is the exact meaning of that? Second, the $\epsilon_\mu$ terms bothers me. From the definition, it is the bellmen error term measured by distribution $\mu$. If the policy $\pi$ is good enough, then the bellman-consistent coefficient blows up. How should one interpret this? It might be better to provide a concrete rate of convergence in Theorem 3 and compare to existing literature. Both Theorem 2 and Theorem 3 does not directly reflect how to choose threshold $G$ in equation (6).

"Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation" might also be a related work, where the distribution shift is measured by a class-restricted $\chi$-square divergence.
summary_of_the_review: I am interested to see any feedback from the author regarding the theory part of the paper. Overall, I think the paper in its current form is slightly below the acceptance bar, but I am willing to raise the rating based on further discussions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to regularize continuous control policies trained offline by forcing them away from actions that are too far from the training data's actions. This is achieved by a learned distance function that predicts distance to the training data's actions. The authors show that such a setup has some interesting properties. Empirically this appears to yield improvements on the D4RL offline benchmark suite.
strength_and_weaknesses: Strengths:
- The proposed method stems from a series of interesting intuitions
- The proposed method seems to be able to capture something about the MDP which increases its performance in an offline RL context

Weaknesses:
- The text is hard to read at times
- The theory within the paper relies heavily on quantities which may not scale intuitively
- There are some strange empirical evaluation choices, in particular:
    - If I understand correctly, the core of the proposed method is to contrain a (continuous) policy to output actions that are not "too far" from the training data's actions. Yet, the main way this is tested is to remove part of the _state space_ from the training data? This is not testing the right thing, and is not even testing what the theory predicts that this method should be better at.
- It's nice to have bounds, but what's even nicer is empiricially verifying that those bounds are meaningful and/or make meaningful predictions about trained models--this is missing from this work.


Additional comments:

> we observe that the value function approximated by DNN can interpolate well but struggles to extrapolate

There are papers showing extrapolation capabilities in deep RL, see e.g. the work of Packer et al. [1]

[1] Assessing Generalization in Deep Reinforcement Learning, Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, Dawn Song, 2018


> Theorem 1

I'm not sure I fully get why the authors think this theorem is important.

1. The NTK assumptions provide kernel-like bounds, and what we get is litterally that the further we get from the dataset the poorer the approximation, this is already known. There is some extra work to make the result a bound over the projection, but that is also fairly straightforward.
2. This result a priori has nothing to do with the convex hull of the training data. (3) and the equation before (2) are the same equation. The main argument seems to be that because the convex hull of a finite set of points is naturally bounded, then (2) is bounded whereas (3) is not. This argument seems moot in practice, since inputs to neural networks (or any ML system really) are almost always bounded (images lie in RGB[0,1], language is a sequence of tokens, etc).
3. The result relies on an Euclidean norm, which quickly loses significance past a number of dimensions. In fact, it is likely that past a certain number of dimensions, the very distinction between interpolation and extrapolation lose meaning (see [2]). Take 100k images from Atari at random, then take a new image by running an agent or again at random. I'd wager that either (a) the new image is within the convex hull, or (b) its distance to the convex hull is upper bounded by Eq (2)'s $B$, because $B$ is close to 1.

I'm also not sure what $x$ stands for here. The error on a $Q$ function evidently depends on whether not just the state but also the action is in the dataset. Is $x$ just the concatenation of the two? What does it mean to concatenate an Atari image and a categorical label and then take norms? What about joint angles and torsion strengths?

[2] Learning in High Dimension Always Amounts to Extrapolation, Randall Balestriero, Jerome Pesenti, Yann LeCun



> State-conditioned distance function

> synthetic noise actions sampled from the uniform distribution

This is a strange choice, why the sampling? We know that MSE regresses to the mean, and we know the mean of $\mathbb{E}_{x'\sim U}[|x-x'|]$. The result (regressing to the centroid) should be the same.

> It is clear that the learned distance function can accurately predict the sample-to-dataset centroid distance

In the action domain, yes, perhaps. But what about the state domain?

> To evaluate the generalization ability of DOGE, we remove small areas of data from the critical pathways to the destination

As above, this does not evaluate the right thing.

I am left wondering after reading this paper if a simpler baseline might be just as effective, one that doesn't require $g$. In the limit, but when each state only appears once in the training data with one associated action*, $g$ is simply the distance between $\mu(s)$ and $a$. What happens when we replace $g(s,a)$ by $\|a-\mu(a)\|$ in (6) and (7)?

*In a continuous state-action space, isn't every state really only "visited once" anyways?


clarity,_quality,_novelty_and_reproducibility: This paper would benefit from being proofread. I realize that English is not a first language for most people, and I am of course not using this to base my judgement of the work, but many sentences are ambiguous and make understanding the details and the arguments within this work difficult.

>  Inspired by this, we propose a new method, DOGE (Distance-sensitive Offline RL with better GEneralization)

Maybe that makes me old school but not every method needs a catchy name, much less one named after a 2010 meme. That just makes us look silly?

The paper is otherwise relatively straightforward. I feel like the mathematics of the paper are a bit more obscuring that they are enlightening, at least I don't obviously see important lessons that we can take from them, nor that are immediately reflected empirically.

I could probably reproduce this paper, perhaps not exactly, but certainly in spirit given the level of detail.
summary_of_the_review: I am a bit conflicted about the paper, the intuitions behind the ideas are nice. The paper spends a lot of time trying to convince us of the usefulness of a variety of bounds (often closely based on prior bounds), used to justify the approach, but I am skeptical that these bounds are reflected in the results. This is worsened by the fact that some predictions made by the theory are not properly tested in practice (see above).

Update: the authors have addressed my concerns, and I find the additions to the paper and the appendix are useful to framing this work. I lean in favor of acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors suggest that current offline RL methods are overly conservative in interpolating regions of the dataset, whereas this should primarily be applied to extrapolation. They suggest that this should be controlled by the distance, which the bound/estimate using a state-conditional action distance estimator. Satisfying this distance constraint then becomes a regularizer for policy optimization. Theoretical analyses show that this provides a tighter bound on performance as compared with a previous method, BEAR. Results are presented on D4RL, comparing with relevant model-free baselines. There are also experiments on an extrapolation version of the task, removing some parts of the state/action space.
strength_and_weaknesses: **Strengths**

- **Strong empirical evaluation**. The authors evaluate their method, DOGE, on a standard offline RL benchmark, D4RL, comparing with the relevant baseline approaches. This is in both the standard locomotion set of tasks, as well as the more difficult AntMaze environments. In all cases, the authors’ approach generally compares favorably or performs similarly. The authors also include an analysis on the AntMaze environments, removing transitions from some regions of the state-space. This allows them to test interpolation abilities of previous methods. I view this empirical evaluation as the strongest point of the paper.

- **Relatively simple method**. While the authors have included a considerable degree of theoretical analysis, the method that they arrive at is ultimately fairly simple, involving a learned, state-conditional action distance estimator trained from offline trajectories. This then enters policy optimization as an additional regularization term. I see this as no more complex than many previous methods, e.g., which learn behavior priors. This simplicity may lead to more widespread adoption of the method in the RL community, making DOGE a useful baseline for future works.

- **Empirical and theoretical analysis**. The authors combine empirical and theoretical analyses. While much of the theoretical analyses are outside of my area of expertise, I appreciate that these can be important. Again, this may broaden the appeal of the paper to the wider RL community.


**Weaknesses**

- **Unclear in places**. At various points throughout the paper, I found it difficult to connect the authors’ theoretical connections and motivations to the particular method proposed. For instance, the authors’ initial motivation is framed in terms of better interpolation of the state-action space of the dataset. Yet, the authors translate this into something akin to estimating the cross entropy of the behavior policy w.r.t. a uniform policy, conditioned on the state. It’s unclear how this exactly relates to the state-action distance initially motivated by the authors. I was also left uncertain about whether the authors are arguing that the state-space, action-space, or both should be interpolated. The initial motivation leads me to believe that it’s both, whereas the proposed method seems to tackle the action-space and the empirical analyses only tackle the state-space. Along similar lines, the state-conditional action-distance penalty is introduced as a regularizer, but it’s unclear how this translates into estimating and propagating state uncertainty. I would have thought that this regularizer would by integrated over space/time by entering the definition of the Q-value (similar to KL-regularizers, e.g., in SAC), but this seems to not be the case. I also found the diagrams difficult to interpret (although I did appreciate that they were included).

- **Unclear how, precisely, this tackles interpolation better than previous methods**. The authors motivate their approach from the perspective of improving the ability to interpolate regions of the dataset, using the perspective of deep Q-networks in the interpolation and extrapolation regimes. In interpolated regions, this error is bounded, whereas in extrapolation regions, it is not (although it will still be correct nearby the data manifold). This then leads the authors to propose their state-conditional action distance estimator. However, previous methods, like BEAR, use ensembles to estimate something like Q-value uncertainty, which is then penalized. Following the authors’ arguments, there should be relatively less uncertainty in interpolated regions as compared with extrapolated regions, allowing these methods to also handle interpolation. Where is my misunderstanding here? Why is this method uniquely suited to handling interpolation, whereas other methods, as the authors show empirically, are not? Ultimately, with a finite dataset and continuous action space, all methods have to interpolate for every single point — does this method just have a wider window of interpolation?

- **Applicability/generality unclear**. While reading the paper, I felt as though there were a number of unstated assumptions that the authors were making. The authors initially motivate their approach by considering interpolation vs. extrapolation in (continuous-action) deep Q-networks, though it seems the same technique could not be applied to discrete action space. The tasks considered by the authors consist of relatively smooth dynamics and reward functions, thus, interpolation in an environment like AntMaze is valid, especially if it allows you to connect two regions of the state-space. However, this strikes me as ultimately a heuristic, which may or may not be respected by the actual environment. Making this heuristic assumption is useful when it works out, but it’s not a general principle that should be relied upon. For example, if we consider a reward function in the shape of an inverted Mexican hat (e.g., imagine an environment with a heat source like a campfire that keeps the agent warm, but would burn the agent if it touches it directly), but you only observe data around the high-reward outer ring, then interpolation will lead you to estimate that the center is also high-value, which is fatally wrong.
clarity,_quality,_novelty_and_reproducibility: **Clarity** -- The clarity of the paper is reasonable, but could be improved somewhat. While it may be my relative lack of familiarity of the area, I found it difficult to follow some of the authors’ lines of reasoning. See weaknesses section above. However, other aspects of the paper, such as the equations and experiments are generally clearly well-defined and explained. For instance, Figure 1 clearly lays out the issue that the authors intend to tackle.

**Quality** -- The quality of the paper appears to be quite high. The authors provide a compelling empirical analysis on a variety of offline RL datasets/environments, comparing with the relevant model-free baselines, coupled with a theoretical analysis of their approach. The authors use figures to explain their motivation, aspects of their approach, and experiments. This all seems to meet the level of a top-level conference publication.

**Novelty** -- The paper is somewhat novel. While the method itself ultimately amounts to adding a regularizer to the policy optimization objective, getting to that point involves a non-trivial amount of insight, which likely took considerable work. Beyond the method, the authors also provide a fair amount of theoretical analysis, as well as a set of empirical analyses around the AntMaze environments looking at interpolation in a more focused way.

**Reproducibility** -- I suspect that the paper is reproducible. The method itself is fairly simple, training an additional distance estimator on the offline dataset and using this as an additional policy regularizer. The authors present results on a standard offline RL benchmark, where they generally perform comparably or favorably, using multiple seeds/evaluations. This suggests that the positive outcome of the results in not merely from random chance.
summary_of_the_review: The paper presents a compelling set of empirical results, alongside theoretical analyses. While I’m fairly convinced that the method performs well against strong baselines in standard offline RL benchmarks, I found it difficult to follow some aspects of the paper, particularly with regard to motivating the approach, e.g., assumptions about the underlying environment, and translating it into a practical training technique.

**UPDATE:**

I appreciate the authors' detailed responses to me and the other reviewers. While there may be some unresolved concerns, particularly around the clarity of the paper, most of my main concerns have been addressed. I now feel that the paper warrants acceptance, and have increased my score from 6 --> 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper observes that deep Q function approximators trained on an offline dataset interpolate well within the convex hull of training data, even on samples that did not exist in that data. This prompts a more lenient view of "conservatism" during offline learning, which refrains from penalizing samples that the deep Q function approximator can in fact perform well on, even if they might in a strict sense be "out-of-distribution" of the offline data. 

Strengths:
- Interesting and intuitive idea, framed in terms of the shape of the offline dataset and how it may permit interpolation to OOD samples.
- Works well on multiple environments.
- Interesting empirical observations about the generalization capabilities of deep Q function approximators.

Weaknesses:
- No clear empirical demonstration of DOGE having achieved a more lenient form of pessimism, or benefiting from it, even though this is the motivation of the approach. Instead, we only see final RL results, which, while impressive, can often be rather sensitive to hyperparameter tuning.
- The connection between the theoretical analysis and the actual algorithm is not communicated very well, and the writing, while improved during the response phase, is still somewhat abstruse.
- This is not a major weakness, but the method is rather close to a slew of offline RL approaches: BEAR, CQL, and TD3-BC, and differs mainly in the insight about interpolatability of deep Q networks. This could be converted to a strength by evaluating direct "plug-in" improvements for many commonly used algorithms today, but this is not currently in the paper.

justification_for_why_not_higher_score: The connection between theoretical results and final algorithm is not very clearly motivated and communicated, and the improved RL results are not clearly shown as being connected to the motivation of the approach. 
justification_for_why_not_lower_score: There is interesting insight and strong empirical results in the paper. Even if it leaves us with some hanging questions, the paper as is is quite interesting, may be practically useful, and the questions may eventually be answered by other offline RL researchers who adopt these techniques.
Decision: Accept: poster