Title: ReDS: Offline RL With Heteroskedastic Datasets via Support Constraints
Venue: NeurIPS 2023
Accepted: yes
Keywords: offline RL

support constraints

heteroskedastic data
Abstract: Offline reinforcement learning (RL) learns policies entirely from static datasets. Practical applications of offline RL will inevitably require learning from datasets where the variability of demonstrated behaviors changes non-uniformly across the state space. For example, at a red light, nearly all human drivers behave similarly by stopping, but when merging onto a highway, some drivers merge quickly, efficiently, and safely, while many hesitate or merge dangerously. Both theoretically and empirically, we show that typical offline RL methods, which are based on distribution constraints fail to learn from data with such non-uniform variability, due to the requirement to stay close to the behavior policy **to the same extent** across the state space. Ideally, the learned policy should be free to choose **per state** how closely to follow the behavior policy to maximize long-term return, as long as the learned policy stays within the support of the behavior policy. To instantiate this principle, we reweight the data distribution in conservative Q-learning (CQL) to obtain an approximate support constraint formulation. The reweighted distribution is a mixture of the current policy and an additional policy trained to mine poor actions that are likely under the behavior policy. Our method, CQL (ReDS), is theoretically motivated, and improves performance across a wide range of offline RL problems in games, navigation, and pixel-based manipulation.

555

D(x,x,))

Success Rate: 54.3 %

D(nn,))

D(x,n,Xs)

CQL,Distributional Constraint

CQL(ReDS),Support Constraint

How are actions modified?

中d

儿

Policy Density

Policy Density

πβ

Tβ

Tre

Bad Actions

Action Space

BadActions

Action Space

Poor actions in support are

Poor actions in support

Resulting Q-Values

QValue

pushed up by CQL

Out of support

QValue

Are pushed down by ReDS

Out of support

are pushed

actions

actions

down by ReDS

are pushed

down by CQL

Action Space

Action Space

Bad Actions

Good Actions

Bad Actions

Behavior Policy Support

Behavior Policy Support

：

Start

Goal

LessVariability

MoreVariability

IQM

Policies

2

0.75

0.90

1.05

1.20

Policies

0.45

0.60

0.75

5

NormalizedScore

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ReDS for offline reinforcement learning (RL) where the variability of demonstrated behaviours changes non-uniformly across the state space. Unlike prior offline RL methods, e.g., CQL and AWR, that directly constrain the learning policy by distribution matching with the behaviour policy, ReDS presents the task as support matching and implements it by reweighting the data distribution in CQL. Both theoretical analysis and experimental results are provided to demonstrate the effectiveness of the resulting algorithm, CQL (ReDS).\n\nSTRENGTHS:\n- The paper is well-written and easy to follow.\n- The paper is well-motivated, theoretically sound, and supported by extensive experiments.\n\nWEAKNESSES:\nThere are several main weaknesses of the paper.\n\n- The main issue of the paper is with its novelty. The idea of reweighting samples by state-dependent weights is not novel. Specifically, [1] has proposed to reweight the samples by trajectory returns / advantages and demonstrated that such a scheme significantly improves the performance of multiple offline RL algorithms, including CQL, IQL, TD3+BC, and BC. Similarly, [2] has proposed to cast the offline RL constraints as a support matching problem, rather than distribution matching, by reweighting the data samples by normalised trajectory returns. The proposed method in [2] has also been demonstrated to improve a wide range of offline RL methods, ranging from CQL, IQL, TD3+BC to CRR. Nevertheless, despite their importance,  highly similar motivation, problem formulation, and solutions, both [1] and [2] are missing from the related works. \n- Although the paper has conducted extensive experiments across different domains, it lacks several important studies. Similar methods [1, 2] have already demonstrated that support matching methods for offline RL can be algorithm-agnostic, i.e., it can be used to boost the performance of different algorithms, ranging from CQL to IQL, TD3+BC, and etc. However, this paper only discusses CQL (ReDS). From the perspective of algorithmic design, CQL, TD3+BC, and AWR (discussed by the authors in line 127) / CRR / IQL (based on the AWR for policy improvement) are all distribution-constraint algorithms. It is important to discuss how ReDS could improve the aforementioned algorithms and how ReDS differs from [1, 2] in its performance.\n\nReferences: \n\n[1] Hong, Z. W., Agrawal, P., des Combes, R. T., & Laroche, R. (2023, February). Harnessing mixed offline reinforcement learning datasets via trajectory weighting. In The Eleventh International Conference on Learning Representations.\n\n[2] Yue, Y., Kang, B., Ma, X., Xu, Z., Huang, G., & Yan, S. (2022). Boosting Offline Reinforcement Learning via Data Rebalancing. arXiv preprint arXiv:2210.09241.\n\nQUESTIONS:\n- Some of the important domains / tasks are missing from the paper. As a standard benchmark for offline RL, the medium-level locomotion tasks, the adroit tasks, and the original antmaze tasks are missing from the paper. Specifically, the original antmaze itself is already heteroskedastic in my opinion, as for most of the states, the ant only needs to go straight and only occasionally it will make turns. How does CQL (ReDS) compare with the well-benchmarked results on these tasks?\n- How to apply ReDS to other distribution-constraint-based methods, including AWR, TD3+BC, etc. If it can be applied, how does it improve them and how does it compare with the aforementioned related works? If ReDS cannot be applied to other algorithms, what is its limitation?\n\nLIMITATIONS:\nThe main limitation of the paper is with its novelty. The idea of reweighting samples and casting the distribution-constrained offline RL, i.e., distribution matching problem, as a support matching problem, has been discussed by prior methods. Also, many important and well-benchmarked domains and tasks are missing in the experiments, which makes it less clear about the performance of the method. Overall, I do think this is an interesting paper, but given the current limitations and issues, it might not be ready for publication yet.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper has identified the presence of heteroskedasticity in realistic offline RL datasets, which negatively impacts the performance of existing offline RL methods that rely on distributional constraints. To tackle this issue, the authors introduce a new approach called ReDS, which transforms distributional constraints into support-based constraints.  Several novel heteroskedastic datasets were introduced in order to showcase the superior performance of ReDS compared to existing offline RL methods.\n\nTo practically achieve the support constraint, the fundamental change based on CQL is to add a new penalty term $\\pi_{re}$, which can be obtained via \n$\\arg \\max_{\\rho_\\psi} \\mathbb{E}_{s \\sim \\mathcal{D}, a \\sim \\pi_\\beta(\\cdot \\mid s)}[\\log \\rho_\\psi(a \\mid s) \\cdot \\exp (-A_\\theta(s, a) / \\tau)]$.\n\nSTRENGTHS:\n1. Well-written, with high-quality examples, and technically sound. \n2. Extensive evaluations over many different tasks.\n\nWEAKNESSES:\n1. The design of the reweighted version of the behavior policy in Eq. 6 is heuristic. I can not see a clear clue why the coefficient is designed as 1/2 and 1/2. \n2. The experiments in the noisy and biased version of Antmaze tasks seem contrived. \n3. Inconsistent of the differential concentrability between theory and real-world practical datasets (std).\n\nQUESTIONS:\n1. Could the policy $\\pi$ in Deﬁnition 3.1 be any policy? What if it is an arbitrary worst policy, i.e., random policy? I do not see clear evidence that $C_{diff}$ is big enough when $D(\\pi, \\pi_{\\beta})$ is arbitrarily large in both $s_1$ and $s_2$. On the other hand, should $s_1, s_2 \\sim d^{\\pi}$ without any other condition? What if they are all sampled from the same place, i.e., the narrow hallways in the didactic example? \n2. Any detail about how to compute the std in Chapter 4.2?\n3. How well does ReDS perform over non-heteroskedastic datasets (random, medium, expert)? Is there any limitation to on non-heteroskedastic datasets?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ReDS, a novel offline RL method designed to handle heteroskedastic datasets. ReDS incorporates support constraints by reweighting the data distribution based on conservative Q-learning (CQL). This allows the learned policy to deviate from the behavior policy within its support while maximizing long-term return.\n\nSTRENGTHS:\n1. The addressed problem of a more fine-grained support constraint in offline RL research is crucial.\n2. The paper includes thorough discussions about the proposed method.\n3. The introduction of the method, which utilizes reweighting distribution constraints to achieve a fine-grained support constraint, is clear and reasonable. The explanation in Figure 3 is particularly intuitive.\n4. The method demonstrates significant performance advantages, especially when applied to heterogeneous datasets.\n\nWEAKNESSES:\n1. The didactic example does not convincingly support the claims. For example, it does not convincingly demonstrate the characteristics of heterogeneous datasets in key states (such as the bottom-left corner of the map) and their impact on the CQL and REDS algorithms. Additionally, the results presented in Figure 1 show that the baseline algorithms AWR and CQL fail, while REDS does not. However, has this demonstration accounted for the potential impact of statistical errors?\n2. The baseline algorithm (CQL) seems somewhat outdated. It would be necessary to discuss and compare with more advanced methods that satisfy the support constraint, such as InAC[1].\n3. There are several spelling errors throughout the paper. A careful proofreading is necessary. For instance, the capitalized \"w\" in \"With\" in the title should be lowercase, and the occurrence of \"WSome\" in line 419 is incorrect.\n\n[1] Xiao, Chenjun, et al. \"The in-sample softmax for offline reinforcement learning.\" arXiv preprint arXiv:2302.14372 (2023).\n\nQUESTIONS:\nThe main concern relates to the points mentioned in the weaknesses section. Additionally, there is another question that requires clarification:\n\nQ: Is the proposed method plug-and-play? If so, why not integrate it with an updated or foundational algorithm (e.g., SAC+vanilla distribution constraint)?\n\nLIMITATIONS:\nThe paper discusses the limitations. And, these limitations seems not appear to require immediate attention.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on heteroskedastic datasets, where the distribution of actions may not be uniform in certain states but close to uniform in other states. The authors propose the ReDS method, which re-weights the actions to penalize \"bad\" actions that perform poorly on the dataset but have a higher probability, while encouraging \"good\" actions with lower probabilities in the dataset's support. ReDS outperforms previous baseline methods in multiple experimental scenarios.\n\nSTRENGTHS:\nThe majority of the paper is written in a clear and easy-to-follow manner. The research problem addressed in this paper is indeed relevant to real-world scenarios, and the authors provide didactic examples to illustrate the potential issues with previous methods. The paper offers a detailed analysis and proposes a concise and intuitive plug-in solution called ReDS, while also demonstrating its effectiveness. The experimental settings are diverse, covering multiple domains.\n\nWEAKNESSES:\nReDS should apply in broader scenarios beyond the specific setting of heteroskedastic datasets. Another point is the choice of experimental settings, as most of them involve manually constructed datasets to satisfy heteroskedastic conditions, rather than utilizing more intuitive scenarios such as autonomous driving or validating whether some existing datasets are really heteroskedastic. Thus the significance is somewhat limited.\n\n\n\n\nMinor issues:\n\nThe figures in the main text, such as Figures 1 and 2, rely heavily on the captions. Adding more conclusive statements in the main text could enhance consistency and clarity.\n\nTable 4 --> Table 2?\n\nLine 326 \"some of the standard D4RL datasets which are not heteroskedastic in and find\" --> extra \"in\"\n\nLine 419 \"WSome prior works ...\"\n\nQUESTIONS:\n1. In Fig.1, I wonder if the datasets cover the whole room B or only a point shown in the figure. If the datasets cover the whole room, AWR and CQL should be able to learn to exit room B, since other actions lead to lower returns.\n2. As Line 326-327 stated, the authors have evaluated ReDS on some of the standard D4RL datasets which are not heteroskedastic and find that the addition of ReDS, as expected, does not help, or hurt on those tasks. Why does ReDS hurt the performance on some of the standard D4RL datasets? By Theorem 1, ReDS can at least have better performance.\n3. Why do the experimental domains contain Atari?\n\nLIMITATIONS:\nThe authors generated new datasets and did not conduct a universal analysis on whether heteroskedastic properties exist in some datasets. It would be beneficial to first employ a suitable method to effectively verify the presence of heteroskedastic properties in commonly used datasets before applying ReDS.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}