Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper identifies the skewed distribution of returns in the offline dataset as a challenge to return-conditioned methods, and proposes to weight the trajectories in the dataset based on their returns. 

strength_and_weaknesses: Strengths

1. The paper is relatively well-written and easy to follow.

2. The proposed idea is intuitive, whose benefits are demonstrated on two different classes of return-conditioned methods (DT and RvS).

Weaknesses

1. The paper makes repeated reference to a "bias-variance trade-off" to motivate their methods. AFAIK, what this trade-off means exactly, and a theoretical or rigorous analyses as to why their method leads to a more favorable trade-off are missing completely.

2. Skewed distribution of returns is a well-known problem in RL, and common technique to tackle this problem is to ensure each batch consists of equal amount of high return and low return episodes (so equivalently, use two bins only). I would have liked to see a comparison to such a simple baseline.

3. My biggest concern is the motivation behind studying return-condition method? Because AFAIK, they under-performs value-based methods, especially on tasks that require stitching (such as antmaze navigation and notshown in the paper). Even in table 2, the performance of return-conditioned methods are under-whelming. I am not saying that the paper should not be accepted because it does not out-perform state-of-the-art, but there should be a reason as to why the class of return-conditioned method is interesting relative to value-based methods.

clarity,_quality,_novelty_and_reproducibility: The paper is very well written and easy to follow. 

The proposed method is novel in the context of return-conditioned method, but I would have like to see a comparison to the simple baseline commonly used in value-based methods (mentioned previously).

The pseudo-code is included so I believe the algorithm is reproducible.
summary_of_the_review: I recommend a weak reject, because it is unclear why we should be interested in return-conditioned methods, especially when they face their own sets of unique challenges, and when enhanced with proposals such as the ones in this paper, still under-perform value-based methods. What are their unique selling points?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes two techniques, trajectory weighting and conservative regularization, to overcome two challenges, the bias-variance tradeoff and out-of-distribution return condition, respectively. Empirical study is conducted on top of two conditional BC offline RL algorithms, Reinforcement Learning via Supervised Learning (RvS) and Decision Transformer (DT). 
strength_and_weaknesses: Strengths:
1. The paper is easy to follow and well structured. 
2. Experiment results support the authors claims. 

Weaknesses:
1. Lack of novelty. The trajectory weighting technique is originated from Kumar & Levine, 2020. The solution for out-of-distribution (ood) returns is naïve and simple by adding positive random noise. The conditional BC offline RL algorithms are prior works, RvS and DT. 
2. The contributions are not very strong in the current state of the paper. The paper develops two techniques specifically for DT and RvS, which are limited in contributions. 
3. Improvement is not consistent via trajectory weighting and the final performance still underperforms SOTA methods. 
4. Experiments are not sufficient. The authors validate their techniques on D4RL locomotion tasks, which are known to be easy tasks. Better add more experiments such as AntMaze, Adroit or Kitchen to make the claims more convincing. 

Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. Advances in Neural Information Processing Systems, 33:5126–5137, 2020.

clarity,_quality,_novelty_and_reproducibility: The main message of the paper is clearly conveyed. The novelty is a concern. The method is easy to reproduce and the code is provided. 


summary_of_the_review: Overall, the depth of this contribution and novelty may not quite yet reach the high bar set for ICLR. The improvement is significant while the improved performance still underperforms the SOTA offline RL algorithms. Only D4RL locomotion tasks are experimented, and adding experiments on more challenging tasks would make the claims more convincing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
This paper considers offline RL with return conditioning, similar to the recent line of works on decision transformers. It considers the problem of return conditioning especially when rewards can be arbitrary depending on the offline data distribution, even if considering expert trajectories. It considers two key methods to tackle arbitrary reward conditioning - trajectory weighting and conservative regularization. These ideas are proposed assuming that expert trajectories can still contain low and high return trajectories. Experimental results are claimed to achieve significant gains of 18% and 8% depending on the state of the art offline RL algorithm, conditioning on returns, being used. 
strength_and_weaknesses: 
	1. The paper tackles an interesting problem of return conditioning in offline RL, where even if there exists expert trajectories, the returns from offline dataset at test time can still be smaller compared to the expert data collecting policy. It argues that conditional BC can be improved to tackle this problem, by a simple mechanism of trajectory reweighting. 
	2. Figure 1 well motivates the problem this paper tries to tackle. However, it would be helpful to understand more about this phenomenon - ie, to what extent does performance actually differ between the original and transformed distributions. This figure tries to argue that distribution of returns can be different depending on the type of data distribution. Existing benchmarks like D4RL considers expert/medium/medium-expert datasets, and it would be interesting to see the significance of this phenomenon in actual benchmark datasets used in practice. 
	3. The core idea proposed in the paper is to consider a reweighted trajectory distribution that conditions more on the high return trajectories, such that conditioning BC on this distribution can improve performance. This leads to the problem of high variance though, since there may be fewer high return samples available. 
	4. Equation 5 is a mechanism inspired from previous work for reweighting the trajectories. I think the main problem is the approximation of equation 5 in this case that is proposed. Moreover, equation 5 in its current form is not completely novel and is a modification of existing approaches that typically re-weights training data distribution as well; 
	5. The mechanism is applied to existing baselines to show that performance can be marginally improved due to the addition of this reweighting mechanism. It is not well explained in the paper why such a form of regularization is needed, given the reweighting mechanism already being proposed. Are there simple demonstrations explaining the need for it? I ask this because regularization schemes in offline RL has been well studied in recent past, all on D4RL too - so unless a different form of dataset or other benchmarks are being used, this conservatism based update seems like a rather forced introduction to this paper?
The paper tries to well motivate the problem, but I am not sure if the experimental results presented in this form are fully convincing or not. I understand the limitations of existing offline RL experiments to d4rl benchmarks only, but recently there are new benchmarks such as v-d4rl as well, with visual observations, that might be worth considering to further make additions to experiments in this paper. Current results comparing to only two baselines seems rather limited. Results on the Atari domains, figure 9 shows marginal improvements only?
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow and the contributions are written clearly. The authors also provide code for the paper which seems easy to understand too. In terms of quality and significance of the work, I am not sure if this work can meet the standard since the contributions are rather not much novel. The paper identifies an important problem of return conditioning on lower quality datasets and the need for trajectory reweighting, but experimental gains are not that significant and such ideas have been considered in the past well.
summary_of_the_review: 
The paper considers an important problem but the novelty, either algorithmic, or significance of experimental gains are rather limited. While the paper is easy to follow and argues for the need for such weighting - it seems like the two contributions are rather forced together to make this into a complete paper. It is not clear why conservative updates are still needed when trajectories are re-weighted, and the two additions to existing baselines are probably not the most important needed to tackle this problem. 

I think this paper is a useful contribution, but perhaps not significantly novel for acceptance in a venue like ICLR. 

I am however willing to re-consider my review and score if there is a more detailed justification of the proposed approach, and the importance of it, to improve existing performance on benchmarks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the offline RL problem and proposed two techniques (trajectory weighting, and reward perturbation) to improve Decision Transformer (DT) and Reinforcement Learning via Supervised Learning (RvS).
Specifically, trajectory weighting transforms the trajectory distribution of the original offline dataset. So the low-return trajectories are down-weighted in the new distribution while the target policy can still learn useful information from these low-return trajectories. Reward perturbation helps improve the performance of RvS when the target return-to-go is too high at test time. This approach is to perturb the return-to-go during RvS training, so that the learned policy yields actions from high-return trajectories when conditioned on out-of-distribution high return-to-go.
The experiments are conducted on the standard offline RL dataset, D4RL. On Hopper, HalfCheetah, and Walker2d, the trajectory-weighting technique turns out to be beneficial for both DT and RvS, and reward perturbation shows improvement for RvS.

strength_and_weaknesses: Strengths:
This paper is generally well-written. The presentation is clear and easy to follow.
The proposed method is technically simple and easy to implement
There are extensive experiments on different environments, locomotion tasks, Atari games, and AntMaze (see Appendix). The comparison with hard-filtering BC in Appendix is a good supplement for the main text.

Weaknesses:
The paper is mainly considering some tricks to improve DT and RvS, which seems incremental. The proposed idea is not general and enlightening enough to inspire follow-up works.
The proposed methods are more like engineering techniques, instead of a principled approach with theoretical foundations or brilliant insight.
The comparison with baselines is not thorough. For example, the trajectory transformer is missing in Table 1&2. Table 6&7 do not have BC, IQL, etc. The proposed method fails to beat trajectory transformer as a SOTA method on locomotion tasks.

clarity,_quality,_novelty_and_reproducibility: Clarity:
The description and analysis in this paper are relatively clear. I especially like Figure 1 and Figure 3, which greatly help the readers to understand the intuition behind trajectory-weighting and the effect of out-of-distribution rewards.

Quality:
Overall, the quality is okay but seems not good enough as a top-tier conference paper.

As for equation (5), there are two hyper-parameters introduced for trajectory weighting. A smaller value of \lambda or \kappa will give more weights for higher-return trajectories in the offline dataset. Then why do we need both of them? Can the trajectory-weighting approach be simplified by discarding one hyper-parameter? As mentioned in the paper, a similar equation was used for model-based optimization with optimality analysis. It will be great if the proposed equation (5) can also be built on a theoretical foundation to verify its correctness and usefulness for offline RL.

There are many hyper-parameters, \lambda and \kappa in equation (5), the number of bins B, the percentile value q, and the weight of C_{RvS} \alpha. When we change to other tasks, is it easy to tune the proposed method? According to Table 5 in Appendix, these hyper-parameters are changed for experiments on Atari environments. Is there any guidance about how to adjust these hyper-parameters when facing a new offline dataset?

Could you please add more metrics (e.g. https://arxiv.org/pdf/2108.13264.pdf) to show the benefit is statistically significant? Especially for the benefit of trajectory-weighting and reward perturbation separately, the improvement for each of these components seems marginal from Table1&2 considering the standard error.

Novelty:

The proposed method is novel in the context of improving DT and RvS. But the idea is not general enough to benefit more methods or inspire more research in the offline RL area. So the significance or impact of this work is not great.

Reproducibility:
There are source codes provided in the supplementary material and detailed hyper-parameters are listed in Appendix. So the reproducibility is okay.

summary_of_the_review: To sum up, this paper proposes simple techniques to improve DT and RvS with extensive experiments. However, the scope is kind of limited, and the performance is not SOTA.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper addresses the data sparseness problem in offline reinforcement learning. Two components are proposed: trajectory reweighting and conservatism introduced through a proposed regularizer. These are evaluated individually and in combination on MuJoCo and Atari tasks.

Initial reviews for this paper appreciated the intuitively appealing idea presented here, along with the empirical benefits the authors demonstrate. 

Several concerns were raised in the initial reviews and the subsequent discussion. Reviewers noted the need to clearly define concepts such as the bias-variance tradeoff referred to in the paper. They asked for a clear motivation for the focus on return-conditioned approaches, and additional empirical validation. Many of the reviewer concerns were addressed by the authors during the rebuttal: clarity was improved on several points, motivation was discussed, and the authors added an additional empirical comparison with a simple weighting scheme.

A key remaining concern, and subject of much discussion, is the novelty and significance of the contribution of this work. Some reviewers have argued that, given that the two proposed modifications are relatively simple and widely known in the research community, both novelty and the size of the contribution are limited. The authors argue that simple approaches can be the basis of very valuable contributions, as can the study of known approaches in new contexts or domains.

After reviewing the paper and discussion in detail, the AC agrees with the perspective that the size of the contribution of the present paper is limited. It is true that simple and/or known approaches can be the basis of valuable contributions. However, to achieve this, the novel insights must have an appropriate depth and provide a corresponding value to the community. Here, depth of analysis does not go very far beyond comparisons of the obtained scores. It would be interesting to see additional analysis to empirically demonstrate the mechanisms behind the obtained performance, i.e., the how and why of the role that the proposed modifications play. Just to give one example, a key contribution is the introduction of conservatism via a regularizer that injects random noise on the return. In RvS paper (one of two baseline approaches used in this work), a whole section of the paper was dedicated to "ARCHITECTURE, CAPACITY, AND REGULARIZATION", identifying this interplay as essential for the performance of these methods. The present work proposes an alternative regularizer, and show empirically that it performs well on the tasks studied. However, in the present paper we gain little further insight on questions such as how the behavior of the proposed regularizer differs or aligns with that observed in the original work.

In sum, the paper presents an appealing approach but does not yet demonstrate a contribution that warrants an accept recommendation.
justification_for_why_not_higher_score: The paper shows promise but does not make a large enough contribution to warrant acceptance.
Decision: Reject