Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors tackle a long-known problem in offline learning where the performance of the RL agent is highly dependent on the distribution of high and low return trajectories. It follows that in datasets that contain a high percentage of low-returning trajectories and a low percentage of high-returning trajectories. The Agent will be overly restrained by the low-return trajectories. 

The authors tackle this problem by implementing trajectory weighting, where different weights are assigned to different trajectories of different importance, naturally a more important trajectory would have a more important weight. After assigning weights to each trajectory, the authors add an entropy regularization term.
strength_and_weaknesses: Main concern: 
- How prevalent is this problem in offline RL? I see the table in A.3, but it does not give the whole image of this problem. Is this problem seen in other mainstream datasets?
- How does this method compare to the other sampling strategy methods? Why should this method be used over the other ones?
- Would making the model learn the policy on offline datasets make the model overfit?
- Is this method robust under different distributions of dataset? For example in Figure 1, what if the dataset has two density spikes, on 0.0, and 0.8.
- In the result part, there is no comparison between the two methods, the authors only gave experiments on their method vs the vanilla method of no weighted distributions.
clarity,_quality,_novelty_and_reproducibility: About Novelty: 
- This problem itself is not novel, as a lot of work in the past has been about solving the distribution skew in offline datasets. But their main approaches are by skewing the sampling methods. Thus the idea of introducing a weight in sampled data is novel. 

About Organization: 
- I feel that the paper’s structure is very clear and concise, from preliminary to results. From what the problem is, and why the problem needs to be addressed, all the way to the author’s methods.
- Section 3 has a lot of reasons for the motivation of the work, especially in figure 1, the analysis is clear and very insightful.
summary_of_the_review: About Challenge: 
- Without collecting additional data, how can the performance of the policy be improved? Previous works have tackled this by skewing their sampling.

About Contribution: 
- The authors reweight the transitions in the dataset, and regularize it by adding an entropy regularization term. Then they further change the reweighting term to equation 11 in the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to reweight episodes in the replay by their advantage values. The paper claims that doing so is especially useful when there are many more low return episodes in the replay buffer compared to high return episodes. 
strength_and_weaknesses: Strength

The proposed idea is intuitive and the claim that the idea is useful when there are many more low return episodes than high return episodes makes sense. 

Weaknesses

Lack of comparison to simple baseline. The issue that the paper studies is well-known, and a common technique to tackle it is to ensure each batch of episodes used to update the networks contain half successful, and half failed episodes. I would have liked to see the comparison to this simple and common baselines.

Parts of the writing are not clear. For example:

- "cold start performance boost" in the Introduction.

- "Most offline RL algorithms are anchored to the behavior policy". What are anchors mean, and where is the evidence that support this claims?

- How should I interpret figure 2? The caption states that the proposed methods outperform baselines, but how is this conclusion supported by the figure?
clarity,_quality,_novelty_and_reproducibility: The idea proposed in the paper is novel, as far as I know. But the paper lacks comparison to simple baseline that is commonly used to tackle this problem. The caption of the figure does not clearly explain the reasoning as to why the information shown demonstrates that the proposed method outperforms baselines.
summary_of_the_review: I look forward to the rebuttal. The idea is simple, and may therefore be used widely. But as it currently stands, it is unclear if the method outperforms simple and common baselines.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores a particular setting of low sparse high reward trajectories in an offline RL datasets. Offline RL algorithms that explicitly do Behavioral Cloning (BC) have a hard time learning good policies in these settings. The authors propose a simple reweighing scheme to induce an artificial dataset with a higher performing behavioral policy. They also show theoretical analysis regarding the bias of this estimate for deterministic settings and also introduce Return Positive Sided Variance as a measure of quantifying the variance in the returns of trajectories among the dataset. Finally the experimental results show that the weighted sampling of the dataset significantly helps in the artificially generated datasets with varying RPSV however the performance gains on true D4RL tasks are marginal especially for methods without explicit BC like CQL and IQL.
strength_and_weaknesses: Strengths:

- The weighted sampling helps combating the need for a near expert dataset for approaches with explicit behavioral cloning terms.
    
- The weighted sampling approach has been demonstrated to not hurt the performance even for well balance datasets across various modern offline rl approaches (CQL and IQL).
    

Weakness:

- The results while impressive is not surprising.  While the gains are signficant for BC and TD3+BC, it is well established that these methods will suffer in sparse high reward trajectory settings. The gains over CQL and IQL are marginal for the current D4RL benchmark.
    
- While the authors introduce RPSV as a measure of tracking the variance in the dataset, it does not answer the question of “When is it necessary to do weighted sampling?”. i.e. The RPSV metric follows a inverted U curve as we increase sigma from 1% to 100% in the artifical datasets that were created.
    
- For this approach to be applicable, the data needs to be collected such that the trajectories are complete; i.e. The behavioral policies have to start from the starting state distribution. While this is not that hard of a constraint it is to be noted that this approach does not make sense if the collected policies start from different parts of the state-space. (i.e. trajectory stitching settings.)
    
- Probability of improvement over uniform sampling 3(b) is very skewed compared to the actual amount of improvement we see in figure 3(a). Here it seems that our approach is equally valid for both IQL and TD3+BC according to fig 3(b) whereas 3(a) tells a different story. overall I feel fig 3(b) does not add much value.

Minor points.

- ““However, offline IL requires expert and random data to be separated while we neither assume separated datasets nor access to expert data.” ([pdf](zotero://open-pdf/library/items/BFZJEQLU?page=9)) ” This is not fully accurate as we are using the rewards to rescale sampling we are assuming proxies to both a “separate dataset” and access to “expert data” offline IL does not require a expert data, its just that it wont work without one, as with the case with this approach.-  
    
- sigma has different scale in page 14 and 15. (0.01 and 1 )
    
- The formatting /Bolding has errors in the tables in appendices.  
    
- It would be wonderful to include plans to release the sampling code snippets for the public.
    
- IQL numbers for hopper is not matching with the author provided numbers, It might be good idea to doublecheck them.
clarity,_quality,_novelty_and_reproducibility: -
summary_of_the_review: Overall the paper presents a relative simple weighted sampling method which can be a valuable tool in an offline-rl toolkit, however, the metric RPSV presented in the paper fails to capture the scenario where the method can/should be applied. This can be an interesting paper with a better metric / analysis on when the approach should be considered during offline RL, however, under the current version, I am leaning toward a rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper is trying to solve the problem that when there exists a scenario in the offline RL dataset: Mostly low return trajectory, few high return trajectory, indicating the mixed policies lying behind, the results could be not ideal.

By introducing the RPSV, the variance of the positive case and negative cases are well captured in the original dataset and is helping to infer the distribution of high-return and low-return policies.

And the author base on the variance of PSV proposed two weighting methods. Abundant experiment on CQL, IQL, TD3+BC, sole BC, etc. from various environments like classic control has justified the effectiveness of different methods. Detailed explanations of experiments and deductions are shown in the appendix.

strength_and_weaknesses: Strength:

- The novelty is clear. The author is trying to analyze the mixed dataset’s efficiency in a theoretical way, representing and capturing the tendency of PSV.

- The author proposed the two methods in a rigorous logic, return-weighting and advantage weighting are straightforward by the definition. 

- The sufficient experiments are evidently showing that the proposed two methods on different baseline methods, could better capture the dataset’s trajectory original feature, and by reweighting the importance of the high-return trajectory, the method’s effectiveness is well improved.

Weakness:

- It’s noticeable from Figure2 that the two weighting methods sometimes have a different appearance, it would be better if the author could analyze the different suitable situations for the above different two methods.

- A little confusing part may be the introduction of negative-sided variance(NSV), because the majority part of the paper is mainly focusing on the RPSV, is NAV the NSV used in the implementation part of two re-weighting methods? 

clarity,_quality,_novelty_and_reproducibility: The work is of good quality, overall clear in the explanation. The example code was not provided, but the instructions on environment and realization details are well presented.
summary_of_the_review: The author is trying to use novel ways to analyze the distribution of offline training datasets. Which helped to improve the performance of the offline RL methods and also justify the applicability to other methods. The paper is presented in an explicit way, also with necessary details from the appendix part. 

It’s meaningful work to help offline RL algorithms improve the exploitation in the limited amount of high return trajectories.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies an important problem, has a novel and intuitive idea (although its novelty is rather limited), is well-written, and the method is supported by the right amount of experimental results (although the results are not that surprising). Overall, it is a well-executed paper that could be useful for the community. I would thank the authors for responding to the reviewers' comments and made changes in their paper. I would suggest they take the rest of the reviewers' comments into consideration in preparing the final version of the paper.  

justification_for_why_not_higher_score: The novelty is limited, the results are not surprising, and the lack of comparison with certain baselines. 
justification_for_why_not_lower_score: Although the novelty of the method is limited, overall, it is a well-executed paper that could be useful for the community.
Decision: Accept: poster