Venue: ICLR 2023 Workshop RRL

Reviews:

final_score: rating: 4: Good paper, strong accept
review_content: # Summary
This paper aims to investigate the use of generated data in reinforcement learning for allowing RL to be more data efficient: using smaller datasets in the offline setting, or with more sample efficiency in the online setting. The paper proposes using diffusion models for modeling existing data, which are then used to generate synthetic data, adding, or replacing the training data for any off-policy RL algorithm, without making any algorithmic changes.  In the offline setting, on the D4RL benchmark, model-free offline RL algorithms can retain performance and sometimes benefit from training on synthetic data,  or allow for reducing the volume of real data. In the online setting, augmenting RL algorithm(s) (SAC) with offline data is shown to increase sample efficiency. They also show their method can outperform explicit data augmentation or algorithms specifically designed for data efficiency (REDQ).  

#  Strengths and Weaknesses
## Strengths
- The paper is clear to follow, and overall goals are well-motivated. Accelerating RL through upsampling prior data is also relevant.   
- While diffusion models have begun to be explored in RL in offline reward-conditioned settings, this seems to be the first work that has used diffusion generative models for increasing efficiency of existing model-free RL algorithms, and in the online setting. 
- Experiments illustrating the quality of synthetic distributions, in comparison to augmentation, and state marginal distributions are also convincing towards diffusion model’s ability to capture dataset statistical. 

## Weaknesses
- Minor: while the reported wall-clock times of SynthER and REDQ are appreciated, it would also be useful to have the wall-clock time of your implementation of SAC. This would be useful to help better understand the wall-clock versus sample-efficiency tradeoff.   
## Questions
- In the online RL experiment, I am confused by the choice to upsample every 10K real samples to 200K (20x scaling), but only use an UTD of 8? Is this ratio of 8, in reference to the original 10K samples, or after upsampling (200k)? 

review_title: Promising results for using synthetic experience

final_score: rating: 4: Good paper, strong accept
review_content: ## summary
The authors propose synthER, a method that can augment data necessary for RL training. Any offline and off-policy RL algorithm can utilize for the synthER. Authors show that performance is at least comparable or superior to using original dataset in offline situations, also show good results with a lower UTD ratio in online situations. The data created is sufficiently diverse and helpful for training a policy.

## clarity
The method and results are simple and clear.

## significance
* It is shown that the diffusion model can well augment the behavior policy data.
* It shows that generative models can be used well in situations where the cost of collecting transition data is high.
review_title: Fits well with the purpose of the workshop

Decision: Accept(Spotlight)