Augmenting Offline Reinforcement Learning with State-only Interactions

Authors: Shangzhe Li, Xinhua Zhang
Primary area: reinforcement learning
Keywords: Offline Reinforcement Learning, Generative Model, Data Augmentation, Trajectory Stitching

Abstract:
Batch offline data have been shown considerably beneficial for reinforcement learning. Their benefit is further amplified by upsampling with generative models. In this paper, we consider a novel opportunity where interaction with environment is feasible, but only restricted to observations, i.e. *no reward* feedback is available. This setting is realistic, because simulators or even real cyber-physical systems are often accessible, while in contrast reward is often difficult or expensive to obtain, similar to imitation learning settings. As a result, the learner must make best sense of the offline data to synthesize the most sample-efficient scheme of querying the transition of observation. Our method first leverages online interactions to generate high-return trajectories via conditional diffusion models. They are then blended with the original offline trajectories through a stitching algorithm, and the resulting augmented data is applied to downstream reinforcement learner. Superior empirical performance is demonstrated over state-of-the-art data augmentation methods that are extended to utilize observation-only interactions.

Decision: reject
Reviewer scores: 5, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730694822452

thread_type:
official_review

<thread_object>
creator: reviewer_awxi

<content>
summary:
This paper proposes a data augmentation strategy DITS to investigate the following question: can we improve trajectory generation, as a data augmentation approach for RL, by making efficient use of state-only interactions?

strengths:
1. The goal of generating high-return trajectories via augmentation is well-motivated and worthwhile to the research community. I expand upon this more below.
1. Empirical setup is adequate. I think it's very appropriate to use behavior cloning (BC) as one of the evaluated algorithms, since DITS aims to generate high-return trajectories. If DITS performs well with BC, then that directly supports this claim.
1. Figure 1 does a nice job illustrating how DITS differs from previous approaches while also highlighting the advantage of DITS.

weaknesses:
I vote to reject primarily because DITS seems to outperform baselines in only half of the considered tasks. 

* **DITS outperforms baselines in only about half of tasks.** Table 1 reports the average return +/- 1 standard deviation (68% confidence interval). DITS's confidence interval overlaps with the intervals of other baselines in about half of the tasks, indicating that DITS does outperforms baselines. For instance:
  * All Med-Expert experiments, both BC and TD3+BC
  * Medium HalfCheetah, both BC and TD3+BC
  * Med Replay HalfCheetah TD3+BC
  * Kitchen Partial, both BC and TD3+BC
  * Antmaze U-Diverse both BC and TD3+BC, Umaze BC

* **Standard errors are not propagated properly.** When averaging performance across multiple experiments, the standard deviations should be added in quadrature and not simply averaged [1]. 

* **Standard deviation is likely not the greatest measure of uncertainty for this empirical analysis, since it assumes the underlying distribution of returns is normally distributed -- which is often not the case in RL.** A bootstrap confidence interval is more appropriate, as it does not assume the returns follow a particular distribution [2]

* **The related work is minimal and excludes a number of relevant data augmentation works.** Corrado et. al [3] focuses on generating expert quality augmented data for offline learning, similar to what DITS aims to do. Pitis et. al [5] is another model-based data augmentation technique, and Sinha et. al [4] consider offline RL when evaluating their augmentation strategy. I also list a few other data augmentation works [6-9] that can provide readers with a more comprehensive overview of the data augmentation research landscape and better contextualize DITS.

* "merely generating high-return trajectories is sub-optimal as it may overfit those regions" This statement seems plausible, though the paper does not provide evidence to support this claim. Kumar et. al [10] provide a theoretical argument that may support it; offline RL performs better with noisy expert data than purely expert data, indicating that high-return data may not be sufficient.

[1] Taylor, John R. An Introduction to Error Analysis: The Study of Uncertainties in Physical Measurements. 1997.

[2] Agarwal et. al. Deep Reinforcement Learning at the Edge of the Statistical Precipice. Neurips 2021. https://arxiv.org/abs/2108.13264

[3] Corrado et. al. Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning. https://arxiv.org/abs/2310.18247

[4] Sinha et. al. S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning in Robotics. https://arxiv.org/abs/2103.06326

[5] Pitis et. al. MoCoDA: Model-based Counterfactual Data Augmentation. NeurIPS 2022. https://arxiv.org/abs/2210.11287

[6] Corrado & Hanna. Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates. ICLR 2024. https://arxiv.org/abs/2310.17786

[7] Pitis et. al. “Counterfactual Data Augmentation using Locally Factored Dynamics.” NeurIPS 2020. https://arxiv.org/abs/2007.02863

[8] Abdolhosseini et. al. “On Learning Symmetric Locomotion.” ACM SIGGRAPH 2019. https://www.cs.ubc.ca/~van/papers/2019-MIG-symmetry/index.html

[9] Adrychowicz et. al. "Hindsight Experience Replay." NeurIPS 2017. https://arxiv.org/abs/1707.01495

[10] Kumar et. al. When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? ICLR 2022. https://arxiv.org/abs/2204.05618

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695786846

thread_type:
official_review

<thread_object>
creator: reviewer_ahba

<content>
summary:
The paper proposes a method to use a diffusion model trained on offline trajectories alongside an inverse dynamics and one-step dynamics model to synthesize new trajectories of higher return from low value trajectories.

strengths:
The proposed method performs well compared to baselines in D4RL.

weaknesses:
1. The algorithm seems like a minor iterative improvement without significant novel contribution. The “state-only trajectory modeling” which the paper highlights alot has been used in several previous works, including Decision Diffuser which the authors acknowledge (and in fact use almost the same setup).
2. The only addition over decision diffuser is the new stitching, but this approach seems very problematic. For one, the new transition \hat{s’} being sampled from a different trajectory will almost always be much less likely than s’ in the predicted trajectory. This is simply because most states in most trajectories are likely very different than any given state compared a predicted next state.
3. The proposed method is also just a minor variation from DiffStitch and SynthER.
3. It is puzzling why the authors use much slower autoregressive generation of the trajectory with the diffusion model instead of joint trajectory diffusion. It is unclear if using a diffusion model here affords any advantage.
4. Even more evidence that a diffusion model is not required is that the one step dynamics model p(s’ | s) actually used to verify if stitching to some state is allowed are simply modeled as Gaussians anyway. Why not just these models and remove the diffusion models?

questions:
1. In section 3.2, the authors claim the last C steps are needed to make the trajectory consistent. However the tasks in D4RL that are evaluated here are all Markovian. Additionally here the authors claim just using s_{t+1} to run the denoising procedure instead of also using the history would make it autoregressive decoding. However, the current process is also autoregressive decoding. It seems like the authors are stating conditioning on history is not autoregressive. I would like some clarification on all this.
2. Why use autoregressive generation to begin with? This seems very inefficient when the diffusion model can be directly used to jointly diffuse whole trajectories (similar to Diffuser).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702226092

thread_type:
official_review

<thread_object>
creator: reviewer_dbgz

<content>
summary:
This paper introduces a new diffusion-based data augmentation method named DITS. DITS considers the setting that only state interactions are available with the interaction. The generator based on conditional diffusion models allows high-return trajectories to be sampled, and the stitching algorithm blends them with the original ones.

strengths:
1. The idea of combining diffusion-based trajectory generating and trajectory stitching is novel and interesting.
2. The writing of the paper is clear and easy to follow.

weaknesses:
1. The main contribution of this paper can be divided into two aspects. One is to generate more accurate transitions based on the state-only observable environment. However, this part is more like directly using diffusion-based planning methods to do data augmentation, while remaining the rewards as predicted. It is doubtful that this scheme is useful enough as it requires too much interaction with the environment like online reinforcement learning although it doesn't need the reward. Also, an additional ablation study is needed to show the effectiveness of this generation pipeline like using Diffuser, DD, or other trajectory-level data augmentation methods to generate trajectories given that the methods can access the dynamics of the environment.

2. The second aspect is to blend the high-returned trajectories with the original trajectories. The current paper lacks an analysis of the importance of the trajectory stitcher.

3. The main results in Table 1. only contain baselines BC and TD3+BC, which ignore many state-of-the-art offline RL baselines including CQL, IQL, DT, etc. 

4. The data augmentation baseline only includes SynthER as the diffusion-based data augmentation methods, while currently there are many latest diffusion-based data augmentation methods like MTDiff. Given the fact that SynthER is a transition-level generation method, the comparison is unfair and incomplete.

questions:
1. In Line 205, What is the meaning of m?
2. What is the meaning of equation 8?
3. How does the iteration rounds K affect the performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714334262

thread_type:
official_review

<thread_object>
creator: reviewer_ms28

<content>
summary:
This paper introduces Diffusion-based Trajectory Stitching (DITS) to enhance offline reinforcement learning by using state-only interactions. DITS consists of two basic parts: a DD-based trajectory generator and a stitcher. First, the generator obtains state-only trajectories from the real environment, and the corresponding actions and rewards are filled by an inverse dynamics model and the diffusion model, respectively. Then, the stitcher finds possible $(s, s')$ to stitch them up, forming new trajectories. Finally, the dataset containing both real trajectories and stitched trajectories are used for the policy training. The experiments show performance gain of DITS over the underlying offline RL algorithms.

strengths:
- The settings of this paper is quite practical. It addresses real-world scenarios where reward feedback is difficult to obtain, making the method broadly applicable to various fields like healthcare and inventory control.
- This paper contains intensive experiments that show the effectiveness and robustness of DITS, proving its potential for real-world application.

weaknesses:
See questions.

questions:
- In Figure 2, the trajectories from DITS trajectory generator are annotated as "trajectories from good policies". However, they are actually obtained by interacting with the real environment using a planning-like method. DITS uses the DD-based trajectory generator to plan the following states, and predicts the action by an IDM. I think this annotation is somehow misleading, and pointing out they are from the **real environment** might be better.
- In Line 16, Algo 2, as you train various offline RL methods besides BC on $\mathcal{D}_{K+1}$, it is imprecise to say "for behavioural cloning training".
- In Algo 4, how do you train the state-value function $V$? Is it trained from scratch for each iteration $k$, or only fine-tuned?
- For experiments, some of the ablation studies takes BC as the base RL algorithm. However, as DITS generates rewards for new transitions, and Appendix E proves the accuracy of the generated rewards, training BC is a little wasting because it does not utilize reward information. TD3+BC, a simple modification of BC, seems more likely to take full advantages of DITS. Could you please conduct ablation studies that comparing DITS against DITS w/o sitcher and DITS w/o generator, as well as the ablation on the number of generated trajectories, using TD3+BC as the base RL policy? If they cannot be completed due to the time limit, I will also appriciate it.
</content>
</thread_object>

</thread>


Metareview: