Question: How was the offline dataset generated?
Evidence: We test the SotA offline RL method conservative Q-learning (CQL) ) and online RL method Randomized Ensembled Double Q-learning (REDQ) under the offline-online setting. Here, we choose a widely used locomotion task Hopper, and use the dataset hopper-medium-replay-v0 in the D4RL benchmark  as the offline dataset, which contains diverse experiences from different policies. Specifically, we first pre-train the agent with the offline dataset for 100K steps. Then, the agent is fine-tuned online by alternately conducting the interaction and updating process, where the agent interacts with the environment for 1K steps, and is updated for 10K steps. Such interleaving process ends until the total online interaction steps reach 90K.
The offline-online setting in this task should be a favourable one for policy learning, because both diverse offline data and online interactions are available here. Therefore, we expect following results: the initial offline training using the dataset will provide a relatively good but not perfect starting point for the agent, and then the agent can be further improved by the online alternating process since online interaction data can be obtained, and finally we may acquire a well-performed policy, even with normalized score close to or better than 100 (i.e., the performance of an expert).
Nonetheless, experiment results are shown in Figure, and they do not totally meet our expectation. Specifically, starting points of two curves show that the offline algorithm CQL and online algorithm REDQ can both obtain normalized scores greater than 0 (i.e., the performance of a random policy) but not very high after the initial offline training process. This result is what we expected, because CQL is designed for the offline setting, and online off-policy methods, such as REDQ, can also learn from the diverse and large offline dataset even though it is fixed. However, although the starting scores (below 30) are far from the expert score and leave a large room for further promoting, both algorithms have troubles in the following online process. REDQ suffers from severe instability issue and its performance drops significantly during online learning, and in the end, the policy almost degenerates into a random one. One the other hand, even though the CQL agent can keep stable during learning, its improvement is very limited.
These results indicate that pure online RL algorithm may be problematic for effectively handling offline data and online interaction data in a single training process, and pure offline RL algorithm cannot make good use of valuable online interaction data due to its conservative updating strategy. Such observation motivates us to couple them in an organic way.
We test the SotA offline RL method conservative Q-learning (CQL) (Kumar et al., 2020) and online RL method Randomized Ensembled Double Q-learning (REDQ) (Chen et al., 2021) under the offline-online setting. Here, we choose a widely used locomotion task Hopper (Todorov et al., 2012), and use the dataset hopper-medium-replay-v0 in the D4RL benchmark (Fu et al., 2020) as the offline dataset, which contains diverse experiences from different policies.
Answer: The offline dataset comes from the commonly used benchmark D4RL[REF4] for offline RL.