Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new offline RL algorithm that aims to adapt the policy to a target domain with insufficient data with the help of abundant data in a source domain. The authors propose their method, DARA, which applies a reward penalty that correct the dynamics shift between the source and target domains. The authors combine DARA with previous model-based and model-free offline RL methods and show that DARA is able to achieve performance improvement over prior offline RL methods without the reward augmentation in both simulated and real-world tasks.
main_review: Pros:
1. The paper is clearly written and easy to understand.
2. DARA appears to be the first method that addresses the problem of domain adaptation in the offline RL setting.
3. The authors show extensive empirical results of DARA on both simulated and real-world experiments and evaluate the performance of DARA combined with most of the prior offline RL methods and observe reasonable improvement.

Cons:
1. I think the algorithm is a bit incremental. It is in some sense a combination of Eysenbach et al. 2021 and conservative penalties introduced in prior model-based and model-free works. It seems to be a bit straightforward and therefore the novelty is a bit limited.
2. DARA introduced additional hyperparameters compared to prior offline RL papers, i.e. the coefficient that controls dynamics shift, which could make the method hard to tune. DARA also requires training two classifiers, which also add another layer of complexity in terms of reproducibility. I also didn't see the mention of the hyperparameter settings. It would be important to know how the authors select hyperparameters for all of the environments since hyperparameter tuning in offline RL is usually expensive and potentially dangerous. It is also important to see the error bars of the results.
3. The real-world experiments seem to use state inputs. It would be really intriguing if the authors could show good transfer results with image inputs in the sim-to-real setting.
summary_of_the_review: I think the paper proposes a new algorithm that tackles a somewhat new problem, but the methodology lacks originality and the hyperparameter details are missing. Therefore, I would vote for a weak reject.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Offline reinforcement learning could enable one to train a data-driven decision making engine from purely offline data. However, the performance of polices learned through existing offline RL methods heavily depend on the quantity and quality of the offline data. When the data is scarce for a task, existing offline RL methods trained on limited offline data could perform poorly. 

This work tackles the data efficiency issue of offline RL by transferring knowledge from a source offline dataset to a target offline dataset following an environment dynamics different from the source offline dataset. The proposed method augments the reward function by penalizing the state-action-next state transition tuples that are rare in the source environment. This reward augmentation strategy directly accounts for the dynamics shift between the source and the target domain. On several simulation a real-world tasks, the proposed method significantly improved the data-efficiency of existing offline RL algorithms. 
main_review: ## Pros
* The proposed framework is well justified for both model-free and model-based approaches. And the framework is easy to implement.
* Figure 3 clearly shows that the reward penalty obtained through domain classifiers can aid the knowledge transfer when the dynamics shift.
* The propose method has been tested on real environment and show improvement over the baseline aggregation strategy. This highlights the importance of dynamics-aware transfer in offline transfer RL for practical applications.

## Cons/questions/suggestions
* __Relationship to importance sampling__ The proposed reward compensation term is strongly related to importance weights. What if one simply weight each of the original offline data point based on the importance weights and train a policy using the combined importance weights source dataset and target dataset? Do you expect any advantage of the proposed method over importance weighting? 
* __Explanation for Lemma 1__ Could the authors elaborate on Lemma 1? Even if the environment has deterministic transition dynamics, given a finite randomly sampled dataset $\mathcal{D}$, the learned dynamics $\hat{T}$ model could still be different from the true dynamics $T$. Consider a simple tabular setting where there is a single initial state, a single allowed action, and two different states that could transition from this initial state and the only allowed action with equal probability. If I was given a dataset with ten state-action-next state tuples, and four of them correspond to the transition state 1 while the other six contain the transition state 2. Then I will get a wrong estimate of the environment dynamics. Please correct me if my understanding of the dynamic shift is incorrect. 
* __Impact of offline dataset quality__ Given different offline dataset quality, the proposed framework's improvement over the simple aggregation of source and target should vary. It would be nice for authors to compute the average improvement for all six algorithms over Hoper and Walker2d and see if there's any consistent trend. If so, it is important to discuss what insights we could learn from the results.

## Minor comments
* Could the authors comment on the sensitivity of the algorithm on the hyperparameter $\eta$ in $r_{t} \leftarrow r_{t}-\eta \Delta r$? How would one pick the value of $\eta$ in practice? 
summary_of_the_review: This well-written paper studies the important problem of transferring knowledge to improve sample efficiency for offline RL. The proposed reward augmentation framework is well justified for both the model-free and model-based settings from the perspective of variational lower bound and model-bias when the dynamics shift, respectively. Also, the experiment results support authors' claims well. Overall, this is a good submission that can inspire future works along this direction.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on the dynamics shift problem in the offline reinforcement learning setting, with the assumption that the estimated MDP from the offline dataset is deterministic. It proposes DARA, a framework that relaxes the requirement of data (both the amount of data and the reward information contained in the data) needed from the target task. By capturing the dynamics shift between the source and target environments, DARA modifies the reward gained in the source environment to discourage transitions that have a smaller probability to happen in the target environment. The framework works with both model-free and model-based settings. 
main_review: The paper shows several strengths:

(+) The method proposed in the paper contributes to improving the data efficiency in the offline reinforcement learning setting. It both reduces the amount of data needed from the target environment and relaxes the requirement on the reward information contained in the target environment’s dataset. 

(+) The theory behind the idea is well explained. The blue and red color letters help distinguish between the source and target MDPs.

(+) The implementation of the method is also discussed, making the method applicable. The hyperparameter settings are reported in the appendix.

In the meanwhile, there exist concerns and limitations:

(-) My concern comes from the experiment section. The experiment results are averaged over 3 seeds only, which does not give an accurate estimation of the true performance. Even though there are 4 environments with 4 dynamic shift cases each and DARA improves the performance in most cases, the risk still exists that the averaged performance of one single test is inaccurate. Therefore it could be better if more random seeds can be tested.

(-) It seems like the proposed method is limited to the case that the target environment is restricting transitions that exist in the source task. It is not clear to me if the method still helps when the target MDP adds new transitions. For example, in the Map task, collecting source dataset with the obstacle and collecting target dataset without the obstacle. In this case, it can be harder for the framework to capture the change in the transition dynamics, thus harder for the agent to figure out the new optimal policy (the shorter path without the obstacle). If this is true, the cases that DARA can be applied to might be limited, though I believe it is still helpful in cases like proposed in the paper.

summary_of_the_review: I recommend acceptance. The paper introduces a framework to adapt to the target environment with a relaxed requirement on data from the target task. I believe this work contributes to improving the data efficiency in the offline reinforcement learning setting, even when the access of the reward function in the target environment is limited. There exists shortages and limitations as listed, but overall, I consider the positives to outweigh the negatives.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The given work proposes the addition of a dynamics aware framework for Offline Reinforcement Learning. It motivates that  offline reinforcement learning suffer from a dynamics shift in SARSA transitions, which need to be effectively handled. They propose modifying reward in source offline dataset, penalizing it with the dynamics aware reward transition, setting the lower bound. The reward augmentation is done with explicit policy constraints as well as implicit policy constraints.They evaluate this approach on various robotic manipulation task from the Gym-MujoCo suite, evaluating and showcasing perforomance on four offline RL algorithms.


main_review: 
Strength:

* Overall, the paper attempts to answer important question in offline reinforcement learning. 
* The given empirical evaluation seems theoretically grounded, results seem reasonably significant, with improved performance over the baseline source dataset.

Weakness:
* The motivation of the proposed approach needs to be expanded further. There is a single line which is hard to fully understand. "Motivated by the (approximate) equivalence1 between the dynamics change and the rewards mod- ification (Kumar et al., 2020b; Eysenbach & Levine, 2019; Eysenbach et al., 2021), we propose to modify rewards in the source offline data to compensate for the dynamics shift. "


* Clarity of writing is a major weakness of the paper. It's not very well organized, logical flow is missing. As one example, the "Related Work"(Section 5) is towards the end of the paper, after the approach has been extensively described. A few more highlighted below to fix:

* "This modification aims to discourage the learning from these source offline transitions (next-state) that are hard to be achieved in the     target environment": Can the latter part of the statement be more precise? This could be misinterpreted as a goat state to be.

* Algorithm 1 Step 1: Perhaps clarify both the models are learnt independently for each of the datasets, ie 4 models.

* Section 4.1: What does it mean for a trajectory to be "optimal"? In reality, is this actually binary?

* Section 4.1 Figure 2 isn't clear to understand

* Section 6.1 There is no dashed green line in Figure 3, as mentioned in Section 6.1

* Section Was the source(noisy) dataset also collected from a D4RL policy?

* "Specifically, we out- line state-action-next-state: besides characterizing the state-action distribution shift, we additionally penalize the agent with a dynamics-aware reward modification.": This line isn't very clear to me.

* Section 4.1: Typo: "the limit"

summary_of_the_review: Overall, the given work is theoretically grounded, providing incremental improvement over the past work in this area of tackling distribution shifts for offline RL learning. The results seem promising. However, the paper is not the easiest to read and interpret.  I'm willing to reconsider my decision, based on the feedback addressed by the authors.

Edit: Updated score based on reviewer feedback
review_title: null

Meta-review: **Summary**

This paper proposes a method to do a sample-efficient offline domain adaptation method where the setting requires one to have abundant amount of data from the source domain but limited amount of data available in the target domain. The proposed approach DARA achieves that by accounting for the the dynamics shift between the source and the target domain via a reward penalty. The paper shows promising experimental results.

**Final Thoughts**
Overall the paper is well-written, and it addresses an important problem. The reviewers are mostly positive about the paper at the end. The authors did a very good job addressing the concerns raised by the reviewers. Reviewer 73ni had concerns about the novelty of the approach, it would be nice if the paper can make it more clear about the novelty of the proposed approach compared to the other existing methods in the paper. I  would also recommend the authors to incorporate the feedback and the suggestions made by the reviewer into the camera-ready version of the paper.
Decision: Accept (Poster)