Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present MISA, a framework for offline RL based on mutual information based regularization over states and actions. The authors motivate their approach step by step and demonstrate how TD3+Behavior cloning and CQL are related to their proposed approach. Finally, the authors provide empirical results relative to key baselines on the D4RL benchmark while also providing ablation studies and analysis. 
strength_and_weaknesses: Strengths: 
- The authors present a clear buildup to their approach in math and justify each step.
- The authors contextualize their approach pretty well with respect to prior art on a theoretical level.
- The approach gets pretty good performance relative to baselines on the D4RL benchmark. 

Weaknesses: 
- The novelty is not terribly high in comparison to past work on offline RL and mutual information estimation. 
- The introduction of the mutual information regularization in equations 5 and 6 struck me as quite intuitive in nature and not really originating from first principles.

Both: 
- I do really appreciate that the authors clearly highlight limitations of their approach when doing learning from data that is using far from an expert policy. However, on the other hand, it is quite a significant limitation relative to baselines. 
clarity,_quality,_novelty_and_reproducibility: The writing quality is pretty good overall and I really appreciate the theoretical discourse of this paper. The novelty is not very high, but I do appreciate that the authors also clearly present their contribution relative to prior art. The authors have provided their code in the supplemental material. 
summary_of_the_review: My biggest concerns about this paper are the relatively low amount of novelty and the fact that the overall motivation is a bit intuitive in nature. However, I lean towards acceptance because I do really appreciate the theoretical discourse and experiments. I feel that in light of this context the authors have provided, the paper is fleshed out enough to make a contribution to the conference, particularly because of its focus on such an important area. 

Update After Author Response: 

I have read through the other reviews and responses to each review. I have kept my score unchanged, but I can't really serve as a strong advocate for this paper either. It is quite outside my area of expertise, so I feel that my confidence is quite low about what the right thing to do here is as I sympathize with all of the points made.

I agree that the novelty is not huge here, but also hear the authors that it is a significant contribution beyond the past literature. However, the results not being too significant could indeed be a potential issue for adoption of this method. Moreover, I think the authors are too quick to discount the importance of not performing well on data from a uniform policy. It is certainly not common for offline RL benchmarks, but if the data compiled by our behavior policy is already "expert" level then offline RL is only improving things at the margins. In typical off-policy RL we start with a uniform policy and while I acknowledge that these are certainly different settings, I think offline RL should be considered harder than off-policy RL (due to lack of access to the underlying behavior policy and lack of access to a simulator during training) and not a strictly easier setting (akin to some slightly more ambitious form of imitation learning).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a regularization method for constraining policy improvements in offline RL. The proposed regularization is based on a lower bound to the mutual information between states and actions, and attempts to mitigate the issue of distribution shift that arises when the policy is queried on out-of-distribution actions.
strength_and_weaknesses: Strengths

1. The proposed regularization is simple to implement and is well-motivated. It can be easily integrated with most offline RL framework as an additional term in the objective. 

2. The practical algorithm is discussed in detail and implementation details are clearly mentioned. 

3. The ablations and embedding visualizations provide nice intuitions about what MISA is learning empirically. 

Weaknesses

1. Conceptually it is unclear how exactly is MUSA better than prior methods like CQL in mitigating distribution shift. There isn't enough intuition provided regarding this, beyond reference to experimental results and "better mutual information estimation" (in section 4.4) Why exactly should the MI estimation be better?

2. The empirical results do not show strong benefit of MISA over baselines, partly because the hopper, cheetah, walker tasks are almost saturated. For the other tasks in Kitchen, Adroit, and Maze, not all baselines are evaluated, so empirical benefits are unclear.

3. The lower bound result in Theorem 4.1 does not discuss anything about how tight the bound is, and whether the bound still holds in practice (i.e. in the experiments) with several approximations, including function approximations.
clarity,_quality,_novelty_and_reproducibility: The contribution is novel in my understanding, and sufficient details for experiments are provided. The writing is clear and easy to follow.
summary_of_the_review: Although there are some weaknesss especially with expeirments that I have pointed out, and which should be addressed, I am leaning towards acceptance because the key insight is simple, neat, and seems to be promising for reducing distribution shift in offline RL. However, the additional experiments/comaprisons are needed to confirm the beenfits.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new framework for handling distribution shift in offline RL, based on a mutual information based regularizer. The mutual information term gives a measure of the data collecting policy's actions on certain states. The regularizer computes a lower bound to the overall policy improvement step and the paper shows how this lower bound can still be useful for leading to overall improvements in offline RL. 
strength_and_weaknesses: 	- The paper proposes a novel regularizer for offline RL and provides intuitions for why such a regularizer might be useful. Existing works have proposed conservative updates to offline RL, but this paper argues the need for such a regularizer. 
	- Mutual information based regularizers have been well studied in online RL literature; so it is interesting to see a use case for it in offline RL; the lower bound is computed using existing approaches in the literature however (lemma 3.1 and 3.2)
	- The main claim of the paper is that this framework can unify two popular offline RL algorithms, namely CQL and TD3+BC. Overall, it leads to the existing algorithms being modified based on equations 5 and 6 and the work shows that this regularizer can lead to a lower bound to the policy improvement step (theorem 4.1)
	- The theoretical justification of this work, along with the derived algorithm, is well explained. However, such techniques are not completely novel and adapts from existing approaches. For example, equation 12 for integrating the regularizer into an offline RL framework is somewhat expected where the additonal term is required based on the approximation of the MI term. 
	- Section 4.4 provides interesting insights for the justification of how the regularizer unifies existing algorithms. 
	- Experiments are primarily done on the D4RL benchmark, and it seems there are marginal improvements due to the addition of the regularizer. However, I am surprised that even in appendix the full result plots are not shown, and it is difficult to judge the actual empirical significance of this work. 
	- Overall, I think the paper is useful in terms of adapting a well known regularizer from online RL into offline RL. However, the resulting approximations for the MI term, and the way it is adapted into offline RL is not completely novel. Even if the algorithm is well derived from existing literature, the empirical significance of the work is not fully clear either. 

clarity,_quality,_novelty_and_reproducibility: - The novelty of this work is somewhat limited, given the approximations for the regularizer have been extensively studied in the past. The paper is however well written and easy to follow, with the theoretical derivation for the resulting updates properly derived. I do have concerns about the reproducibility of the work, since the results are only presented in the form of a table, without actual return plots, and it is not clear to me whether there is a bias variance trade-off due to the addition of this regularizer. Full results not being presented is a major concern for the empirical significance of this work. 
summary_of_the_review: The work proposes a novel regularizer based on mutual information. This form of regularizer has been popular in online RL literature, and past works studied approximations to such MI terms. Therefore, the contributions and novelty of this work seems somewhat limited to me. Additionally, the empirical significance of the work is not clear enough. I am willing to re-evaluate my score if the paper clarity can be improved, and experimental significance made clearer.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to:

1. Estimate the mutual information between state and action

2. Use the estimate as regularization, by adding it to the policy and value function objective.
strength_and_weaknesses: Strength 

The idea of estimating the mutual information between state and action pair and using the estimates to improve policy evaluation and improvement are intuitive.

Weaknesses

The writing is not very clear. I list a few examples below: 

- In the abstract, "Intuitively, mutual information … reflects how a behavior agent reacts to certain environment states during data collection". While this is factually true, I fail to see how this is different from simply estimating the behavior policy. 

- The abstract also writes "equivalent to maximizing the likeli- hood of a one-step improved policy on the offline dataset. In this way, we constrain the policy improvement direction to lie in the data manifold." I do not see how the former leads to the latter, or even what the latter means at all. AFAIK, the data manifold is not defined in the paper at all.

- The introduction writes "Though these methods are effective at alleviating the distributional shift problem of the learning policy, the im- proved policy is unconstrained and might still deviate from the data distribution." This sentence seems self-contradictory. If these methods are effective at alleviating the issue of distribution shift, then how can the learned policy still deviate from the data distribution.

In addition to writing, the paper also has two other weaknesses:

- The paper claims TD3+BC and CQL are special cases, but it is unclear why their method outperforms these methods. The paper claims that their method leads to a better mutual information estimation. But what does "better" mean, and why does that lead to better performance compared to CQL? This is not explained at all.

-  The method does not perform well when learning from uniform data. How about data where the behavior is uniform in some state, but much more constrained in others, which is the more realistic settings? The limitations of the methods seem to be an afterthought that does not receive adequate analysis.
clarity,_quality,_novelty_and_reproducibility: The method is novel as far as I know. However, the writing is not clear and too generic. The paper does not release code, so I also do not have high rating as far as reproducibility is concerned.
summary_of_the_review: The proposed idea is novel and intuitive. But the paper writing can be significantly improved, and why the method outperforms baselines deserve more analysis to be more convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: (a) Summary: This paper proposes an algorithm for offline RL which attempts to account for the distribution shift problem by constraining policy improvements to satisfy a lower bound on mutual information between states and actions, and empirically evaluates the proposed algorithm.

(b) Strengths: Most reviewers agreed that the paper was very clearly written, with each step well justified.  The authors responsibly highlights that their method performs poorly when the demonstration trajectories are generated using far-from-expert policies such as uniform randomization.

(c) Weaknesses: There were two marginally-positive reviews, but neither were willing to advocate for the paper's acceptance.  Many reviewers were unsure that the novelty of the approach was high enough for acceptance.  The method performs well against baselines, but did not convincingly out-perform other approaches in the literature.  One of the marginally-positive reviewers was especially concerned about the poor performance on uniformly-generated data; they pointed out that a method that only works on data from close-to-expert policies can only eke out marginal improvements almost by definition.
justification_for_why_not_higher_score: Novelty and empirical performance are really the two main things I'd want to see in a published paper, and the reviewers had serious concerns about both.
justification_for_why_not_lower_score: n/a
Decision: Reject