Venue: NeurIPS 2022 Workshop Offline_RL

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper is well written and introduces a novel framework of incorporating a mutual information regularizer in the policy evaluation and policy improvement step. They show how this state-action mutual information regularizer framework is a generalization of other behavior-regularized approaches like TD3+BC and conservative-value approaches like CQL.

Experimentally they show that their method generally outperforms the relevant baselines on several of the D4RL tasks. Additionally, they include many insightful ablations on how different variants of their MISA algorithm performs.

Pros:
* Interesting idea that generalizes many previous SOTA offline RL approaches
* Well supported final practical algorithm
* Improves results on many D4RL tasks
* Good ablations

Cons:
N/A

review_title: Interesting idea, good theory, and great results

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a novel MISA framework to approach offline RL from the perspective of Mutual Information between States and Actions in the dataset by directly constraining the policy improvement direction. Intuitively, mutual informa- tion measures the mutual dependence of actions and states, which reflects how a behavior agent reacts to certain environment states during data collection. To effectively utilize this information to facilitate policy learning, MISA constructs lower bounds of mutual information parameterized by the policy and Q-values.

This paper shows that optimizing this lower bound is equivalent to maximizing the likelihood of a one-step improved policy on the offline dataset. In this way, we constrain the policy improvement direction to lie in the data manifold. 

I believe the result is good overall, however, why the standard deviation is not report in the table? Those uncertainty measure can further measure how good this MISA method is.
review_title: Official Review

Decision: Accept