Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an offline RL algorithm based on the principles of in-sample maximization (when fitting the Q function) and maximum-entropy RL. It introduces the in-sample softmax, which is a straightforward modification of the usual softmax used in MaxEnt RL to only include actions in the support of the behavior policy, similarly to Batch-Constrained Q-Learning (BCQ). The convergence of in-sample soft policy iteration to an optimal policy (for an appropriate definition of optimal) is proved.

The main novelty is in showing that this in-sample softmax can be written as an expectation over samples from the dataset, reducing dependence on the behavior policy. Using this identity, the authors develop an algorithm similar to Soft Actor-Critic (SAC), which they dub In-sample Actor-Critic (InAC). Experiments demonstrate that InAC is effective in avoiding out-of-distribution (OOD) actions and attains high performance in both purely offline and offline-to-online settings.
strength_and_weaknesses: Strengths:
* The paper presents a clean motivating theory that is not too far removed from the practical implementation.
* Unlike some other offline RL algorithms, InAC does not enforce constraints between the learned policy and the estimated behavior policy. This is good because imperfections in the estimated behavior policy can bias learning.
* Experiments show that the algorithm can effectively avoid OOD actions.
* The performance of the algorithm is similar to or better than existing algorithms, both for offline and offline-to-online learning.

Weaknesses:
* Judging by Figure 3, it seems that some of the algorithms have not converged within the 0.8M iterations. Obviously it is good that InAC learns faster, but it’s possible that the other algorithms could reduce the gap if given more iterations. Since training time is typically not the constraining factor in offline RL, I think all algorithms should be run to convergence.
clarity,_quality,_novelty_and_reproducibility: The paper is generally clear and well-written.

The in-sample softmax, while a combination of existing ideas, is novel to my knowledge. More importantly, the authors show identity (5), which is the core algorithmic innovation and addresses a key issue in offline RL.

The authors include detailed descriptions of the experimental setup and hyperparameters for reproducibility purposes.

A small note: in equation (47), I think the final equality is the wrong way. (The logic would be correct and match the statement of the lemma if the sign were flipped.)
summary_of_the_review: I think the paper presents a useful idea and algorithm with compelling results, and therefore should be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In maximum entropy RL, we use a soft Bellman optimality equations where the standard max operation is replaced by the log-sum-exp operation. Just as researchers have proposed to use in-sample max instead of max over the entire action space, this paper propose to use in-sample softmax to improve the softmax operation, such that we only uses actions well-covered by the dataset. Some theoretical study are provided to show the convergence property similar to softmax. Experimental results demonstrate the effectiveness of the proposed in-sample softmax operation for a SAC-type algorithm on various offline RL benchmarks.
strength_and_weaknesses: Strength:
- The paper is well-written and easy to follow.
- Nice theoretical characterization.
- Experiments results seem promising.

Weaknesses/Questions:
- The idea is a bit straightforward (though not necessarily a weakness), as we have in-sample max to improve max, it seems not surprising to use in-sample softmax to replace softmax.
- The explanation/derivation in page 4 before section 4 may not be necessary as the math seems trivial. Also some argument could be more accurate (e.g. when $\pi_D(a|s) = 0$, equation (7) is undefined since $\log(\pi_D(a|s) )$ does not exist).
- The in-sample softmax optimal policy approaches the in-sample optimal policy as the temperature approaches zero. In practice, how do you choose the temperature. Have you considered annealing the temperature for better performance?
clarity,_quality,_novelty_and_reproducibility: See above for more detailed comments.
summary_of_the_review: The paper propose to use in-sample softmax instead of softmax for maximum entropy offline RL, which leads to in-sample SAC that shows promising results on some offline RL benchmarks. The idea is interesting, though being a straightforward extension of existing ideas.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an Actor-Critic RL algorithm able to learn from an offline dataset, while addressing the problem of out-of-distribution actions. The core contribution of the paper is the identification of how nicely soft policies and Q-updates (based on the softmax over actions) are amenable to updates robust to out-of-distribution actions. The resulting algorithm looks like the Soft Actor-Critic, but with the actor and critic losses slightly modified to introduce a term that depends on the estimated behavior policy used to collect samples in the dataset. Experimental results show that the resulting algorithm outperforms several baselines on standard benchmarks, and a theoretical analysis explains why the algorithm makes sense.
strength_and_weaknesses: Strengths:

- The paper is very interesting to read and the idea is exciting! Some parts of the paper, that may appear simple (such as Equation 5, the trick to reduce the important of $\pi_D$), are eye-opening.
- The proposed algorithm is a relatively simple modification of the standard soft actor-critic losses. Nothing extra to compute, no resampling of the dataset to do. The algorithm seems easy to implement, and at almost no computational cost compared to vanilla SAC.
- The paper is very clear and well-presented. Just the right amount of maths is given to understand how the new losses are designed, why they make sense, and how they can be implemented. Equations 14 to 16 are nicely grouped together to give a direct overview of the full (modified) SAC algorithm.
- The sanity check is very nice to have, more papers should have a small and simple sanity check like this.

Weaknesses:

- The empirical evaluation is not as precise and complete as in other papers. Even in the appendix, I did not find absolute scores obtained by the agents in the environments (to be easy able to verify that the baseline scores match what other papers report, and to easily compare the scores obtained by IAC with baselines not present in the paper). Normalized scores are not that useful and usually hide more information than they reveal. Evaluation on AntMaze could also have been interesting: some people motivate that AntMaze is more difficult than the Mujoco tasks due to its sparse reward function, that better stresses the ability of an RL agent to bootstrap (instead of just following $r_t$).
clarity,_quality,_novelty_and_reproducibility: - Clarity: high. The paper is easy to read and to follow, takes time to explain everything needed to understand the algorithm, and does not have any distracting part.
- Quality: high. The proposed method is sound, easy to implement, cheap to compute, and compatible with state-of-the-art RL algorithms of the family of Soft Actor-Critic.
- Originality: high. The proposed algorithm is, to my knowledge, novel. Its derivation is interesting and novel too.
summary_of_the_review: Very well-written paper with sufficient (but can be improved) empirical results. I would strongly advocate for the authors to include raw (un-normalized) scores in the final version of the paper, for ease of comparison against algorithms not present in this particular paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper studies offline RL. The main novelty is showing that in-sample softmax can be written as an expectation over samples from the dataset, reducing dependence on the behavior policy. It proves the convergence of the in-sample soft policy iteration. Experimental results show that the resulting algorithm outperforms several baselines on standard benchmarks.

Strength:
-  Clean motivating theory,  theory inspired practical implementation.
-  no constraints enforced between the learned policy and the estimated behavior policy. 
- Experiments show that the algorithm can effectively avoid OOD actions and have good performance.

Weakness:
- The empirical evaluation is not as precise and complete as in other papers.



justification_for_why_not_higher_score: The experiments are not as precise and complete as in previous papers.
justification_for_why_not_lower_score: The proposed algorithm is simple and effective and have good empirical performance.
Decision: Accept: notable-top-25%