Venue: ICLR 2023 Conference

Reviews:

final_score: rating: null
review_content: There already exists a paper in the offline RL context that introduces the acronym "DROP".
>Jian Shen et al.,, Model-based Offline Policy Optimization with Distribution Correcting Regularization, 2021\
>"... in this paper we present density ratio regularized offline policy learning (DROP) ..."

I would therefore suggest that the authors use a different acronym.
review_title: The Acronym DROP

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tries to summarize current offline RL algorithms into a bi-level optimization problem and offers a new algorithm in the non-iterative camp (that is sufficiently different from previous non-iterative work such as RvS). 
strength_and_weaknesses: This is a very novel paper that provides a fascinating new insight into Offline RL based on the COMs [1] framework.
The extension on top of COMs is simple, but the observations made in the paper (about how we learn $Q$ and how we use $Q$ to find policy) are nontrivial and fascinating. 

Technical Questions:
1. For deployment-time adaptation (Figure 2 Right side, Algorithm 2), what exactly is adapting? It doesn't seem like the authors are performing parameter updates in $\beta(a|s, z^*)$ (contextual policy) or score function $f(s, a, z)$. So what is the adaptation here? When I first read it, I thought the authors were doing online learning (aka online fine-tuning), but it doesn't seem to be the case.
2. At test time, performing gradient ascent to get $z^*$ seems computationally expensive. Can authors comment on how computationally efficient this algorithm is (compared to other algorithms like CQL)? 
3. How does the introduction of context $z$ help solve the OOD penalization problem? In another way, why searching $\arg\max_a Q(s, a)$ for policy is bad, while searching $z^* = \arg\max_z f(s, a, z)$ then decode as $\beta(a|s, z^*)$ is better? Because the process seems similar, the authors also trained $f$ to output a lower score when $z$ is OOD. I don't know why this is better than the pessimistic approach on the $(s, a)$ space.
4. For Table 1, would all the iterative Offline RL Algorithms (i.e., BCQ, CQL, COMBO, AWAC) satisfy all Q1, Q2, Q3? So DROP essentially is just an improvement over supervised learning for RL methods? Although, I should point out that DROP is doing TD-error learning in Equation (8), so I'm not sure why the authors think it's comparable to RvS, Decision Transformer, etc. 

Strengths:
1. The experiments are very comprehensive.
2. The authors demonstrated expert-level knowledge, summarizing and abstracting prior work into a new framework that could potentially provide insight into the problem domain.

[1]: Conservative Objective Models for Effective Offline Model-Based Optimization
clarity,_quality,_novelty_and_reproducibility: The quality of the paper is high. The method is novel. The reviewer cannot assess the reproducibility of the experiments.
summary_of_the_review: I'm currently giving this paper a 6 because I think it's well-written and offers interesting insights. I also have a series of questions regarding the technical and conceptual parts of the paper. Based on the answer, I will re-assess my score for this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents DROP, a non-iterative framework for policy optimization in offline RL, which separates the model training on offline data from deploying the model for testing/adaptation. The approach first splits the data into N subsets to learn distinct behavior policies, which are further used to learn a contextual behavior policy and task embeddings for a conservative model optimization to be used for policy inference. Experiments on D4RL benchmark data show that the proposed work performs better or on par compared to several competitors.
strength_and_weaknesses: Strengths:
+ The paper is well-written and visually appealing.
+ The idea of decoupling conservative behavioral cloning from policy inference is interesting and somewhat novel.
+ The experiments are extensive and reproducible, and the results are promising on the D4RL benchmark data. 



Weaknesses:
- The clarity of the paper in terms of contributions, approach, and argumentation should be improved
- The literature is not sufficiently reviewed and the position of the paper in the domain is not clear.
- The significance of the contributions on the model-based offline RL domain is not completely clear (see comments below)
clarity,_quality,_novelty_and_reproducibility: + The approach seems to be reproducible and some practical implications are presented.

- Although the paper is well-written, some parts are a bit hard to understand. The clarity of contributions, approach, and argumentation should be improved. In addition, providing some context/background (e.g., on the terms) from the beginning would be helpful.

- The connection of paper with the related work can be clarified further, particularly to the most closely related work. In addition, a line of research on "model-based" offline RL is missing from the paper (see references mentioned below). In this area, model is referred to the dynamics model of the environment which is first estimated and then used for policy learning. Whereas, the model in this paper is interpreted differently (i.e., parametric embeddings). This connection should be clarified/discussed.

- Apart from "model", the term "offline" seems to be loosely used. The approach delays the policy inference to the testing time when the model can acquire new samples (see Fig.2 and Alg. 2). While to my knowledge, in offline learning, we don't have access to online rollouts  whatsoever. Hence, it is not clear to me how the proposed ideas contribute to the field of MB offline RL (both in terms of methodology and empirical study).

- Accordingly, from where P(.|s,a) in Alg. 2 comes from?

- Could you elaborate how splitting the data into N subsets benefits the approach? An ablation study on the effect of N and also when N=1 would be helpful.


[1] Yu et al., MOPO: Model-based Offline Policy Optimization (2020)
[2] Kidambi et al., MOReL: Model-Based Offline Reinforcement Learning (2020)
[3] Yu et al., COMBO: Conservative Offline Model-Based Policy Optimization (2021)
[4] Rigter et al., RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning (2022)
summary_of_the_review: The paper is well-written and introduces a novel approach for offline RL that separates policy inference from the offline training of a conservative model. However, the position of the contributions in the domain of model-based offline RL is not clear and the paper should be improved in terms of clarity. I hence vote for a reject.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a non-iterative actor-critic approach for offline RL, which tackles the erroneous overestimation problem of the critic by combining the ideas of the non-iterative training procedure, test-time policy adaptation (both from Brandfonbrener et al., 2021), and conservative objective optimization (from Trabucco et al., 2021). 

The key insight of this paper is to fit the assumed multiple modes of offline data distribution by learning a mode-specific embedding as the condition of the actor and the critic. To this end, specific training methods include: 
1. Divide the offline training datasets into multiple sub-tasks based on certain decomposition rules.
2. In the offline training phase, the policy network and the embedding network are learned jointly with behavior cloning, and the score model is learned with TD-errors. 
3. In the online testing phase, slightly different from optimizing the entire policy network (Trabucco et al., 2021), the authors perform gradient ascent in the embedding space to find the optimal z* that can maximize the estimated objectives of the learned score model.

Compared with previous work, the technical contributions of this paper are:
1. The idea of task decomposition;
2. The efficient policy adaptation approach at test time by performing optimization in the compact embedding space.
strength_and_weaknesses: Strength:
1. It is a reasonable idea to split the offline datasets into N sub-tasks and condition both the score model and the policy model on a task embedding z. The proposed method has two advantages: 
- By explicitly considering the multi-modal distribution of the offline datasets and inferring the task embedding at test time, it can improves the domain adaptation ability of the policy. 
- By optimizing z (which is compact), it can ease the high-dimensional optimization difficulty and thus extend mode-based optimization (Trabucco et al., 2021) to test-time policy adaptation. 
2. Good ablation study on the embedding inference approaches and the decomposition rules (better to move into the main body of the paper).

Weaknesses:
1. The writing of the paper can be largely improved. 
- The structure of this paper is not very well organized. Frankly speaking, I spent a lot of time reading this paper so that I could understand Q1-Q3 and find out how they relate to each contribution of the proposed method. It might be better to describe the overall framework and the technical novelty before going to each detailed component in the method section. When I read the method section, I felt a little bit lost until I finished Sec 3.3 which introduced the last test-time adaptation module. I suggest describing the general pipeline of the model in the first paragraph of Sec 3 for a better reading experience.
- Fig 1 is less informative, which only shows the differences between iterative and non-iterative actor-critic offline RL methods, rather than illustrating the unique properties of the proposed method. Since the work of OneStep (Brandfonbrener et al., 2021) also follows the non-iterative framework, this figure does not shed a light on the novel contribution of DROP.
- The main body of the paper is not self-contained. I have to frequently flip between the main text and the appendix. In my view, at least the core algorithms and some important quantitative experimental results (such as the full comparisons with existing offline RL methods) should be placed in the main text. 
- Misleading notations. In Sec 2.1, "mu" stands for the initial state distribution, while in Eq (8), mu is the distribution of the latent sub-task embedding "z". This might raise confusion. Suggest using two different notations here.
- Use ``xxx'' for the quotation marks in LaTeX, instead of ''xxx''.
2. For the experiments,
- In Table 2, the proposed model is only compared with OneStep and COMs in the AntMaze environments, which is severely inadequate to support the advantages of the proposed method.
- The model was claimed to outperform previous value regularized methods including BEAR/BCQ/CQL on 19/33 tasks (Page 8), however, in Table 10, CQL achieves a better performance on 16/33 tasks than the proposed model. These results undoubtedly weaken the effectiveness of the proposed method, especially considering that it also borrows the regularization term from CQL.
- As a further ablation study, it would be good to show the experimental results without deployment adaptation, which can better verify the effectiveness of performing outer-level optimization in the embedding space rather than the action space from the original COMs.







clarity,_quality,_novelty_and_reproducibility: 1. Clarity and quality: Needs to be improved. Please see comments above.
2. Novelty: Good. Please see the Strength part above.
3. Reproducibility: Source code provided.
summary_of_the_review: In general, I think this paper proposes an interesting model to tackle the out-of-distribution learning problem in offline RL tasks, and contains rich ablation studies of different task decomposition rules and embedding inference methods. I was inclined to accept this paper given its novelty, however, 
- The paper lacks sufficient results to demonstrate the performance advantage over previous iterative value regularization offline RL methods, especially considering the model incorporates the CQL regularization when learning the score model. These experiments seem unconvincing to me. I strongly suggest further validation of the model by comparing its effectiveness with CQL in the revised paper. 
- The overall writing should be improved. Core algorithms and important results should not be placed in the appendix part. Please amend them accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper deals with offline RL. It takes the position that iterated policy evaluation can be given so little trust that behavior cloning is preferable.
It makes the assumption that it is useful to decompose the data set into episodes and (assuming that different policies were active in different episodes) to create individual behavior clonig policies for individual episodes or appropriate groups of episodes.
The algorithm is evaluated on the D4RL benchmark, where exactly this assumption is realized.
strength_and_weaknesses: **Strengths**
* The algorithm may be useful in special cases that may arise in practice.

**Weaknesses**
* The presentation needs to be improved significantly wrt structure and writing.
* The paper proposes an algorithm that exploits a property of the offline data in the D4RL benchmarks. It does not discuss whether this property can be expected in practice.  
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper needs improvements wrt structure and is very difficult to read. Terms are used before they are explained and there is no reference that they will be explained later. So you get the impression that it makes more sense to read the paper from back to front than the other way round. 

**Quality**

The quality of the presentation needs to be improved drastically.

**Novelty**

The main concepts are not really new, the main ideas already exist.

**Reproducibility**

Good

**Further comments**
* The first sentence „Offline reinforcement learning describes the task of learning policy from previously collected static data.“  is wrong in my opinion. It should be „Offline reinforcement learning describes the task of learning a policy from previously collected static data.“
or 
„Offline reinforcement learning describes the task of learning policies from previously collected static data.“
* Since the statement "Then, these methods directly deploy the learned policy in the online environment to test the performance." is wrong (the algorithms mentioned are offline procedures and do not test online), the authors mean something different than usual by "online environment".
* What is meant by "online environment" here?
* What is meant by "employed in testing"?
* „To enable this model-based optimization in offline RL, we are required to decompose an offline RL task into multiple sub-tasks, each of which thus corresponds to a behavior policy-return (parameters-return) pair.“ Why multiple sub-tasks? Doesn't this simply mean that supervised learning is used to learn a predictive model of the dynamic behavior and rewards of the environment from offline data (system identification (J. Sjöberg, H. Hjalmarsson, L. Ljung, Neural Networks in System Identification, 1994)) and then use this model to optimize a policy? See e.g. A.M. Schaefer, Reinforcement Learning with Recurrent Neural Networks, 2008.
* The term sub-tasks is very misleading, what is meant here are sub-sets of the data under the assumption that for each sub-set exactly one policy was active and that for different sub-sets different policies were active.
* What is meant by "unconfident embeddings"?
* The terms "low-dimensional embeddings for decomposed sub-tasks", "deployment adaptation" are used before they were explained.
* The statement "Such decomposition also comes with an additional benefit that it provides an avenue to exploit the hybrid modes in offline data D, because that D is often collected using hybrid data-generating behavior policies (Fu et al., 2020)," is, in my opinion, unacceptable. I read this as: because the benchmark datasets were created in this particular way, an algorithm is now proposed that exploits this property of the benchmark datasets. There is no practical benefit from this.



summary_of_the_review: The paper is very hard to read.
The paper proposes an algorithm that exploits a property of the offline data in the D4RL benchmarks. It does not discuss whether this property can be expected in practice.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a non-iterative bi-level learning method for offline RL, where the inner and outer levels correspond to training and testing, respectively.  The inner level learns a score model after partitioning the offline data, and a behavior embedding is learned with a conservative regularization.  Adaptive inference can be achieved across states in testing.  Experiments show DROP achieves similar or better performance than existing methods.

The paper addresses an important problem, and the experimental results appear encouraging.  The presentation can be improved in various places, especially for non-expert readers.  The technical novelty is somewhat limited.  Some assumptions are made on the dataset which holds for D4RL, but more investigations are needed for its generality.

justification_for_why_not_higher_score: The presentation can be improved in various places, especially for non-expert readers.  The technical novelty is somewhat limited.  Some assumptions are made on the dataset which holds for D4RL, but more investigations are needed for its generality.
Decision: Reject