Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces LODAC, a variant offline actor critic method, which first embeds the state space into a latent space and also include risk measure into consideration. On the theory side, the authors show that minimizing risk measure in the latent space and in the natural space. On the empirical side, the authors consider an artificial environemnt (walker) which forces the deterministic reward function to be stochastic and justifies the efficacy of the proposed method. 
strength_and_weaknesses: Strength: 
 - The equivalence between minimizing risk measure in the latent space and in the natural space seems interesting. 

Weaknesses:
 - Third paragraph in the intro: I don’t think I buy the logic here. “Classical” RL methods can also be evaluated under “stochastic” environment since their goal is to maximize **expected** total reward. The importance of developing risk-sensitive methods is to consider risk while maximizing total reward, which should not be counted toward training policies under stochastic environment. 
 - The paper seems to be a naive combination of risk-sensitive offline RL and latent offline RL. I did not find any novelty beyond these two points. I suggest the authors further classify this. 
 - Experiment seems too simple. I believe walker environment itself is already a low-dim env (18, see (Tassa et al., 2018)). It seems there is no need to further embed into low dimension. I suggest the authors consider a more complex environment. 
 - The authors should also compare the results with other SOTA offline RL methods: COMBO, CQL, etc. There should be a trade-off between reward and risk. 
clarity,_quality,_novelty_and_reproducibility: Clarity: Yes. 

Quality: Yes. 

Novelty: little. 

Reproducibility: I believe it should be fine. 
summary_of_the_review: The paper has an overall good presentation and some motivation. However, given the weaknesses stated above, I recommend reject.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an approach for Offline RL by explicitly modeling risk through CVaR. The paper targets stochastic environments and makes the case that modeling risk is essential for  performing well in these settings by demonstrating improved performance in standard control tasks with noise injection in the rewards. 
strength_and_weaknesses: Strengths

1. The proposed approach for risk-sensitive RL in a latent space is simple to implement, by combining with several prior offline RL algorithms.
2. The motivation for necessity of modeling risk in stochastic environments is on point, and should be helpful in developing algorithms for these environments broadly. 

Weaknesses

1. The practical implementation section is very vague and it is unclear how exactly the components of LODAC differ from prior approaches like Rafailov et al. (2021) More details regarding the algorithm is essential for better understanding the contributions

2. The experimental settings do not conform to the motivations for risk-sensitive RL. The only stochasticity induced is a form of Bernoulli noise added to the rewards, and the dynamics are still deterministic. I think it is important to perform analyses on environments with stochastic dynamics to really understand the benefits of risk-sensitive algorithms.  

3. The experimental comparisons are insufficient and are significantly minimal compared to evaluations that offline RL algorithms usually do. For example, only a single walker agent is considered for evaluation. I would suggest at least evaluating on the entire D4RL suite and comapring with other baselines like CQL, IQL, COMBO etc. 
clarity,_quality,_novelty_and_reproducibility: The motivations of the paper re interesting and important. However insufficient details are provided about the practical algorithm and experiments. 
summary_of_the_review: The paper has interesting insights and motivations, but the experimental comparisons are incomplete and missing external baselines, evaluation environments and experiments settings. Please refer to the Weaknesses section above. 

--- AFTER REBUTTAL ---

After reading the authors' responses, I am keeping my score and not recommending accept for the paper because only my first concern is reasonably addressed, and I am not convinced by the justification for significantly minimal experimental evaluations.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to train a distributional actor-critic algorithm in a latent space for high-dimensional data in offline reinforcement learning. Risk measure instead of the classic expected return is optimized under the risk-sensitive RL framework. 
strength_and_weaknesses: Strengths: 
1. This paper combines the methods from prior methods. 

Weaknesses:
1. This paper lacks novelty. The latent model for encoding high-dimensional states is borrowed from Rafailov et al. (2021), and the distributional Q-learning is originated from Ma et al. (2021).
2. The writing is coarse, and many typos appear abruptly.  
3. Theoretical analysis doesn’t support the proposed algorithm due to unsatisfactory assumptions. 
4. Experiments are only conducted on customized walker datasets. 

Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pp. 1154–1168. PMLR, 2021.

Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.

clarity,_quality,_novelty_and_reproducibility: The writing is coarse and the originality is limited. Code is provided for reproducibility. 
summary_of_the_review: Overall, I am concerned that the originality and depth of this contribution may not quite yet reach the high bar set for ICLR. The writing errors make reading stumble. 

Writing errors: 
- Section 1, paragraph 2, “it may be be more …”
- Section 3, paragraph - Latent variable model, “Using this factorization, the evidence lower bound (ELBO) (Odaibo, 2019) and a really interesting theoretical approach (Levine, 2018), the following objective function for the latent variable model is derived … ” by any reference? 
- Section 3, paragraph - Latent variable model, “ … and D a decoder”
- Section 3, paragraph – Distributional RL, “For example, O-RAAC (Urp´ı et al., 2021) and following a previous work (Fujimoto et al., 2019), decomposes the actor into two different component an imitation actor and a perturbation model.”
- Section 3, paragraph – Distributional RL, “… , is introduced to prevent for too optimistic estimation”
- Section 4, starting paragraph, “Since training with high dimensional space directly failed and following previous work (Rafailov et al., 2021; Lee et al., 2019), we encode our …”
- ……

Theoretical concern: the first two assumptions require the encoder $\phi$ to be a one-to-one mapping function, while this is hard to be satisfied when using non-linear neural networks state encoders, for example, $\phi$ occasionally maps both two states $s$ and $s’$ to zero vectors.  This conflicts with the use of trained variational encoder in the algorithm part, so the theoretical analysis doesn’t support the proposed algorithm. 

Experiments are not standard and only walker_walk task is considered. The dataset is collected by running SAC based on classic states provided by DeepMind Control suite. If the dataset is collected by policy which is trained not based on pixel input, why you trained your model on this dataset via pixel input. Limited number of baselines are considered here, and the paper didn’t include implementation details of these baseline methods. 

Is LODAC-O a O-RAAC trained with your latent encoder?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the risk measure instead of the classically expected return in the offline RL setting. This paper shows that under some assumptions it is equivalent to minimize a risk measure in the latent space and in the natural space. A practical approach is proposed to use CVaR as the objective function.
strength_and_weaknesses: Strengths: easy to follow, good performance
Weaknesses: Incremental idea, insufficient experiments
clarity,_quality,_novelty_and_reproducibility: Clarity: good
Quality: good
Novelty: poor. the idea is incremental
Reproducibility: good
summary_of_the_review: The paper is easy to follow and combines distributional RL with offline RL. My concerns are:
1. the idea is incremental and the performance is not surprising. The method simply combines CVaR and an RL method. There is no specific design for the offline setting. Since distributional RL methods like C51 can achieve better performance than classical RL methods, the performance shown in this paper is not surprising.
2. For experiments, only one environment is used, which is not very convincing. More results should be provided.

typos: the equation above Eq. 3: s_i+1a_i -> s_i+1, a_i; a MDP -> an MDP; equation before Lemma 4.1.1: \phi(s_0) -> \phi(s_H)
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: ## Summary
This paper proposes a method that embeds high dimensional state space into a low dimensional latent space with a method they call as LODAC. which is a type of offline actor critic algorithm. The paper also suggests to minimize a risk measure instead of expected total rewards. The proposed method works with stochastic reward functions as well. The paper shows some empirical results on `walker_walk` task from DeepMind Control suite using pixel observations.

In a nutshell, the authors agreed that this paper is not ready for publication yet. Below I will list some of the strengths and weaknesses pointed out by the reviewers.

## Strengths

- Interesting relationship between minimizing risk measure in the latent space and in the natural space.
- The proposed approach is simple and easy to implement.

## Weaknesses

- The reviewers agreed that the proposed method is not novel and the results are not surprising.
- Limited experiments only on walker task. It is not clear how general this approach is since there are no experiments on other environments. 
- Lack of comparisons against other SOTA offline RL tasks.
- Unclear and vague writing. For instance, the intro of the paper needs some work. In general, th writing is coarse, and there are several typos in the paper.
- Lack of experiments on environments with stochastic dynamics.
- Theoretical analysis doesn’t support the proposed algorithm due to unsatisfactory assumptions.
justification_for_why_not_higher_score: The reviewers unanimously agreed to reject this paper. 
justification_for_why_not_lower_score: There are no lower score.
Decision: Reject