Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims at achieving robustness in uncertain environments. The proposed approach, a mean-standard deviation formulation, seemingly reduces the desired level of robustness to one parameter tuning. Experiments are conducted on discrete and continuous control domains. 
strength_and_weaknesses: The idea of relating distributional RL for the sake of robustness is interesting and novel, although the connection is quite natural. However, I think there are two major flaws in this paper, in terms of 1/ novelty; 3/ quality. I detail my concerns below.


clarity,_quality,_novelty_and_reproducibility: 1/ Novelty

It is not clear to me what novelty this paper brings, may it be from a distributional, a regularization, or a robust RL perspective. What are the theoretical results? Robust MDPs with f-divergence uncertainty sets have been studied in [1]. It seems that this paper is a particular case of that work: [1] considers s-rectangular uncertainty sets as opposed to the (s,a)-rectangular sets of this one. Indeed, Bellman recursions and critic estimates are all considered in terms of Q, not v. By design, this implicitly means that uncertainty sets are independently defined for each state-action pair. 

From a robust/regularized RL perspective, learning the variance besides the expected reward has been done in [2] already, while [3] has trained the exact same mean-std objective as here. I would have expected [3] to appear as a baseline in the experiments. This yields my second question to the authors: what are the experimental contributions? 



2/ Quality/Clarity

In my opinion (please correct me if I am wrong), there are some misunderstandings regarding some cited works. I detail my claim below:

- In the 2d paragraph (Sec 1), it says « in general, this problem is NP-hard due to the complex max-min problem, making it challenging to solve in a discrete state action space and to scale to a continuous action space »: Actually, the complexity of computing a robust optimal policy is unrelated to the size of the state-action space, but rather, it is related to the structure of the uncertainty set. See [4] - for the problem to be tractable we basically need some rectangularity assumption, as mentioned above. 

- The 3rd § of Sec 1 is unclear to me. The authors claim that they focus on more general, continuous state space, but they do not provide any theoretical result. Additionally, although not formally studied in most RL works, most standard properties such as contractive operator, greediness, etc., still hold for continuous action and state spaces, as long as the action space is compact and the state space Polish + measurable (see [5])

- p. 2 - §2: «  The problem of uncertainty under the distribution of the environment is transformed into a problem with uncertainty over the distribution of the rewards, which makes it tractable»: In fact, this is arguable. As shown in [6], it depends on the type of regularization being used. For example, policy regularization alone may not account for uncertainty in dynamics, whereas value regularization (as in this paper) does. This point should be made very clear, as I think this represents the main interest in doing distributional RL for robustness purposes. 

- In general, related works are being cited and described without being compared to this study (for example in §4 of Sec. 1)


summary_of_the_review: Overall, I think this paper holds an interesting intuition by relating distributional RL to robustness, but it should be further investigated and analyzed: formally, why is the std penalty adding robustness to varying P, theoretically? How different is it from the policy regularization perspective? Why not just do TD learning on the variance as in [2]? 

References

[1] Ho, Chin Pang, Marek Petrik, and Wolfram Wiesemann. "Robust Phi-Divergence MDPs." arXiv preprint arXiv:2205.14202 (2022).

[2] Tamar, Aviv, Dotan Di Castro, and Shie Mannor. "Learning the variance of the reward-to-go." The Journal of Machine Learning Research 17.1 (2016): 361-396.

[3] Pan, Xinlei, et al. "Risk averse robust adversarial reinforcement learning." 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.

[4] Wiesemann, Wolfram, Daniel Kuhn, and Berç Rustem. "Robust Markov decision processes." Mathematics of Operations Research 38.1 (2013): 153-183.

[5] Puterman, Martin L. "Markov decision processes." Handbooks in operations research and management science 2 (1990): 331-434.

[6] Derman, Esther, Matthieu Geist, and Shie Mannor. "Twice regularized MDPs and the equivalence between robustness and regularization." Advances in Neural Information Processing Systems 34 (2021): 22274-22287.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies risk-averse reinforcement learning problems where the robust objective can be formulated as a penalization term on the standard deviation of the stochastic total return. The authors propose two algorithms for the setting with discrete action space and the setting with continuous action space respectively. Experiments are conducted to compare the proposed method with SAC on Mujoco environments.

strength_and_weaknesses: This paper provides a clear derivation for the reformulation of the robust RL objective to a variance-regularized objective, which is very helpful for readers to understand the motivation. The experimental results seem to be promising and show the effect of regularization on robust performance. However, the experimental comparisons are not complete and thus may not justify the proposed algorithm and its advantage in robust RL compared with other baselines.  

clarity,_quality,_novelty_and_reproducibility: There are multiple typos in the current manuscript. The writing needs more polishing. The algorithm described in Section 3.3 is a little bit confusing and hard to read.  

summary_of_the_review: As mentioned in the previous sections, I appreciate the mathematical derivation of the reformulation of the robust RL objective. The main shortcoming that I think the authors could improve is its relationship with related methods, which I elaborate in the next paragraph. 

There is no comparison with other robust RL methods in the literature. The authors might need to complete this comparison both from an algorithmic design perspective and the experiment perspective. In particular, since there are multiple ideas that motivate the development of the algorithms in this paper such as that of Kumar et al. (2022) and Moskovitz et al. (2021), the authors should at least have some discussion about the performance comparison with them. 

“We show that is it possible to improve the Robustness of RL algorithms with variance/standard deviation regularisation”: it is …

Can you explain why the first equality in (1) holds?

In the sentence “...to obtain better gradients when u is small, the (asymmetric) Huber quantile loss”, what do you mean by better gradients?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a risk-sensitive criterion based on the expected value and standard deviation of a distributional RL trianing scheme. The authors evaluate their method w.r.t. robustness over repetitions and find more robust behavior
strength_and_weaknesses: Strengths:
- the paper is in a relevant field of research
- the mathematical derivations appear sound
- relevant work is metioned

Weaknesses:

- Novelty: 
   - using a mean/std approach for risk-sensitive RL has been around for some time and is standard procedure with many publications
   - the results are very much to be expected, adding a standard deviation makes training more stable at the cost of performance

- Flexibility:   Compared to  Smirnova et al. (2019) the authors state that "since they use the variance estimate under a Gaussian assumption of distributions while we use a standard deviation penalization without any distribution assumptions" (Page 1). 
    - However, parameterizing a distribution using its first and second moment corresponds to moment matching (which is minimizing the KL divergence between the target distribution and a Gaussian variational distribution.
    - E.g. if the distribution Z would be multi-modal, the standard deviation  and thus \mu - \sigma estimate would not be meaningful

- Applicability: the fact that the parameter must be carefully chosen, not only its specific value but even its order of magnitude (compare 
( Hopper-v3 and Walker2d-v3 w) makes it hard to apply in practice. Ideally, the author would have proposed an estimation procedure
to migitate this effect

-  Empirical evaluation: 
    - minor things like y-axis labeling on all figures, and unclear what "Mean over 20 trajectories" would mean and no mention of experiment repetitions
    - The only quality metric studied is performance(?) and spread of performance over trajectories. There would be more metrics to estimate robustness, such as worst-case (or some 10,20 percentile) performance. Variation over experiment repetitions etc.
  

clarity,_quality,_novelty_and_reproducibility: The paper is quite clear, although I was not able to check the mathematical derivations

I have doubts about the novelty of the approach, specifically using a mean/std approach as objective function is well established in safe, risk-sensitve, and curiosity based RL and other fields of research (e.g. active learning).

summary_of_the_review: Overall, I cannot recommend acceptance, because I have concerns regarding the novelty and applicability of the proposed method.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work targets the robust reinforcement learning problem with an uncertainty set with respect to $\chi^2$. It proposes an alternative 'Q-value' $Z$ and arrives at the mean standard deviation RL. Then the algorithms in the discrete case and continuous case are proposed separately and evaluated in different experiments with comparisons to baselines.
strength_and_weaknesses: -Strength:
1. The intuition and derivation of the designing algorithm in robust RL are well-explained by showing the relationship between robust RL and mean standard deviation optimization.
2. The presentation and writing of this work are very clear and easy to follow.

-Weaknesses:
1. It seems the main contribution is that this work proposes a different 'Q-value' function by using mean standard deviation optimization and minus $\alpha$ multiplying the standard deviation. However,  the experiments do not show that such designing of 'Q-value' has benefits if I didn't miss something. Such as in Figure 1 and Figure 2, the algorithm with $\alpha=0$ (vanilla Q-value without extra design) and distributed critics works even better or the same as when $\alpha>0$ (proposed methods in this work). It seems there does not exist a  great $\alpha$ for all environments.
clarity,_quality,_novelty_and_reproducibility: -For clarity, some minors:
1. "ovoid" in the last several lines on Page 5
2. The notations of $N or $C$, $n$ and $c$ keep rotating is very confusing in the last several lines on Page 5 and Page 6.
3. The index $i$ in Equation 4 is not mentioned before?
4. Is there a missing of $Z$ in Equation 6 and the equation above Equation 6?
5. There are also some confusing notations in Algorithm 2, such as what is $i$?

-The novelty and originality of this work seem limited. The tailored design of the 'Q-value' does not have convincing results in experimental results.
summary_of_the_review: In summary, although this work is motivated by a clear intuition between the robust RL and the mean standard derivation optimization, the proposed technical 'Q-value' does not have convincing results in evaluated experiments. So I suggest for rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a reinforcement learning (RL) algorithm that is robust against uncertainties in the environment.  The proposed approach is well theoretically grounded with clean derivation, which constitute the main strength of the paper.

Major concerns are in the limited novelty and practical advantages over existing robust or risk-sensitive RL methods.  The proposed approach is essentially the mean -  standard deviation formulation, which has been studied extensively in the literature.  The experiments only compare the proposed approach against SAC and PPO, and the empirical advantages of the proposed approach to existing robust or risk-sensitive RL methods are unclear.
justification_for_why_not_higher_score: The proposed approach has limited novelty, and its empirical advantages over existing methods are unclear.
Decision: Reject