Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper investigates the problem of computing the robust Bellman operator efficiently when the uncertainty set is defined using $\phi$-divergence. The basic idea is to calculate the robust Bellman update via simplex projections, which can be solved efficiently. Multiple examples of $\phi$-divergence defined uncertainty sets are given, including Kullback-Leibler Divergence, Burg Entropy, Variation Distance, $\chi^2$-divergence. Numerical experiments are provided to demonstrate the efficiency of the proposed method.
strengths_and_weaknesses: Strengths:
This paper investigates a more general notation of uncertainty set, s-rectangular, which is less conservative than the (s,a)-rectangular uncertainty set model. A large class of distance metrics, i.e., $\phi$-divergence defined uncertainty sets are studied. The proposed method is claimed to be more computationally efficient. Theoretical results and numerical experiements validates these claims. Overall, the contribution is useful in practice.


Weaknesses:
The major weakness of this paper lies in the presentation. In section 4, the author claim that the robust Bellman operator can be solved efficiently if the projection problem in (4) can be solved efficently. However, it is not discussed at all how the projection problem in (4) is related to the robust Bellman operator. It is thus not clear to this reviewer that once the projection in (4) is solved, how it is going to be used in the evaluation of the robust Bellman operator. As in section 5, the discussion is focused on solving the projection in (4) for various types of $\phi$-divergences. The overall presentation of the paper needs to be carefully revised.

Second, the setting focused in this paper is the s-rectangular uncertainty set. One question is that if the problem is reduced to the (s,a)-rectangular uncertainty set, is the challenging in evaluating the robust Bellman operator still exist? As for the (s,a)-rectangular uncertainty set, the problem is reduced to a distributionally robust optimization problem for each (s,a) pair. Therefore, the motivation of considering the s-rectangular uncertainty set shall be carefully discussed in order to motivate the research in this paper. 

Third, this paper mainly focus on the model-based planning setting, where the uncertainty set is assumed to be known. Therefore, the approach in this paper might be difficult to be generalized to the practical reinforcement learning setting with large or even continuous state and action spaces. 

Fourth, there is another line of research focus on model-free robust reinforcement learning, where the uncertainty set is not known, and samples from the center of the uncertainty set can be obtained. Different uncertainty set models are considered, e.g., ellipsoid type uncertainty set models and $\epsilon$-contamination uncertainty set model. Those algorithms are computationally efficient (the complexity at each iteration is independent of S and A), and can be easily extended to the case with function approximation for large state-action sapces. 
Reinforcement learning under model mismatch. NIPS 2017.
Online Robust Reinforcement Learning with Model Uncertainty. NeurIPS 2021.
Robust reinforcement learning using least squares policy iteration with provable performance guarantees. ICML 2021.
Policy Gradient Method For Robust Reinforcement Learning, ICML 2022.



questions: Please see Strengths And Weaknesses.
limitations: NA.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed equivalence between the robust Bellman operator and a projected optimization problem. Also, several solutions to the projected optimization problems are developed for different distance measures. 
strengths_and_weaknesses: Strengths: (1). The $\phi$-divergence is general hence several distance measures are included; (2). The proposed method works for many different uncertainty sets. (3). The complexity matches the one for non-robust problems. 

Weakness: (1). Although this is a model-based method, many model-free methods aren't discussed, e.g., [1,2,3]. More related works should be discussed. 
(2). Theoretically, the authors didn't explain clearly why the method in this paper is better than the value iteration or dynamic programming method for robust RL problems.



[1 ]Wang, Yue, and Shaofeng Zou. "Online robust reinforcement learning with model uncertainty." Advances in Neural Information Processing Systems 34 (2021): 7193-7206.
[2] Panaganti, Kishan, and Dileep Kalathil. "Sample Complexity of Robust Reinforcement Learning with a Generative Model." International Conference on Artificial Intelligence and Statistics. PMLR, 2022.
[3] Badrinath, Kishan Panaganti, and Dileep Kalathil. "Robust reinforcement learning using least-squares policy iteration with provable performance guarantees." International Conference on Machine Learning. PMLR, 2021.
questions: See the above part.
limitations: This paper mainly focuses on the model-based setting, which limits the contributions. Also, some external methods like binary search or other optimization methods are introduced and hence will complexify the method.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper studies robust MDPs with s-rectangular ambiguity sets defined by $\phi$-divergence. The authors then propose a number of fast algorithms with complexity analyses for different popular $\phi$-divergences. Numerical results were then provided to support the methodologies.  The paper is well-written. The problem studied is highly relevant and of importance. 
strengths_and_weaknesses: ### Strengths:

Robust MDP is an important topic in ML/AI and the question of how to quickly solve a robust MDP problem is highly relevant. It is known in the literature that to maintain tractability, one needs to stick with (s)- or (sa)-rectangular ambiguity sets. The paper focuses on several types of (sa)-rectangular sets and provides rigorous investigations, which is good and would be a good contribution to the research topic. 

### Weaknesses:

My main concern goes to the novelty and the significance of the work. Robust MDP is already widely studied and well understood. The paper does not provide new robust MDP models, but only focuses on how to quickly solve the robust MDO problem under some well-known uncertainty settings (the use $\phi$-divergence ambiguity sets in robust MDP is not new). Note that $\phi$-divergence (sa)-rectangular ambiguity sets were already studied in the literature, thus the work here is just an extension to (s)-rectangularity settings.

**Other comments**

The paper only discusses value iteration; how's about policy evaluation and policy improvement?

The experiments are quite limited since the authors only compare their algorithms with MOSEK solvers. I can see that the Bellman equation has a standard max-min form with a linear objective function, for which several efficient algorithms exist. 

Discussions on why $\phi$-divergence (s)-rectangular ambiguity sets are useful and how to construct them seem missing. 

 
questions: - Why are $\phi$-divergence (s)-rectangular ambiguity sets useful? 
- Can we handle ambiguity sets defined by several $\phi$-divergences?
- Can the algorithms developed apply to policy improvement?
limitations: I don't see any limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This is a somewhat borderline paper. The reviewers were unanimously positive, but they all had concerns. In reading through the concerns and responses, it seems that many (though perhaps not all) of the concerns could be addressed with additional references and some expository modifications. 

If the paper makes the final cut, I encourage the authors to follow through with their proposed modifications and do their best to address the concerns expressed by the reviewers.
confidence: Less certain
Decision: Accept