Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Offline RL has been receiving enormous attention recently due to its practical application. However, as the offline data generation method could be noisy, identifying a robust solution by accounting uncertain model parameters has great impact. The paper proposes a robust fitted Q-learning (RFQI) algorithm that used a dual formulation to work with nominal training model. The dual problem is reformulated with function optimization and solved with risk minimization techniques, which is then used to approximate robust Bellman update. Theoretical upper bounds are derived for the RFQI algorithm. Experimental results on simple toy environments demonstrate that RFQI outperforms non-robust algorithms.
strengths_and_weaknesses: Robust offline RL is a practically important and challenging problem. RFQI reformulated the challenging estimation problem to a function optimization model and used it to update the robust Q-function. I appreciate the theoretical results and insights provided in Theorem 1. Experiments are performed on simple gym environments which are not exhaustive enough. Furthermore, only non-robust benchmark algorithms are used to validate the performance. Comparing against something like R-LSVI (Zhang et. al. 2022) would have been great.

References:
1.	Zhang, Xuezhou, Yiding Chen, Xiaojin Zhu, and Wen Sun. "Corruption-robust offline reinforcement learning." In International Conference on Artificial Intelligence and Statistics, pp. 5757-5773. PMLR, 2022. 

questions: 1. How is your work fundamentally different from R-LSVI?
2. Are you planning to open-source the codes for reproducibility purposes?

limitations: The paper missed citing some recent work on offline robust RL (e.g., Zhang et. al. 2022). Experimental results are not sufficient to empirically validate the claims. Validating the performance against state-of-the-art robust offline RL methods on a set of real-world simulated environments will strengthen the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper suggests a method to solve robust MDP using offline data. A fitted Q-learning is proposed and the authors developed its convergence and sample complexity. 
strengths_and_weaknesses: S: This work considered the RL problems under the offline setting, which is more challenging. A global optimality is also proposed. 

W: (1).The main challenge in robust RL is to find the worst case, however in this paper, it is solved by assuming a fail-state. This assumption greatly reduces the difficulties of the problem hence I'm afread it is too strong. Although it can be justified in practice, it is still too strong for me. 
questions: (1). See Weakness above. 
(2). Assumption 4 assumes the representative ablity of the function class, can the authors provide some example? Linear function approximation? Neural network? 
(3) Assumption 1 assumes an upper bound of $\mu$, and hence we have $\min_{s,a} \mu(s,a)>0$. As the number of samples increases, does this means we are getting samples of all $\mathcal{S}\times\mathcal{A}$? So does this situation in some sense recover the generated model setting (for any policy $\pi$, the offline data can be viewed as a generated sample)?
limitations: This paper is well-written, but some of the assumptions seem to be a little strong to me and hence limit the contribution of the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper deals with robust offline RL using function approximators. Robustness is a broad term, here it means a special robustness defined in the RMDP framework. An algorithm is outlined and numerous proofs of the properties of the algorithm are proved under strong assumptions. The algorithm is evaluated on two deterministic benchmarks (cart pole and hopper). 
Robustness to changes in system dynamics is investigated, and promising performance is achieved on the two selected benchmarks relative to the selected comparison algorithms.

strengths_and_weaknesses: **Strengths**
* The paper makes extensive theoretical contributions.
* The proposed practical algorithm could be promising.

**Weaknesses**
* The paper makes exaggerated claims.\
"In this work, we presented a novel robust RL algorithm called Robust Fitted Q-Iteration algorithm
with provably optimal performance for an RMDP with arbitrarily large state space, using only offline data with function approximation."\
Optimal? Provably optimal? The limitations and assumptions should be named. The non-optimality evident in Figure 3 should be explained.\
„We also demonstrated the superior performance of the proposed algorithm on standard benchmark problems.“\
Based on figures 1, 2 and 3, there is too little evidence, that the performance is superior. Furthermore, the statement can only be made for the examined alternative algorithms, furthermore there are only two benchmarks.

* The empirical evidence of the usefulness of the method is weak.\
Only two benchmarks, moreover both contain only deterministic dynamics. Due to the lack of investigations on benchmarks with stochastic dynamics, it seems questionable to me whether the experiments are at all suitable to test the claim of the proposed method.
* The paper seems incomplete without the Appendix.\
Algotithm 2. More experiments.

questions: The proofs make several assumptions, it would be interesting to see to what extent these assumptions are met in practice. Specifically, it would be interesting to see how often it happens in the experiments that the assumptions of the proofs are fulfilled.

In Figure 1 and 2 the algorithms FQI, DQN, SR-DQN and RFQI are examined, in Figure 3 it is FQI, TD3, SRDDPG and RQI. Why were the same algorithms not compared on all benchmarks? In addition, the same colors are used in the three figures to identify the algorithms, although they are not the same algorithms. This should be explained or changed.

In Figure 3, it is noticeable that for small values of "lag_joint_stiffness" of 0, 5 and 10 TD3 is clearly superior to the proposed RFQI. This should be mentioned in the text. Possibly it should be mentioned as a limitation. Also, it should be explained why this does not contradict claim „we presented a novel robust RL algorithm called Robust Fitted Q-Iteration algorithm with provably optimal performance for an RMDP with arbitrarily large state space, using only offline data with function approximation“. Furthermore, claim „We also demonstrated the superior performance of the proposed algorithm on standard benchmark problems“ seems to be an exaggeration.

„Parameter uncertainty commonly occurs in many real-world RL applications due to simulator modeling errors, changes in the real-world system dynamics over time, and ad\ersarial disturbances“\
I am skeptical that "Parameter uncertainty commonly occurs in many real-world RL applications due to ... adversarial disturbances". I don't think you can write it that way without giving a reference to the source of this assessment.


Lesser importance:\
„Robust RL is typically formulated as a max-min problem“\
I would rather say this is one variant, others use percentile performance optimization.

"Learning policies for RL problems with large state-action spaces is computationally intractable."\
This claim is not correct in this form (backgammon, go, chess, manifold applications with continuous state and action spaces).

In the bibliography there are a few wrong lower case letters, like: bayesian, mdp 




limitations: There are, in my opinion, a few limitations that should still be addressed in the text.

To what extent the conditions for the validity of the proofs are realized in practice should be discussed as a possible limitation.

Furthermore, the limitation that the benchmarks used have deterministic dynamics throughout should be mentioned.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper endeavors to design a robust algorithm in the offline RL setting with function approximation. The proposed algorithm is called Robust Fitted Q-Iteration, hopefully addressing challenges in this specific setting. The detailed derivation of the proposal is given under certain assumptions, followed by a near-optimal guarantee or sample complexity result. Experiments are conducted on three typical environments.


strengths_and_weaknesses: ### Strength.
1.Related works and preliminaries are thorough and comprehensive.

2.The paper is well organized and easy to follow.


### Weakness
1.In the design process of Robust Fitted Q-iteration, the sequentially added assumptions, e.g., Assumptions 3 and 4, lack demonstration empirically. In practice, we are not fully sure whether these assumptions can be satisfied or not, and how they impact on empirical performance.

2.Emprical demonstration is insufficient as authors only compare their algorithm with a limited number of baselines, e.g., SR-DQN, on only two or three Mujoco games. Researchers tend to be concerned about whether the empirical superior performance of the proposed algorithm can be observed as well on other games.

3.The proof sketch might be wordy and hard to understand.

questions: 1.In Eq.10, I am wondering why seeking the optimal dual function $g$ is equivalent to an optimal dual variable, and what the approximation gap between $\hat{g_{f^\prime}}$ and $g^*_{f^\prime}$?

2.The paper [1] also considers the robust RL setting, so what is the difference between it with this paper? Relevant discussion should be added in the related work.

3.Three challenges mentioned in Section 1 seems only to involve Robust RL. If additionally considering the offline setting, is there any new challenge for offline robust RL?

[1] Corruption-Robust Offline Reinforcement Learning (AISTAT 2022)

limitations: 1.The paper designs the robust algorithm only based on Total variation, which might be limited in terms of practice. I am not sure why the authors chose this distance rather than KL or others? Is TV the most typical choice in robust RL literature? An explanation is expected in the future.

2.The empirical improvement over other baselines seems not to be significant. More experiments are needed, including on more games. Also, a detailed comparison of the computational cost is also suggested.

In summary, I may not be familiar with some technical details, but as far as I can tell, they are basically sound. In particular, the dual reformulation as well as the following assumptions serves as the key to the proposed algorithm, which is the critical contribution of this paper. Therefore, I think it is a borderline paper as the algorithmic contribution is notable, while the practicability of the proposed algorithms might be questionable and needs to be further demonstrated.
review_title: null

Meta-review: metareview: The reviewers are in general positive about the paper. The main concerns were about the assumptions used in the analysis. The AC is satisfied by the response from authors, and also thinks the assumptions are reasonable (standard in offline RL literature). The AC also thinks the setting studied in this paper is important. 
confidence: Certain
Decision: Accept