Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper the authors provide uniform convergence results for offline policy evaluation, a minimax lower bound for global uniform OPE and an upper bound for local uniform OPE, with a new design analysis tool called singleton absorbing MDP. Besides, the authors generalize the results to new settings, including offline task-agnostic and offline reward-free rl. 
main_review: Originality: The results presented in the paper is new and tighter than the prior works (Yin et.al 2021), and the proving idea of using singleton-absorbing MDP technique is novel.  Also, the new settings proposed in this paper (offline task-agnostic and reward-free rl ) are interesting. 

Quality & Clarity: Overall the paper is well-written and results are well-presented. The visualizations of singleton absorbing mdp in the appendix are really nice and provide intuitions for understanding. 

Significance: To me this is a nice theoretical paper and the paper makes contributions to the theoretical offline rl community. 
limitations_and_societal_impact: I have two questions regarding the offline new settings presented in this paper:

1. For the offline task-agnostic learning settings, if we using a simple union bound over the K tasks, can we get similar results?

2. For the Offline Reward-free Learning, why this is an interesting settings and what we can benefit if we consider this setting?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies uniform off-policy evaluation in the episodic setting. In particular, the authors consider a stationary episodic setting. The goal is to evaluate the value function corresponding to a given policy, up to some given accuracy, uniformly among all the policies. The idea is that a uniform off-policy evaluation can be used for off-policy learning directly, due to the equation (1).
The authors first characterize a minimax lower bound for global uniform OPE, and they show that this lower bound is proportional to the cardinality of the state space. Next they characterize a local uniform upper bound for plug-in estimator for OPE. 
Next, the authors consider offline task-agnostic and offline reward-free learning. They provide the sample complexity of model based methods for these tasks, and they show that these bounds are optimal.
main_review: - My main concern about this paper is on the relation between OPE and off-policy learning. The authors claim that global OPE can be directly translated to off-policy learning by an exhaustive search over space of all the policies. However, Search over space of all the policies is exponentially expensive in the cardinality of state and action space. Am I missing something here?
- Typo: Line 284: supports our design
limitations_and_societal_impact: Lack of experimental results.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers uniform convergence in model-based off-policy evaluation and learning in time-homogeneous setting, and provide sharp error bound in different scenarios.
main_review: The authors considers uniform convergence in model-based offline reinforcement learning, and generalize the results to two slightly different setting, termed as task-agnostic and reward-free reinforcement learning. Central to the analysis is an absorbing MDP technique to decouple the statistical dependency on the empirical and the terms obtained from the empirical model, which is a generalization from the infinite horizon case and eventually lead to a sharp result on the leading term. The results are clearly presented. One minor issue is that, the higher order term are not very satisfactory, especially the results in Theorem 5.6, that need samples proportion to the number of states in linear MDP with anchor points, which should be avoided in the setting with function approximation (as we want to deal with the large state space with function approximation).
limitations_and_societal_impact: The authors have discussed them properly.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This work studies uniform convergence for offline policy evaluation (OPE) with model-based methods for episodic MDP. Particularly, uniform convergence for all near-empirically optimal policies is established.
main_review: The motivation for considering uniform OPE is insufficient, since it is not suitable to studying offline learning. The bound established based on (local) uniform OPE is too loose compared with existing results for offline learning (both model based approach and model free approach, also known as asynchronous Q-learning). Hence, I think this work is of limited importance. In addition, the proof is based on Agarwal et al. 2020. It is not evident to me what new challenges are in the proof. 

A. Agarwal, S. Kakade, and L. F. Yang. Model-based reinforcement learning with a generative model is minimax optimal. Conference on Learning Theory, 2020.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: The paper studies the uniform UPE problem in the offline RL setting with both upper and lower bound. Most of the reviewers believe the paper is well-written. There is one major concern about the epsilon range. The current results give the bound for eps<1/sqrt{S}. The paper would be much stronger if results about eps>1/sqrt{S} is also provided. 
Decision: Accept (Poster)