Before revision:  In this paper, we establish an \Omega(H^2 S/d_m\epsilon^2) lower bound (over model-based family) for the global uniform OPE , where d_m is the minimal state-action probability induced by the behavior policy. Next, our main result establishes an episode complexity of O(H^2/d_m\epsilon^2) for  \emph{local} uniform convergence that applies to all \emph{near-empirically optimal} policies for the MDPs with \emph{stationary} transition.
After revision:  In this paper, we establish an \Omega(H^2 S/d_m\epsilon^2) lower bound (over model-based family) for the global uniform OPE and our main result establishes an episode complexity of O(H^2/d_m\epsilon^2) for \emph{local} uniform convergence that applies to all \emph{near-empirically optimal} policies for the MDPs with \emph{stationary} transition.