Before revision:  We present a novel model-free upper value iteration procedure (}{\sf  (x) - V^{\pi}(x) from above and to construct confidence intervals for V^\star.
After revision:  We present a novel model-free upper value iteration procedure (}{\sf ) that allows us to estimate the suboptimality gap V^{\star(x) - V^{\pi}(x) from above and to construct confidence intervals for V^\star.