Before revision:  We present a novel model-free upper value iteration procedure (}{\sf  (x) - V^{\pi}(x) from above and to construct confidence intervals for V^\star.
After revision:  We present a novel model-free upper value iteration procedure (}{\sf UVIP (x) - V^{\pi}(x) from above and to construct confidence intervals for V^\star.