Before revision: In this work we present an approach for building tight model-free confidence intervals for the optimal value function V ^\star in general infinite horizon MDPs via the upper solutions.
After revision: Policy evaluation is an important instrument for the comparison of different algorithms in Reinforcement Learning (RL). Yet even a precise knowledge of the value function V ^\star in general infinite horizon MDPs via the upper solutions.