When given a state x, you learn the projected cost via value iteration. When you use q-learning and take action a while in state x, you get the promised discounted cost.