Title: Variational characterizations in Markov decision processes