It is possible to generalize this model so that it takes into account both the long run average and the amount of initial reward than can be gained. In the generalized, bias optimal model, a policy is preferred if it maximizes the long-run average and ties are broken by the initial extra reward. Figure 2 contrasts these models of optimality by providing an environment in which changing the model of optimality changes the optimal policy. In this example, circles represent the states of the environment and arrows are state transitions. There is only a single action choice from every state except the start state, which is in the upper left and marked with an incoming arrow.
All rewards are zero except where marked.
Since the choice of optimality model and parameters matters so much, it is important to choose it carefully in any application. A system with a hard deadline would be appropriately modeled this way. All unlabeled arrows produce a reward of zero.