These results are discussed in a more general setting in the next two chapters. At the current time this is an open question in the sense that no one has been able to prove mathematically that one method converges faster than the other. In practice, however, TD methods have usually been found to converge faster than MC methods on stochastic tasks, as illustrated in the following example. Random Walk . All episodes start in the center state, C , and proceed either left or right by one state on each step, with equal probability.
Episodes terminate either on the extreme left or the extreme right.
Because this task is undiscounted and episodic, the true value of each state is just the probability of terminating on the right if starting from that state. Thus, the true value of the center state is . The true values of all the states, A through E , are , and . In all cases the approximate value function was initialized to the intermediate value , for all s . A small Markov process for generating random walks.