Intuitively it seems like a good choice, at least better than the uniform distribution. For example, if you are learning to play chess, you study positions that might arise in real games, not random positions of chess pieces. The latter may be valid states, but to be able to accurately value them is a very different skill from evaluating positions in real games. We also know from the previous chapter that the on-policy distribution has significant advantages when function approximation is used. At the current time this is the only distribution for which we can guarantee convergence with general linear function approximation.
Results are for randomly generated tasks of two sizes and various branching factors, b .
Focusing on the on-policy distribution could be beneficial because it causes vast, uninteresting parts of the space to be ignored, or it could be detrimental because it causes the same old parts of the space to be backed up over and over. We conducted a small experiment to assess the effect empirically. The tasks were undiscounted episodic tasks, generated randomly as follows. From each of the states, two actions were possible, each of which resulted in one of b next states, all equally likely, with a different random selection of b states for each state-action pair. The branching factor, b , was the same for all state-action pairs.