However, even in a Markovian task, the single job of learning the direct mapping can be difficult. Only when the task is non Markovian is it necessary to equip the agent with some sort of internal state. However, even in a Markovian task, the single job of learning the direct mapping can be difficult. One often investigated problem results from dealing with a large state space. L.-J. Lin and T. Mitchell.
Memory approaches to reinforcement learning in non-markovian domains .
The desired quantity at each step of a backgammon game is the continuous valued probability of winning, whereas the desired output of a phoneme recognition problem is a classification decision among 50 or more discrete choices. L.-J. Lin and T. Mitchell. Memory approaches to reinforcement learning in nonMarkovian domains . One way to include memory in a reinforcement learning algorithm like Q learning is to use a recurrent neural network to represent the Qvalues. This approach works well on small problems but may suffer from local optima in complex problems.