This can lead not only to logical problems in setting up the agent models, but also to greatly increasing levels of computational complexity with the depth of recursion. In this scheme, the computational requirements greatly increase with the level of modeling, and furthermore, there is no adequate way for an agent to model other agents as being at the same level of depth. We consider two basic heuristic approaches to avoiding an infinite recursion of opponent models. In this case, the infinite recursion is cut off by the finite depth of the search. For example, the chess machines Deep Thought and Deep Blue give the impression of generating sophisiticated positional understanding as an emergent property from deep searches plus simple positional knowledge built into the evaluation function.
Policy iteration is guaranteed to converge to the optimal agent policy for single-agent MDPs.
As a general caveat, we should point out that both classes of algorithms mentioned above seek deterministic optimal policies. However, it may the case that such deterministic policies do not exist.