A computational inquiry into navigation, with particular reference to the hippocampus

Goal-directed navigation is the art of traversing space in order to get to a goal. Animals can be expected to learn the capacity to navigate optimally, that is, learn to be able to take direct paths from where they are to where they want to go. However, optimal local behaviours are determined by global constraints. A common theoretical suggestion is that in order to navigate animals build and use a global representation in the form of a map. Maps pose two difficult problems, however: a global-from-local problem in building a consistent map, and, of greater difficulty, a local-from-global problem in reading the map. An alternative to using a map is to learn appropriate local behaviours directly from interacting with the world, using only local representations. This thesis reviews evidence that a neural structure in the mammalian brain known as the hippocampus provides a representation of space that is purely local. The thesis then reviews a selection of computationally efficient, neurally plausible, reinforcement learning methods that can use local representations to learn optimal actions in navigation-like tasks. The key disadvantages usually associated with the methods are slow learning, and inflexibility to change. Both disadvantages were investigated in this thesis in the context of learning to navigate. A first model is presented which learns to perform two hippocampally dependent tasks, one involving navigation to a single goal, the other involving navigation to multiple goals. Animals gradually acquire the single goal task, but in the multiple goals task, gradually acquire the ability to navigate directly to a novel goal on only the second trial to that goal. One component of the model uses place cells within a standard reinforcement learning scheme, temporal difference learning in an actor-critic. This component by itself captures performance in the single goal task, but fails to capture one-trial learning in the multiple goals task. A second component of the model learns globally consistent coordinates from local self-motion information, in a novel application of temporal difference learning. This coordinate learning is relatively independent of the behaviour of the animal, enabling the gradual acquisition of one-trial learning in the multiple goals task to be captured. Two purely behavioural predictions follow which were tested experimentally. First, once a coordinate system has been learned, simple placement of an animal at a novel goal should provide the animal with sufficient information to allow direct paths on the next trial, and this was shown to be true. Second, since coordinates offer no principled way of circumnavigating barriers, the model predicts that animals will be unable to learn the task in the presence of barriers, but this was shown to be false. A second model is presented, based on the idea that unsupervised learning can be used to find structure in value functions (the predictions of reward learned by reinforcement learning methods), that corresponds to structure in the environment, eg due to the presence of barriers. Decompositions of various environments are found of different types – flat, linear and hierarchical. Results show that that structure learned from just a few goals compares well with that learned from all possible goals, and that augmenting a standard state representation with these decompositions leads to faster acquisition, again within the purview of the local, neurally plausible reinforcement learning methods. These results emphasise the importance of representational learning, and suggest that with appropriate representations, simple learning methods can rival more rigidly specified and complex methods.