A system comprising one or more computers and one or more storage devices storing instructions that when executed cause the one or more computers to implement: a mapping subsystem configured to, at each time step of a plurality of time steps, generate a characterization of an environment from an image of the environment at the time step, wherein the characterization comprises an environment map identifying locations in the environment having a particular characteristic, wherein the particular characteristic is learned by the mapping subsystem without supervision, and wherein generating the characterization comprises, for each time step: obtaining the image of the environment at the time step, wherein the image is an image captured by an agent moving through the environment, processing the image using a neural network to generate a first initial characterization of the environment for the time step, obtaining a final characterization of the environment for a previous time step preceding the time step, obtaining a measure of movement of the agent between the previous time step and the time step, processing the final characterization of the environment for the previous time step to generate a second initial characterization of the environment for the time step, comprising applying a differentiable warping function to the final characterization of the environment for the previous time step and the measure of movement to generate the second initial characterization; and combining the first initial characterization and the second initial characterization to generate a final characterization of the environment for the time step; and a planning subsystem configured to, for each of the plurality of time steps: obtain the final characterization for the time step from the mapping subsystem, and process the final characterization to select a proposed action to be performed by the agent interacting with the environment at the time step; wherein the mapping subsystem is trained by backpropagating errors from the planning subsystem, the errors being between selected actions and optimal actions. a mapping subsystem configured to, at each time step of a plurality of time steps, generate a characterization of an environment from an image of the environment at the time step, wherein the characterization comprises an environment map identifying locations in the environment having a particular characteristic, wherein the particular characteristic is learned by the mapping subsystem without supervision, and wherein generating the characterization comprises, for each time step: obtaining the image of the environment at the time step, wherein the image is an image captured by an agent moving through the environment, processing the image using a neural network to generate a first initial characterization of the environment for the time step, obtaining a final characterization of the environment for a previous time step preceding the time step, obtaining a measure of movement of the agent between the previous time step and the time step, processing the final characterization of the environment for the previous time step to generate a second initial characterization of the environment for the time step, comprising applying a differentiable warping function to the final characterization of the environment for the previous time step and the measure of movement to generate the second initial characterization; and combining the first initial characterization and the second initial characterization to generate a final characterization of the environment for the time step; and obtaining the image of the environment at the time step, wherein the image is an image captured by an agent moving through the environment, processing the image using a neural network to generate a first initial characterization of the environment for the time step, obtaining a final characterization of the environment for a previous time step preceding the time step, obtaining a measure of movement of the agent between the previous time step and the time step, processing the final characterization of the environment for the previous time step to generate a second initial characterization of the environment for the time step, comprising applying a differentiable warping function to the final characterization of the environment for the previous time step and the measure of movement to generate the second initial characterization; and combining the first initial characterization and the second initial characterization to generate a final characterization of the environment for the time step; and a planning subsystem configured to, for each of the plurality of time steps: obtain the final characterization for the time step from the mapping subsystem, and process the final characterization to select a proposed action to be performed by the agent interacting with the environment at the time step; obtain the final characterization for the time step from the mapping subsystem, and process the final characterization to select a proposed action to be performed by the agent interacting with the environment at the time step; wherein the mapping subsystem is trained by backpropagating errors from the planning subsystem, the errors being between selected actions and optimal actions.