A system comprising: a recurrent neural network implemented by one or more computers and configured to, at each of a plurality of time steps: receive a preceding action for a preceding time step, the preceding action being an action performed at the preceding time step by an agent interacting with an environment or to be performed at the preceding time step by the agent; update a preceding initial hidden state of the recurrent neural network from the preceding time step using the preceding action to generate an initial hidden state for the time step; update a preceding cell state of the recurrent neural network from the preceding time step using at least the initial hidden state for the time step to generate a cell state for the time step; and determine a final hidden state for the time step using the cell state for the time step, wherein, at each of one or more initial time steps of the plurality of time steps: an encoded representation of a preceding actual observation is also received, the preceding actual observation being an image characterizing an actual state of the environment at a preceding time step; and the preceding cell state of the recurrent neural network from the preceding time step is updated using the encoded representation of the preceding actual observation as well as the initial hidden state for the time step to generate the cell state for the time step; and a decoder neural network implemented by the one or more computers and configured to, at each of one or more of the plurality of time steps: receive the final hidden state for the time step; and process the final hidden state to generate a predicted observation, wherein the predicted observation is a predicted image of the environment at the time step characterizing a predicted state of the environment at the time step. a recurrent neural network implemented by one or more computers and configured to, at each of a plurality of time steps: receive a preceding action for a preceding time step, the preceding action being an action performed at the preceding time step by an agent interacting with an environment or to be performed at the preceding time step by the agent; update a preceding initial hidden state of the recurrent neural network from the preceding time step using the preceding action to generate an initial hidden state for the time step; update a preceding cell state of the recurrent neural network from the preceding time step using at least the initial hidden state for the time step to generate a cell state for the time step; and determine a final hidden state for the time step using the cell state for the time step, wherein, at each of one or more initial time steps of the plurality of time steps: an encoded representation of a preceding actual observation is also received, the preceding actual observation being an image characterizing an actual state of the environment at a preceding time step; and the preceding cell state of the recurrent neural network from the preceding time step is updated using the encoded representation of the preceding actual observation as well as the initial hidden state for the time step to generate the cell state for the time step; and receive a preceding action for a preceding time step, the preceding action being an action performed at the preceding time step by an agent interacting with an environment or to be performed at the preceding time step by the agent; update a preceding initial hidden state of the recurrent neural network from the preceding time step using the preceding action to generate an initial hidden state for the time step; update a preceding cell state of the recurrent neural network from the preceding time step using at least the initial hidden state for the time step to generate a cell state for the time step; and determine a final hidden state for the time step using the cell state for the time step, wherein, at each of one or more initial time steps of the plurality of time steps: an encoded representation of a preceding actual observation is also received, the preceding actual observation being an image characterizing an actual state of the environment at a preceding time step; and the preceding cell state of the recurrent neural network from the preceding time step is updated using the encoded representation of the preceding actual observation as well as the initial hidden state for the time step to generate the cell state for the time step; and an encoded representation of a preceding actual observation is also received, the preceding actual observation being an image characterizing an actual state of the environment at a preceding time step; and the preceding cell state of the recurrent neural network from the preceding time step is updated using the encoded representation of the preceding actual observation as well as the initial hidden state for the time step to generate the cell state for the time step; and receive the final hidden state for the time step; and process the final hidden state to generate a predicted observation, wherein the predicted observation is a predicted image of the environment at the time step characterizing a predicted state of the environment at the time step.