A neural network system implemented by one or more computers, the neural network system comprising: an encoder neural network, wherein the encoder neural network is a recurrent neural network that is configured to, for each input data item processed by the encoder neural network and at each time step of a plurality steps: receive a glimpse captured by reading from the input data item, receive a decoder hidden state vector of a decoder neural network for the preceding time step, and process the glimpse, the decoder hidden state vector, and an encoder hidden state vector of the encoder neural network from the preceding time step to generate an encoder hidden state vector for the time step; a decoder neural network, wherein the decoder neural network is a recurrent neural network that is configured to, for each of the plurality of time steps: receive a decoder input for the time step, and process the decoder hidden state vector for the preceding time step and the decoder input to generate a decoder hidden state vector for the time step; and a subsystem, wherein the subsystem is configured to, for each of the time steps: read the glimpse from the input data item using the decoder hidden state vector for the preceding time step; provide the glimpse as input to the encoder neural network; generate the decoder input for the decoder neural network from the encoder hidden state vector at the time step; provide the decoder input as input to the decoder neural network for the time step; generate a neural network output update for the time step from the decoder hidden state vector for the time step; and combine the neural network output update for the time step with a current neural network output to generate an updated neural network output. an encoder neural network, wherein the encoder neural network is a recurrent neural network that is configured to, for each input data item processed by the encoder neural network and at each time step of a plurality steps: receive a glimpse captured by reading from the input data item, receive a decoder hidden state vector of a decoder neural network for the preceding time step, and process the glimpse, the decoder hidden state vector, and an encoder hidden state vector of the encoder neural network from the preceding time step to generate an encoder hidden state vector for the time step; receive a glimpse captured by reading from the input data item, receive a decoder hidden state vector of a decoder neural network for the preceding time step, and process the glimpse, the decoder hidden state vector, and an encoder hidden state vector of the encoder neural network from the preceding time step to generate an encoder hidden state vector for the time step; a decoder neural network, wherein the decoder neural network is a recurrent neural network that is configured to, for each of the plurality of time steps: receive a decoder input for the time step, and process the decoder hidden state vector for the preceding time step and the decoder input to generate a decoder hidden state vector for the time step; and receive a decoder input for the time step, and process the decoder hidden state vector for the preceding time step and the decoder input to generate a decoder hidden state vector for the time step; and a subsystem, wherein the subsystem is configured to, for each of the time steps: read the glimpse from the input data item using the decoder hidden state vector for the preceding time step; provide the glimpse as input to the encoder neural network; generate the decoder input for the decoder neural network from the encoder hidden state vector at the time step; provide the decoder input as input to the decoder neural network for the time step; generate a neural network output update for the time step from the decoder hidden state vector for the time step; and combine the neural network output update for the time step with a current neural network output to generate an updated neural network output. read the glimpse from the input data item using the decoder hidden state vector for the preceding time step; provide the glimpse as input to the encoder neural network; generate the decoder input for the decoder neural network from the encoder hidden state vector at the time step; provide the decoder input as input to the decoder neural network for the time step; generate a neural network output update for the time step from the decoder hidden state vector for the time step; and combine the neural network output update for the time step with a current neural network output to generate an updated neural network output.