A neural network system for processing a sequence of system inputs to generate a sequence of system outputs, the neural network system comprising: a controller neural network configured to receive a neural network input at each of a plurality of time steps and to process the controller neural network input to generate a neural network output for the time step, wherein the controller neural network input for each time step is a respective system input from the sequence of system inputs, wherein, for a first time step, the neural network input includes the system input for the first time step and pre-determined data and, for subsequent time steps, the neural network input includes the system input for the respective time step and data read from the external memory, and wherein each neural network output includes: a read key, and a write vector; an external memory, the external memory being external to the controller neural network; and a sparse memory access subsystem that is configured to, for each of the plurality of time steps: generate a sparse set of reading weights that includes a respective reading weight for each of a plurality of locations in the external memory using the read key, read data (214, 310) from the plurality of locations in the external memory in accordance with the sparse set of reading weights, generate a set of writing weights that includes a respective writing weight for each of the plurality of locations in the external memory, and write the write vector to the plurality of locations in the external memory in accordance with the writing weights; wherein the sparse memory access subsystem is configured to generate the sparse set of reading weights by: determining the K locations of the plurality of locations that are most similar to the read key according to a similarity measure; assigning a zero reading weight to each location of the plurality of locations other than the K locations; and determining the reading weight for each of the K locations from the similarity measure between the location and the read key. a controller neural network configured to receive a neural network input at each of a plurality of time steps and to process the controller neural network input to generate a neural network output for the time step, wherein the controller neural network input for each time step is a respective system input from the sequence of system inputs, wherein, for a first time step, the neural network input includes the system input for the first time step and pre-determined data and, for subsequent time steps, the neural network input includes the system input for the respective time step and data read from the external memory, and wherein each neural network output includes: a read key, and a write vector; a read key, and a write vector; an external memory, the external memory being external to the controller neural network; and a sparse memory access subsystem that is configured to, for each of the plurality of time steps: generate a sparse set of reading weights that includes a respective reading weight for each of a plurality of locations in the external memory using the read key, read data (214, 310) from the plurality of locations in the external memory in accordance with the sparse set of reading weights, generate a set of writing weights that includes a respective writing weight for each of the plurality of locations in the external memory, and write the write vector to the plurality of locations in the external memory in accordance with the writing weights; wherein the sparse memory access subsystem is configured to generate the sparse set of reading weights by: generate a sparse set of reading weights that includes a respective reading weight for each of a plurality of locations in the external memory using the read key, read data (214, 310) from the plurality of locations in the external memory in accordance with the sparse set of reading weights, generate a set of writing weights that includes a respective writing weight for each of the plurality of locations in the external memory, and write the write vector to the plurality of locations in the external memory in accordance with the writing weights; wherein the sparse memory access subsystem is configured to generate the sparse set of reading weights by: determining the K locations of the plurality of locations that are most similar to the read key according to a similarity measure; assigning a zero reading weight to each location of the plurality of locations other than the K locations; and determining the reading weight for each of the K locations from the similarity measure between the location and the read key.