A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement: a memory (108; 114) configured to store a respective memory vector at each of a plurality of memory locations in the memory; a neural network that comprises a memory module for relational reasoning, wherein the neural network is configured to: at each of a plurality of time steps: receive an input; determine an update to the memory; update the memory using the determined update to the memory; and generate an output for the current time step using the updated memory; wherein determining the update comprises the memory module applying an attention mechanism over the memory vectors in the memory and the received input; the memory module maintains a plurality of parameter sets; and the memory module comprises a plurality of self-attention layers, each self-attention layer corresponding to: i) a respective parameter set, ii) a respective sub-memory of the memory that is a subset of the memory vectors in the memory, and iii) a respective sub-memory of an appended memory that is created by appending the received input to the memory; wherein at each time step, each self-attention layer is configured to determine a proposed update to the respective sub-memory by applying an attention mechanism over memory vectors in the respective sub-memory and the respective sub-memory of the appended memory using the respective parameter set; wherein the self-attention layers operate in parallel and the attention mechanism is applied over the plurality of memory locations at a single time step; and wherein the memory module is further configured to, at each time step, combine the proposed updates generated by the plurality of self-attention layers to generate the update for the memory. a memory (108; 114) configured to store a respective memory vector at each of a plurality of memory locations in the memory; a neural network that comprises a memory module for relational reasoning, wherein the neural network is configured to: at each of a plurality of time steps: receive an input; determine an update to the memory; update the memory using the determined update to the memory; and generate an output for the current time step using the updated memory; wherein receive an input; determine an update to the memory; update the memory using the determined update to the memory; and generate an output for the current time step using the updated memory; wherein determining the update comprises the memory module applying an attention mechanism over the memory vectors in the memory and the received input; the memory module maintains a plurality of parameter sets; and the memory module comprises a plurality of self-attention layers, each self-attention layer corresponding to: i) a respective parameter set, ii) a respective sub-memory of the memory that is a subset of the memory vectors in the memory, and iii) a respective sub-memory of an appended memory that is created by appending the received input to the memory; wherein at each time step, each self-attention layer is configured to determine a proposed update to the respective sub-memory by applying an attention mechanism over memory vectors in the respective sub-memory and the respective sub-memory of the appended memory using the respective parameter set; wherein the self-attention layers operate in parallel and the attention mechanism is applied over the plurality of memory locations at a single time step; and wherein the memory module is further configured to, at each time step, combine the proposed updates generated by the plurality of self-attention layers to generate the update for the memory.