A neural network system for generating a system output from a system input, the neural network system comprising: a memory storing a set of register vectors and data defining a plurality of modules, wherein each module is a respective function that takes as input one or more first vectors and outputs a second vector; a controller neural network configured to, for each of a plurality of time steps: receive a neural network input for the time step; and process the neural network input for the time step to generate a neural network output for the time step; a subsystem configured to, for each of the plurality of time steps: determine, from the neural network output, inputs to each of the plurality of modules; process, for each of the modules, the input to the module using the module to generate a respective module output; determine, from the neural network output, updated values for the register vectors using the module outputs; and generate a neural network input for the next time step from the updated values of the register vectors; and an external variable-sized memory tape, wherein the plurality of modules comprises a first module that reads from the external variable-sized memory tape in accordance with the input to the first module and a second module that writes to the external variable-sized memory tape in accordance with the input to the second module; and the subsystem is further configured to read a pointer as input from the variable-sized memory tape. a memory storing a set of register vectors and data defining a plurality of modules, wherein each module is a respective function that takes as input one or more first vectors and outputs a second vector; a controller neural network configured to, for each of a plurality of time steps: receive a neural network input for the time step; and process the neural network input for the time step to generate a neural network output for the time step; receive a neural network input for the time step; and process the neural network input for the time step to generate a neural network output for the time step; a subsystem configured to, for each of the plurality of time steps: determine, from the neural network output, inputs to each of the plurality of modules; process, for each of the modules, the input to the module using the module to generate a respective module output; determine, from the neural network output, updated values for the register vectors using the module outputs; and generate a neural network input for the next time step from the updated values of the register vectors; and determine, from the neural network output, inputs to each of the plurality of modules; process, for each of the modules, the input to the module using the module to generate a respective module output; determine, from the neural network output, updated values for the register vectors using the module outputs; and generate a neural network input for the next time step from the updated values of the register vectors; and an external variable-sized memory tape, wherein the plurality of modules comprises a first module that reads from the external variable-sized memory tape in accordance with the input to the first module and a second module that writes to the external variable-sized memory tape in accordance with the input to the second module; and the subsystem is further configured to read a pointer as input from the variable-sized memory tape.