A system comprising one or more computers configured to implement a neural network system for invoking a sequence of computer programs selected from a set of computer programs (108A, 108N) to control a mechanical agent to perform a task to change a position of an object within an environment, each of the set of computer programs (108A, 108N) causing the mechanical agent to perform a corresponding act to move an object within the environment to cause the environment to transition into a different state, wherein the different state represents the position of the object within the environment after the object has been moved, wherein for each computer program in the set of computer programs (108A, 108N), a key is stored in a memory identifying the computer program and an embedding for the computer program, the embedding for the computer program comprising one or more vectors representing the computer program, the neural network system comprising: a core recurrent neural network configured to receive a sequence of neural network inputs and to generate a sequence of neural network outputs; and a subsystem configured to, for each neural network output: determine, from the neural network output, whether or not to end a currently invoked computer program and to return to a calling computer program; determine, from the neural network output, a next computer program to be called; determine, from the neural network output, contents of arguments to the next computer program to be called; receive a representation of a current state of the environment representing a current position of the object within the environment; and generate a next neural network input from an embedding for the next computer program to be called and the representation of the current state of the environment, the embedding for the next computer program to be called comprising one or more vectors representing the next computer program to be called, wherein generating the next neural network input comprises: extracting a fixed-length state encoding from the representation of the current state of the environment using a domain-specific encoder; and combining the fixed-length state encoding and the embedding for the next computer program to be called to generate the next neural network input. a core recurrent neural network configured to receive a sequence of neural network inputs and to generate a sequence of neural network outputs; and a subsystem configured to, for each neural network output: determine, from the neural network output, whether or not to end a currently invoked computer program and to return to a calling computer program; determine, from the neural network output, a next computer program to be called; determine, from the neural network output, contents of arguments to the next computer program to be called; receive a representation of a current state of the environment representing a current position of the object within the environment; and generate a next neural network input from an embedding for the next computer program to be called and the representation of the current state of the environment, the embedding for the next computer program to be called comprising one or more vectors representing the next computer program to be called, wherein generating the next neural network input comprises: extracting a fixed-length state encoding from the representation of the current state of the environment using a domain-specific encoder; and combining the fixed-length state encoding and the embedding for the next computer program to be called to generate the next neural network input. determine, from the neural network output, whether or not to end a currently invoked computer program and to return to a calling computer program; determine, from the neural network output, a next computer program to be called; determine, from the neural network output, contents of arguments to the next computer program to be called; receive a representation of a current state of the environment representing a current position of the object within the environment; and generate a next neural network input from an embedding for the next computer program to be called and the representation of the current state of the environment, the embedding for the next computer program to be called comprising one or more vectors representing the next computer program to be called, wherein generating the next neural network input comprises: extracting a fixed-length state encoding from the representation of the current state of the environment using a domain-specific encoder; and combining the fixed-length state encoding and the embedding for the next computer program to be called to generate the next neural network input. extracting a fixed-length state encoding from the representation of the current state of the environment using a domain-specific encoder; and combining the fixed-length state encoding and the embedding for the next computer program to be called to generate the next neural network input.