A system configured to select actions from a set of actions to be performed by a mechanical agent interacting with a real-world environment, the system comprising one or more computers configured to: implement a dueling deep neural network, the dueling deep neural network comprising: one or more initial neural network layers configured to: receive an observation captured by one or more sensors of the environment and/or the mechanical agent and characterizing one or more properties of the real-world environment; process the observation to generate a representation of the observation characterizing a current state of the environment; a value subnetwork configured to: receive the representation of the observation; and process the representation of the observation to generate a value estimate, the value estimate being an estimate of an expected return resulting from the environment being in the current state; an advantage subnetwork configured to: receive the representation of the observation; and process the representation of the observation to generate a respective advantage estimate for each action in the set of actions that is an estimate of a relative measure of the return resulting from the mechanical agent performing the action when the real-world environment is in the current state relative to the return resulting from the mechanical agent performing other actions when the real-world environment is in the current state; a combining layer configured to, for each action, combine the value estimate and the respective advantage estimate for the action to generate a respective Q value for the action, wherein the respective Q value is an estimate of an expected return resulting from the mechanical agent performing the action when the real-world environment is in the current state; and instructions that when executed by the one or more computers, cause the one or more computers to select an action to be performed by the mechanical agent in response to the observation using the respective Q values for the actions in the set of actions. one or more initial neural network layers configured to: receive an observation captured by one or more sensors of the environment and/or the mechanical agent and characterizing one or more properties of the real-world environment; process the observation to generate a representation of the observation characterizing a current state of the environment; receive an observation captured by one or more sensors of the environment and/or the mechanical agent and characterizing one or more properties of the real-world environment; process the observation to generate a representation of the observation characterizing a current state of the environment; a value subnetwork configured to: receive the representation of the observation; and process the representation of the observation to generate a value estimate, the value estimate being an estimate of an expected return resulting from the environment being in the current state; receive the representation of the observation; and process the representation of the observation to generate a value estimate, the value estimate being an estimate of an expected return resulting from the environment being in the current state; an advantage subnetwork configured to: receive the representation of the observation; and process the representation of the observation to generate a respective advantage estimate for each action in the set of actions that is an estimate of a relative measure of the return resulting from the mechanical agent performing the action when the real-world environment is in the current state relative to the return resulting from the mechanical agent performing other actions when the real-world environment is in the current state; receive the representation of the observation; and process the representation of the observation to generate a respective advantage estimate for each action in the set of actions that is an estimate of a relative measure of the return resulting from the mechanical agent performing the action when the real-world environment is in the current state relative to the return resulting from the mechanical agent performing other actions when the real-world environment is in the current state; a combining layer configured to, for each action, combine the value estimate and the respective advantage estimate for the action to generate a respective Q value for the action, wherein the respective Q value is an estimate of an expected return resulting from the mechanical agent performing the action when the real-world environment is in the current state; and instructions that when executed by the one or more computers, cause the one or more computers to select an action to be performed by the mechanical agent in response to the observation using the respective Q values for the actions in the set of actions.