A computer-implemented method for operating a reinforcement learning system that is configured to select actions to be performed by a mechanical agent interacting with a real-world environment from a continuous action space of actions, the method comprising: training the reinforcement learning system, wherein the training comprises: obtaining an experience tuple identifying a training observation characterizing a training state of the environment, an action performed by the mechanical agent in response to the training observation, a reward received as a result of the mechanical agent performing the action in response to the training observation, and a subsequent observation characterizing a subsequent state of the environment; processing the training observation using a value subnetwork of the reinforcement learning system to generate a first value estimate, the first value estimate being an estimate of an expected return resulting from the environment being in the training state; processing, using a policy subnetwork of the reinforcement learning system and in accordance with current values of the parameters of the policy subnetwork, the training observation to generate an ideal point in the continuous action space for the training observation; wherein the ideal point represents an action that, if performed in response to the training observation, is expected to produce a maximum Q value for all actions in the continuous space of actions; generating an advantage estimate for the training action from a distance between the ideal point and a particular point representing the training action in the continuous action space; generating a Q value for the training action that is an estimate of an expected return resulting from the mechanical agent performing the training action when the environment is in the current state by combining the advantage estimate and the value estimate; processing the subsequent observation using the value subnetwork to generate a new value estimate for the subsequent state, the new value estimate being an estimate of an expected return resulting from the environment being in the subsequent state; combining the reward and the new value estimate to generate a target Q value for the training action; and determining an update to the current values of the parameters of the policy subnetwork using an error between the Q value for the training action and the target Q value; after training of the reinforcement learning system, receiving, by the policy subnetwork, an observation characterizing a current state of the environment; processing, by the policy subnetwork, the observation to generate an ideal point in the continuous action space; and selecting, by the reinforcement learning system, an action for the mechanical agent to perform using the generated ideal point and causing the mechanical agent to perform the selected action. obtaining an experience tuple identifying a training observation characterizing a training state of the environment, an action performed by the mechanical agent in response to the training observation, a reward received as a result of the mechanical agent performing the action in response to the training observation, and a subsequent observation characterizing a subsequent state of the environment; processing the training observation using a value subnetwork of the reinforcement learning system to generate a first value estimate, the first value estimate being an estimate of an expected return resulting from the environment being in the training state; processing, using a policy subnetwork of the reinforcement learning system and in accordance with current values of the parameters of the policy subnetwork, the training observation to generate an ideal point in the continuous action space for the training observation; wherein the ideal point represents an action that, if performed in response to the training observation, is expected to produce a maximum Q value for all actions in the continuous space of actions; generating an advantage estimate for the training action from a distance between the ideal point and a particular point representing the training action in the continuous action space; generating a Q value for the training action that is an estimate of an expected return resulting from the mechanical agent performing the training action when the environment is in the current state by combining the advantage estimate and the value estimate; processing the subsequent observation using the value subnetwork to generate a new value estimate for the subsequent state, the new value estimate being an estimate of an expected return resulting from the environment being in the subsequent state; combining the reward and the new value estimate to generate a target Q value for the training action; and determining an update to the current values of the parameters of the policy subnetwork using an error between the Q value for the training action and the target Q value; after training of the reinforcement learning system, receiving, by the policy subnetwork, an observation characterizing a current state of the environment; processing, by the policy subnetwork, the observation to generate an ideal point in the continuous action space; and selecting, by the reinforcement learning system, an action for the mechanical agent to perform using the generated ideal point and causing the mechanical agent to perform the selected action.