A computer-implemented method of training a policy neural network having a plurality of policy network parameters, wherein the policy neural network is configured to receive a policy input comprising an observation characterizing a current state of an environment and to process the network input in accordance with the policy network parameters to generate a policy output that defines a probability distribution over a space of possible actions to be performed by an agent interacting with the environment, wherein the policy neural network is trained jointly with a Q-value neural network (i) having a plurality of Q network parameters and (ii) configured to receive a Q network input comprising data identifying an action and the observation and to process the Q network input in accordance with the Q network parameters to generate a Q value for the action, and wherein the method comprises: sampling a mini-batch comprising one or more observation - action - reward tuples generated as a result of interactions of a first agent with a first environment; determining an update to current values of the Q network parameters by minimizing a robust entropy-regularized temporal difference error that accounts for possible perturbations of the states of the first environment represented by the observations in the sampled mini-batch of observation - action - reward tuples, wherein the robust entropy-regularized temporal difference error measures, for each tuple, an error between (i) a sum of the reward in the tuple and an infimum or average of Q values for perturbed next states and (ii) a Q value for the observation-action pair in the tuple, wherein each perturbed next state represents a perturbation of the first environment; determining, using the Q-value neural network, an update to the policy network parameters using the sampled mini-batch of observation - action - reward tuples; and wherein the policy neural network is configured to control a mechanical agent to perform a task in a real-world environment. wherein the policy neural network is configured to receive a policy input comprising an observation characterizing a current state of an environment and to process the network input in accordance with the policy network parameters to generate a policy output that defines a probability distribution over a space of possible actions to be performed by an agent interacting with the environment, wherein the policy neural network is trained jointly with a Q-value neural network (i) having a plurality of Q network parameters and (ii) configured to receive a Q network input comprising data identifying an action and the observation and to process the Q network input in accordance with the Q network parameters to generate a Q value for the action, and wherein the method comprises: sampling a mini-batch comprising one or more observation - action - reward tuples generated as a result of interactions of a first agent with a first environment; determining an update to current values of the Q network parameters by minimizing a robust entropy-regularized temporal difference error that accounts for possible perturbations of the states of the first environment represented by the observations in the sampled mini-batch of observation - action - reward tuples, wherein the robust entropy-regularized temporal difference error measures, for each tuple, an error between (i) a sum of the reward in the tuple and an infimum or average of Q values for perturbed next states and (ii) a Q value for the observation-action pair in the tuple, wherein each perturbed next state represents a perturbation of the first environment; determining, using the Q-value neural network, an update to the policy network parameters using the sampled mini-batch of observation - action - reward tuples; and wherein the policy neural network is configured to control a mechanical agent to perform a task in a real-world environment. sampling a mini-batch comprising one or more observation - action - reward tuples generated as a result of interactions of a first agent with a first environment; determining an update to current values of the Q network parameters by minimizing a robust entropy-regularized temporal difference error that accounts for possible perturbations of the states of the first environment represented by the observations in the sampled mini-batch of observation - action - reward tuples, wherein the robust entropy-regularized temporal difference error measures, for each tuple, an error between (i) a sum of the reward in the tuple and an infimum or average of Q values for perturbed next states and (ii) a Q value for the observation-action pair in the tuple, wherein each perturbed next state represents a perturbation of the first environment; determining, using the Q-value neural network, an update to the policy network parameters using the sampled mini-batch of observation - action - reward tuples; and wherein the policy neural network is configured to control a mechanical agent to perform a task in a real-world environment.