A computer-implemented method of training a policy neural network that is to be used to control a real-world robot to perform a task that involves navigating within and/or moving an object within a real-world environment, wherein the policy neural network has a plurality of policy parameters and is configured to receive as input a state representation characterizing a state of the environment and to process the state representation in accordance with the policy parameters to generate a policy output that defines a probability distribution over a set of actions that can be performed by the robot, wherein, when the policy neural network is in use to control the robot within the real-world environment, a) the state representation is generated from sensor data captured by one or more sensors of the robot and b) an action to be performed by the robot is selected based on the policy output, wherein the policy neural network is to be used to control the robot to perform the task that involves navigating within and/or moving an object within a real-world environment by repeatedly causing the robot to perform actions that are selected based on policy outputs generated by the policy neural network, wherein the method comprises performing multiple iterations of: obtaining trajectory data comprising one or more tuples, each tuple identifying a state representation characterizing a state of the environment, an action performed by the robot when the environment was in the state characterized by the state representation, a reward received in response to the robot performing the action, and a next state representation characterizing a next state of the environment after the robot performed the action; updating, using the trajectory data, current values of the policy parameters using a maximum entropy reinforcement learning technique that maximizes both (i) a reward term that measures total rewards in the tuples in the trajectory data and (ii) an entropy term that measures an entropy of probability distributions defined by policy outputs generated by processing the state representations in the tuples in the trajectory data in accordance with the current values of the policy parameters, wherein a relative weight between the entropy term and the reward term in the maximization is determined by a temperature parameter; and updating, using the probability distributions defined by the policy outputs generated in accordance with the current values of the policy parameters for the tuples in the trajectory data, the temperature parameter to regulate an expected entropy of the probability distributions to at least equal a minimum expected entropy value, wherein the minimum expected entropy value is a fixed value. obtaining trajectory data comprising one or more tuples, each tuple identifying a state representation characterizing a state of the environment, an action performed by the robot when the environment was in the state characterized by the state representation, a reward received in response to the robot performing the action, and a next state representation characterizing a next state of the environment after the robot performed the action; updating, using the trajectory data, current values of the policy parameters using a maximum entropy reinforcement learning technique that maximizes both (i) a reward term that measures total rewards in the tuples in the trajectory data and (ii) an entropy term that measures an entropy of probability distributions defined by policy outputs generated by processing the state representations in the tuples in the trajectory data in accordance with the current values of the policy parameters, wherein a relative weight between the entropy term and the reward term in the maximization is determined by a temperature parameter; and updating, using the probability distributions defined by the policy outputs generated in accordance with the current values of the policy parameters for the tuples in the trajectory data, the temperature parameter to regulate an expected entropy of the probability distributions to at least equal a minimum expected entropy value, wherein the minimum expected entropy value is a fixed value.