An off-policy reinforcement learning actor-critic neural network system, comprising one or more computers in one or more locations, to select actions to be performed by an agent, wherein the agent is a robot, interacting with a real-world environment to perform a task, the system comprising: a first input to receive an observation comprising state data characterizing a state of the environment, and reward data representing a reward from operating with an action in the environment; at least one actor neural network, coupled to receive the state data and configured to define a policy function mapping the state data to action data defining an action, wherein the at least one actor neural network has an output to provide the action data for the agent to perform the action, and wherein the environment transitions to a new state in response to the action; at least one critic neural network, coupled to receive the action data, the state data, and return data derived from the reward data, and configured to define a value function which generates an error signal; a replay buffer to store reinforcement learning transitions comprising operation transition data from operation of the system, wherein the operation transition data comprises tuples of said state data, said action data, said reward data and new state data representing said new state; and a second input to receive training data defining demonstration transition data, the demonstration transition data comprising a set of said tuples from a demonstration of the task within the environment, wherein the demonstration transition data comprises kinesthetic data from manipulation of the robot, wherein reinforcement learning transitions stored in the replay buffer further comprise the demonstration transition data; and wherein the neural network system is configured to train the at least one actor neural network and the at least one critic neural network off-policy using the error signal and using stored tuples from the replay buffer comprising tuples from both the operation transition data and the demonstration transition data, wherein the system further comprises a sample selection system to sample the reinforcement learning transitions according to a sampling probability, wherein the sample selection system samples both the demonstration transition data and the operation transition data from the replay buffer, and wherein the replay buffer stores an actor-critic error signal, wherein the actor-critic error signal is a Temporal-Difference error signal, and wherein the sampling probability further prioritizes sampling according to the Temporal Difference error signal. a first input to receive an observation comprising state data characterizing a state of the environment, and reward data representing a reward from operating with an action in the environment; at least one actor neural network, coupled to receive the state data and configured to define a policy function mapping the state data to action data defining an action, wherein the at least one actor neural network has an output to provide the action data for the agent to perform the action, and wherein the environment transitions to a new state in response to the action; at least one critic neural network, coupled to receive the action data, the state data, and return data derived from the reward data, and configured to define a value function which generates an error signal; a replay buffer to store reinforcement learning transitions comprising operation transition data from operation of the system, wherein the operation transition data comprises tuples of said state data, said action data, said reward data and new state data representing said new state; and a second input to receive training data defining demonstration transition data, the demonstration transition data comprising a set of said tuples from a demonstration of the task within the environment, wherein the demonstration transition data comprises kinesthetic data from manipulation of the robot, wherein reinforcement learning transitions stored in the replay buffer further comprise the demonstration transition data; and wherein the neural network system is configured to train the at least one actor neural network and the at least one critic neural network off-policy using the error signal and using stored tuples from the replay buffer comprising tuples from both the operation transition data and the demonstration transition data, wherein the system further comprises a sample selection system to sample the reinforcement learning transitions according to a sampling probability, wherein the sample selection system samples both the demonstration transition data and the operation transition data from the replay buffer, and wherein the replay buffer stores an actor-critic error signal, wherein the actor-critic error signal is a Temporal-Difference error signal, and wherein the sampling probability further prioritizes sampling according to the Temporal Difference error signal.