Patent ID: 11868866
Assignee: DEEPMIND TECHNOLOGIES LIMITED
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A method comprising:
receiving a current observation characterizing a current state of an environment being interacted with by an agent;
sampling a plurality of actions from a set of possible actions that can be performed by the agent to interact with the environment, and wherein each action in the set of possible actions comprises a respective action value for each of a plurality of sub-actions, the sampling comprising:
sampling one or more random actions randomly from the set of possible actions;
sampling one or more proposal actions, comprising, for each proposal action:
sampling action values for the proposal action auto-regressively one after the other according to an ordering of the sub-actions, comprising, for each sub-action:
processing (i) the current observation and (ii) sampled action values for any sub-actions before the particular sub-action in the ordering using a proposal neural network having a plurality of proposal network parameters, wherein the proposal neural network is configured to process (i) the current observation and (ii) the sampled action values in accordance with current values of the proposal network parameters to generate a proposal output for the sub-action that defines a sub-action probability distribution over a set of possible action values for the sub-action, and
sampling the action value for the proposal action from the sub-action probability distribution for the sub-action;
processing the current observation and each sampled random and proposal action using a Q neural network having a plurality of Q network parameters, wherein, for each sampled action, the Q neural network is configured to process the current observation and the sampled action in accordance with current values of the Q network parameters to generate a Q value for the sampled action that is an estimate of a return that would be received if the agent performed the sampled action in response to the current observation;

selecting an action using the Q values generated by the Q neural network for the sampled actions; and
causing the agent to perform the selected action.