Patent ID: 11972339
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G  B | IPC B  G

Claim 19:
20. A method implemented by one or more processors, comprising:
identifying free-form natural language that describes navigation to a goal location;
generating a memory network output based on processing the free-form natural language input using a memory network;
for each of a plurality of iterations:
receiving a corresponding instance of simulated or real vision data;
generating at least one corresponding vision embedding using at least one vision branch comprising a plurality of convolutional neural network layers, wherein generating the at least one corresponding vision embedding comprises:
generating semantic vision data, wherein generating the semantic vision data is based on processing at least some of the corresponding instance of simulated or real vision data using one or more additional neural networks, and wherein the semantic vision data includes natural language labels of objects captured in the at least some of the corresponding instance of simulated or real vision data, and
generating a semantic embedding of the at least one corresponding vision embedding, wherein generating the semantic embedding is based on processing the semantic vision data using a semantic vision branch of the at least one vision branch of the neural network model;

generating a corresponding natural language embedding based on processing the memory network output using an attention layer conditioned at least in part on the corresponding vision embedding;
generating a corresponding action prediction output based on processing of the at least one corresponding vision embedding and the corresponding natural language embedding using a plurality of feedforward layers that represents an action-value function, wherein the corresponding action prediction output indicates a corresponding robotic action, of a plurality of candidate robot actions, to be performed based on the corresponding instance of simulated or real vision data and the free-form natural language input; and
controlling a real or simulated robot based on the corresponding action prediction output;
determining whether controlling the real or simulated robot based on the corresponding action prediction output caused the real or simulated robot to reach the goal location; and
when it is determined that controlling the real or simulated robot based on the corresponding action prediction output caused the real or simulated robot to reach the goal location:
using a reward, that corresponds to reaching the goal location, in updating at least the feedforward layers that approximate the action value function.