Patent ID: 11941504
Assignee: GOOGLE LLC
Field: nan (nan)
Classification: CPC nan | IPC B  G

Claim 18:
19. A method implemented by one or more processors, comprising:
receiving free-form natural language input, the free-form natural language input generated based on user interface input provided by a user via one or more user interface input devices;
generating a memory network output based on processing the free-form natural language input using a memory network;
for each of a plurality of iterations:
receiving a corresponding instance of vision data, the corresponding instance of vision data generated based on corresponding output from one or more vision sensors of a vision component of a robot for the iteration;
generating at least one corresponding vision embedding using at least one vision branch comprising a plurality of convolutional neural network layers, wherein generating the at least one corresponding vision embedding comprises:
generating semantic vision data, wherein generating the semantic vision data is based on processing at least some of the corresponding instance of vision data using one or more additional neural networks, and wherein the semantic vision data includes natural language labels of objects captured in the at least some of the corresponding instance of vision data, and
generating a semantic embedding of the at least one corresponding vision embedding, wherein generating the semantic embedding is based on processing the semantic vision data using a semantic vision branch of the at least one vision branch of the neural network model;

generating a corresponding natural language embedding based on processing the memory network output using an attention layer conditioned at least in part on the corresponding vision embedding;
generating a corresponding action prediction output based on processing of the at least one corresponding vision embedding and the corresponding natural language embedding using a plurality of feedforward layers, wherein the corresponding action prediction output indicates a corresponding robotic action, of a plurality of candidate robot actions, to be performed based on the corresponding instance of vision data and the free-form natural language input; and
controlling one or more actuators of a robot based on the corresponding action prediction output, wherein controlling the one or more actuators of the robot causes the robot to perform the corresponding robotic action indicated by the corresponding action prediction output.