Patent ID: 11972339
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G  B | IPC B  G

Claim 17:
18. The method of claim 1, further comprising:
after controlling one or more actuators of a robot based on the action prediction output:
receiving an additional instance of vision data, the additional instance of vision data generated based on additional output from one or more vision sensors of a vision component of a robot;
generating at least one additional vision embedding based on processing the additional instance of vision data using the at least one vision branch of the neural network model;
generating an additional natural language embedding based on processing the free-form natural language input using the language branch of the neural network model, and based on attention weighting, during the processing, that is based on the additional vision embedding;
generating an additional action prediction output based on processing of the at least one additional vision embedding and the additional natural language embedding using the action prediction layers of the neural network model, wherein the additional action prediction output indicates an additional robotic action to be performed based on the additional instance of vision data and the free-form natural language input; and
controlling the one or more actuators of the robot based on the additional action prediction output, wherein controlling the one or more actuators of the robot causes the robot to perform the additional robotic action indicated by the additional action prediction output.