Patent ID: 11861263
Assignee: X DEVELOPMENT LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 19:
20. A method implemented by one or more processors, the method comprising:
receiving natural language input that includes a request to control a computer application;
generating a request embedding, generating the request embedding being based on processing the natural language input;
processing the request embedding to determine at least a first candidate action set for the request and a second candidate action set for the request, the first candidate action set including a first group of the actions in a first sequence, and the second candidate action set including the first group of the actions in a second sequence or a second group of the actions in a third sequence,
wherein each of the actions is implementable through interaction with the computer application;

prior to any interacting, responsive to the request, with the computer application to control the computer application:
performing a first simulation of controlling the computer application using the first candidate action set;
performing a second simulation of controlling the computer application using the second candidate action set;
generating user interface output that is based on first simulated data from the first simulation and second simulated data from the second simulation;
causing the user interface output to be rendered in response to receiving the natural language input;
receiving user interface input in response to rendering the user interface output; and
determining, based on the user interface input, to automatically select the first candidate action set in lieu of at least the second candidate action set; and

in response to the request and in response to determining to automatically select the first candidate action set:
interacting with the computer application to control the computer application using the first candidate action set.