Patent ID: 11960636
Assignee: MAGIC LEAP, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC F  G

Claim 17:
18. A method carried out by a wearable system including a head-mounted display (HMD), an eye gaze tracking device, and a gesture tracking device, the method comprising:
receiving a multimodal input comprising:
first user input from a hardware component of the wearable system, wherein the first user input is associated with a first mode of user input, and
a second user input, wherein the second user input is associated with a second mode of user input;

identifying a multimodal input command comprising:
a subject comprising a selected portion of text, and
a command operation comprising an interaction with the selected portion of text, and

executing the multimodal input command to cause a result of the multimodal input to be displayed to a user on the HMD;
wherein identifying comprises identifying the selected portion of text by:
determining a first confidence score for the selected portion of text utilizing eye gaze direction data received from the eye gaze tracking device;
determining a second confidence score for the selected portion of text utilizing hand gesture data received from the gesture tracking device;
calculating an aggregated confidence score for the selected portion of text based, at least in part, on at least the first confidence score and the second confidence score; and

utilizing the aggregated confidence score to identify the selected portion of text.