Patent Description:
Even though voice interaction with electronic devices has advanced in the past decade, there still lacks the ease and intelligence of human-to-human interaction.

Voice interfaces capable of disambiguating voice input are known, e.g., patent application <CIT>. However, such voice interfaces are often limited in scope and capabilities when compared to other modalities of interaction.

The solution to the problem is described by the description and defined by the claims.

The following description is made for the purpose of illustrating the general principles of one or more embodiments and is not meant to limit the inventive concepts claimed herein. Further, particular features described herein can be used in combination with other described features in each of the various possible combinations and permutations. Unless otherwise specifically defined herein, all terms are to be given their broadest possible interpretation including meanings implied from the specification as well as meanings understood by those skilled in the art and/or as defined in dictionaries, treatises, etc..

One or more embodiments provide for improving, for example, an intelligent assistant's response to voice input based on contextual information provided by other mechanisms/modalities. Multiple advantages may be achieved by the context based interaction that include, but are not limited to the following: controlling a piece of content (e.g., a music file, video file, etc.) amidst a myriad of options (e.g., play, forward, pause, etc.), conditional individual selection, shortening navigation, disambiguation of voice command options, consolidation of actions to an item, transferring properties between items without knowing their details, disambiguating in large collections of content, allows on-screen user interface (UI) to interact with off screen UI/ processes, replacing multiple onscreen UIs, perform spatial tasks more efficiently, disambiguation of service to be used to complete an action, dynamic refinement of UI based on real-time voice and hover input, contextual query, conditional mass selection/ interaction, allows a more conversational interface using demonstratives (terms like 'this' or 'that'), use demonstratives to add contextual intelligence to various activities such as selection, exclusion, assignment, query etc., uninterrupted, real-time modification of interaction properties, disambiguation between multiple voice enabled devices - allows spatially targeting input to a particular device, etc., which are further described below.

The examples described below show how various data including different inputs may be correlated together to improve user interactions and controls, according to some embodiments. While some examples show how gesture inputs or controlling device inputs may be used to provide contextual information to facilitate voice inputs, in some embodiments voice inputs may be used to provide contextual information to support gesture inputs or controlling device inputs.

Spatial context is a crucial component of most human to human interaction. For example, when a parent says "pick this up and put it over there," words such as 'this' and 'there' (i.e., demonstratives), serve as critical links in the child's understanding of the parent's intention and the objects to which they are targeted to. Voice interfaces of electronic devices can understand the action in the command (e.g., pick, put, etc.) based on natual language processing and understanding, but will have difficulty understanding the full intent of the command without spatial context.

Users are able to speak about what they know. It is common, however, that users are not aware of the full description of an item or object that they would like to interact with or obtain information about. Voice interfaces can use speech synthesis to list the user's options, but they often end up being lengthy, cumbersome experiences rife with recall issues for the users. In other times, it may be difficult in distinguishing similar sounds and/or pronunciation.

In a multi-modal world, context from one modality would impact interaction on another. Current voice interfaces hardly use such cross modal inputs for richer context. When there are multiple pieces of content that may all activate with the same "key word," it may cause disruptive overlap because there is no spatial disambiguation for voice commands. The same could be said about multiple devices activated via the same keyword.

It is also cumbersome and unnatural to perform spatial tasks with a voice only interface. This is especially true in a multi-device scenario, when the user is in front of a cluster of voice enabled devices, but wants to speak or emit commands to a specific device. Users are forced to perform associations with the desired device for example, using the device names, and are forced to remember such details. Additionally, this also lengthens interactions by adding one more step of device selection via voice before any interaction could happen.

In situations such as shopping using voice agents, users have to provide detailed information such as the complete name of the product, the flavor, the size etc., which becomes a lengthy interaction. Various embodiments have addressed the challenges identified above.

<FIG> is a schematic view of a communications system <NUM>, in accordance with one embodiment. Communications system <NUM> may include a communications device that initiates an outgoing communications operation (transmitting device <NUM>) and a communications network <NUM>, which transmitting device <NUM> may use to initiate and conduct communications operations with other communications devices within communications network <NUM>. For example, communications system <NUM> may include a communication device that receives the communications operation from the transmitting device <NUM> (receiving device <NUM>). Although communications system <NUM> may include multiple transmitting devices <NUM> and receiving devices <NUM>, only one of each is shown in <FIG> to simplify the drawing.

Any suitable circuitry, device, system or combination of these (e.g., a wireless communications infrastructure including communications towers and telecommunications servers) operative to create a communications network may be used to create communications network <NUM>. Communications network <NUM> may be capable of providing communications using any suitable communications protocol. In some embodiments, communications network <NUM> may support, for example, traditional telephone lines, cable television, Wi-Fi (e.g., an IEEE <NUM> protocol), BLUETOOTH®, high frequency systems (e.g., <NUM>, <NUM>, and <NUM> communication systems), infrared, other relatively localized wireless communication protocol, or any combination thereof. In some embodiments, the communications network <NUM> may support protocols used by wireless and cellular phones and personal email devices (e.g., a Blackberry®). Such protocols may include, for example, GSM, GSM plus EDGE, CDMA, quadband, and other cellular protocols. In another example, a long-range communications protocol can include Wi-Fi and protocols for placing or receiving calls using VOIP, LAN, WAN, or other TCP-IP based communication protocols. The transmitting device <NUM> and receiving device <NUM>, when located within communications network <NUM>, may communicate over a bidirectional communication path such as path <NUM>, or over two unidirectional communication paths. Both the transmitting device <NUM> and receiving device <NUM> may be capable of initiating a communications operation and receiving an initiated communications operation.

The transmitting device <NUM> and receiving device <NUM> may include any suitable device for sending and receiving communications operations. For example, the transmitting device <NUM> and receiving device <NUM> may include, but are not limited to mobile telephone devices, television systems, cameras, camcorders, a device with audio video capabilities, tablets, wearable devices, smart appliances, smart picture frames, and any other device capable of communicating wirelessly (with or without the aid of a wireless-enabling accessory system) or via wired pathways (e.g., using traditional telephone wires). The communications operations may include any suitable form of communications, including for example, voice communications (e.g., telephone calls), data communications (e.g., e-mails, text messages, media messages), video communication, or combinations of these (e.g., video conferences).

<FIG> shows a functional block diagram of an architecture system <NUM> that may be used for contextual interaction with voice, hand or pointing devices, and for interacting with display devices <NUM>-N <NUM> (e.g., tablets, monitors, digital photo display frames, computing displays, television displays, projected displays, etc.; where N is = <NUM>) using an electronic device <NUM> (e.g., mobile telephone devices, television (TV) systems, cameras, camcorders, a device with audio video capabilities, tablets, pad devices, wearable devices, smart appliances, smart picture frames, smart lighting, etc.). Both the transmitting device <NUM> (<FIG>) and receiving device <NUM> may include some or all of the features of the electronics device <NUM>. In one embodiment, the electronic device <NUM> may comprise a display <NUM>, a microphone <NUM>, an audio output <NUM>, an input mechanism <NUM>, communications circuitry <NUM>, control circuitry <NUM>, a camera <NUM>, a context interaction app <NUM> (for contextual interaction with voice, gesture or devices, and for interacting with display devices <NUM>-N <NUM>), and any other suitable components. In one embodiment, applications <NUM>-N <NUM> are provided and may be obtained from a cloud or server <NUM>, a communications network <NUM>, etc., where N is a positive integer equal to or greater than <NUM>.

In one embodiment, all of the applications employed by the audio output <NUM>, the display <NUM>, input mechanism <NUM>, communications circuitry <NUM>, and the microphone <NUM> may be interconnected and managed by control circuitry <NUM>. In one example, a handheld music player capable of transmitting music to other tuning devices may be incorporated into the electronics device <NUM>.

In one embodiment, the audio output <NUM> may include any suitable audio component for providing audio to the user of electronics device <NUM>. For example, audio output <NUM> may include one or more speakers (e.g., mono or stereo speakers) built into the electronics device <NUM>. In some embodiments, the audio output <NUM> may include an audio component that is remotely coupled to the electronics device <NUM>. For example, the audio output <NUM> may include a headset, headphones, or earbuds that may be coupled to communications device with a wire (e.g., coupled to electronics device <NUM> with a jack) or wirelessly (e.g., BLUETOOTH® headphones or a BLUETOOTH® headset).

In one embodiment, the display <NUM> may include any suitable screen or projection system for providing a display visible to the user. For example, display <NUM> may include a screen (e.g., an LCD screen, LED screen, OLED screen, etc.) that is incorporated in the electronics device <NUM>. As another example, display <NUM> may include a movable display or a projecting system for providing a display of content on a surface remote from electronics device <NUM> (e.g., a video projector). Display <NUM> may be operative to display content (e.g., information regarding communications operations or information regarding available media selections) under the direction of control circuitry <NUM>.

In one embodiment, input mechanism <NUM> may be any suitable mechanism or user interface for providing user inputs or instructions to electronics device <NUM>. Input mechanism <NUM> may take a variety of forms, such as a button, keypad, dial, a click wheel, mouse, visual pointer, remote control, one or more sensors (e.g., a camera or visual sensor, a light sensor, a proximity sensor, a capacitive hover sensor, etc., or a touch screen). The input mechanism <NUM> may include a multi-touch screen.

In one embodiment, communications circuitry <NUM> may be any suitable communications circuitry operative to connect to a communications network (e.g., communications network <NUM>, <FIG>) and to transmit communications operations and media from the electronics device <NUM> to other devices within the communications network. Communications circuitry <NUM> may be operative to interface with the communications network using any suitable communications protocol such as, for example, Wi-Fi (e.g., an IEEE <NUM> protocol), Bluetooth®, high frequency systems (e.g., <NUM>, <NUM>, and <NUM> communication systems), infrared, GSM, GSM plus EDGE, CDMA, quadband, and other cellular protocols, VOIP, TCP-IP, or any other suitable protocol.

In some embodiments, communications circuitry <NUM> may be operative to create a communications network using any suitable communications protocol. For example, communications circuitry <NUM> may create a short-range communications network using a short-range communications protocol to connect to other communications devices. For example, communications circuitry <NUM> may be operative to create a local communications network using the Bluetooth® protocol to couple the electronics device <NUM> with a Bluetooth® headset.

In one embodiment, control circuitry <NUM> may be operative to control the operations and performance of the electronics device <NUM>. Control circuitry <NUM> may include, for example, a processor, a bus (e.g., for sending instructions to the other components of the electronics device <NUM>), memory, storage, or any other suitable component for controlling the operations of the electronics device <NUM>. In some embodiments, a processor may drive the display and process inputs received from the user interface. The memory and storage may include, for example, cache, Flash memory, ROM, and/or RAM/ DRAM. In some embodiments, memory may be specifically dedicated to storing firmware (e.g., for device applications such as an operating system, user interface functions, and processor functions). In some embodiments, memory may be operative to store information related to other devices with which the electronics device <NUM> performs communications operations (e.g., saving contact information related to communications operations or storing information related to different media types and media items selected by the user).

In one embodiment, the control circuitry <NUM> may be operative to perform the operations of one or more applications implemented on the electronics device <NUM>. Any suitable number or type of applications may be implemented. Although the following discussion will enumerate different applications, it will be understood that some or all of the applications may be combined into one or more applications. For example, the electronics device <NUM> may include applications <NUM>-N <NUM> including, but not limited to: an automatic speech recognition (ASR) application, OCR application, a dialog application, a map application, a media application (e.g., QuickTime, MobileMusic. app, or MobileVideo. app), social networking applications (e.g., FACEBOOK®, INSTAGRAM®, TWITTER®, etc.), a calendaring application (e.g., a calendar for managing events, appointments, etc.), an Internet browsing application, etc. In some embodiments, the electronics device <NUM> may include one or multiple applications operative to perform communications operations. For example, the electronics device <NUM> may include a messaging application, an e-mail application, a voicemail application, an instant messaging application (e.g., for chatting), a videoconferencing application, a fax application, or any other suitable application for performing any suitable communications operation.

In some embodiments, the electronics device <NUM> may include a microphone <NUM>. For example, electronics device <NUM> may include microphone <NUM> to allow the user to transmit audio (e.g., voice audio) for speech control and navigation of applications <NUM>-N <NUM>, during a communications operation or as a means of establishing a communications operation or as an alternative to using a physical user interface. The microphone <NUM> may be incorporated in the electronics device <NUM>, or may be remotely coupled to the electronics device <NUM>. For example, the microphone <NUM> may be incorporated in wired headphones, the microphone <NUM> may be incorporated in a wireless headset, the microphone <NUM> may be incorporated in a remote control device, etc..

In one embodiment, the camera module <NUM> comprises one or more camera devices that include functionality for capturing still and video images, editing functionality, communication interoperability for sending, sharing, etc. photos/videos, etc..

In one embodiment, the context interaction app <NUM> comprises processes and/or programs for integrating contextual interaction with voice, gesture or devices for playing, manipulating, selecting, using, exploring, etc. content elements. The content elements may include: visual content, graphic images, video content, photos, music content, etc. The context interaction app <NUM> may include any of the processing for, but not limited to, the examples as described below.

In one embodiment, the electronics device <NUM> may include any other component suitable for performing a communications operation. For example, the electronics device <NUM> may include a power supply, ports, or interfaces for coupling to a host device, a secondary input mechanism (e.g., an ON/OFF switch), or any other suitable component.

In some embodiments, the following classifications may be used for the examples described below:.

Additionally, other classifications may be implemented, such as voice + gaze, etc..

<FIG> shows an example of controlling a content element amidst a myriad of options, according to some embodiments. In one example, when multiple video episodes from a season of a television (TV) show are selectable for content playback, instead of saying the name of the show, the season and the specific episode number, in some embodiments the user can simply point to the content (with their hand/finger) that they are interested in playing and saying only "Play that one. " The context interaction processing (e.g., using system <NUM> (<FIG>), device <NUM>, context interaction app <NUM> or any combination thereof) correlates the gesture input with the voice input to determine the correct content to be selected and the action to be taken on the content. In the example of <FIG>, a display <NUM> is shown with three content elements (e.g., videos, music files, etc.). The user points with a finger for gesture <NUM> at the content element <NUM> and speaks an utterance <NUM> "play that one. " The microphone (e.g., microphone <NUM>) receives the utterance and a speech processing application determines the ambiguity of the terms "that one. " Without further interaction, a typical system would have to query further as to which content element the user wants the "play" command to apply. In some embodiments, however, the context interaction processing uses the additional context of the gesture <NUM> as captured by an input mechanism <NUM> in combination with the spoken ambiguity (utterance <NUM>) to select content element <NUM> and play (action) the selected content element <NUM> on an electronic device <NUM>. In some embodiments, upon detection of a gesture <NUM>, the context interaction processing determines that the gesture <NUM> is pointing toward the content element <NUM>, and uses the determination as a selection, along with the action term "play" (determined by the utterance <NUM>), and selects an app (e.g., an app from applications <NUM>-N <NUM>) to perform the action (e.g., a content player). Any media device contains several different types of content such as images, music, videos, etc. Depending on the content type, only a certain set of actions makes sense for that particular content type. For example, "play"-ing the content makes sense for music and video, but not for a photo or a document. Therefore, in some embodiments each content is mapped with a narrow set of commands that are relevant to that content type. Similarly, each application can only play certain content types. When the user utters the word "play" while gesturing at a content, the context interaction app <NUM> (or context interaction processing <NUM>) checks to see if this command matches with the set of commands supported by this content type. If it does, processing proceeds to accomplish that command using an application that supports the selected content type. In some environments, it could be desirable to know that the gesture and voice command were issued by the same person. It may be observed here that the person issuing the gesture has to be close to the input mechanism (on the device or in range of the pointer) while using the electronic device <NUM>, and it would be unnecessary to process any voice input coming from afar. The distance range of a microphone (e.g., microphone <NUM>) may be dynamically modified by altering its signal to noise ratio. As a result, a far-field microphone can be instantly switched to near-field mode, only picking up sounds within a smaller region. In some embodiments, when the electronic device <NUM> senses engagement by the user via a gesture input, it automatically switches to near-field mode, reducing the voice input range to a much smaller region and hence only picking up voice commands by the user who is in close proximity to the electronic device <NUM>. This can allow the electronic device <NUM> to be used in moderately noisy or loud environments ranging from group home movie watching experiences to public spaces such as airports.

In one or more embodiments, a combination of face recognition, face identification, lip movement recognition, and posture recognition techniques may be used to identify if the voice commands being received are also from the user who is performing the gesture. In scenarios where two users are sitting side-by-side with each other while only one of them is interacting with the electronic device <NUM>, these techniques can allow to associate voice input to the user who is performing the gesture and selectively process them, while ignoring the other user who might also be speaking but not necessarily directing commands to the device.

In some embodiments, the category of within device disambiguation refers to a class of techniques in which context is used to enhance voice input to improve interactions within a single display <NUM>. There are several scenarios within the state of a single display <NUM>, where voice input alone is inadequate/insufficient to control the display's <NUM> state. In some embodiments, context may be achieved using additional modalities, resulting in an improved user experience. For Voice + Hover, voice input may be of particular interest to users when they interact with computing devices <NUM> that they cannot or prefer not to touch directly. For example, when the user tries to follow digital recipes in the kitchen, or practice certain sports following digital instructions.

<FIG> shows an example of a conditional individual selection, according to some embodiments. In some embodiments, conditional selection is accomplished when considering both voice input and hover input (e.g., a gesture). In this example, when a user's finger hovers (gesture <NUM>) over a specific object (content element <NUM>) displayed at a display <NUM> surface of an electronic device <NUM>, not only can they just provide simple action commands such as 'play' to activate that object, but they can also provide compound conditional statements as instructions to activate the object. The computing device processes these conditional commands even before opening/activating/launching the chosen content, to display them in a manner desired by the user. As shown, the user provides an utterance <NUM> of "play this one starting at <NUM> minutes. " The context interaction processing determines the action term "play" and the condition "starting at <NUM> minutes" to apply to the content element <NUM>. The context interaction processing uses an input mechanism (e.g., input mechanism <NUM>, <FIG>) to identify what the gesture <NUM> is pointing at on the display <NUM>) along with the action and condition to launch the content element <NUM> at the point <NUM> of the length of play for the content element <NUM> using a voice recognition process or application.

<FIG> shows an example of shortening navigation, according to some embodiments. In a multi-hierarchical menu <NUM>, a user would step through multiple levels of menus to get down to the desired piece of content, which adds considerable number of steps and delay the selection process. Instead, users can use the processing described above to shorten navigation. As shown, the user can hover over the parent multi-hierarchical menu <NUM> that they are interested in and express the particular child/grandchild/great grandchild, etc., menu item that is associated with the content element <NUM> that they are interested in, and jump to selecting the specific content element <NUM> directly by providing an utterance <NUM> of "play the episode about whales, start at <NUM> seconds in. " The voice recognition process or application determines whether the specified content element <NUM> is in the multi-hierarchical menu <NUM> based on the gesture <NUM> pointing to the multi-hierarchical menu <NUM> (of the two available menus on the display <NUM>) and a search within the multi-hierarchical menu <NUM> based on the terms "about whales. " Once the content element <NUM> is found, the context interaction processing starts the appropriate app/application for playing the content element <NUM> at the point <NUM>, which is <NUM> seconds from the start of the video.

<FIG> shows an example of disambiguation of voice command options, according to some embodiments. This example shows how voice helps a user understand what she can do with a particular content element <NUM> shown on a display <NUM>. When the gesture <NUM> for a user's finger hovers over a specific object (content element <NUM>) displayed at a display <NUM> surface of an electronic device (e.g., an electronic device <NUM>), there may be many actions available. By saying, for example, "options," the electronic device is able to use both the hover input (gesture <NUM>) and the voice input (utterance <NUM>) to determine that the user desires to see (or hear) options for only the object her finger(s) hovers over. Without the hover input, the electronic device may not be able to understand the voice command, or might end up providing options available for the whole display <NUM> system instead of for the specific content (content element <NUM>) desired. Without the voice input (utterance <NUM>), gestures more than hovering may be required for the electronic device to decide what the user's intention is. When the options <NUM> pertaining to the content element <NUM> are displayed, the user may speak out the name of one of these options <NUM> to select them - for example, the user would stay on the play icon with gesture <NUM> and speak out "look up" to select that particular option. This reduces travel/movement of hands required for a set of operations. Alternatively, the user can choose to move their hands to hover over the 'look up' object and say "select" to invoke that option.

<FIG> shows an example of consolidation of actions to an item, according to some embodiments. As shown, hover inputs using gesture <NUM> assists in mass editing of content elements <NUM> (shown on display <NUM>) using a voice utterance <NUM>. With the hover input of gesture <NUM>, an electronic device (e.g., electronic device <NUM>, <FIG>) is able to identify one specific object (content element <NUM>) of multiple content elements <NUM> (e.g., a photo gallery) and apply one or more edits to it. The example utterance <NUM> includes the terms "make sepia rotate right and call this oldtime setting. " The voice recognition process or application determines the action words meanings (of "make sepia," "rotate right," and "call") and uses the gesture <NUM> (using input mechanism, e.g., input mechanism <NUM>, <FIG>) to determine the user's intent of selecting content element <NUM>. The computing device is further able to save these editing steps as one template "oldtime setting. " The result of using the context interaction processing is that the content element <NUM> becomes content element <NUM> with the changes made (i.e., color is now sepia, and the content element is rotated to the right <NUM> degrees (e.g., default rotation).

<FIG> shows another example of consolidation of actions to an item, according to some embodiments. With the hover input changes from the example in <FIG>, the electronic device is able to understand that another object (content element <NUM>) is the focus for the next voice command (utterance <NUM>) based on the hover input (gesture <NUM>), and applies the saved template of "oldtime setting" to the content element <NUM> to arrive at the modified content element <NUM> that now has the color sepia, and is rotated to the right.

<FIG> show an example of transferring properties between items (e.g., content elements) without knowing their details, according to some embodiments. As shown, the voice input utterance <NUM> identifies a command ("copy photo filter settings") and the hover input gesture <NUM> helps narrow down to which object the command should be applied. The display <NUM> includes multiple content elements <NUM>, and the gesture <NUM> is pointing towards content element <NUM>. In <FIG>, the context interaction processing applies the voice utterance <NUM> of "paste photo filter settings" to the content element <NUM> selected by the hover gesture <NUM>. In these operations, the user is able to perform a copy of properties from content element <NUM> (<FIG>) to content element <NUM> without having to know the details involved in the setting. In some embodiments, the context interaction processing uses the voice recognition processing or application to determine the term "settings" for the content element <NUM> (e.g., applied photo settings), for example from a photo gallery application. The context interaction processing then applies each of the settings to the content element <NUM> arriving at the modified content element <NUM>.

<FIG> show an example of actions applied to multiple pieces of content <NUM>, according to some embodiments. In <FIG>, upon receipt of the first voice input of utterance <NUM> ("rotate"), the electronic device (e.g., electronic device <NUM>, <FIG>) using the context interaction processing enters into a "Rotating" mode <NUM>. Next, the hover input gesture <NUM> assists the electronic device to determine a first object (content element <NUM>) for application of the subsequent voice command utterance <NUM> "Right," which causes the first object to rotate to the right, shown as object <NUM>.

<FIG> shows another example of actions applied to multiple pieces of content, according to some embodiments. In <FIG>, another gesture <NUM> hovers over a second object (content element <NUM>). While hovering a finger over the second object with gesture <NUM>, the electronic device is able to apply the subsequent voice command utterance <NUM> "Left" to the second object and causes the second object to rotate to the left while in the rotating mode resulting in the modified second object <NUM>. That is, once the rotating mode (or any other mode, such as sizing, filtering, locating, etc.), uttered commands are recognized and interpreted as actions and applied using the gestures based on the input mechanism (e.g., input mechanism <NUM>, <FIG>) and context interaction processing recognizing the hover location relative to the displayed object.

<FIG> show an example of allowing an on-screen user interface (UI) to interact with off screen UI/processes, according to some embodiments. As shown, the context interaction processing provides that instead of having users navigate through multiple menus, hover input plus voice input allow continuous speech across an application. In <FIG>, the electronic device (e.g., an electronic device <NUM>, <FIG>) using the context interaction processing is able to add a track <NUM> selected by the hover gesture <NUM> from a first Album <NUM> (first UI) to playlist "serene" <NUM> using utterance <NUM> of "add to playlist serene" and gesture <NUM>. In <FIG>, the user does not need to repeat the command but can simplify the voice command (utterance <NUM>, "and this one") to add another track <NUM> selected by the hover gesture <NUM> from a second Album <NUM> to the same (serene playlist <NUM>) playlist.

<FIG> show examples of performing spatial tasks efficiently, according to some embodiments. In some embodiments, voice input along with the context provided by spatial input modalities, such as hover input, allow for performing spatial tasks more elegantly and efficiently. <FIG> shows the hover gesture <NUM> along with voice utterance <NUM> of "pickup" applied to a content element <NUM> (e.g., a photograph) displayed on display <NUM>. The context interaction processing provides for commands, such as Pick up, drag, Drop, drop here, etc. for providing the action of picking up, dragging and dropping/placing the content element <NUM>. That is, the context interaction processing provides for the determined intent of selecting, dragging and dropping of the content element <NUM>. As user moves their finger in the direction of arrow <NUM> and stops at gesture <NUM>, the content element <NUM> is linked to its position using the input mechanism (e.g., input mechanism, <NUM>, <FIG>) and the context interaction processing. When the user provides the utterance <NUM> "Drop" (or "drop it here," etc.) and moves their hand away as gesture <NUM> indicates, the content element <NUM> is now anchored to that new position.

In <FIG>, the commands, such as Pick Up, Then Drop (no Drag) voice utterances (utterance <NUM> "pickup" and utterance <NUM> "drop") provides action for picking up and dropping photo on the display <NUM>. The content element <NUM> gains, for example, a highlight, but remains in place. The user moves their finger around hover to gesture <NUM> (e.g., in the direction of arrow <NUM>) and once they find suitable location speaks the utterance <NUM> "drop. " The content element <NUM> then "teleports" to the new location.

In <FIG>, the context interaction processing provides for directional commands. As shown, the user points to an available space with gesture <NUM> and uses voice utterance <NUM> of "Dock photos here," which the electronic device <NUM> (<FIG>) using the context interaction processing moves the content elements <NUM> (e.g., a group of photos, etc.) to a new location.

<FIG> shows an example of disambiguation of service used to complete an action, according to some embodiments. The example shows where a hover input (e.g., gesture <NUM>) serves as a mechanism to disambiguate the service <NUM> (among multiple services) to be used while executing an action. In a voice only interface, if a user requests the electronic device <NUM> (<FIG>) to complete a task, the electronic device will complete the action by picking one of the services that can achieve the action. To play a song by Beyonce if the user provides utterance <NUM> "Play Halo by Beyonce" the electronic device would pick the default application (e.g., Google Play) to play that track over other services such as SPOTIFY® or AMAZON® music. If the user prefers to play the track via a specific application, they have to explicitly mention the same in their voice command "Play Halo by Beyonce using Spotify. " In some embodiments, the user can instead hover over an icon for a compatible service (e.g., gesture <NUM>) and speak of an action (utterance <NUM>), which automatically gets channeled to the service hovered over. As shown the context interaction processing causes the electronic device <NUM> (<FIG>) to play the track <NUM> via the selected service <NUM> instead of the default service or other service options.

<FIG> show examples of dynamic refinement of a UI based on real-time voice input, according to some embodiments. Every interface/modality has its own strength and weakness. Some interfaces/modalities are better at certain tasks than others. Hover/touch/pointing interfaces are screen space input modalities. These modalities are great at selection and disambiguation from choices available within that screen, but their effectiveness is often limited to those visible choices. Typically, these modalities will need to go through multiple menu items to get to choices that are not provided in the current page/screen. Voice input, on the other hand is great for abstract starting points and are not limited by what could fit within a screen. Users can use a voice query to start interaction on elements that are not currently visible in a screen in a single step. On the other hand, they are not good for spatial disambiguation and spatial selection. In some embodiments, dynamic refinement plays on the strength of each interface for rapid and intuitive interaction. As users start to speak their commands, the visual interface using the input mechanism (e.g., input mechanism <NUM>, <FIG>) and context interaction processing filters and reconfigures itself to elements that fit within the context of the command executed. The more the user speaks, the more refined the interface becomes based on using, for example, machine learning models. Simultaneously, users also use their hands to hover and filter the interface even further to arrive at the final choices for selection.

In one example embodiment, when the user is browsing through content elements <NUM> (e.g., photos), as they say the utterance <NUM> "Show me music by Beyonce" the electronic device (e.g., electronic device <NUM>, <FIG>) using the context interaction processing to identify cues such as music and Beyonce to reconfigure the visual interface to only show music albums <NUM> by Beyonce. The user then hovers with gesture <NUM> over an album <NUM> and says the utterance <NUM> "play that," which evokes another tray of services <NUM> to execute the action "play" with. The user can either walk away for the electronic device to select the default service to complete this action, or the user can hover over a service of choice (music service <NUM>) with gesture <NUM> and say the utterance <NUM> "On this. " Here, the voice input serves as a catalyst for continuous refinement of the interface for better hover interaction.

<FIG> show examples of contextual queries, according to some embodiments. In some embodiments, the electronic device (e.g., electronic device <NUM>, <FIG>) uses the context interaction processing for Voice + Cursor combinations. As shown, the user may use a pointing device, e.g., a mouse <NUM> having a left and right click available for interaction. In this example, the right click on a mouse is used as a dedicated input to bring up a voice/speech artificial intelligence (AI) process to assist with directed content.

In <FIG>, when the mouse is right clicked <NUM>, the conventional response is to provide a menu of actions <NUM> that can be taken to the object pointed to by the cursor as a result of mouse movements. However, in some embodiments, there can be many actions for the user to choose.

In <FIG>, with voice command utterance <NUM>, the electronic device is able to take immediate action without further user inputs. In the example embodiment, on the display device <NUM>, the mouse cursor <NUM> is pointing to the word "Texas" on a display of information <NUM> regarding Paris, Texas. The user performs right clicking <NUM> on the mouse <NUM> and provides the utterance <NUM> of "how far away is it," which may also modify the mouse cursor color, shape, intensity, etc. to mouse cursor <NUM>. In some embodiments, the context interaction processing may determine what is shown on the display <NUM> by knowing a search term that resulted in the display of the information <NUM>, using an optical character recognition (OCR) app, etc. The context interaction processing causes the electronic device to determine how far away Paris, Texas is to the user's current location using a map app, distance app, Internet app, etc., to determine distance using either the user's known location, using the electronic device's IP address, etc., the display <NUM> then shows the result <NUM>.

<FIG> shows another example of a contextual query, according to some embodiments. In the example shown, the mouse cursor <NUM> is used with voice to help resolve ambiguity, when one name ("Paris") may be mapped to various instances. The text <NUM> pointed at by the mouse cursor <NUM> helps the electronic device (e.g., electronic device <NUM>, <FIG>) using the context interaction processing understand that it is Paris in Texas of the U. that the user is interested in, instead of Paris, the capital city of France. That is, the context interaction processing can determine the displayed text and use that information to assist in determining the user's intent. In this example, a right click <NUM> is made with the mouse cursor <NUM> pointing to the information <NUM>. The mouse cursor changes to mouse cursor <NUM> from the right click <NUM>, and the user issues the utterance <NUM> of "How far away is PARIS. " The context interaction processing then provides the information <NUM>. In some embodiments for situations when the cursor is not pointing to a specific area in the screen, when a user issues the utterance <NUM> "How far away is PARIS," the electronic device <NUM> (<FIG>) will still try to use the context of what is displayed on the screen to answer the query.

<FIG> shows an example of a conditional mass selection/interaction, according to some embodiments. In some embodiments, the conditional mass selection/interaction uses Voice + Pointer inputs. For this example, multiple objects (e.g., music tracks/ files) are shown on display <NUM>. In this example embodiment, the voice input utterance <NUM> enables the electronic device <NUM> (<FIG>) using the context interaction processing to enter into a "Selecting" mode <NUM>. The user is using a controlling device <NUM> (e.g., a remote control) having a microphone selection <NUM> and pointing element <NUM> that invokes a pointer displayed on the display <NUM>. The user may use the controlling device <NUM> to point to one or more objects displayed. As a result, these objects that satisfy conditions laid out by the initial voice command become selected objects. In this example, the user provides the utterance <NUM> of "Make selection, exclude blues. " In the example, the blues elements <NUM> are desired to be excluded. The user points at the display <NUM> invoking the pointing element <NUM> and makes a selection moving the pointer as shown by the pathway (dashed line) <NUM>. The selected elements are shown as <NUM> outlined boxes while the blues elements <NUM> are not selected.

<FIG> shows an example of providing a more conversational interface using demonstratives, according to some embodiments. In this example embodiment, the electronic device <NUM> (<FIG>) using the context interaction processing uses a pointing element <NUM> of the controlling device <NUM> to identify a part of visual information, upon which a subsequent voice command can further act. As shown, an object shown on the display <NUM> may include a person wearing a shirt <NUM> on a video. The user says an utterance <NUM> "pause" using the microphone selection <NUM> of the controlling device <NUM> to pause the video. The user points <NUM> at the shirt <NUM> with the pointing element <NUM> and issues the utterance <NUM> of "What is that. " The context interaction processing shows a display of the shirt <NUM> with information <NUM> about the shirt <NUM>. In one embodiment, the electronic device and the context interaction processing identify the object (shirt <NUM> shown on the display <NUM>), performs a lookup or search for the object (e.g., using the Internet, a database of product placements, etc.) to recognize the object, and performs a further search if necessary to find further information.

<FIG> shows an example of using demonstratives to add contextual intelligence to various activities, according to some embodiments. Using the electronic device (e.g., electronic device <NUM>) and the context interaction processing, demonstratives may be used to add contextual intelligence to various activities such as selection, exclusion, assignment, query, etc. In this example, a collection of selectable objects (e.g., content elements) <NUM> are shown on a display <NUM>. The user uses a controlling device <NUM> with microphone selection <NUM> and provides the utterance <NUM> of "Select.

<FIG> shows an example of selection of content but excluding elements from the selection in real time using voice control, according to some embodiments. Once the utterance <NUM> (<FIG>) is provided, the controlling device <NUM> may be used to select a pathway <NUM> with the utterance <NUM> to combine the utterance <NUM> of "Select" with the utterance <NUM> of "This and This," the utterance <NUM> of "Not this," and the utterance <NUM> of "and this and this and this. " The utterance <NUM> excludes object <NUM> along the pathway <NUM>. The final selection <NUM> is then left on the display <NUM>.

<FIG> shows an example of uninterrupted, real-time modification of interaction properties, according to some embodiments. In this example, the electronic device (e.g., electronic device <NUM>) and the context interaction processing use Voice + Touch inputs. In this example embodiment, a drawing application executing on the electronic device is presented on the display <NUM>. The user has drawn a curved line <NUM> using the drawing application. When the user wants to change, for example, both a drawing tool and painting color, they may achieve that by touching the screen with gesture <NUM> at a position where they want to draw, and simultaneously issue the utterance <NUM> stating "Water color brush RED. " The user does not have to go to separate pallets to change the color and the tool. The context interaction processing provides the drawing <NUM> with the new color and different drawing shape as the user moves their finger with gesture <NUM>.

<FIG> shows an example of voice plus computer vision, according to some embodiments. In this example, the electronic device (e.g., electronic device <NUM>) and the context interaction processing use Voice + Vision inputs. As described above, using context as a way to disambiguate among on screen UI elements (virtual objects). In some embodiments, context may be used to extend the concept to physical objects. There are several scenarios in the day-to-day life of a user where their interaction with a physical object initiates a virtual interaction within an electronic device. For example, when the user finishes a large bottle of tea in their refrigerator, they may invoke the voice assistant on their electronic device <NUM> and ask to order a six pack of the same. Although interaction with the physical object invokes virtual interaction, the former does not in any way help with or enhance the latter.

With the prevalence of cameras on many smart devices in the home, improved computational capability of the electronic device and the camera becomes a viable additional modality for providing visual context for voice input. Advances in the fields of computer vision, machine learning and deep learning allow detection and labeling of objects within view of the camera. This capability may provide rich additional context that may be used with voice input. As shown, the user uses the camera <NUM> of the electronic device <NUM> to show an object (banana) <NUM> in the display and ask a voice assistant "What can I cook with this?" and the context interaction processing would detect the object <NUM> on the scene, make use of keywords in the utterance command ('cook' and 'this') and natural language processing to determine what the user's intention is. The electronic device <NUM> then shows information <NUM> as a series of recipe in text or videos that the user may select to view or watch. The electronic device <NUM> may make use of other information about the user such as history, location, items available in, for example, a smart refrigerator, upcoming events on a calendar application, etc., to appropriately adjust its recommendations to better suit the user's life. The electronic device <NUM> may use a combination of the list of objects detected in its view, their relative sizes and the context of the command to determine the most likely object that the user is directing their command to. In situations when multiple objects of varying types are visible in the view (e.g., carrots and tomatoes), the electronic device <NUM> may interact with the user via speech synthesis to disambiguate the selection.

<FIG> shows another example of voice plus computer vision, according to some embodiments. Using voice plus computer vision, some embodiments may be used for online shopping. For example, if the user had just finished the last tea bag, they can point the box <NUM> (or packaging) at a camera (e.g., camera <NUM>) of an electronic device (e.g., electronic device <NUM>) and say the utterance "add a couple of these to my AMAZON® cart" or "Buy this" <NUM>. The context interaction processing performs label matching with AMAZON®'s database, identifies the product and adds it to the cart. As shown, display <NUM> then displays the items purchased <NUM> and provides further information <NUM>. In some embodiments, natural language processing is used to determine how many (e.g., a couple) to add to the cart. In other embodiments, the electronic device <NUM> may interact back and forth with the user via speech synthesis to obtain more exact details about the order such as desired date of delivery, address of delivery etc..

<FIG> shows yet another example of voice plus computer vision, according to some embodiments. In one example, a user may point to a brochure <NUM> for a local museum exhibit and say the utterance of "add this to my calendar. " The electronic device (e.g., electronic device <NUM>, <FIG>) using the context interaction processing performs OCR on the frame <NUM> to read the text details on the brochure <NUM>, performs language processing to determine details <NUM> such as time and date, and adds this event to the corresponding time and date in the calendar app of the electronic device or over multiple devices. In all of these scenarios that involve voice + vision, the question might arise as to how does the electronic device <NUM> know when to use the camera for an input. When voice input begins, the electronic device <NUM> looks to see if there is engagement in any of the other input, such as hover, cursor, point touch, etc. When none of these other input modalities are engaged with, it then decides to use the camera output to check if any vision based information was being conveyed by the user.

<FIG> shows an example of tracked points using, for example, LEAP MOTION® hand tracking, according to some embodiments. In some embodiments, visual disambiguation for multiplicity of matches is performed using an electronic device and the context interaction processing. Classification based on part of the body (e.g., hand <NUM>) is possible using computer vision to differentiate between different parts of the body such as the fingers (index <NUM>, middle <NUM>, ring <NUM> and pinky <NUM>), thumb <NUM>, the palm position <NUM> and the wrist position <NUM>. Additionally, positions of different bones <NUM> (e.g., distal <NUM>, intermediate <NUM>, proximal <NUM> and metacarpal <NUM>), tip position <NUM> and arm <NUM> may be differentiated as well.

<FIG> shows an example of a detected device that overlaps with the palm/ fingers and another electronic device, according to some embodiments. In this example, the user is wearing a smart watch <NUM> and holding an electronic device <NUM> (e.g., a smart phone), and says "order me this" (meant to buy the phone). Considering that each of the three devices involved in this scenario, the display <NUM>, the smart watch <NUM> and the electronic device <NUM> (e.g., a smart phone) are smart devices, they might each be running an intelligent assistant of their own. As a result, they might all start listening to a user's voice command as they start speaking. But, only device <NUM> supports the usage of demonstratives such as 'this' due to the visual disambiguation it supports by combining voice with the camera. Therefore, the voice command is only completed by device <NUM>, whereas the other devices do not. Because the camera (e.g., camera <NUM>, <FIG>) has a wide lens and there could potentially be many matching items in the same scene, it becomes important to provide specific means to visually disambiguate the item of interest. The specific case considered is that of a user wearing a smart watch <NUM> and holding the electronic device <NUM> (e.g., a smart phone) in their hand and commanding to the device, for example "Can you order this?" In this example, the user needs to express to the context interaction processing that the item they are interested in ordering is the electronic device <NUM> (smart phone) and not the smart watch <NUM>.

When the context interaction processing detects the electronic device <NUM> and the smart watch <NUM> on the screen <NUM> using the camera device (e.g., camera <NUM>, <FIG>), the context interaction processing can also estimate which among the devices overlaps with the area of the palm or fingers <NUM> (as opposed to overlapping with the wrist region <NUM>). By default, the context interaction processing may assume that the user would hold an object they are trying to order and thereby go with the electronic device <NUM> (smart phone) as the item of interest and order the same.

As shown in <FIG>, further identification based on what the user points to with the other hand. In this embodiment, the context interaction processing detects the pointing direction <NUM> of the index finger of the other hand <NUM> and detects which item it points to/overlaps with and use that as a way of identifying the target item.

<FIG> shows an example of identifying all items in a scene and explicitly providing for selection or disambiguation, according to some embodiments. Some embodiments provide for explicit selection in a second action. In this example, the context interaction processing identifies all the items in a scene and explicitly allows for selection or disambiguation by the user in a second action via voice or other input modalities. As shown, the user is holding an electronic device <NUM> in their hand and is wearing a smart watch <NUM>. The context interaction processing using an electronic device (e.g., electronic device <NUM>, <FIG>) and camera (e.g., camera <NUM>), determines the captured electronic device <NUM> (smart phone) and palm and fingers position <NUM>, and wrist position <NUM> shown on display <NUM>. The context interaction processing identifies the smart phone <NUM> as a first object, and the smart watch <NUM> as the second object as options <NUM> and <NUM>. A voice utterance <NUM> is received by the microphone (e.g., microphone <NUM>). For example, the utterance <NUM> may be "select option <NUM>. " The context interaction processing may then show option <NUM> including the smart phone <NUM> and provides further selections <NUM>, for example in a shopping scenario, confirmed purchase.

<FIG> shows an example of using voice input, along with two gestural input technologies, according to some embodiments. In this example, the electronic device (e.g., electronic device <NUM>) and the context interaction processing use Voice + Hover + touch inputs. In this example embodiment, besides the voice input, two gestural input technologies provide additional context. One of the two gestural inputs serve as the primary input and other serves as the secondary input. When a user views an object such as a video file, they can use the primary input - touch in this particular case to tap on a video on a touch screen and have it start playing. In the event that the user does not know enough about the video file, they can hover over the video and use voice in order to see more options for the tasks they can perform on it. This way of using hover + voice as a secondary input provides more information only when the user requests it, but does not get in the way when the user already knows what to do. As shown, once the user uses a gesture <NUM> (e.g., touch on input mechanism (touch screen) <NUM>, <FIG>) on a content element <NUM> (e.g., a video file), the content element may just be played based on gesture <NUM>, or the user may elect to issue an utterance <NUM> of, for example, "options" along with a second gesture <NUM> of hovering over the content element <NUM>. The result is displaying options <NUM> associated with the content element <NUM>.

<FIG> shows an example of disambiguation between multiple voice enabled devices, according to some embodiments. With the fast adoption of voice (virtual) agents and their addition to many smart electronic devices in the home, their prevalence in a multitude of devices in the home is inevitable. Voice input, although great for a very many tasks, is not well suited for spatial selection. Performing spatial tasks with voice input conventionally remains, at best, an awkward and cumbersome experience. In one example embodiment, a cluster of voice enabled displays are kept together, such as in the case of a group of smart digital picture frames (smart picture frames <NUM>, <NUM>, <NUM> and <NUM>) set up on a wall for decor. When the user stands in front of this cluster of displays and speaks out a voice command (e.g., "switch to my wedding album"), conventionally there is no clear way of determining which device the user intended to target their command to. Users could potentially use ways of disambiguating between these devices by mentioning the device's name in addition to their voice command. While this can work, this puts the burden of remembering the device names or other identification on the user. Another option is that the name of the user can be displayed on each display, but still the need to have to read the name of a display, speak it back and then follow up with the voice commands is unintuitive and cumbersome. It should be noted that conventionally the user has to repeat this device name every single time.

In some embodiments, providing context via an additional modality is a quicker way of targeting commands to a specific device. A block-to-speak model may be used for disambiguation among a cluster of displays. Each device in the cluster may be setup with an inexpensive sensor, such as Infra-red, time of flight, a camera, etc., that detects obstructions in the area immediately in front of the electronic device. In some embodiments, the user performs an action - blocking <NUM> the sensor (without touching) on a smart digital picture frame <NUM> to engage with that electronic device and speak voice commands to the blocked device. Only the device that has its sensors blocked within a threshold (e.g., one inch away, two inches away, five inches away, etc.) will open a microphone and listen for an utterance from the user. The neighboring devices either do not listen or do not process the voice input even though they are close enough to hear the user. This example may be used as a way of targeting speech input to an intended device.

Alternate embodiments may include other modalities such as touch or pointing with a remote control or controlling device maybe used as effective alternatives to blocking to achieve the same effect mentioned above. In some embodiments, blocking a sensor of a display with pointing from afar may be employed to result in a point-to-speak interaction model. The user points their hands at an electronic device and starts speaking. The voice commands target the pointed to device. This may be achieved using, for example, a KINECT® or similar type of camera that detects the human, their arm orientation and use that direction to determine which electronic device it is pointed to via a pre-calibrated map.

<FIG> shows an example of using demonstratives in combination with physical gestures, according to some embodiments. In some embodiments, instead of voice input being available on each individual device, a central hub handles voice input for all the electronic devices in a cluster of devices. In one example, a user walks up and start speaking to the cluster as a whole, but use demonstratives in combination with physical gestures just as blocking/pointing only while targeting to a particular display. This allows emanating batch commands. As shown, the electronic devices <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and <NUM> (e.g., electronic devices <NUM>) make up a cluster. The user issues an utterance of "load Anna's daughter pictures on this, this, and this" with each utterance of 'this' pointing to/blocking a different display in the cluster. Here, the user provides an utterance of "this" with a gesture <NUM> pointing to electronic device <NUM>, with a gesture <NUM> pointing to electronic device <NUM> and with a gesture <NUM> pointing to electronic device <NUM>. The result is displaying the intended photos (e.g., of Anna's daughter) on the intended electronic devices (electronic devices <NUM>, <NUM> and <NUM>).

<FIG> shows a block diagram of a process <NUM> for using demonstrables for cues, according to some embodiments. A sequence of logical steps provides cues to the voice assistant to use image data when appropriate. In block <NUM>, a user speaks a voice command (e.g., to an electronic device <NUM> (<FIG>) having a microphone <NUM>) that is captured, sent to a natural language processing system (e.g., a voice recognition program, virtual agent, an artificial intelligence (AI) system, etc.), and a text transcription of the voice command is received by context interaction processing (e.g., using system <NUM> (<FIG>), device <NUM>, context interaction app <NUM> or any combination thereof). The processed command is received back by the electronic device with tags (action, date, time, application to use, etc.). In block <NUM> if the subject of the command is clear, the electronic device proceeds to execute the action. Conversely, if the subject of the command is a demonstrable such as "this" or "these," it does not have sufficient information to execute the command (i.e., the electronic device knows what action to perform, but not on whom or what to perform it on). In block <NUM>, process <NUM> resolves the ambiguity based on one of but not limited to, the following demonstrables (i.e., the electronic device knows the following demonstrables are the possible scenarios):.

In block <NUM>, if the system does not detect any of the modalities listed in <NUM>-<NUM> (e.g., hand not within tracking area, hand not touching screen, pointer not within screen space, etc.), the electronic device determines that context provided via the camera (e.g., camera <NUM>) may be the option (i.e., the camera is the medium used for context). The electronic device using context interaction processing switches to the camera (unless an image is already captured) for an additional modality to resolve what 'this' means. In block <NUM>, the electronic device activates the camera (if no image is already present), takes a picture, scans for objects and identifies the objects. If the identified object is recognized (e.g., matches an element in a database, etc.) the electronic device executes the corresponding action for that object.

<FIG> shows a block diagram of a process <NUM> for context interaction, according to some embodiments. In block <NUM>, a voice input is received at an electronic device (e.g., electronic device <NUM>, <FIG>). In block <NUM>, an ambiguity of the voice input is determined (e.g., a voice utterance of "this" or "that"). In block <NUM>, the ambiguity is resolved based on contextual data (e.g., within device disambiguation: Voice + Hover, Voice + Cursor, Voice + Point, Voice + Touch, Voice + Show, or combinations thereof). In some embodiments, the contextual data includes at least one of: an image (e.g., on a display <NUM>), a non-voice input (e.g., a gesture, a pointer of a pointing device, a touch on a touch screen (e.g., input mechanism <NUM>), or a combination thereof.

In some embodiments, in process <NUM> the ambiguity relates to identification of an object or a position that the voice input applies to. The image may be captured by a camera (e.g., camera <NUM>, <FIG>) connected to the electronic device, displayed on a display device (e.g., display <NUM>), or a combination thereof. In process <NUM>, the non-voice input may be sensed by at least one sensor (e.g., input mechanism <NUM>) connected to the electronic device.

In some embodiments, process <NUM> may further include resolving the ambiguity by determining identification of the object based on the image containing the object (e.g., captured by a camera <NUM>), or the non-voice input indicating the object (e.g., a hand or finger gesture, a pointer from a pointing device, a touch, a hover, etc.). In process <NUM>, the ambiguity may include a demonstrative determiner (e.g., terms like 'this' or 'that', see, e.g., <FIG>, <FIG>, <FIG>), and the contextual data further includes information affecting an action applicable to the object (e.g., apply a setting, etc.).

In some embodiments, process <NUM> may further include resolving the ambiguity by enabling the camera upon a determination that no non-voice input occurred (e.g., no occurrence of: the user providing context via voice + hover (hover-to-speak), the user providing context via voice + cursor, the user providing context via voice + point, the user providing context via voice + touch, etc.).

In some embodiments, in process <NUM> the object may be the electronic device (itself), and the electronic device adjusts an interface corresponding to the voice input with the ambiguity resolved. Process <NUM> may include that the action applicable to the object includes: receiving information (e.g., information about the object from an Internet search), assisting with a purchase (e.g., finding the object or similar object for sale on-line, adding to a virtual shopping cart, etc.), calendaring an event (e.g., using a calendaring app on the electronic device(s)), applying features to content (e.g., applying settings, copying and pasting settings to an object, etc.), selecting at least one content associated with the object (e.g., selecting a photo from a gallery of photos, etc.), moving the object on a display (e.g., picking up, dragging, dropping, etc.), or a combination thereof.

<FIG> is a high-level block diagram showing an information processing system comprising a computing system implementing one or more embodiments. The system <NUM> includes one or more processors <NUM> (e.g., ASIC, CPU, etc.), and may further include an electronic display device <NUM> (for displaying graphics, text, and other data), a main memory <NUM> (e.g., random access memory (RAM), cache devices, etc.), storage device <NUM> (e.g., hard disk drive), removable storage device <NUM> (e.g., removable storage drive, removable memory, a magnetic tape drive, optical disk drive, computer-readable medium having stored therein computer software and/or data), user interface device <NUM> (e.g., keyboard, touch screen, keypad, pointing device), and a communication interface <NUM> (e.g., modem, wireless transceiver (such as Wi-Fi, Cellular), a network interface (such as an Ethernet card), a communications port, or a PCMCIA slot and card).

The communication interface <NUM> allows software and data to be transferred between the computer system and external devices through the Internet <NUM>, mobile electronic device <NUM>, a server <NUM>, a network <NUM>, etc. The system <NUM> further includes a communications infrastructure <NUM> (e.g., a communications bus, cross bar, or network) to which the aforementioned devices <NUM> through <NUM> are connected.

The information transferred via communications interface <NUM> may be in the form of signals such as electronic, electromagnetic, optical, or other signals capable of being received by communications interface <NUM>, via a communication link that carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, a radio frequency (RF) link, and/or other communication channels.

In one implementation of one or more embodiments in a mobile wireless device (e.g., a mobile phone, tablet, wearable device, etc.), the system <NUM> further includes an image capture device <NUM>, such as a camera <NUM> (<FIG>), and an audio capture device <NUM>, such as a microphone <NUM> (<FIG>). The system <NUM> may further include application processing or processors as MMS <NUM>, SMS <NUM>, email <NUM>, social network interface (SNI) <NUM>, audio/video (AV) player <NUM>, web browser <NUM>, image capture <NUM>, etc..

In one embodiment, the system <NUM> includes context interaction processing <NUM> that may implement processing similar as described regarding context interaction app <NUM> (<FIG>), processing for process <NUM>, and process <NUM> as described above. In one embodiment, the context interaction processing <NUM> along with an operating system <NUM> may be implemented as executable code residing in a memory of the system <NUM>. In another embodiment, the context interaction processing <NUM> may be provided in hardware, firmware, etc..

In one embodiment, the main memory <NUM>, storage device <NUM> and removable storage device <NUM>, each by themselves or in any combination, may store instructions for the embodiments described above that may be executed by the one or more processors <NUM>.

As is known to those skilled in the art, the aforementioned example architectures described above, according to said architectures, can be implemented in many ways, such as program instructions for execution by a processor, as software modules, microcode, as computer program product on computer readable media, as analog/logic circuits, as application specific integrated circuits, as firmware, as consumer electronic devices, AV devices, wireless/wired transmitters, wireless/wired receivers, networks, multi-media devices, etc. Further, embodiments of said Architecture can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements.

One or more embodiments have been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to one or more embodiments. Each block of such illustrations/ diagrams, or combinations thereof, can be implemented by computer program instructions. The computer program instructions when provided to a processor produce a machine, such that the instructions, which execute via the processor create means for implementing the functions/operations specified in the flowchart and/or block diagram. Each block in the flowchart/block diagrams may represent a hardware and/or software module or logic, implementing one or more embodiments. In alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures, concurrently, etc..

The terms "computer program medium," "computer usable medium," "computer readable medium", and "computer program product," are used to generally refer to media such as main memory, secondary memory, removable storage drive, a hard disk installed in hard disk drive. These computer program products are means for providing software to the computer system. The computer readable medium allows the computer system to read data, instructions, messages or message packets, and other computer readable information from the computer readable medium. The computer readable medium, for example, may include non-volatile memory, such as a floppy disk, ROM, flash memory, disk drive memory, a CD-ROM, and other permanent storage. It is useful, for example, for transporting information, such as data and computer instructions, between computer systems. Computer program instructions may be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/ or block diagram block or blocks.

Computer program instructions representing the block diagram and/or flowcharts herein may be loaded onto a computer, programmable data processing apparatus, or processing devices to cause a series of operations performed thereon to produce a computer implemented process. Computer programs (i.e., computer control logic) are stored in main memory and/or secondary memory. Computer programs may also be received via a communications interface. Such computer programs, when executed, enable the computer system to perform the features of the embodiments as discussed herein. In particular, the computer programs, when executed, enable the processor and/ or multi-core processor to perform the features of the computer system. Such computer programs represent controllers of the computer system. A computer program product comprises a tangible storage medium readable by a computer system and storing instructions for execution by the computer system for performing a method of one or more embodiments.

Claim 1:
A method performed by an electronic device (<NUM>), comprising:
receiving a voice input of a user;
determining that the voice input is ambiguous, if the voice input includes a demonstrative determiner indicating an object;
if the voice input is determined ambiguous, resolving ambiguity related to the voice input based on contextual data including an image containing the object by:
obtaining the image containing at least one object using a camera (<NUM>) of the electronic device (<NUM>),
scanning the obtained image for objects,
identifying the object from the obtained image, and
recognizing the identified object based on the identified object matching an element in a database;
determining intention of the user associated with the recognized object using keywords of the voice input and natural language processing; and
performing an action corresponding to the voice input based on the determined intention of the user.