Patent Description:
This relates generally to digital assistant and, more specifically, to determining when to enable various sensors of an electronic device using a digital assistant in various computer-generated reality technologies.

Intelligent automated assistants (or digital assistants) can provide a beneficial interface between human users and electronic devices. Such assistants can allow users to interact with devices or systems using natural language in spoken and/or text forms. For example, a user can provide a speech input containing a user request to a digital assistant operating on an electronic device. The digital assistant can interpret the user's intent from the speech input and operationalize the user's intent into tasks. The tasks can then be performed by executing one or more services of the electronic device, and a relevant output responsive to the user request can be returned to the user. In some cases, a user may provide a request that is ambiguous, particularly when in use with various computer-generated reality technologies; for example, a user request such as "what is that?". Thus, it may be difficult for the digital assistant to determine an appropriate response to the request.

In document <CIT> (<NUM>-<NUM>-<NUM>), a digital assistant receives the spoken utterance and detects a trigger in the utterance. Responsive to detecting the trigger, the digital assistant activates the camera. The camera then captures an image of the object of interest and passes the utterance, the captured image, and context information to the image integrated query system.

The invention is defined by the appended independent claims, with the dependent claims providing further preferred embodiments.

Determining, based on the semantic analysis, a likelihood that the electronic device requires additional contextual data to satisfy the request allows a digital assistant to efficiently determine whether to enable one or more sensors of an electronic device. For example, determining whether additional contextual data is required in this manner allows the digital assistant to selectively determine which sensors may be helpful and enable them in a quick and efficient manner. Thus, this provides for more efficient use of the electronic device (e.g., by only enabling the sensors which will be helpful), which, additionally, reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Further, only enabling the one or more sensors of the electronic device when required provides privacy benefits as everything a user does or interacts with is not captured. Rather, specific activities that will be helpful to the user may be captured with the enabled sensors while all others are not captured.

Various examples of electronic systems and techniques for using such systems in relation to various computer-generated reality technologies are described.

A physical environment (or real environment) refers to a physical world that people can sense and/or interact with without aid of electronic systems. Physical environments, such as a physical park, include physical articles (or physical objects or real objects), such as physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment, such as through sight, touch, hearing, taste, and smell.

In contrast, a computer-generated reality (CGR) environment refers to a wholly or partially simulated environment that people sense and/or interact with via an electronic system. In CGR, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner that comports with at least one law of physics. For example, a CGR system may detect a person's head turning and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), adjustments to characteristic(s) of virtual object(s) in a CGR environment may be made in response to representations of physical motions (e.g., vocal commands).

A person may sense and/or interact with a CGR object using any one of their senses, including sight, sound, touch, taste, and smell. For example, a person may sense and/or interact with audio objects that create a 3D or spatial audio environment that provides the perception of point audio sources in 3D space. In another example, audio objects may enable audio transparency, which selectively incorporates ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and/or interact only with audio objects.

Examples of CGR include virtual reality and mixed reality.

A virtual reality (VR) environment (or virtual environment) refers to a simulated environment that is designed to be based entirely on computer-generated sensory inputs for one or more senses. A VR environment comprises a plurality of virtual objects with which a person may sense and/or interact. For example, computer-generated imagery of trees, buildings, and avatars representing people are examples of virtual objects. A person may sense and/or interact with virtual objects in the VR environment through a simulation of the person's presence within the computer-generated environment, and/or through a simulation of a subset of the person's physical movements within the computer-generated environment.

In contrast to a VR environment, which is designed to be based entirely on computer-generated sensory inputs, a mixed reality (MR) environment refers to a simulated environment that is designed to incorporate sensory inputs from the physical environment, or a representation thereof, in addition to including computer-generated sensory inputs (e.g., virtual objects). On a virtuality continuum, an MR environment is anywhere between, but not including, a wholly physical environment at one end and a VR environment at the other end.

In some MR environments, computer-generated sensory inputs may respond to changes in sensory inputs from the physical environment. Also, some electronic systems for presenting an MR environment may track location and/or orientation with respect to the physical environment to enable virtual objects to interact with real objects (that is, physical articles from the physical environment or representations thereof). For example, a system may account for movements so that a virtual tree appears stationary with respect to the physical ground.

Examples of MR include augmented reality and augmented virtuality.

An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed over a physical environment, or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person may directly view the physical environment. The system may be configured to present virtual objects on the transparent or translucent display, so that a person, using the system, perceives the virtual objects superimposed over the physical environment. Alternatively, a system may have an opaque display and one or more imaging sensors that capture images or video of the physical environment, which are representations of the physical environment. The system composites the images or video with virtual objects, and presents the composition on the opaque display. A person, using the system, indirectly views the physical environment by way of the images or video of the physical environment, and perceives the virtual objects superimposed over the physical environment. As used herein, a video of the physical environment shown on an opaque display is called "pass-through video," meaning a system uses one or more image sensor(s) to capture images of the physical environment, and uses those images in presenting the AR environment on the opaque display. Further alternatively, a system may have a projection system that projects virtual objects into the physical environment, for example, as a hologram or on a physical surface, so that a person, using the system, perceives the virtual objects superimposed over the physical environment.

An AR environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, a system may transform one or more sensor images to impose a select perspective (e.g., viewpoint) different than the perspective captured by the imaging sensors. As another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof, such that the modified portion may be representative but not photorealistic versions of the originally captured images. As a further example, a representation of a physical environment may be transformed by graphically eliminating or obfuscating portions thereof.

An augmented virtuality (AV) environment refers to a simulated environment in which a virtual or computer generated environment incorporates one or more sensory inputs from the physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but people with faces photorealistically reproduced from images taken of physical people. As another example, a virtual object may adopt a shape or color of a physical article imaged by one or more imaging sensors. As a further example, a virtual object may adopt shadows consistent with the position of the sun in the physical environment.

There are many different types of electronic systems that enable a person to sense and/or interact with various CGR environments. Examples include head mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields having integrated display capability, windows having integrated display capability, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop/laptop computers. A head mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head mounted system may be configured to accept an external opaque display (e.g., a smartphone). The head mounted system may incorporate one or more imaging sensors to capture images or video of the physical environment, and/or one or more microphones to capture audio of the physical environment. Rather than an opaque display, a head mounted system may have a transparent or translucent display. The transparent or translucent display may have a medium through which light representative of images is directed to a person's eyes. The display may utilize digital light projection, OLEDs, LEDs, uLEDs, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one example, the transparent or translucent display may be configured to become opaque selectively. Projection-based systems may employ retinal projection technology that projects graphical images onto a person's retina. Projection systems also may be configured to project virtual objects into the physical environment, for example, as a hologram or on a physical surface.

<FIG> and <FIG> depict exemplary system <NUM> for use in various computer-generated reality technologies.

In some examples, as illustrated in <FIG>, system <NUM> includes device 100a. Device 100a includes various components, such as processor(s) <NUM>, RF circuitry(ies) <NUM>, memory(ies) <NUM>, image sensor(s) <NUM>, orientation sensor(s) <NUM>, microphone(s) <NUM>, location sensor(s) <NUM>, speaker(s) <NUM>, display(s) <NUM>, and touch-sensitive surface(s) <NUM>. These components optionally communicate over communication bus(es) <NUM> of device 100a.

In some examples, elements of system <NUM> are implemented in a base station device (e.g., a computing device, such as a remote server, mobile device, or laptop) and other elements of the system <NUM> are implemented in a head-mounted display (HMD) device designed to be worn by the user, where the HMD device is in communication with the base station device. In some examples, device 100a is implemented in a base station device or a HMD device.

As illustrated in <FIG>, in some examples, system <NUM> includes two (or more) devices in communication, such as through a wired connection or a wireless connection. First device 100b (e.g., a base station device) includes processor(s) <NUM>, RF circuitry(ies) <NUM>, and memory(ies) <NUM>. These components optionally communicate over communication bus(es) <NUM> of device 100b. Second device 100c (e.g., a head-mounted device) includes various components, such as processor(s) <NUM>, RF circuitry(ies) <NUM>, memory(ies) <NUM>, image sensor(s) <NUM>, orientation sensor(s) <NUM>, microphone(s) <NUM>, location sensor(s) <NUM>, speaker(s) <NUM>, display(s) <NUM>, and touch-sensitive surface(s) <NUM>. These components optionally communicate over communication bus(es) <NUM> of device 100c.

In some examples, system <NUM> is a mobile device. In some examples, system <NUM> is a head-mounted display (HMD) device. In some examples, system <NUM> is a wearable HUD device.

System <NUM> includes processor(s) <NUM> and memory(ies) <NUM>. Processor(s) <NUM> include one or more general processors, one or more graphics processors, and/or one or more digital signal processors. In some examples, memory(ies) <NUM> are one or more non-transitory computer-readable storage mediums (e.g., flash memory, random access memory) that store computer-readable instructions configured to be executed by processor(s) <NUM> to perform the techniques described below.

System <NUM> includes RF circuitry(ies) <NUM>. RF circuitry(ies) <NUM> optionally include circuitry for communicating with electronic devices, networks, such as the Internet, intranets, and/or a wireless network, such as cellular networks and wireless local area networks (LANs). RF circuitry(ies) <NUM> optionally includes circuitry for communicating using near-field communication and/or short-range communication, such as Bluetooth®.

System <NUM> includes display(s) <NUM>. In some examples, display(s) <NUM> include a first display (e.g., a left eye display panel) and a second display (e.g., a right eye display panel), each display for displaying images to a respective eye of the user. Corresponding images are simultaneously displayed on the first display and the second display. Optionally, the corresponding images include the same virtual objects and/or representations of the same physical objects from different viewpoints, resulting in a parallax effect that provides a user with the illusion of depth of the objects on the displays. In some examples, display(s) <NUM> include a single display. Corresponding images are simultaneously displayed on a first area and a second area of the single display for each eye of the user. Optionally, the corresponding images include the same virtual objects and/or representations of the same physical objects from different viewpoints, resulting in a parallax effect that provides a user with the illusion of depth of the objects on the single display.

In some examples, system <NUM> includes touch-sensitive surface(s) <NUM> for receiving user inputs, such as tap inputs and swipe inputs. In some examples, display(s) <NUM> and touch-sensitive surface(s) <NUM> form touch-sensitive display(s).

System <NUM> includes image sensor(s) <NUM>. Image sensors(s) <NUM> optionally include one or more visible light image sensor, such as charged coupled device (CCD) sensors, and/or complementary metal-oxide-semiconductor (CMOS) sensors operable to obtain images of physical objects from the real environment. Image sensor(s) also optionally include one or more infrared (IR) sensor(s), such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real environment. For example, an active IR sensor includes an IR emitter, such as an IR dot emitter, for emitting infrared light into the real environment. Image sensor(s) <NUM> also optionally include one or more event camera(s) configured to capture movement of physical objects in the real environment. Image sensor(s) <NUM> also optionally include one or more depth sensor(s) configured to detect the distance of physical objects from system <NUM>. In some examples, system <NUM> uses CCD sensors, event cameras, and depth sensors in combination to detect the physical environment around system <NUM>. In some examples, image sensor(s) <NUM> include a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical objects in the real environment from two distinct perspectives. In some examples, system <NUM> uses image sensor(s) <NUM> to receive user inputs, such as hand gestures. In some examples, system <NUM> uses image sensor(s) <NUM> to detect the position and orientation of system <NUM> and/or display(s) <NUM> in the real environment. For example, system <NUM> uses image sensor(s) <NUM> to track the position and orientation of display(s) <NUM> relative to one or more fixed objects in the real environment.

In some examples, system <NUM> includes microphones(s) <NUM>. System <NUM> uses microphone(s) <NUM> to detect sound from the user and/or the real environment of the user. In some examples, microphone(s) <NUM> includes an array of microphones (including a plurality of microphones) that optionally operate in tandem, such as to identify ambient noise or to locate the source of sound in space of the real environment.

System <NUM> includes orientation sensor(s) <NUM> for detecting orientation and/or movement of system <NUM> and/or display(s) <NUM>. For example, system <NUM> uses orientation sensor(s) <NUM> to track changes in the position and/or orientation of system <NUM> and/or display(s) <NUM>, such as with respect to physical objects in the real environment. Orientation sensor(s) <NUM> optionally include one or more gyroscopes and/or one or more accelerometers.

<FIG> depicts exemplary digital assistant <NUM> for determining a response to user requests. In some examples, as illustrated in <FIG>, digital assistant <NUM> includes input analyzer <NUM>, sensor interface <NUM>, and output generator <NUM>. In some examples, digital assistant <NUM> may optionally include a reference resolution module, as discussed further below. In some examples, digital assistant <NUM> is implemented on electronic device <NUM>. In some examples, digital assistant <NUM> is implemented across other devices (e.g., a server) in addition to electronic device <NUM>. In some examples, some of the modules and functions of the digital assistant are divided into a server portion and a client portion, where the client portion resides on one or more user devices (e.g., electronic device <NUM>) and communicates with the server portion through one or more networks.

It should be noted that digital assistant <NUM> is only one example of a digital assistant, and that digital assistant <NUM> can have more or fewer components than shown, can combine two or more components, or can have a different configuration or arrangement of the components. The various components shown in <FIG> are implemented in hardware, software instructions for execution by one or more processors, firmware, including one or more signal processing and/or application specific integrated circuits, or a combination thereof. In some examples, digital assistant <NUM> connects to one or more components and/or sensors of electronic device <NUM> as discussed further below.

Digital assistant <NUM> receives spoken input <NUM> including a request from a user and provides spoken input <NUM> to input analyzer <NUM>. After receiving spoken input <NUM>, input analyzer <NUM> performs a semantic analysis on spoken input <NUM>. In some examples, performing the semantic analysis includes performing automatic speech recognition (ASR) on spoken input <NUM>. In particular, input analyzer <NUM> can include one or more ASR systems that process spoken input <NUM> received through input devices (e.g., a microphone) of electronic device <NUM>. The ASR systems extract representative features from the speech input. For example, the ASR systems pre-processor performs a Fourier transform on the spoken input <NUM> to extract spectral features that characterize the speech input as a sequence of representative multi-dimensional vectors.

Further, each ASR system of input analyzer <NUM> includes one or more speech recognition models (e.g., acoustic models and/or language models) and implements one or more speech recognition engines. Examples of speech recognition models include Hidden Markov Models, Gaussian-Mixture Models, Deep Neural Network Models, n-gram language models, and other statistical models. Examples of speech recognition engines include the dynamic time warping based engines and weighted finite-state transducers (WFST) based engines. The one or more speech recognition models and the one or more speech recognition engines are used to process the extracted representative features of the front-end speech pre-processor to produce intermediate recognition results (e.g., phonemes, phonemic strings, and sub-words), and ultimately, text recognition results (e.g., words, word strings, or sequence of tokens).

In some examples, performing semantic analysis includes performing natural language processing on spoken input <NUM>. In particular, once input analyzer <NUM> produces recognition results containing a text string (e.g., words, or sequence of words, or sequence of tokens) through ASR, input analyzer <NUM> may deduce an intent of spoken input <NUM>. In some examples, input analyzer <NUM> produces multiple candidate text representations of the speech input. Each candidate text representation is a sequence of words or tokens corresponding to spoken input <NUM>. In some examples, each candidate text representation is associated with a speech recognition confidence score. Based on the speech recognition confidence scores, input analyzer <NUM> ranks the candidate text representations and provides the n-best (e.g., n highest ranked) candidate text representation(s) to other modules of digital assistant <NUM> for further processing.

In some examples, performing the semantic analysis includes determining whether the request of spoken input <NUM> includes an ambiguous term. In some examples, the ambiguous term is a deictic reference. A deictic reference is a word or phrase that ambiguously references something like an object, time, person, or place. Exemplary deictic references include but are not limited to "that," "this," "here," "there," "then," "those," "them," "he," "she," etc. particularly when used with a question such as the questions "what is this?," "where is that?," and "who is he?" Accordingly, input analyzer <NUM> determines whether the request includes one of these words or words like them and thus, whether the use of the word is ambiguous. For example, in the spoken input "what is that?" input analyzer <NUM> may determine that "that" is a deictic reference through ASR and/or NLP. Similarly, in spoken input <NUM> "when was this built?" input analyzer <NUM> determines that "this" is a deictic reference. In both examples, input analyzer <NUM> may determine "that" and "this" to be ambiguous because the user input does not include a subject or object that could be referred to with "that" or "this.

After performing the semantic analysis, input analyzer <NUM> determines a likelihood that additional contextual data is required to satisfy the request. In accordance with the invention, the likelihood that additional contextual data is required to satisfy the request is based on movement of electronic device <NUM> during receipt of spoken input <NUM>. For example, when electronic device <NUM> is a head mounted device, the user may move their head and thus electronic device <NUM> while providing the word "that" of spoken input <NUM>. Accordingly, input analyzer <NUM> may determine that the user was indicating a possible object with the reference "that" because electronic device <NUM> moved near the same time the user provided "that" in spoken input <NUM>. Input analyzer may then determine a high likelihood that additional contextual data is required to satisfy the request because of the ambiguous reference "that" and the movement provided at the same time, indicating an object.

It should be understood that gestures or other information detected near the same time as words provided in spoken input <NUM> may be detected at the same time as the words in spoken input <NUM> or at substantially the same time as the words in spoken input <NUM>. For example, the gestures and other information discussed below may be received at the same as the spoken input <NUM>, a short time before spoken input <NUM> (e.g., <NUM> seconds, <NUM> second, <NUM> milliseconds, <NUM> milliseconds, etc.) or a short time after spoken input <NUM> (e.g., <NUM> seconds, <NUM> second, <NUM> milliseconds, <NUM> milliseconds, etc.).

As another example, when electronic device <NUM> is a handheld electronic device such as a smart phone, the user may gesture with electronic device <NUM> by moving electronic device <NUM> towards an object while providing the word "that" of spoken input <NUM>. Accordingly, similar to the example above, input analyzer <NUM> may determine that the user was indicating a possible object with the reference "that" because electronic device <NUM> moved towards an object near the same time as the user provided "that" in spoken input <NUM>. Input analyzer may then determine a high likelihood that additional contextual data is required to satisfy the request because of the ambiguous reference "that" and the movement.

In some examples, when electronic device <NUM> is a handheld electronic device such as a smart phone, the user may gesture towards a screen of electronic device <NUM> (e.g., pointing at a portion of the screen) or on a screen of electronic device (e.g., tapping a portion of the screen) while providing "that" of spoken input <NUM>. Accordingly, input analyzer <NUM> may determine that the user was indicating a possible object with the reference "that" because electronic device <NUM> detected a gesture towards or on a screen of electronic device <NUM> near the same time as the user provided "that" in spoken input <NUM>. For example, the screen of electronic device <NUM> may be displaying multiple landmarks and the user may point at one while saying "that," and thus, input analyzer <NUM> may determine that the user is gesturing towards the one object and thus intends to reference that object. Input analyzer <NUM> may then determine a high likelihood that additional contextual data is required to satisfy the request because of the ambiguous reference "that" and the movement towards or on the screen of electronic device <NUM>.

In some examples, the likelihood that additional contextual data is required is based on whether movement of electronic device <NUM> ceases during receipt of spoken input <NUM>. For example, while receiving the spoken input "what is that over there?" electronic device <NUM> may stop moving (e.g., linger) for a brief time while the user provides "that" of spoken input <NUM>. Accordingly, input analyzer <NUM> may determine that the user was indicating a possible object with the reference "that" because electronic device <NUM> stopped moving near the same time as "that" was uttered in spoken input <NUM>. Input analyzer may then determine a high likelihood that additional contextual data is required to satisfy the request because of the ambiguous reference "that" and the ceasing of movement of electronic device <NUM>.

In contrast, while receiving the spoken input "what is that over there?" electronic device <NUM> may continuously move because, for example, the user is scanning the horizon while providing spoken input <NUM>. Accordingly, input analyzer <NUM> may determine that the movement or ceasing of movement did not indicate any potential object the user is referencing and thus determine a low likelihood that additional contextual data is required to satisfy the request.

In some examples, the likelihood that additional contextual data is required is based on movement of electronic device <NUM> for a predetermined time after receiving spoken input <NUM>. Thus, as discussed above with reference to movement or ceasing of movement detected during receipt of spoken input <NUM>, input analyzer <NUM> may determine whether electronic device <NUM> moves during a predetermined time (e.g., <NUM> second, <NUM> seconds, <NUM> seconds, <NUM> seconds, etc.) after receiving spoken input <NUM>. If electronic device <NUM> moves during that predetermined time, input analyzer <NUM> may determine that the movement was indicating an object and thus determine a high likelihood that additional contextual data is required.

In some examples, determining whether movement of electronic device <NUM> ceases includes determining whether movement of electronic device <NUM> is below a threshold for a predetermined time. The movement threshold includes six inches of movement, a foot of movement, two feet of movement, or any other amount of movement useful for determining whether the user intends to move electronic device <NUM>. The predetermined time includes one second, five seconds, ten seconds, etc. For example, while electronic device <NUM> receives spoken input <NUM>, electronic device <NUM> may detect small movements indicative of the normal movements a user makes when not intending to provide a gesture or any other meaningful movement of electronic device <NUM>. Thus, the movements may be less than the threshold of one foot of movement for five seconds. Accordingly, input analyzer <NUM> may determine that electronic device <NUM> has ceased moving because the movement is below the threshold for the predetermined time.

In some examples, the likelihood that additional contextual data is required is based on a field of view of electronic device <NUM> near in time to receiving spoken input <NUM>. In particular, the user may change the field of view of electronic device <NUM> by moving from looking at something close by to looking at something far away and near the same time provide the spoken input "what is that?". For example, electronic device <NUM> may be receiving a field of view of a tree and the user may glance behind the tree at a tower while providing the spoken input "what is that?". Accordingly, input analyzer <NUM> may determine that the user was indicating the tower with the reference "that" because electronic device <NUM> detected that the field of view of electronic device <NUM> changed from the tree to the tower near the same time as the user provided "that" in spoken input <NUM>.

In some examples, the likelihood that additional contextual data is required is based on a pose of electronic device <NUM> after receiving spoken input <NUM>. For example, after receiving spoken input <NUM> of "what is in that direction?" input analyzer <NUM> may determine that electronic device <NUM> is rotated in a pose pointing a new direction. Accordingly, input analyzer <NUM> may determine a high likelihood that additional contextual data that would indicate the direction is required to help determine a response to spoken input <NUM>.

In some examples, the likelihood that additional contextual data is required is based on a detected gaze of the user during receipt of spoken input <NUM>. In some examples, digital assistant <NUM> detects the gaze of the user based on movement or orientation of electronic device <NUM>. For example, when electronic device <NUM> is a wearable device like a head mounted display, the view of electronic device <NUM> is also the view of a user wearing electronic device <NUM>. Thus, digital assistant <NUM> may determine the user gaze associated with spoken input <NUM> to be the direction that electronic device <NUM> is facing or is oriented towards. Accordingly, digital assistant <NUM> may determine that the user is looking in a specific direction and thus input analyzer <NUM> may determine a high likelihood that additional contextual data is required.

In some examples, digital assistant <NUM> detects the user gaze based on a front facing camera or other sensor of electronic device <NUM>. Thus, when electronic device <NUM> is a phone, the user may look at the display of electronic device <NUM>. Accordingly, electronic device <NUM> may receive an image of the user's face with a front facing camera and based on this image digital assistant <NUM> can determine where the user is looking while providing spoken input <NUM>, thus determining a user gaze associated with spoken input <NUM>. Accordingly, digital assistant <NUM> may determine that the user gaze is looking at a specific point on the display and thus input analyzer <NUM> may determine a low likelihood that additional contextual data is required. Conversely, input digital assistant <NUM> may determine that the user gaze is looking away from the display and thus input analyzer <NUM> may determine a high likelihood that additional contextual data is required because the user is likely referencing something not being displayed.

In some examples, the likelihood that additional contextual data is required is based on a location of electronic device <NUM> during or after receiving spoken input <NUM>. For example, when digital assistant <NUM> receives the spoken input <NUM> "what is that?" digital assistant <NUM> may also receive data indicating that the user is near several landmarks such as the Brooklyn Bridge and the Statue of Liberty. Accordingly, input analyzer <NUM> may determine that because the user is near several landmarks and has provided spoken input <NUM> of "what is that?" there is a high likelihood that additional contextual data is required to determine which landmark the user intended with "that. " Conversely, digital assistant <NUM> may receive the same spoken input <NUM> "what is that?" while the user is standing directly next to the Eiffel Tower. Accordingly, input analyzer <NUM> may determine that because the user is directly next to the Eiffel Tower (and possibly facing or gesturing towards it as described above) there is a low likelihood that additional contextual data is required to satisfy the user's request.

In some examples, the likelihood that additional contextual data is required is based on historical interaction data between digital assistant <NUM> and a user. For example, digital assistant <NUM> may receive spoken input <NUM> of "what do they eat?" after providing an output of "that animal is an opossum. " Accordingly, input analyzer <NUM> may then determine that because user input <NUM> includes "they" spoken input <NUM> likely references the recent exchange between digital assistant <NUM> and the user related to opossums. Thus, input analyzer <NUM> may determine there is a low likelihood that additional context is required to satisfy the user's request.

In some examples, the likelihood that additional contextual data is required is based on whether a virtual reality mode or an augmented reality mode of electronic device <NUM> is active. In some examples, digital assistant <NUM> determines whether a virtual reality mode or an augmented reality mode of electronic device <NUM> is active based on whether one or more virtual reality or augmented reality objects are being generated and/or displayed. For example, digital assistant <NUM> may determine that a virtual reality object such as an airplane is being generated and displayed to the user and thus that a virtual reality mode is active. Accordingly, when digital assistant <NUM> receives spoken input <NUM> "who makes this?" input analyzer <NUM> may determine that spoken input <NUM> is likely referencing the virtual reality airplane and thus determine there is a low likelihood that additional contextual data is required to satisfy the user's request.

Conversely, in some examples, the likelihood that additional contextual data is required is not based on whether a virtual reality mode of electronic device <NUM> is active and instead on one of the other factors described herein. For example, digital assistant <NUM> may determine that electronic device <NUM> is generating a virtual reality environment including several paintings. Digital assistant <NUM> may then receive spoken input <NUM> "who painted that one?". Thus, input analyzer <NUM> may determine that "that one" of spoken input <NUM> is ambiguous because it is unclear which of the virtual paintings the user is referencing. Accordingly, input analyzer <NUM> may determine a high likelihood that additional contextual data is required to satisfy the user's request. Additionally, in some examples, input analyzer <NUM> may detect a gesture of the user, a gaze of the user, etc. to further inform the likelihood that additional contextual data is required, as described above.

It will be understood that the factors described above used to determine a likelihood that additional contextual data is required to satisfy the request of spoken input <NUM> may be examined in combinations of one or more by input analyzer <NUM> to determine the likelihood based on the data available from electronic device <NUM> at one time.

After determining the likelihood that additional contextual data is required, input analyzer <NUM> determines if the likelihood that additional contextual data is required exceeds a predetermined threshold. In some examples, the predetermined threshold is a threshold that indicates that the likelihood that additional contextual data is required is sufficiently high that digital assistant <NUM> should acquire additional data in order to determine how to respond to a user's request. The predetermined threshold may be any number indicative of this importance including, for example, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, etc..

In some examples, the predetermined threshold may be adjusted as more requests are received and successfully answered by digital assistant <NUM> based on whether the additional contextual data was necessary to respond to the user's request of spoken input <NUM>. For example, if input analyzer <NUM> determines that additional contextual data is required because the likelihood that additional contextual data is required is over a predetermined threshold of <NUM> and the contextual data <NUM> received by one or more sensors was not necessary to respond to the user's request, the predetermined threshold may be increased to, for example, <NUM>. Similarly, if input analyzer <NUM> determines that additional contextual data is not required because the likelihood that additional contextual data is required is below a predetermined threshold of <NUM> and digital assistant <NUM> later determines that contextual data <NUM> received by one of the sensors was necessary to respond to the user's request, the predetermined threshold may be decreased to, for example, <NUM>.

If the likelihood that additional contextual data is required exceeds a predetermined threshold, digital assistant <NUM> causes sensor interface <NUM> to enable one or more sensors of electronic device <NUM> such as image sensors <NUM>, orientation sensors <NUM>, and location sensors <NUM>. The sensors of electronic device <NUM> includes one or more cameras (e.g., image sensors <NUM>), gyroscopes (e.g., orientation sensors <NUM>), accelerometers (e.g., orientation sensors <NUM>), altimeters (e.g., orientation sensors <NUM>), GPS sensors (e.g., location sensors <NUM>), and network detectors (e.g., location sensors <NUM>). Accordingly, when digital assistant <NUM> determines that the likelihood exceeds the predetermined threshold, digital assistant <NUM> causes sensor interface <NUM> to enable one of the camera, gyroscope, accelerometer, altimeter, GPS sensor, or network detectors of electronic device <NUM>. Thus, the contextual data received includes a picture, a video stream, acceleration data, altitude data, GPS data, network data, rotation data, speed data, etc..

In some examples, before or after the likelihood determination described above, input analyzer <NUM> may determine whether one or more ambiguous terms or deictic references of spoken input <NUM> references an object. For example, as shown in <FIG>, electronic device <NUM> and digital assistant <NUM> may receive spoken input <NUM> "what animal is that?" and input analyzer <NUM> may determine that the likelihood that additional contextual data is required to satisfy the request of spoken input <NUM> is over a predetermined threshold as described above. Accordingly, digital assistant <NUM> enables a camera (e.g., image sensor <NUM>) of electronic device <NUM> to receive picture <NUM> as contextual data <NUM>. Input analyzer <NUM> may then determine which of objects <NUM> and <NUM> the deictic reference of spoken input <NUM> references. In particular, input analyzer <NUM> may determine based on the use of "animal" in spoken input <NUM>, that spoken input <NUM> is likely referencing object <NUM> which is an opossum of picture <NUM>. In some examples, the determination of whether one or more ambiguous terms or deictic references of spoken input <NUM> references an object may be performed by a different module of digital assistant <NUM> or by input analyzer <NUM> and one or more other modules of digital assistant <NUM>.

In some examples, determining whether one or more ambiguous terms or deictic references of spoken input <NUM> references an object includes performing image processing techniques such as edge detection, edge extraction, optical character recognition, image segmentation, texture analysis, motion analysis, etc. These image processing techniques can be performed through the use of machine learning models, neural networks, deep learning networks, or any other acceptable image processing software and/or module. These image processing techniques may allow input analyzer <NUM> to determine objects in the image, text in the image, edges in the image, etc..

In some examples, the sensor of electronic device <NUM> automatically receives contextual data <NUM> when the sensor is launched by sensor interface <NUM>. For example, digital assistant <NUM> may cause sensor interface <NUM> to enable a camera of electronic device <NUM> when electronic device <NUM> is a head mounted device. Accordingly, the camera of electronic device <NUM> may automatically receive contextual data <NUM> of a picture of the user/electronic device <NUM>'s view. As another example, digital assistant <NUM> may cause sensor interface <NUM> to enable a camera of electronic device <NUM> when electronic device <NUM> is a smart phone. Accordingly, the camera of electronic device <NUM> may automatically receive contextual data <NUM> of a picture of the view of electronic device <NUM>, which is distinct from the view of the user in this example. As yet another example, digital assistant <NUM> may cause sensor interface <NUM> to enable an accelerometer of electronic device <NUM>. Accordingly, the accelerometer of electronic device <NUM> may automatically receive contextual data <NUM> of a speed of electronic device <NUM>.

In some examples, the sensor of electronic device <NUM> is launched in the background. Accordingly, a user interface for the sensor or any other indication that the sensor is launched is not provided to a user of electronic device <NUM>. As described above, digital assistant <NUM> may cause sensor interface <NUM> to enable a camera of electronic device <NUM> and thus, the camera of electronic device <NUM> may receive contextual data <NUM> of a picture in the background, without providing any user interface or other indication. This may also occur with a gyroscope, accelerometer, altimeter, GPS sensor, network detectors, or other sensor of electronic device <NUM>.

In some examples, launching the sensor of electronic device <NUM> includes displaying a user interface associated with the sensor on a display of electronic device <NUM>, as shown in <FIG>. For example, when a camera of electronic device <NUM> is launched by sensor interface <NUM>, electronic device <NUM> may display user interface <NUM> associated with the camera on display <NUM> of electronic device <NUM>. In some examples, displaying the user interface associated with the sensor includes displaying one or more affordances associated with receiving contextual data <NUM> on the user interface associated with the sensor. For example, as shown in <FIG>, user interface <NUM> includes affordance <NUM> associated with taking picture <NUM> including objects <NUM> and <NUM>. In some examples, affordance <NUM> or another affordance of user interface <NUM> may be associated with taking a video stream in addition to or instead of picture <NUM>.

In some examples, the sensor of electronic device <NUM> receives contextual data when prompted by a user. In some examples, prompting the sensor of electronic device <NUM> to receive contextual data includes selecting a button of electronic device <NUM>. In some examples, prompting the sensor of electronic device <NUM> to receive contextual data includes selecting an affordance of the user interface associated with the sensor. Continuing the example discussed above, when electronic device <NUM> displays user interface <NUM> associated with the camera, affordance <NUM> for taking a picture is displayed in user interface <NUM>. Accordingly, the user may select affordance <NUM> for taking a picture thus prompting the camera of electronic device <NUM> to receive contextual data <NUM> of picture <NUM>. Thus, the camera of electronic device <NUM> receives contextual data <NUM> (e.g., picture <NUM>) when prompted by the user.

As another example, when an accelerometer of electronic device <NUM> is launched by sensor interface <NUM>, electronic device <NUM> may display a user interface associated with the accelerometer which may include an affordance for taking a speed of electronic device <NUM>. Accordingly, the user may select the affordance and prompt the accelerometer of electronic device <NUM> to receive contextual data <NUM> of the speed of electronic device <NUM>. Thus, the accelerometer of electronic device <NUM> receives contextual data <NUM> of the speed of electronic device <NUM> when prompted by the user.

In some examples, the input is a voice input confirming the prompt. For example, digital assistant <NUM> may provide a spoken output of "Would you like to take a picture?" as a prompt to the user. Accordingly, the user may respond with "Yes" to confirm that the camera of electronic device <NUM> should receive contextual data <NUM> of a picture or "No" to stop the camera of electronic device <NUM> from receiving contextual data <NUM> of the picture.

In some examples, the user interface associated with the sensor is displayed in another user interface associated with digital assistant <NUM>. For example, as shown in <FIG>, electronic device <NUM> may display user interface <NUM> including the camera interface inside a user interface associated with the digital assistant <NUM> on display <NUM>. In this way digital assistant <NUM> can preserve continuity during the interaction between the user and digital assistant <NUM>. Thus, contextual data <NUM> of picture <NUM> may also be displayed in user interface <NUM>, providing contextual data <NUM> to the user.

In some examples, the user interface associated with the sensor belongs to an application associated with the sensor. In some examples, the application is a first party application. For example, electronic device <NUM> may have a camera application that is pre-installed. Accordingly, the user interface associated with the camera may belong to the camera application that is pre-installed. In some examples, the application is a third party application. For example, as an alternative or in addition to the first party camera application electronic device <NUM> may also have a third party camera application installed. Accordingly, the user interface associated with the camera may belong to the third party camera application.

In some examples, digital assistant <NUM> provides a prompt confirming that contextual data <NUM> should be received. Digital assistant <NUM> further receives an input confirming that contextual data <NUM> should be received or stopping contextual data <NUM> from being received. In some examples, the input is a selection of a button of electronic device <NUM> or an affordance of a user interface. For example, as shown in <FIG>, digital assistant <NUM> may provide prompt <NUM> asking "Would you like to take a picture?" in user interface <NUM> on display <NUM> of electronic device <NUM>. Prompt <NUM> may further include affordances <NUM> and <NUM> including "yes" and "no," respectively, as options for the user to select. The user may provide an input selecting one of affordances <NUM> and <NUM> to confirm that picture <NUM> should be taken or to stop the taking of picture <NUM>. Accordingly, if the user selects affordance <NUM> including "yes" electronic device <NUM> receives picture <NUM>. Conversely, if the user selects affordance <NUM> including "no" electronic device <NUM> does not receive picture <NUM>.

After receiving the contextual data, digital assistant <NUM> provides the spoken input <NUM> and contextual data <NUM> to response generator <NUM> and response generator <NUM> determines response <NUM> to the request based on contextual data <NUM>. For example, in response to spoken input <NUM> "what is that animal?" response generator <NUM> may provide response <NUM> "that animal is an opossum," after conducting a search based on contextual data <NUM> of a picture including the opossum, as described below. As another example, in response to spoken input "where is this?" response generator <NUM> may provide response <NUM> "Paris, France" after conducting a search based on contextual data <NUM> of GPS coordinates of electronic device <NUM>.

In some examples, response generator <NUM> determines response <NUM> by performing a search based on contextual data <NUM>. In some examples, the search includes a search of one or more databases of electronic device <NUM> or connected electronic devices (e.g., servers). In some examples, the search includes a search on the internet, using a search engine, a website, or similar tools. In some examples, the search includes using an image classifier, object detector, or other neural network or machine learning model to process contextual data <NUM> for additional information. For example, when contextual data <NUM> is a picture including an animal, response generator <NUM> may perform a search with the animal of the picture in local databases and on the internet to determine response <NUM> to the request of spoken input <NUM>. Accordingly, response generator <NUM> may determine based on image classifier and database search results of the animal of the picture that the animal is an opossum and generate the response <NUM> "that animal is an opossum.

In some examples, the search is based on other data in addition to contextual data <NUM>. For example, when contextual data <NUM> is a picture of the Eiffel Tower, response generator <NUM> may perform a search for the picture along with location data (e.g., GPS coordinates) of electronic device <NUM> to inform the search results based on the picture. Accordingly, response generator <NUM> may determine that the search results indicating pictures similar to contextual data <NUM> and the determined location data are likely the correct response to the user's request. Thus, response generator <NUM> may generate the response <NUM> "Paris, France," to provide to the user.

In some examples, response generator <NUM> generates response <NUM> including the results of the search performed based on contextual data <NUM>. For example, when response generator <NUM> generates response <NUM> "that animal is an opossum," response generator <NUM> may also include results of the internet search for the animal in response <NUM>. Accordingly, response <NUM> may include hyperlinks to websites that provide information about opossums or other references or information the user may find helpful in answering their request.

Digital assistant <NUM> then provides response <NUM> to the request. In some examples, the response to the request is provided as an audio output. For example, digital assistant <NUM> may provide response <NUM> "that animal is an opossum," as an audio output. In some examples, the response to the request is provided on a display of electronic device <NUM>. For example, digital assistant <NUM> may provide response <NUM> "that animal is an opossum," on a display of electronic device <NUM>. In some examples, digital assistant <NUM> provides the response on a display of electronic device <NUM> and echoes the displayed response as an audio output. Accordingly, digital assistant <NUM> may both display response <NUM> "that animal is an opossum," on a display of electronic device <NUM> while providing response <NUM> "that animal is an opossum," as an audio output.

In some examples, digital assistant <NUM> provides other information in addition to response <NUM> on a display of electronic device <NUM> while providing response <NUM> as an audio output. For example, digital assistant <NUM> may provide response <NUM> "that animal is an opossum," as an audio output and then provide on a display of electronic device <NUM> the same response in addition to the search results related to the opossum including one or more facts, hyperlinks, or other information that may be helpful to the user.

In some examples, after providing the response to the request digital assistant <NUM> stores (e.g., saves) contextual data <NUM>. For example, after providing response <NUM> "that animal is an opossum," digital assistant <NUM> may store or save contextual data <NUM> of the picture including the opossum for future reference by digital assistant <NUM> and/or the user. In some examples, digital assistant <NUM> stores the results of the search and/or the response in addition to contextual data <NUM>. Continuing the example above, digital assistant <NUM> may also store or save the search results related to the opossum for further reference by digital assistant <NUM> and/or the user to answer further requests or provide further information.

In some examples, digital assistant <NUM> discards (e.g., deletes) contextual data <NUM>. For example, after providing the response digital assistant <NUM> may determine that contextual data <NUM> was unhelpful or is unnecessary for further responses and may thus discard or delete contextual data <NUM>.

Based on the disclosure above, it will be understood that the methods and structure described allow a digital assistant and an electronic device to use one or more sensors to determine if additional sensors should be enabled and data received to respond to one or more requests provided by a user. In this way, sensors may be selectively activated as required, reducing the processing needed at one time and conserving battery.

<FIG> is a flow diagram illustrating a process for determining a response to a request, according to various examples. Method <NUM> is performed at a device (e.g., device <NUM>, <NUM>) with one or more input devices (e.g., a touchscreen, a mic, a camera), and a wireless communication radio (e.g., a Bluetooth connection, WiFi connection, a mobile broadband connection such as a <NUM> LTE connection). In some embodiments, the electronic device includes a plurality of cameras. In some embodiments, the electronic device includes only one camera. In some examples, the device includes one or more biometric sensors which, optionally, include a camera, such as an infrared camera, a thermographic camera, or a combination thereof. Some operations in method <NUM> are, optionally, combined, the orders of some operations are, optionally, changed, and some operations are, optionally, omitted.

At block <NUM>, a spoken input (e.g., spoken input <NUM>) including a request is received. In some examples, the request includes an ambiguous term.

At block <NUM>, a semantic analysis on the spoken input (e.g., spoken input <NUM>) is performed. In some examples, performing the semantic analysis on the spoken input further comprises determining whether the request includes an ambiguous term. In some examples, in accordance with a determination that the request includes the ambiguous term, whether the ambiguous term references an object (e.g., object <NUM>, object <NUM>) is determined.

At block <NUM>, a likelihood that additional contextual data (e.g., contextual data <NUM>, picture <NUM>) is required to satisfy the request is determined based on the semantic analysis. In accordance with the invention, determining, based on the semantic analysis, the likelihood that additional contextual data is required to satisfy the request further comprises determining movement of the electronic device (e.g., electronic device <NUM>, electronic device <NUM>) during the reception of the spoken input (e.g., spoken input <NUM>). In some examples, determining, based on the semantic analysis, the likelihood that additional contextual data is required to satisfy the request further comprises determining movement of the electronic device is below a threshold for a predetermined time after receiving the spoken input.

In some examples, determining, based on the semantic analysis, the likelihood that additional contextual data (e.g., contextual data <NUM>, picture <NUM>) is required to satisfy the request further comprises determining a pose of the electronic device (e.g., electronic device <NUM>, electronic device <NUM>) after receiving the spoken input (e.g., spoken input <NUM>). In some examples, determining, based on the semantic analysis, the likelihood that additional contextual data is required to satisfy the request further comprises determining a gaze of a user while receiving the spoken input. In some examples, determining, based on the semantic analysis, the likelihood that additional contextual data is required to satisfy the request further comprises determining a location of the electronic device after receiving the spoken input.

At block <NUM>, in accordance with the determined likelihood exceeding a threshold a camera (e.g., image sensor <NUM>) of the electronic device (e.g., electronic device <NUM>, electronic device <NUM>) is enabled. In some examples, the camera of the electronic device is enabled in the background. In some examples, a picture (e.g., picture <NUM>) is taken with the camera of the electronic device.

In some examples, a user interface (e.g., user interface <NUM>, user interface <NUM>) associated with the camera (e.g., image sensor <NUM>) of the electronic device (e.g., electronic device <NUM>, electronic device <NUM>) is displayed. In some examples, the user interface associated with the camera of the electronic device belongs to a camera application. In some examples, a prompt (e.g., prompt <NUM>) confirming that the picture (e.g., picture <NUM>) should be taken is provided. In some examples, a user input confirming that the picture should be taken is received. In some examples, a picture is taken with the camera of the electronic device.

At block <NUM>, a response (e.g., response <NUM>) to the request is determined based on the contextual data (e.g., contextual data <NUM>, picture <NUM>) received by the camera (e.g., image sensor <NUM>) of the electronic device (e.g., electronic device <NUM>, electronic device <NUM>). In some examples, determining the response to the request based on contextual data received by the camera of the electronic device further comprises performing a search based on the data received by the camera and providing the response to the request based on the results of the search. In some examples, the search is based on other contextual data in addition to the contextual data received by the camera. In some examples, the contextual data received by the camera is saved. In some examples, after providing the response to the request, the contextual data received by the camera is discarded.

As described above, one aspect of the present technology is the gathering and use of data available from various sources to reference and object determination of a request. The present disclosure contemplates that in some instances, this gathered data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data can include demographic data, location-based data, telephone numbers, email addresses, twitter IDs, home addresses, data or records relating to a user's health or level of fitness (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used to deliver accurate responses to requests that are of greater interest to the user. Accordingly, use of such personal information data enables users calculated control of response resolution. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure. For instance, health and fitness data may be used to provide insights into a user's general wellness, or may be used as positive feedback to individuals using technology to pursue wellness goals.

In particular, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data.

Despite the foregoing, the present disclosure also contemplates examples in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and/or software elements can be provided to prevent or block access to such personal information data. For example, in the case of enabling sensors, the present technology can be configured to allow users to select to "opt in" or "opt out" of participation in the collection of personal information data during registration for services or anytime thereafter. In another example, users can select to limit the length of time captured data and/or requests are maintained or entirely prohibit the development of saving the data or requests. In addition to providing "opt in" and "opt out" options, the present disclosure contemplates providing notifications relating to the access or use of personal information. For instance, a user may be notified upon downloading an app that their personal information data will be accessed and then reminded again just before personal information data is accessed by the app.

De-identification may be facilitated, when appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of data stored (e.g., collecting location data at city level rather than at an address level), controlling how data is stored (e.g., aggregating data across users), and/or other methods.

Claim 1:
A method, comprising:
at an electronic device with one or more processors and memory:
receiving a spoken input including a request;
performing a semantic analysis on the spoken input;
determining, based on the semantic analysis , a likelihood that additional contextual data is required to satisfy the request;
wherein determining, based on the semantic analysis, the likelihood that additional contextual data is required to satisfy the request comprises determining movement of the electronic device during the reception of the spoken input,
in accordance with the determined likelihood exceeding a threshold:
enabling a camera of the electronic device; and
determining a response to the request based on contextual data received by the camera of the electronic device.