Patent Description:
Intelligent personal assistants are becoming commonplace in today's homes. Products such as Amazon's Echo, Google's Google Home, and Apple's Siri are all examples of these assistants. Typically, such assistants are installed at home, coupled to an existing home Wi-Fi network and placed in a convenient location where they may be used most frequently, such as in a family room or kitchen.

Generally, each device listens for a wake word to be spoken, such as "Alexa" for Amazon's Echo and "OK Google" for Google's Home device, followed by a question or a command. The question is typically sent over the Internet to a voice recognition server that interprets the question or command and provides a response that is sent back over the internet for verbal playback via a speaker that is typically integrated into each device, or causing some action to occur, such as lighting lights, playing music, etc..

One of the drawbacks of these assistants is that the wake word must be spoken each time a new question or command is asked by a user. This can become annoying, especially when a series of questions or commands are asked in succession.

It would be desirable to eliminate the need to utter the wake word every time a question is posed to such assistants.

<CIT> is prior art under Article <NUM>(<NUM>) EPC. It discusses a more natural way to interact with an intelligent personal assistant. An intelligent personal assistant comprises a camera that provides images of an area surrounding the assistant. The assistant monitors images provided by the camera to determine when a user is addressing the assistant. When voice input is received after determining that a user is addressing the assistant, the assistant understands that the voice input is intended for the assistant, and acts on the voice input to respond to the user.

<CIT> discusses an interactive head-mounted eyepiece with an integrated processor for handling content for display and an integrated image source for introducing the content to an optical assembly through which the user views a surrounding environment and the displayed content, wherein the eyepiece includes event and sensor input triggered user action capture device control.

<CIT> discusses techniques for determining when to perform an action associated with a voice command and when to disregard the voice command. In some instances, the techniques reference an identity of a user that utters a command when making this determination. For instance, if a first user awakens a device or initially begins providing voice commands to the device, the device or another computing device may perform actions associated with subsequent voice commands uttered by the same user, while disregarding voice commands from other users. That is, because the device is engaging in a dialog with a first user, the device may refrain from engaging in a dialog with a different user.

There is provided a method performed by an intelligent personal assistant, according to claim <NUM>. There is provide an intelligent personal assistant according to claim <NUM>.

The features, advantages, and objects of the present invention will become more apparent from the detailed description as set forth below, when taken in conjunction with the drawings in which like referenced characters identify correspondingly throughout, and wherein:.

Embodiments of the present invention are directed towards a more natural way to interact with an intelligent personal assistant, by reducing or eliminating the need to utter a wake word or phrase.

<FIG> is a perspective view of one embodiment of an intelligent personal assistant <NUM>, shown as a cylindrical unit comprising several cameras having camera lenses <NUM>, several audio transducers <NUM> and a speaker <NUM> supported by amplification circuitry. The assistant <NUM> monitors for audio input received in a vicinity of the intelligent personal assistant in order to determine when a wake word or phrase is spoken (in one embodiment) and in order to provide audio information to a remote server for determining a question or a command from a user. Responses to questions are provided audibly by the intelligent personal assistant through speaker <NUM>, as well as playing music. The intelligent personal assistant further comprises wireless networking circuitry to connect to a local area network and on to the Internet for communication with the remote server. Other uses for intelligent personal assistant <NUM> comprises ordering products, and controlling things such as lights, thermostats, security systems, etc.. The intelligent personal assistant <NUM>, in this embodiment, can determine when a user is addressing the assistant, so that the use of wake words and phrases can be minimized or eliminated entirely.

The remote server may provide Alexa Voice Services (AVS) to the intelligent personal assistant, which is a service offered by Amazon that analyzes audio information to determine if human speech is present in the audio information provided by the intelligent personal assistant, to interpret the speech, and to provide responses to the speech back to the assistant <NUM> over the Internet. Generally, the assistant <NUM> constantly listens for a wake word or phrase that must be spoken by a user in order to activate the assistant. For example, Amazon's Echo uses the wake word, "Alexa", while Google Home uses the wake phrase "Ok Google". When the assistant <NUM> determines that the wake word has been uttered, it records the next few seconds of audio information provided by the audio transducer(s) and then provides the recording to the remote server for processing. In prior art assistants, the wake word must be spoken each time an interaction with the assistant was desired. This typically becomes annoying after a series of interactions with the assistant.

In order to minimize or completely eliminate the need for a wake word, in one embodiment, assistant <NUM> utilizes one or more cameras <NUM> to determine when a user is addressing the intelligent personal assistant. The camera generates digital images and/or video and provides it to a processor that determines whether someone is addressing the assistant <NUM>, for example, whether someone is addressing the assistant <NUM>, for example by looking or "gazing" at assistant <NUM>, and, in one embodiment, whether someone is looking or gazing at assistant <NUM> for more than a predetermined time period (such as two seconds), or, in another embodiment, whether someone is looking or gazing at assistant <NUM> as human speech is being received by the assistant. In one embodiment, the processor does not process the digital images/video, but rather provides this information from the camera(s) to a remote server for processing, much the same way that it provides audio information to a remote server.

In one embodiment, after the assistant <NUM> determines that a wake word has spoken, assistant <NUM> records a sample of following audio input received by the audio transducer(s) and provides the sample to the remote server, as well-known in the art. However, the wake word may additionally cause the intelligent personal assistant to begin processing digital images provided by the camera to determine if any follow-on voice input is provided by a user while the user looks at assistant <NUM>. When further voice input is received as a user looks at assistant <NUM>, assistant <NUM> records the voice input and sends it to the remote server for processing. A conversation between a user and assistant <NUM> may comprise multiple questions posed by a user in order to receive an acceptable answer. For example, the following exchange may occur:.

As one can see, the above exchange is much more natural than having to speak the wake word each time a question is posed to assistant <NUM>.

In another embodiment, a wake word is not used at all. In this embodiment, assistant <NUM> continuously monitors the area surrounding assistant <NUM> via the one or more cameras <NUM>. When intelligent personal assistant <NUM> determines that a user is looking or gazing at assistant <NUM>, assistant <NUM> begins recording audio information from the audio transducer(s), generally for a predetermined time period, such as five seconds, and then sends the audio recording to the remote server. Further verbal input from the user may be recorded and sent to the remote server any time that assistant <NUM> determines that a user is looking or gazing at assistant <NUM>. Thus, a very natural "conversation" may be had between a user and assistant <NUM> without the use of a wake word or phrase.

In yet another embodiment, assistant <NUM> monitors both audio input from two or more audio transducers and digital images/video from the camera to determine when a user is attempting to communicate with assistant <NUM>. In this embodiment, assistant <NUM> comprises two or more audio transducers, and typically a series of audio transducers formed along a circumference of assistant <NUM>, in order to provide directionality to the audio input. Then, processing circuitry can determine an approximate location of a user as the user speaks to assistant <NUM>. This can aid in determining whether the same user is still speaking to assistant <NUM>. For example, assistant <NUM> may listen to audio input provided by eight audio transducers located around a circumference of assistant <NUM>. When assistant <NUM> determines that a user is looking or gazing at assistant <NUM> for more than three seconds, assistant <NUM> records a sample of the audio input received by one or more of the audio transducers after determining that a user is looking or gazing at assistant <NUM>. Assistant <NUM> additionally determines that the user is located approximately in alignment with audio transducer number seven of the eight audio transducers. This may be determined by measuring a signal strength of the audio provided by each of the audio transducers, and determining a user location using the strongest signal, or the top two or three strongest signals from adjacent audio transducers. Then, when assistant <NUM> again determines that someone is looking or gazing at assistant <NUM>, assistant <NUM> determines whether the person looking at assistant <NUM> is in the same location as the previous determination. If so, assistant may use this information to better determine whether someone is addressing assistant <NUM> or not.

It should be understood that the number, type and arrangement of the camera lenses <NUM>, audio transducers <NUM> and speaker <NUM> could be different than what is shown in <FIG>. In some embodiments, however, it is advantageous to have some physical correspondence between the camera lenses <NUM> and the audio transducers <NUM>, as will be explained later herein.

<FIG>a is a perspective view of another embodiment of the intelligent personal assistant <NUM> shown in <FIG>. In this embodiment, intelligent personal assistant <NUM> is shown comprising a reflector <NUM> located at the top of the unit, in this embodiment shaped as a hemisphere or half-dome. In other embodiments, reflector <NUM> is parabolic in shape. In any case, the sloped nature of reflector <NUM> causes light from the area surrounding intelligent personal assistant <NUM> to be reflected downwards into the interior of intelligent personal assistant <NUM>. A single camera is housed within intelligent personal assistant <NUM> pointing upwards toward an apex <NUM> of reflector <NUM>. Reflector <NUM> is shown mounted to three mounting posts along a circumference, however, in other embodiments, it may be mounted using more, or fewer, mounting posts, and that in one embodiment, a single mounting post may be used, coupled to apex <NUM> and extending downwardly to the camera lens. In yet another embodiment, the circumference of the dome could be supported by a clear, cylinder, so that nothing obstructs light being reflected off of the dome. Reflector <NUM> is manufactured from any material that reflects light, such as metal or a mirrored surface, and is shaped to reflect light in an area surrounding the assistant <NUM> towards the camera. Thus, the camera is able to "see" in a <NUM> degree view around assistant <NUM>.

Operation of this embodiment is similar to that described with respect to <FIG>, except that the only source of visual information is from the single camera.

<FIG> is a perspective view of an alternative embodiment of the intelligent personal assistant shown in <FIG>a. The intelligent personal assistant in this embodiment generally comprises the same components as intelligent personal assistant <NUM>, except that reflector <NUM> is located near the bottom of the assistant, rather than the top, and the camera located inside of the assistant is pointing down towards reflector <NUM>.

<FIG> is a perspective view of yet another embodiment of the intelligent personal assistant as shown in <FIG>, this time an intelligent personal assistant <NUM> fashioned as a ceiling-mounted unit. As in the other embodiments, assistant <NUM> comprises one or more audio transducers for receiving audio input, a speaker for providing audio responses and one or more cameras. In one embodiment, a reflector is not used, as the camera may be capable of receiving light directly from all or a great majority of the area surrounding assistant <NUM>. In one embodiment, the camera comprises a fish-eye lens.

<FIG> is a functional block diagram of one embodiment of the intelligent personal assistants as shown in <FIG>, <FIG>a and <NUM>b. The same functional components are also used in the embodiment shown in <FIG>, with the reflector omitted. <FIG> shows processor <NUM>, memory <NUM>, network interface <NUM>, one or more audio transducers <NUM>, one or more digital cameras <NUM>, reflector <NUM> and user output <NUM>. It should be understood that in some embodiments, some functionality has been omitted for purposes of clarity, such as a power supply.

Processor <NUM> comprises one or more general-purpose microprocessors, microcontrollers and/or custom or semi-custom ASICs, and/or discrete components able to carry out the functionality required for operation of the intelligent personal assistant. Processor <NUM> may be selected based on processing capabilities, power-consumption properties, and/or cost and size considerations. In the case of a microprocessor, microcontroller, or ASIC, processor <NUM> generally executes processor-executable instructions stored in memory <NUM> that control the functionality of the intelligent personal assistant. Examples of memory include one or more electronic memories such as RAM, ROM, hard drives, flash memory, EEPROMs, UVPROMs, etc. or virtually any other type of electronic, optical, or mechanical memory device, but excludes propagated signals.

Network interface <NUM> comprises circuitry necessary to transmit and receive digitized data packets comprising audio recordings from the one or more audio transducers <NUM> and/or visual recordings received by camera <NUM>, between the intelligent personal assistant and a remote server over the Internet or other wide-area network, either directly or through a local network such as a router, modem, gateway, security panel, or other home-networked device(s). Such circuitry is well known in the art and may comprise BlueTooth, Wi-Fi, or RF circuitry, among others.

One or more audio transducers <NUM> comprise one or more devices used to convert sound pressure into electronic signals for use by processor <NUM>. An example of such audio transducers are piezo-electric microphones, selected due to their inexpensive cost and small size. Another example are mems microphones.

Digital camera(s) <NUM> comprises a device that converts light into electronic signals and provides the electronic signals to processor <NUM>. A wide variety of inexpensive, high-quality cameras are available on the market today. As used herein, the term "camera" can mean, simply, a lens and an image sensor in combination. Such sensors are typically CMOS devices, and offer enough resolution to distinguish facial features at a distance of up to ten feet or more. Moreover, the subject devices may utilize a single sensor device in the form of color sensor, such as a <NUM>-channel (RGB) photodiode sensitive to the blue, green and red regions of the spectrum, or an infrared sensor, such as a passive infrared sensor (PIR sensor) that measures infrared (IR) light radiating from objects in its field of view, in combination with the reflector <NUM> to determine the presence of users within an area of the intelligent personal assistant over <NUM> degrees. In this embodiment, the use of a color sensor or a passive infrared sensor has the advantage of providing an output (which may, in turn be provided to the cloud for processing as described above) in which the image information is generally physically distorted thus providing some privacy to the individuals being monitored. As shown in <FIG>, camera <NUM> is mounted so as to face towards and centered under the apex of the reflector <NUM> to thereby provide the desired <NUM> degrees of uninterrupted coverage. In this example, the image information obtained by the sensor may be utilized to detect the presence of one or more people within the monitored space, identify how many people are within the monitored space, track the movement of people within the monitored space, determine the location of people within the monitored space relative to the personal assistant <NUM>, etc. for the purposes described previously. In some cases, the sensors may also be capable of collecting information that is sufficient for the system to distinguish between persons within the monitored space and/or to determine if one or more persons are looking at the device again for the same purposes as described above. In this manner, this device may not only perform the functions noted above, but may also use the sensed presence of a user to simply turn on and off the device such that the device is not continually in a listening mode.

User output <NUM> comprises a speaker and related electronics and/or a video display for providing answers to users' questions or status information to a user. For example, a user may ask an intelligent personal assistant what the weather will be like the following day. In response, the intelligent personal assistant may audibly inform the user, via a speaker, that rain is likely in the forecast. In another example, if a user asks the intelligent personal assistant to play a particular song, and the intelligent personal assistant may play the song for the user via a speaker. If the user output additionally comprises a video display, the video display may indicate the song name, performer, album name, or other information related to the song.

<FIG> is a flow diagram of one embodiment of a method, performed by an intelligent personal assistant, for interacting with a user in a more natural manner. It should be understood that the steps described in this method could be performed in an order other than what is shown and discussed and that some minor method steps may have been omitted for clarity and simplicity.

At block <NUM>, a user of intelligent personal assistant in the vicinity of the intelligent personal assistant utters a wake word or phrase that is received by the one or more microphones and converted into an electronic signal representative of the wake work or phrase.

At block <NUM>, processor <NUM> receives digitized audio information from the one or more microphones on the intelligent personal assistant and determines that the wake word or phrase was uttered by comparing the digitized audio information electronic to a wake word or phrase stored in memory <NUM>, using techniques well-known in the art.

At block <NUM>, in response to determining that a user uttered the wake word or phrase, processor <NUM> begins recording audio information from the one or more microphones and stores the audio information in memory <NUM>. Typically, the audio information will comprise a question or a command from the user. The audio recording is generally terminated by processor <NUM> after a predetermined time period, such as five seconds, or upon determination by processor <NUM> that the user has stopped speaking, using techniques well-known in the art.

At block <NUM>, processor <NUM> may estimate a location of the user in relation to the intelligent personal assistant using the audio information. For example, the intelligent personal assistant may comprise eight microphones lining a circumference of the intelligent personal assistant, each spaced equally apart from one another. Processor <NUM> may determine a signal strength of the audio information provided by each microphone, and use this information to determine a location of the user based on the strongest signal(s) from one or more of the microphones. For example, if microphone number <NUM> produced the strongest signal while either the wake word/phrase was spoken, or during a subsequent audio recording following detection of the wake word/phrase, processor <NUM> may conclude that the user is in an area adjacent to microphone number <NUM> or along an acoustic trajectory away from microphone number <NUM>. In other embodiments, well-known microphone beamforming techniques may be used to estimate the location of a user.

At block <NUM>, processor <NUM> causes the audio recording to be sent from memory <NUM> to a remote server via network interface <NUM>, to determine whether the audio recording contains human speech and, if so, what the user has said, again using techniques well-known in the art. In another embodiment, the audio recording is processed locally by processor <NUM>.

At block <NUM>, the remote server then sends a response back to the intelligent personal assistant with an audible response and/or it provides a command to another cloud-based server for an action to occur at the location of the intelligent personal assistant. For example, if the user's voice recording was interpreted by the remote server to "turn on the kitchen lights", the remote server may send a command to another server via the Internet that is capable of remotely controlling the kitchen lights at the user's home. When the other server receives the command, it in turn sends a signal over the wide-area network to a control device inside the home for the device to turn the kitchen lights on. In an embodiment where processing is performed locally by processor <NUM>, the response is provided by processor <NUM> to user output <NUM>, in the case of an audible response or, in the case of causing an action to occur in the user's home, sends a signal to one or more networked home automation controllers or devices, to perform such actions such as turn on/off lights, set a thermostat, order goods online, etc..

At block <NUM>, after a response has been provided to the user, visual information from the camera is evaluated in order to determine if the user, or someone else, is looking, or "gazing" at the intelligent personal assistant. In one embodiment, visual information is evaluated only for a predetermined time after the wake word or phrase was uttered, or within a predetermined time from when a response to the user's request or command following the wake word or phrase was provided. This embodiment utilizes the fact that a user is more likely to have a follow-on question or command immediately after receiving a response to his or her initial question or command, and that the user is likely looking at the intelligent personal assistant when a follow-up question or command is given.

In one embodiment, digital video and/or still images from the camera are evaluated by processor <NUM> to determine whether someone is looking or gazing at the intelligent personal assistant. In another embodiment, processor <NUM> causes the digital video and/or still images from the camera to be sent via network interface <NUM> to the same, or a different, server that processed the audio recording sent at block <NUM>, in order to evaluate the visual information from camera <NUM>. In either case, a determination may be made by determining whether two eyes and a mouth, representative of the presence of a person in the vicinity of the intelligent personal assistant, are present in the visual information. In one embodiment, an inverse function related to a curvature of the reflector is applied by processor <NUM>, or the remote server, to the digital information from camera <NUM> to eliminate or reduce visual distortion caused by curvature of the reflector.

When the remote server determines that someone is looking or gazing at the intelligent personal assistant, the remote server sends a signal to the intelligent personal assistant via the Internet and network interface <NUM>. In one embodiment, determining whether someone is looking or gazing at intelligent personal assistant comprises determining that someone is looking or gazing for more than a predetermined time period, such as two seconds. This makes it more likely that someone is intending to interact with the intelligent personal assistant, verses merely glancing at the intelligent personal assistant.

At block <NUM>, when processor <NUM> determines, or is notified by the remote server, that someone is looking or gazing at the intelligent personal assistant, and in one embodiment, for more than a predetermined time period, processor <NUM> begins recording audio information from the one or more microphones and stores the audio information in memory <NUM>. The recording is generally terminated by processor <NUM> after a predetermined time period, such as five seconds. In one embodiment, recording is stopped when processor <NUM> determines that someone has stopped speaking, using techniques well-known in the art. This may be used in conjunction with a predetermined time period. For example, the recording may be stopped upon detection that someone stopped speaking or a predetermined time period, whichever comes first. In one embodiment, the recording continues only for as long as the personal intelligent assistant is being addressed.

At block <NUM>, also in response to determining that someone is looking or gazing at the intelligent personal assistant, processor <NUM> may provide an indication to the user that the intelligent personal assistant has determined that the user is looking or gazing at the intelligent personal assistant, in order to provide feedback to the user that the user's gaze has been acknowledged by the intelligent personal assistant. Knowing this, a user may pause to receive the indication before speaking to the intelligent personal assistant. The indication may comprise, simply, of illuminating an LED located on top of the intelligent personal assistant, for example.

At block <NUM>, also in response to determining that someone is looking or gazing at the intelligent personal assistant, processor <NUM> may determine a location of the user in relation to the intelligent personal assistant using the visual information. For example, the intelligent personal assistant may comprise eight lenses lining a circumference of the intelligent personal assistant, each spaced equally apart from one another. Processor <NUM> may determine that two eyes and a mouth are present in signals provided by lenses <NUM> and <NUM>. Thus, processor <NUM> may determine that the user is located along a line of sight of both lenses <NUM> and <NUM>. In an embodiment where a reflector is used in conjunction with a single camera, the executable code that processes the images from the camera can comprise instructions that can determine a location of the user in relation to the intelligent personal assistant, based on, for example, a detection of two eyes and a mouth at a point along the <NUM> degrees of view.

At block <NUM>, processor <NUM> causes the audio recording at block <NUM> to be sent from memory <NUM> to the remote server via network interface <NUM>, to determine whether any human speech was uttered and, if so, what the user has said. In another embodiment, processing is conducted locally by processor <NUM>.

In one embodiment, processor <NUM> sends the audio recording to the remote server only when the estimated user location determined by the visual information matches the estimated user location determined by the audio information when the key word/phrase was spoken. This helps ensure that a follow-up question or command is being spoken from the user who provided the original question/command in block <NUM>, rather than a false signal as a result of someone else simply looking at the intelligent personal assistant.

<FIG> is a flow diagram of another embodiment of a method, performed by an intelligent personal assistant, for interacting with a user in a more natural manner, without the use of wake words or phrases. It should be understood that the steps described in this method could be performed in an order other than what is shown and discussed and that some minor method steps may have been omitted for clarity and simplicity.

At block <NUM>, processor <NUM> monitors digitized visual information from digital camera <NUM> to determine if anyone is looking or gazing at the intelligent personal assistant. In one embodiment, the digitized visual information comprises digital video and/or still images from the camera that are evaluated by processor <NUM>. In another embodiment, processor <NUM> causes the digitized visual information from the camera to be sent via network interface <NUM> to the same, or a different, server that processed the audio recording sent at block <NUM>, in order to evaluate the visual information from camera <NUM>.

In one embodiment, processor <NUM> performs an inverse function related to a curvature of the reflector to the digitized visual information from camera <NUM> to eliminate or reduce visual distortion caused by curvature of the reflector.

When the remote server determines that someone is looking or gazing at the intelligent personal assistant, the remote server sends a signal to the intelligent personal assistant via the Internet and network interface <NUM>. In one embodiment, determining whether someone is looking or gazing at intelligent personal assistant comprises determining that someone is looking or gazing for more than a predetermined time period, such as two seconds. In any case, a determination that someone is looking or gazing at the intelligent personal assistant may be made by determining whether two eyes and a mouth, representative of the presence of a person in the vicinity of the intelligent personal assistant, are present in the digitized visual information.

At block <NUM>, processor <NUM> may determine a location of the user in relation to the intelligent personal assistant using the digitized visual information, as explained above with respect to the method illustrated in <FIG>.

At block <NUM>, processor <NUM> may estimate a location of the person who is addressing the intelligent personal assistant, as described above in the description to the method illustrated in <FIG>.

At block <NUM>, processor <NUM> causes the audio recording to be sent from memory <NUM> to a remote server via network interface <NUM>, to determine whether the audio recording contains human speech and, if so, what the user has said, again using techniques well-known in the art. In one embodiment, the audio recording is sent only when the location of the strongest audio signal from the microphone(s) matches the estimated location of a user via the visual information. This helps confirm that an actual question or command is being addressed to the intelligent personal assistant, rather than some random noise. In another embodiment, the audio recording is processed locally by processor <NUM> in order to determine whether the audio recording contains human speech and, if so, what the user has said.

At block <NUM>, the remote server then sends a response back to the intelligent personal assistant with an audible response and/or it provides a command to another cloud-based server for an action to occur at the location of the intelligent personal assistant. For example, if the user's voice recording was interpreted by the remote server to "turn on the kitchen lights", the remote server may send a command to another server via the Internet that is capable of remotely controlling the kitchen lights at the user's home. When the other server receives the command, it in turn sends a signal over the wide-area network to a control device inside the home for the device to turn the kitchen lights on. In another embodiment, processor <NUM> sends an audible response to user output <NUM> and/or causes a signal to be transmitted via network interface <NUM> to other network-enabled devices or controllers, in order to perform such actions such as turning on/off lights, set thermostats, order goods online, etc..

At block <NUM>, after a response has been provided to the user, processor <NUM> continues to evaluate, or have evaluated by the remote server, visual information from the camera in order to determine if the user is looking at the intelligent personal assistant. If so, blocks <NUM>-<NUM> are repeated.

<FIG> is a flow diagram of yet another embodiment of a method, performed by an intelligent personal assistant as shown in <FIG>, <FIG>, <FIG>, or <NUM>, for interacting with a user in a more natural manner, using a combination of a wake word or phrase in combination with a determination that a user is addressing the intelligent personal assistant. This embodiment is useful to reduce the number of "false alarms" that may occur, for example, as a user is listening to a news or technical report describing an intelligent personal assistant. Often times, these reports audibly mention a key word or phrase in their discussions, which may cause a prior art intelligent personal assistant to react and, over a number times, become annoying. It should be understood that the steps described in this method could be performed in an order other than what is shown and discussed and that some minor method steps may have been omitted for clarity and simplicity.

At block <NUM>, processor <NUM> may estimate a location of a user who uttered the wake word/phrase in relation to the intelligent personal assistant, as described above in the description to the method illustrated in <FIG>.

At block <NUM>, processor <NUM> monitors digitized visual information from one or more digital cameras <NUM> to determine if anyone is addressing the intelligent personal assistant, as described above in the description to the method illustrated in <FIG>.

At block <NUM>, processor <NUM> may determine a location of the user who is addressing the intelligent personal assistant using the digitized visual information, as explained above with respect to the method illustrated in <FIG>.

At block <NUM>, in response to determining that a user uttered the wake word or phrase, and that someone is addressing the intelligent personal assistant, processor <NUM> begins recording audio information from the one or more microphones and stores the audio information in memory <NUM>. It should be understood that blocks <NUM> and <NUM> could be reversed. That is, processor <NUM> may determine, first, that a user is addressing the intelligent personal assistant using the visual information and then determine that a wake word or phrase has been uttered. In one embodiment, the recording commences when the wake word/phrase is uttered within a predetermined time from when the intelligent personal assistant was addressed, for example, one second. In one embodiment, the recording continues only for as long as the personal intelligent assistant is being addressed. In another embodiment, the recording is terminated after processor <NUM>, or a remote server, determines that a user has stopped speaking. In one embodiment, the recording is commenced only when the estimated location of the person who uttered the wake word/phrase matches the estimated location based on the visual information, or vice-versa, in order to better ensure that the same person who uttered the wake word/phrase is the same person who is addressing the intelligent personal assistant.

At block <NUM>, processor <NUM> causes the audio recording to be sent from memory <NUM> to a remote server via network interface <NUM>, to determine whether the audio recording contains human speech and, if so, what the user has said, again using techniques well-known in the art. In another embodiment, the audio recording is processed locally by processor <NUM> in order to determine whether the audio recording contains human speech and, if so, what the user has said.

At block <NUM>, the remote server then sends a response back to the intelligent personal assistant with an audible response and/or it provides a command to another cloud-based server for an action to occur at the location of the intelligent personal assistant, as explained above with respect to the method illustrated in <FIG>.

At block <NUM>, after a response has been provided to the user, blocks <NUM>-<NUM> are repeated.

<FIG> is a perspective view of another embodiment of an intelligent personal assistant <NUM>. In this embodiment, intelligent personal assistant <NUM> is shown comprising a retractable, reflective reflector <NUM> located on top of the unit, shaped as a hemisphere or half-dome. In other embodiments, reflector <NUM> is parabolic in shape. In any case, the sloped nature of reflector <NUM> causes light from the area surrounding intelligent personal assistant <NUM> to be reflected downwards into the interior of intelligent personal assistant <NUM>. As in other embodiments, a camera is housed within intelligent personal assistant <NUM> pointing upwards toward an apex <NUM> of reflector <NUM>, and intelligent personal assistant <NUM> further comprises one or more audio transducers <NUM> and one or more speakers <NUM>. The audio transducer(s) receives audio input from a user in proximity to intelligent personal assistant <NUM> and converts the audio input into electronic signals for use by a processor within intelligent personal assistant <NUM>. The audio input is typically questions or commands from a user, the commands typically asking intelligent personal assistant <NUM> to control certain electronic devices coupled to intelligent personal assistant <NUM> via a local area network, such as a home Wi-Fi network. For example, a user may ask intelligent personal assistant <NUM> to turn on one or more lights, turn a TV on or off, are or disarm a home security system, etc..

Reflector <NUM> comprises a spherical structure with a reflective coating or made from reflective material, to allow camera <NUM> to view <NUM> degrees around an intelligent personal assistant. In one embodiment, reflector <NUM> comprises a sphere, which may add to the aesthetic appearance of the intelligent personal assistant. Camera <NUM> is positioned either below or above reflector <NUM>, pointed directly upwards, or downwards, depending on implementation, towards reflector <NUM>, which enables camera <NUM> to receive light reflected from around the assistant in <NUM> degrees, unobstructed by any mechanisms to support reflector <NUM>.

Reflector <NUM> is shown mounted inside a rigid, transparent support structure <NUM>, in this embodiment, in the shape of a short, clear cylinder, so that nothing obstructs light from reaching reflector <NUM>. Reflector <NUM> is manufactured from any material that reflects light, such as metal or a mirrored surface, and is shaped to reflect light in an area surrounding the assistant <NUM> towards the camera. Thus, the camera is able to "see" in a <NUM> degree view around assistant <NUM> without obstruction. Generally, reflector <NUM> comprises a rigid, transparent support structure <NUM> comprises a rigid, clear material such as glass, Lucite®, one or more of a wide variety of clear plastics, etc. In one embodiment, during construction of intelligent personal assistant <NUM>, a cylindrical mold is partially filled with liquefied or gelatinous clear material which, when cool, forms rigid, transparent support structure <NUM>. While the clear, rigid support structure is in the liquefied or gelatinous state, reflector <NUM> is placed into the cylindrical mold, with apex <NUM> in a down-facing position. When the liquefied or gelatinous cools, reflector <NUM> is secured within and to the clear, rigid material. In another embodiment, transparent support structure <NUM> is formed with a depression sized and shaped to receive reflector <NUM>. Then, reflector <NUM> is placed into the depression and secured in place by traditional methods, such as clear adhesive, screws, rivets, clips, etc. In yet another embodiment, reflector <NUM> is formed by first forming structure <NUM> with a depression formed into a top of structure <NUM> resembling reflector <NUM>, and then coating this surface with reflective material, such as quicksilver, paint, or the like, or reflective metal.

The various components of intelligent personal assistant <NUM> are generally the same components as the intelligent personal assistant shown in <FIG>, shown as functional block diagrams as shown in <FIG>.

<FIG> is a perspective view of the intelligent personal assistant <NUM> of <FIG>, with reflector <NUM> stowed inside of intelligent personal assistant <NUM>. Some of the features as shown in <FIG> have been omitted for clarity. In this view, rigid, transparent support structure <NUM> has been pushed downward by a user, inside the housing of intelligent personal assistant <NUM>. This is typically accomplished by the user pressing downward on surface <NUM> while intelligent personal assistant <NUM> is in the position as shown in <FIG>, until rigid, transparent support structure <NUM> reaches the position as shown in <FIG>. Rigid, transparent support structure <NUM> is held into either an extended position, as shown in <FIG> or a retracted position, as shown in <FIG>, by one or more spring-loaded, push-to-lock mechanisms <NUM>. Push-to-lock mechanisms <NUM> typically comprise a spring acting on an extension that pushes against rigid, transparent support structure <NUM>, causing it to extend into the position as shown in <FIG>. The spring is coiled as a user presses downward on surface <NUM> and when rigid, transparent support structure <NUM> reaches a point just below that as shown in <FIG>, a locking mechanism of push-to-lock mechanism <NUM> is engaged, holding structure <NUM> in place. To extend structure <NUM>, a user pushes down on surface <NUM>, thus releasing the locking mechanism inside push-to-lock mechanism <NUM>, causing the spring and extension to push structure <NUM> upwards into the extended position. In one embodiment, push-to-lock mechanism <NUM> further comprises a dampening mechanism that causes structure <NUM> to more slowly move from the stowed position to the extended position. Push-to-lock mechanism <NUM> is a well-known mechanical device in the art.

Digital camera <NUM> is shown in hidden lines in <FIG>, mounted in a fixed position, pointing upwards towards an apex of reflector <NUM>. Camera <NUM> converts light reflected from reflector <NUM> into digital signals for processing by processor <NUM>. When structure <NUM> is stowed, reflector <NUM> cannot receive light from the area surrounding intelligent personal assistant <NUM>. When structure <NUM> is extended, camera <NUM> receives light from the area surrounding intelligent personal assistant <NUM>. Structure <NUM> is configured to position reflector <NUM> a predetermined distance from digital camera <NUM> such that images captured by camera <NUM> are in focus.

In one embodiment, when structure <NUM> is in the stowed position, processer <NUM> stops processing information from camera <NUM>, either by processor <NUM> ignoring further information from camera <NUM>, by camera <NUM> ceasing to provide digital signals to processor <NUM>. Processor stops processing information when it receives a signal from a mechanical, optical, or magnetic switch <NUM> located within intelligent personal assistant <NUM> and activated when structure <NUM> is stowed, typically by mechanical interaction with the extension and switch <NUM>, or by direct interaction between structure <NUM> and switch <NUM>.

In one embodiment, when a user presses on surface <NUM> to stow or retract reflector <NUM>, camera <NUM> may record a fingerprint from the user as the user presses downward on surface <NUM>. In this embodiment, surface <NUM> is made from a rigid, transparent material such as clear plastic or glass, and switch <NUM> (or some other switch capable of detecting when reflector <NUM> is being moved by a user) is used to detect movement of reflector <NUM> by a user. Processor <NUM> receives a signal from switch <NUM> during movement of reflector <NUM>, and processor <NUM> records one or more images and/or video of the user's finger or thumb pressing downwards on surface <NUM> via aperture <NUM> formed into apex <NUM>. Aperture <NUM> is small enough not to substantially alter the light reaching camera <NUM>, but large enough to allow camera <NUM> to view a user's finger or thumb when it pushed downward on surface <NUM>. In one embodiment, where reflector is six inches in diameter, the diameter of aperture <NUM> is typically between one-half inch and one inch.

When processor <NUM> captures an image of a user's finger or thumbprint, processor <NUM> may compare the image to one or more images stored in memory <NUM> to determine if the user is authorized to stow or retract reflector <NUM>, as stowing reflector <NUM> prevents intelligent personal assistant <NUM> from processing visual information in the vicinity of intelligent personal assistant <NUM>. Users may pre-store one or more fingerprints during setup of intelligent personal assistant <NUM> (or during a "learn" mode, where intelligent personal assistant <NUM> stores images/fingerprints of authorized users) where processor <NUM> stores the images in memory <NUM>. The fingerprints may be stored in association with an image of a user, or with attributes determined by processor <NUM> as a result of processing images of the user. If processor <NUM> determines that an un-registered user is stowing or retracting reflector <NUM>, for example, no match is found between user fingerprint images captured during stowage/retraction to pre-stored fingerprint images in memory <NUM>, processor may generate an alert and send it to network interface <NUM>, alerting an authorized user that an un-authorized person is operating intelligent personal assistant <NUM>. The alert may be addressed to a pre-registered user (i.e., a person who has previously submitted at least one fingerprint during setup of intelligent personal assistant <NUM>) by retrieving an email address, telephone number, or some other information used to contact a user), and then addressing the alert using the contact information stored in memory <NUM>.

<FIG> is a flow diagram of a method, performed by intelligent personal assistant <NUM>, for interacting with a user. The method begins as intelligent personal assistant <NUM> is in an extended position and has been preprogrammed to interact with at least one electronic device over a local area network, i.e., to turn at least one electronic device on or off. It should be understood that the steps described in this method could be performed in an order other than what is shown and discussed and that some minor method steps may have been omitted for clarity and simplicity.

At block <NUM>, processor <NUM> monitors an area surrounding intelligent personal assistant <NUM> by processing digitized video or images from camera <NUM>. Generally, camera <NUM> captures image information in an unobscured, <NUM> degree view around intelligent personal assistant <NUM> due to reflective reflector being suspended inside structure <NUM>.

At block <NUM>, a user enters a room where intelligent personal assistant <NUM> is located, and the user may instruct intelligent personal assistant <NUM> to enter a setup or "learn" mode of operation, causing intelligent personal assistant <NUM> to remember visual features of the user's face and/or body for future reference. In one embodiment, the user may cause intelligent personal assistant <NUM> to enter the setup or learn mode using a predetermined voice command in connection with a wake word. In another embodiment, the user may use an app on a mobile device such as a smart phone or tablet computer to wireless transmit a command to place intelligent personal assistant <NUM> into the setup or learn mode.

In one embodiment, only the lower portion of a user's body is recorded for future reference to identify persons in the room where intelligent personal assistant <NUM> is located. In this embodiment, camera <NUM> may only be capable of viewing the lower portion of a user's body, especially if the user is close to intelligent personal assistant <NUM>, due to the inherent nature of a half-dome generally being limited to reflecting light at a height of the upper-most portion <NUM> of reflector <NUM> or below.

At block <NUM>, after intelligent personal assistant <NUM> has entered the setup or learn mode, the user may face intelligent personal assistant <NUM> so that processor <NUM> may capture one or more images or video of the user via camera <NUM>. The image or video information may be stored in memory <NUM> or provided to a server for storage over the local area network and the Internet. Processor stores this information in association with an identity of the user, which is typically provided by the user prior to, during or after processor <NUM> has captured the image of video information of the user. In another embodiment, processor may determine certain physical characteristics of the user, such as an approximate height or weight of the user, the user's eye color, hair color, skin tone, presence or absence of facial hair, or other distinguishing physical characteristics from the images and/or video and store this information, rather than raw visual information. In another embodiment, only a portion of a user is recorded in an embodiment where light cannot be reflected by reflector <NUM> at a height greater than an upper-most portion <NUM> of reflector <NUM>. Thus, later user identification is performed using information not associated with a user's head, such as using only a user's leg length, waist size, garments, gait, etc. as factors in identifying a user.

At block <NUM>, processor <NUM> may provide an indication to the user that the image/video capture/identification process has been completed by causing user output <NUM> to sound a predetermined tone or other audio signal, and/or to cause one or more lights to become illuminated on an exterior surface of intelligent personal assistant <NUM>.

At block <NUM>, processor <NUM> may receive instructions from the user to cause one or more predetermined actions to occur when intelligent personal assistant <NUM> identifies the presence of the user in proximity to intelligent personal assistant <NUM>. One or more electronic devices may be electronically coupled to intelligent personal assistant <NUM> via a local area network, such as a home Wi-Fi network, and the instructions may comprise commands to operate or control one or more functions of one or more of these electronic devices. For example, a Wi-Fi enabled stereo and television may be electronically coupled to intelligent personal assistant <NUM>, and the instructions could comprise a voice command from the user to turn both the stereo and the TV on, to set the stereo to accept audio information from the TV, and to set the stereo at a certain volume. The user may, as another example, provide instructions to disarm a home security system, or to change an operating mode of such a home security system, such as from "Armed-Away" to "Armed-Home". In another example, an HVAC system could be coupled to intelligent personal assistant <NUM>, and the instructions could comprise instructions that cause the HVAC system to set the temperature of the room where intelligent personal assistant <NUM> is located to a predetermined temperature.

At block <NUM>, processor <NUM> stores the instructions in memory <NUM> and associates the instructions with the image/video information received at block <NUM>.

At some time later, at block <NUM>, when most or all of the electronic devices in the room where intelligent personal assistant <NUM> is located are in the off position, and structure <NUM> is in an extended position, camera <NUM> monitors the area around intelligent personal assistant <NUM> by receiving unobstructed light from the area surrounding intelligent personal assistant <NUM> to detect when someone has entered the room.

At block <NUM>, the user enters the room.

At block <NUM>, camera <NUM> receives light reflected from reflector <NUM>, capturing the user soon after entering the room. Camera <NUM> converts the light into one or more digitize images/video and provides it to processor <NUM>.

At block <NUM>, processor <NUM> determines an identity of the user that entered the room by comparing the digitized images/video to previously-stored images/video of one or more users at block <NUM> using known image comparison techniques. In another embodiment, processor <NUM> sends the captured image/video information to a remote sever via, for example, the local area network and the Internet, where the remote server performs the identity of the user by comparing the image/video data to previously provided image/video data using known image comparison techniques.

At block <NUM>, if the identity of the user is determined, processor <NUM> determines one or more actions to perform, based on the instructions from the identified user as provided at block <NUM>.

At block <NUM>, processor <NUM> provides one or more instructions or commands to one or more electronic devices coupled to intelligent personal assistant <NUM> via a local area network. In response, the one or more electronic devices perform an intended function, such as to turn one or more lights on or off, set the room temperature to a predetermined temperature, open or close blinds, set a stereo and/or TV to predetermined settings, such setting the stereo to receive a certain music stream, or setting the TV to a predetermined channel.

At block <NUM>, at any time, a user may wish to stow structure <NUM>, so that intelligent personal assistant <NUM> no longer processes visual information in the vicinity of intelligent personal assistant <NUM>. The user may stow structure <NUM> to allow the user more privacy, i.e., knowing that visual information is not being processed by intelligent personal assistant <NUM> or by a server in a remote location over the Internet. However, audio input <NUM> may still monitor the area surrounding intelligent personal assistant <NUM> for audible queries or instructions from a user.

At block <NUM>, processor <NUM> detects that structure <NUM> has been stowed, as indicated by switch <NUM>. In response, processor <NUM> stops processing signals from camera <NUM>, and/or camera <NUM> is disabled from providing digitized images/video to camera <NUM>.

<FIG> is a perspective view of yet another embodiment of an intelligent personal assistant. In this embodiment, intelligent personal assistant <NUM> is similar to the intelligent personal assistant as shown in <FIG> and <NUM>B, with the exception of reflector <NUM> being replaced by a pair of reflectors <NUM>. In one embodiment, reflector <NUM> comprises a primary, parabolic reflector <NUM> and a secondary, hyperbolic reflector <NUM>. Light from the area surrounding intelligent personal assistant <NUM> is reflected down through through-hole <NUM> into intelligent personal assistant <NUM>. A camera centrally mounted inside intelligent personal assistant <NUM> and pointing upward captures the light, similar to the embodiments shown in <FIG>, <FIG>, <FIG>and <FIG>. Although reflectors <NUM> and <NUM> are shown in <FIG> as shown, in other embodiments, they could comprise different dimensions, such as the apexes of both reflectors extending towards each other, different sizes, shapes, etc. Further details of reflector <NUM> is provided below.

<FIG> illustrates reflector <NUM>. Reflector <NUM> comprises a primary, parabolic reflector <NUM> and a secondary, hyperbolic reflector <NUM> spaced apart from each other. In this embodiment, secondary reflector <NUM> is shown as being smaller than secondary reflector <NUM> shown in <FIG>. Primary reflector <NUM> comprises through-hole <NUM> located at the center of primary reflector <NUM> that allows light from secondary reflector <NUM> to pass to camera <NUM>. The surfaces of primary reflector <NUM> and secondary reflector <NUM> are reflective, comprising quicksilver, paint, or the like, or reflective metal. In one embodiment, secondary reflector <NUM> is suspended above primary reflector <NUM> by forming a transparent cylinder <NUM> and coupling primary reflector <NUM> to one end and secondary reflector <NUM> to the other end, or by forming one end of the transparent cylinder in a parabolic shape and at least a portion of the opposing end in a hyperbolic shape, and then coating these shapes with a reflective material.

In operation, light from an area surrounding intelligent personal assistant <NUM> passes through transparent cylinder <NUM> and reflected off of primary reflector <NUM>, where it is focused on secondary reflector <NUM>. Secondary reflector <NUM>, in turn, focuses the light downwards through through-hole <NUM>, to camera <NUM>. While the light striking camera <NUM> may be distorted due to the parabolic and hyperbolic reflectors, software executed by processor <NUM> can be used to remove the distortion enough to enable processor <NUM> to make visual comparisons of persons in a room where intelligent personal assistant <NUM> to stored images and/or information pertaining to such persons. Finally, similar to the embodiment shown in <FIG> and <FIG>, transparent cylinder <NUM> may be stowed inside intelligent personal assistant <NUM> by mounting transparent cylinder <NUM> to a mechanism that allows for such stowage.

The methods or steps described in connection with the embodiments disclosed herein may be embodied directly in hardware or embodied in machine-readable instructions executed by a processor, or a combination of both. The machine-readable instructions may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. In the alternative, the processor and the storage medium may reside as discrete components.

Accordingly, an embodiment of the invention may comprise a non-transitory processor-readable media embodying code or machine-readable instructions to implement the teachings, methods, processes, algorithms, steps and/or functions disclosed herein.

Claim 1:
A method, performed by an intelligent personal assistant (<NUM>, <NUM>), comprising:
reflecting light from an area around the intelligent personal assistant by a reflector (<NUM>, <NUM>) into an interior of the intelligent personal assistant, wherein the reflector is supported by a rigid, transparent support structure (<NUM>) that maintains the reflector in a position above the housing;
receiving, while the rigid, transparent support structure (<NUM>) is in an extended position, the reflected light from the reflector by a digital camera (<NUM>) positioned inside the intelligent personal assistant and converting the light into electronic signals;
receiving the electronic signals from the digital camera by a processor (<NUM>);
determining, by the processor, an identity of a user in proximity to the intelligent personal assistant based on the electronic signals;
causing, by the processor, an action to be performed based on the identity of the user, and
stowing the rigid, transparent support structure inside the housing, thereby placing the rigid, transparent support structure (<NUM>) into a retracted position and preventing the camera from receiving light from the area surrounding the intelligent personal assistant (<NUM>, <NUM>),
wherein stowing the rigid, transparent support structure (<NUM>) inside the intelligent personal assistant (<NUM>, <NUM>) comprises receiving manual input from the user, pushing the rigid, transparent support structure inside the intelligent personal assistant.