System and method for conversation-based notification management

A method for dynamic notification management at a head mounted display (HMD) includes presenting, at the HMD, an augmented reality display, receiving, at a processor controlling the HMD, a notification from an application for display at the HMD, receiving, at the processor at a first time, first sensor data from one or more of a camera or a microphone, determining, based on the first sensor data, a first value of one or more factors associated with a probability that a user of the HMD is currently in a real-world conversation, determining an importance value of the received notification at the first time, and determining whether to display the notification from the application based on a comparison of the first value of the one or more factors associated relative to the importance value of the received notification.

TECHNICAL FIELD

This disclosure relates generally to augmented reality (AR) and systems and methods for managing AR displays. More specifically, this disclosure relates to a system and method for conversation-based notification management.

BACKGROUND

Portable networked devices with displays, such as tablets and smartphones have, within the course of little over a decade, gone from being luxury items to supplanting desktop computers, fax machines and landline telephones as the default device for personal and business communications. The smartphone's dominance as the primary communication interface for millions of people is reflected by the frequency with which many users check their devices and the steady rise in screen time among users.

In addition to the emergence of portable networked devices as a dominant communication medium, the past decade has also witnessed the emergence of new display technologies, including virtual reality (VR) and augmented reality (AR), which harness the possibilities of small form processors and lightweight displays to provide displays which supplant or augment the feed of visual information to viewers' eyes from the physical world. Typically, AR displays are provided through head-mounted apparatus, such as AR glasses or other head-mounted displays (“HMD”)

Similar to hand-held or worn devices, such as smartphones and smart watches, notifications, such as pop-up messages, from applications can be provided to a user as part of an AR display presented through an HMD. However, unlike a smartphone, where a user has one or more mechanical options (for example, putting her phone in her pocket or purse) by which she can postpone seeing a notification provided through the device, the fact that the HMD is worn in front of the user's eyes, means that HMD users have less agency to defer reviewing notifications from applications than users of other apparatus. The immersive character of the AR experience means that AR HMD users must rely primarily on the control logic providing the AR display to perform an analysis of the operating context of the HMD and present notifications in a way that is contextually appropriate.

Analyzing the operational context of an HMD and determining appropriate times to present notifications through an AR display presents a nuanced, multi-faceted problem that resists simple rule-based solutions. For example, a simple rule suspending presenting notifications through the HMD whenever the outside sound level exceeds a threshold amplitude, while perhaps effective for ensuring that notifications do not interrupt a user's real-world conversations, would almost certainly be over-exclusive, and prevent the display of important or urgent notifications.

Accordingly, providing effective, context-aware notification management at AR devices, particularly where human voices have been detected, remains a source of technical challenges and opportunities for improvement in the art.

SUMMARY

This disclosure provides a system and method for conversation-based notification management.

In a first embodiment, a method for dynamic notification management at an HMD includes presenting, at the HMD, an augmented reality (AR) display comprising items of digital content incorporated into a field of view of the HMD, receiving, at a processor controlling the HMD, a notification from an application for display at the HMD, receiving, at the processor at a first time, first sensor data from one or more of a camera or a microphone, the one or more of the camera or the microphone being located proximate to the HMD, determining, based on the first sensor data, a first value of one or more factors associated with a probability that a user of the HMD is currently in a real-world conversation, determining an importance value of the received notification at the first time, and determining whether to display the notification from the application based on a comparison of the first value of the one or more factors associated with the probability that the user is currently in the real-world conversation relative to the importance value of the received notification at the first time.

In a second embodiment, an apparatus for performing for dynamic notification management includes a processor communicatively connected to a head mounted display and a memory. The memory contains instructions, which when executed by the processor, cause the apparatus to present, at an HMD, an AR display comprising items of digital content incorporated into a field of view of the HMD, receive, a notification from an application for display at the HMD, receive, at a first time, first sensor data from one or more of a camera or a microphone, the one or more of the camera or the microphone being located proximate to the HMD, determine, based on the first sensor data, a first value of one or more factors associated with a probability that a user of the HMD is currently in a real-world conversation, determine an importance value of the received notification at the first time, and determine whether to display the notification from the application based on a comparison of the first value of the one or more factors associated with the probability that the user is currently in the real-world conversation relative to the importance value of the received notification at the first time.

DETAILED DESCRIPTION

FIG.1illustrates a non-limiting example of a device100for providing conversation-based notification management at an HMD according to some embodiments of this disclosure. The embodiment of the device100shown inFIG.1is for illustration only, and other configurations are possible. However, suitable devices come in a wide variety of configurations, andFIG.1does not limit the scope of this disclosure to any particular implementation of a device. For example, device100may be implemented as an HMD, or as a separate device (for example, a smartphone) controlling an AR display presented at a connected (for example, through a BLUETOOTH or ZIGBEE connection) HMD. Depending on the form factor, battery life, and other performance requirements of the system providing the AR display, other embodiments are possible and within the contemplated scope of this disclosure.

As shown in the non-limiting example ofFIG.1, the device100includes a communication unit110that may include, for example, a radio frequency (RF) transceiver, a BLUETOOTH transceiver, or a WI-FI transceiver, etc., transmit (TX) processing circuitry115, a microphone120, and receive (RX) processing circuitry125. The device100also includes a speaker130, a main processor140, an input/output (I/O) interface (IF)145, input/output device(s)150, and a memory160. The memory160includes an operating system (OS) program161and one or more applications162.

Applications162can include games, social media applications, applications for geotagging photographs and other items of digital content, virtual reality (VR) applications, augmented reality (AR) applications, operating systems, device security (e.g., anti-theft and device tracking) applications or any other applications which access resources of device100, the resources of device100including, without limitation, speaker130, microphone120, input/output devices150, and additional resources180. According to some embodiments, applications162include applications which can consume image data from physical objects in a field of view of a camera of electronic device100and provide AR or VR content through a display of device100, or a display of a separate device. Further, one or more of applications162are configured to periodically provide a user with application-related notifications through an HMD connected to device100.

The communication unit110may receive an incoming RF signal, for example, a near field communication signal such as a BLUETOOTH or WI-FI signal. The communication unit110can down-convert the incoming RF signal to generate an intermediate frequency (IF) or baseband signal. The IF or baseband signal is sent to the RX processing circuitry125, which generates a processed baseband signal by filtering, decoding, or digitizing the baseband or IF signal. The RX processing circuitry125transmits the processed baseband signal to the speaker130(such as for voice data) or to the main processor140for further processing (such as for web browsing data, online gameplay data, notification data, or other message data). Additionally, communication unit110may contain a network interface, such as a network card, or a network interface implemented through software.

The TX processing circuitry115receives analog or digital voice data from the microphone120or other outgoing baseband data (such as web data, e-mail, or interactive video game data) from the main processor140. The TX processing circuitry115encodes, multiplexes, or digitizes the outgoing baseband data to generate a processed baseband or IF signal. The communication unit110receives the outgoing processed baseband or IF signal from the TX processing circuitry115and up-converts the baseband or IF signal to an RF signal for transmission.

The main processor140can include one or more processors or other processing devices and execute the OS program161stored in the memory160in order to control the overall operation of the device100. For example, the main processor140could control the reception of forward channel signals and the transmission of reverse channel signals by the communication unit110, the RX processing circuitry125, and the TX processing circuitry115in accordance with well-known principles. In some embodiments, the main processor140includes at least one microprocessor or microcontroller.

The main processor140is also capable of executing other processes and programs resident in the memory160. The main processor140can move data into or out of the memory160as required by an executing process. In some embodiments, the main processor140is configured to execute the applications162based on the OS program161or in response to inputs from a user or applications162. Applications162can include applications specifically developed for the platform of device100, or legacy applications developed for earlier platforms. Additionally, main processor140can be manufactured to include program logic for implementing methods for monitoring suspicious application access according to certain embodiments of the present disclosure. The main processor140is also coupled to the I/O interface145, which provides the device100with the ability to connect to other devices such as laptop computers and handheld computers. The I/O interface145is the communication path between these accessories and the main processor140.

The main processor140is also coupled to the input/output device(s)150. The operator of the device100can use the input/output device(s)150to enter data into the device100. Input/output device(s)150can include keyboards, head mounted displays (HMD), touch screens, mouse(s), track balls or other devices capable of acting as a user interface to allow a user to interact with electronic device100. In some embodiments, input/output device(s)150can include a touch panel, a (digital) pen sensor, a key, or an ultrasonic input device.

Input/output device(s)150can include one or more screens, which can be a liquid crystal display, light-emitting diode (LED) display, an optical LED (OLED), an active-matrix OLED (AMOLED), or other screens capable of rendering graphics.

The memory160is coupled to the main processor140. According to certain embodiments, part of the memory160includes a random access memory (RAM), and another part of the memory160includes a Flash memory or other read-only memory (ROM). AlthoughFIG.1illustrates one example of a device100. Various changes can be made toFIG.1.

For example, according to certain embodiments, device100can further include a separate graphics processing unit (GPU)170.

According to certain embodiments, electronic device100includes a variety of additional resources180which can, if permitted, be accessed by applications162. According to certain embodiments, additional resources180include an accelerometer or inertial motion unit182, which can detect movements of the electronic device along one or more degrees of freedom. Additional resources180include, in some embodiments, a dynamic vision sensor (DVS)184, one or more cameras186of electronic device100.

AlthoughFIG.1illustrates one example of a device100for managing notifications provided through an AR display, various changes may be made toFIG.1. For example, the device100could include any number of components in any suitable arrangement. In general, devices including computing and communication systems come in a wide variety of configurations, andFIG.1does not limit the scope of this disclosure to any particular configuration. WhileFIG.1illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

FIG.2illustrates an example of an HMD200according to one or more embodiments of this disclosure. According to some embodiments, HMD200may operate as an accessory device to another device (for example, a smartphone.

Referring to the non-limiting example ofFIG.2, HMD200includes an externally oriented camera205. For the purposes of explaining this non-limiting example, the arrow201is provided. Arrow201points externally, towards a field of view away from the direction of projection of an internal-facing display of HMD200. According to various embodiments, externally oriented camera205is an RGB digital video camera (for example, a camera using a CMOS sensor). According to some embodiments, externally oriented camera205is a camera capable of detecting light at wavelengths outside the visible range of the human eye (for example, infrared). In certain embodiments, externally oriented camera205is a dynamic vision sensor (DVS), which provides an event stream of changes in the intensity of light received at pixels of a sensor of the DVS. In this non-limiting example, externally oriented camera205generates image data, either as an event stream or as discrete image frames, from which objects in a field of view of HMD200may be recognized.

Referring to the non-limiting example ofFIG.2, HMD200includes display207. According to various embodiments, display207displays, in an internally facing direction (e.g., in a direction having a component that is opposite to arrow201) items of AR content in conjunction with views of objects in an externally facing field of view. According to some embodiments, the display207is clear (similar to, for example, the displays used in “smart glasses” or “heads-up displays” on the cockpit glass of an airplane) and views of objects in externally facing fields of view come from light passing through display207. According to various embodiments, (sometimes referred to as “mixed reality”) the display207is opaque, and views of objects in the externally facing fields of view come from image data from the externally oriented cameras (for example, the externally oriented camera205).

According to various embodiments, the HMD200includes second camera209. In some embodiments, the second camera209is an externally oriented camera of the same type as the externally oriented camera205, thereby forming a stereoscopic pair that can generate image data comprising depth estimation. In certain embodiments, the second camera209is an externally oriented camera having a different sensor type than the externally oriented camera205. For example, in some embodiments, to extend battery life and minimize processor usage, the externally oriented camera205is a DVS sensor, and the second camera209is a CMOS type camera, which, while less efficient than a DVS sensor, can provide additional image data that is useful for object recognition. For example, the data can be data regarding colors and elements of a scene whose brightness may not change at a level detectable by a DVS sensor. According to various embodiments, the second camera209is an internally facing camera that tracks the motion of the eyes of a user, and by implication, a direction of a gaze of the user. Gaze tracking can be used to support foveal rendering of items of AR content, which can conserve battery and processor resources by rendering items of AR content away from a viewer's gaze at lower resolutions.

According to certain embodiments, the HMD200includes processor211and memory213. In certain embodiments, the memory213contains program code that, when executed by processor211, causes the HMD200to execute an AR application.

Referring to the non-limiting example ofFIG.2, the HMD200includes an inertial measurement unit215that generates location data associated with the motion of HMD200along one or more degrees of freedom. In certain embodiments, a data output from IMU215is usable for positioning (for example, to confirm a geospatial position of HMD200), or to obtain image stabilization data (for example, data indicating the direction and periodicity of a camera shake) to facilitate object recognition.

In some embodiments, the HMD200includes input/output interface217. According to various embodiments, the I/O interface217provides communicative connectivity between the HMD200and at least one other electronic device, such as a smartphone, or computer to which the HMD200is an accessory device. The I/O interface217is, in certain embodiments, a wireless communication interface, such as a BLUETOOTH transceiver, or communication hardware supporting communications over one or more longer range wireless systems (for example, communication unit110inFIG.1).

FIG.3illustrates an example of a notification appearing in a field of view at an HMD according to various embodiments of this disclosure. The example field of view inFIG.3is for illustration only and other examples could be depicted without departing from the scope of the present disclosure.

Referring to the non-limiting example ofFIG.3, a scene300is shown from a viewing point along an internally facing direction relative to an HMD305. In this illustrative example, scene300includes first tree301a, second tree301b, house302and stream303.

According to certain embodiments, HMD305includes an internally facing display309. In this non-limiting example, the internally facing display309is at least partially transparent, and permits light from first tree301aand house302to pass to the viewing point. According to some embodiments, the internally facing display309comprises one or more regions in which items of AR content (for example, notification311) can be displayed. Depending upon the number (quantity) of applications that are providing content through the internally facing display309, additional notifications may be presented to the viewer. In the simplified example ofFIG.3, and assuming the user is not driving or engaged in another activity requiring undivided attention, the notification311presents a limited distraction to the user, as there is little activity in scene300competing with notification311for a user's immediate attention.

FIGS.4A and4Bprovide an example visualization of one problem specifically addressed by certain embodiments according to this disclosure. Referring to the illustrative example ofFIG.4A, a man and two women are depicted as conducting a real-world conversation. As shown in the example depicted inFIG.4A, the man405is wearing an HMD410comprising a pair of smart glasses. As shown inFIG.4B, the man405receives, through his HMD410, a set of notifications415from applications executing on the HMD410or an accessory device.

As shown in the visualization ofFIG.4B, given the timing of their appearance, notifications415are functionally equivalent to a fourth person loudly interrupting the parties' conversation, immediately drawing the man's attention from the present conversation. Depending upon the context of the notification and the state of the conversation, immediately presenting the notification (for example, if the notification pertained to a clear and present risk to the user's health, such as a public safety announcement for the area) may be appropriate. However, in some contexts, for example, if the notification advised of a multi-day sale at a supermarket, immediately presenting the notification would be inappropriate. A further contextual wrinkle to be considered is the present state of the conversation, and more specifically, the likelihood that the conversation has concluded, or is still ongoing. Returning to the example of an application alerting an HMD-wearer of a sale at a supermarket, if the conversation between the man405and the two women had concluded, then it may be appropriate to display an otherwise low-priority notification to the man405.

As discussed with reference to the examples ofFIGS.5-8, certain embodiments according to the present disclosure solve the technical problem described with reference toFIGS.4A-4Bby deferring display of notifications whose present priority value does not outweigh a value associated with the current conversational state.

FIG.5illustrates an example architecture500for performing conversation-based notification management according to various embodiments of this disclosure. The architecture descried with reference toFIG.5may, in some embodiments, be performed at a single processing platform (for example, HMD200inFIG.2), or at a plurality of processing platforms that includes an HMD providing an AR display that includes notifications from applications.

Referring to the illustrative example ofFIG.5, architecture500comprises one or more applications505(for example, applications162inFIG.1) executing on a processing platform (for example, the device100inFIG.1or the HMD200inFIG.2), and which are configured to provide notifications (for example, pop-up notification311inFIG.3, or notifications401inFIG.4B) to a user through an AR display provided at the HMD). As shown inFIG.5, notifications510generated by application(s)505are passed to notification manager550, which implements control logic for conversation-based notification management.

As shown in the explanatory example ofFIG.5, architecture500further comprises program code515periodically determining a value of the function UserInConversationInstant( ) which returns a value based on one or more factors determined based upon, at a minimum, sensor data from one or more of a microphone or a camera proximate to the HMD, as to the instant probability that the user wearing the HMD is presently in a real-world conversation. In some embodiments, the microphone and camera are components of the HMD. In some embodiments, the microphone and camera may be components of an accessory device (for example, a smartphone or smart watch) that is sufficiently proximate to the HMD that it can provide sensor data that is an effective proxy for data collected at the HMD.

In certain embodiments, in addition to determining an instant value of the probability that the HMD user is presently in a real-world conversation from values based on the instant sensor data, program code515may further compute a weighted average of the probability that the HMD user is in a conversation. According to various embodiments, the weighted average may be calculated by assigning a time-based weighting factor (for example, assigning less weight to the oldest determined values of UserInConversationInstant( ) and more weight to the most recent values of UserInConversationInstant( )) of the most recently calculated value of UserInConversationInstant( ) and updating the weighted average value of UserInConversationInstant( ). According to certain embodiments, providing a time-weighted average value of UserInConversationInstant( ) to notification manager550can avoid management errors associated with instantaneous pauses in the conversation or misleading camera data (for example, where an HMD wearer turns her head and her conversation partners are no longer in the field of view of the HMD camera).

Referring to the illustrative example ofFIG.5, according to various embodiments, notification manager550performs at least the following operations. First, at operation555, the notification manager determines a current importance weight of the notification(s)510. As discussed in greater detail herein, the importance of a notification is dynamic, with certain notifications (for example, a notification from a vehicle parking application informing a user that she needs to move her vehicle) becoming more pressing as time passes and others becoming less important as time goes on (for example, a notification from a social media application indicating that a friend engaged with an item of shared content). Additionally, the importance of a notification may further change in response to contextual factors.

As shown inFIG.5, notification manager550further performs operation560, wherein the notification manager compares a value of the current importance weight of the notification against a current value of a probability that the HMD user is currently in a conversation. In some embodiments, the current value of the probability that the HMD user is currently in a real-world conversation may be the most recently determined value of USERINCONVERSATIONINSTANT( ). In some embodiments, the current value of the probability that the HMD user is currently engaged in a real-world conversation is a time-weighted average of values of USERINCONVERSATIONINSTANT( ) over a specified time window.

Depending upon the outcomes of operations555and560, notification manager550can output one of handling instructions565athrough565dfor managing the display of notification(s)510. According to some embodiments, notification manager550can output instruction565a, which uses the standard logic of the HMD or other processing platform managing the AR display for handling notifications. For example, if notification manager550determines that the HMD user is not engaged in a conversation, the processing platform's default policies for handling applies. Similarly, if an overriding factor (for example, an extremely important message, or data showing a user is driving a car or otherwise cannot be distracted), conversation-based notification management is bypassed, and the platform's default notification management logic applies.

According to various embodiments, notification manager550can output instruction565b, wherein notification(s)510are displayed to the user. For example, instruction565bcan be issued when the current importance of notification(s)510is high relative to the current probability that the HMD user is in a real-world conversation (i.e., where the notification is presently important and/or the probability that the user is in a conversation is low).

As shown inFIG.5, in some embodiments, notification manager550may output instruction565c, wherein notification(s)510are not presently displayed but are instead added to an unseen notification stack maintained in a memory of the HMD. For example, instruction565bcan be issued when the current importance of notification(s)510is low relative to the current probability that the HMD user is in a real-world conversation (i.e., where the notification is relatively unimportant and/or the probability that the user is in a conversation is high). Returning to the example ofFIGS.4A and4B, instruction565cmight be issued when notification415pertains to a multi-day sale at a local supermarket, and thus does not require immediate attention.

In certain embodiments, notification manager550can output instruction565d, wherein an unseen notification(s) stack is cleared, and notifications are pulled from a queue of notifications to be displayed. In some embodiments, the unseen notification(s) stack can be cleared in conjunction with another instruction, such as instruction565b, as displayed notification(s) do not need to be maintained in an unseen notification stack. In certain embodiments, the current importance weight of the notification is periodically recalculated, and the importance of the notification(s) in the unseen notification(s) stack may drop below a threshold value for display. For example, where an application has issued subsequent similar notifications, making the unseen notification(s) duplicative and/or irrelevant.

FIG.6illustrates a process600for determining an instant value of a variable embodying an instant probability (for example, a calculated value of UserInConversationInstant( )) that a user is in a real-world conversation. WhileFIG.6depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. The process600depicted can be implemented by one or more processors in mobile electronic device, such as by one or more processors140of an electronic device100or processor211of HMD200. In certain embodiments, the operations of process600may be performed at a single processing platform (i.e., an HMD providing an AR display), or across multiple processing platforms (for example, an HMD connected to another processing platform, such as a smartphone or tablet computer). In certain embodiments, process600is performed continuously, at a predetermined intervals (for example, at a first, faster rate if a user has been determined to presently be in conversation, and at a second, slower rate if a user has not been determined to be in conversation recently) whenever the HMD is in use.

At operation605, the processor determining an instant value of UserInConversationInstant( ) obtains predefined weights for one or more factor modules. As used in this disclosure, the expression factor module encompasses a set of rules or control logic for determining, based on application data and/or sensor data (for example, data obtained from a camera of the HMD or accessory device with a field of view overlapping that of HMD user's field of view, or a microphone of the HMD or accessory device capable of picking up some or all of the same audio as the HMD user's ears) whether the received data satisfies the criteria for a conversation event.

A non-limiting set of example of factor modules according to the present disclosure is provided in Table 1, below:

“In Conversation” EventFactor ModuleInputsTriggered By:Face in defined regionSensor data from cameraIf a face is within a definedOutput of facialregion (for example, anrecognition algorithmestimated center of the HMDuser’s field of view, or alocation within a specifiedrange of the user) for longerthan a threshold time, InConversation event istriggered.Face recognizedSensor data from cameraIf a face recognized as belongOutput of facialto a database of facesrecognition algorithmassociated with the HMD useris in a defined region forlonger than a threshold time,an In Conversation event istriggered.Voice in defined regionSensor data from one orUsing one or more of themicrophonesHMD microphone and theOutput of audiomicrophone of a connectedtriangulation algorithmaccessory device, detecting thelikely location of audiorecognized as a voice. Whenthe tracked voice is within adefined region (for example,within a given proximity, or infront of the HMD user) for atleast a threshold time, an InConversation event istriggered.Recognized voiceSensor data from one orUsing one or more of themicrophonesHMD microphone and theOutput of voicemicrophone of a connectedrecognition algorithmaccessory device, detecting thelikely location of audiorecognized as a previouslyrecognized voice. When thetracked voice is within adefined region (for example,within a given proximity, or infront of the HMD user) for atleast a threshold time, an InConversation event istriggered.

Further examples of factor modules include a factor module assessing an instant value of a likelihood that a user is in conversation based on a combination of location and/or movement of a device. For example, if location and movement data show that a user is engaged in an activity where conversation is theoretically possible, but less likely (for example, running or cycling at a high rate of exertion), then the output of a location/movement factor may increase or decrement the probability of a user being in conversation to reflect such data. Other possible factor modules include a factor module whose inputs are drawn from a log of user interactions with the HMD. Thus, where an interaction log data shows that a user has recently interacted with one or more applications presented through the HMD, the factor module may return a value indicating that the user's attention is more likely to be focused on digital content than a real-world conversation. Similarly, where an interaction log indicates that a user has not recently interacted with any applications, the factor module may return a value indicating that the user's attention is more likely presently focused on real-world events, rather than AR content provided through the HMD. Although several factor modules are depicted, others could be used without departing from the scope of the present disclosure.

At operation605, each of the factor modules may be given a predefined weight. According to some embodiments, the assignment of predefined weights may be purely-rule based and responsive to various contextual factors. As one illustrative example, a GPS sensor or WI-FI signal indicates that a user is at home, the weights assigned to predefined weights for detected voices may be decreased, as conversations with familiar voices may be a baseline condition of the environment, and not strongly correlative of an HMD user being a real-world conversation. Alternatively, or additionally, the predefined weights for the one or more factor modules may be dynamic and tuned based on user experience. In some embodiments, an HMD user may provide feedback data as to the contextual appropriateness of notification management, from which the notification manager (for example, notification manager550inFIG.5) may be trained. Further, in certain embodiments, certain factor modules may be selectively activated or deactivated (i.e., given a factor weight of zero) depending upon context. In one example, a calendar application indicates that, for a given time window, a user is going to be at an inherently noisy event (for example, a concert or sporting event at a stadium) the voice-related factor weights may be reduced or set to zero to compensate for the noisiness and abundance of ambient conversation at the event.

According to various embodiments, at operation610, each factor module (for example, the factor modules described in Table 1 of this disclosure) is run based on the most recently available data as to the HMD user's environment (for example, sensor data from cameras, microphones, and motion and location sensors). In some embodiments, each factor module for which a non-zero weight assigned at operation605is run in parallel and independently of each of the other factor modules. In some embodiments, factor modules are run conditionally with one another, such that an “in conversation” event is returned only when the triggering conditions of multiple factor modules are satisfied within a designated interval. For example, certain contexts (for example, when an HMD user is at home) the presence of a recognized face or hearing a known voice, may, by themselves be poorly correlative of whether an HMD user is in a real-world conversation, as, given the nature of shared households, the user is likely to frequently see their family members and hear their voices somewhere in the house. However, the confluence of seeing a person and triangulating their voice to a particular location relative to the HMD user, can, in certain settings, be highly predictive of whether a user is presently engaged in a conversation. Thus, in certain embodiments according to this disclosure, running each factor module at operation610comprises running logical combinations (for example, combinations defined by Boolean operators such as AND, OR, NAND, etc.) of one or more factor modules.

Referring to the illustrative example ofFIG.6, at operation615, a check is performed to determine whether any of the factor modules return a blocking value. For example, in some embodiments, a factor module may return one of the following three values of UserInConversationInstant( ): “0” indicating that no “in conversation” event was detected; “1” indicating that an In Conversation event was detected, and a blocking value “−1” indicating that the factor module detected a condition precluding display of a notification. Non-exhaustive examples of conditions that can trigger a blocking value include sensor data showing a particular individual (for example, the HMD user's spouse, or an individual recognized as a police officer) in a field of view for a threshold portion of time, or detecting a voice whose volume exceeds that of other sounds by a threshold amount (i.e., a factor indicating that someone is likely shouting to get the HMD user's attention). Where one or more factor modules return a blocking value, process600proceeds to operation620, wherein notification(s) are blocked from being displayed as part of the AR display. In this way, certain embodiments according to the present disclosure can implement rules-based notification management policies for clear cases of when a notification is to be blocked (for example, when a voice recognized as an HMD user's teacher is speaking), but also perform a balancing test between the current likelihood of a user is in a real-world conversation against the current importance of the unseen messages for subtler, more nuanced situations.

If no blocking values are returned from running the factor module(s), process600proceeds to operation625, where the results of each (for example, “in conversation” event values) factor module are multiplied by the weighting factors assigned at operation605and summed to provide, for a given point in time, an instant value of a function (for example, UserInConversationInstant( )) associated with an instant probability, that based on sensor and application data obtained at a specified time interval, the HMD user is currently in a real-world conversation.

According to various embodiments, at operation630, weighted results of each factor module are returned to a notification manager (for example, notification manager550inFIG.5). In certain embodiments, a time weighted average of the value of the function associated with the probability that the user is in a real-world conversation is updated and recalculated based on a set of values which includes the most recently obtained value of the function determined at operation625is used to determine, and from which the oldest value of the function is removed from the calculation set.

FIG.7illustrates a process700for determining current value associated with importance of unseen notification(s) (for example, notifications510inFIG.5) received by a notification manager, according to various embodiments of this disclosure. WhileFIG.7depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. The process700depicted can be implemented by one or more processors in mobile electronic device, such as by one or more processors140of an electronic device100or processor211of HMD200. The process700can be performed across one or more processing platforms. For example, in some embodiments, process700may be performed entirely at an HMD through which an HMD user is viewing an AR display. In various embodiments, process700is performed at a processor of an accessory device (for example, a smartphone) connected to and supporting an AR display provided through the HMD.

Referring to the non-limiting example ofFIG.7, at operation705, a notification manager (for example, notification manager550inFIG.5) according to various embodiments of this disclosure, receives one or more notifications to be displayed in an AR display provided by the HMD. In some embodiments, the notifications are provided by applications running on the HMD or may be notifications provided by the HMD itself (for example, a low battery warning notification). At operation705, the received notifications are added to an unseen notification(s) stack maintained by the notification manager, pending analysis at operations710-725to determine a current notification priority value or whether any factors superseding calculating a current notification priority value are applicable.

As shown in the explanatory example ofFIG.7, at operation710, the notification manager assigns an initial weight to each notification. According to various embodiments, the initially assigned weights may be static, and set by platform developers. In some embodiments, the initially assigned weights may be user-tunable, with users able to specify aspects of their preferences for how long presentation of notifications is deferred. In some embodiments, the assigned weights may be initially set by platform developers but modified through learned adjustments based on training data from user interactions with the HMD. Additionally, in some embodiments, at operation710, the notification manager analyzes the received notifications to determine whether the notifications belong to a type designated as superseding calculation (i.e., notifications that should be presented to a user, regardless of whether they are in a conversation). Non-limiting examples of notifications superseding a calculation include notifications regarding calls or texts from predefined users (for example, an HMD user's mother) or safety related notifications (for example, a flash flood warning from a weather application). Where a notification belongs to a type designated as superseding calculation, it is displayed, and removed from the unseen notification stack, and the remaining notifications in the unseen notification stack are passed to operation715.

In certain embodiments, the determination of initial weights at operation710is based on a further analysis of simply the type of each received notification in the unseen notification stack. For example, in some embodiments, the notification weights assigned at operation710may be based on patterns in the unseen notification stack (for example, where two calls from the same number are received, the total priority weight for the calls may be higher than if the calls were received separately). As another non-limiting example, where one notification supersedes another notification (for example, where a restaurant application provides a first notification advising the HMD user of a 15-minute wait time, and a second notification advising the user that their table is ready). In this example, the second notification advising that the HMD user's table is ready moots the first notification advising of a 15-minute wait for a table.

At operation715, a sum of the unseen notification weights in the unseen notifications is determined. According to various embodiments, where unseen notifications accumulate in the unseen notification stack, the baseline expectation is that the priority of the notifications increases. Put differently, while certain embodiments according to this disclosure may apply a gain factor or other adjustment to account for the effects of time on the relative priority of the unseen notifications, the pre-gain adjustment expectation is that, the more unseen notifications, the greater the priority that a user look at their unseen messages.

As noted previously, certain embodiments according to this disclosure account for the fact that the real priority of notifications is dynamic, with certain types of notification having a priority that increases over time, and other types of notification having a decaying priority. To reflect this fact, at operation720, one or more gain functions are applied to the summed weights of the unseen notifications obtained at operation715. According to various embodiments, a single time-based gain factor may be added to the sum of the unseen notification weights, such that the priority of the unseen notifications increases linearly with time. According to various embodiments, a multi-variable gain factor may be applied. For example, the notifications in the unseen notifications stack may be divided into notifications whose priority increases with time and notifications whose priority decreases with time, and positive gain values (for the applications with increasing priority) and negative gain values (for applications with decreasing priority) are determined and summed to determine an overall time-based gain factor. In this way, an HMD user's real-world conversations may be less likely to be interrupted with unwanted notifications, when the unseen notification stack fills up with low-priority notifications.

As shown inFIG.7, at operation720, the time-based gain factor is applied to the summed weights of the unseen notifications in the unseen notifications stack to determine a current value of the importance of the unseen notifications currently in the unseen notification stack. The current value of the importance of the unseen notifications currently in the unseen notification stack is returned to the notification manager to be compared against a present value of the likelihood (based on an instant value or time weighted average value of a function associated with the probability of a user being in a real-world conversation) that the HMD user is currently engaged in a real-world conversation.

Table 2, below, provides an illustrative example of notification management according to various embodiments of this disclosure over a series of points in time. In the example of Table 2, the variable “UC” refers to the current moving average of a variable (for example, UserInConversationInstant( ) associated with the probability that an HMD user is in a real-world conversation. In this example, the variable “NRW” is a quantification of the relative weights of the notifications in an unseen notification stack.

TABLE 2TimeActionUCNRWComments1Notification received by01Notification displayed, as user is not innotification manager, whileconversationthe HMD user is notpresently in conversation.2User begins real-world00conversation3Using camera and40microphone data, systemrecognizes a face and a voice4User receives two social42Social media notifications are not displayedmedia notifications.because UC > NRW5Conversation continues.42.1NRW increases according to a gainfunction adding 0.1 to NRW for eachminute a notification goes unseen.6Conversation continues for44.1Notifications shown to user, because20 minutes.NRW > UC at this time.7Conversation continues.40Unseen notifications stack is cleared asnotifications were displayed to user.8User begins to walk away,20sensors detect conversationpartner’s voice, but face isno longer detected in cameradata.9User walking away,1.50Moving average of UC begins moving toconversation partner’s voicezero as factor modules no longer return Inno longer audible, face noConversation events.longer visible.

FIG.8illustrates a process800for performing conversation-based notifications according to various embodiments of this disclosure. WhileFIG.8depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. The process800depicted can be implemented by one or more processors in mobile electronic device, such as by one or more processors140of an electronic device100or processor211of HMD200. The operations of process800may, in embodiments, be performed at a single computing platform (for example, HMD200inFIG.2). Alternatively, or additionally, the operations of process800may be performed in whole, or in part, at a processing platform communicatively connected to the HMD. For example, in some cases, it may be desirable from a battery life and heat management perspective, to minimize, wherever possible, the amount of processing performed at the HMD. Accordingly, in such embodiments, notification management may be offloaded to an accessory device, such as a smartphone or smart watch communicatively connected to the HMD.

Referring to the illustrative example ofFIG.8, one or more applications providing an AR display at the HMD is executing and providing an AR display to a viewer. The AR display comprises elements of a real-world view from the perspective of the HMD user (for example, what the user sees through a clear screen of the HMD) and can also comprise elements of digital content provided in conjunction with the real-world view.

According to various embodiments, at operation810, a notification manager (for example, notification manager550inFIG.5) executing at the HMD or an accessory device, receives a notification from an application or a processing platform (for example, a system notification, such as a low battery warning or a system update reminder) to be presented through the HMD. In certain embodiments, the received notification is added to an unseen notification(s) stack, pending determination of factors associated with a likelihood that a user is presently in a real-world conversation and determination of the current priority of the unseen notifications in the unseen notification(s) stack.

As shown in the illustrative example ofFIG.8, at operation810, the system receives first sensor data obtained at a first time. According to various embodiments, the first sensor data is obtained from one or more of a camera or a microphone located on the HMD, or an accessory device at another location (for example, a smartphone in a user's shirt pocket) providing a useful proxy for the sounds and sights obtained at perspective of the user's eyes and ears. According to various embodiments, application and system data, for example, data showing the location or movement of the HMD user, may also be obtained at operation815.

According to various embodiments, at operation820, the sensor data obtained at operation815is provided to one or more factor modules (for example, one or more of the factor modules described in Table 1 of this disclosure), to obtain values of one or more factors associated with the probability of the user being in a real-world conversation. In some embodiments, at operation820, a weighting factor may be applied to the values output by each of the factor modules. When the obtained sensor data is provided to more than one factor module, a sum of the weighted outputs of the factor modules may be calculated. Additionally, in certain embodiments, at operation820, the system may perform a determination (for example, as shown in operation615inFIG.6) to see if any factor modules return a blocking value, superseding the need for further analysis of conversation factors. According to some embodiments, at operation820, the system determines a current value of a moving average of the value of the probability that the user is presently in a real-world conversation.

As shown in the explanatory example ofFIG.8, at operation825, the system or a component thereof (for example, notification manager550inFIG.5) determines an importance value of the notifications currently in the unseen notification(s) stack. According to various embodiments, determining the importance of the received notification(s) comprises assigning predefined weights to each notification, taking a sum of the predefined weights and then applying a time-based gain factor to the sum of the predefined weights.

According to various embodiments, at operation830, the notification manager performs a comparison of the current value of the probability of the HMD user being in a real-world conversation at the first time against the current importance value of the unseen notification(s). Where the current importance value of the unseen notification(s) exceeds that of the current value of the probability of the HMD user being in a real-world conversation, the notifications are displayed through the HMD, and the unseen notification(s) stack is cleared. Similarly, where the current importance value of the unseen notification(s) is less than that of the current value of the probability of the HMD user being in a real-world conversation, the notification(s) are kept in the unseen notification(s) stack, and the relative values may be recalculated based on sensor data obtained at a second time.