Patent Description:
Two examples of approaches for determining a user's perception of a stimulus are as follows. One approach, used in current consumer electronics, relies on explicit user interaction. Perception is assumed once a user manually acknowledges the item or stimulus or opens the application and/or conversation that generated the notification. For example, an email notification that appears may only be dismissed once a user takes some explicit action associated with the email application, e.g., bringing the email application to the foreground, manually opening the message, or labelling it as "read". Similarly, an incoming video call within an XR environment may continue generating a notification until the user has acknowledged it, either by answering or declining the call (or a timeout occurs and the call is automatically terminated).

Another approach attempts to predict if a signal will be perceived prior to its delivery based on user context attributes and properties of the stimulus. For example, based on a user's age and current activity, a prediction may be made of whether a vibrotactile signal with a particular vibration intensity is likely to be perceived by the user. However, such reliance on discrete activity recognition as a means of predicting perception can be impractical. A prediction of whether a vibrotactile signal will be perceived can be improved by using aggregated continuous accelerometer measurements to model the amount of interfering haptic stimuli prior to the delivery of the vibrotactile signal. Such a method allows for the adjustment of stimulus properties to increase the probability that it is perceived, but it does not confirm post-presentation whether the user did, in fact, perceive the signal, without relying on explicit user interaction.

<NPL>, evaluated perception of smartphone notifications using skin conductance measurements. The reliability of such a technique, however, can be diminished by user activities that corrupt the skin conductance signal. <CIT> "Determining user response to notifications based on a physiological parameter" describes the use of a physiological parameter to determine user response to notifications. <CIT> "Rendering of a Notification on a Head Mounted Display" describes a method of determining occurrence of a virtual information region event indicating a change of information that is allocated to a virtual information region. Further pertinent prior art is described in the patent documents <CIT>, <CIT>, <CIT>, and <CIT>.

Current XR systems are generally incapable of accurately determining whether a notification, message or in-experience stimulus has been successfully perceived without explicit interaction from the user, e.g., a button-press acknowledgement. This can result in two problematic situations. One situation is a false positive (type I error), where the system erroneously assumes that the stimulus was perceived. In such case, users may miss some content related to the XR experience or delivered through the XR system to communicate critical information related to the physical world. For example, in most systems, vibrotactile and auditory notifications are only delivered once using a simple haptic or auditory cue. Failure to perceive these cues will necessarily delay the user's response to the event, potentially resulting in missed communication opportunities. Another situation is a false negative (type II error), where the system incorrectly assumes that a notification has not been perceived. In such case, it may repeat the notification redundantly, significantly disrupting users as they are engaging with their primary activity. For example, an incoming call will keep "ringing" until the user decides either to answer or dismiss it. Similarly, a visual notification may remain on the screen for an extended period of time, despite the fact that it was perceived by the user. These stimuli place undue demands on the user's limited sensory and cognitive resources, ultimately impacting their productivity and well-being.

A method according to the invention is defined in claim <NUM>. An apparatus according to the invention is defined in claim <NUM>. The invention as claimed is best understood in light of the embodiments described in the context of <FIG> and <FIG>. Other embodiments may or may not describe the complete combination of features as claimed, but are useful in understanding the invention as claimed.

A method according to the invention, comprises: obtaining information indicating a current activity of a user; obtaining a first signal representing a change in a first physiological parameter of the user between a time before and a time after presentation of an information item to the user is initiated; obtaining a second signal representing a change in a second physiological parameter of the user between a time before and a time after presentation of the information item to the user is initiated; and determining, using at least the first and second signals, whether the user has perceived the information item, wherein a contribution of at least the first signal to the determination is weighted by an amount based on the current activity.

In some embodiments, obtaining the first signal comprises obtaining a first measurement of the first physiological parameter of a user from a time before presentation of the information item to the user;
obtaining a second measurement of the first physiological parameter of the user from a time after the presentation of an information item to the user is initiated; and generating the first signal based on a difference between the first measurement and the second measurement.

In embodiments according to the invention, the weighting of the first signal is determined based on a corruption coefficient associated with the current activity.

In embodiments according to the invention, the corruption coefficient is based at least in part on an amount by which the current activity of the user is expected to interfere with the first physiological parameter.

In some embodiments, determining whether the user has perceived the information item comprises applying a first weight to the first signal and a second weight to the second signal, the second weight being different from the first weight.

In some embodiments, the information item is a notification.

In some embodiments, the information item is presented in an extended reality (XR) experience.

In some embodiments, the first physiological parameter is a parameter selected from the group consisting of: skin conductance, photoplethysmography (PPG), electrocardiography (ECG), electroencephalography (EEG), electromyography (EMG), electrooculography (EOG), skin temperature, heat flux, abdominal respiration, thoracic respiration, pupillometry, and gaze tracking.

In some embodiments, in response to a determination that the user has perceived the information item, dismissing the information item.

A method according to some embodiments comprises: in response to a determination to present an information item to a user, initiating a first measurement of a physiological parameter of the user for a first time period; delaying presentation of the information item to the user until after the first time period; conducting a second measurement of the physiological parameter for a second time period after initiation of the presentation of the information item to the user; and determining, based at least in part on a comparison between the first measurement and the second measurement, whether the user has perceived the information item.

Some embodiments further comprise, in response to a determination that the user has perceived the information item, dismissing the information item.

Some embodiments further comprise obtaining information indicating a current activity of a user, wherein the determination of whether the user has perceived the information item is based at least in part on the current activity of the user.

An apparatus according to the invention comprises a processor configured to perform at least: obtaining information indicating a current activity of a user; obtaining a first signal representing a change in a first physiological parameter of the user between a time before and a time after presentation of an information item to the user is initiated; obtaining a second signal representing a change in a second physiological parameter of the user between a time before and a time after presentation of the information item to the user is initiated; and determining, using at least the first and second signals, whether the user has perceived the information item, wherein a contribution of at least the first signal to the determination is weighted by an amount based on a first corruption coefficient associated with the current activity, the first corruption coefficient being based at least in part on an amount by which the current activity interferes with the first physiological parameter.

In example embodiments, a method includes obtaining a first measurement of at least a first physiological parameter of a user from a time before presentation of an information item to the user. The information item may be, for example, a notification, an advertisement, or an emergency alert, among other possibilities. The information item may be another experience-relevant event or information, such as an event occurring in a movie, a games, or the like. A second measurement of the first physiological parameter is obtained from a time after a beginning of the presentation of the information item to the user (e.g. during the presentation). Based at least on a comparison between the first and second measurements, a determination is made of whether the user has perceived the information item. The determination may also be based on a corruption coefficient indicating an amount by which an activity of the user is likely to interfere with the first physiological parameter.

In some embodiments, the corruption coefficient is based on a determined activity being performed by the user. In some embodiments, the corruption coefficient is further based on motion specific to an on-body measurement site, such as the user's arms or legs.

In some embodiments, pre- and post-presentation measurements of a plurality of physiological parameters (possibly weighted based on respective corruption coefficients) are used in determining whether the user has perceived the information item.

As shown in <FIG>, the communications system <NUM> may include wireless transmit/receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN <NUM>, a CN <NUM>, a public switched telephone network (PSTN) <NUM>, the Internet <NUM>, and other networks <NUM>, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and/or network elements.

Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN <NUM>, the Internet <NUM>, and/or the other networks <NUM>.

The base station 114a may be part of the RAN <NUM>, which may also include other base stations and/or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and/or the base station 114b may be configured to transmit and/or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown).

<FIG> is a system diagram illustrating an example VIΓΓRU <NUM>.

More specifically, the VIΓΓRU <NUM> may employ MIMO technology.

It will be appreciated that the WTRU <NUM> may acquire location information by way of any suitable locationdetermination method while remaining consistent with an embodiment.

The peripherals <NUM> may include one or more sensors, the sensors may be one or more of a gyroscope, an accelerometer, a hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric or other physiological sensor, and/or a humidity sensor.

In an embodiment, the WTRU <NUM> may include a half-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).

11e DLS or an <NUM>. The IBSS mode of communication may sometimes be referred to herein as an "adhoc" mode of communication.

In view of <FIG>, and the corresponding descriptions, one or more, or all, of the functions described herein may be performed by one or more emulation devices (not shown). For example, the emulation devices may be used to test other devices and/or to simulate network and/or VIΓΓRU functions.

Example embodiments use changes in physiological signals in response to the delivery of notifications and other information presentation in an extended reality (XR) experience to determine whether the stimulus was perceived by the user. Some embodiments combine multiple streams of physiological data (e.g., skin conductance, heart rate, gaze trajectories, EEG, etc.) based on the user's current activity and instantaneous motion measurements, such that measurements acquired at body sites that are more susceptible to motion artifacts carry less weight in the perception classification algorithm.

Example embodiments provide an XR system (or other system capable of presenting notifications) with automated perception confirmation of visual, auditory, haptic, gustatory and/or olfactory stimuli through detection of changes in users' physiological signals caused by the perception of the stimulus. These changes occur in response to the perception of the in-experience stimuli, which may be caused by the stimuli's sensory characteristics, psychophysiological impact and/or relevance to the user. In addition, example methods employ a context-dependent combination of physiological signals to improve signal quality and redundancy. Examples of such physiological signals include, but are not limited to, combinations of skin conductance, skin temperature, heart rate, respiration rate, pupil dilation, eye blink and gaze trajectories extracted using wearable and/or remote sensors.

In one example embodiment, an XR system or other system determines that there is a notification or other sensory information to be presented to the user. This information may be presented using a single sensory modality or a combination of different sensory modalities depending on the hardware available to the user. The system initiates the collection of multi-channel physiological signals using wearable and/or remote physiological sensing solutions prior to the presentation of the sensory information to gather contextualized baseline measurements. The system delivers the sensory information to the user using available peripherals, e.g., headset, hand controllers, and the like. The system collects the physiological signals until a prescribed time has passed post-stimulation. The system computes or otherwise determines appropriate features for each physiological sensing channel for the baseline measurements and from post-stimulation measurements. The system uses the user's current application in combination with a context-based activity recognition system to estimate the likelihood of signal corruption for one or more of the physiological sensing channels. The likelihood of corruption may be represented in some embodiments by corruption likelihood coefficients. The system compares pre- and post-stimulation features per-channel to determine whether significant changes were caused by perceiving the stimulus. The corruption likelihood coefficients may be used to weight the contribution of each physiological sensing channel in the overall classifier. In some embodiments, this is performed using a meta-learning algorithm. If no difference or a sufficiently small difference is apparent between the pre- and post-stimulation features, this may be interpreted as an indication that the user has not perceived the stimulation. If, however, a difference is detected above a threshold (e.g. a statistical threshold) between the pre and post-stimulation features, this may be interpreted as an indication that the stimulus has been perceived. In some embodiments, further perceptual and experiential information such as changes in psychophysiological states may be extracted from the physiological signals by investigating effects within a channel and/or cross-channel effects. The system may then return a signal that indicates whether the information was perceived, along with, when applicable, the changes in psychophysiological state induced by the information being presented. The system may repeat the information presentation or other stimulation (e.g. in response to an indication that the information was not perceived), silence or otherwise dismiss the information (e.g. in response to an indication that the information was perceived), or take any other action as appropriate to its interaction objectives.

In some embodiments, there is no need for users to train, or even learn how to use the interface, since the system responds to natural changes in users' physiological states.

By automatically acknowledging the perception of information (e.g., notifications, messages or other stimuli) being delivered to users, the system may allow the users to remain focused on their primary task and preserve their cognitive budget for activities that are aligned with their current objective. For example, an email notification delivered during the design of a new product in an XR environment can be automatically dismissed based on the perception feedback provided by the proposed system, allowing users to allocate their cognitive reserves to their primary task.

In some embodiments, by offering a more natural means of interacting with, and automatically acknowledging the perception of in-experience stimuli, users can maintain better immersion in the XR experience. The psychophysiological feedback offered in some embodiments through the combination of multiple channels of physiological sensors allows for adaptation of the content to provide richer experiences to XR users.

To allow the confirmation of XR stimulus perception, example embodiments use at least one physiological sensor positioned adequately to acquire signals from a user. The physiological sensor(s) may be wearable, or they may use remote physiological sensing techniques. The sensors may be in communication with a processor or other computer controlling the XR experience, as illustrated in <FIG>.

<FIG> is a schematic block diagram illustrating hardware architecture employed in some embodiments. In the example of <FIG>, a plurality of physiological sensors <NUM>, <NUM>, <NUM> collect physiological data from a user <NUM> of an XR system. Data from the physiological sensors is provided to a processor <NUM> that is configured to perform any of the methods as described herein to determine whether a user has perceived an information item.

<FIG> is a visualization of a timeline of pre- and post-stimulation measurements used in some embodiments. At time <NUM>, a determination is made to present an information item to a user. During period <NUM>, baseline physiological measurements are collected. At time <NUM>, the presentation of the information item to the user is initiated. During period <NUM> after initiation of the presentation of the information item, additional physiological measurements are collected. At time <NUM> a determination is made based on the physiological measurements of whether the user has perceived the information item, and that determination may be relayed to the system that presented the information item. For example, if a determination has been made that the information item has been perceived, then the item may be marked as read and/or may be dismissed.

In some embodiments, confirmation of stimulus perception from physiological signals may be achieved using the following procedure, as illustrated in <FIG>.

The system (e.g. an XR controller) initiates physiological signal data collection before the presentation of the notification, message, haptic effect, or other sensory stimulus. In some embodiments, initiation of data collection may begin <NUM> to <NUM> seconds before the presentation, although other time periods may be used.

The exact amount of data collection time before and after stimulation may be dependent on the physiological signals used in a specific embodiment of the invention. For example, pupil dilation, EEG, heart rate and respiration signals operate on a much shorter time scale than skin conductance and skin temperature. Physiological signals that operate on a shorter time scale may be monitored for a shorter data collection time (e.g. <NUM>-<NUM> seconds), and physiological signals that operate on a longer time scale may be monitored for a longer data collection time (e.g. <NUM>-<NUM> seconds) to be able to make practical perception inference. In some embodiments, particularly if power consumption is not a concern, physiological signals may be continuously monitored and logged.

In some embodiments, contextual information is collected to predict the user's current activity and the user's level of physical engagement in a current task. In some embodiments, the contextual information is collected simultaneously with the collection of pre-stimulation physiological signals. The collection of contextual information may be used to identify a discrete activity (e.g., standing up, walking, sitting, running, jumping) and combinations of discrete activities associated with the user's task (e.g., in-place running and jumping, sitting and standing up, etc.). An initial set of signal corruption coefficients may be determined based on the identified discrete activity or activities (see Table <NUM> for examples). This information may be obtained directly from an analysis of the XR experience, or by using one or more known activity recognition frameworks.

Some embodiments further operate to determine the user's level of physical engagement with the task at hand. In some embodiments, this is achieved by aggregating passive sensor data (e.g., accelerometer, gyroscope, magnetometer, motion tracking data, etc.) at one or more body locations to determine, given the discrete activity, which body parts are the most physically active during the user's activity and how active they are in relation to one another. For example, when sitting, the user's head usually experiences less motion than hands and upper body. However, in some cases, the user may be moving their head a lot because of an XR experience that calls for them to look around themselves. The resulting order may then be, from the less active to most active upper body, head, hands. A engagement-based corruption coefficient ranging between, for example, <NUM> and <NUM> may then be attributed based on the ranked list of active body parts, as shown in the example of Table <NUM>.

These results may be combined using, for example, the equation below to compute a corruption likelihood coefficient for each available physiological channel. The discrete activity may be used to obtain initial corruption likelihood coefficients for each physiological channel based on the user's activity. The physical engagement parameters may be used to customize the activity-based corruption coefficients such that they are better representative of the user's context. A weight array (w) used in the meta learning algorithm may be computed from the activity-based corruption coefficient array (CorrAct) and engagement-based corruption coefficient array (CorrEng) using the following formula: <MAT>.

The system (e.g. the XR controller) causes the sensory information to be presented to the user using the headset, haptic controller or any other relevant interface.

The system may stop collecting physiological data after stimulus presentation. For example, the system may stop collecting physiological data <NUM>-<NUM> seconds, or some other period, after stimulus presentation.

As illustrated in <FIG>, an example method includes performing activity recognition <NUM> to determine an activity of a user. Data is collected (<NUM>) of a first physiological parameter before presentation of an information item, and data is collected (<NUM>) of a second physiological parameter before presentation of the information item. In some embodiments, the collection of physiological data before the presentation of the information item may be performed for between <NUM> and <NUM> seconds.

At <NUM>, the presentation of an information item, such as a notification, is initiated. In some embodiments, the presentation of the notification may be delayed to allow collection of the physiological data. For example, in response to a determination to present physiological data, the data collection <NUM> and <NUM> may be initiated, and the notification may be presented to the user only after sufficient data has been collected to establish a baseline, e.g. between <NUM> and <NUM> seconds, or between <NUM> seconds to <NUM> seconds. The length of the delay and of the attendant data collection may depend on the physiological parameter(s) being collected. The use of a delay may help to limit the consumption of energy and computing resources that may otherwise be used in embodiments where physiological data is collected continuously.

After presentation of the information item has been initiated, additional data is collected (<NUM>) of the first physiological parameter, and additional data is collected (<NUM>) of the second physiological parameter. In some embodiments, the data collection after the presentation of the information item is initiated continues for between <NUM> and <NUM> seconds. In some embodiments, data collection of the physiological parameters may be continuous, with some data being collected before and some collected after the presentation of the information item is initiated. In some embodiments, the information item continues to be displayed after the presentation of the information item is initiated, and display of the information item may continue, for example, for a predetermined amount of time or until it is determined that the user has perceived the information item.

Based on the data collected regarding the first physiological parameter, a first signal is prepared or otherwise obtained (<NUM>) that represents a change in the first physiological parameter of the user between a time before and a time after presentation of an information item to the user. A second signal is prepared or otherwise obtained (<NUM>) that represents a change in the second physiological parameter of the user between a time before and a time after presentation of the information item to the user. In embodiments in which data is collected for additional physiological parameters, additional signals may be prepared or otherwise obtained that represent changes in those respective parameters between a time before and a time after presentation of the information item to the user.

The signals that represent changes in physiological parameters may be obtained using one or more of a variety of techniques. In some embodiments, the signal may represent a difference (including, e.g., an absolute value of a difference or a square difference) between a pre-stimulus average value of the parameter and a post-stimulus average value of the parameter, or between a pre-stimulus maximum or minimum, value of the parameter and a post-stimulus maximum or minimum value, or the like. In some embodiments, the signal may represent a difference between a pre-stimulus frequency of the parameter and a post-stimulus frequency of the parameter. In some embodiments, the signal may represent a difference between a pre-stimulus variability or range of the parameter and a post-stimulus variability or range of the parameter.

Using the first and second signals, a determination is made (<NUM>) of whether the user perceived the information item. In some embodiments, a contribution of the first signal to the determination is weighted (<NUM>), and/or a contribution of the second signal to the determination is weighted (<NUM>). In some embodiments, the strength (or other quantity) of the weights used for the first and/or second signal are determined (<NUM>) based on the user's activity as determined by the activity recognition <NUM>. In some embodiments, the weights are corruption coefficients or are determined based on corruption coefficients. In some embodiments, a measurement or other estimate is made of the magnitude of the user's motion (<NUM>), and the determination of the one or more of the weights is further based on an estimate of the magnitude of the motion.

The collected physiological signals may be pre-processed (e.g., filtered, cleaned, etc.) using existing automated signal cleaning procedures as deemed relevant depending on the physiological signal. For example, a band-pass filter may be applied to skin conductance measurements to a range of approximately <NUM>-<NUM> to remove low frequency trends, noise introduced by small changes in pressure, and power frequencies (<NUM>-<NUM>). Signal correction methods may also be employed to automatically "correct" signal segments that are sensitive to motion artifacts or other large sources of noise.

Features may be extracted from one or more of the physiological sensing channels within the pre- and post-stimulation measurements. Such features may be dependent on specific physiological signals. Table <NUM> presents examples of such relevant features for examples of physiological signals. The physiological signals may be, but are not limited to, any combination of one or more of the following: skin conductance, photoplethysmography (PPG), electrocardiography (ECG), electroencephalography (EEG), electromyography (EMG), electrooculography (EOG), skin temperature, heat flux, abdominal and/or thoracic respiration, pupillometry and gaze tracking. These measurements may be achieved using, for example, traditional wearable sensors, head mounted display (HMD) integrated sensing apparatuses, or remote physiological sensing methods.

A binary classification algorithm or other technique may be employed to compare extracted features to determine, for each channel, whether the channel indicates that the stimulus was perceived by the user. <FIG> is an example illustrating a method in which pre-stimulation and post-stimulation measurements are compared, e.g. for skin conductance signals. The classification results for the available physiological channels may be combined using meta learning algorithms (e.g., ensemble methods or weighted majority algorithms) which may use the weights obtained above to increase the classification accuracy and reduce the impact of an individual channel on the classification outcome.

In an example method illustrated in <FIG>, pre-stimulation physiological measurements (<NUM>) and post-stimulation physiological measurements (<NUM>) are collected for at least one physiological parameter. In this example, each of the physiological measurements is processed using discrete deconvolution analysis, although other preprocessing techniques may be used in other embodiments. At <NUM>, a comparator compares the preprocessed signals to generate a signal representing a change in the physiological parameter of the user between a time before and a time after presentation of an information item to the user. At <NUM>, a classifier operates to determine whether the user has perceived the stimulation, and at <NUM>, the results of the classification are returned. The results may be used, for example, to determine whether to dismiss a notification and/or mark it as read in a case where the stimulus is a notification.

In some embodiments, in response to a determination that the information item was perceived, information from the different physiological signal channels may be combined to estimate the changes in psychophysiological states that they caused to the user. Interpretation of physiological signals may be performed using techniques such as those described in <NPL>. A decision tree using increase or decrease of features between pre and post-stimulation may be used to provide an estimate of the user's emotional state. More advanced machine learning methods may alternatively be applied to increase the emotion recognition accuracy.

The perception feedback may be relayed to the controller (e.g. XR controller) for further action.

In some embodiments, the processor receiving the physiological signals may be different from the one controlling the XR experience. In such case, a low latency communication channel may be established between the processors to provide timely initiation of physiological measurement and communication of the classification results.

In some embodiments, particularly where energy consumption is not an issue, the system may continuously measure physiological signals. In such embodiments, it may not be necessary for the system to wait for pre-stimulation measurements before presenting a stimulus, potentially making the overall process faster.

In some embodiments, particularly when the physiological signals have a high quality, richer information may be extracted from the physiological signals than a binary perception classification. In some embodiments, a classification is performed not only of whether a stimulus was perceived, but also how it was perceived, such as whether the stimulus negatively influenced the user's mood, cognitive load or other psychological construct.

Some embodiments may be implemented in an emergency alarm system. Residents in elderly and long-term health-care facilities often suffer from one or more sensory impairments. These sensory deficits may negatively impact their perception of emergency alarms (e.g., fire, carbon monoxide, evacuation order), to a point where they become partially or completely imperceptible. This introduces significant risks for the residents, but also for emergency responders that have to assist them. To address this issue, some facilities are equipped with visual indicators that activate simultaneously with the auditory alarms to catch the attention of residents with hearing impairments. However, not all facilities are equipped with such systems, nor does this resolve the perceptual challenges faced by individuals who also have visual impairments. In some embodiments, systems and methods are provided to confirm residents' perception of emergency alerts.

In an example of such an embodiment, residents may wear at least one physiological sensor in communication with a processor capable of implementing methods as described herein. When an alert is presented, this information may be communicated (e.g. simultaneously) to the patient's computing unit to activate the analysis of physiological signals. Alternatively, residents' physiological signals may be streamed by the processor to an intelligent alarm system to synchronize alarm presentation and residents' physiological responses.

In some embodiments, the residents' computing unit may be equipped with a microphone to capture and detect alarms. Once an alarm is detected, the perception prediction system may be activated to confirm whether the associated resident has perceived the alarm. The knowledge of which residents are aware of the alarm, combined with information on their mobility limitations and location may allow the care team and/or emergency responders to more efficiently organize an evacuation or otherwise direct emergency resources.

Some embodiments may be employed in the context of other types of alarms and alerts. For example, an embodiment used by a pedestrian or driver may operate to determine whether a user has perceived a car horn, an emergency vehicle siren, or a traffic or train signal, among other possibilities. The embodiment may operate to provide an additional alert to a user who has not perceived the outside stimulus. Some embodiments operate through communication, with, for example, an autonomous or semi-autonomous vehicle. In some embodiments, the vehicle may receive information from a system associated with a pedestrian or a driver of a different vehicle indicating whether that person perceived a particular stimulus, and the autonomous or semi-autonomous vehicle may use that information in its own operation. For example, the vehicle may give a wider berth or otherwise operate more cautiously around a pedestrian or a driver who has not been confirmed to have perceived an alert. In some embodiments, a system implemented in an autonomous or semi-autonomous vehicle may operate according to techniques described herein to determine whether its own driver and/or operator has perceived particular stimuli. Such a system may communicate that information to other vehicles and/or may use that information the operation of its own associated vehicle. For example, in response to a determination that the driver/operator has failed to perceive a stimulus, the vehicle may provide a higher level of automation and/or it may provide a supplemental alert (e.g. an audio signal or a light within the vehicle) that is more likely to be perceived by the user.

Some embodiments may be employed in an advertisement system. The pricing and effectiveness of advertisement campaigns are often based on anticipated "eyeball count" or similar measure. For example, an ad placed on a busy street will be more expensive, and is thought to be more effective, than one in a back alley, due to the number of people it is likely to reach. Some embodiments operate to confirm perception and/or interest towards an advertisement in, for example, an XR, web, TV, radio or physical environment.

In an example of such an embodiment, a viewer may wear at least one physiological sensor in communication with a processor capable of implementing methods as described herein. When an ad is presented, this information may be communicated to the viewer's physiological sensing unit to activate the analysis of physiological signals. In some embodiments, viewers' physiological signals may be streamed to nearby intelligent advertisement hubs to synchronize ad presentation and their physiological responses.

In some embodiments, the viewer's physiological monitoring system may be equipped with a microphone, camera or other sensor system to capture and detect displayed ads. Once an advertisement is detected, the perception prediction system may be activated to confirm that it was perceived. In some embodiments, the system may estimate the viewer's interest towards the ad.

Depending on the physiological sensors available, different levels of perceptual information may be inferred. A single channel may suffice in some embodiments to determine whether an ad was perceived. However, as more channels are used, more granular perception inferences may be implemented to estimate, for example, whether the content was significant/relevant to the user and/or what emotions the content induced. Such information may be used, for example, in an advertisement pricing scheme based on the probability that an advertisement is perceived either because of its saliency, or its relevance to the viewer, as reported using techniques described herein. Such methods may also allow advertisers to present content tailored to the viewer's history of perceived advertisements, allowing for a continuity in the ad experience and better control over the campaign's effectiveness.

In some embodiments, this perception history may be combined with information on other viewer activities, such as their internet browsing and recent purchases. Beyond determining viewers' perception of advertisements and understanding customer behavior, such combination of data may be used in determining a physiological "purchase intent" index. Such an index may be iteratively updated over time to optimize its predictive performance for the general population as well as specific users/user groups.

A use case for some embodiments is in a collaborative XR meeting. In such an embodiment, a user may be engaged in a collaborative XR experience in which she and her colleagues are discussing a new smartphone prototype they are developing. They are standing in circle around a small table, interacting with the smartphone prototype using their hands. While the user is explaining a new feature that was added to the prototype, she receives an email from her boss, which triggers a notification (visual, auditory and haptic) in her XR display. At that moment, she is standing, moving her hands while her head and upper body remains relatively stable. The signal corruption coefficients resulting from the combination of the activity-based coefficient and the engagement-based coefficient results can be found in Table <NUM>.

When perceiving the incoming email, the user's pupils dilate, her gaze is attracted towards the new information, her skin conductance peaks and her heart rate variability decreases. These changes in physiological signals are picked up by the system which uses them, with the weight array computed from the corruption coefficients, to determine that the user has perceived the incoming message. A schematic representation of the changes in each signal is provided in <FIG>. In addition to the binary perception classification, the changes in her signals are classified and the system comes to the conclusion that this event was stress-inducing to her.

In an example where the XR system employs a perception feedback system, the message is automatically placed in the background once it is perceived. The fact that this was a stressful event for the user is integrated in the information presentation system. The system may then avoid presenting this type of information to avoid increasing the user's stress level, promoting a richer and more positive experience in the future. The user in this example may continue her explanation of the new feature without any of her colleagues noticing the reception and acknowledgment of the email.

<FIG> is a schematic representation of physiological signals plotted over time illustrating changes in each physiological signal following the perception of a piece of relevant information in an XR environment. The time at which the information item is presented is illustrated by the dashed line <NUM>. Physiological data collected before presentation of the information item is illustrated in regions <NUM>, <NUM>, <NUM>, and <NUM>. Physiological data collected after presentation of the information item is illustrated in regions <NUM>, <NUM>, <NUM>, and <NUM>. Comparison between physiological data collected before versus after the presentation of the information item is used in determining whether the user has perceived the information item.

<FIG> is a sequence diagram illustrating a method performed in some embodiments using an XR controller <NUM>, an XR display <NUM>, a notification system <NUM>, at least one physiological sensor <NUM>, a signal pre-processor <NUM>, a perception classifier <NUM>, and an activity recognition module <NUM>. In an example method, the XR controller notifies the notification system of an event (e.g. an incoming email). The notification system requests corruption coefficient information from the activity recognition module. The notification system requests collection of physiological data from the physiological sensor. The notification system requests that the XR display render the notification. The notification system requests that the signal preprocessor process the collected physiological data. The notification system inquires of the perception classifier whether the notification was perceived. Based in part on the corruption coefficient(s) determined by the activity recognition module, the perception classifier determines whether the notification was perceived, and it informs the notification system of the result. If the notification is determined to have been perceived, the notification system requests that the XR display dismiss the notification.

<FIG> are graphs schematically illustrating the training and use of a classifier according to some but not necessarily all embodiments. <FIG> is a graph schematically illustrating data collection for classifier training according to some embodiments. The horizontal axis represents a first signal representing a change in a first physiological parameter of the user between a time before and a time after presentation of an information item to the user. The vertical axis represents a second signal representing a change in a second physiological parameter of the user between a time before and a time after presentation of an information item to the user. Although <FIG> illustrate only two signals to allow for illustration on a two-dimensional page, it should be understood that the principles may be extended to the use of more than two physiological parameters.

Illustrated on the graph of <FIG> are several hypothetical data points of the type that may be collected from one or more individuals, with each data point being associated with a different stimulus. The position of the data point on the graph represents the values of the first and second signals. The color of the data points represents whether or not the individual perceived the stimulus, which may be based on reporting by those individuals during a training or experimentation phase. Filled circles represent stimuli that were perceived by the individual, and white circles represent stimuli that were not perceived. Based on the data, a boundary <NUM>, which may be a line, curve, plane, or other surface as appropriate, may be fit to the data to separate regions representing perceived stimuli from regions representing un-perceived stimuli. As in the example of <FIG>, some data points may be outliers that do not strictly comply with the boundary.

Based on the boundary determined as described with respect to <FIG>, a system may operate to determine whether a user has perceived particular stimuli, including but not limited to XR notifications. In the example of <FIG>, various stimuli have been provided to a user at different times. For each of the stimuli, a first signal is obtained representing a change in the first physiological parameter of the user between a time before and a time after the respective stimulus, and a second signal is obtained representing a change in the second physiological parameter between a time before and a time after the stimulus. The resulting data points <NUM>, <NUM>, and <NUM> are shown on the graph of <FIG>. Based on the position of these data points relative to the boundary <NUM>, a classifier may make a determination that the stimulus corresponding to data point <NUM> was not perceived but that the stimuli corresponding to data points <NUM> and <NUM> were perceived.

In some embodiments, as illustrated with respect to <FIG>, the classifier may apply weights to one or more of the signals. The weight(s) may be based on a determined activity of the user. Known techniques may be used to determine the activity of a user. For example, a combination of heartrate monitor, GPS, and accelerometer may be used to determine whether a user is walking, running, or otherwise exercising. Such activities may cause a user to sweat to varying degrees; accordingly, some physiological parameters such as skin conductance may become less representative of the user's psychological response during exercise-related activities, and the signal(s) related to such parameters may be weighted accordingly. In the example shown in <FIG>, the system obtains information indicating that the user is engaged in an activity that increases the likelihood of corruption of the first signal. For example, based on the activity, the system may determine that a corruption coefficient of <NUM> should be applied to the first signal. As a result, the system may apply a weight of <NUM> (= <NUM> - <NUM>) to the first signal. In some embodiments, weights are determined directly without the use of a corruption coefficient. When the weight is applied, data points <NUM>, <NUM>, <NUM> are repositioned to points <NUM>, <NUM>, <NUM>. Using the weighted contribution of the first signal, the system may determine that the weighted signals represented by point <NUM> still correspond to a perceived notification, and that the weighted signals represented by point <NUM> still correspond to an un-perceived notification, but that the weighted signals represented by point <NUM> correspond to an un-perceived notification.

In some embodiments, when the contribution of one signal is reduced due to weighting, the contributions of other signals may be increased to normalize or otherwise balance the total contribution of signals. In some embodiments, the boundary used by the classifier may be shifted when weighted signals are used.

A method according to some embodiments comprises: obtaining a first measurement of a first physiological parameter of a user from a time before presentation of an information item to the user; obtaining a second measurement of the first physiological parameter of the user from a time after a beginning of the presentation of the information item to the user; and determining, based at least on a comparison between the first measurement and the second measurement, whether the user has perceived the information item.

Some embodiments further include determining a first corruption coefficient corresponding to the first physiological parameter, wherein the determination of whether the user has perceived the information item is based at least in part on the first corruption coefficient. The first corruption coefficient may be based at least in part on an amount by which a current activity of the user is expected to interfere with the first physiological parameter.

Some embodiments further include obtaining a third measurement of a second physiological parameter of the user from a time before presentation of the information item to the user; and obtaining a fourth measurement of the second physiological parameter of the user from a time after the beginning of the presentation of the information item to the user; wherein the determination of whether the user has perceived the information item is further based on a comparison between the third measurement and the fourth measurement.

Some embodiments further include determining a first corruption coefficient corresponding to the first physiological parameter and a second corruption coefficient corresponding to the second physiological parameter, wherein the determination of whether the user has perceived the information item is based at least in part on the first corruption coefficient and the second corruption coefficient.

In some embodiments, the first corruption coefficient is based at least in part on an amount by which a current activity of the user is expected to interfere with the first physiological parameter and the second corruption coefficient is based at least in part on an amount by which a current activity of the user is expected to interfere with the second physiological parameter.

In some embodiments, the information item is a notification. In some embodiments, the information item is presented in an extended reality (XR) experience.

In some embodiments, at least one of the first and the second physiological parameter is a parameter selected from the group consisting of: skin conductance, photoplethysmography (PPG), electrocardiography (ECG), electroencephalography (EEG), electromyography (EMG), electrooculography (EOG), skin temperature, heat flux, abdominal and/or thoracic respiration, pupillometry and gaze tracking.

Some embodiments further include, in response to a determination that the user has perceived the information item, dismissing the information item.

In some embodiments, the time after a beginning of the presentation of the information item to the user is a time during the presentation of the information item to the user.

In some embodiments, the information item is an advertisement or an emergency alert.

An apparatus according to some embodiments comprises a processor configured to perform any of the methods described herein.

Various hardware elements of one or more of the described embodiments are referred to as "modules" that carry out (i.e., perform, execute, and the like) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by those of skill in the relevant art for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and it is noted that those instructions could take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and/or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc..

Claim 1:
A mathod comprising:
obtaining (<NUM>) information indicating a current activity of a user;
obtaining (<NUM>) a first signal representing a change in a first physiological parameter of the user between a time before and a time after presentation of an information item to the user is initiated;
obtaining (<NUM>) a second signal representing a change in a second physiological parameter of the user between a time before and a time after presentation of the information item to the user is initiated;
determining (<NUM>, <NUM>), using at least the first and second signals, whether the user has perceived the information item,
characterised in that
a contribution of at least the first signal to the determination is weighted (<NUM>) by an amount based on a first corruption coefficient associated with the current activity, the first corruption coefficient being based (<NUM>) at least in part on an amount by which the current activity interferes with the first physiological parameter.