Patent Description:
Modem vehicles commonly include multiple devices that occupants of a vehicle can interact with to perform various functions. For example, and without limitation, a vehicle could include an interactive navigation system with which a vehicle occupant could obtain driving instructions. Such systems may be display based (center console display, display cluster, head-up display (HUD)), and have other components (haptics, remote haptics, shape-shifting controllers such as shape-shifting rotary knobs and shape-shifting steering wheels). The vehicle could also include an interactive stereo system with which a vehicle occupant could play music. In addition, occupants of vehicles may bring portable and wearable interactive devices into vehicles. For example, and without limitation, a vehicle occupant could bring a mobile phone, a tablet computer, a portable video game console, a smart phone, a head-mounted display device (for notifications and augmented reality content), smart glasses, and other devices with which the occupant could interact.

Each of these different devices may be configured to generate notifications that may require interaction from vehicle occupants or otherwise cause those occupants to divert attention. For example, and without limitation, the mobile phone mentioned above could ring, indicating that an incoming call should be answered. Alternatively, the interactive navigation system noted above could generate a visual and/or auditory notification indicating that a selection of a driving route is needed. As a general matter, any interactive device that is integrated with or brought into a vehicle may produce one or more types of notification at any given time. These notifications pose specific problems to the driver of the vehicle.

In particular, such notifications can be distracting to the driver of the vehicle and may cause the driver to divert attention away from driving. For example, and without limitation, if a mobile phone within the vehicle rings, the driver could be compelled to divert attention away from driving in order to answer the call. This problem is worsened when multiple devices generate notifications simultaneously. For example, and without limitation, if the driver's mobile phone rings at the same time that the navigation system requests that a route be selected, then the driver may become overwhelmed with input and then pay an insufficient amount of attention to driving. The above-described problem may also be worsened if one or more devices generate notifications during a period of time during which the driver needs to focus an elevated level of attention on the act of driving. For example, and without limitation, if the driver's tablet computer generates a notification while the driver is attempting to merge into traffic, the driver may not be able to coordinate the merge operation safely due to the distracting notification. Situations such as these are exceedingly problematic because distracted driving is recognized as one of the leading causes of vehicle accidents.

As the foregoing illustrates, more effective techniques for mitigating distractions associated with device notifications within a vehicle would be useful.

The invention sets forth a computer-implemented method for transmitting notifications to a driver of a vehicle, the method including determining a first mental load on the driver based on sensor data, determining a second mental load on the driver associated with a first notification, determining, based on the first mental load and the second mental load, that a mental load threshold will be exceeded, determining a priority level associated with the first notification, and transmitting, based on the priority level, either the first notification or a modified version of the first notification to one or more output devices to be output within the vehicle, thereby causing a third mental load on the driver that does not exceed a maximum load threshold.

One advantage of the approach described above is that the driver of the vehicle may maintain sufficient cognitive and mental resources to safely operate the vehicle. Accordingly, the techniques described herein represent a technological advancement over conventional systems that do not coordinate the delivery of notifications.

So that the manner in which the above recited features can be understood in detail, a more particular description of various embodiments of the invention briefly summarized above, may be had by reference to certain embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments and are therefore not to be considered limiting of scope, for the contemplated embodiments may admit to other equally effective embodiments.

In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments of the present invention.

As discussed above, devices within a vehicle can generate notifications at unpredictable times, thereby leading to situations in which the driver of the vehicle can become distracted. This situation is especially problematic when multiple notifications arrive simultaneously and/or when one or more notifications arrive during periods of time when the driver needs to focus an elevated level of attention on the act of driving. As a general matter, because in-vehicle devices generate notifications unpredictably and independently of context, those devices may distract a driver of a vehicle and/or the vehicle occupants.

To address these issues, embodiments of the invention include a notification control system (NCS) configured to coordinate the delivery of notifications issued from a set of devices to the driver of a vehicle. The NCS registers all interactive devices residing in a vehicle and causes those devices to conform to specific directives generated by the NCS. The NCS analyzes and predicts the cognitive and emotional load on the driver and also estimates the cognitive and emotional impact on the driver potentially caused by any incoming notifications. The NCS then coordinates the delivery of notifications to the driver in a manner that avoids overwhelming the cognitive and/or emotional capacity of the driver. The NCS may suppress notifications, delay the delivery of notifications, and/or transcode notifications into other media associated with different sensory modalities. The NCS may perform these operations in real-time based on the cognitive and emotional state of the driver and/or based on the current and predicted driving context.

One advantage of the approach described above is that the driver of the vehicle may be afforded sufficient cognitive and mental resources to safely operate the vehicle. Because the NCS coordinates the delivery of notifications based upon the state of the driver and also the driving context, the NCS avoids situations where device notifications cause the driver to unsafely divert attention away from driving. Accordingly, the techniques described herein represent a technological advancement over conventional systems that do not coordinate the delivery of notifications.

<FIG> illustrates a notification control system configured to implement one or more aspects of the various embodiments. As shown, a notification control system (NCS) <NUM> includes a computing device <NUM> and sensor arrays <NUM>. Computing device <NUM> is configured to couple with various interactive devices <NUM> that reside within a vehicle <NUM>. Vehicle <NUM> transports occupants, including driver <NUM> and passenger <NUM>. Vehicle <NUM> may include a car, truck, construction machine and/or heavy machinery, motorcycle, boat, submarine, jet ski, snowmobile, airplane, spaceship, exoskeleton, and a power loader, among others. Sensors <NUM> are configured to measure data that reflects a plurality of different physiological attributes of driver <NUM>.

Computing device <NUM> implements a discovery process to establish communications with interactive devices <NUM> and assume control over when and how those devices output notifications to driver <NUM>. Computing device <NUM> also processes data gathered via sensors <NUM> to determine the cognitive and/or emotional load on driver <NUM> at any given time. Computing device <NUM> then analyzes notifications to be scheduled for delivery and estimates the cognitive and/or emotional impact of those notifications on driver <NUM>. Finally, computing device <NUM> coordinates the delivery of notifications to driver <NUM> to avoid causing driver <NUM> excessive mental load, as described in greater detail below in conjunction with <FIG>.

<FIG> illustrate various graphs depicting how device notifications may be throttled in response to mental load, according to various embodiments. As shown in each of these Figures, a graph <NUM> includes load axis <NUM> and impact axis <NUM>. Load axis <NUM> corresponds to the mental load of driver <NUM> at any given time. Impact axis <NUM> corresponds to the predicted additional cognitive/emotional load driver may potentially experience in response to a given notification. As a general matter, in the context of this invention, the term "impact" refers to additional mental load caused by sensory input driver <NUM> receives. Plot <NUM> indicates the maximum allowed impact of a given notification output to driver <NUM> as a function of the mental load on driver <NUM>.

As shown in <FIG>, graph <NUM>(A) includes plot <NUM>(A) plotted against load axis <NUM> and impact axis <NUM>. According to plot <NUM>(A), when mental load is low, as in region <NUM>, higher impact notifications can be output to driver <NUM>. Conversely, when mental load is high, as in region <NUM>, only lower impact notifications can be output to driver <NUM>. Region <NUM> may correspond to driving contexts that do not require elevated attention from driver <NUM>. For example, and without limitation, region <NUM> may correspond to driving along a straight country road with little traffic. Region <NUM>, on the other hand, may correspond to driving contexts that do require elevated attention from driver <NUM>. For example, and without limitation, region <NUM> may correspond to driving in heavy traffic during rush hour after a long day at the office. According to graph <NUM>(A), notifications having higher impact levels can be output to driver <NUM> until the mental load on driver <NUM> reaches threshold <NUM>, after which only lower impact notifications can be output. <FIG> illustrate other ways notifications can be throttled.

As shown in <FIG>, graph <NUM>(B) includes plot <NUM>(B) plotted against load axis <NUM> and impact axis <NUM>. According to plot <NUM>(B), the maximum impact level of notifications that can be output to driver <NUM> gradually decreases as the mental load on driver <NUM> increases. With this approach, the maximum impact level of any notification to be output to driver <NUM> is approximately and/or indirectly proportional to the current mental load of driver <NUM>.

As shown in <FIG>, graph <NUM>(C) includes plot <NUM>(C) plotted against load axis <NUM> and impact axis <NUM>. Plot <NUM>(C) includes regions <NUM>, <NUM>, and <NUM> corresponding to low, medium, and high mental load. Plot <NUM>(C) may also include other regions as well. When the mental load on driver <NUM> falls within any given region, only notifications having less than or equal to the corresponding maximum impact level can be output to driver <NUM>.

Referring generally to <FIG>, NCS <NUM> may buffer notifications with predicted impact exceeding the current maximum impact for later delivery. Additionally or alternatively, NCS <NUM> may transcode those notifications into a different delivery mechanism that may be less distracting to driver <NUM>. These particular techniques are described in greater detail below in conjunction with <FIG>.

With the above approach, NCS <NUM> coordinates notifications output by interactive devices <NUM> in a manner that avoids consuming excessive attention from driver <NUM>. Thus, NCS <NUM> may avoid situations where driver <NUM> is distracted from driving by incoming device notifications. Various hardware and software configured to implement the above techniques are described in greater detail below in conjunction with <FIG>.

<FIG> is a more detailed illustration of the computing device of <FIG>, according to various embodiments. As shown, computing device <NUM> includes processor <NUM>, input/output (I/O) devices <NUM>, and memory <NUM>, and is coupled to sensor arrays <NUM> and interactive devices <NUM>.

Processor <NUM> includes any technically feasible set of hardware units configured to process data and execute software applications. For example, and without limitation, processor <NUM> could include one or more of a central processing unit (CPU), a graphics processing unit (GPU), and an application specific integrated circuit (ASICs). I/O devices <NUM> include any technically feasible set of devices configured to perform input and/or output operations, including, for example, and without limitation, a display device, a keyboard, a mouse, a touchscreen, a loudspeaker, haptic actuators, mid-air haptic transducers, and shape-shifting interfaces, among others. Memory <NUM> includes any technically feasible set of storage media configured to store data and software applications, such as a hard disk, a random-access memory (RAM) module, and a read-only memory (ROM), for example. Memory <NUM> includes a throttling application <NUM> and a database <NUM>. Throttling application <NUM> includes program code that, when executed by processor <NUM>, analyzes data captured by sensor arrays <NUM> to determine the mental load on driver <NUM> at any given time.

Sensor arrays <NUM> may include pupil sensors, heart rate sensors, breathing rate sensors, galvanic skin response sensors, optical devices configured for facial recognition, neural activity sensors, audio sensors, muscle contraction sensors, eye gaze detectors, voice analyzers, gesture sensors, radar sensors, thermal sensors, and any other technically feasible type of sensor configured to measure any attribute of a human being. Sensor arrays <NUM> may also include gesture detectors, touch detectors, kinematic and/or dynamic sensors for measuring the dynamics of vehicle <NUM>, state sensors for determining the mechanical state of vehicle <NUM>, and environmental sensors for detecting the ambient conditions inside and outside of vehicle <NUM>. Sensor arrays <NUM> may further include devices configured to gather information from third party services that may provide weather, traffic, and road condition data, among other types of data. Sensor arrays <NUM> may also include outward-facing sensors configured to collect information associated with the environment surrounding vehicle <NUM>, including geolocation sensors, range sensors, daylight sensors, and so forth.

Based on data captured via sensor arrays <NUM>, throttling application <NUM> determines the current cognitive and/or emotional load on driver <NUM>. Throttling application <NUM> may also predict the cognitive/emotional load on driver <NUM> at any future time based on contextual data, such as driving conditions, weather conditions, road conditions, among other examples and without limitation. Throttling application <NUM> also discovers and registers interactive devices <NUM> and assumes control over notifications output by those devices. Throttling application <NUM> may implement a cognitive and/or emotional model of driver <NUM> in order to predict the cognitive and/or emotional impact of a given notification on driver <NUM>. Throttling application <NUM> then coordinates the delivery of notifications to driver <NUM> to prevent the cognitive and/or emotional load on driver <NUM> from exceeding a maximum level. Various modules within throttling application <NUM> responsible for performing these operations are described in greater detail below in conjunction with <FIG>.

<FIG> is a more detailed illustration of the throttling application of <FIG>, according to various embodiments. As shown, throttling application <NUM> includes load analysis and prediction engine <NUM>, impact modeling and estimation engine <NUM>, prioritization engine <NUM>, and scheduling engine <NUM>.

Load analysis and prediction engine <NUM> includes cognitive load module <NUM> and emotional load module <NUM>. Cognitive load module <NUM> analyzes data captured from sensor arrays <NUM> to determine the cognitive load on driver <NUM>. Cognitive load module <NUM> may include a machine learning (ML) model of driver <NUM> that is updated based on the cognitive responses of driver <NUM> to various types of events, including, for example and without limitation, notifications generated by interactive devices <NUM> as well as various driving-related events. The ML model may initially include a baseline model that is generated based on analyzing the responses of a plurality of other drivers of other vehicles. That baseline model can be fine-tuned over time to better approximate the cognitive load of driver <NUM> based on sensor data.

Emotional load module <NUM> analyzes data captured from sensor arrays <NUM> to determine the emotional load on driver <NUM>. Similar to cognitive load module <NUM>, emotional load module <NUM> may include an ML model of driver <NUM> that is updated based on the emotional responses of driver <NUM> to various types of events, including, for example and without limitation, notifications and other events. This ML model may initially include a baseline model that is generated based on analyzing the emotional responses of other drivers. The baseline model can be fine-tuned over time to better approximate the emotional load of driver <NUM> given various sensor readings. Load analysis and prediction engine <NUM> synthesizes the determined cognitive load and emotional load of driver <NUM> to generate a combined load metric, referred to herein as "mental load" or simply "load. " Load analysis and extension <NUM> outputs this load metric to scheduling engine <NUM>.

In parallel with the operation of load analysis and prediction engine <NUM>, impact modeling and estimation engine <NUM> analyzes notifications associated with interactive devices <NUM> in order to estimate the cognitive and/or emotional impact of those notifications on driver <NUM>. Impact modeling and estimation engine <NUM> includes a cognitive impact module <NUM> and an emotional impact module <NUM>. Cognitive impact module <NUM> is configured to predict the cognitive impact on driver <NUM> of any given notification. Cognitive impact module <NUM> may include an ML model of driver <NUM> that, in some embodiments, is derived from the ML model included in cognitive load module <NUM> and trained in like fashion. Similarly, emotional impact module <NUM> is configured to predict the emotional impact on driver <NUM> of any given notification. Emotional impact module <NUM> may include an ML model of driver <NUM> that, in some embodiments, is derived from the ML model included in emotional load module <NUM> and trained in like fashion. Impact modeling and estimation engine <NUM> synthesizes the predicted cognitive and emotional impact of any given notification on driver <NUM> to generate an impact metric. Impact modeling and estimation engine <NUM> outputs this impact metric to scheduling engine <NUM>.

In parallel with the above-described engines, prioritization engine <NUM> analyzes received notifications using a set of source weightings <NUM>. Each source weighting <NUM> indicates a priority associated with a particular source from which a notification may be received. For example, and without limitation, a given source weighting <NUM> could indicate that email messages received from the spouse of driver <NUM> should take priority over email messages received from the boss of driver <NUM>. In another example, and without limitation, a given source weighting <NUM> could indicate that vehicle notifications associated with driving should take priority over all other notifications. Prioritization engine <NUM> assigns priorities to different notifications and outputs these priorities to scheduling engine <NUM>.

Scheduling engine <NUM> analyzes the current load on driver <NUM>, the predicted impact of received notifications on driver <NUM>, and the priorities assigned to those notifications, and then coordinates the delivery of notifications to driver <NUM> in a manner that avoids causing the cognitive and/or emotional load of driver <NUM> to exceed a maximum level. For example, and without limitation, scheduling engine <NUM> could identify that the load on driver <NUM> is elevated, and then determine that a given notification should be delayed because that notification could potentially cause the load on driver to exceed a maximum level. In addition, scheduling engine <NUM> may implement transcoding module <NUM> to transcode a high priority notification into a different medium that allows more immediate delivery to driver <NUM>. For example, and without limitation, scheduling engine <NUM> could identify that a text message was received from the spouse of driver. Then, scheduling engine <NUM> could determine that although driver <NUM> is visually engaged with driving, the text message can be transcoded into speech and delivered to driver <NUM> audibly without causing driver <NUM> any visual distractions. <FIG> set forth more detailed examples of how scheduling engine <NUM> schedules notifications for delivery.

<FIG> illustrate examples of how the notification control system of <FIG> coordinates delivery of multiple notifications to the driver of a vehicle, according to various embodiments.

As shown in <FIG>, a graph <NUM>(A) includes time axis <NUM> and load axis <NUM>. A function, L(t) <NUM>, is plotted against time axis <NUM> and load axis <NUM>. L(t) <NUM> represents the load of driver <NUM> over time. A portion of L(t) <NUM> is determined based on the real-time, current load computed based on sensor readings. Another portion of L(t) <NUM> is estimated based on contextual information, including traffic data, driving data, weather information, road conditions, and so forth. For example, and without limitation, peak <NUM> of L(t) <NUM> corresponds to a predicted lane change. Generally, L(t) <NUM> represents the load of driver <NUM> within a time window starting with the current time. In addition, maximum load <NUM> represents the maximum allowable load that driver <NUM> should experience at any time. As described below, NCS <NUM> coordinates the delivery of notifications to driver <NUM> to keep driver load beneath maximum load <NUM>.

As shown in <FIG>, graph <NUM>(B) includes input axis <NUM> and impact axis <NUM>. Notifications <NUM> and <NUM> are displayed along input axis <NUM>. These notifications may be received via interactive devices <NUM> and then transmitted to NCS <NUM> for delivery to driver <NUM>. Notification <NUM> corresponds to a viral media clip, while notification <NUM> corresponds to a text message from the spouse of driver <NUM>. NCS <NUM> analyzes these notifications and determines that notification <NUM> will likely have a high impact on the load of driver <NUM> and therefore may be distracting. NCS <NUM> also determines that notification <NUM> may have a lower impact on driver <NUM> and therefore be less distracting. NCS <NUM> generally makes these predictions using the various models discussed above in conjunction with <FIG>. In addition, NCS <NUM> also determines that notification <NUM> has a higher priority than notification <NUM> based on the source of those notifications using the techniques described above in conjunction with <FIG>. Based on the priority and estimated impact associated with the different notifications, NCS <NUM> schedules those notifications for delivery to driver <NUM> in a manner that does not cause the load of driver <NUM> to exceed maximum load <NUM>.

As shown in <FIG>, NCS <NUM> schedules notification <NUM> to be delivered to driver <NUM> at time t<NUM> and schedules notification <NUM> to be delivered to driver <NUM> at time t<NUM>. Both of these notifications increase the load on driver <NUM>. However, notification <NUM> only increases the load by a small amount, notification <NUM> increases the load by larger amount, and over time neither notification causes the load on driver <NUM> to exceed the maximum load <NUM>. In addition, because NCS <NUM> distributes the delivery of notifications across time, driver <NUM> can process those notifications separately and therefore may not become overloaded. NCS <NUM> also transcodes notifications into different media to expedite the delivery of certain notifications while avoiding exceeding the maximum load of driver <NUM>, as described in greater detail below in conjunction with <FIG>.

<FIG> illustrate examples of how the notification control system of <FIG> transcodes notifications between sensory modalities, according to various embodiments. As shown in <FIG>, a graph <NUM>(A) includes time axis <NUM> and load axis <NUM>. Two functions <NUM> are plotted against time axis <NUM> and load axis <NUM>. Lopt(t) <NUM>(<NUM>) represents the load of driver <NUM> over time caused by optical inputs, which may include visual inputs falling within the visual range of humans as well as other optical transmissions. Laud(t) <NUM>(<NUM>) represents the load of driver <NUM> over time caused by auditory inputs. NCS <NUM> may determine Lopt(t) <NUM>(<NUM>) and Laud(t) <NUM>(<NUM>) by analyzing sensor data associated with driver <NUM>, analyzing information associated with vehicle <NUM>, and analyzing contextual information related to driving, among others. For example, and without limitation, NCS <NUM> could analyze real-time driving information and determine that driver <NUM> is currently engaged with navigating heavy traffic. As is shown, heavy traffic begins at peak <NUM>. NCS <NUM> could then compute Lopt(t) <NUM>(<NUM>) to indicate that the optical load on driver <NUM> is elevated. In another example, and without limitation, NCS <NUM> could analyze sensor data indicating that the interior cabin of vehicle <NUM> is relatively quiet. NCS <NUM> could then compute Laud(t) <NUM>(<NUM>) to indicate that the auditory load on driver <NUM> is minimal, as is shown. The overall load on driver <NUM> at any given time may be derived from the individual loadings associated with each different sensory modality. Based on real-time estimates and predictions of Lopt(t) <NUM>(<NUM>) and Laud(t) <NUM>(<NUM>) computed in the manner described above, NCS <NUM> transcodes notifications between sensory modalities to prevent the overall load on driver <NUM> from exceeding maximum load <NUM>.

As shown in <FIG>, graph <NUM>(B) includes input axis <NUM> and impact axis <NUM>. Notifications <NUM> and <NUM> are displayed along input axis <NUM>. These notifications may be received via interactive devices <NUM> and then transmitted to NCS <NUM> for delivery to driver <NUM>. Notification <NUM> corresponds to an email sent by the boss of driver <NUM>, while notification <NUM> corresponds to an indication of the final score of a sporting event (potentially received via a sports app). NCS <NUM> analyzes the content of these notifications and then estimates the predicted impact on driver <NUM>. In doing so, NCS <NUM> determines that each of these notifications includes optical content, and therefore may demand an elevated level of optical processing to be performed by driver <NUM>. NCS <NUM> also prioritizes notifications <NUM> and <NUM> and determines that notification <NUM> is low priority and can be delayed, while notification <NUM> is higher priority and should be delivered expeditiously. However, because the optical load on driver <NUM> is already elevated, NCS <NUM> determines that notification <NUM> cannot be delivered to driver <NUM> optically and should instead be transcoded into an auditory format.

As shown in <FIG>, NCS <NUM> transcodes notification <NUM> into an auditory medium, such as spoken and/or dictated text, for example and without limitation, and then schedules notification <NUM> to be delivered to driver <NUM> at time t<NUM>. Because notification <NUM> does not demand driver <NUM> to perform optical processing, Lopt(t) <NUM>(<NUM>) is not significantly increased and therefore the load on driver <NUM> does not exceed maximum load <NUM>. Laud(t) <NUM>(<NUM>) is elevated, but similarly does not cause the load of driver <NUM> to surpass maximum load <NUM>. NCS <NUM> also schedules notification <NUM> to be delivered to driver <NUM> at time t<NUM>. NCS <NUM> need not transcode notification <NUM> into an auditory format because at time t<NUM> Lopt(t) <NUM>(<NUM>) decreases sufficiently that driver <NUM> can spare optical processing resources. However, NCS <NUM> may optionally transcode notification <NUM> anyway in order to allow Lopt(t) <NUM>(<NUM>) to decrease to a lower level. Although the example described in conjunction with these figures pertains to just two sensory modalities, NCS <NUM> may select between any number of sensory modalities when transcoding notifications.

In one embodiment, NCS <NUM> may also transcode notifications during delivery if the measured mental load on driver <NUM> begins increasing beyond expected or acceptable levels. In addition, NCS <NUM> may transcode and summarize notifications for quick delivery, and then deliver the original notification at a later time in the original sensory modality.

Referring generally to <FIG>, NCS <NUM> coordinates the delivery of notifications, potentially transcoding those notifications between sensory modalities, in order to control the cognitive and/or emotional load placed on driver <NUM> and keep that load below a maximum level. Importantly, this approach reduces potential distractions while still allowing driver <NUM> to receive important information delivered via notifications. Although the techniques described above are discussed relative to driving, persons skilled in the art will understand that these techniques can be applied to control any source of distraction that can interfere with any human activity.

In one embodiment, NCS <NUM> analyzes the behavior of driver <NUM> and then determines a degree to which driver <NUM> is focused on driving. If the degree of focus indicates that driver <NUM> is not sufficiently focused on driving, then NCS <NUM> may suppress all notifications and instruct driver <NUM> to pay more attention to driving. If the degree of focus indicates that driver <NUM> is focused on driving but can spare cognitive and/or emotional resources, then NCS <NUM> may schedule notifications in the manner described above. If the degree of focus indicates that driver <NUM> is focused on driving and cannot spare cognitive and/or emotional resources, then NCS <NUM> may delay or transcode notifications as also described above. In this manner, NCS <NUM> helps keep the attention level of driver <NUM> within a "sweet spot" that may facilitate safe driving.

In another embodiment, NCS <NUM> relies on cloud-based data to update and/or generate machine learning models used to predict the cognitive and/or emotional load on driver <NUM> in response to various events. The cloud-based data may include indications of specific events that cause predictable increases in mental loading across a range of other drivers. For example, and without limitation, NCS <NUM> could acquire cloud-based data indicating that a particular stretch of highway includes a sequence of distracting billboards, and then anticipate that the mental load on driver <NUM> when driving along that stretch of highway will also increase.

In yet another embodiment, NCS <NUM> coordinates the delivery of notifications to passengers within vehicle <NUM> using a different approach than that used to deliver notifications to driver <NUM>. For example, and without limitation, NCS <NUM> could determine that passenger <NUM> is sufficiently far away from driver <NUM> that a mobile device in the possession of passenger <NUM> should be allowed to deliver notifications to passenger <NUM> without restrictions. In another example, NCS <NUM> could determine that a backseat infotainment system cannot be seen by driver <NUM>, and therefore should not be restricted from issuing optical notifications. NCS <NUM> may also deliver notifications according to specific sound zones within vehicle <NUM>. For example, and without limitation, NCS <NUM> could cause notifications to be output within sound zones that are inaudible to driver <NUM>, thereby allowing notifications to be delivered to vehicle passengers without distracting driver <NUM>.

<FIG> set forth a flow diagram of method steps for coordinating delivery of multiple notifications to the driver of a vehicle, according to various embodiments. Although the method steps are described in conjunction with the systems of <FIG>, persons skilled in the art will understand that the method steps can be performed in any order by any system.

As shown, a method <NUM> begins at step <NUM>, where NCS <NUM> registers a set of devices for centralized management. The set of devices generally includes interactive devices that can generate notifications of various types. At step <NUM>, NCS <NUM> captures physiological data associated with driver <NUM>. At step <NUM>, NCS <NUM> determines the mental load on driver over a timespan based on the physiological data and current driving context. NCS <NUM> may implement a cognitive model and/or an emotional model in order to estimate the cognitive and/or emotional load on driver <NUM> at any given time. NCS <NUM> may also model the predicted cognitive and/or emotional load on driver <NUM> in response to future events, such as anticipated traffic, predicted weather, and so forth.

At step <NUM>, driver <NUM> receives one or more notifications generated by one or more registered devices. At step <NUM>, NCS <NUM> estimates the mental impact on driver for each received notification. In doing so, NCS <NUM> may test each notification against a cognitive and/or emotional model of driver <NUM>. At step <NUM>, NCS <NUM> prioritizes notifications based on a priority configuration. The priority configuration may indicate weights associated with different sources of notifications, or priorities associated with specific types of content, among other things. At step <NUM>, NCS <NUM> identifies at least one notification that should be transcoded to a different sensory modality. The identified notification could be, for example and without limitation, a high priority notification that cannot be delivered to driver <NUM> in an original form because driver <NUM> is already engaged with a driving task. At step <NUM>, NCS <NUM> transcodes the identified notification into another sensory modality where driver <NUM> can spare cognitive and/or emotional resources. At step <NUM>, NCS <NUM> coordinates the delivery of notifications to driver <NUM> across multiple sensory modalities without causing mental load on driver to exceed a maximum load.

At step <NUM>, NCS <NUM> captures physiological data associated with driver <NUM> in response to the delivered notifications. At step <NUM>, NCS <NUM> determines the mental load on driver in response to each notification. Finally, at step <NUM>, NCS <NUM> updates the cognitive and/or emotional model of driver <NUM> based on the estimated mental impact computed at step <NUM>. In this manner, NCS <NUM> continuously improves the accuracy with which the cognitive and/or emotional impact of driver <NUM> can be predicted.

In sum, a notification control system (NCS) is configured to coordinate the delivery of notifications issued from a set of devices to the driver of a vehicle. The NCS registers all interactive devices residing in a vehicle and causes those devices to conform to specific directives generated by the NCS. The NCS analyzes and predicts the cognitive and/or emotional load on the driver and also estimates the cognitive and/or emotional impact on the driver that may be caused by any incoming notifications. The NCS then coordinates the delivery of notifications to the driver in a manner that avoids overwhelming the cognitive and/or emotional capacity of the driver. The NCS may suppress notifications, delay the delivery of notifications, and/or transcode notifications into other media associated with different sensory modalities. The NCS performs these operations in real-time based on the cognitive and/or emotional state of the driver and/or based on the current and predicted driving context.

One advantage of the approach described above is that the driver of the vehicle may maintain sufficient cognitive and mental resources to safely operate the vehicle. Because the NCS coordinates the delivery of notifications based upon the state of the driver and also the driving context, the NCS avoids situations where device notifications cause the driver to unsafely divert attention away from driving. Accordingly, the techniques described herein represent a technological advancement over conventional systems that do not coordinate the delivery of notifications.

Aspects of the present embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "module" or "system. " In addition, any hardware and/or software technique, process, function, component, engine, module, or system described in the present invention may be implemented as a circuit or set of circuits. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

Aspects of the present invention are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks.

Claim 1:
A computer-implemented method (<NUM>) for transmitting notifications to a driver (<NUM>) of a vehicle (<NUM>), the method comprising:
determining a first mental load (<NUM>, <NUM>) on the driver (<NUM>) based on sensor data;
determining a second mental load on the driver (<NUM>) associated with a first notification (<NUM>, <NUM>, <NUM>, <NUM>);
determining, based on the first mental load (<NUM>, <NUM>) and the second mental load, that a mental load threshold (<NUM>, <NUM>, <NUM>) will be exceeded;
determining a priority level associated with the first notification (<NUM>, <NUM>, <NUM>, <NUM>); and
transmitting, based on the priority level, either the first notification (<NUM>, <NUM>, <NUM>, <NUM>) or a modified version of the first notification to one or more output devices (<NUM>) to be output within the vehicle (<NUM>), thereby causing a third mental load on the driver (<NUM>) that does not exceed a maximum load threshold (<NUM>).