Adaptive Heads-Up Display Using a User Monitoring System and Remote Systems

Methods and systems for a vehicle for adaptively controlling messages displayed on a heads-up display. A plurality of requests is received by a first controller to display a message associated with each of the plurality of requests on a heads-up display from one or more operational applications. The plurality of requests are filtered by an arbitration gateway based on user attention data gathered from a user monitoring system and a user behavior history. A priority is assigned to each of the plurality of filtered requests based on a user interest prediction model, at least a portion of the plurality of the filtered requests are selected based on the assigned priority, and the messages associated with the selected portion of the plurality of filtered requests are displayed on the heads-up display.

BACKGROUND

A heads-up display is a transparent display that provides information to a driver while allowing the driver to maintain their gaze toward the road, minimizing driver distraction. Generally, heads-up displays utilize a projector to project images onto the windshield or other transparent screen within the driver's field of vision while gazing at the roadway. In aspects, the windshield, or screen, includes a phosphor material that is activated by a beam of light or a reflective mirror to display projected images. A number of heads-up displays include augmented reality, where information is projected onto the display and overlays, or is combined with, features in the environment visible through the windshield or transparent screen. Heads-up display systems project data gathered from, body electronics such as the speedometer or tachometer, navigation systems, infotainment systems and other on-board or remote systems. For example, a heads-display may provide a driver with information regarding the status of the vehicle such as the vehicle speed, engine rotations per minute, battery charging, or applications running on infotainment systems such as maps or playing media.

Thus, while heads-up displays achieve their intended purpose, room remains for improvement in displaying information on heads-up displays.

SUMMARY

According to several aspects, the present disclosure relates to a method of adaptively controlling messages displayed on a heads-up display. The method includes receiving a plurality of requests to display a message associated with each of the plurality of requests on a heads-up display from one or more operational applications. The method further includes filtering the plurality of requests by an arbitration gateway based on user attention data gathered from a user monitoring system and a user behavior history. In addition, the method includes assigning a priority to each of the plurality of filtered requests based on a user interest prediction model, selecting at least a portion of the plurality of the filtered requests based on the assigned priority, and displaying the messages associated with the selected portion of the plurality of filtered requests on the heads-up display.

In embodiments, the method includes dropping the message associated with at least one of the plurality of filtered requests that was not selected based on the assigned priority. In further embodiments, the method includes displaying the dropped message associated with at least one of the plurality of filtered requests on the heads-up display that was not selected based on the assigned priority. In additional further embodiments, the method includes providing at least one of a sound and a haptic response when the dropped message is related to safety.

In additional embodiments, the method includes gathering user attention data using an attention sensor. The attention sensor may include at least one of a time-of-flight sensor and a camera.

In any of the above embodiments, the method further includes calculating the user interest prediction model based upon a surrounding environmental event and the user behavior history.

In any of the above embodiments, the method further includes filtering the plurality of requests by calculating a filtering tuple for each of the plurality of requests. In further embodiments, the method includes calculating the filtering tuple by characterizing each of the plurality of requests according to the following elements: an ID, a priority, an emergency level, a user attention status, a user behavior history, a heads-up display attribute, and raw data. In yet further embodiments, the method includes assigning the priority of each of the plurality of filtered requests by weighting the filtering tuple for each of the plurality of filtered requests, weighting a prioritizing tuple for each of the plurality of filtered requests, and adding the weighted filtering tuple and the weighted prioritizing tuple. In additional further embodiments, the method further includes calculating the prioritizing tuple by characterizing each of the plurality of requests according to the following elements: an event_ID, a priority, an emergency level, a predictive user interest, a heads-up display attribute, a lead time, and raw event data.

According to several additional aspects, the present disclosure also relates to an on-board adaptive heads-up display system for a vehicle. The on-board adaptive heads-up display system includes a first controller, a heads-up display connected the first controller, and a user monitoring system connected to the first controller. The first controller is configured to execute instructions to perform the method according to any of the above-described embodiments of adaptively controlling messages displayed on a heads-up display. The first controller is configured to execute instructions to receive a plurality of requests to display a message associated with each of the plurality of requests on a heads-up display from one or more operational applications and filter the plurality of requests by an arbitration gateway based on user attention data gathered from the user monitoring system and a user behavior history. The first controller is further configured to execute instructions to assign a priority to each of the plurality of filtered requests based on a user interest prediction model, select at least a portion of the plurality of the filtered requests based on the assigned priority, and display the messages associated with the selected portion of the plurality of the filtered requests on the heads-up display. In further embodiments, the controller is also configured to execute instructions to drop the message associated with the at least one of the plurality of filtered requests that was not selected based on the assigned priority. In yet further embodiments, the controller is also configured to execute instructions to provide at least one of a sound and a haptic response when the dropped message is related to safety.

In any of the above embodiments, the controller is further configured to gather user attention data using an attention sensor, wherein the attention sensor includes at least one of a time-of-flight sensor and a camera. In further embodiments, the controller is configured to execute instructions to configure a heads-up display attribute of the messages using user inputs received from a user input device.

In any of the above embodiments, the controller is further configured to execute instructions to filter the plurality of requests using a filtering tuple to characterize each of the plurality of requests by the arbitration gateway including the following elements: an ID, a priority, an emergency level, a user attention status, a user behavior history, a heads-up display attributes, and raw data. In further embodiments, the controller is configured to execute instructions to assign a priority to the plurality of filtered requests by weighting the filtering tuple for each of the plurality of filtered requests; weighting a prioritizing tuple based on the user interest prediction model; and adding the weighted filtering tuple and the weighted prioritizing tuple to calculate a prioritizing score. In yet further embodiments, controller is configured to execute instructions to calculate the prioritizing tuple by characterizing each of the plurality of filtered requests according to the following elements: an event_ID, a priority, an emergency level, a predictive user interest, a heads-up display attribute, a lead time, and raw event data.

According to several additional aspects, the present disclosure also relates to an adaptive heads-up display system for a vehicle. The system includes an on-board adaptive heads-up display system including a first controller, a heads-up display connected the first controller, a user monitoring system connected to the first controller, and a first communication system connected to the first controller. The system further includes a remote system. The remote system includes a remote controller and a remote communication system connected to the remote controller. The remote controller is configured to wirelessly communicate information to the first communication system. The first controller is configured to execute instructions to execute the method of any of the above-described embodiments of adaptively controlling messages displayed on a heads-up display. The first controller is configured to receive a plurality of requests to display a message associated with each of the plurality of requests on a heads-up display from one or more operational applications and filter the plurality of requests by an arbitration gateway based on user attention data gathered from the user monitoring system and a user behavior history. The first controller is further configured to assign a priority to each of the plurality of filtered requests based on a user interest prediction model calculated by the remote system, select at least a portion of the plurality of the filtered requests based on the assigned priority, and display the messages associated with the selected portion of the plurality of the filtered requests on the heads-up display.

In embodiments of the above, the remote controller is further configured to execute instructions to calculate a user prediction model based upon prior behavior gathered from the user monitoring system and data regarding a surrounding environment.

DETAILED DESCRIPTION

The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding introduction, summary, or the following detailed description. It should be understood that throughout the drawings, corresponding reference numerals indicate like or corresponding parts and features.

As used herein, the term “vehicle” is not limited to automobiles. While the present technology is described primarily herein in connection with automobiles, the technology is not limited to automobiles. The concepts can be used in a wide variety of systems, such as in connection with motorcycles, mopeds, locomotives, aircraft, marine craft, and other vehicles, or other automated systems including navigation and mapping functionalities. Systems include, for example, driving navigation, air navigation, marine navigation, robotic navigation, and navigation in enclosed spaces or enclosed geological features, to name a few.

Reference will now be made in detail to several examples of the disclosure that are illustrated in accompanying drawings. Whenever possible, the same or similar reference numerals are used in the drawings and the description to refer to the same or like parts or steps. The drawings are in simplified form and are not to precise scale.

The present disclosure is directed to an adaptive heads-up display for messages using a user monitoring system and a remote system, such as a cloud service. In aspects, a method and systems are proposed to show messages on a heads-up display adaptively configured in real time according to user is looking determined by the direction of user's eyes or head, leveraging a user monitoring system. Simultaneously, the remote system predictively prioritizes messages using current environmental conditions and accumulated data user behavior exhibited during previous driving events. The combination of real-time and predictive capability prioritizes the messages108and eliminates low priority messages108for users in an orderly manner, optimizing system resources, reducing distraction, and enhancing outcomes. In aspects, the user is a driver, or other vehicle occupants or remote operator who has authorization to operate the vehicle100.

FIG.1illustrates a vehicle100according to aspects of the present disclosure including an adaptive heads-up display system101. The adaptive heads-up display system101includes both an on-board adaptive heads-up display system102and a remote system104, such as a cloud service. On-board the vehicle100, the on-board adaptive heads-up display system102projects messages108(seeFIGS.2A and2B), such as various images or characters in the form of text or numbers, onto a heads-up display106. In the illustrated aspect, the heads-up display106is visible on the interior of a windshield. In addition, or alternatively, other transparent displays in the user's field of vision may be used. Further, the heads-up display106may be an augmented reality heads-up display, where messages108are displayed relative to the environment visible through the heads-up display.

The on-board adaptive heads-up display system102also includes a projector110, a user monitoring system112, user input devices114, and one or more trip monitoring sensors116. The various devices, i.e., projector110, user monitoring system112, user input devices114, and one or more trip monitoring sensors116, are coupled to a controller118through one or more wired or wireless connections. The connections provide communication of information in the form of signals, or changes in electrical voltage or current, to and from the various devices in the on-board adaptive heads-up display system102and controller118. The controller118includes one or more processors120, computer readable media122, a communication system124and a buffer126. As noted above, the adaptive heads-up display system101also includes a remote system104, i.e., cloud services, that delivers predictive information on demand to the vehicle100. The remote system104performs various functions in adapting the messages108displayed on the heads-up display106using user behavior history and surrounding environmental events.

With reference toFIG.2AandFIG.2B, the projector110is connected to the center top of the dashboard130. However, it should be appreciated that the projector110may alternatively be connected to or integrated into the rearview mirror132, the roof of the vehicle, or the A-pillars. Further, more than one projector may be provided. The projector110may display images across the entire heads-up display106or in limited locations on the heads-up display106.FIG.2Aillustrates a heads-up display106including messages108represented by various images. The images are not prioritized and represent information having varying degrees of importance to the user. The messages108clutter the heads-up display106, distracting from messages108that may be relatively more valuable to the user.FIG.2Billustrates a heads-up display106including a limited number of messages108. The messages108displayed are prioritized, wherein messages108that are deemed less important is minimized or omitted altogether, highlighting the importance of the messages108that are displayed. The display of particular messages108being determined by a combination of user preference, user behavior history, and other characteristics of the information presented in the messages108.

In addition, a user monitoring system112is illustrated inFIG.1,FIG.2AandFIG.2B. The user monitoring system112is used to determine where a user, such as the operator or driver, is directing their attention. In embodiments, the user monitoring system112includes at least one attention sensor136, such as a time-of-flight (ToF) sensor and one or more cameras. In further embodiments, the cameras capture visible light or infrared irradiation. While the user monitoring system112is illustrated as being positioned near the top of the heads-up display106, the attention sensor136of the user monitoring system112may be located in any position that allows measurement of at least one of the user's head position, the user's eye direction, or the user's body. For example, a user monitoring system112may be connected to or integrated into the dashboard130, the rearview mirror132, an infotainment system138, or the roof interior. The attention sensor136provides user attention data, in the form of a signal, i.e., a change in voltage or current, to the controller118.

With further reference toFIGS.2A and2B, user input devices114are provided to allow the user to utilize, manipulate, and control the heads-up display system101through user inputs. User input devices114include, for example, one or more buttons or sensors in the center console, on the infotainment system138, and on steering wheel134. User input devices114may also include a microphone, which may be integrated into the steering wheel134or the infotainment system138. In embodiments, the user input devices114may be push buttons, knobs, switches, selectors, dials, or toggle buttons. The user input devices114include electromechanical or capacitance sensors. In further embodiments the user input devices114also provide a haptic response. In embodiments, the infotainment system138may include an interactive display140including sensors, such as capacitance sensors or resistance sensors for detecting touch. In yet further embodiments, a user may utilize an external communication device144for providing input to the controller118through the communication system124. The user input devices114are connected to the controller118, either wired or wirelessly, and allow for the user to configure the messages108, and the priority of the messages108and associated requests from operational applications, displayed on the adaptive heads-up display system101.

The external communication device144may include a smart phone, tablet or computer that communicates to the vehicle100wirelessly through one or more wireless communication protocols such as through one or more networks utilizing local area networks using IEEE 802.11 a, b, g, n, p, ac, ax protocols, BLUETOOTH®, cellular networks including 2G, 3G, 4G/LTE, and 5G networks using various communication protocols such as global system for mobile communications (GSM), code division multiple access (CDMA), general packet radio service (GPRS), wideband code division multiple access (W-CDMA), enhanced general packet radio service (E-GPRS), CDMA2000, and universal mobile technology system (UTMS), low power wide-area networks (LPWAN), mobile satellite communications, and combinations thereof.

With reference again toFIG.1, the trip monitoring sensors116provide driving environment perception and includes one or more sensors used to monitor various aspects of the vehicle100and the environment surrounding the vehicle100, particularly while the vehicle100is in motion. The trip monitoring sensors116include, for example, one or more of the following: one or more visible light cameras, one or more infrared radiation cameras, one or more radars, one or more light detection and ranging (lidar) sensors, one or more odometers, one or more ground penetrating radar (GPR) sensors, one or more ground positioning receivers, one or more steering angle sensors, one or more tire pressure sensors, one or more cameras (e.g., optical cameras and/or thermal cameras, such as a rear camera and/or a front camera), one or more gyroscopes, one or more accelerometers, one or more speed sensors, one or more steering angle sensors, one or more ultrasonic sensors, one or more inertial measurement units (IMUs) and/or other sensors. Each of the trip monitoring sensors116are configured to generate a signal, received by the controller118, that represents information regarding the sensed observable conditions of the environment surrounding the vehicle100.

As noted above, the controller118includes a processor120. In embodiments, the processor120is a custom made or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors associated with the on-board adaptive heads-up display system102, a semiconductor-based microprocessor (in the form of a microchip or chip set), a macroprocessor, a combination thereof, or generally a device for executing instructions. The computer readable media122, in embodiments, includes volatile and nonvolatile storage in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM), for example. KAM is a persistent or non-volatile memory that may be used to store various operating variables while the processor120is powered down. The computer readable media122may be implemented using a number of memory devices such as PROMs (programmable read-only memory), EPROMs (electrically PROM), EEPROMs (electrically erasable PROM), flash memory, or another electric, magnetic, optical, or combination memory devices capable of storing data, some of which represent executable instructions, used in conjunction with the user monitoring system112to discern the direction the user is looking or used in conjunction with the trip monitoring sensors116to analyze the environment surrounding the vehicle100.

The controller118is programmed to execute instructions for adapting the messages108on the heads-up display106with the projector110using data collected from the user monitoring system112, the trip monitoring sensors116, the user input devices114, and remote system104. The instructions may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions. The instructions, when executed by the processor120, receive and process signals from the user monitoring system112, the trip monitoring sensors116, the user input devices114, and remote system104, performs logic, calculations, methods and/or algorithms. In embodiments, the processor120is integrated into the controller118as shown inFIG.1or separate from the controller118.

In further embodiments, the user monitoring system112, includes a processor (not illustrated) for processing image data generated by the attention sensor136, which sends data regarding the images to the processor120. In addition, computer readable media (not illustrated) and a buffer may also be provided in the user monitoring system112. In further embodiments, the infotainment system138includes a processor (not illustrated) for processing data received by the user input devices114as well as for providing an auxiliary display. In addition, computer readable media (not illustrated) and a buffer may also be provided in the infotainment system138.

The controller118also receives requests to display messages108on the adaptive heads-up display system101received from various operating applications being executed by the controller118or other controllers and processors throughout the vehicle100. Requests to display messages108may also be received by the controller118from external communication devices144. Operational applications are understood as a set of instructions that are designed to carry out a specific task such as, e.g., advanced driver assistance systems, body electronics (vehicle speed, engine rpm, battery charge status, etc.), infotainment systems, autonomous driving systems, telematics systems (including e-call, ONSTAR®, etc.) and power train systems.

The computer readable media122stores data for use in making determinations regarding which requests to display messages108on the heads-up display106should be selected. The computer readable media122stores data relating to the user monitoring system112, the trip monitoring sensors116, the user input devices114, and received from the remote system104. In various embodiments, the computer readable media122stores images for comparison to images captured by the attention sensor136using the processor120or stores messages108to display on the heads-up display106. The computer readable media122is non-transitory and can include one or more storage devices, articles of manufacture, or the like. In embodiments, computer readable media122include computer system memory, e.g., RAM (random access memory), ROM (read only memory); semiconductor memory, e.g., EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), flash memory; magnetic or optical disks or tapes; and/or the like. The computer readable media122may also include computer-to-computer connections, for example, when data is transferred or provided over a network or another communications connection (either wired, wireless, or a combination thereof). Any combination(s) of the above examples is also included within the scope of the computer readable media122. The computer readable media122may be part of the controller118or separate from the controller118.

The computer readable media122includes a buffer126. The buffer126is a physical region of computer readable media122that is used to temporarily store data between devices such as the processors120and the remaining portions of the computer readable media122or the processors120and the communication system124. Alternatively, or additionally, the buffer126is implemented in an application executed by the processors120and the data is stored in a physical region of computer readable media122. The buffer126is used to store, e.g., real-time adaptive data.

The communication system124is in communication with the controller118and is configured to wirelessly communicate information to and from the remote system104, such as but not limited to, other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems at a remote call center (e.g., ON-STAR by GENERAL MOTORS), and external communication device144. In certain embodiments, the communication system124is a wireless communication system configured to communicate wirelessly through one or more wireless communication protocols such as through one or more networks utilizing local area networks using IEEE 802.11 a, b, g, n, p, ac, ax protocols, BLUETOOTH®, cellular networks including 2G, 3G, 4G/LTE, and 5G networks using various communication protocols such as global system for mobile communications (GSM), code division multiple access (CDMA), general packet radio service (GPRS), wideband code division multiple access (W-CDMA), enhanced general packet radio service (E-GPRS), CDMA2000, and universal mobile technology system (UTMS), low power wide-area networks (LPWAN), mobile satellite communications, and combinations thereof. However, additional or alternate communication methods, such as a dedicated short-range communications (DSRC) channel, are also considered within the scope of the present disclosure. DSRC channels refer to one-way or two-way short-range to medium-range wireless communication channels specifically designed for automotive use and a corresponding set of protocols and standards. Accordingly, the communication system124may include one or more antennas and/or transceivers for receiving and/or transmitting signals, such as cooperative sensing messages (CSMs).

The remote system104, including, e.g., cloud services, also includes a controller150. The controller150is connected to a remote processor152and remote computer readable media154as well as a remote communication system156. Remote is understood herein to indicate that the remote processor, remote computer readable media, remote communication system, etc. are not located on-board the vehicle100. In embodiments, the remote processors152is a custom made or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors associated with the controller150, a semiconductor-based microprocessor (in the form of a microchip or chip set), a macroprocessor, a combination thereof, or generally a device for executing instructions. The remote computer readable media154, in embodiments, includes volatile and nonvolatile storage in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM), for example. KAM is a persistent or non-volatile memory that may be used to store various operating variables while the remote processor152is powered down. The remote computer readable media154may be implemented using a number of memory devices such as PROMs (programmable read-only memory), EPROMS (electrically PROM), EEPROMs (electrically erasable PROM), flash memory, or another electric, magnetic, optical, or combination memory devices capable of storing data, some of which represent executable instructions, used by the controller150in controlling the vehicle100. The controller150is programmed to execute instructions for adapting messages108by predicting user interest in certain messages108, using data previously collected by the user monitoring system112, and events occurring in the surrounding environment.

Again, the instructions may include one or more separate applications, each of which comprises an ordered listing of executable instructions for implementing logical functions. The instructions, when executed by the remote processor152, receive and process signals from the user monitoring system112, user input devices114, and trip monitoring sensors116, perform logic, calculations, methods and/or algorithms for displaying on the heads-up display the adapted information, forwarded by the communication system124in the controller118. The remote system controller150also receives information regarding user behavior history to make predictions regarding the information the user may be interested in seeing displayed on the heads-up display106.

The remote computer readable media154stores data related to the surrounding environment and the user behavior history. The remote computer readable media154is non-transitory and can include one or more storage devices, articles of manufacture, or the like. In embodiments, remote computer readable media154includes computer system memory, e.g., RAM (random access memory), ROM (read only memory); semiconductor memory, e.g., EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), flash memory; magnetic or optical disks or tapes; and/or the like. The remote computer readable media154may also include computer-to-computer connections, for example, when data is transferred or provided over a network or another communications connection (either wired, wireless, or a combination thereof). Any combination(s) of the above examples is also included within the scope of the remote computer readable media154. The remote computer readable media154may be part of the controller150or separate from the controller150.

The remote computer readable media154includes a remote buffer158. The buffer is a physical region of remote computer readable media154that is used to temporarily store data between devices such as the between the remote processor152and the remainder of the remote computer readable media154or between the remote processor152and remote communication system156. Alternatively, or additionally, the remote buffer158is implemented in an application executed by the remote processor152and the data is stored in a physical region of remote computer readable media154. The remote buffer158is used to store the predictive data as determined by the remote processor152.

The remote communication system156is in communication with the remote controller150and is configured to wirelessly communicate information to and from the controller118in the on-board adaptive heads-up display system102of the vehicle100and external communication devices144. In certain embodiments, the remote communication system156is a wireless communication system configured to communicate wirelessly through one or more wireless communication protocols such as through one or more networks utilizing local area networks using IEEE 802.11 a, b, g, n, p, ac, ax protocols, BLUETOOTH®, cellular networks including 2G, 3G, 4G/LTE, and 5G networks using various communication protocols such as global system for mobile communications (GSM), code division multiple access (CDMA), general packet radio service (GPRS), wideband code division multiple access (W-CDMA), enhanced general packet radio service (E-GPRS), CDMA2000, and universal mobile technology system (UTMS), low power wide-area networks (LPWAN), mobile satellite communications, and combinations thereof. However, additional or alternate communication methods, such as a dedicated short-range communications (DSRC) channel, are also considered within the scope of the present disclosure. DSRC channels refer to one-way or two-way short-range to medium-range wireless communication channels specifically designed for automotive use and a corresponding set of protocols and standards. Accordingly, the remote communication system156may include one or more antennas and/or transceivers for receiving and/or transmitting signals, such as cooperative sensing messages (CSMs). The remote communication system156is configured to wirelessly communicate information between the vehicle100and another vehicle. Further, the remote communication system156is configured to wirelessly communicate information between the remote system104and infrastructure or other vehicles.

FIG.3, with references toFIGS.1,2A, and2B, illustrates a method300for the adaption of the messages108displayed on the heads-up display106of a vehicle100by determining which requests from operational applications should be selected for display. The method utilizes two simultaneous processes, one on-board the vehicle100using the on-board adaptive heads-up display system102and the other using the remote system104. The method300or parts thereof may be implemented in a computer program product embodied in the computer readable media122found on-board the vehicle as well as in the remote computer readable media154and includes instructions usable, or executable, by the processor120of the controller118on-board the vehicle100and in the remote processor152of the remote controller150in the remote system104across which processing and data storage may take place. The computer program product may include one or more software programs comprised of program instructions in source code, object code, executable code or other formats; one or more firmware programs; or hardware description language (HDL) files; and any program related data. The data may include data structures, look-up tables, or data in any other suitable format. The program instructions may include program modules, routines, programs, objects, components, and/or the like. The computer program may be executed on one computer or on multiple computers in communication with one another. It is therefore to be understood that the method300may be at least partially performed by any electronic articles and/or devices capable of carrying out instructions corresponding to one or more steps of the method300. The method300is repeated, wherein the repetition is based on a selected time period such as once every 0.01 seconds to once every 1 second or triggered by a request by an operational application to display a message108on the heads-up display106.

On board the vehicle100, the method300begins at block302when the controller118receives a number of requests303through303nfrom one or more operational applications to display a message108on the heads-up display106. Again, the operational applications may include, e.g., advanced driver assistance systems, body electronics (vehicle speed, engine rpm, battery charge status, etc.), infotainment systems, autonomous driving systems, telematics systems (including e-call, ONSTAR®, etc.) and power train systems. At block304the requests are received by an arbitration gateway, which is embodied by instructions executed by the processor120associated with the controller118. The arbitration gateway receives the requests to display one or more messages108, prioritizes the requests to display the message108associated with each request, and filters the requests and associated messages108that are low-priority, conflicted, or out-of-scope of the user's interest. In embodiments, the filtered requests may be dropped and not displayed on the heads-up display106. The arbitration gateway performs these functions through the execution of instructions by the processor120using, e.g., the steps illustrated in method400illustrated inFIG.4.

Referring now toFIG.4, at block402the arbitration gateway checks the ID of a request received. The ID is understood as a signifier that provides identifying information regarding the request, which is allocated by the controller118. The output of block402is a first tuple (a set of ordered elements) for each request, the first tuple including the following elements: the ID, the priority, heads-up display attributes, and raw data. The priority may be normalized and ranked on a scale of greater than zero and up to and including one, wherein the higher priority the lower the ranking. In embodiments, heads-up display attributes include the format of the messages108to be displayed, including the location the messages108should be projected on the heads-up display106, the format of the messages108such as the shape of the icon or characters, the color of the messages108, whether the messages108is solid or blinking, etc. The configuration of heads-up display attributes is performed during user preference configuration, wherein the user configures the heads-up display attributes using the user input devices114. User preference configuration may occur upon initialization of the on-board adaptive heads-up display system102, such as before the user's first trip operating the vehicle100, after the vehicle100is first turned on, or after the vehicle100is switched from a demo mode to a standard mode. Finally, the raw data is the data providing the basis of the request. In embodiments, the raw data may include the vehicle speed, the tachometer reading, the name of a song playing on the infotainment system, the name of a road that is being traversed by the vehicle100, an overlay image representing a particular hazard, etc. The data is converted into the messages108displayed.

At block404, the arbitration gateway, as executed by instructions with the processor120, is used to determine which messages108are to be displayed on the heads-up display106based on real-time information and user behavior history. The arbitration gateway polls various the operating applications to extract emergency attributes and safety related data from the different applications and then further categorizes the information from the first tuple. Again, in embodiments, the emergency level may be normalized on a scale of greater than zero and up to and including one, wherein the lower value indicates a greater urgency to display the messages108on the heads-up display106. A second tuple is defined for each request including the following elements: the ID, the priority, emergency level, heads-up display attributes, and raw data.

At block406a third tuple is derived for each request including the following elements: the ID, the priority, emergency level, user attention status, heads-up display attributes, and raw data. The user attention status is derived from the user attention data from the user monitoring system112at block306ofFIG.3and indicates the eye and head location of the user. The user monitoring system112may collect data, continuously repeating at given intervals of time or upon the request of the arbitration gateway. If the user's focus is far from the location of the where the messages108are displayed, or a specific message108is displayed, on the heads-up display106, a normalized user attention status is calculated and in embodiments, is ranked on a scale of greater than zero and up to and including one, wherein the lower the number the more focus the user has on the location of the messages108in the heads-up display106.

At block408, the user behavior history, of the heads-up display106is referenced at block308ofFIG.3to further augment the third data tuple to create a fourth data tuple which includes the following elements: the ID, the priority, emergency level, user attention status, user behavior history, heads-up display attributes, and raw data. For example, if the user has aggressive driving behavior such as relatively high acceleration times, relatively high average steering wheel changing ratio, relatively high jerk ratio, etc., or relatively high confidence in their driving skills without depending on, e.g., advanced driver assistance systems such as electronic stability control, adaptive cruise control, or lane departure warning, the data generated from active safety operational applications may not be as useful to the user. Thus, the fourth tuple, a filtering tuple, includes a user interest factor that quantifies the level of interest of the user in the data. In embodiments, the user interest is normalized on a scale of greater than zero and up to and including one, wherein a value closer to zero represents more interest the user has shown in the data. Information regarding the user behavior history may be updated continuously at given intervals or updated upon request by the arbitration gateway.

With reference again toFIG.3, using the filtering tuple in the arbitration gateway at block304, the requests for displaying messages108are filtered and a buffer of real-time adaptive and customized messages108requested by the various operational applications is created at block310. The buffer126of messages108for display is maintained in, for example, the computer readable media122.

While the controller118executes instructions in the onboard adaptive heads-up display system102, the remote system104proceeds at block312. Block312is a service pool, stored on e.g., the computer readable media154, where programs and models are saved and ready to be used. For example, loading the map data, pre-trained models, predefined user behavior models (generalized information with statistical information and prior knowledge) and previously requested ID's for displaying messages108on the heads up display106. At block314a user interest prediction model is executed by the remote controller150. The user interest prediction model polls data regarding the user behavior history collected at block308. In addition, the user interest prediction model also polls data collected at block316regarding the surrounding environment. This data includes, weather data, speed limits, work zone locations, and accident information. The data may further include mapping information.

The user interest prediction model executed by the controller150at block314is, in embodiments, a multi-category regression model, which predicts the level of interest of a user in messages108received from various operational applications at block302at different traffic events. For example, based on prior behavior gathered from the user monitoring system112at block306it may be understood that a user has more interest in traffic lights and work zone detours due to behavior exhibited when the user previously approached traffic lights and work zone detours. The output of the user prediction model is a fifth, prioritizing tuple including the following data: event_ID, priority, emergency level, predictive user interest, heads-up display attributes, lead time, raw event data. The event_ID is understood as an identifier for a particular event that occurred in the past at a particular location and is allocated by the controller150. The priority may be ranked on a scale of greater than zero and up to and including one, wherein the higher priority the lower the ranking, the emergency level may also be normalized on a scale of on a scale of greater than zero and up to and including one and determined based upon the historical data used in the arbitration gateway at block304. Similarly, the predictive user interest may also be normalized on a scale of greater than zero and up to and including one based on data gathered from the user monitoring system112at block306. Heads-up display attributes include the user preference configuration or an altered user preference configuration such as previous changes to the configuration of the format of the messages108that occurred during previous driving events. Lead time is the amount of time messages108should be shown on the heads-up display before an event occurs, such as arriving at an intersection, arriving at an upcoming accident, etc., and may take into account the vehicle speed, etc. Finally, the raw data is the data that provides the basis of the request. In embodiments, the raw data may include the vehicle speed, the tachometer reading, the name of a song playing on the infotainment system, the name of a road that is being traversed by the vehicle100, etc. The output of the user interest prediction model at block314is saved in a remote buffer158at block318.

At block320the predictive data, i.e., stored in a remote buffer158at block318, and the real-time adaptive data, stored in buffer126at block310are combined using a prioritization procedure to create a prioritizing score and assign a priority to each request to display a message108. The data may be combined in either the controller118or in the remote controller150. The prioritization procedure executed at block320weights the data provided in the fourth tuple and the fifth tuple. In embodiments, the final prioritization score is the sum of a) a product of the ID, the priority, emergency level, user attention status, user behavior history of the fourth tuple is weighted by a factor w and b) the product of the event_ID, priority, emergency level, predictive user interest of the fifth tuple is weighted by a factor (1−w). The default value for w is, in embodiments, e.g., 0.7. The smaller the value of the product, the relatively higher priority level the request to display a message108on the heads-up display106has. If the score is higher than a threshold prioritization level, e.g., 0.5, the request and associated message108are considered a low priority request. The request and associated message108will be dropped and not displayed on the heads-up display106. If it is determined that a dropped request and associated message108is safety related, feedback may be provided to the user using at least one of sound emitted by the infotainment system138and a haptic response created by one or more vibration motors142(seeFIG.1) in the steering wheel134at block322. However, if the user specifically raises the priority of the dropped request and associated message108, then the message108will be displayed on the heads-up display106.

At block326, the associated messages108of requests assigned the highest priority, for example, the top three or four, are then displayed on the heads-up display106using adaptive color and status based configured by the user using the user input devices114at block328. The user preference learning model at block328is calculated from the data received by the user monitoring system112and the user preference configuration. As noted above, the user preference configuration may be configured upon initialization of the on-board adaptive heads-up display system102, such as before the users first trip using the vehicle100after the vehicle100is turned on, or after the vehicle100is switched from a demo mode to a standard mode.

In embodiments, a number of parameters may be learned, rather than being predefined based upon field testing or the use of a shadow system where user behavior is being recorded by the user monitoring system112whether the vehicle100is being used in an autonomous mode, a semi-autonomous mode, or a manual mode. These parameters include the weight factor w, the threshold prioritization level, the number of highest priority requests displayed on the heads-up display106, the weights and relationships used to determine the user interest factor and the predictive user interest factor, as well as the lead time, which may be altered according to, e.g., an ongoing situation.

Several advantages are offered by the adaptive heads-up display described herein, including prioritizing information of interest to the user exercising control over the vehicle. An additional advantage includes the reduction of distraction by minimizing information displayed on the heads-up display, such as by omitting information deemed to be of less importance to the user. A further advantage includes the relative optimization and maximization of resources for user. Yet a further advantage is that data calculated by the user interest prediction model may be transferred between users displaying similar user behavior history.