IDENTIFYING PERSON OF INTEREST WITH AUGMENTED REALITY HEAD-UP DISPLAY

A system for identifying a person of interest for an occupant within a vehicle with a head-up system includes a controller in communication with an external scene camera adapted to capture images of an external environment in proximity to the vehicle, identify at least one person of interest located within the external environment in proximity to the vehicle, characterize the at least one person of interest, determine appropriate augmentation graphics related to the identified at least one person of interest to be displayed for the occupant, and display, with the head-up display system, the augmentation graphics onto an inner surface of a windshield of the vehicle.

The present disclosure relates to systems and methods for displaying appropriate augmentation graphics in a vehicle to alert an occupant therein to the presence of persons of interest in proximity to the vehicle.

To increase occupant awareness and convenience, vehicles may be equipped with display systems which are configured to provide information about an environment surrounding the vehicle to an occupant. Display systems may use human-interface devices, such as, for example, touchscreens, to provide information to the occupant. Display systems may also use head-up displays (HUD) to provide information to the occupant. To control and/or illuminate the HUD, a graphics projector may be used to project graphics upon a surface (e.g., a windscreen) of the vehicle. However, current graphics projectors may utilize high-intensity, concentrated light sources, such as, for example, lasers. High-intensity, concentrated light sources may cause the occupant discomfort when reflecting off of surfaces of the vehicle. Additionally, high-intensity, concentrated light sources may increase weight and resource use of the vehicle.

Thus, while current graphics projectors achieve their intended purpose, there is a need for a new and improved system for graphics projection for a vehicle which is adapted to display appropriate augmentation graphics to alert an inform an occupant to the presence and nature of persons of interest in proximity to the vehicle.

SUMMARY

According to several aspects of the present disclosure, a method of identifying a person of interest for an occupant within a vehicle with a head-up system includes: identifying, with a controller in communication with an external scene camera adapted to capture images of an external environment in proximity to the vehicle, at least one person of interest located within the external environment in proximity to the vehicle; characterizing, with the controller, the at least one person of interest; determining, with the controller, appropriate augmentation graphics related to the identified at least one person of interest to be displayed for the occupant; and displaying, with the head-up display system, the augmentation graphics onto an inner surface of a windshield of the vehicle.

According to another aspect, the method further includes sending, via a communication network within the vehicle, information related to the characterization of the at least one person of interest to a vehicle control module of the vehicle, and initiating, with the vehicle control module, vehicle maneuvers based on the characterization of the identified at least one person of interest.

According to another aspect, the identifying, with a controller in communication with an external scene camera adapted to capture images of an external environment in proximity to the vehicle, at least one person of interest located within the external environment in proximity to the vehicle further includes: scanning, with the external scene camera, the external environment in proximity to the vehicle; and identifying, with a computer vision algorithm within the controller, people located within the external environment in proximity to the vehicle.

According to another aspect, the identifying, with a computer vision algorithm within the controller, people located within the external environment in proximity to the vehicle further includes: collecting, from remote sources in communication with the controller, data related to the identified people located within the external environment in proximity to the vehicle; and identifying, with the controller, at least one person of interest within the identified people based on the data collected from the remote sources.

According to another aspect, the collecting, from remote sources in communication with the controller, data related to the identified people located within the external environment in proximity to the vehicle further includes collecting data related to the identified people located within the external environment in proximity to the vehicle from at least one of social media applications, criminal records, business networks, motor vehicle records, real estate records, voting records, and school records.

According to another aspect, the identifying, with the controller, at least one person of interest within the identified people based on the data collected from the remote sources further includes identifying at least one person of interest within the identified people who are at least one of: a follower of the occupant on social media, followed by the occupant on social media, a friend of the occupant, a family member of the occupant, a celebrity, a co-worker of the occupant, a person who has similar interest as the occupant, and a person who works in the same industry as the occupant.

According to another aspect, the characterizing, with the controller, the at least one person of interest further includes characterizing, with a cloud computing algorithm in communication with the controller, the at least one person of interest as one of friendly or not-friendly based on the data related to the identified people collected from the remote sources.

According to another aspect, the displaying, with the head-up display system, the appropriate augmentation graphics onto an inner surface of a windshield of the vehicle further includes: determining, with an occupant monitoring system, a location of the occupant's head and eyes; calculating a location to display the appropriate augmentation graphics on the inner surface of the windshield based on the position of the head and eyes of the occupant, wherein the augmentation graphics appear in proximity to the identified at least one person of interest when viewed by the occupant; and displaying the appropriate augmentation graphics on the inner surface of the windshield by: projecting, with a light source of a digital light projector, an excitation light; focusing, with a condensing lens, the excitation light from the light source; splitting, with a color filter, the focused excitation light into red, green and blue light; focusing, with a shaping lens, the excitation light passing through the color filter; re-directing, with a digital micro-mirror device (DMD), the excitation light; receiving, with a projection lens, the excitation light from the digital micro-mirror device; and projecting, with the projection lens, the excitation light to the inner surface of the windshield at a location on the inner surface of the windshield based on the position of the head and eyes of the occupant, wherein light emitting particles within the windshield emit visible light in response to absorbing the excitation light, and the appropriate augmentation graphics appear in proximity to the identified at least one person of interest when viewed by the occupant.

According to another aspect, the method further includes: determining, with the controller, if the windshield is in a privacy mode, wherein the windshield is opaque and the external environment is not visible to the occupant through the windshield and the occupant is not visible to persons within the external environment; and when the windshield is in the privacy mode, deactivating the privacy mode prior to displaying appropriate augmentation graphics on the inner surface of the windshield.

According to several aspects of the present disclosure, a system for identifying a person of interest for an occupant within a vehicle with a head-up system includes: a controller in communication with an external scene camera adapted to: capture images of an external environment in proximity to the vehicle; identify at least one person of interest located within the external environment in proximity to the vehicle; characterize the at least one person of interest; determine appropriate augmentation graphics related to the identified at least one person of interest to be displayed for the occupant; and display, with the head-up display system, the augmentation graphics onto an inner surface of a windshield of the vehicle.

According to another aspect the system further includes a communication network within the vehicle adapted to send information related to the characterization of the at least one person of interest to a vehicle control module of the vehicle, the vehicle control module adapted to initiate vehicle maneuvers based on the characterization of the identified at least one person of interest.

According to another aspect, when identifying at least one person of interest located within the external environment in proximity to the vehicle, the controller is further adapted to: scan, with the external scene camera, the external environment in proximity to the vehicle; and identify, with a computer vision algorithm within the controller, people located within the external environment in proximity to the vehicle.

According to another aspect, when identifying at least one person of interest located within the external environment in proximity to the vehicle, the controller is further adapted to: collect, from remote sources in communication with the controller, data related to the identified people located within the external environment in proximity to the vehicle; and identify at least one person of interest within the identified people based on the data collected from the remote sources.

According to another aspect, the controller is adapted to collect data related to the identified people located within the external environment in proximity to the vehicle from at least one of social media applications, criminal records, business networks, motor vehicle records, real estate records, voting records, and school records.

According to another aspect, the controller is adapted to identify at least one person of interest who is at least one of: a follower of the occupant on social media, followed by the occupant on social media, a friend of the occupant, a family member of the occupant, a celebrity, a co-worker of the occupant, a person who has similar interest as the occupant, and a person who works in the same industry as the occupant.

According to another aspect, the controller is adapted to utilize a cloud computing algorithm in communication with the controller to characterize the at least one person of interest as one of friendly or not-friendly based on the data related to the identified people collected from the remote sources.

According to another aspect, the digital light projector includes: a light source adapted to project an excitation light; a condensing lens adapted to focus the excitation light from the light source; a color filter adapted to split the focused excitation light into red, green and blue light; a shaping lens adapted to focus the excitation light passing through the color filter; a digital micro-mirror device (DMD) adapted to re-direct the excitation light; and a projection lens adapted to receive the excitation light from the digital micro-mirror device and project the excitation light to the inner surface of the windshield; and, the head-up system includes an occupant monitoring system adapted to receive information related to a position of the head and eyes of the occupant within the vehicle, wherein the digital light projector is adapted to display the augmentation graphics at a location on the inner surface of the windshield based on the position of the head and eyes of the occupant such that the augmentation graphics appear in proximity to the identified at least one person of interest when viewed by the occupant.

According to another aspect, the windshield is operable in a privacy mode, wherein the windshield is opaque and the external environment is not visible to the occupant through the windshield and the occupant is not visible to persons within the external environment, the controller adapted to determine if the windshield is in a privacy mode, and, when the windshield is in the privacy mode, deactivate the privacy mode prior to displaying appropriate augmentation graphics on the inner surface of the windshield.

The figures are not necessarily to scale and some features may be exaggerated or minimized, such as to show details of particular components. In some instances, well-known components, systems, materials or methods have not been described in detail in order to avoid obscuring the present disclosure. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present disclosure.

DETAILED DESCRIPTION

The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding technical field, background, brief summary or the following detailed description. It should be understood that throughout the drawings, corresponding reference numerals indicate like or corresponding parts and features. As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic, and/or processor device, individually or in any combination, including without limitation: application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that executes one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality. Although the figures shown herein depict an example with certain arrangements of elements, additional intervening elements, devices, features, or components may be present in actual embodiments. It should also be understood that the figures are merely illustrative and may not be drawn to scale.

As used herein, the term “vehicle” is not limited to automobiles. While the present technology is described primarily herein in connection with automobiles, the technology is not limited to automobiles. The concepts can be used in a wide variety of applications, such as in connection with aircraft, marine craft, other vehicles, and consumer electronic components.

In accordance with an exemplary embodiment,FIG.1shows a vehicle10with an associated head-up display system within the vehicle10in accordance with various embodiments. In general, the head-up display system11works in conjunction with other systems within the vehicle10to display various information and infotainment content for the passenger. The vehicle10generally includes a chassis12, a body14, front wheels16, and rear wheels18. The body14is arranged on the chassis12and substantially encloses components of the vehicle10. The body14and the chassis12may jointly form a frame. The front wheels16and rear wheels18are each rotationally coupled to the chassis12near a respective corner of the body14.

In various embodiments, the vehicle10is an autonomous vehicle and the system11is incorporated into the autonomous vehicle10. An autonomous vehicle10is, for example, a vehicle10that is automatically controlled to carry passengers from one location to another. The vehicle10is depicted in the illustrated embodiment as a passenger car, but it should be appreciated that any other vehicle including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), etc., can also be used. In an exemplary embodiment, the vehicle10is equipped with a so-called Level Four or Level Five automation system. A Level Four system indicates “high automation”, referring to the driving mode-specific performance by an automated driving system of all aspects of the dynamic driving task, even if a human driver does not respond appropriately to a request to intervene. A Level Five system indicates “full automation”, referring to the full-time performance by an automated driving system of all aspects of the dynamic driving task under all roadway and environmental conditions that can be managed by a human driver.

As shown, the vehicle10generally includes a propulsion system20, a transmission system22, a steering system24, a brake system26, a sensor system28, an actuator system30, at least one data storage device32, a vehicle controller34, and a communication system36. In an embodiment in which the vehicle10is an electric vehicle, there may be no transmission system22. The propulsion system20may, in various embodiments, include an internal combustion engine, an electric machine such as a traction motor, and/or a fuel cell propulsion system. The transmission system22is configured to transmit power from the propulsion system20to the vehicle's front wheels16and rear wheels18according to selectable speed ratios. According to various embodiments, the transmission system22may include a step-ratio automatic transmission, a continuously-variable transmission, or other appropriate transmission. The brake system26is configured to provide braking torque to the vehicle's front wheels16and rear wheels18. The brake system26may, in various embodiments, include friction brakes, brake by wire, a regenerative braking system such as an electric machine, and/or other appropriate braking systems. The steering system24influences a position of the front wheels16and rear wheels18. While depicted as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of the present disclosure, the steering system24may not include a steering wheel.

The sensor system28includes one or more sensing devices40a-40nthat sense observable conditions of the exterior environment and/or the interior environment of the autonomous vehicle10. The sensing devices40a-40ncan include, but are not limited to, radars, lidars, global positioning systems, optical cameras, thermal cameras, ultrasonic sensors, and/or other sensors. The cameras can include two or more digital cameras spaced at a selected distance from each other, in which the two or more digital cameras are used to obtain stereoscopic images of the surrounding environment in order to obtain a three-dimensional image. The plurality of sensing devices40a-40nis used to determine information about an environment surrounding the vehicle10. In an exemplary embodiment, the plurality of sensing devices40a-40nincludes at least one of a motor speed sensor, a motor torque sensor, an electric drive motor voltage and/or current sensor, an accelerator pedal position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor. In another exemplary embodiment, the plurality of sensing devices40a-40nfurther includes sensors to determine information about the environment surrounding the vehicle10, for example, an ambient air temperature sensor, a barometric pressure sensor, and/or a photo and/or video camera which is positioned to view the environment in front of the vehicle10. In another exemplary embodiment, at least one of the plurality of sensing devices40a-40nis capable of measuring distances in the environment surrounding the vehicle10. In a non-limiting example wherein the plurality of sensing devices40a-40nincludes a camera, the plurality of sensing devices40a-40nmeasures distances using an image processing algorithm configured to process images from the camera and determine distances between objects. In another non-limiting example, the plurality of vehicle sensors40a-40nincludes a stereoscopic camera having distance measurement capabilities.

In one example, at least one of the plurality of sensing devices40a-40nis affixed inside of the vehicle10, for example, in a headliner of the vehicle10, having a view through the windshield56of the vehicle10. In another example, at least one of the plurality of sensing devices40a-40nis affixed outside of the vehicle10, for example, on a roof of the vehicle10, having a view of the environment surrounding the vehicle10. It should be understood that various additional types of sensing devices, such as, for example, LiDAR sensors, ultrasonic ranging sensors, radar sensors, and/or time-of-flight sensors are within the scope of the present disclosure. The actuator system30includes one or more actuator devices42a-42nthat control one or more vehicle10features such as, but not limited to, the propulsion system20, the transmission system22, the steering system24, and the brake system26.

In various embodiments, one or more instructions of the vehicle controller34are embodied in a trajectory planning system and, when executed by the at least one data processor44, generates a trajectory output that addresses kinematic and dynamic constraints of the environment. For example, the instructions receive as input process sensor and map data. The instructions perform a graph-based approach with a customized cost function to handle different road scenarios in both urban and highway roads.

The vehicle controller34is a non-generalized, electronic control device having a preprogrammed digital computer or processor, memory or non-transitory computer readable medium used to store data such as control logic, software applications, instructions, computer code, data, lookup tables, etc., and a transceiver [or input/output ports]. Computer readable medium includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory. A “non-transitory” computer readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals. A non-transitory computer readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device. Computer code includes any type of program code, including source code, object code, and executable code.

Referring toFIG.2, the controller34of the head-up system11is in communication with an external scene camera52. The various sensing devices40a-40ncollect and provide information to the controller34. The sensing devices40a-40ninclude the external scene camera52or image capturing device taking periodic or sequential images representing a view from the vehicle10. The controller34is adapted to capture images of an external environment54in proximity to the vehicle10with the external scene camera52. The controller34is further adapted to identify at least one person of interest56located within the external environment54in proximity to the vehicle10. When identifying at least one person of interest56located within the external environment54in proximity to the vehicle10, the controller34is further adapted to scan, with the external scene camera52, the external environment54in proximity to the vehicle10, and to identify, with a computer vision algorithm58within the controller34, people located within the external environment54in proximity to the vehicle10.

Referring toFIG.3, the external scene camera52captures images of a group of people60standing in proximity to the vehicle10. The controller34, using the computer vision algorithm58, identifies the individual people comprising the group60, and further, collects, from remote sources48in communication with the controller34, data related to the identified people located within the external environment54in proximity to the vehicle10. In an exemplary embodiment, the controller34accesses remote sources48including at least one of social media applications, criminal records, business networks, motor vehicle records, real estate records, voting records, and school records to identify the individuals comprising the group of people60. Such remote sources48can includes any publicly accessible database.

The controller34is further adapted to identify at least one person of interest56within the identified group of people60based on the data collected from the remote sources48. By analyzing the data collected from the remote sources48, the controller34identifies individuals within the group of people60who are a person of interest56to the occupant80within the vehicle10. A person of interest56may be any one or more of the following: a follower of the occupant80on social media, followed by the occupant80on social media, a friend of the occupant80, a family member of the occupant80, a celebrity, a co-worker of the occupant80, a person who has similar interest as the occupant80, or a person who works in the same industry as the occupant80. Any person who has some connection to the occupant80, as determined by the controller34analyzing data from the remote sources48, may be identified as a person of interest56to the occupant80.

The controller34is further adapted to characterize the at least one person of interest56. In an exemplary embodiment, the controller34is adapted to utilize a cloud computing algorithm92in communication with the controller34to characterize the at least one person of interest56as one of friendly or not-friendly based on the data related to the identified group of people60collected from the remote sources48. The controller34analyzes the data collected from remote sources48to determine if each one of the identified persons of interest56have positive or negative feelings toward the occupant80. The controller34may identify a person of interest56as un-friendly or even hostile depending on communications or comments made by the person of interest56and/or the occupant80on social media or other remote sources48.

In an exemplary embodiment, after characterizing any persons of interest56, the controller34is adapted to send, via a communication network106within the vehicle10, information related to the characterization of the at least one person of interest56to a vehicle control module104of the vehicle10. The vehicle control module104is adapted to initiate vehicle maneuvers based on the characterization of the identified at least one person of interest56. For example, if the identified person of interest56is friendly, the vehicle control module104may cause the vehicle10to stop, giving the occupant80an opportunity to interact with the identified person of interest56. Further, the controller34and the vehicle control module104may prompt vehicle infotainment systems to provide audible cues to alert the occupant80of the presence of a friendly person of interest56. Further, the controller34may initiate internal and external speakers and microphones to allow the occupant to communicate with the identified person of interest56. Alternatively, if the identified person of interest56is un-friendly, the controller34and the vehicle control module104may cause the vehicle10to keep moving past the un-friendly person of interest56, or to alter the route of the vehicle10, to avoid the un-friendly person of interest56altogether. Here also, the controller34and the vehicle control module104may prompt vehicle infotainment systems to provide audible cue to alert the occupant80of the presence of the un-friendly person of interest56.

The controller34is further adapted to determine appropriate augmentation graphics94related to the identified at least one person or persons of interest56to be displayed for the occupant80, and to display, with the head-up display system11, the appropriate augmentation graphics94onto an inner surface96of a windshield98of the vehicle10. Appropriate augmentation graphics94are adapted to inform the occupant80of the identity and categorization of the identified persons of interest56. Referring toFIG.4, the windshield98, as viewed by the occupant80is shown. The occupant80can see the group of people60standing in proximity to the vehicle10. The system11has captured an image of the group of people60, accessed remote sources48to identify the individuals and has identified two persons of interest56for the occupant80. The system11has then displayed appropriate augmentation graphics94for each of the two persons of interest56.

For a first person of interest56A, the system11has identified that the individual is Sally56A, a friend of the occupant80as indicated by labels and conversations from social media sites. The proper augmentation graphic94for Sally56A includes an outline100A to distinguish Sally56A as a person of interest56to the occupant80. Further, the appropriate augmentation graphic94for Sally56A includes a text box102A attached to the outline100A including descriptive information about Sally56A. As shown inFIG.4, the text box102A includes Sally's name, “Sally”, as well as the word “Friend”, indicating that Sally56A is a friend of the occupant80. The appropriate augmentation graphic94may be tailored further based on the categorization of Sally56A. If the system11categorizes Sally56A as “friendly”, then the outline100A may be displayed in a color indicating such, for example, the outline100A may be green.

For a second person of interest56B, the system11has identified that the individual is Lori56B, a co-worker of the occupant80as indicated by labels and conversations from social media sites, and possibly a website of their employer. The proper augmentation graphic94for Lori56B includes an outline100B to distinguish Lori56B as a person of interest56to the occupant80. Further, the appropriate augmentation graphic94for Lori56B includes a text box102B attached to the outline100B including descriptive information about Lori56B. As shown inFIG.4, the text box102B includes Lori's name, “Lori”, as well as the word “Co-Worker”, indicating that Sally56B is a co-worker of the occupant80. The appropriate augmentation graphic94may be tailored further based on the categorization of Lori56B. If the system11categorizes Lori56B as “un-friendly”, then the outline100B may be displayed in a color indicating such, for example, the outline100B may be red.

It should be understood that the appropriate augmentation graphic94may include any information deemed relevant to the occupant80. In an exemplary embodiment, the amount of information provided in a text box102A,102B associated with an identified person of interest56may include more or less details depending on preferences of the occupant80, and the nature of the information. If the system11identifies a person of interest56is particularly hostile toward the occupant80, detailed information may be provided regardless of preferences selected by the occupant80. For example, if the occupant80is a celebrity or politician, and the system11identifies that a person of interest56in proximity to the vehicle10has made threats against the occupant80, an appropriate augmentation graphic94may be adapted to ensure that the occupant80is made aware of such information. In such circumstances, the system11may provide an appropriate augmentation graphic94including an outline that is of a color (i.e. red) or alternating colors and/or includes dynamic features, such as pulsing or increasing brightness/intensity.

Referring again toFIG.3, the head-up display system11includes a digital light projector (DLP)50adapted to project appropriate augmentation graphics94onto the inner surface96of the windshield98of the vehicle10. The DLP50projects appropriate augmentation graphics94that contain various information, as described above. The DLP50is capable of projecting two-dimensional and three-dimensional images that depict objects such as pedestrians, buildings and other vehicles in the external environment54surrounding the vehicle10that may be includes in appropriate augmentation graphics94. In an exemplary embodiment, the DLP50includes a light source62adapted to project an excitation light64, a condensing lens66adapted to focus the excitation light64from the light source62, a color filter68(color wheel) adapted to split the focused excitation light64into red, green and blue light, a shaping lens70adapted to focus the excitation light64passing through the color filter68, a digital micro-mirror device (DMD)72adapted to re-direct the excitation light64, and a projection lens74adapted to receive the excitation light64from the digital micro-mirror device (DMD)72and project the excitation light64to the inner surface96of the windshield98.

Referring again toFIG.2, the vehicle controller34includes a DLP engine76. The DLP engine76includes a programmable processor including programming to monitor various inputs and determine what information should be included within appropriate augmentation graphics94. The DLP engine76can communication directly with various systems and components, or the DLP engine76can alternatively or additionally communicate over a LAN/CAN system. The DLP engine76utilizes information regarding the operational environment of the vehicle10derived from a number of inputs.

The windshield98is equipped with features capable of displaying appropriate augmentation graphics94projected thereupon while remaining transparent or substantially transparent such that the occupant80of the vehicle10can clearly observe outside of the vehicle10through the windshield98. It should be understood that, as depicted, the windshield98is in the front of the vehicle10, other surfaces within the vehicle10could be used for projection, including side windows and a rear window. Additionally, the view on the front windshield98could be continued upon the front vehicle “A-pillars” and onto the side windows as a continuous image.

The DLP engine76includes display software or programming translating requests to display information from the DLP engine76in graphical representations describing the information. The DLP engine76includes programming to compensate for the curved and tilted surface of the windshield98and any other surfaces onto which the appropriate augmentation graphics94are to be projected. The DLP engine76controls the light source62which includes a laser or projector device producing an excitation light64to project the appropriate augmentation graphics94.

An occupant monitoring system78includes sensors known in the art to approximate a location of the head of an occupant80and further the orientation or gaze location of the eyes of the occupant80. Based upon the output of the occupant monitoring system78and input data tracking location information regarding the environment around the vehicle10, the DLP engine76can accurately position the appropriate augmentation graphics94such that the occupant80sees the appropriate augmentation graphics94overlaid with visual images (identified person of interest56) through the windshield98. Referring toFIG.3, the outline100A and text box102A for the first identified person of interest56A and the outline100B and text box102B for the second identified person of interest56B are displayed on the inner surface96of the windshield98at a position between the eyes of the occupant80and the first and second identified person of interest56A,56B, such that, referring toFIG.4, the outline100A and text box102A for the first person of interest56A appears to the occupant80to be located in proper relation to the first person of interest56A, and the outline100B and text box102B for the second person of interest56B appears to the occupant80to be located in proper relation to the second person of interest56B.

The system11described above includes eye sensing and head sensing devices allowing estimation of eye location, allowing registration of images upon the windshield98such that the images correspond to a view of the occupant80. However, it will be appreciated that estimation of head and eye location can be achieved through a number of methods. For example, in a process similar to adjusting rearview mirrors, an operator can use a calibration routine upon entering a vehicle to align graphics to a detected object. In another embodiment, seat position longitudinally in the vehicle10can be used to estimate a position of the occupant's80head. In another embodiment, manual adjustment of a rearview mirror or mirrors can be used to estimate location of an occupant's80eyes. It will be appreciated that a combination of methods, for example, seat position and mirror adjustment angle, can be utilized to estimate occupant80head location with improved accuracy. Many methods to accomplish accurate registration of graphics upon the windshield98are contemplated, and the disclosure is not intended to be limited to the particular embodiments described herein.

The windshield98functions as a medium through which relevant features are observable while serving as a display device upon which the appropriate augmentation graphics94may be displayed. The windshield98is both transparent and capable of displaying images projected by an excitation light64. Referring toFIG.5andFIG.6, an occupant80is able to see an arbitrary object (e.g. cube82) through a substrate84positioned on the windshield98. The substrate84may be transparent or substantially transparent. While the occupant80sees the arbitrary object82through the substrate84, the occupant80can also see images (e.g. circle86and triangle88) that are created at the substrate84. The substrate84may be part of the windshield98, a glass substrate, a plastic substrate, a polymer substrate, or other transparent (or substantially transparent) medium that would be appreciated by one of ordinary skill in the art. Other substrates may complement the substrate84to provide for tinting, substrate protection, light filtering (e.g. filtering external ultraviolet light), and other functions.

FIG.5depicts illumination of transparent displays illuminated with excitation light64(e.g. ultraviolet light or infrared light) from a light source62. The substrate84receives excitation light64from the light source62. The received excitation light64may be absorbed by light emitting material90at the substrate84. When the light emitting material receives the excitation light64, the light emitting material90emits visible light. Accordingly, images (e.g. circle86and triangle88) may be created at the substrate84by selectively illuminating the substrate84with excitation light64.

In an exemplary embodiment, the light emitting material90includes transparent phosphors that are embedded into the substrate84. The transparent phosphors are light emitting particles which fluoresce in response to being excited by the excitation light64. In an exemplary embodiment, the transparent phosphors are red, green, and blue (RGB) phosphors, allowing full color. The use of monochrome and/or two-color phosphors is also within the scope of the present disclosure. When excitation light64is absorbed by the transparent phosphors, visible light is emitted by the transparent phosphors. The excitation light64is provided by the light source62. Use of the substrate84and light emitting material90to display graphics is discussed in greater detail in U.S. application Ser. No. 17/749,464 titled “HYBRID AUGMENTED REALITY HEAD-UP DISPLAY FOR CREATING AN EDGE-TO-EDGE AUGMENTED REALITY VIEW” filed on May 20, 2022, the entire contents of which is hereby incorporated by reference.

The excitation light64may be ultraviolet light, in accordance with embodiments of the present disclosure. If the excitation light64is ultraviolet light, then when the light emitting material90emits visible light in response to the ultraviolet light, a down-conversion physical phenomenon occurs. Specifically, ultraviolet light has a shorter wavelength and higher energy than visible light. Accordingly, when the light emitting material90absorbs the ultraviolet light and emits lower energy visible light, the ultraviolet light is down-converted to visible light because the ultraviolet light's energy level goes down when it is converted into visible light.

It should be understood, other technology could also be used, such as a holographic grating (Bragg gratings) based technology which uses visible light projectors, wherein RGB LEDs are used to make a vector graphics projector and an RGB-DLP projector.

As shown inFIG.2andFIG.3, the excitation light64is output by the light source62of the DLP50. In an exemplary embodiment, the light source62is a micro-mirror array (MMA) projector (e.g. a digital light processing (DLP) projector50). In the DLP50, the appropriate augmentation graphics94are created by microscopically small mirrors laid out in a matrix on a semiconductor chip within the DMD72. An MMA projector that outputs ultraviolet light may be similar to a MMA projector that outputs visible light, except that the color filter68has light filters that are tailored to the ultraviolet light spectrum. In other embodiments, the DLP50is a liquid crystal display (LCD) projector. In embodiments, the DLP50may be a liquid crystal on silicon (LCOS) projector. In embodiments, the DLP50may be an analog projector (e.g. a slide film projector or a movie film projector). One of ordinary skill in the art would appreciate other types of projectors which may be used to project ultraviolet light on the substrate84.

In an exemplary embodiment, the windshield98is operable in a privacy mode, wherein the windshield98is opaque and the external environment54is not visible to the occupant80through the windshield98and the occupant80is not visible to persons within the external environment54. The controller34is adapted to determine if the windshield98is in a privacy mode, and, when the windshield98is in the privacy mode, deactivate the privacy mode prior to displaying appropriate augmentation graphics94on the inner surface96of the windshield98.

Additionally, after characterization of any identified person of interest56, the controller may activate or de-activate the privacy mode of the windshield98based on the characterization of the identified person or persons of interest56. For example, if the system11identifies and characterizes a person of interest56as friendly, and the windshield98is in the privacy mode, the controller will de-activate the privacy mode of the windshield98, allowing the occupant80to see the friendly person of interest56and allowing the friendly person of interest56to see the occupant80. Alternatively, if the system11identifies and characterizes a person of interest56as un-friendly, and the windshield98is in the privacy mode, the controller will keep the windshield in the privacy mode. If the windshield98is not in the privacy mode, the controller will activate the privacy mode of the windshield98, thus preventing the occupant80from seeing the un-friendly person of interest56and preventing the un-friendly person of interest56from seeing the occupant80.

In an exemplary embodiment, to achieve privacy mode, a layer of liquid crystals104is positioned between two layers of glass106of the windshield98. Referring toFIG.7A, the liquid crystals104respond to an electrical charge, provided through conducting rods108, by aligning parallel and letting light through, as indicated by arrows110. Referring toFIG.7B, when the electrical charge is absent, the liquid crystals104are randomly oriented, stopping much of the light from passing through the windshield98, as indicated by arrows112and making the windshield98opaque or translucent. With liquid crystals104, the glass is either clear or translucent, depending on the presence or absence of electrical current, and there is no intermediate setting.

Referring toFIG.8, a method200of identifying a person of interest56for an occupant80within a vehicle10with a head-up system11is shown. Starting at block202, an occupant80rides within the vehicle10. Moving to block204, if the occupant80does not initiate the system11, then, moving to block206, no action is taken, and the method200reverts back to block202.

If, at block204, the occupant80does initiate the system11, then, moving to block208, the method200includes identifying, with a controller34in communication with an external scene camera52adapted to capture images of an external environment54in proximity to the vehicle10, at least one person of interest56located within the external environment54in proximity to the vehicle10.

Moving to block210, if the system11does not identify any person of interest56, then, then no action is taken, and the system11reverts back to block208. If, at block210, the system11does detect at least one person of interest56, then, moving to block212, the method200includes characterizing, with the controller34, the at least one person of interest56. Moving to block214, the method200further includes determining, with the controller34, appropriate augmentation graphics94related to the identified at least one person of interest56to be displayed for the occupant80, and, moving to block216, displaying, with the head-up display system11, the appropriate augmentation graphics94onto an inner surface96of a windshield98of the vehicle10. Moving to block218, the system11checks to see if any identified persons of interest56are still present. If no person of interest56is still present, then the method200reverts back to block208, to identify, with the controller34any person of interest56. If at least one person of interest is still present, then the method200reverts to block216and continues displaying, with the head-up display system11, the appropriate augmentation graphics94onto the inner surface96of the windshield98of the vehicle10.

In an exemplary embodiment, after characterizing, with the controller34, the at least one person of interest56at block212, the method further includes, moving to block220, sending, via a communication network106within the vehicle10, information related to the characterization of the at least one person of interest56to a vehicle control module104of the vehicle10, and, moving to block222, initiating, with the vehicle control module104, vehicle maneuvers based on the characterization of the identified at least one person of interest56.

In another exemplary embodiment, the identifying, with a controller in communication with an external scene camera adapted to capture images of an external environment in proximity to the vehicle, at least one person of interest located within the external environment in proximity to the vehicle at block208further includes, moving to block224, scanning, with the external scene camera52, the external environment54in proximity to the vehicle10, and, moving to block226, identifying, with a computer vision algorithm58within the controller34, people located within the external environment54in proximity to the vehicle10.

In another exemplary embodiment, the identifying, with a computer vision algorithm58within the controller34, people located within the external environment54in proximity to the vehicle10at block226further includes, moving to block228, collecting, from remote sources48in communication with the controller34, data related to the identified people located within the external environment54in proximity to the vehicle10, and, moving to block230, identifying, with the controller34, at least one person of interest56within the identified people based on the data collected from the remote sources48. The collecting, from remote sources48in communication with the controller34, data related to the identified people located within the external environment54in proximity to the vehicle10at block228further includes collecting data related to the identified people located within the external environment54in proximity to the vehicle10from at least one of social media applications, criminal records, business networks, motor vehicle records, real estate records, voting records, and school records. Further, the identifying, with the controller34, at least one person of interest56within the identified people based on the data collected from the remote sources48at block230further includes identifying at least one person of interest56within the identified people who are at least one of: a follower of the occupant80on social media, followed by the occupant80on social media, a friend of the occupant80, a family member of the occupant80, a celebrity, a co-worker of the occupant80, a person who has similar interest as the occupant80, and a person who works in the same industry as the occupant80.

In an exemplary embodiment, the characterizing, with the controller34, the at least one person of interest56at block212further includes characterizing, with a cloud computing algorithm92in communication with the controller34, the at least one person of interest56as one of friendly or not-friendly based on the data related to the identified people collected from the remote sources48.

In another exemplary embodiment, the displaying, with the head-up display system11, the appropriate augmentation graphics94onto an inner surface96of a windshield98of the vehicle10at block216further includes, moving to block232, determining, with an occupant monitoring system78, a location of the occupant's80head and eyes, moving to block234, calculating a location to display the appropriate augmentation graphics94on the inner surface96of the windshield98based on the position of the head and eyes of the occupant80, wherein the appropriate augmentation graphics94appear in proximity to the identified at least one person of interest56when viewed by the occupant80, and, moving to block236, displaying the appropriate augmentation graphics94on the inner surface96of the windshield98by projecting, with a light source62of a digital light projector50, an excitation light64, focusing, with a condensing lens66, the excitation light64from the light source62, splitting, with a color filter68, the focused excitation light64into red, green and blue light, focusing, with a shaping lens70, the excitation light64passing through the color filter68, re-directing, with a digital micro-mirror device (DMD)72, the excitation light64, receiving, with a projection lens74, the excitation light64from the digital micro-mirror device72, and projecting, with the projection lens74, the excitation light64to the inner surface96of the windshield98at a location on the inner surface96of the windshield98based on the position of the head and eyes of the occupant80, wherein light emitting particles90within the windshield98emit visible light in response to absorbing the excitation light64, and the appropriate augmentation graphics94appear in proximity to the identified at least one person of interest56when viewed by the occupant80.

In an exemplary embodiment, the method200includes, moving to block238, determining, with the controller34, if the windshield98is in a privacy mode, wherein the windshield98is opaque and the external environment54is not visible to the occupant80through the windshield98and the occupant80is not visible to persons within the external environment54. If at block238, the windshield is not in the privacy mode, then the method200moves directly to block236, wherein the appropriate augmentation graphics94are displayed. If, at block238, the windshield is in privacy mode, then, moving to block240, the method200includes deactivating the privacy mode prior to displaying appropriate augmentation graphics94on the inner surface96of the windshield98.

A system and method of the present disclosure offers the advantage of providing information to an occupant within a vehicle of the presence and nature of persons of interest in proximity to the vehicle.