Virtual driver display for autonomous vehicle

Systems and methods for operating a vehicle are provided. In one example, an apparatus comprises a portion of a window of the vehicle having a configurable light transmittance, an image output device, and a controller configured to operate in a first mode when the vehicle is partially or fully controlled by a driver inside the vehicle and to operate in a second mode when the vehicle is not controlled by a driver inside the vehicle. In the first mode, the controller can adjust a light transmittance of the portion of the window to a first value to enable the driver inside the vehicle to see through the window. In the second mode, the controller can adjust the light transmittance of the portion of the window to a second value lower than the first value, and control the image output device to display an image on the portion of the window.

BACKGROUND

Autonomous driving technology has experienced rapid development in recent years. An autonomous vehicle may be able to operate without the control by a driver inside the vehicle. An autonomous vehicle may be configured to operate in different modes, such as an autonomous mode or in a manual mode. In an autonomous mode, the autonomous vehicle may be controlled by, for example, an automated control system, a remote driver and/or system, etc. In a manual mode, the autonomous vehicle may be controlled by a person inside the vehicle. The person may still sit on the driver seat when autonomous vehicle operates in the autonomous mode, but the person does not need to control the autonomous vehicle and does not need to focus on the road.

One commonly encountered situation that poses a difficult challenge to autonomous driving is lack of communication between an autonomous vehicle and other users of the road (e.g., other vehicles, pedestrians, etc.). A road user typically relies on the driver of an approaching vehicle to provide certain signals (e.g., a gesture, eye movement, other types of body language, verbal communication, etc.) to predict what the approaching vehicle will do. The road user can then take certain actions (e.g., cross the street or drive through an intersection in front of the approaching vehicle) based on the prediction. However, in a case where an autonomous vehicle operates in the autonomous mode, there may be either no human driver in the vehicle to provide the signal, or a person sitting on the driver seat who is not paying attention to the road and cannot provide the signal.

Therefore, to improve safety and to facilitate the safe usage of the road by other road users, there is a need to enable an autonomous vehicle to signal a future action of the vehicle to other road users in an effective and intuitive way.

SUMMARY

The present disclosure provides an apparatus that can be part of a vehicle. In one example, the apparatus includes a portion of a window of the vehicle having a configurable light transmittance, an image output device, and a controller configured to operate in a first mode when the vehicle is partially or fully controlled by a driver inside the vehicle and to operate in a second mode when the vehicle is not controlled by a driver inside the vehicle. In the first mode, the controller is configured to adjust a light transmittance of the portion of the window to a first value to enable the driver inside the vehicle to see through the window. In the second mode, the controller is configured to: adjust the light transmittance of the portion of the window to a second value lower than the first value, and control the image output device to display an image on the portion of the window while the transmittance of the portion of the window is at the second value, wherein the image is visible from outside the vehicle.

In some aspects, content of the image includes an object that resembles a driver.

In some aspects, the apparatus further comprises one or more sensors. The controller is configured to determine the content of the image based on data collected from the one or more sensors.

In some aspects, the controller is further configured to: determine, based on data from the one or more sensors, whether a pedestrian is on a curbside in front of a crosswalk within a predetermined distance from the vehicle; and determine the content of the image based on whether the pedestrian is on the curbside in front of the crosswalk.

In some aspects, the controller is configured to, based on determining that a pedestrian is on the curbside in front of the crosswalk, include a driver making a gesture in the content of the image to indicate to the pedestrian to use the crosswalk.

In some aspects, the apparatus further comprises an audio output device and an audio input device. The controller is configured to control the image output device, the audio output device and the audio input device to provide a two-way communication session between a remote person associated with the vehicle and a person outside of the vehicle.

In some aspects, the one or more sensors further comprise a light intensity sensor configured to detect an ambient light intensity. The controller is configured to decrease the light transmittance of the portion of the window based on the ambient light intensity exceeding a threshold.

In some aspects, the portion of the window of the vehicle also has a configurable light reflectivity controllable by the controller. The controller is configured to, in a third mode of operation: control the image output device to display the image based on an input from an occupant of the vehicle; set the light reflectivity of the portion of the window to reflect light of the displayed image towards the occupant inside the vehicle; and set the light transmittance of the portion of the window to substantially block the light of the image.

In some aspects, the image is displayed on the portion of the window by an image projection device.

The present disclosure further provides a method of operating a vehicle. In one example, the method comprises: in a first mode when the vehicle is partially or fully controlled by a driver inside the vehicle, adjusting a light transmittance of a portion of a window of the vehicle to a first value to enable the driver inside the vehicle to see through the window; and in a second mode when the vehicle is not controlled by a driver inside the vehicle: adjusting the light transmittance of the portion of the window to a second value lower than the first value, and controlling an image output device to display an image on the portion of the window while the transmittance of the portion of the window is at the second value such that the image becomes visible from outside the vehicle.

In one aspect, content of the image includes an object that resembles a driver.

In one aspect, the method further comprises: collecting data from one or more sensors; and determining the content of the image based on the data collected from the one or more sensors.

In one aspects, the method further comprises: determining, based on the data from the one or more sensors, whether a pedestrian is on a curbside in front of a crosswalk within a predetermined distance from the vehicle; and determining the content of the image based on whether the pedestrian is on the curbside in front of the crosswalk.

In one aspect, the method further comprises: based on determining that a pedestrian is on the curbside in front of the crosswalk, including a driver making a gesture in the content of the image to indicate to the pedestrian to use the crosswalk.

In one aspect, the data collected from the one or more sensors include a measurement of an ambient light intensity. The method further comprises decreasing the light transmittance of the portion of the window based on the ambient light intensity exceeding a threshold.

In one aspect, the method further comprises, in a third mode of operation: controlling the image output device to display the image based on an input from an occupant of the vehicle; setting a light reflectivity of the portion of the window to reflect light of the displayed image towards the occupant inside the vehicle; and setting the light transmittance of the portion of the window to substantially block the light of the image.

The present disclosure further provides a non-transitory computer readable medium storing instructions that, when executed by a processor of a vehicle, cause the processor to perform: in a first mode when the vehicle is partially or fully controlled by a driver inside the vehicle, adjusting a light transmittance of a portion of a window of the vehicle to a first value to enable the driver inside the vehicle to see through the window; and in a second mode when the vehicle is not controlled by a driver inside the vehicle: adjusting the light transmittance of the portion of the window to a second value lower than the first value, and controlling an image output device to display an image on the portion of the window while the transmittance of the portion of the window is at the second value such that the image becomes visible from outside the vehicle.

In one aspect, content of the image includes an object that resembles a driver.

In one aspect, the instructions, when executed by the processor of a vehicle, cause the processor to perform: determining, based on data from one or more sensors, whether a pedestrian is on a curbside in front of a crosswalk within a predetermined distance from the vehicle; and determining the content of the image based on whether the pedestrian is on the curbside in front of the crosswalk.

In one aspect, the instructions, when executed by the processor of the vehicle, cause the processor to perform, in a third mode of operation: controlling the image output device to display the image on the portion of the window based on an input from an occupant of the vehicle; setting the light reflectivity of the portion of the window to reflect light of the displayed image towards the occupant inside the vehicle; and setting the light transmittance of the portion of the window to substantially block the light of the image.

DETAILED DESCRIPTION OF THE INVENTION

The present disclosure relates to techniques for signaling a future action of an autonomous vehicle to other road users. The techniques can be implemented in a vehicle with a window and an image output device capable of outputting an image through at least a portion of the window. The window may include, for example, a windshield, a side window, a rear window, etc. The at least a portion of the window may have configurable light transmittance. When the vehicle operates in a non-autonomous mode (e.g., being controlled by a driver inside the vehicle), the light transmittance of the portion of the window can be set to a first value to enable the driver inside the vehicle to see through the window. When the vehicle operates in an autonomous mode (e.g., being controlled by an automated control system, by a remote driver and/or system, etc.), the light transmittance of the portion of the window can be set to a second value lower than the first value, and the image output device can be controlled to output an image using the portion of the window while the transmittance of the portion of the window is at the second value, such that the image becomes visible from outside the vehicle.

In some examples, the portion of the window may be in front of a driver seat (e.g., in a case where the window is the front windshield), and the image can include a virtual driver object resembling a driver (e.g., a realistic depiction of a human being, a cartoon figure, etc.) providing a gesture to signal a future action of the vehicle. The gesture included in the image can be determined based on, for example, a location of the vehicle, whether a foreign object (e.g., a pedestrian, another vehicle, etc.) is detected at a pre-determined distance from the vehicle, etc.

With the disclosed techniques, an autonomous vehicle can be configured to use the virtual driver to signal the vehicle's future action (e.g., to stop or to keep moving) to the other road user in a way similar to what a human driver would have done. Such arrangements can recreate a human interaction between the road user and the autonomous vehicle, and can ensure that the signaling can be readily understood by the road user without causing confusion. This not only improves safety, but also facilitates the usage of the road by other road users by providing them with information to predict the future action of the vehicle. Moreover, by providing a window with configurable light transmittance, the autonomous vehicle can also support a manual mode where a driver within the vehicle controls the vehicle and needs to have a clear view through the window. By allowing the autonomous vehicle to alternate between an autonomous mode and a manual mode, the operational flexibility of the vehicle can be improved as well.

The embodiments described in the present disclosure may be used in vehicles that offer various degrees of automated driving capabilities, ranging from partial driver assistance to full automation of the driving task. The National Highway Traffic Safety Administration (NHTSA) and the Society of Automotive Engineers (SAE) International define levels of vehicle autonomy as follows: Level 0, where the driver is in full control of the vehicle; Level 1, where a driver assistance system controls steering or acceleration/deceleration; Level 2, where the driver assistance system controls steering and acceleration/deceleration, and where the driver performs all other aspects of the driving task; Level 3, where all aspects of driving are performed by the driver assistance system, but where the driver may have to intervene if special circumstances occur that the automated vehicle is unable to safely handle such a situation; Level 4, where all aspects of driving are performed by the driver assistance system, even in situations where the driver does not appropriately respond when requested to intervene; and Level 5, where the vehicle drives fully autonomously in all driving situations, with or without a passenger. It should be noted that the term “autonomous vehicle” is sometimes used in the art to refer to any level of automation. However, in this document, “automated vehicle” is used to refer to level 1 through level 3 of automation when the vehicle is being driven in an automated mode, and the term “autonomous vehicle” is used to refer to levels 4 and 5 of the automation with little or no intervention of a human driver.

FIG. 1shows an example of an environment in which the disclosed techniques may be used. In the example ofFIG. 1, a vehicle100is approaching crosswalk102, while a pedestrian104is waiting to cross the street over crosswalk102. Pedestrian104may see vehicle100approaching crosswalk102, and may look at windshield portion106(which is in front of driver seat108) to look for any signaling from the driver, such as a signal to pedestrian104to cross the street. In the example ofFIG. 1, vehicle100may operate in an autonomous mode (e.g., SAE levels 4-5) and may not have a human sitting on the driver seat of the vehicle. Pedestrian104may not receive the signal he or she expects to receive in this situation (e.g., a waving gesture from a driver to signal that pedestrian104can cross the street), and pedestrian104may not be able to predict whether vehicle100is going to stop in front of crosswalk102. As a result, pedestrian104may decide to stay on the curbside not knowing that vehicle100will stop in front of crosswalk102. The lack of communication can add delay to the decision of pedestrian104to cross the street and can create inconvenience for pedestrian104. The lack of communication between pedestrian104and vehicle100also creates uncertainty. For example, pedestrian104may also take a risk and use crosswalk102while vehicle100is still moving towards him or her, thinking that vehicle100is going to stop. Meanwhile, vehicle100may fail to detect pedestrian104and is controlled to move across crosswalk102without stopping. Such uncertainly not only can have serious consequence for road safety but also can add psychological stress to pedestrian104and other road users alike.

FIGS. 2A and 2Billustrate examples of techniques that can be used to provide signaling of a vehicle's future action to other road users. In the example ofFIG. 2A, a vehicle200may include a windshield202with a windshield portion204. Windshield portion204may be in front of a driver seat (e.g., in front of a steering wheel) of vehicle200, or in a location of the windshield of vehicle200where another road user (e.g., pedestrian104) expects to see a driver. It should be noted that although in this figure, a portion of the windshield is used for displaying the image, the image can be displayed on any of the windows of the vehicle (e.g., side window on the driver side or passenger side, windshield, rear windows, etc.), without departing from teachings of the present disclosure. An image output device (not shown inFIGS. 2A and 2B) may output an image206through windshield portion204so that image206is externally visible to other road users (e.g., pedestrian104). In some examples, image206may include a virtual driver object that resembles a driver (e.g., a realistic depiction of a human being, a cartoon figure, etc.) and can be used to convey information about a future action of vehicle200.

Image206can be adapted based on an environment and/or various driving conditions in which vehicle200is situated. For example, inFIG. 2A, vehicle200may be operating on a road208unobstructed, and image206may be adapted to depict a driver driving vehicle200. Image206can be seen at windshield portion204by other drivers and can be used to convey, for example, that vehicle200will continue moving at its current speed and is not stopping imminently. In another example shown inFIG. 2B, vehicle200may approach crosswalk102, while pedestrian104is waiting to cross the street over crosswalk102. In this situation, an image216may be generated (e.g., based on image206) to depict the driver providing a hand waving gesture, which can be seen by pedestrian104. Based on the hand waving gesture, pedestrian104may understand that vehicle200will stop in front of crosswalk102to yield to pedestrian104. Pedestrian104may then cross the street using crosswalk102(while vehicle200is still moving towards crosswalk102) to expedite his/her movement.

FIG. 3AandFIG. 3Billustrate components of an example display apparatus300that can be used to provide signaling of a vehicle's future action to other road users, according to certain embodiments. Display apparatus300can be part of vehicle200ofFIG. 2AandFIG. 2B.FIG. 3Aillustrates a front (or back) view of display apparatus300. Display apparatus300may include a window302(which can be a windshield, a side window, a rear window, etc.), an image output device304, and a controller312. Window302may include a dimmable portion308coupled with electrodes310, which can be used to change the light transmittance of dimmable portion308. Image output device304may output image206through dimmable portion308so that image206can be visible to a viewer external to the vehicle. In some examples, dimmable portion308can extend across the entirety of window302, whereas image206can be output at the location of window302in front of, behind, or on a side of a passenger seat (e.g., on a portion of the windshield in front of a driver seat). Display apparatus300further includes a controller312to control the operations of image output device304and electrodes310. For example, controller312may include an image source314to provide data of image206to image output device304. Controller312may also include a dimming controller316to control electrodes310to change the light transmittance of dimmable portion308. As to be discussed below, controller312may receive data from one or more sensors (not shown inFIGS. 4A and 4B) and can control image source314to change the image data provided to image output device304based on the sensor data. Controller312may also control dimming controller316to change the light transmittance of dimmable portion308based on the sensor data.

FIG. 3Billustrates a side view of the example display apparatus300in a case where window302is part of the windshield of a vehicle. In the example shown inFIG. 3B, window302(e.g., dimmable portion308) and image output device304may form a rear projection system. Image output device304may be an image projector positioned within a compartment of a vehicle. For example, image output device304may be positioned on a console320and above a steering column322of a vehicle. Image output device304can project light324a-324cof an image (e.g., image206) towards window302and dimmable portion308. Dimmable portion308can allow some or all of light324a-324cto penetrate through. Dimmable portion308can also provide a diffusive medium (e.g., performing similar functions as a projection film) which can diffuse light324a-324cto enable an external viewer326to view the projected image from various angles and without distorting the projected image.

In some examples (not shown inFIG. 3B), window302(e.g., dimmable portion308) and image output device304may also form a front projection system. For example, image output device304can be positioned external to the compartment of the vehicle (e.g., next to the windshield wipers) and can project the image light onto dimmable portion308, which can provide diffuse reflection of the image light towards external viewer326.

Besides forming an image projection system, image output device304and window302can also be integrated together as a single piece of display device (not shown inFIG. 3B). For example, window302may include a set of transparent active display devices (e.g., transparent organ light emitting diodes (OLED)) which can be connected to transparent electrodes controlled by image source314. In this case, window302can be configured as a display device to generate image light, rather than passively transmitting or reflecting image light generated by another source. In general, the windshield and/or any of the windows of the vehicle (or any portion thereof) can be a display device with adjustable light transparency that can be adjusted based on the mode of operation of the vehicle.

In some examples, dimmable portion308may be formed by attaching a film made of electrochromic material on a surface of window302, or integrating the film within window302(e.g., by sandwiching the film between two glass layers to form window302). A variable electric field can be applied across dimmable portion308using electrodes310to change the light transmittance of the electrochromic film as well as dimmable portion308. One example of electrochromic material that can be used in display apparatus300is polymer dispersed liquid crystal. For example, when electrodes310supply no electric field (or a relatively weak electric field), the liquid crystals in the PDLC film can be positioned randomly, which can scatter and/or reflect the incident light, and reduce a quantity of light transmitted through dimmable portion308. The light transmittance of dimmable portion308can be reduced to the extent that dimmable portion308can become completely opaque. When dimmable portion308becomes completely opaque, it may also reflect light incident upon dimmable portion308. On the other hand, when electrodes310supply a relatively strong electric field, some or all of the liquid crystals in the PDLC can be aligned. The number of liquid crystals that are aligned (or oriented in a pre-determined direction) can control a quantity of light passing through the PDLC film, as well as the light transmittance of dimmable portion308. Other examples of electrochromic materials may also be used to create dimmable portion308such as, for example, micro-blinds, nanocrystals, etc.

In some examples, the film may also include active display devices (e.g., transparent OLED devices) which can be controlled to generate light. The active display devices may be used to generate light to block the view of a person via the windshield, which can change the perceived transparency of the windshield to achieve similar result as adjusting the light transmittance of the windshield.

In some examples, the light transmittance of dimmable portion308can be adjusted based on an ambient light intensity to improve the quality of the image as seen by an external viewer. One aspect of quality improvement can be in the contrast ratio, which defines the ratio of the luminance of the brightest color (white) to that of the darkest color (black) of image206. A high contrast ratio is desirable which allows different colors across the color spectrum between the white and black colors to become more easily discernable. When image206is output via a completely transparent windshield, the ambient light intensity of the environment in which the image is viewed may set a limit in the contrast ratio. For example, the ambient light intensity can set a lower bound of a luminance of the black level of image206, whereas the upper bound of a luminance of the white level of image206may be bounded by the output power of the image output device. If image206is viewed in an environment with very strong ambient light (e.g., under strong sunlight), the luminance of the black level may increase, which reduces the contrast ratio. On the other hand, by outputting image206via a windshield with reduced light transmittance, the luminance of the black level of image206may be reduced with respect to the luminance of the white level of image206, and the contrast ratio can be increased.

To improve the contrast ratio of image206as seen by an external viewer, controller312can reduce the light transmittance of dimmable portion308(e.g., by reducing the electric field applied by electrodes310) in an environment with strong ambient light (e.g., daytime with sunny weather). Controller312can also increase the light transmittance of dimmable portion308(e.g., by increasing the electric field applied by electrodes310) in an environment with weak ambient light (e.g., nighttime and/or daytime with cloudy weather). In some examples, vehicle200may include optical sensors mounted on the body of the vehicle to measure the ambient light intensity, and provide the light intensity measurement data to controller312, which can adjust the light transmittance of dimmable portion308based on the light intensity measurement data.

In some examples, the light transmittance of dimmable portion308can also be adjusted by controller312based on other inputs. For example, the light transmittance of dimmable portion308can be adjusted manually by a driver, or can be adjusted based on a sensor inside vehicle200(e.g., a proximity sensor, a weight sensor, a camera, a driving monitoring system etc.) detecting a person sitting on the driver seat. The adjustment may be to, for example, maximize the light transmittance of dimmable portion308when the vehicle is operated in a manual mode so that the driver can have a clear view of the road condition in front via window302. In some examples, the light transmittance of dimmable portion308can also be adjusted based on a proximity sensor inside vehicle200. In some examples, the light transmittance of dimmable portion308(as well as the entirety of window302) may also be set to minimum when the vehicle is parked and no passenger is in the vehicle. Such arrangements can prevent people outside the vehicle, including potential thieves, from looking through window302of a parked vehicle, to improve security and privacy).

FIGS. 4A and 4Billustrate examples of operations of display apparatus300.FIG. 4Aillustrates operations402and404. In operation402, vehicle200can be operated in a manual mode to be controlled by a driver inside vehicle200(e.g., SAE levels 1 and 2). To provide the driver with a clear view of the road condition, as well as pedestrian104, in front of vehicle200so that the driver can control vehicle200in a safe manner, window302(and dimmable portion308) can be controlled to have maximum light transmittance. Moreover, image output device304can be disabled and and/or controlled to not project any light onto window302, to avoid obstructing the view of the driver. In operation402, the light transmittance of dimmable portion308can be adjusted by controller312based on, for example, a manual input from the driver (e.g., to configure the vehicle to operate in a manual mode), detection of a person sitting on the driver seat by a proximity sensor or a weight sensor, etc.

In operation404, vehicle200can be operated in an autonomous mode where the vehicle is controlled by an automated control system (e.g., SAE levels 4-5) and/or controlled by a remote driver/system. With vehicle200in the autonomous mode, the driver (or any other passenger within vehicle200) does not need to control the vehicle, and does not need to have a clear view of the road condition (and pedestrian104) in front of vehicle200. In operation404, the light transmittance of dimmable portion308(or the entirety of window302) can be reduced, and image output device304can be enabled to project image206onto window302such that image206can be externally visible to pedestrian104. In operation404, the light transmittance of dimmable portion308can be adjusted by controller312based on, for example, a manual input from the driver (e.g., to configure the vehicle to operate in the autonomous mode), ambient light measurement data provided by external optical sensors, etc.

FIG. 4Billustrates operations406and408. In operation406, the light transmittance of dimmable portion308can be further reduced with respect to operation404. For example, dimmable portion308can be configured to become completely opaque. In this operation, image output device304can be controlled to output an image407to be viewed by a driver and/or other passengers within vehicle200. Image407may include, for example, a video, a software application interface, etc. Due to the complete opacity of dimmable portion308, image407may not be externally visible to other people outside of vehicle200, and may be reflected towards the interior of the vehicle. In operation406, the light transmittance of dimmable portion308can be adjusted by controller312based on, for example, a manual input from the driver and/or passengers to convert dimmable portion308into an internal image projection screen. This mode can be used, for example, when a passenger uses a parked vehicle as an office or when the vehicle is being driven in fully autonomous mode.

In some examples, display apparatus300may include two image output devices and two dimmable portions. Each image output device and each dimmable portion can form an independent display unit. One display unit can be used for displaying images to be externally visible to people outside of vehicle200, whereas the other display unit can be used for displaying images to the driver and/or passengers inside vehicle200. Operation408illustrates an example of operating two dimmable portions308aand308bwith two image output devices310aand310b. Dimmable portion308aand image output device310amay be located on the driver side of window302to output image206, whereas dimmable portion308band image output device310bcan be on the passenger side of window302to output image407. Dimmable portion308acan have higher light transmittance than dimmable portion308bsuch that image206can be externally visible whereas image407is mostly visible internally for the passenger (or other people behind window302). In operation408, dimmable portion308aand image output device310amay be configured as a rear projection device, whereas dimmable portion308band image output device310bcan be configured as a front projection device. In some embodiments, the dimmable portion308aand308bmay overlap and the images206and407may be projected on overlapping portions of the windshield (e.g., one from a projection device inside the vehicle and one from a projection device outside the vehicle).

As discussed above, the image output by image output device310acan be updated based on data collected by one or more sensors. The data collected by the sensors can provide information about an environment in which vehicle200is located to the controller. As an example, the one or more sensors may include camera(s), ultrasonic sensor(s), RADAR(s), LiDAR(s), and any other types of sensors without departing from the teachings of the present disclosure. The image is output by image output device310ato be adapted to reflect the instantaneous environment. Given that the automated control system that controls vehicle200is also likely to determine a future action based on the instantaneous environment, the adaptation of the image according to the instantaneous environment can enable other road users to better predict the future action of the vehicle, and safety can be improved.

FIGS. 5A and 5Billustrate various measurements related to an environment for output image determination. The environment inFIG. 5Amay be similar to the environment depicted inFIG. 1, in which vehicle200is approaching crosswalk102, while pedestrian104is waiting to cross the street over crosswalk102, and controller312may control image output device310ato output an image that depicts a driver providing a hand waving gesture to signal pedestrian104to use the crosswalk.

To determine to output an image that depicts a driver providing a hand waving gesture to signal pedestrian104to use the crosswalk, controller312may determine that vehicle200is approaching crosswalk102. The determination can be based on sensing a location of vehicle200and determining, based on the location information and a map of crosswalks, that vehicle200is close to crosswalk102. The sensing of the location of vehicle200can be based on signals provided by satellite502(e.g., Global Positioning System (GPS) signals), by cell towers504(e.g., Positioning Reference Signal (PRS)), etc. By comparing the location of the vehicle and known locations of crosswalks from the map, controller312may determine a distance between vehicle200and crosswalk102. In another example, the controller may process images received from one or more cameras to determine if the vehicle is approaching a crosswalk.

In addition, controller312may also determine that a physical object (e.g., a pedestrian) is on one end of crosswalk102. The determination can be based on a combination of data from, for example, radar sensor506, camera508and LiDAR514. For example, radar sensor506can transmit radio waves and monitor for reflected radio waves, based on which radar sensor506(or controller312) can detect a physical object which reflects the radio waves. Based on a timing difference between the transmitted radio waves and the reflected radio waves, controller312can also determine a distance between vehicle200and the physical object. Based on the distance as well as the locations of vehicle200and crosswalk102, controller312may determine that an object is located on one end of crosswalk102. Moreover, controller312may also use camera508to capture an image of the object and determine, from the image, that the object is a pedestrian. Controller312may perform image analysis to identify, for example, a human head from the image, an outline of a human body from the image, or other suitable image features associated with a human. Based on the image features, controller312may determine that a pedestrian (e.g., pedestrian104) may be standing on one end of crosswalk102waiting to cross the street, and the distance between vehicle200and pedestrian104.

Moreover, upon determining that the distance between vehicle200and pedestrian104drops to within a pre-determined threshold distance (e.g., a typical human visual range for an image output by image output device304), and that vehicle200is approaching crosswalk102, controller312may then control image source314to provide image216of a driver providing a hand waving gesture to signal pedestrian104to cross the street over crosswalk102.

In one example, controller312may also output, via a speaker510, an audio signal to pedestrian104, prior to (or concurrent with) providing image216to image output device304for displaying on window302. Controller312may determine that the pedestrian likely does not see vehicle200, and may output audio signal to pedestrian104to draw his or her attention to image216. The determination can be based on, for example, determining a gaze direction of pedestrian104from the image data provided by camera508and comparing the gaze direction (and/or an orientation of the head) of pedestrian104with a direction window302is facing. Based on determining that pedestrian104is likely not looking at vehicle200(and likely not looking at window302), controller312may output audio signal via a speaker510when the distance between vehicle200and pedestrian104is within the pre-determined threshold distance (or within another threshold distance based on a typical human audio range).

In one example, display apparatus300can also be used to provide two-way communication between pedestrian104(or other human beings, such as law enforcement personnel) and a remote driver controlling vehicle200. For example, via satellite502and/or cell towers504, a remote operator (e.g., a remote driver or a remote person responsible for a fleet of autonomous vehicles) can transmit his or her voice and image data, in real-time, to controller312, which can then output the received image and voice data to, respectively, image output device304and speaker510. Camera508, as well as an external microphone512, may capture image and voice data from pedestrian104(e.g., a law enforcement officer), which controller312can transmit back to the remote operator via satellite502and/or cell towers504. In one example cameras and microphone can be located on the side of the vehicle to enable communication with pedestrian located on the side of the vehicle. The two-way communication enables exchange of information (between pedestrian104and the remote operator of vehicle200, e.g., real-time information about the environment provided by pedestrian104who is physically in the environment, the remote driver's planned operation of the vehicle, etc.), which can improve the operation of the vehicle by the remote driver and enhance safety and user experience. In another example, the display apparatus can be used for two-way communication between law-enforcement personnel and a remote person associated with the vehicle. For example, the law-enforcement officer may perform a traffic stop on the vehicle (e.g., an autonomous vehicle) and need some information (e.g., vehicle registration, etc.) from the remote operator. The remote operator may send the requested information through the communication link to the vehicle to be displayed on the window of the vehicle to be seen by the law-enforcement personnel.

In addition to providing a signal to pedestrians, the disclosed techniques can also be used to provide a signal to a human driver at an intersection.FIG. 5Billustrates an environment in which the output image is determined based on detection of another vehicle in an intersection. In the environment ofFIG. 5B, vehicle200may be approaching an intersection530. To determine what image to display on window302, controller312may first determine that vehicle200is approaching intersection530(e.g., based on the location of vehicle200provided by satellite502and/or cell towers504and a map of intersections and their known locations, or detected by RADAR, LiDAR, Cameras, etc.), and then determine whether vehicle200has the right-of-way when it crosses intersection530. If controller312determines that vehicle200has the right-of-way, controller312may provide image206(e.g., a driver driving without providing a gesture) to image output device304.

On the other hand, if controller312detects that another vehicle540is approaching intersection530(e.g., based on data from radar sensor506), and that vehicle540has the right-of-way, controller312may provide image216(e.g., a driver providing a hand waving gesture) to image output device304for displaying on window302, to signal to the other driver in vehicle540to cross intersection530. Controller312may provide image216to image output device304when, for example, the distance between vehicle200and stop line542of intersection530(e.g., where vehicle200is supposed to stop) is within the aforementioned pre-determined threshold distance (e.g., a typical human visual range for an image output by image output device304).

There are various ways by which controller312can determine that vehicle200(or vehicle540) has the right-of-way. For example, controller312may determine that vehicle200is not approaching a stop sign, which indicates that vehicle200has the right-of-way. Controller312may also determine that vehicle200has the right-of-way based on a traffic light (e.g., green), or other traffic signs. Moreover, in the case where the traffic signs and/or traffic lights indicate that vehicle200may need to stop, controller312may also determine whether vehicle540will have the right-of-way when vehicle200reaches intersection530. The determination can be based on, for example, the traffic signs/lights facing vehicle540(which can be determined from a map of traffic lights and signs), the speed of vehicle540, etc. As an illustrative example, if both vehicles200and540are approaching a stop sign, and that vehicle540is likely to reach the stop sign earlier than vehicle200(or within a pre-determined window before or after vehicle200reaches the stop sign), controller312may determine that vehicle540has the right-of-way, and provide image216with a driver providing a hand waving gesture to image output device304to signal to the driver of vehicle540to cross intersection530.

In some examples, controller312may also control image output device304to output the virtual driver image (e.g., a driver driving, a driver providing a gesture, etc.) only at times and locations where other road users are expected to pay attention to the virtual driver image. For example, controller312may control image output device304to output the virtual driver image only when vehicle200is at a location where communication with the other road user is typical (e.g., at a crosswalk, at an intersection, etc.), and can disable image output device304when vehicle200is at other locations (e.g., on a highway). Such arrangements can be adopted to, for example, reduce unnecessary distraction of other drivers and/or pedestrians, to reduce power consumption, etc.

FIG. 6illustrates additional components of controller312and their connections with various sensors and/or other input devices. As shown inFIG. 6, dimming controller316is coupled with a manual control module602, light intensity sensors604, and driver sensors606. Driver sensors606may include, for example, a proximity sensor, a weight sensor and/or a camera to sense whether a driver sits on the driver seat. These sensors enable the light transmittance of different portions of window302(e.g., dimmable portions308aand308b) to be set either manually (from manual control module602), based on ambient light intensity (based on light intensity data provided by light intensity sensors604, which can be part of camera508ofFIG. 5B), and/or based on whether a person sits on the driver seat (based on measurements provided by driver sensors606).

Moreover, controller312further includes a vehicle detection module608to detect a vehicle and its distance from vehicle200. Vehicle detection module608may perform the detection based on measurements provided by perception sensors610, which may include, for example, radar sensors (e.g., radar sensors506), image sensors (e.g., camera508), LiDAR, etc. Controller312further includes a pedestrian detection module612which may perform the detection also based on measurements provided by perception sensors610. For example, based on measurements provided by radar sensors506, pedestrian detection module612may determine a distance between vehicle200and a pedestrian. Moreover, based on image data provided by camera508, pedestrian detection module612may determine that an object in the image is a pedestrian. Pedestrian detection module612may also determine a gaze direction of the detected pedestrian, and control audio source614to output audio signals if the gaze direction indicates that the pedestrian is not looking at vehicle200. Both vehicle detection module608and pedestrian detection module612can provide the detection results to image source314.

In addition, image source314may also obtain, from location sensors616, a location of vehicle200. By combining the location information, map information (e.g., a map of intersections and crosswalks), as well as the detection results from vehicle detection module608and pedestrian detection module612, image source314may identify the conditions described inFIGS. 5A and 5Band determine what image to provide to image output device304for displaying on window302, whereas dimming controller316can adjust the light transmittance of window302based on ambient light intensity to improve the display quality of the image.

FIGS. 7A-7Cillustrate a method700for operating a display device of a vehicle (e.g., image output device304and dimmable portion308of vehicle200). In some examples, method700can be performed by, for example, controller312ofFIG. 3AandFIG. 6.

Referring toFIG. 7A, at702, controller312may determine a mode of operation of the display device. The determination can be based on, for example, whether vehicle200is in an autonomous mode where the vehicle is controlled by an automatic control system (e.g., SAE levels 4-5) and/or by a remote driver/system, or vehicle200is in an manual or automated mode (e.g., SAE levels 1-3) where a driver within the vehicle (an internal driver) is controlling the vehicle or may need to take control of the vehicle in short notice. If the vehicle is controlled by an internal driver (at704), controller312may operate the display device in a first mode, at706. If the vehicle is in the autonomous mode (at704), controller312may operate the display device in a second mode, at708.

FIG. 7Billustrates an operation of the display device in the first mode. At710, controller312may maximize the light transmittance of the windshield (or set it to a first value that exceeds a reference) by, for example, configuring electrodes310to output a strong electric field. At712, controller312may disable image output device304, to provide the internal driver with an unobstructed view through dimmable portion308of window302.

FIG. 7Cillustrates an operation of the display device in the second mode. At720, controller312may determine a location of vehicle200based on, for example, signals received by location sensors616(e.g., GPS signals, PRS signals, etc.). Based on the location of vehicle200, camera information as well as map information, controller312may determine whether vehicle200is at an intersection (at722). If the vehicle is at an intersection, controller312may further determine whether vehicle200has the right-of-way based on the techniques disclosed above, at724. If vehicle200does not have the right-of-way (at724), controller312may generate an image including a driver showing a gesture (e.g., a hand waving gesture to signal to the other party to move), at726.

Referring back to722, if controller312determines that vehicle200is not at an intersection, controller312may determine whether vehicle200is at a crosswalk, at730. If vehicle200is at a crosswalk, controller312may determine whether a pedestrian is detected waiting at the crosswalk (e.g., based on data from radar sensors and cameras), at732. If a waiting pedestrian is detected (at732), controller312may proceed to generate an image including a driver showing a gesture at726.

Referring back to724,730, and732, if controller312determines that vehicle200has the right-of-way (at724), not at a crosswalk (at730), or that no waiting pedestrian is detected (at732), or after the image including the driver showing a gesture is generated (726), controller312may set the light transmittance of at least a portion of a window of vehicle200based on ambient light intensity to improve a display quality of the image, at734. Controller312can then provide the image to image output device304, which enables image output device304to output the image on dimmable portion308of window302, at736.

Some portions of this description describe the embodiments of the disclosure in terms of algorithms and operations. These operations are understood to be implemented by computer programs or equivalent electrical circuits, machine code, or the like. Furthermore, it has also proven convenient, at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, and/or hardware.

Steps, operations, or processes described may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. Although the steps, operations, or processes are described in sequence, it will be understood that in some embodiments the sequence order may differ from that which has been described, for example with certain steps, operations, or processes being omitted or performed in parallel or concurrently.

In some embodiments, a software module is implemented with a computer program product comprising a non-transitory computer-readable storage medium containing computer program code, which can be executed by one or more computer processors for performing any or all of the steps, operations, or processes described. Examples of a non-transitory storage medium include, but are not limited to, a magnetic disk or tape, optical storage media such as compact disk (CD) or digital versatile disk (DVD), flash memory, or other memory devices.