Multi-plane augmented reality image generation

A set of training images are input into a neural network that outputs a pixel-wise phase matrix identifying pixels in the training images. Based on the pixel-wise phase matrix, a spatial light modulator (SLM) is actuated to output, onto a vehicle windshield, an augmented reality (AR) image including a plurality of sub-images each output in one of a plurality of focal planes. Each training image corresponds to one respective sub-image. A feedback image of the AR image is obtained via an image sensor. An offset is determined based on comparing the training images to the feedback image. Parameters of a loss function are updated based on the offset, and the updated parameters are provided to the neural network to obtain an updated offset.

BACKGROUND

Vehicles typically include various displays to provide users with a variety of information. For example, vehicles can include instrument panels to provide a vehicle operator with data about a vehicle's speed, fuel status, engine temperature, etc. A vehicle can include a touchscreen display or the like to provide a variety of information and accept user input, e.g., for a climate control system, an audio system, etc. Yet further, a vehicle can include a heads-up display or the like to display information to be viewed by a vehicle operator or other user in combination with the user's view of a vehicle windshield and/or a roadway.

DETAILED DESCRIPTION

A vehicle can include a heads-up display (HUD) that can display content such as information about the vehicle and/or objects around the vehicle to an occupant of the vehicle. The HUD can project images onto a windshield of the vehicle. The HUD can provide content as an augmented reality (AR) image that includes a plurality of sub-images. The HUD can provide the AR image such that the sub-images, when viewed by the occupant, are overlayed with the objects around the vehicle. Thus, the HUD can display images in a manner to allow the occupant to view the images while also viewing a roadway along which the vehicle is traveling. However, projecting each sub-image into a single focal plane, e.g., on the windshield, may result in misalignment and low quality between one or more sub-images and the respective object to which the sub-image is being overlayed. Such misalignment and low quality may result in an undesirable aesthetic appearance and unclarity of the AR image for the occupant.

Advantageously, a neural network can be trained to accept a set of training images and to generate a pixel-wise phase matrix identifying pixels in the training images. Each training image corresponds to one respective sub-image of the AR image. A spatial light modulator (SLM) is actuated to output the AR image based on the pixel-wise phase matrix. That is, the SLM outputs each sub-image into one of a plurality of focal planes. For example, one sub-image can be output to appear on the windshield, and another sub-image can be output to appear to be exterior and forward of the windshield. An offset is determined between the training images and a feedback image including the AR image. The offset is used to update parameters of a loss function of the neural network. The updated parameters are used to train the neural network to output an updated pixel-wise phase matrix. Techniques disclosed herein improve displaying content via a HUD by providing the plurality of sub-images in respective focal planes, which can improve the alignment and quality of the sub-images with the corresponding objects around the vehicle thereby providing a desirable aesthetic appearance with high quality of the AR image to the occupant. Further, the techniques disclosed herein can train the neural network to optimize the pixel-wise phase matrix based on the windshield and the SLM.

The present disclosure includes system, comprising: a spatial light modulator (SLM) arranged to output, onto a vehicle windshield, an augmented reality (AR) image including a plurality of sub-images each output in one of a plurality of focal planes; an image sensor positioned to obtain a feedback image of the AR image; and a computer including a processor and a memory, the memory storing instructions executable by the processor programmed to: input a set of training images into a neural network that outputs a pixel-wise phase matrix identifying pixels in the training images, wherein each training image corresponds to one respective sub-image; actuate the SLM to output the AR image based on the pixel-wise phase matrix; determine an offset based on comparing the training images to the feedback image; and update parameters of a loss function based on the offset and provide the updated parameters to the neural network to obtain an updated offset.

The instructions can further include instructions to update parameters of the loss function until the updated offset is less than a predetermined threshold. The neural network can be trained to output a pixel-wise phase matrix calibrated to the vehicle windshield and the SLM when the updated offset is less than the predetermined threshold.

The image sensor can be spaced from the vehicle windshield to correspond to an expected pose of an occupant. The instructions can further include instructions to select the focal planes for the sub-images based on a user input. Each focal plane can be defined by a respective distance from an expected pose of an occupant. The instructions can further include instructions to, while training the neural network, actuate a projector to provide the training images to the SLM.

The instructions can further include instructions to, while training the neural network, select the training images from a plurality of training images. The system of claim1, the instructions can further include instructions to: after training the neural network, generate a three-dimensional (3D) image based on vehicle operation data; and determine, via the trained neural network, a calibrated pixel-wise phase matrix for the 3D image. The instructions can further include instructions to, upon generating a set of masked images from the 3D image, input the masked images to the trained neural network that outputs the calibrated pixel-wise phase matrix for the 3D image. The system of claim10, the instructions can further include instructions to generate the masked images by masking a plurality of pixels in the 3D image based on the focal planes, each masked pixel being a pixel of the 3D image having a depth corresponding to one of the focal planes. The system of claim9, the instructions can further include instructions to actuate the SLM to output the AR image based on the calibrated pixel-wise phase matrix. The instructions can further include instructions to actuate a projector to provide the 3D image to the SLM.

Further according to this disclosure, a method can comprise: inputting a set of training images into a neural network that outputs a pixel-wise phase matrix identifying pixels in the training images; based on the pixel-wise phase matrix, actuating a spatial light modulator (SLM) to outputting, onto a vehicle windshield, an augmented reality (AR) image including a plurality of sub-images each output in one of a plurality of focal planes, wherein each training image corresponds to one respective sub-image; obtaining a feedback image of the AR image via an image sensor; determining an offset based on comparing the training images to the feedback image; and updating parameters of a loss function based on the offset and providing the updated parameters to the neural network to obtain an updated offset.

The method can further comprise updating parameters of the loss function until the updated offset is less than a predetermined threshold. The neural network can be trained to output a pixel-wise phase matrix calibrated to the vehicle windshield and the SLM when the updated offset is less than the predetermined threshold. The image sensor can be spaced from the vehicle windshield to correspond to an expected pose of an occupant. The focal planes for the sub-images can be selected based on a user input. Each focal plane can be selected based on a respective distance from an expected pose of an occupant. The can further comprise, while training the neural network, actuating a projector to provide the training image to the SLM.

With reference toFIGS.1-6, an example control system100includes a vehicle105, a remote computer140, and an image sensor145. The vehicle105includes a spatial light modulator (SLM)150. A vehicle computer110in the vehicle105receives data from sensors115. The SLM150is arranged to output, onto a vehicle windshield310, an augmented reality (AR) image300including a plurality of sub-images305each output in one of a plurality of focal planes P1, P2, P3. The image sensor145is positioned to obtain a feedback image of the AR image300. The vehicle computer110is programmed to actuate the SLM150using a trained neural network400, as discussed below.

To train the neural network400to determine sub-images305for the focal planes P1, P2, P3, the remote computer140is programmed to input a set of training images402into the neural network400that outputs a pixel-wise phase matrix410identifying pixels in the training images402. Each training image402corresponds to one respective sub-image305. The remote computer140is further programmed to actuate the SLM150to output the AR image300based on the pixel-wise phase matrix410. The remote computer140is further programmed to determine a pixel-wise offset between the training images402and the feedback image500. The remote computer140is further programmed to update parameters of a loss function based on the offset and provide the updated parameters to the neural network400to obtain an updated offset.

Turning now toFIG.1, the vehicle105includes the vehicle computer110, sensors115, actuators120to actuate various vehicle components125, and a vehicle105communication module130. The communication module130allows the vehicle computer110to communicate with remote computers140, and/or other vehicles, e.g., via a messaging or broadcast protocol such as Dedicated Short Range Communications (DSRC), cellular, and/or other protocol that can support vehicle-to-vehicle, vehicle-to infrastructure, vehicle-to-cloud communications, or the like, and/or via a packet network135.

The vehicle computer110includes a processor and a memory such as are known. The memory includes one or more forms of computer-readable media, and stores instructions executable by the vehicle computer110for performing various operations, including as disclosed herein. The vehicle computer110can further include two or more computing devices operating in concert to carry out vehicle105operations including as described herein. Further, the vehicle computer110can be a generic computer with a processor and memory as described above and/or may include a dedicated electronic circuit including an ASIC that is manufactured for a particular operation, e.g., an ASIC for processing sensor115data and/or communicating the sensor115data. In another example, the vehicle computer110may include an FPGA (Field-Programmable Gate Array) which is an integrated circuit manufactured to be configurable by a user. Typically, a hardware description language such as VHDL (Very High Speed Integrated Circuit Hardware Description Language) is used in electronic design automation to describe digital and mixed-signal systems such as FPGA and ASIC. For example, an ASIC is manufactured based on VHDL programming provided pre-manufacturing, whereas logical components inside an FPGA may be configured based on VHDL programming, e.g., stored in a memory electrically connected to the FPGA circuit. In some examples, a combination of processor(s), ASIC(s), and/or FPGA circuits may be included in the vehicle computer110.

The vehicle computer110may operate and/or monitor the vehicle105in an autonomous mode, a semi-autonomous mode, or a non-autonomous (or manual) mode, i.e., can control and/or monitor operation of the vehicle105, including controlling and/or monitoring components125. For purposes of this disclosure, an autonomous mode is defined as one in which each of vehicle105propulsion, braking, and steering are controlled by the vehicle computer110; in a semi-autonomous mode the vehicle computer110controls one or two of vehicle105propulsion, braking, and steering; in a non-autonomous mode a human operator controls each of vehicle105propulsion, braking, and steering.

The vehicle computer110may include programming to operate one or more of vehicle105brakes, propulsion (e.g., control of acceleration in the vehicle105by controlling one or more of an internal combustion engine, electric motor, hybrid engine, etc.), steering, transmission, climate control, interior and/or exterior lights, horn, doors, etc., as well as to determine whether and when the vehicle computer110, as opposed to a human operator, is to control such operations.

The vehicle computer110may include or be communicatively coupled to, e.g., via a vehicle communication network such as a communications bus as described further below, more than one processor, e.g., included in electronic controller units (ECUs) or the like included in the vehicle105for monitoring and/or controlling various vehicle components125, e.g., a transmission controller, a brake controller, a steering controller, etc. The vehicle computer110is generally arranged for communications on a vehicle communication network that can include a bus in the vehicle105such as a controller area network (CAN) or the like, and/or other wired and/or wireless mechanisms.

Via the vehicle105network, the vehicle computer110may transmit messages to various devices in the vehicle105and/or receive messages (e.g., CAN messages) from the various devices, e.g., sensors115, actuators120, ECUs, etc. Alternatively, or additionally, in cases where the vehicle computer110actually comprises a plurality of devices, the vehicle communication network may be used for communications between devices represented as the vehicle computer110in this disclosure. Further, as mentioned below, various controllers and/or sensors115may provide data to the vehicle computer110via the vehicle communication network.

Vehicle105sensors115may include a variety of devices such as are known to provide data to the vehicle computer110. For example, the sensors115may include Light Detection And Ranging (LIDAR) sensor115(s), etc., disposed on a top of the vehicle105, behind a vehicle105front windshield310, around the vehicle105, etc., that provide relative locations, sizes, and shapes of objects surrounding the vehicle105. As another example, one or more radar sensors115fixed to vehicle105bumpers may provide data to provide locations of the objects, second vehicles, etc., relative to the location of the vehicle105. The sensors115may further alternatively or additionally, for example, include camera sensor(s)115, e.g., front view, side view, etc., providing images from an area surrounding the vehicle105. As another example, the vehicle105can include one or more sensors115, e.g., camera sensors115, mounted inside a cabin of the vehicle105and oriented to capture images of users in the vehicle105cabin. In the context of this disclosure, an object is a physical, i.e., material, item that has mass and that can be represented by physical phenomena (e.g., light or other electromagnetic waves, or sound, etc.) detectable by sensors115. Thus, the vehicle105, as well as other items including as discussed below, fall within the definition of “object” herein.

The vehicle computer110is programmed to receive data from one or more sensors115, e.g., substantially continuously, periodically, and/or when instructed by a remote computer140, etc. The data may, for example, include a location of the vehicle105. Location data specifies a point or points on a ground surface and may be in a known form, e.g., geo-coordinates such as latitude and longitude coordinates obtained via a navigation system, as is known, that uses the Global Positioning System (GPS) and/or dead reckoning. Additionally, or alternatively, the data can include a location of an object, e.g., a vehicle105, a sign, a tree, etc., relative to the vehicle105. As one example, the data may be image data of the environment around the vehicle105. In such an example, the image data may include one or more objects and/or markings, e.g., lane markings, on or along a road. As another example, the data may be image data of the vehicle105cabin, e.g., including users and seats in the vehicle105cabin. Image data herein means digital image data, i.e., comprising pixels, typically with intensity and color values, that can be acquired by camera sensors115. The sensors115can be mounted to any suitable location in or on the vehicle105, e.g., on a vehicle105bumper, on a vehicle105roof, etc., to collect images of the environment around the vehicle105.

The vehicle105actuators120are implemented via circuits, chips, or other electronic and or mechanical components that can actuate various vehicle105subsystems in accordance with appropriate control signals as is known. The actuators120may be used to control components125, including braking, acceleration, and steering of a vehicle105.

In the context of the present disclosure, a vehicle component125is one or more hardware components adapted to perform a mechanical or electro-mechanical function or operation—such as moving the vehicle105, slowing or stopping the vehicle105, steering the vehicle105, etc. Non-limiting examples of components125include a propulsion component (that includes, e.g., an internal combustion engine and/or an electric motor, etc.), a transmission component, a steering component (e.g., that may include one or more of a steering wheel, a steering rack, etc.), a suspension component (e.g., that may include one or more of a damper, e.g., a shock or a strut, a bushing, a spring, a control arm, a ball joint, a linkage, etc.), a brake component, a park assist component, an adaptive cruise control component, an adaptive steering component, one or more passive restraint systems (e.g., airbags), a movable seat, etc.

The vehicle105further includes a human-machine interface (HMI)118. The HMI118includes user input devices such as knobs, buttons, switches, pedals, levers, touchscreens, and/or microphones, etc. The input devices may include sensors115to detect a user input and provide user input data to the vehicle computer110. That is, the vehicle computer110may be programmed to receive user input from the HMI118. The user may provide the user input via the HMI118, e.g., by selecting a virtual button on a touchscreen display, by providing voice commands, etc. For example, a touchscreen display included in an HMI118may include sensors115to detect that a user selected a virtual button on the touchscreen display to, e.g., select or deselect an operation, which input can be received in the vehicle computer110and used to determine the selection of the user input.

The HMI118typically further includes output devices such as displays (including touchscreen displays), speakers, and/or lights, etc., that output signals or data to the user. The HMI118is coupled to the vehicle communication network and can send and/or receive messages to/from the vehicle computer110and other vehicle sub-systems.

The vehicle105further includes the SLM150. An “SLM” is an object that imposes spatially varying modulation on a beam of light. The SLM150is arranged to receive light from a projector155and to modulate the light according to a pixel-wise phase matrix to output the light onto the windshield310to provide and AR image300that can appear to be exterior to the vehicle105. The SLM150is coupled to the vehicle communication network and can send and/or receive messages to/from the vehicle computer110and other vehicle sub-systems.

The vehicle105further includes a projector155. The projector155can be arranged to display images in a field of view of an occupant of the vehicle105. The projector155can be arranged to display images vehicle-forward of the occupant to provide information about vehicle surroundings, vehicle operations, etc. For example, the projector155can project light onto the windshield310. Specifically, the projector155can project light through the SLM150to the windshield310. The light is reflected by the windshield310to provide the AR image300in the light of sight of the occupant so as to be viewable by and understood by the occupant. Although the AR image300is projected onto the windshield310, the AR image300appear to the occupant to be exterior to the vehicle105to provide an augmented reality display of surroundings of the vehicle105. Specifically, the plurality of sub-images305of the AR image300appear to be in respective focal planes P1, P2, P3forward of the vehicle105, as discussed below. The projector155is coupled to the vehicle communication network and can send and/or receive messages to/from the vehicle computer110and other vehicle sub-systems.

In addition, the vehicle computer110may be configured for communicating via a vehicle-to-vehicle communication module or interface with devices outside of the vehicle105, e.g., through a vehicle-to-vehicle (V2V) or vehicle-to-infrastructure (V2X) wireless communications (cellular and/or DSRC, etc.) to another vehicle, and/or to a remote computer140(typically via direct radio frequency communications). The communication module could include one or more mechanisms, such as a transceiver, by which the computers of vehicles may communicate, including any desired combination of wireless (e.g., cellular, wireless, satellite, microwave and radio frequency) communication mechanisms and any desired network topology (or topologies when a plurality of communication mechanisms are utilized). Exemplary communications provided via the communications module include cellular, Bluetooth, IEEE 802.11, Ultra-Wideband (UWB), Near Field Communication (NFC), dedicated short range communications (DSRC), and/or wide area networks (WAN), including the Internet, providing data communication services.

The network135represents one or more mechanisms by which a vehicle computer110may communicate with remote computing devices, e.g., the remote computer140, another vehicle computer, etc. Accordingly, the network135can be one or more of various wired or wireless communication mechanisms, including any desired combination of wired (e.g., cable and fiber) and/or wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any desired network topology (or topologies when multiple communication mechanisms are utilized). Exemplary communication networks135include wireless communication networks (e.g., using Bluetooth®, Bluetooth® Low Energy (BLE), UWB, NFC, IEEE 802.11, vehicle-to-vehicle (V2V) such as Dedicated Short Range Communications (DSRC), etc.), local area networks (LAN) and/or wide area networks (WAN), including the Internet, providing data communication services.

The remote computer140can be a conventional computing device, i.e., including one or more processors and one or more memories, programmed to provide operations such as disclosed herein. Further, the remote computer140can be accessed via the network135, e.g., the Internet, a cellular network, and/or some other wide area network.

During operation, the vehicle computer110can receive vehicle operation data of one or more components125from one or more sensors115. In this context, “vehicle operation data” are data describing operation of vehicle components, i.e., operation data are data measuring various component attributes as the vehicle operates on a road. The operation data can include, e.g., speed data, acceleration data, braking data, steering angle data, etc. That is, as vehicles operate, the vehicle operation data provide measurements describing how the vehicles operate.

Additionally, the vehicle computer110can receive environment data of the environment around the vehicle105from one or more sensors115. In this context, “environment data” are data describing objects around the vehicle105as the vehicle105operates on the road. The vehicle computer110can identify the detected object160based on the environment data. For example, object identification techniques can be used, e.g., in the vehicle computer110based on LIDAR sensor115data, camera sensor115data, etc., to identify a type of object160, e.g., a user, an animal, a vehicle, etc., as well as physical features of objects205.

Any suitable techniques may be used to interpret sensor115data. For example, camera and/or LIDAR image data can be provided to a classifier that comprises programming to utilize one or more conventional image classification techniques. For example, the classifier can use a machine learning technique in which data known to represent various objects, is provided to a machine learning program for training the classifier. Once trained, the classifier can accept as input vehicle sensor115data, e.g., an image, and then provide as output, for each of one or more respective regions of interest in the image, an identification of a user or an indication that no user is present in the respective region of interest. Further, a coordinate system (e.g., polar or cartesian) applied to an area proximate to the vehicle105can be applied to specify locations and/or areas (e.g., according to the vehicle105coordinate system, translated to global latitude and longitude geo-coordinates, etc.) of a user identified from sensor115data. Yet further, the vehicle computer110could employ various techniques for fusing (i.e., incorporating into a common coordinate system or frame of reference) data from different sensors115and/or types of sensors115, e.g., LIDAR, radar, and/or optical camera data.

The vehicle computer110can determine a first distance between a detected object160and the vehicle105based on sensor115data. For example, a sensor115, e.g., a radar sensor115, mounted to the vehicle105can measure an amount of time elapsed from emitting a radio wave to receiving the radio wave reflected from the object160. Based on the time elapsed and a speed of light, the vehicle computer110can determine the first distance between the object160and the vehicle105.

Additionally, the vehicle computer110can determine navigation data for the vehicle105. In this context, “navigation data” are data describing a planned path of the vehicle105, i.e., navigation data are data measuring various features of the planned path as the vehicle105operates along the planned path. For example, the navigation data can include a second distance between the vehicle105and a location of a specified vehicle operation for maintaining operation of the vehicle along a planned path. As one example, the specified vehicle operation can be turning the vehicle105from a current road to, e.g., a new road, a parking lot, etc. As another example, the specified vehicle operation can be to change a lane of operation so as to permit the vehicle105to depart the current road, e.g., via a lane permitting turning and/or merging maneuvers. As yet another example, the specified vehicle operation can be to maintain the vehicle105on the current road, e.g., via a lane change operation.

The vehicle computer110can determine the second distance between the vehicle105and the location of the specified vehicle operation, e.g., by comparing respective geo-coordinates. For example, the vehicle computer110may receive a location of the vehicle105, e.g., from a sensor115, a navigation system, a remote computer140, etc. The vehicle computer110can determine the geo-coordinates for the location of the specified vehicle operation based on the planned path. As one example, the planned path may specify the geo-coordinates for the location of the specified vehicle operation. As another example, the vehicle computer110may overlay the planned path onto map data, e.g., received from the remote computer140. In such an example, the map data may specify geo-coordinates for various locations. The vehicle computer110can determine the geo-coordinates for the location of the specified vehicle operation based on selecting a location on the map corresponding to the specified vehicle operation of the planned path.

The vehicle computer110can generate the planned path, e.g., to avoid detected objects, to reach a destination specified by a user input, etc. As used herein, a “path” is a set of points, e.g., that can be specified as coordinates with respect to a vehicle coordinate system and/or geo-coordinates, that the vehicle computer110is programmed to determine with a conventional navigation and/or path planning algorithm. A path can be specified according to one or more path polynomials. A path polynomial is a polynomial function of degree three or less that describes the motion of a vehicle on a ground surface. Motion of a vehicle on a roadway is described by a multi-dimensional state vector that includes vehicle location, orientation, speed, and acceleration. Specifically, the vehicle motion vector can include positions in x, y, z, yaw, pitch, roll, yaw rate, pitch rate, roll rate, heading velocity and heading acceleration that can be determined by fitting a polynomial function to successive 2D locations included in the vehicle motion vector with respect to the ground surface, for example.

Further for example, the path polynomial p(x) is a model that predicts the path as a line traced by a polynomial equation. The path polynomial p(x) predicts the path for a predetermined upcoming distance x, by determining a lateral coordinate p, e.g., measured in meters:
p(x)=a0+a1x+a2x2+a3x3(1)
where a0an offset, i.e., a lateral distance between the path and a center line of the vehicle105at the upcoming distance x, a1is a heading angle of the path, a2is the curvature of the path, and a3is the curvature rate of the path.

The vehicle computer110can generate a three-dimensional (3D) image based on the vehicle operation data, the environment data, including the first distance, and the navigation data, including the second distance. For example, the vehicle computer110can maintain a look-up table, or the like, that associates various graphics with corresponding vehicle operation data, the environment data, and the navigation data. Upon determining the vehicle operation data, the environment data, and the navigation data, the vehicle computer110can access the look-up table and determine corresponding graphics. For example, the vehicle computer110can select graphics associated with stored vehicle operation data, stored environment data, and stored navigation data that corresponds to, i.e., substantially matches, the determined vehicle operation data, the determined environment data, and the determined navigation data. The vehicle computer110can then input the graphics, the vehicle operation data, the environment data, and the navigation data into an image generator (e.g., a neural network, such as an encoder-decoder neural network, a generative adversarial network, etc.) that outputs the 3D image, e.g., according to known image generation techniques. The 3D image includes the selected graphics each displayed in one of a plurality of focal planes P1, P2, P3. The image generator determines the focal plane P1, P2, P3for each graphic based on the vehicle operation data, the environment data, and the navigation data.

Each focal plane P1, P2, P3is a plane positioned a specified distance D1, D2, D3forward of an expected pose of an occupant in which a sub-image305of the AR image300is projected (seeFIG.3). That is, the sub-image305will appear to the occupant to be in front of the vehicle105by the specified distance D1, D2, D3corresponding to the focal plane P1, P2, P3. A sub-image305is a portion, i.e., less than all, of the AR image300. The vehicle computer110can actuate the SLM150to output an AR image300including a plurality of non-overlapping sub-images305, as discussed below. The distances can be determined or selected based on empirical testing or simulation, for example, to determine desired distances of the focal planes P1, P2, P3from the perspective of the occupant.

The vehicle computer110can determine to provide one or more of the plurality of focal planes P1, P2, P3as output to the HMI118based on a user input selecting the focal plane(s). An occupant may wish to enable or disable a focal plane based of various factors, e.g., relevance of or interest in its content at a current time or during a current or planned vehicle operation. For example, the vehicle computer110can actuate and/or instruct the HMI118to display virtual buttons corresponding to various focal planes P1, P2, P3that the user can select to specify the focal planes P1, P2, P3. In other words, the HMI118may activate sensors that can detect the first user selecting virtual buttons to specify the focal planes P1, P2, P3. Upon detecting the user input, the HMI118can provide the user input to the vehicle computer110, and the vehicle computer110can determine the focal planes P1, P2, P3for output or display based on the user input. For example, the user input may specify three focal planes P1, P2, P3at three specified distances D1, D2, D3. (seeFIG.2). In this example, a first focal plane P1may be positioned forward of the expected pose of the occupant by a first distance D1. The first distance D1may be specified such that the sub-image305provided in the first focal plane P1appears to be on the windshield310. Additionally, a second focal plane P2may be positioned forward of the expected pose of the occupant by a second distance D2. The second distance D2may be specified such that the second distance D2is farther away from the expected pose of the occupant than the first distance D1, i.e., the sub-image305provided in the second focal plane P2may appear to be exterior to the windshield310. Additionally, a third focal plane P3may be positioned forward of the expected pose of the occupant by a third distance D3. The third distance D3may be specified such that the third distance D3is farther away from the expected pose of the occupant than the second distance D2, i.e., the sub-image305provided in the third focal plane P3may appear to be exterior to the windshield310and farther than the sub-image305provided in the second focal plane P2.

The expected pose of an occupant (from which distances D1, D2, D3are measured) can, for example, be determined empirically, e.g., based on testing that allows for determining an average (or some other statistical measure) height and average (or some other statistical measure) vehicle seat position (e.g., specified according to the vehicle coordinate system) for various occupants. In this situation, the expected pose can be stored, e.g., in a memory of the vehicle computer110.

As another example, the vehicle computer110can determine the expected pose based on receiving a second user input specifying the expected pose of the occupant. For example, the vehicle computer110can actuate and/or instruct the HMI118to display virtual buttons corresponding to various expected poses that the user can select to specify the expected pose, as just discussed above regarding the first user input. In such an example, the vehicle computer110can receive a plurality of estimated poses, e.g., from a remote computer140. The plurality of expected poses may be determined empirically, e.g., based on physically measuring occupants having various heights in specified seat positions.

As yet another example, the vehicle computer110can selected an expected pose that is closest to an actual pose of the occupant. In such an example, the vehicle computer110can receive the plurality of estimated poses, e.g., from a remote computer140. The vehicle computer110can then determine the actual pose of the occupant and compare the actual pose to each of the expected poses. Upon determining differences between each expected pose and the actual pose, the vehicle computer110can select the expected pose corresponding to a minimum difference.

The vehicle computer110can determine the actual pose for the occupant based on sensor115data. For example, the vehicle computer110can obtain an image from an image sensor115positioned to face the occupant when the occupant is seated inside the vehicle105. The vehicle computer110can then input the image to a machine learning program that identifies keypoints. The machine learning program can be a conventional neural network trained for processing images, e.g., OpenPose, Google Research and Machine Intelligence (G-RMI), DL-61, etc. For example, OpenPose receives, as input, an image and identifies keypoints in the image corresponding to human body parts, e.g., hands, feet, joints, etc. OpenPose inputs the image to a plurality of convolutional layers that, based on training with a reference dataset such as Alpha-Pose, identify keypoints in the image and output the keypoints. The keypoints include depth data that the image alone does not include, and the vehicle computer110can use a machine learning program such as OpenPose to determine the depth data to identify the actual pose of the occupant in the image. That is, the machine learning program outputs the keypoints as a set of three values: a length along a first axis of a 2D coordinate system in the image, a width along a second axis of the 2D coordinate system in the image, and a depth from the image sensor115to the vehicle occupant, the depth typically being a distance along a third axis normal to a plane defined by the first and second axes of the image. The vehicle computer110can then connect the keypoints, e.g., using data processing techniques, to determine the actual pose of the occupant.

Upon generating the 3D image, the vehicle computer110can generate a set of two-dimensional (2D) depth masked images based on the 3D image. To generate a 2D depth masked image, the vehicle computer110can mask pixels of the 3D image based on one of the focal planes P1, P2, P3. In this context, to “depth mask” pixels, the vehicle computer110performs a pixel wise filtering operation to hide pixels based on depths of the pixels. For example, the 3D image may include a set of 3D coordinates for each pixel in the 3D image, e.g., with respect to a vehicle coordinate system. The vehicle computer110can identify pixels in the 3D image that have a depth outside of a predetermined range associated with the one focal plane P1, P2, P3and can mask the identified pixels. That is, the vehicle computer110can generate a 2D depth masked image with values of 0 for pixels identified to be masked for the one focal plane P1, P2, P3. The vehicle computer110can continue to generate 2D depth masked images in this manner until one 2D depth masked image is generated for each focal plane P1, P2, P3. The predetermined range may be determined based on a distance between the one focal plane P1, P2, P3and the occupant (or nearest intermediate focal plane P1, P2, P3between the occupant and the one focal plane P1, P2, P3).

The vehicle computer110can determine a pixel-wise phase matrix for the 3D image by inputting the set of 2D depth masked images into a neural network, such as a deep neural network (DNN)400(seeFIG.4). The DNN400can be trained (as discussed below) to accept the set of 2D depth masked images as input and generate an output of a calibrated pixel-wise phase matrix for the 3D image. A calibrated pixel-wise phase matrix is a matrix identifying pixels in the 3D image and specifying a pixel phase for each pixel. A “pixel phase” is an adjustment to a point in time that a sample is taken in an analog-digital conversion. A pixel phase allows for synchronizing pixel (or dot) clocks of the vehicle computer110and a projector. A “pixel clock” is a speed at which pixels are transmitted such that a full frame of pixels fits within one refresh cycle. Unsynchronized pixel clocks can result in pixel banding, i.e., multiple pixels end at the same pixel coordinates, which reduces the resolution of the AR image300. The pixel-wise phase matrix for the 3D image is calibrated based on the physical properties of the windshield310, the SLM, and the projector. That is, the calibrated pixel-wise phase matrix allows the vehicle computer110to account for distortion, lack of focus, or low quality of the AR image300resulting from the physical properties of the windshield310, the SLM150, and/or the projector155, as discussed below.

Upon determining the calibrated pixel-wise phase matrix, the vehicle computer110actuates the projector155to provide the 3D image to the SLM150. Providing the 3D image to the SLM150allows the vehicle computer110to output an AR image300onto the windshield310. (seeFIG.3). The AR image300as shown inFIG.3provides an example view as would be seen by the occupant of the vehicle105, e.g., a driver. The AR image300may indicate to the occupant the presence of objects in view of the AR image300. In the example shown inFIG.3, the AR image300indicates an animal on the left side of the roadway and a second vehicle ahead of the vehicle105. The objects may be indicated by lights, colors, or verbiage within the AR image300. Additionally, the AR image300may indicate vehicle measurements and information about the operation of the vehicle105, e.g., speedometer, odometer, tachometer, fuel status, turn-by-turn navigation instructions, etc.

Specifically, the vehicle computer110actuates the SLM150based on the calibrated pixel-wise phase matrix to output the AR image300including sub-images305into respective focal planes P1, P2, P3. That is, the SLM150receives the 3D image as input and spatially modulates the 3D image according to the calibrated pixel-wise phase matrix to output the AR image300. The AR image300, and specifically, the sub-images305at the respective focal planes P1, P2, P3, is provided in the line of sight of the occupant so as to be viewable by and understood by the occupant. Each sub-image305may include, for example, information about vehicle operation and/or objects in the environment around the vehicle105. As discussed above, providing the sub-images305of the AR image300in respective focal planes P1, P2, P3can improve the quality of the AR image300.

The windshield310may include a photopolymer film arranged to receive the AR image300on the windshield310. The light from the projection155may be reflected by photopolymer film making the AR image300visible to occupant. The photopolymer film may be any suitable material such that the light from the projector115may be reflected by the photopolymer film. For example, the photopolymer film may be RGB light sensitive for recording volume holograms. The photopolymer film may, for example, be between layers of polyvinyl butyral (“PVB”) and glass of the windshield310.

FIG.4is a diagram of a deep neural network (DNN)400. The DNN400can be a software program executing on the remote computer140. Once trained, the DNN400can be downloaded to the vehicle computer110. The vehicle computer110can use the DNN400to generate an AR image300having a plurality of sub-images305output in respective focal planes P1, P2, P3. Advantageously, the trained DNN400allows for a computer110or the like to output higher quality images at desired depths, e.g., at respective focal planes P1, P2, P3. Feedback images captured by a sensor focusing at different focal planes during training are typically of lower than desired quality, e.g., blurry, out-of-focus, noisy, containing artifacts, etc., caused by the nonlinearity of SLM150, varying windshield geometry, and/or optical aberrations of the windshield and other optical elements in the system. The DNN400can be trained to increase the quality of such images.

The DNN400can include a plurality of convolutional layers (CONV)404that process input images (IN)402by convolving the input images402using convolution kernels to determine latent variables (LV)406. The DNN400includes a plurality of fully-connected layers (FC)408that process the latent variables406to produce a pixel-wise phase matrix (PM)410. The DNN400can input a set of 2D depth masked images402, as discussed above, to determine a pixel-wise phase matrix410. The pixel-wise phase matrix410is a matrix identifying pixels in the training images402and specifying a pixel phase for each pixel.

The DNN400is trained by processing a dataset that includes a plurality of training images402. Prior to training the DNN400, the focal planes P1, P2, P3are defined as described above, e.g., based on desired distances D1, D2, D3from the perspective of an occupant. (Note that the examples herein discuss three focal planes, but in other examples there could be other numbers of focal planes.) Each of the training images402may be depth masked based on one of the focal planes P1, P2, P3. The training images402may be stored, e.g., in a memory of the remote computer140. To train the DNN400, the remote computer140selects a set of training images402from the plurality of training images402. The remote computer140can select one training image402for each respective focal plane P1, P2, P3. That is, each training image402in the selected set may be depth masked based on one respective focal plane P1, P2, P3. The remote computer140inputs the set training images402into the DNN400that outputs a pixel-wise phase matrix410for the selected set of training images402.

Upon determining the pixel-wise phase matrix410for the selected set of training images402, the remote computer140actuates the projector155to provide the selected set of training images402to the SLM150, as discussed above. For example, the remote computer140can provide, e.g., via the network135, instructions to the projector155to provide the selected set of training images402. The remote computer140then actuates the SLM150based on the pixel-wise phase matrix410to output an AR image300of the selected set of training images402. For example, the remote computer140can provide, e.g., via the network135, instructions to the SLM150to actuate based on the pixel-wise phase matrix410. That is, the SLM150outputs the selected set of training images402into respective focal planes P1, P2, P3. The selected set of training images402is reflected by the windshield310to provide the AR image300of the selected set of training images402, as discussed above.

Upon detecting the AR image300of the selected set of training images402on the windshield310, the remote computer140can determine an offset between the selected set of training images402and a feedback image500at corresponding focal planes. An offset specifies an intensity difference between features and/or pixels in the selected set of training images402and the feedback image500. An example of an offset can be seen by comparingFIGS.5A and5B.FIG.5Ashows an expected feedback image502, andFIG.5Bshows an actual feedback image500. As seen inFIG.5B, the actual feedback image500locates objects at different locations, i.e., locations of pixels are offset, than in the expected feedback image500.

The offset may result from various factors. As one example, the windshield310may cause the offset. For example, due to manufacturing tolerances and/or being repaired, windshields in various vehicles may have different physical properties (e.g., a contour, transparency, composition, alignment relative to a windshield opening of the vehicle and/or an instrument panel, etc.). As another example, features of various projectors and SLMs may be different relative to each other based on manufacturing tolerances and/or being repaired. Determining the offset between the selected set of training images402and the feedback image500allows the DNN400to be trained to determine a pixel-wise phase matrix that minimizes the offset for the various factors in the vehicle105that contribute to the offset, i.e., is calibrated to the vehicle105.

To determine the offset, the remote computer140can generate an expected feedback image502(seeFIG.5A) based on the selected set of training images402. For example, the remote computer140can overlay the selected set of training images402based on the masked pixels of the respective images402. That is, the remote computer140can align the selected set of training images402such that the masked pixels of some of the training images402are hidden by corresponding unmasked pixels of other training images402. The remote computer140can then compare the expected feedback image502to the feedback image500, e.g., using known image differencing techniques. For example, the remote computer140can determine differences between corresponding pixel values between the expected feedback image502and the feedback image500and can generate an image based on the determined differences in pixel values.

As another example, the remote computer140can determine distances (in pixel coordinates) between corresponding features in the expected feedback image502and the feedback image500, e.g., using known image processing techniques. As one example, the remote computer140can determine the offset based on an average distance between corresponding features, e.g., by using a mean square error (MSE). In this situation, the offset is determined from the average distance. As another example, the remote computer140can use an intersection over union (IoU) to determine a ratio of an area of intersection between corresponding features in the expected feedback image502and the feedback image500. In this situation, the offset is determined from the ration.

The remote computer140can detect the AR image300via a feedback image500, e.g., using known image recognition techniques, as discussed above. A feedback image500is obtained from the image sensor145. The image sensor145can be separate from the vehicle105. For example, the image sensor145can be mounted in the vehicle105while training the DNN400and removed after training is complete. That is, the image sensor145can be remote from the vehicle105during operation of the vehicle105by an occupant.

The image sensor145can be deployed in a fixed or stationary manner, e.g., mounted to a fixture, mounted to a vehicle seat, etc. The image sensor145may be positioned in the vehicle105based on an estimated pose of an occupant (as discussed above). For example, as shown inFIG.6, the image sensor145may be positioned adjacent to a headrest of a vehicle seat. That is, the image sensor145may be positioned to have a field of view that substantially corresponds to a field of view of an occupant in the vehicle seat. Specifically, the field of view of the image sensor145includes the windshield310. That is, the image sensor145is positioned to view the windshield310from substantially the same perspective as an occupant in the vehicle seat.

The remote computer140can update parameters of a loss function based on the offset. Back-propagation can compute a loss function based on the expected feedback image502and the feedback image500. A loss function is a mathematical function that maps values such as the expected feedback image502and the feedback image500into real numbers that can be compared to determine a cost during training. In this example, the cost is the offset. The loss function determines how closely the expected feedback image502matches the feedback image500and is used to adjust the parameters or weights that control the DNN.

Parameters or weights include coefficients used by linear and/or non-linear equations included in fully-connected layers. Fully-connected layers process the latent variables output by other hidden layers. Upon determining the offset, the remote computer140can update the parameters of the loss function. For example, the remote computer140can systematically vary these parameters or weights and compare the output results to a desired result minimizing the loss function. As a result of varying the parameters or weights over a plurality of trials over a plurality of input images, a set of parameters or weights that achieve a result that minimizes the loss function can be determined. As another example, the remote computer140can optimize parameters of the loss function by applying gradient descent to the loss function. Gradient descent calculates a gradient of the loss function with respect to the current parameters. The gradient indicates a direction and magnitude to move along the loss function to determine a new set of parameters. That is, the remote computer140can determine a new set of parameters based on the gradient and the loss function. Applying gradient descent reduces an amount of time for training by using the loss function to identify specific adjustments to the parameters as opposed to selecting new parameters at random.

The remote computer140can then provide the updated parameters to the DNN400. The remote computer140can then determine an updated offset based on the selected set of training images402and the updated DNN400. For example, the remote computer140can input the selected set of training images402to the updated DNN400that can output an updated pixel-wise phase matrix410for the selected set of training images402. The remote computer140can then actuate the SLM150based on the updated pixel-wise phase matrix410to output an updated AR image300, as discussed above. The remote computer140can then obtain an updated feedback image500from the image sensor145, as discussed above. The remote computer140can then determine an updated offset based on the expected feedback image502and the updated feedback image500, e.g., in substantially the same manner as discussed above with respect to determining the offset.

The remote computer140can subsequently determine updated parameters, e.g., in substantially the same manner as discussed above with respect to updating the parameters of the loss function, until the updated offset is less than a predetermined threshold. That is, parameters controlling the DNN400processing are varied until output AR images300match, within the predetermined threshold, the input selected set of training images402for each of the plurality of training images402in the dataset. The predetermined threshold may be determined based on, e.g., empirical testing to determine a maximum offset at which a vehicle computer110can actuate the SLM150to output an AR image300achieving desired quality. Upon determining the offset, the remote computer140can compare the offset to the predetermined threshold. The predetermined threshold may be stored, e.g., in a memory of the remote computer140. When the updated offset is less than the predetermined threshold, the DNN400is trained to accept a set of 2D depth masked images402generated from a 3D image as input and to generate a calibrated pixel-wise phase matrix for the 3D image.

FIG.7is a diagram of an example process700executed in a remote computer140according to program instructions stored in a memory thereof for training a neural network400to accept input images402and to generate a pixel-wise phase matrix410for the input images402. Process700includes multiple blocks that can be executed in the illustrated order. Process700could alternatively or additionally include fewer blocks or can include the blocks executed in different orders.

Process700begins in a block705. In the block705, the remote computer140selects a set of training images402from a plurality of training images402, as discussed above. Each training image402in the selected set corresponds to one of a plurality of focal planes P1, P2, P3, as discussed above. The process700continues in a block710.

In the block710, the remote computer140determines a pixel-wise phase matrix410for the selected set of training images402. For example, the remote computer140can input the selected set of training images402into a DNN400that outputs the pixel-wise phase matrix410, as discussed above. The process700continues in a block715.

In the block715, the remote computer140actuates an SLM150in a vehicle105to output an AR image300onto a vehicle windshield310, as discussed above. Specifically, the remote computer140actuates the SLM150based on the pixel-wise phase matrix410determined in the block610. Actuating the SLM150based on the pixel-wise phase matrix410allows the SLM150to output a plurality of sub-images305of the AR image300into respective focal planes P1, P2, P3, as discussed above. The process700continues in a block720.

In the block720, the remote computer140determines an offset between an expected feedback image502and a feedback image500. The remote computer140receives the feedback image500from an image sensor145, as discussed above. The remote computer140determines the expected feedback image502based on the selected set of training images402, as discussed above. The remote computer140can determine the offset according to image differencing techniques, as discussed above. The process700continues in a block725.

In the block725, the remote computer140determines whether the offset is less than a predetermined threshold. The remote computer140can compare the offset to the predetermined threshold. If the offset is less than the predetermined threshold, then the process700continues in a block735. Otherwise, the process700continues in a block730.

In the block730, the remote computer140updates parameters of a loss function based on the offset, as discussed above. The remote computer140can then provide the updated parameters to the DNN400. The process700returns to the block710.

In the block735, the remote computer140determines that the DNN400is trained to output a calibrated pixel-wise phase matrix. The remote computer140can then provide the DNN400, e.g., data describing the DNN400, to a vehicle computer110, e.g., via the network135. The process700ends following the block735.

FIG.8is a diagram of an example process800executed in a vehicle computer110according to program instructions stored in a memory thereof for outputting an AR image300onto a vehicle windshield310. Process800includes multiple blocks that can be executed in the illustrated order. Process800could alternatively or additionally include fewer blocks or can include the blocks executed in different orders.

Process800begins in a block805. In the block805, the vehicle computer110receives data from one or more sensors115, e.g., via a vehicle network, and/or from the remote computer140, e.g., via the network135. For example, the vehicle computer110can receive vehicle operation data, environment data, and navigation data, as discussed above. Additionally, the vehicle computer110can determine a first distance between the vehicle105and an object160and a second distance between the vehicle105and a location of a specified vehicle operation based on the received data, as discussed above. The process800continues in a block810.

In the block810, the vehicle computer110generates a 3D image based on the received data, the first distance, and the second distance, e.g., using an image generator, as discussed above. The process800continues in a block815.

In the block815, the vehicle computer110generates a set of 2D depth masked images from the 3D image, as discussed above. Each 2D depth masked image in the set is depth masked based on one respective focal plane P1, P2, P3, as discussed above. The process800continues in a block820.

In the block820, the vehicle computer110determines a calibrated pixel-wise phase matrix for the 3D image using the trained DNN400. For example, the vehicle computer110can input the set of 2D depth masked images into the trained DNN400. The DNN400can the output the calibrated pixel-wise phase matrix for the 3D image. The process800continues in a block825.

In the block825, the vehicle computer110actuates the SLM150to output an AR image300onto the vehicle windshield310. Specifically, the vehicle computer110actuates the SLM150based on the calibrated pixel-wise phase matrix. Actuating the SLM150based on the calibrated pixel-wise phase matrix allows the SLM150to output a plurality of sub-images305of the AR image300into respective focal planes P1, P2, P3, as discussed above. The process800ends following the block825.

Memory may include a computer-readable medium (also referred to as a processor-readable medium) that includes any non-transitory (e.g., tangible) medium that participates in providing data (e.g., instructions) that may be read by a computer (e.g., by a processor of a computer). Such a medium may take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media may include, for example, optical or magnetic disks and other persistent memory. Volatile media may include, for example, dynamic random-access memory (DRAM), which typically constitutes a main memory. Such instructions may be transmitted by one or more transmission media, including coaxial cables, copper wire and fiber optics, including the wires that comprise a system bus coupled to a processor of an ECU. Common forms of computer-readable media include, for example, RAM, a PROM, an EPROM, a FLASH-EEPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

With regard to the media, processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes may be practiced with the described steps performed in an order other than the order described herein. It further should be understood that certain steps may be performed simultaneously, that other steps may be added, or that certain steps described herein may be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments and should in no way be construed so as to limit the claims.