Head up display apparatus and display control method thereof

In relation to a HUD apparatus having an AR function, the shift between an object and a virtual image (AR image) is reduced. The HUD apparatus extracts a predetermined object based on an image taken by an external camera, acquires object information including an object position in space, viewpoint position information of a driver and the amount of movement of the viewpoint position in space based on an image taken by an internal camera, and information including a position of a virtual image region in space. The HUD apparatus generates the image to be superimposed on the object, and corrects a display position of the image in the virtual image region including at least a position in a horizontal direction, and performs conversion processing at the time of the correction so that a virtual image region is set as a display position of the image after correction.

TECHNICAL FIELD

The present invention relates to a technology of a display apparatus such as a head up display (HUD) apparatus, and relates to a technology for providing image information of augmented reality (AR) to a driver.

BACKGROUND ART

As an AR display apparatus having an AR function and a system thereof, there is a HUD apparatus mounted on an automobile or the like. In the HUD apparatus (sometimes referred to as an AR-HUD), a virtual image is displayed to be superimposed on a transmitted real image by projecting image light to a visible region of a windshield, a combiner, or the like in a field of view of a user such as a driver. The visible region is a region where the user can visually recognize the image. Hereinafter, a virtual image may be described as an AR image. The virtual image includes, for example, image information for driving assistance and the like relative to a real image including a road, a car, and the like in front of an own vehicle. Specific examples of the virtual image include an image of vehicle information such as a vehicle speed, an arrow image for navigation of a traveling direction, and an icon image for alerting or highlighting the approach of a car, a person, a bicycle, or the like. Further, as the original data for generating the virtual image, information of an in-vehicle sensor, an engine control unit, a car navigation system, and the like can be used.

The HUD apparatus includes, for example, a display element, a light source, and an optical system. The optical system includes a lens, a mirror, and the like for guiding image light to the visible region. The image light from the display element is projected through the mirror and the like to a predetermined region within the visible region of a windshield, a combiner, or the like. The light reflected in the predetermined region enters the eyes of the driver and forms an image on the retina, and is recognized as a virtual image by the driver. A virtual image region which is a region where the virtual image can be seen is formed in front of the viewpoint of the driver through the visible region so as to correspond to the predetermined region.

Japanese Patent Application Laid-Open Publication No. 2010-70066 (Patent Document 1) is an example of a related art regarding an in-vehicle HUD apparatus. Patent Document 1 describes an invention for a head up display, in which visibility of displayed information is improved by reducing a projection position of an image from being shifted from a line of sight of one eye due to the vibration of a vehicle or the change in the posture of a driver.

RELATED ART DOCUMENTS

Patent Documents

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

In a conventional in-vehicle system including a HUD apparatus, a virtual image including an AR image is displayed to be superimposed on a real image of an object in a virtual image region through a visible region of a windshield or the like by using an AR function. At that time, depending on the position of the driver's eyes (may be referred to as a viewpoint position), a position shift may occur between the object and the AR image. In other words, the display position of the AR image determined so as to match the object position in design and calculation by the HUD apparatus does not match the object position in actual visual recognition by the driver in some cases. For example, the cause of the shift between the object and the AR image is that the eye position is displaced from a viewpoint position of a basic setting (eye box corresponding to a range including the viewpoint position) due to an action of changing the posture, an action of moving the head by the driver, or the like.

As an example, when the viewpoint position moves to the left or right in the horizontal direction from the viewpoint position of the basic setting, the shift in the horizontal direction occurs between the object position and the display position of the AR image as viewed from the viewpoint position after movement. The larger the amount of movement of the viewpoint position, the larger the shift. As the shift becomes larger, the visual recognition in which the object and the AR image are associated with each other becomes more difficult or impossible for the driver. Such AR images are unsuitable.

An object of the present invention is to provide a technology capable of realizing a suitable AR display by reducing a shift between an object and a virtual image (AR image) in relation to a technology of a HUD apparatus having an AR function.

Means for Solving the Problems

A typical embodiment of the present invention is a head up display apparatus or the like, and is characterized by having the following configuration.

A head up display apparatus according to an embodiment is configured to project an image to a windshield or a combiner of a vehicle, thereby displaying a virtual image to be superimposed on a real image in front of the vehicle for a driver, the head up display apparatus comprises: a controller configured to perform control to display the virtual image in a visible region of the windshield or the combiner; and a display unit configured to display the virtual image in the visible region in accordance with the control of the controller, the controller extracts a predetermined object based on an image taken by an external camera, the controller acquires information including object information including an object position of the object in space, viewpoint information including a viewpoint position of the driver and the amount of movement of the viewpoint position in space based on an image taken by an internal camera, and virtual image information including a position of a virtual image region which is a range in which the virtual image can be displayed in space, the controller generates the image displayed to be superimposed on the object, the controller corrects a display position of the image in the virtual image region including at least a position in a horizontal direction by using the acquired information, the controller performs the control to the display unit by using data after the correction, and the controller performs conversion process at the time of the correction so that a position of an intersection between a straight line when viewing the object position through the visible region from a viewpoint position after movement from a viewpoint position of basic setting of the driver and the virtual image region is set as a display position of the image after correction.

Effects of the Invention

According to a typical embodiment of the present invention, it is possible to realize a suitable AR display by reducing a shift between an object and a virtual image (AR image) in relation to a technology of a HUD apparatus having an AR function.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Hereinafter, an embodiment of the present invention will be described in detail with reference to drawings. Note that the same members are denoted by the same reference characters in principle throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted.

Embodiment

A head up display apparatus (HUD apparatus) according to an embodiment of the present invention will be described with reference toFIG. 1toFIG. 23. The HUD apparatus according to the embodiment is an AR display apparatus having an AR function, and the case where the HUD apparatus is mounted as an in-vehicle AR-HUD will be described. A display control method according to the embodiment is a method including steps executed by the HUD apparatus according to the embodiment.

The HUD apparatus according to the embodiment has a function of correcting a display position of an AR image in a virtual image region when displaying a virtual image including the AR image. This correction function is a function of automatically correcting the display position of the AR image in accordance with the movement of the viewpoint position of the driver so that the shift between the real image of an object and the AR image is eliminated or reduced. In this correction function, when the viewpoint position of a driver moves from the viewpoint position of the basic setting to the left or right position in the horizontal direction, the display position of the AR image is corrected in accordance with the object position and the viewpoint position after movement. Also, in this correction function, the correction of the display position of the AR image is controlled in accordance with the setting of the position of the virtual image region with respect to the viewpoint position and the relationship of the object position with respect to the position of the virtual image region. In this correction function, the correction of the display position of the AR image is realized by the process for correcting the content of the AR data by the software program processing of the controller.

With this correction function, even if the viewpoint position moves in the horizontal direction from the viewpoint position of the basic setting due to the movement of the head of the driver during driving or the like, the AR image can be displayed at a position accurately superimposed on the object. Namely, the shift between the object and the AR image can be eliminated or reduced. Consequently, the driver can visually recognize the object and the AR image suitably.

Supplementary descriptions for the problems and the like will be given with reference toFIG. 20toFIG. 23. As a comparative example with respect to the embodiment, the in-vehicle system including the HUD apparatus of the related art has the following problems. The HUD apparatus according to the comparative example does not have the function of correcting the shift between the real image of the object and the AR image. Therefore, when the viewpoint position of the driver moves in the horizontal direction, a shift occurs between the object and the AR image in the horizontal direction, so that it becomes difficult for the driver to perform the visual recognition in which the object and the AR image are associated with each other.

Examples of the case in which the viewpoint position of the driver moves left or right in the horizontal direction include the following cases. The viewpoint position of the driver moves to some extent during driving, and it is impossible that the viewpoint position does not move at any time. If the change of the viewpoint position is within the range of the predetermined eye box, there is little influence of the shift between the object and the AR image. If the change of the viewpoint position exceeds a certain amount, the influence of the shift is large, and it becomes difficult for the driver to perform the visual recognition in which the object and the AR image are associated with each other.

For example, when a vehicle turns right or left, changes lanes, enters an intersection, or the like, the driver tends to move the viewpoint position in order to check the surrounding conditions. In addition, when entering a corner or the like, the head and the viewpoint position are likely to move because the driver leans his/her body in accordance with the corner in advance or due to the effect of centrifugal force. Also, during one-hand driving, the viewpoint position is likely to move because the center of gravity of the body is inclined. Further, in a situation where the visibility is poor at an intersection or the like, the driver sometimes leans forward to check the surrounding situation, so that the viewpoint position is likely to move.

Note that, in the HUD technology, how it looks differs depending on the position of the eyes of a person who views the virtual image region (for example, a driver), for example, the relative positional relationship between the real image and the virtual image is shifted. The state of the image formed on the retina of the viewer is determined by the relationship between the position of the virtual image formed by the HUD apparatus and the position of the eyes of the viewer. When the display position of the virtual image by the HUD apparatus and the position of the eyes of the viewer do not satisfy the predetermined relationship, the viewer cannot or cannot easily recognize the virtual image visually. The range in which the position of the eyes must enter for enabling a viewer to visually recognize the image is referred to as an eye box.

FIG. 20shows a viewpoint, an object, an AR image, a shift, and the like in the case of a first HUD apparatus in a related art example as a first comparative example.FIG. 20shows a plane (X-Z plane), which is formed by the horizontal direction (X direction) and the front-back direction (Z direction), viewed from above. The viewpoint position of the driver of the own vehicle is denoted by a triangle mark. A viewpoint position E1of the basic setting and a viewpoint position Ex after movement, for example, when moved to the right in the X direction are shown.

The object position is denoted by a black dot. In this example, the case where the objects are located at three positions having relatively different distances at the back of the virtual image region7in the Z direction is shown. The positions are indicated as a long-distance position PA, a middle-distance position PB, and a short-distance position PC. For example, the position PA is located at a position with a distance of about 40 to 60 m, for example, 45 m from the viewpoint position. The position PB is located at a position with a distance of about 20 to 30 m, for example, 25 m from the viewpoint position. The position PC is located at a position with a distance of 5 m from the viewpoint position. The distance from the viewpoint position to the object position (for example, position PC) in the Z direction is indicated as an object distance FA. In this case, at each position in the Z direction, an object is located at each of the left, center, and right positions in the X direction as viewed from the viewpoint position E1.

In the first HUD apparatus, the virtual image region7is set at a position with a predetermined distance relatively close in a forward direction (Z direction) from the viewpoint position E1of the basic setting in design. The position of the virtual image region7in the Z direction is defined as a virtual image region position P1. The distance from the viewpoint position (0 m) to the virtual image region7is indicated as a virtual image distance FB. In the first HUD apparatus, the virtual image distance FB at the virtual image region position P1is 2 to 3 m, and is 2.5 m in this example. Also, the distance between the object position (for example, position PC) and the virtual image region position P1in the Z direction is indicated as an object virtual image distance FC. A display position of the AR image displayed to be superimposed on the object in the virtual image region7at the virtual image region position P1, the display position being set in accordance with the object position of each object as viewed from the viewpoint position E1, is denoted by a white dot. A dash-dotted line extending in the Z direction as viewed from the viewpoint position E1indicates the line of sight for viewing the center of the virtual image region7and the center object in the X direction. Each solid line indicates the line of sight for viewing the left object and the right object. When viewing each object and the AR image from the viewpoint position E1, the positions of the object and the AR image overlap on the line of sight, and there is no shift.

As viewed from the viewpoint position Ex after movement, dotted lines indicate the lines of sight for viewing the objects, and broken lines indicate the lines of sight for viewing the AR images. When viewing the object and the AR image from the viewpoint position Ex, the lines of sight for the object and the AR image do not overlap (the angles formed by the lines of sight differ), and there is a position shift in the X direction.

Note that a triangular region including the viewpoint position E1of the basic setting and the left and right ends of the virtual image region7corresponds to a region where the virtual image can be displayed, and the outside thereof basically corresponds to a region where the virtual image cannot be displayed. If the virtual image region position can be adjusted, the displayable region and the non-displayable region also change accordingly.

FIG. 21similarly shows a viewpoint, an object, a virtual image, a shift, and the like in the case of a second HUD apparatus as a second comparative example. A HUD apparatus1according to the embodiment is based on the second HUD apparatus. In the second HUD apparatus, the virtual image region7is set at a position with a predetermined distance relatively far in a forward direction (Z direction) from the viewpoint position E1of the basic setting in design. The virtual image region position P1is a position with the virtual image distance FB of 20 to 30 m, and the case where P1=PB=25 m is shown in this example.

As the object position and the object distance FA in the Z direction, a case where objects are each located in front of and at the back of the virtual image region position P1is shown. A position P3(particularly defined as a long-distance position PA) indicated as (A) corresponds to the case where the object is located at the back of the virtual image region7(virtual image region position P1), and a position P2(particularly defined as a short-distance position PC) indicated as (B) corresponds to the case where the object is located in front of the virtual image region position P1of the virtual image region7. In addition, at each position in the Z direction, an object is located at each of the left, center, and right positions in the X direction as viewed from the viewpoint position E1in this example. The object at the position P3has the object virtual image distance FC of +20 m, and the object at the position P2has the object virtual image distance FC of −20 m.

A display position of the AR image when the AR image is displayed to be superimposed on each object in the virtual image region7at the virtual image region position P1as viewed from the viewpoint position E1is denoted by a white dot. When viewing the object and the AR image from the viewpoint position E1, the object and the AR image overlap on the line of sight (dash-dotted line, solid line) and there is no shift. When viewing the object and the AR image from the viewpoint position Ex after movement, the line of sight (dotted line) for viewing the object and the line of sight (broken line) for viewing the AR image do not overlap each other and have different angles, and there is a position shift in the X direction.

As in the examples ofFIG. 20andFIG. 21, the position shift occurs between the object and the AR image in visual recognition in accordance with the object position, the object distance, and the viewpoint position after movement. In particular, in the case of the second HUD apparatus, the influence of the shift is relatively larger when the object is in front of the virtual image region7as in the position P2in (B) than when the object is at the back of the virtual image region7as in the position P3in (A). When viewed from the driver, it is more difficult to focus on the object and the AR image and to perform the visual recognition in which the object and the AR image are associated with each other in the case where the object is in front of the virtual image region7.

FIG. 22simply shows examples of the shift in accordance with the movement of the viewpoint position in the X direction in the case of the first HUD apparatus. InFIG. 22and others, the distance in the Z direction is illustrated as being small and the distance in the X direction is illustrated as being emphasized for the sake of explanation. (A) ofFIG. 22shows the case where the viewpoint position moves leftward and the object4is located at the center position PB with the middle distance. A viewpoint position E2after the movement to the left from the viewpoint position E1is shown. The amount of movement H1of the viewpoint position is shown. An AR image6aindicates an AR image in the case of no correction. When viewing the object4and the AR image6afrom the viewpoint position E1of the basic setting, the angle formed by the lines of sight is 0 degrees, and there is no shift in the X direction. When viewing the object4and the AR image6afrom the viewpoint position E2after movement, there is a shift h1between the two in the X direction (shown as the shift considered at the position PB). An angle α1indicates an angle formed between the line of sight when viewing the object4from the viewpoint position E2and the Z direction. An angle β1indicates an angle formed between the line of sight when viewing the AR image6afrom the viewpoint position E2and the Z direction. The angle α1and the angle β1are different from each other (α1<β1).

(B) ofFIG. 22similarly shows the case where the viewpoint position moves rightward and the object4is located at the center position PB with the middle distance. A viewpoint position E3after the movement to the right from the viewpoint position E1is shown. The amount of movement H2of the viewpoint position is shown. When viewing the object4and the AR image6afrom the viewpoint position E3after movement, there is a shift h2between the two. An angle α2indicates an angle when viewing the object4from the viewpoint position E3. An angle β2indicates an angle when viewing the AR image6afrom the viewpoint position E3. The angle α2and the angle β2are different from each other (α2<β2).

As described above, there is the shift in accordance with the moving direction and the amount of movement (moving distance) of the viewpoint position. The angle at the viewpoint position after movement is larger than the angle at the viewpoint position E1, and the larger the angle, the larger the shift.

(C) ofFIG. 22similarly shows the case where the viewpoint position moves leftward and the object position is located at a long-distance position PA and a short-distance position PC. When viewing the object4aat the position PA and the uncorrected AR image6afrom the viewpoint position E2after movement, there is a shift h3between the two. An angle α3indicates an angle when viewing the object4afrom the viewpoint position E2. An angle β3indicates an angle when viewing the AR image6afrom the viewpoint position E2. Also, when viewing the object4cat the position PC and the uncorrected AR image6afrom the viewpoint position E2, there is a shift h4between the two. Note that the shift h3and the shift h4are illustrated as the shifts considered at the position PA for comparison. An angle α4indicates an angle when viewing the object4cfrom the viewpoint position E2. The angle α3and the angle α4are different from each other (α3<α4). The angle difference from the angle β3is larger in the case of the angle α3at the position PA than in the case of the angle α4at the position PC, and the shift is also larger in the case of the angle α3than in the case of the angle α4. As described above, there is the shift in accordance with the object position with each object distance.

(D) ofFIG. 22similarly shows the case where the viewpoint position moves leftward and the object is located at each of the left and right positions in the X direction at the middle-distance position PB. At the position PB, an object4b1is located at the position on the left side from the center, and an object4b2is located at the position on the right side. When viewing the object4b1on the left side and the uncorrected AR image6a1from the viewpoint position E2after movement, there is a shift h5between the two. Further, when viewing the object4b2on the right side and the uncorrected AR image6a2from the viewpoint position E2, there is a shift h6between the two. The angle differs between the case where the object and the AR image are viewed from the viewpoint position E1of the basic setting and the case where the object and the AR image are viewed from the viewpoint position E2after movement, and the shift is larger in the latter case. For example, an angle γ1when viewing the object4b2and the AR image6a2from the viewpoint position E1, an angle γ2when viewing the object4b2from the viewpoint position E2, and an angle γ3when viewing the AR image6a2from the viewpoint position E2are shown. γ1is smaller than γ2, and γ2is smaller than γ3(γ1<γ2<γ3). Thus, there is the shift in accordance with the object position in the X direction and the moving direction of the viewpoint.

As described above, when the object4and the AR image6of the virtual image region7are viewed from the viewpoint position of the driver through the visible region5of the windshield9, the angle formed by the line of sight and the like are different between the case where they are viewed from the viewpoint position of the basic setting and the case where they are viewed from the viewpoint position after movement. When viewed from the viewpoint position after movement, the shift in the X direction occurs between the object position and the display position of the AR image6. As the shift becomes larger, it becomes more difficult for the driver to perform the recognition in which the object4and the AR image6are associated with each other, and the driver cannot perform the suitable visual recognition.

In (A) ofFIG. 22, a position201indicates an ideal position where the AR image6is desired to be superimposed in the virtual image region7in the case of the viewpoint position E2. The position201is located at the intersection between the line of the virtual image region7in the X direction and the line of sight for viewing the object4from the viewpoint position E2. When the AR image6is displayed to be superimposed at the position201, the object4and the AR image6overlap as viewed from the viewpoint position E2, and the shift is eliminated. Similarly, a position202in (B), positions203and204in (C), and positions205and206in (D) indicate ideal display positions of the AR image6.

FIG. 23shows examples of the shift in accordance with the movement of the viewpoint position in the X direction in the case of the second HUD apparatus. (A) ofFIG. 23shows the case where the viewpoint position moves leftward from the viewpoint position E1. The viewpoint position E2after movement and the amount of movement H1are shown. As the object position, a case where the object4ais located at a position P3(=PA) at the back of the position PB corresponding to the virtual image region position P1and a case where the object4cis located at a position P2(=PC) in front of the position PB are shown. When viewing the object4aat the position P3and the AR image6abefore correction from the viewpoint position E2, there is a shift h11between the two in the X direction. When viewing the object4cat the position P2and the AR image6abefore correction from the viewpoint position E2, there is a shift h12between the two. Note that the shifts h11and h12at the virtual image region position P1are shown for comparison. An angle cdl indicates the angle when viewing the object4aat the position P3. An angle α12indicates the angle when viewing the object4cat the position P2. An angle β11indicates the angle when viewing the AR image6aat the virtual image region position P1. α11is smaller than α12(α11<α12), α11is smaller than β11(α11<β11), and α12is larger than β11(α12>β11). The shift at the position P2is larger than that at the position P3(h11<h12).

(B) ofFIG. 23similarly shows the case where the viewpoint position moves rightward. The viewpoint position E3after movement and the amount of movement H2are shown. When viewing the object4aat the position P2and the AR image6abefore correction from the viewpoint position E3, there is a shift h13between the two. When viewing the object4cat the position P2and the AR image6abefore correction from the viewpoint position E3, there is a shift h14between the two. The shift at the position P2is larger than that at the position P3(h13<h14).

As described above, the shift occurs in accordance with each viewpoint position and each object position. In particular, the influence of the shift is larger in the case where the object4is located in front of the virtual image region7as in the position P2than the case where the object4is located at the back of the virtual image region7as in the position P3.

In (A) ofFIG. 23, positions211and212indicate ideal positions where the AR image6is desired to be superimposed in the virtual image region7. For example, the position211is located at the intersection between a line of the virtual image region7in the X direction and a line of sight for viewing the object4afrom the viewpoint position E2(straight line connecting the viewpoint position E2and the position of the object4a). When the AR image6is displayed to be superimposed on the position211, the object4aand the AR image6overlap as viewed from the viewpoint position E2, and the shift is eliminated. In the HUD apparatus1according to the embodiment, these positions are obtained as display positions of the AR image6after correction. Similarly, positions213and214in (B) indicate ideal display positions of the AR image for the respective object positions and display positions after correction in the embodiment.

As described above, in the first HUD apparatus and the second HUD apparatus, the virtual image region7is set at a position (virtual image region position P1) with a predetermined distance (virtual image distance FB) from the viewpoint position E1in the forward direction (Z direction). The virtual image region7is a HUD display range in which a virtual image can be displayed by the HUD apparatus. The virtual image including the AR image is displayed in the virtual image region7with the virtual image distance FB at the virtual image region position P1. In the first HUD apparatus, the virtual image region position P1is a position having the virtual image distance FB of 2 to 3 m and is relatively close to the driver and the windshield9. On the other hand, in the second HUD apparatus and the HUD apparatus1according to the embodiment, the virtual image region position P1is a position having the virtual image distance FB of 20 to 30 m and is relatively far from the driver and the windshield9, in other words, the position closer to the object. Further, in the HUD apparatus1according to a modification of the embodiment, the virtual image region position P1in the Z direction can be variably set and controlled within a predetermined setting range (for example, 20 to 30 m).

Therefore, in the second HUD apparatus and the HUD apparatus1according to the embodiment, the AR image can be displayed to be superimposed at a distance and position as close as possible to the object as compared with the first HUD apparatus. In other words, the object virtual image distance FC can be made relatively small. When a driver views an object and an AR image associated with the object from a viewpoint position, the sense of perspective and the focus fit more easily as the distance between the object and the AR image becomes closer in the Z direction. Therefore, the driver can easily recognize the object and the AR image and visually recognize the object and the AR image more naturally and suitably.

As described above, the first HUD apparatus and the second HUD apparatus have respective characteristics regarding the shift between the object and the AR image viewed from the viewpoint position. In particular, in the second HUD apparatus, the influence of the shift differs in accordance with the relationship of the object position and the object distance with respect to the virtual image region position P1in the Z direction. As shown inFIG. 21andFIG. 23, the influence of the shift is larger when the object is in front of the virtual image region than when the object is at the back of the virtual image region. Even if the object virtual image distance FC is the same in the case where the object is located in front of the virtual image region and the case where the object is located at the back of the virtual image region, the shift is more likely to be perceived visually in the case where the object is located in front of the virtual image region.

In the HUD apparatus1according to the embodiment, a function of correcting the display position of the AR image is devised in consideration of the problems of the first HUD apparatus and the second HUD apparatus described above. In this correction function, the display position of the AR image including at least the position in the X direction is appropriately corrected in accordance with the relationship between the object position and the viewpoint position and the relationship between the virtual image region position and the object position.

FIG. 1shows an overview of the HUD apparatus1according to the embodiment. InFIG. 1and others, the X direction, the Y direction, and the Z direction are defined as directions for explanation. The X direction is a first horizontal direction and a left-right direction based on a vehicle and a driver, and corresponds to a lateral direction of the virtual image region7. The Y direction is a vertical direction and an up-down direction based on the vehicle and the driver, and corresponds to a longitudinal direction of the virtual image region7. The Z direction is a second horizontal direction and a front-back direction based on the vehicle and the driver, and corresponds to a direction of viewing the virtual image region7and the like.

The HUD apparatus1according to the embodiment is a part of an in-vehicle system (FIG. 2) and includes a controller10and a display unit20, and an external camera2, an internal camera3, and the like are connected to the HUD apparatus1. The display unit20is configured of a projector (projection image display apparatus) or the like and includes an optical system. The HUD apparatus1detects the position and the like of the object4located in front of (Z direction) the vehicle based on the image taken by the external camera2. In addition, the HUD apparatus1detects a viewpoint position and the like of the driver based on the image of the driver taken by the internal camera3. In the HUD apparatus1, the controller10generates AR data for displaying a virtual image including an AR image based on the detected object4, and provides the display unit20with the generated AR data. The AR data is display data for displaying the virtual image by forming the predetermined virtual image region7ahead through the visible region5of the windshield9as viewed from the driver. The display unit20generates and emits image light to be projected onto the visible region5of the windshield9based on the AR data. The emitted image light is projected through the optical system to the region in the visible region5of the windshield9, is reflected in the region, and enters the eyes of the driver. Consequently, the driver visually recognizes the image light as the virtual image including the AR image6displayed to be superimposed on the object4in the virtual image region7seen through the visible region5.

The controller10of the HUD apparatus1implements the correction function that is a correction function of a display position of an AR image. In this correction function, the display position of the AR image6in the virtual image region7is corrected in accordance with the positional relationship among the object position, the viewpoint position, and the virtual image region position. In this correction function, setting information such as the virtual image region position P1is referred to. In this correction function, based on the display position of the AR image6aof the viewpoint position E1of the basic setting, the display position of the AR image6bafter correction as viewed from the viewpoint position Ex after movement is calculated in accordance with the amount of movement of the viewpoint position. In this correction function, the correction is performed so that the display position of the AR image6bafter correction is set at the intersection with the virtual image region7on a straight line for viewing the object position from the viewpoint position Ex after movement. In this correction function, the display position of the AR image6bafter correction is obtained by a predetermined conversion process in the controller10. In this correction function, AR data including information of the display position of the AR image6bafter correction is generated by the process of the controller10and is provided to the display unit20. The display unit20projects and displays the AR image6bof the virtual image region7in accordance with the AR data.

InFIG. 1, the virtual image region7is set at the virtual image region position P1with a predetermined distance ahead through the visible region5of the windshield9as viewed from the viewpoint position E1of the basic setting of the driver. In this example, a target object4on which the AR image6is displayed to be superimposed is located at the position P2in front of the virtual image region7in the Z direction. The AR image6abefore correction and the AR image6bafter correction are shown. A position Qa indicates the display position of the AR image6abefore correction in the X direction. A position Qb indicates the display position of the AR image6bafter correction in the X direction.

In the AR function, the AR image6is displayed to be superimposed on the real image of the object4in the virtual image region7as viewed from the viewpoint position E1of the basic setting. The driver sees the real image of the object4and the AR image6of the virtual image region7ahead through the visible region5of the windshield9from the viewpoint position E1or the like. On this premise, if the viewpoint position of the driver moves from the viewpoint position E1to the left or right position in the horizontal direction (X direction), for example, to the viewpoint position Ex after movement to the right, the shift occurs between the object4and the AR image6ain the state before correction. In other words, a difference and a distance are made between the object position and the AR image display position as viewed from the viewpoint position Ex. As the shift becomes larger, it becomes more difficult for the driver to perform the recognition in which the object and the AR image are associated with each other.

On the other hand, in the HUD apparatus1according to the embodiment, the display position (position Qa) of the AR image6ais corrected to the display position (position Qb) of the AR image6bin accordance with the object position and the viewpoint position by the correction function. In the state after correction, when the object4and the AR image6bare viewed from the viewpoint position Ex, the position shift is eliminated.

The HUD apparatus1detects the position and the distance of the object4with respect to the position of the own vehicle and the viewpoint position based on the image of the external camera2. Further, the HUD apparatus1detects the viewpoint position of the driver of the own vehicle and the amount of movement of the viewpoint position based on the image of the internal camera3. Then, the HUD apparatus1controls the correction based on the relationship between the virtual image region position P1and the object position in the Z direction. The HUD apparatus1corrects the display position of the AR image in accordance with the object position, the viewpoint position, and the virtual image region position P1. When displaying and correcting the virtual image, the controller10first determines the display position of the AR image6afrom the viewpoint position E1of the basic setting as a basis. Next, the controller10determines by correction the display position of the AR image6bafter correction as viewed from the viewpoint position Ex after movement in accordance with the amount of movement of the viewpoint position.

Note that, inFIG. 1and others, the viewpoint position E1and the virtual image region7of the basic setting are illustrated as being located at the center in the X direction with respect to the visible region5of the windshield9for the sake of explanation, but the configuration is not limited to this.

FIG. 2shows a configuration of an in-vehicle system100including the HUD apparatus1according to the embodiment. The in-vehicle system100is a system mounted on an automobile. A driver who is a user operates and uses the in-vehicle system100and the HUD apparatus1. In particular, the HUD apparatus1is an AR-HUD apparatus having an AR function.

The in-vehicle system100includes an ECU (Engine Control Unit)101, the HUD apparatus1, an external photographing unit102including the external camera2, a video data storage unit103, a communication unit104, a GPS (Global Positioning System) receiver105, a car navigation unit106, a sensor unit108, a DB unit109, and the like, and these are connected to an in-vehicle bus and a CAN (Car Area Network)110. The in-vehicle system100further includes an audio output unit, an operation unit, a power supply unit, and the like (not shown).

The HUD apparatus1includes the controller10and the display unit20. The HUD apparatus1has the AR function. The HUD apparatus1can give various types of information to the driver by displaying the virtual image including the AR image in the virtual image region7through the visible region5of the windshield9by using the AR function. The AR function of the HUD apparatus1includes a correction function. The correction function is a function of automatically correcting the display position of the AR image in the virtual image region7.

The HUD apparatus1can also provide an audio output such as the guidance or alarm by the car navigation function and the AR function to the driver by using the audio output unit in combination. The HUD apparatus1further includes an operation unit such as an operation panel and operation buttons, so that the manual operation input by a driver, for example, turning on/off of the AR function, the user setting, the adjustment of a mirror angle of the optical system24, and the like are also possible.

The controller10controls the entire HUD apparatus1. The controller10includes hardware such as a CPU, a ROM, and a RAM and corresponding software. The controller10and other units may be implemented by hardware such as a microcomputer and an FPGA. The controller10realizes each unit such as the image input unit11by, for example, reading a program from the ROM by the CPU and executing a process in accordance with the program. The controller10stores various types of data and information in an internal memory or an external memory as necessary, and performs reading and writing. The controller10stores setting information and the like for the AR function in a nonvolatile memory of the memories. The setting information includes setting information for conversion process of the correction unit14and user setting information. The user setting information includes the viewpoint position and virtual image region position of the basic setting, the setting information of the mirror angle of the optical system24, and the like.

The controller10receives an input of an image taken by the external camera2, generates AR data for displaying a virtual image in the virtual image region7of the visible region5with using the acquired information, and provides the display unit20with the AR data. The controller10corrects the display position of the AR image in the virtual image region7when generating the AR data. In addition, the controller10can adjust the state of the optical system24such as the mirror angle by controlling the display unit20. Thus, the position of the virtual image region7can be adjusted.

The display unit20is configured of a projection image display apparatus (projector) or the like. The display unit20includes a display drive circuit21, a display element22, a light source23, the optical system24, and a drive unit25, and these are connected to each other. The display unit20generates image light for displaying a virtual image based on the control from the controller10and the video data (AR data), and projects the generated image light to the region within the visible region5. The display drive circuit21generates a display drive signal for AR display in accordance with the video data from the AR display unit15and supplies the display drive signal to the display element22and the light source23to control the drive of the display element22and the light source23.

The light source23generates illumination light for the display element22based on the display drive signal. The light source23is configured of, for example, a high-pressure mercury lamp, a xenon lamp, an LED element, a laser element, or the like. The light from the light source23is incident on the display element22through an illumination optical system (not shown). The illumination optical system collects the illumination light, makes it uniform, and irradiates the light to the display element22.

The display element22generates image light based on the display drive signal and the illumination light from the light source23, and emits the image light to the optical system24. The display element22is configured of, for example, an SLM (Spatial Light Modulator), a DMD (Digital Micromirror Device, registered trademark), a MEMS device, an LCD (Transmissive liquid crystal panel or reflective liquid crystal panel), or the like.

The optical system24includes elements such as a lens and a mirror for guiding the image light from the display element22to the visible region5of the windshield9. The drive unit25is connected to the optical system24. The image light from the display element22is magnified or the like by the lens of the optical system24, is reflected or the like by the mirror, and is projected to a part of the visible region5(FIG. 3). The image light reflected in the region enters the eyes of the driver and forms an image on the retina. Consequently, when viewed from the viewpoint of the driver, the virtual image region7that is seen through the visible region5is formed so as to correspond to the region within the visible region5, and a virtual image is visually recognized in the virtual image region7.

The drive unit25is an optical system driver for driving the optical system24, and includes components such as a motor for driving a lens, a mirror, and the like. The drive unit25drives the optical system24in accordance with the manual operation input of the driver or the control from the controller10, thereby changing the angle of the mirror or the like. The drive unit25includes an operation button for adjusting the angle of the mirror or the like. The driver can change the angle of the mirror to positive or negative with reference to the standard angle by manual operation of moving the operation button up and down. For example, while a first portion of the operation button is being pressed, the angle of the mirror changes in the positive direction (the direction of increasing the angle), and while a second portion is being pressed, the angle of the mirror changes in the negative direction (the direction of decreasing the angle).

The ECU101performs vehicle control including engine control and overall control of the in-vehicle system100. The ECU101is, in other words, a vehicle controller. The ECU101may have advanced functions for driving assistance and automatic driving control. In this case, the ECU101may control the HUD apparatus1by outputting information related to the function to the HUD apparatus1, thereby causing the HUD apparatus1to display a virtual image related to the function. The ECU101obtains vehicle information such as a vehicle speed based on the detection information from the sensor unit108and uses it for control. Further, the HUD apparatus1can acquire vehicle information from the ECU101and use it for control.

The external photographing unit102includes one or more external cameras2, and captures a situation including the front of the own vehicle by using the external camera2when the own vehicle is stopped or running, thereby acquiring video data (including time-series image frames) and vehicle surrounding information. The external photographing unit102stores the video data and the like in the video data storage unit103or outputs the video data and the like to the ECU101and the HUD apparatus1.

The external camera2is installed at a predetermined position of the vehicle so as to have a predetermined direction and an angle of view (FIG. 3). The position of the external camera2is, for example, near the front bumper of the vehicle, near the side of the windshield9, or near the rearview mirror at the side of the vehicle. The external camera2takes an image in a predetermined shooting direction and in a range of a predetermined angle of view so as to include the front of the vehicle and the driver, and outputs video data.

The external photographing unit102may or may not include a signal processor that processes the image of the external camera2. The signal processor may be provided in the ECU101or the HUD apparatus1. The signal processor may process the images of one or more external cameras2and obtain the vehicle surrounding information and the like by calculation. The external photographing unit102may determine the situation such as another vehicle, a person, a building, a road surface, terrain, weather, and the like around the own vehicle based on the analysis of the image of the external camera2.

The external photographing unit102may have a function of measuring the distance between the own vehicle and the object (object distance) and the position of the object in space relative to the own vehicle (object position) based on the image of the external camera2. When two or more cameras or a stereo camera are provided as the external camera2, the object distance can be calculated by a known distance measurement method based on binocular disparity by using two images taken by the two left and right cameras. Further, even when only one external camera2is provided, the object position in space, the object distance, and the like can be schematically calculated by a predetermined calculation based on the object position in the image of the external camera2, the detection information of other sensors, and the like.

The video data storage unit103stores video data and the like from the external camera2. The video data storage unit103may be provided inside the external photographing unit102or inside the HUD apparatus1. Further, the video data storage unit103may store the video data of the internal camera3of a driver photographing unit107and the like.

The communication unit104includes a communication interface device that performs communication with a mobile network outside the vehicle, the Internet, and the like. The communication unit104can communicate with, for example, a server or the like on the Internet based on the control from the ECU101, the HUD apparatus1, and the like. For example, the HUD apparatus1may refer to and acquire original data and related information to be used for AR display from the server via the communication unit104. The communication unit104may include a vehicle-to-vehicle communication wireless receiver, a road-to-vehicle communication wireless receiver, a VICS (Vehicle Information and Communication System) receiver, and the like. The vehicle-to-vehicle communication is the communication between the own vehicle and other vehicles around the own vehicle. The road-to-vehicle communication is the communication between the own vehicle and roads and devices such as traffic lights around the own vehicle.

The GPS receiver105acquires current position information (for example, latitude, longitude, altitude, and the like) of the own vehicle based on signals from GPS satellites. The ECU101, the HUD apparatus1, and the car navigation unit106can acquire the current position information of the own vehicle from the GPS receiver105and use it for control.

The car navigation unit106is a part of an existing car navigation system mounted on a vehicle, and is configured to hold map information, position information acquired from the GPS receiver105, and the like and perform the known navigation process by using the information. The ECU101and the HUD apparatus1can perform control by acquiring the information from the car navigation unit106. The HUD apparatus1may refer to the information from the car navigation unit106and use it as original data for AR display. For example, the HUD apparatus1may generate an arrow image for the navigation of a traveling direction to a destination as an example of the AR image based on the original data.

The sensor unit108has a group of known sensors mounted on the vehicle, and outputs detection information. The ECU101and the HUD apparatus1perform control by acquiring the detection information. Examples of sensor devices in the sensor unit108include a vehicle speedometer, an acceleration sensor, a gyro sensor, a geomagnetic sensor (electronic compass), an engine start sensor, a shift position sensor, a steering wheel angle sensor, a headlight sensor, an external light sensor (chromaticity sensor and illuminance sensor), an infrared sensor (proximity object sensor), a temperature sensor, and the like. The acceleration sensor and the gyro sensor detect acceleration, angular velocity, angle, and the like as the state of the own vehicle.

The sensor unit108may include a distance sensor that measures the distance between the own vehicle and the object. The distance sensor can be realized by, for example, an optical sensor, and can calculate the distance based on the time until the emitted light hits the object and returns. When the distance sensor is provided, object distance information can be acquired from the distance sensor.

The DB unit109is configured of a storage or the like, and the DB stores original data and information that can be used for AR display. Examples of the original data include basic image data for generating an AR image and the like. Examples of information include basic information and related information of a target object, a reference image, and the like. The DB unit109may be provided inside the HUD apparatus1, may be combined with the car navigation unit106, or may be provided in a data center or the like on the communication network outside the in-vehicle system100. Information acquired from the outside via the communication unit104may be stored in the DB.

The windshield9is a part of an automobile, and is configured of glass having transparency and rigidity, a film having predetermined optical characteristics, and the like (FIG. 3). The virtual image region7is formed in a part of the visible region5of the windshield9when the AR function is used. Note that a dedicated AR display panel (combiner or the like) may be provided in front of the windshield9. The virtual image region7can be set within the range of the visible region5.

The image input unit11receives an input of an image taken by the external camera2and extracts a predetermined object for AR from the image. An information acquisition unit12acquires object information, viewpoint information, vehicle information, virtual image information, and the like as information necessary for AR display, correction, and the like. The object information is information including an object position and an object distance. The object position is two-dimensional coordinates in an image, three-dimensional coordinates in space, or the like. The object distance is a distance between the own vehicle and the object. The vehicle information is information including a vehicle speed, a traveling direction, and the like. The viewpoint information is information such as a viewpoint position of the driver. The virtual image information is information such as a set position of the virtual image region7and the like.

An AR image generation unit13generates basic AR image data displayed to be superimposed on an object. The basic AR image has a basic display position in the virtual image region7. The correction unit14is, in other words, a display position conversion unit, and performs a correction process of the display position of the AR image in accordance with the correction function. The correction unit14corrects the display position of the AR image in the virtual image region7by using the basic AR image generated by the AR image generation unit13and each information acquired by the information acquisition unit12. The correction unit14performs the correction as a predetermined conversion process. The conversion process is defined in advance in, for example, a conversion table. The correction unit14outputs data including the display position of the AR image after correction. The AR display unit15generates AR data for displaying and superimposing the virtual image including the AR image on the virtual image region7by using the data after correction, and performs display control of the display unit20based on the AR data.

FIG. 3shows an overview of the viewpoint position, the virtual image region position, and the like in a plane in which a driver's seat of an automobile is seen from a side (X direction). The state inFIG. 3shows the basic setting state, in which the driver sees the AR image6in the virtual image region7and the object4that can be seen through the visible region5of the windshield9from the viewpoint position E1of the basic setting. The windshield9is present in front of the driver's seat and the visible region5is present as apart of the windshield9, and the virtual image region7is formed so as to correspond to a partial region307in the visible region5. The HUD apparatus1, the car navigation unit106, and the like are accommodated and installed at a position of a dashboard in front of the driver's seat, for example, at a position of a console panel.

The viewpoint position E1of the basic setting based on the position of the eyes of the driver is shown. Also, an eye box300including the viewpoint position E1is shown. Note that the viewpoint position may be different from the eye position. For example, the viewpoint position may be calculated as an intermediate point between the left and right eyes, or may be calculated as a center point of a head or a face. In addition, the line of sight301when viewing the AR image6in the virtual image region7and the object4from the viewpoint position E1through the region307in the visible region5is indicated by a dash-dotted line.

Examples of an installation position p2of the external camera2in a vehicle and a shooting direction302are shown. In this case, the shooting direction302is the Z direction. Examples of an installation position p3of the internal camera3in a vehicle and a shooting direction303are shown. Installation positions of the external camera2and the internal camera3are not limited to these and any installation positions are possible.

FIG. 3shows a mirror501in the optical system24in the HUD apparatus1. The mirror501has an angle θ and an angle ϕ as biaxial variable angles (FIG. 5). The image light304reflected by the mirror501is projected to the region307in the visible region5inside the windshield9and is reflected toward the eyes of the driver. A center position pC of the region307corresponds to the projection position of the image light304. Note that the configuration of the optical system24is not limited to this, and any configuration is possible as long as the position and direction in which the image light is projected to the visible region5are variable.

The AR image6in the virtual image region7is formed at a forward position through the visible region5so as to correspond to a position pA of the object4in space as viewed from the viewpoint position E1on the calculation of the HUD apparatus1. The center point of the virtual image region7in space and the display position of the AR image6when displayed at the center point are indicated as a position pB. The line of sight301passes through the position pC, the position pB, and the position pA, and the real image of the object4and the AR image6superimposed thereon are viewed along the line of sight301. In this case, the virtual image region7is shown as a plane having a sight line direction301as a normal line.

The mirror501is a free-form mirror or the like, for example, a concave mirror. The mirror501has an angle θ and an angle ϕ indicating the orientation. The mirror501has, for example, the angle θ of a rotation axis (X axis) extending in the X direction and the angle ϕ of a rotation axis (Y axis) extending in the Y direction. The angle θ and the angle ϕ of the mirror501can be varied within a predetermined range by the driving from the drive unit25. The projection direction and the projection position (position pC) of the image light304can be varied in accordance with the state of the angle θ and the angle ϕ. Further, a function of distortion correction for adapting to the region307having a curved surface can also be provided by the design of the curved surface of the mirror501or the like.

Each position such as the viewpoint position has three-dimensional coordinates in space. For example, the viewpoint position E1has (X1, Y1, Z1), the position pA has (XA, YA, ZA), the position pB has (XB, YB, ZB), and the position pC has (XC, YC, ZC). The controller10obtains the three-dimensional coordinates of each position by calculation.

The relationship among the position p2of the external camera2, the position p3of the internal camera3, and the viewpoint position is as follows. In the in-vehicle system100and the HUD apparatus1, the position p2of the external camera2and the position p3of the internal camera3in space are set in advance. Further, the viewpoint position of the driver generally falls within a predetermined range corresponding to the vicinity of the driver's seat, and is obtained by the driver photographing unit107. Although the three-dimensional coordinates of each position are different, the position p2, the position p3, and the viewpoint position (for example, the viewpoint position E1) have a predetermined relationship. These positions can be mutually converted by calculation based on the predetermined relationship, and can be aligned at the time of calculation such as the correction. In addition, the position of the own vehicle has a relationship with these positions, and can be mutually converted similarly. For example, the object position and the object distance viewed from the viewpoint position can be calculated from the object position and the object distance viewed from the position p2of the external camera2and the distance between the position p2and the viewpoint position E1. Further, as a modification, it is also possible to perform the control while omitting the calculation of the mutual conversion among the positions and considering each of the positions as substantially the same position, and a commensurate effect can be obtained.

FIG. 4simply shows the visible region5of the windshield9and the virtual image region7viewed from a front seat of a vehicle.FIG. 4shows a state where there is no shift between the real image and the virtual image (AR image). A steering wheel401and the like are provided on the right side, and a console panel402and the like are provided near the center of the dashboard, in which the HUD apparatus1, the car navigation unit106, and the like are accommodated. On the console panel402, operation input and screen display of the HUD apparatus1and the car navigation unit106are performed. The windshield9has the visible region5with a curved surface. On an upper side of the windshield9, an internal rearview mirror403and the like are installed. In this example, the internal camera3is installed at the upper right position of the windshield9. Although not shown, the external camera2is installed at any given position of the vehicle.

In an example of the real image inFIG. 4, another vehicle (oncoming vehicle) is present as the target object4on the road in front of the own vehicle, near the intersection. The image light is projected through an opening404of the HUD apparatus1to a region in the visible region5of the windshield9, so that the virtual image region7(indicated by a broken line frame and not displayed in actual) viewed from the driver is formed. The AR image6is displayed to be superimposed on the object4in the virtual image region7. In this example, the AR image6is an alert icon image for alerting the driver about the oncoming vehicle, and has an arch shape (ring shape having a notch).

FIG. 5shows the basic setting and the like of the viewpoint position E1and the virtual image region7in accordance with the individual driver. The viewpoint position E1of the basic setting is shown. The viewpoint position E1is set as a user setting value based on the initial setting value of the HUD apparatus1. The driver adjusts the driver's seat and the like in accordance with his or her physical size and posture, and takes a basic posture. Thereafter, the driver performs the basic setting of the virtual image region7and the viewpoint position E1of the HUD apparatus1. The driver sets the virtual image region7to be in a suitable state as viewed from the viewpoint of the driver by adjusting the angles θ, ϕ, and the like of the mirror501of the optical system24as necessary. The position of the virtual image region7can be changed in the X direction and the Y direction within a predetermined setting range. In a modification, the position of the virtual image region7can be further changed in the Z direction within a predetermined setting range.

At the time of the basic setting, the HUD apparatus1displays a guide AR image503(for example, a frame indicating the virtual image region7or a mark indicating the center position) for the basic setting in the virtual image region7. For example, the driver adjusts the angle of the mirror501and the like manually while watching the guide AR image503so as to be in a suitable state in accordance with the line of sight from his or her own viewpoint position.

The HUD apparatus1has the mirror501as a part of the optical system24. The HUD apparatus1has the drive unit25that variably drives the angles θ and ϕ of the mirror501. The drive unit25changes the angles θ and ϕ of the mirror501by driving a motor or the like based on the control from the controller10or the operation input from the operation unit (for example, operation buttons of the console panel402). The projection direction and the projection position (position pC) are determined in accordance with the angles θ and ϕ of the mirror501. By adjusting the angle θ, the position of the virtual image region7in the Y direction as viewed from the driver can be adjusted. In addition, by adjusting the angle ϕ, the position of the virtual image region7in the X direction as viewed from the driver can be adjusted. In the example ofFIG. 5, the case where the virtual image region7is adjusted to a position on the right side within the windshield9by adjusting the angle ϕ is shown.

At the time of the basic setting, the HUD apparatus1photographs the driver with the internal camera3, detects a viewpoint position based on, for example, an intermediate point of both eyes, and sets it as the viewpoint position E1of the basic setting. Further, the HUD apparatus1detects and sets the angles θ and ϕ of the mirror501in the state corresponding to the viewpoint position E1via the drive unit502. Note that the HUD apparatus1may automatically perform the basic setting by adjusting the angle of the mirror501while omitting the adjustment operation by the driver.

Further, in the HUD apparatus1according to a modification of the embodiment, the position and the like of the virtual image region7in the front-back direction (Z direction) can be variably set and controlled. Whether or not the display position of the AR image is corrected and the correction method can be set depending on the position of the virtual image region. As one of the setting information of the controller10, there are a virtual image region position and a virtual image distance. The HUD apparatus1can change the virtual image distance in accordance with the use or situation or in accordance with the user setting, and can set the virtual image region7at the position of the virtual image distance. For example, there are virtual image distances suitable for the traveling on a general road and the traveling on an expressway. The controller10of the HUD apparatus1can switch and set the virtual image distance in accordance with the determination of these situations (for example, the determination based on the vehicle speed). Further, the virtual image distance can be changed in accordance with an arbitrary input operation by the driver.

[Setting Example in Virtual Image Region]

In the AR region602, the AR image6(for example, an alert icon image) is displayed to be superimposed on the target object4, and is basically set as a target object to which the correction function is applied. There are various types of AR images6to be displayed in the AR region602. For each type of the object4, the type of the AR image6, whether or not the correction function is applied, and the like can be set. A table on the lower side ofFIG. 6shows an example of AR setting when displaying an AR image in the AR region602of the virtual image region7by the AR function and the correction function. In this table, information items include an object type, an AR image type, application or non-application of the correction function, and the like. The AR image type, the application or non-application of correction, and the like can be set for each object type. As a setting example, the setting in which, when the target object is a car corresponding to the object type K1, the AR image type G1is applied and the correction of the AR image display position by the correction function is applied is shown.

[Relationship Between Position and Distance]

FIG. 7shows the relationship between the position and the distance of the viewpoint, the object, and the virtual image (virtual image region7). The viewpoint position E1of the basic setting and the viewpoint position Ex after movement to the right in the X direction are shown. The amount of movement H (viewpoint moving distance) from the viewpoint position E1to the viewpoint position Ex is shown. As the object positions, the position P2(particularly on the right side in the X direction) in the case of the object4blocated in front of the virtual image region position P1of the virtual image region7in the Z direction and the position P3(particularly on the left side in the X direction) in the case of the object4alocated at the back of the virtual image region position P1of the virtual image region7in the Z direction are shown. The object4is denoted by a solid circle. The AR image6is denoted by a dotted circle. The distance from the viewpoint position to the object position (for example, position P2or position P3) is indicated as an object distance LA. The distance between the viewpoint position and the virtual image region position P1is indicated as a virtual image distance LB. The distance between the virtual image region position P1and the object position is indicated as a virtual image object distance LC (relative distance).

The virtual image region7is set at the virtual image region position P1with the predetermined virtual image distance LB from the viewpoint position E1in the forward direction (Z direction) through the windshield9. The virtual image distance LB is set by the HUD apparatus1and is variable in the case of a modification. The virtual image distance LB is 20 to 30 m (for example, 25 m) as in the above-described second HUD apparatus. When the object4is at the virtual image region position P1, the display position of the AR image6is the same as the position of the object4.

The display position of the AR image6of each object4in the X direction within the virtual image region7is denoted by a white dot. For example, in the case of the object4aat the object position P3, the AR image6A after correction viewed from the viewpoint position Ex after movement is shown based on the AR image6before correction viewed from the viewpoint position E1. In the case of the object4bat the object position P2, the AR image6B after correction viewed from the viewpoint position Ex after movement is shown based on the AR image6before correction viewed from the viewpoint position E1. For example, with respect to the object4b, the position Qa of the AR image before correction and the position Qb of the AR image6B after correction are shown.

In the correction function, the HUD apparatus1corrects the display position of the AR image6in accordance with the relationship between the position and the distance of the viewpoint, the object, and the virtual image region7as described above. In particular, the effect of the position shift differs between the case where the object4is located in front of the virtual image region7and the case where the object4is located at the back of the virtual image region7. Therefore, for example, the HUD apparatus1corrects the display position of the AR image6when the object4is located in front of the virtual image region7.

FIG. 8shows a main process flow of the controller10of the HUD apparatus1. This process includes the conversion process for the correction of the display position of the AR image. The controller10performs such process in real time while the own vehicle is running and the AR function is being used.FIG. 8includes steps S1to S10. Hereinafter, the steps will be described in order.

(S1) The HUD apparatus1performs a basic setting process based on the operation of the driver. As shown inFIG. 5, the viewpoint position E1, the virtual image region position P1, and the like of the basic settings are set. The HUD apparatus1performs the following process while the AR function is on.

(S2) The image input unit11of the controller10sequentially receives an input of the image frame of the video data of the external camera2of the external photographing unit102by, for example, reading it from the video data storage unit103.

(S3) The image input unit11performs a process of extracting a region of a predetermined object or the like for AR from the input image. The predetermined object is, for example, a car, a person, a bicycle, or the like on which an AR image is displayed to be superimposed, and a lane line or a mark on a road surface, a traffic sign, or the like used for AR control. Note that this extraction process can be realized by a known technology including image analysis such as the feature extraction and the image matching.

(S4) The image input unit11detects the two-dimensional coordinates (x, y) of the object position in the image, the object distance between the viewpoint position and the object position, and the like, with respect to the object4extracted from the input image. Alternatively, the image input unit11detects the three-dimensional coordinates (X, Y, Z) of the object position in space.

(S5) The image input unit11or the driver photographing unit107receives an input of an image of video data from the internal camera3. If the viewpoint position and the like can be detected by the function of the driver photographing unit107, the information acquisition unit12may be configured to acquire information of the viewpoint position and the like detected by the driver photographing unit107.

(S6) The image input unit11or the driver photographing unit107detects the current viewpoint position of the driver from the image of the internal camera3. The image input unit11or the driver photographing unit107detects two-dimensional coordinates (x, y) of the viewpoint position in the image or three-dimensional coordinates (X, Y, Z) of the viewpoint position in space. In addition, the image input unit11or the driver photographing unit107detects the amount of movement, the moving direction, and the like related to the viewpoint position Ex after movement from the viewpoint position E1of the basic setting.

(S7) The information acquisition unit12acquires object information, viewpoint information, vehicle information, virtual image information, and the like as information necessary for the virtual image display and the correction. The object information includes information such as an object position, an object distance, an object type (for example, a car, a person, and a bicycle), and a size. The viewpoint information includes a viewpoint position, the amount of movement, and the like. The vehicle information includes a vehicle speed, a traveling direction, and the like. The virtual image information includes setting information such as the virtual image region position P1.

The information acquisition unit12may acquire the object distance information from, for example, the external photographing unit102or the sensor unit108(for example, distance sensor). The information acquisition unit12may acquire the vehicle information from, for example, the ECU101or the sensor unit108. The information acquisition unit12may acquire the current position information of the own vehicle and the like from, for example, the sensor unit108, the GPS receiver105, or the car navigation unit106. The information acquisition unit12may acquire the information from outside via the DB unit109or the communication unit104.

(S8) The AR image generation unit13generates basic AR image data of the AR image6to be displayed and superimposed on the target object4by using each image and the acquired information. The basic AR image data generated here has a basic display position before correction. The AR image generation unit13generates an AR image in accordance with the object type or the like. The AR image generation unit13generates an AR image (for example, alert icon image) in accordance with the extracted object (for example, another vehicle). Also, the AR image generation unit13may generate an AR image (for example, navigation arrow image) with reference to data and information as appropriate from the ECU101, the car navigation unit106, the DB unit109, and the like. These depend on what information is provided as the AR function, and are not particularly limited. The AR image generation unit13outputs the generated AR image data and the like.

(S9) The correction unit14performs a conversion process for correcting the display position of the AR image6in the virtual image region7with respect to the basic AR image data generated by the AR image generation unit13. In this correction process, the correction unit14determines the display position of the AR image after correction in accordance with the object position, the object distance, the amount of movement of the viewpoint position, the front-back relationship of the object with respect to the virtual image region position P1, and the like. In other words, the amount of correction from the display position of the AR image before correction is determined.

The correction unit14determines the display position of the AR image after correction by performing a conversion process based on a predetermined conversion table (or a predetermined conversion formula). The correction unit14performs conversion process with reference to the conversion table with using the object position, the amount of movement of viewpoint position, the virtual image region position, and the like as input values, and outputs the display position of the AR image after correction and the like as output values.

The correction unit14performs the correction process for the object4located in front of the virtual image region7with using, for example, a second method described later as a correction method, and obtains the display position of the AR image after correction. In addition, the correction unit14compares the amount of movement of the viewpoint position in the X direction with a predetermined threshold value as a correction method, and performs the correction process when the amount exceeds the threshold value. The threshold value can be set in accordance with the eye box or the like.

(S10) The AR display unit15generates AR data which is the display data for the virtual image display in which the correction and the like up to S9are reflected. Further, when generating the AR data, the AR display unit15also performs the coordinate transformation for adapting from the plane (virtual image plane) on the calculation in the HUD apparatus1to the curved region of the windshield9(region307inFIG. 3). This coordinate transformation can be realized by a known projection transformation or the like, and is a transformation for forming the planar virtual image region7as a result of the projection onto a curved surface.

The display unit20generates and emits image light based on the AR data from the AR display unit15of the controller10. The image light is projected through the optical system24to the region307in the visible region5of the windshield9and is reflected. Consequently, the driver visually recognizes the AR image superimposed on the object in the virtual image region7. When the AR function is on, the main process as described above is similarly repeated.

[Acquisition of Object Position and Viewpoint Position]

FIG. 9shows the acquisition of the object position and the viewpoint position. (A) ofFIG. 9shows an image901of the external camera2. When the image901includes the object4(for example, another vehicle), two-dimensional coordinates (xm, ym) are detected as the position of the object4in the image901. Further, on the lower side of (A), the position of the own vehicle900, the object position of another vehicle (object4), and the distance D between the own vehicle900and another vehicle as the object distance are shown on the X-Z plane in space. Three-dimensional coordinates of the position of the own vehicle (Xm1, Ym1, Zm1) (associated with the viewpoint position as described above), three-dimensional coordinates of the object position (Xm2, Ym2, Zm2), and the distance D (DX, DY, DZ) are shown. The three-dimensional coordinates of the object position are position coordinates in a three-dimensional coordinate system based on the position of the own vehicle (viewpoint position). The distance DX in the distance D is a component in the X direction, the distance DY is a component in the Y direction, and the distance DZ is a component in the Z direction. The controller10obtains the three-dimensional coordinates and the like of the object position by using the image901and the above-described means.

(B) ofFIG. 9shows an image902of the internal camera3. As the viewpoint positions of the driver, the viewpoint position E1of the basic setting, the viewpoint position Ex after movement, the amount of movement H, the moving direction (including at least the X direction), and the like can be detected from the image902. As the two-dimensional coordinates of the viewpoint position in the image902, the viewpoint position E1(x1, y1) and the viewpoint position Ex (x2, y2) are shown. The viewpoint position E1and the viewpoint position Ex may be detected as two-dimensional coordinates, or may be detected as three-dimensional coordinates. The three-dimensional coordinates can be obtained based on the two-dimensional coordinates by a predetermined calculation. As the three-dimensional coordinates of the viewpoint positions, the viewpoint position E1(X1, Y1, Z1) and the viewpoint position Ex (X2, Y2, Z2) are shown. From the difference between these viewpoint positions, the amount of movement H of the viewpoint position and the like can be detected.

Note that, although the correction of the display position is realized by using the three-dimensional information as described above in the case of correcting the display position of the AR image by the correction function of the HUD apparatus1according to the embodiment, the correction may be simply performed by using the two-dimensional information. Note that, if the three-dimensional coordinates of the viewpoint position and the three-dimensional coordinates of the object position can be detected, it means that a straight line connecting those two points can be detected. Moreover, the straight line may be used for control by schematically regarding it as a line of sight.

In the HUD apparatus1according to the embodiment, at least the two-dimensional coordinates or three-dimensional coordinates of the viewpoint position are detected by calculation by using the internal camera3. Not limited to this, if the sight line direction (the line of sight of the eyes viewing a real image or a virtual image) and the blinking of the eyes can be detected as the function of the driver photographing unit107, the information can be used for controlling the correction function. For example, the display position of the AR image may be controlled in accordance with the viewpoint position and the sight line direction. For example, the control in which the correction is not performed when it is found that the virtual image region7(the object or the AR image) is not viewed by the detection of the sight line direction is also possible.

Further, as a modification, other than the correction of the display position of the AR image in accordance with the viewpoint position and the sight line direction, the control for processing the AR image may be added. Consequently, more accurate correction and AR presentation that can be visually recognized more easily become possible. For example, when the display position of the AR image is being corrected, the process of changing the shape or color of the AR image may be performed so that the state of correction (that is, the state of changing the viewpoint position) is understood by the driver. Combination with various known technologies for visually recognizing the AR image more easily is possible.

FIG. 10shows the basics when the AR image generation unit13generates the AR image6in the virtual image region7. On the upper side ofFIG. 10, a basic display position (also referred to as a display position before correction) of the AR image6with respect to the object position in a calculation plane (described as a virtual image plane8) of the HUD apparatus1corresponding to the virtual image region7and the like are shown. In the two-dimensional coordinate system (x, y) in the virtual image plane8, the AR image6(corresponding AR image data) is arranged for each object position of the target object4. Based on the object position and the viewpoint position E1, the AR image generation unit13determines the two-dimensional coordinates (xa, ya) of the basic display position of the AR image6with respect to the two-dimensional coordinates (xm, ym) of the object position on the virtual image plane8. The two-dimensional coordinates (xa, ya) of the basic display position of the AR image on the virtual image plane8are obtained by a predetermined calculation based on the three-dimensional coordinates of the object position and the viewpoint position and the distance between the two.

In this example, the case where the object4is another vehicle (oncoming vehicle) and an alert icon image for alerting is displayed to be superimposed as the AR image6is shown. An example of the alert icon image has an arch shape and is displayed so as to appear in parallel to the road surface. In this example, the center of the lower side of the object region of the object4is defined as the basic display position of the AR image, but the basic display position is not limited to this. Based on the basic image of the alert icon image, the AR image whose inclination and the like are adjusted in accordance with the road surface is generated. Then, the AR image is arranged at the basic display position in the virtual image plane8. Further, the color, size, and the like of the AR image may be adjusted at this time. For example, the AR image may be adjusted to have a color in accordance with the type of the object or a size in accordance with the size of the object region.

On the lower side ofFIG. 10, another example of the AR image is shown. In this example, the object4is another vehicle and the AR image6is a frame image surrounding the object4(object region). Further, in this example, an arrow image for navigation of the traveling direction is displayed as the AR image6at a forward position on the road. Also, when the object4is a predetermined facility or signboard, a word balloon image is displayed as the AR image6at a related position slightly apart from the object position in this example. In the word balloon image, information of the object4(for example, information of a facility or a signboard based on the reference to the car navigation unit106and the DB unit109) is displayed in characters or the like.

As in the example described above, various types of AR images are possible. Note that the correction function can be similarly applied even when the display position is slightly apart from the object position as in the word balloon image as a type of the AR image. It is also possible to set whether to apply the correction function to the word balloon image. When the word balloon image is corrected, the positional relationship between the object and the word balloon image is maintained before and after the correction. When the correction is performed, how it looks such as the distance between the object and the word balloon image changes in accordance with the viewpoint position. Although not shown, another example of the AR image may be a lane line image (image for highlighting the lines at the left and right ends of the road) or the like.

In the embodiment, as the correction function, the content of the correction of the display position of the AR image is controlled in accordance with the object position, the amount of movement of the viewpoint position, and the positional relationship between the virtual image region and the object. For this correction function, the following methods are used.

First, as shown inFIG. 1, when correcting the display position of the AR image by the correction function in the HUD apparatus1according to the embodiment, the process of correcting the content of the AR data (display data given to the display unit20) is performed in the controller10. The correction process can be realized by the conversion process using, for example, a conversion table based on the software program processing of the controller10.

Regarding the means for realizing this correction, as the HUD apparatus according to a modification, the control for changing the angles θ, ϕ, and the like of the mirror501of the optical system24of the display unit20in the hardware of the HUD apparatus may be applied. Alternatively, the control using the information processing of the AR data described above and the drive control of the optical system24in combination may also be applied.

FIG. 11shows a correction method of the display position of the AR image in the correction function of the HUD apparatus1according to the embodiment. In this correction method, in particular, the correction is controlled in accordance with the front-back positional relationship between the object4and the virtual image region7in the Z direction. In the correction function, the display position of the AR image is corrected by using at least one of the methods described below. In the HUD apparatus1, the correction function based on at least one of the methods is implemented in advance as a design matter. Alternatively, the HUD apparatus1may be configured so that the method selected by a user from the plurality of methods can be set as the method to be used by using a user setting function. Further, in the HUD apparatus1, the method to be used may be switched among a plurality of methods depending on the situation. The effect of correction in accordance with the used method can be realized.

(A) First method: First, as the basic correction method, in the first method, the correction is similarly applied regardless of the position of the object4with respect to the virtual image region7in the Z direction. The HUD apparatus1determines the display position after correction of the AR image6in the X direction in the virtual image region7in accordance with the object position and the amount of movement H of the viewpoint position.

(B) Second method: Further, in the second method and the like, whether or not to apply the correction is controlled in at least two sections of the case where the object4is located at the back of the virtual image region7(virtual image region position P1) in the Z direction and the case where the object4is located in front of the virtual image region7. As shown inFIG. 7, the case where the object4is located at the back of the virtual image region7is the case of the object4aat the position P3, in other words, the case where the object distance LA is larger than the virtual image distance LB (LA LB). The case where the object4is located in front of the virtual image region7is the case of the object4bat the position P2, in other words, the case where the object distance LA is smaller than the virtual image distance LB (LA<LB). In the second method, the HUD apparatus1corrects the display position of the AR image6when the object4is located in front of the virtual image region7as the positional relationship, and the HUD apparatus1does not correct the display position when the object4is located at the virtual image region position P1or at the back of the virtual image region position P1. In the example ofFIG. 11, the correction is performed for the object4blocated in front of the virtual image region7as viewed from the viewpoint position Ex after movement, and the AR image6bafter correction is shown. For the object4alocated at the back of the virtual image region7, the AR image6afrom the viewpoint position E1is not corrected, and the AR image6abefore correction is shown.

(C) Third method: As a modification of the second method, the third and following methods may be used. In the third method, a threshold position for control is set at a position802with a predetermined distance801in front of the virtual image region position P1in the Z direction. In the third method, the HUD apparatus1performs the correction when the object position is located in front of the position802, and does not perform the correction when the object position is located at the back of the position802. In the third method, as a concept, the correction is not performed when the object position is located between the virtual image region position P1and the position802because the object4and the virtual image region7are relatively close and the influence of the shift is relatively small.

(D) Fourth method: In the fourth method, a range803with a predetermined distance extending in front of and at the back of the virtual image region position P1is set as a virtual image region proximity range which is a control range. In the fourth method, three sections such as the range803and those located in front of and at the back of the range803are provided. In the fourth method, the HUD apparatus1does not perform the correction when the object position is located in the range803. When located in the range803, the correction is not performed because the object4and the virtual image region7are relatively close and the influence of the shift is relatively small. Further, in the fourth method, the correction is performed when the object position is outside the range803and in front of the range803, and the correction is not performed when the object position is outside the range803and at the back of the range803.

(E) Fifth method: As a modification of the fourth method, in the fifth method, the correction is not performed when the object position is within the range803, the correction is performed when the object position is outside the range803and in front of the range803, and the correction is performed when the object position is outside the range803and at the back of the range803.

Since the influence of the shift becomes larger as the distance from the object position to the virtual image region position P1(virtual image object distance FC) becomes larger, there is the higher necessity and effectiveness of the correction. In addition, since the influence of the shift becomes larger when the object is located in front of the virtual image region7, there is the higher necessity and effectiveness of the correction. In each of the above-described correction methods, at least when the object is located in front of the virtual image region7, the correction is reliably performed, so that the shift is eliminated or reduced.

FIG. 12shows a correction method in accordance with the amount of movement of the viewpoint position as a correction method additionally applied in the HUD apparatus1according to the embodiment. In this correction method, whether or not to apply the correction to the display position of the AR image is controlled in accordance with the magnitude of the amount of movement H of the viewpoint position in the X direction, that is, whether the amount of movement H is within a predetermined threshold value or exceeds the threshold value.

(A) ofFIG. 12shows the case where the amount of movement H is the amount of movement Hx1at the viewpoint position Ex1after movement to the right from the viewpoint position E1of the basic setting and the amount of movement Hx1is within a predetermined threshold value Ht. The case where the object4alocated at the position P3at the back of the virtual image region7and the object4blocated at the position P2in front of the virtual image region7are present is shown. In this case, since the amount of movement Hx1is within the threshold value Ht, the HUD apparatus1does not perform the correction using the above-described first method, second method, and the like. As a concept, in this case, the shift between the object4aor the object4band the AR image6abefore correction is within an acceptable range in the visual recognition from the viewpoint position Ex1. For this reason, the correction process is omitted to improve the efficiency.

Similarly, (B) ofFIG. 12shows the case where the amount of movement His the amount of movement Hx2(Hx2>Hx1) at the viewpoint position Ext after movement from the viewpoint position E1of the basic setting and the amount of movement Hx2exceeds the predetermined threshold value Ht. In this case, since the amount of movement Hx2exceeds the threshold value Ht, the HUD apparatus1performs the correction using the above-described first method, second method, and the like. As a concept, in this case, the shift between the object4aor the object4band the AR image6abefore correction is out of the acceptable range in the visual recognition from the viewpoint position Ext. For this reason, the correction process is performed to eliminate the shift.

At the virtual image region position P1, a position121indicates the display position of the AR image6abefore correction. When the object4bis located at the position P2in front of the virtual image region7and the correction is performed, a position122indicates the display position of the AR image after correction. Similarly, when the object4ais located at the position P3at the back of the virtual image region7and the correction is performed, a position123indicates the display position of the AR image after correction.

FIG. 13shows correction and conversion of the display position of the AR image by the correction unit14. In the conversion process, the correction unit14receives, as input values, the object position (for example, three-dimensional coordinates), the amount of movement from the viewpoint position E1of the basic setting, the virtual image region position P1, and the display position of the AR image before correction, and acquires the display position (or the amount of correction) of the AR image after correction as an output value as a result of the conversion.

The example ofFIG. 13shows the case where the object positions are the position P3at the back of the virtual image region position P1and the position P2in front of the virtual image region position P1, and are the positions displaced slightly to the left from the center in the X direction. Further, the case where the viewpoint position moves rightward from the viewpoint position E1of the basic setting to the viewpoint position Ex after movement with the amount of movement Hx is shown as the movement of the viewpoint position.

In the virtual image region7, the display position when displaying the AR image6aligned with the object4aat the position P3or the object4bat the position P2from the viewpoint position E1is indicated as a position QB of the AR image6B before correction. Although not shown, when an object is located at the position QB at the virtual image region position P1=PB, the display position of the AR image is at the same position QB.

In the case of the first method and the like, the case where the display position is corrected for the object4alocated at the back of the virtual image region7is as follows. The HUD apparatus1corrects the display position (position QB) of the AR image6B of the object4aat the position P3in accordance with the viewpoint position Ex after movement. By the conversion process, the display position after correction is determined as a position QA. The position QA is obtained as an intersection between a line of the virtual image region7in the X direction and a straight line for viewing the object4afrom the viewpoint position Ex. The HUD apparatus1displays the AR image6A at the position QA after correction.

In the case of the first method, the second method, and the like, the case where the display position is corrected for the object4blocated in front of the virtual image region7is as follows. The HUD apparatus1corrects the display position (position QB) of the AR image6B of the object4bat the position P2in accordance with the viewpoint position Ex after movement. By the conversion process, the display position after correction is determined as a position QC. The position QC is obtained as an intersection between the line of the virtual image region7in the X direction and a straight line for viewing the object4bfrom the viewpoint position Ex. The HUD apparatus1displays the AR image6C at the position QC after correction.

FIG. 14shows an example of the relationship between the position and the distance of the viewpoint, the object, and the virtual image. An example in which the object virtual image distance LC and the like are different will be described.FIG. 14shows an X-Z plane in which the road and the like are seen from above. For example, the virtual image region position P1(including the AR image display position) is set within a range of 20 to 30 m (setting range140) as the virtual image distance LB from the viewpoint position of the own vehicle. In this example, the case where the virtual image region position P1is set at a position with the virtual image distance FB of 25 m is shown.

The case where a person151, a cone152, a car153, and the like are the objects4to be the targets for AR is shown. The person151(pedestrian or the like) is located at a relatively close position in front of the virtual image region position P1with a distance of 5 m from the viewpoint position. The object virtual image distance LC is −20 m (the object virtual image distance LC in the case where the object is in front of the virtual image region position P1is defined as negative). The cone152is located at a position near the virtual image region position P1, that is, within the setting range140with a distance of 20 to 30 m from the viewpoint position. The car153(oncoming car or the like) is located at a relatively far position at the back of the virtual image region position P1with a distance of about 40 to 60 m (for example, 45 m) from the viewpoint position. The object virtual image distance LC is +20 m.

First, when the object4is located at the position near the virtual image region7(virtual image region position P1) within the setting range140like the cone152, the influence of the shift of the display position of the AR image6is small, and a suitable AR image display is possible.

The car153is located at the position at the back of the virtual image region position P1, and the object virtual image distance LC is +20 m. For the AR image to be superimposed on the position of the car153, there is the influence of the shift, but the influence is smaller as compared with the case of the object located in front of the virtual image region position P1like the person151. Therefore, the AR image of the car153may be corrected based on the first method and the like, or the correction may be omitted based on the second method and the like.

The person151is located at the position in front of the virtual image region position P1, and the object virtual image distance LC is −20 m. For the AR image superimposed on the position of the person151, the influence of the shift is larger as compared with the case where the object is located at the back of the virtual image region position P1like the car153. Therefore, for the AR image of the person151, it is important to eliminate the shift, and the correction is reliably performed.

For example, when the above-mentioned second method is applied, the correction process for an object such as the car153located at the back of the virtual image region7is omitted to improve efficiency, and the shift is eliminated by the correction for an object such as the person151located in front of the virtual image region7, thereby achieving the visual recognition suitable for the driver.

FIG. 15shows an example of the object and the AR image corresponding to the example ofFIG. 14. (A) on the upper side ofFIG. 15simply shows an example of the object4viewed from the viewpoint position E1of the basic setting. The person151is present as the object4on the road at a position relatively close to the own vehicle. In addition, the car153is present as the object4at a position relatively far from the own vehicle. Further, at an intermediate position between them, the cone152is present as the object4placed near the corner. As an object position154, the two-dimensional coordinates (x, y) of the object position in the case of the person151are shown, and the object position154is set at the center point of an object region155in this case. The object region155corresponds to the region surrounding the person151as the object4(for example, a rectangle including the shape of the person151), and is indicated by a broken line frame. The object region is not limited to a rectangle, and any shape can be adopted for the object region.

(B) on the lower side ofFIG. 15shows an example of the superimposed display of the AR image6on the object4of (A). The example in which an alert icon image is displayed to be superimposed as the AR image6on each of the person151, the cone152, and the car153is shown. For example, a display position156indicates the basic display position of the AR image6before correction in the case of the person151. The case where the display position of the AR image6for the object position154and the object region155is set to the center of the lower side of the object region155located below the object position154is shown, but the display position is not limited to this, and the object position may be directly used as the display position of the AR image.

FIG. 16shows the position shift between the object and the AR image and the correction corresponding to the examples ofFIG. 14andFIG. 15. The virtual image regions7before and after movement of the viewpoint position are simply shown. A virtual image region7A indicates the virtual image region viewed from the viewpoint position E1of the basic setting before movement. A virtual image region7B indicates the virtual image region viewed from the viewpoint position Ex after movement to right from the viewpoint position E1. The amount of movement H is, for example, 100 mm. In the state before correction, there is a position shift between the person151as the object4and the AR image6a(AR image before correction) as viewed from the viewpoint position Ex. In the state after correction, the position shift has been eliminated between the person151and the AR image6b(AR image after correction) as viewed from the viewpoint position Ex. By the correction, the display position156of the AR image6abefore correction is changed to the display position157of the AR image6bafter correction.

In particular, when the object4is located in front of the virtual image region7and the object virtual image distance LC is relatively large as in the example of the person151, the influence of the shift is large, and thus the correction of the display position is effective. Although the object virtual image distance LC is 20 m for both the person151and the car153, the influence of the shift is larger for the person151than for the car153. Therefore, the display position of the AR image is reliably corrected for the person151by using the second method and the like.

[Modification: Variable Setting of Virtual Image Region]

FIG. 17shows variable setting of the position of the virtual image region7in the Z direction in the HUD apparatus1according to a modification. The virtual image region position P1can be variably set within the predetermined setting range140(20 to 30 m). A position P1aindicates a position of a virtual image region7ain the case where the position with the virtual image distance FB of 30 m is set as the virtual image region position. A position P1bindicates a position of a virtual image region7bin the case where the position with the virtual image distance FB of 20 m is set as the virtual image region position. Also, the case where the object4is located at the positions (PA, PB, PC) with respective object distances is shown. The object4A at the position PA, the object4B at the position PB, and the object4C at the position PC are shown.

Even in the case of setting each virtual image region7, the correction function can be similarly applied. For example, it is assumed that the above-described second method is applied. In the case of the virtual image region7a, the correction is performed for the object4C and the object4B because they are located in front of the virtual image region7a, and the correction is not performed for the object4A because it is located at the back of the virtual image region7a. In the case of the virtual image region7b, the correction is performed for the object4C because it is located in front of the virtual image region7b, and the correction is not performed for the object4A and the object4B because they are located at the back of the virtual image region7b.

Note that the plane of the virtual image region7(for example, a rectangular plane) set based on the road surface, the vehicle, and the conditions of the driver may be an X-Y plane or a plane having the line of sight301from the viewpoint position E1as a normal line as in the example ofFIG. 3. The correction function can be similarly applied in each case.

FIG. 18shows a setting state of the virtual image region7according to another modification. In this manner, the virtual image region7is not limited to a plane perpendicular to the line of sight301from the viewpoint position E1, but may be a plane inclined obliquely. In this example, the virtual image region7is set as a plane having an angle δ with respect to the line of sight301in the Z direction. Further, the angle δ may be variably set. In this case, the virtual image region7has a width181in the Z direction with the virtual image region position P1in the Z direction and the position pB on the line of sight301as a center. The width181has a size within the setting range140. Also in this case, the correction function can be similarly applied. In this example, an AR image6y1at a position Qy1and an AR image6y2at a position Qy2are shown as the AR images6in the virtual image region7. The position Qy1is a position above the line of sight301in the Z direction and a position at the back of the virtual image region position P1. The position Qy2is a position below the line of sight301in the Z direction and a position in front of the virtual image region position P1.

In the case of this modification, for example, the fourth method of (D) inFIG. 11and the fifth method of (E) inFIG. 11can be similarly applied as the correction method. For example, whether or not to apply the correction may be controlled by making the width181correspond to the range803of (D).

[Control in Case of Outside of Virtual Image Region]

FIG. 19shows control in the case of outside of the virtual image region7as a part of the correction function. The control in the case where the display position of the AR image after correction goes out of the virtual image region7in accordance with the amount of movement of the viewpoint position will be described.

(1) As this control, in the first control, when the display position of the AR image6related to the object4is within the virtual image region7in consideration of the virtual image region7(corresponding virtual image plane) viewed from the viewpoint position, the AR image can be displayed and the display position of the AR image is corrected. Further, in this control, when the display position of the AR image6related to the object4(the display position calculated at the time of correction) is outside the virtual image region7in consideration of the virtual image region7(corresponding virtual image plane) viewed from the viewpoint position, the correction of the display position of the AR image is not performed, and the AR image is not displayed. An outside190of virtual image region (undisplayable region) shows an example of an outside region on a right side of the virtual image region7. A position191is a display position of the AR image6acalculated in accordance with the viewpoint position E1of the basic setting. A position192is a display position calculated for correction in accordance with the viewpoint position Ex after movement. The position192is located in the outside190of virtual image region. In the first control, in the case of the outside190of virtual image region, the AR image is not displayed and corrected simply. When the viewpoint position returns, that is, when the display position after correction returns to the inside of the virtual image region7, correction is performed and the AR image is displayed again.

(2) In the second control, when the HUD apparatus1determines that the viewpoint position views the inside of the virtual image region7or the display position after correction is inside the virtual image region7in the normal state, the HUD apparatus1automatically turns on the correction function and performs the correction and display. When the HUD apparatus1determines that the viewpoint position does not view the virtual image region7or the display position after correction is outside the virtual image region7, the HUD apparatus1automatically switches the correction function to the off state and does not perform the correction and display. In the latter case, it is judged that the driver cannot see the AR image in the virtual image region7or does not see the AR image intentionally, and the HUD apparatus1turns off the correction function. Depending on the situation, when the driver does not want to see the virtual image or wants to see only the real image, the driver may move the viewpoint position for that purpose. Even in such a case, the driver can easily see only the object without seeing the AR image by applying the above control.

(3) In the third control, when the display position of the AR image after correction goes out of the virtual image region7based on the viewpoint position, the HUD apparatus1does not perform correction and displays the AR image while maintaining the display position before correction. Namely, the AR image before correction remains displayed at the position near the left or right end of the virtual image region7. When returning to the inside of the virtual image region7, the necessity of correction is determined again. In the case of this control, the shift between the object and the AR image remains in the visual recognition by the driver, and the driver recognizes the shift. If the driver wants to see the object by changing the viewpoint, the driver can easily see the object because the superimposition of the AR image is displaced.

As described above, with the HUD apparatus1according to the embodiment, the shift between an object and a virtual image (AR image) can be reduced, and a suitable AR display can be realized. In particular, it is possible to realize suitable correction in accordance with the viewpoint position and the virtual image region position.

In the foregoing, the present invention has been specifically described based on the embodiment, but the present invention is not limited to the embodiment and various changes can be made within the scope of the present invention. The addition, deletion, separation, connection, replacement, and combination of the component of the embodiment are possible. A part or all of the function of the embodiment may be implemented by hardware such as an integrated circuit or by software program processing. Each software may be stored in the apparatus in advance at the time of the product shipment or may be acquired through communication from the external apparatus after the product shipment.

The correction by the correction function according to the embodiment can be similarly applied not only in the horizontal direction (X direction) but also in the vertical direction (Y direction). The present invention can be applied to various uses in addition to the in-vehicle system.

REFERENCE SIGNS LIST