Patent Description:
A display apparatus, such as a head-up display, superimposes a virtual image of an indicator image for indicating various kinds of information on the scenery in front of the vehicle by projecting the indicator image onto the front windshield of the vehicle, so that the virtual image is visible to the driver of the vehicle.

For example, PTL <NUM> discloses a technique for recognizing traffic signs such as road signs, road surface markings, and the like in front of the vehicle on the basis of an image of the forward view from the vehicle obtained by a camera attached to the vehicle and superimposing virtual images of traffic sign icon images at positions of the road that are visible in the forward view, so that the road to which the traffic signs are applied are displayed to the driver of the vehicle.

Furthermore, <CIT> discloses a method including acquiring an image by a camera of an assistance system. A traffic sign is determined within the captured image. An orientation of the particular traffic sign with respect to a vehicle is determined. A display device of the assistance system is actuated depending on the orientation. A position of the traffic sign is determined with respect to the vehicle. The orientation of the particular traffic sign is determined from a perspective distortion.

<CIT> discloses a driver assistance system having a camera configured to acquire traffic signs in front of the vehicle, a controller configured to identify the target path and the driving lanes in front of the vehicle and a head-up display configured to display the traffic sign icons superimposed on the road in front of the vehicle.

However, according to the technique of PTL <NUM>, in a case where a path such as a road to which a traffic sign is applied or the traffic sign is visually recognized in proximity to an edge portion and the like of the forward view, the display apparatus is required to project a wide-angle view in order to superimpose virtual images of predetermined images, such as traffic sign icon images, at positions of traffic signs in the forward view. As a result, the size and the cost of the display apparatus may increase.

It is an object of the present disclosure to be able to display a path to which a traffic sign is applied or display the traffic sign even when projecting a narrow-angle view.

According to a first aspect of the invention, a display apparatus for being mounted on a moving body and for displaying a virtual image of a projection image to a driver of the moving body is provided. The display apparatus includes an image processing apparatus configured to execute image processing on an image of a forward view from the moving body and to output position information, an identification unit configured to obtain a traffic sign icon image based on a sign image received from the image processing apparatus. The display apparatus further includes an image generation unit configured to generate the projection image including a predetermined image obtained based on a traffic sign, and a direction indication unit configured to output either an indication direction in which a target path, to which the traffic sign is applied, is present or an indication direction in which the traffic sign is present, based on a position of the traffic sign and a position of a path in front of a moving body, and to cause the image generation unit to generate the projection image including the indication direction.

The indication direction indicates a direction in which the target path or the traffic sign is present with respect to a reference position. The image generation unit is configured to generate the projection image of a predetermined image size with the reference position being adjusted with respect to a forward image such that a traffic sign icon image is arranged in the direction of the indication direction with respect to the reference position.

According to another aspect of the invention a display method is defined for causing a display apparatus to display a virtual image of a projection image to a driver of a moving body. The method comprises obtaining a predetermined image obtained based on a traffic sign;causing an image generation unit to generate the projection image including the predetermined image; and causing a direction indication unit to output either an indication direction in which a target path, to which the traffic sign is applied, is present or an indication direction in which the traffic sign is present, the indication direction being determined based on a position of the traffic sign and a position of a path in front of the moving body; and causing the image generation unit to generate the projection image including the indication direction, the indication direction indicating a direction in which the target path or the traffic sign is present with respect to a reference position, wherein the projection image of a predetermined image size is generated with the reference position being adjusted with respect to a forward image such that a traffic sign icon image is arranged in the direction of the indication direction with respect to the reference position.

A path to which a traffic sign is applied or the traffic sign can be displayed even when projecting a narrow-angle view.

Hereinafter, modes for carrying out the invention are explained with reference to drawings. In the drawings, similar constituent elements are denoted with the same reference numerals, and repeated description thereabout may be omitted.

A display apparatus according to an embodiment superimposes a virtual image of a projection image including various kinds of information on a front windshield and the like, so that the virtual image is visible to the driver of a moving body such as a vehicle.

In addition, the display apparatus according to the embodiment generates a projection image including a predetermined image such as a traffic sign icon image obtained based on a traffic sign, and displays, to the driver, an indication direction in which either a target path to which the traffic sign is applied or the traffic sign is present, on the basis of a position of the traffic sign and a position of a path in front of the moving body.

Because the display apparatus according to the embodiment displays the direction, it is not necessary to increase the angle of view of projection to superimpose a virtual image of a predetermined image such as a traffic sign icon image at a position of the target path that are visible in a forward view. Therefore, even when the angle of view of projection of the display apparatus is small, either the target path or the traffic sign can be displayed.

An image includes both of a still image and a moving image. The image is assumed to include a video.

The forward view refers to the scenery in front of a moving body that is visible to the driver of the moving body. For example, the scenery in front of the moving body seen by the driver through the front windshield of the moving body corresponds to the scenery in front of the vehicle.

The driver refers to the person who drives or operates the moving body. For example, the driver is a driver who drives the vehicle.

The traffic sign is a sign that provides information with regard to traffic to a person who uses the road such as the driver of a vehicle. The traffic sign includes a display member for displaying information, an image for displaying information, a marking for displaying information, or the like. Specifically, traffic signs may include road signs, road surface markings, or the like. The road sign may be a signboard disposed in proximity to or above a road to provide information with regard to traffic to a person who uses the road such as a driver of a vehicle. The road surface marking is a marking on the road surface for providing information for the purpose of control, guidance, or regulation of traffic. The traffic sign includes a figure, a symbol, characters and the like defined in advance according to information to be provided.

The moving body refers to an object that can move. Examples of moving bodies include: land moving bodies capable of moving on the land, such as an automobile, a vehicle, a train, a locomotive, and a forklift; air moving bodies capable of moving in the air, such as an airplane, a balloon, and a drone; and water moving bodies capable of moving on the water, such as a ship, a vessel, a steamship, and a boat.

The path refers to a road, a route, a trace, or a direction in which a moving body moves. Examples of paths include traffic routes, traveling directions, cruising directions, and the like, and include roads, aisles, passages, roadways, and railroad tracks, on which the land moving bodies move; an airway and an air route along which the air moving bodies move; and a waterway, a sea route or a watercourse in which the water moving bodies move.

The traffic sign icon image refers to an image of a virtual traffic sign. In other words, the traffic sign icon image is an image for displaying either a figure corresponding to a figure displayed by a traffic sign or characters corresponding to characters displayed by a traffic sign, or displaying both the figure and the characters. For example, if the traffic sign is a "Do Not Enter" sign, an image of the same pattern as that displayed by the "Do Not Enter" sign corresponds to the traffic sign icon image.

The projection image refers to a projected image that is visible to the driver. The projection image includes traveling speed information about the vehicle, information to be provided to the driver such as traffic sign icon images, and the like.

The predetermined image refers to an image included in the projection image. For example, the predetermined image is a character image and the like including characters substituted for the traffic sign icon image and the traffic sign.

The predetermined position refers to a position at which the predetermined image is arranged in the projection image in a case where, as a result of movement of the moving body, the traffic sign moves out of the forward view and information about the traffic sign is no longer obtained.

The target path refers to the path to which information indicated by a traffic sign is applied. For example, if a traffic sign indicates that the speed limit of the road is <NUM>/h, i.e., a path along which an automobile moves, the target path is the road to which a speed limit of <NUM>/h is applied. In this case, the road is referred to as the target road. The target road is an example of a target path.

The indication direction refers to the direction in which either the target path or the traffic sign is present as seen from the driver.

Hereinafter, an embodiment is described with reference to a head-up display (HUD) mounted on a vehicle capable of moving on a road. In this case, the road is an example of a path, the vehicle is an example of a moving body, and the HUD is an example of a display apparatus.

<FIG> is a drawing illustrating an example of a configuration in which a display apparatus <NUM> according a first embodiment is mounted on a vehicle <NUM>. As illustrated in <FIG>, the display apparatus <NUM> includes a control unit <NUM> and a projection unit <NUM>.

The display apparatus <NUM> is embedded in the dashboard of the vehicle <NUM>. The display apparatus <NUM> emits an image toward a front windshield <NUM> through an emission window <NUM> on the upper surface of the display apparatus <NUM>. The projection image is displayed as a virtual image I in a positive Y direction with respect to the front windshield <NUM> and is visible to the driver V of the vehicle <NUM>. By seeing the virtual image I, the driver V can obtain information useful for driving with less eye movement while the driver V keeps his or her eyes on other vehicles, the road surface, and the like located in positive the Y direction with respect to the vehicle <NUM>.

The display apparatus <NUM> may be installed on the ceiling or a sun visor in addition to the dashboard, as long as the image can be projected on the windshield <NUM>.

As illustrated in <FIG>, the display apparatus <NUM> includes, as main constituent elements, a projection unit <NUM> and a control unit <NUM>. The projection unit <NUM> projects an image toward the front windshield <NUM>. A laser scanning method and a panel method are known as image projection methods for the projection unit <NUM>. The laser scanning method is a method for scanning a laser beam emitted from a laser light source with a two-dimensional scanning device to form an intermediate image (a real image projected on a screen described later). The panel method is a method for forming an intermediate image with an imaging device such as a liquid crystal panel, a Digital Micro-mirror Device (DMD) panel, or a vacuum fluorescent display (VFD). The projection image includes both a projection image projected by the laser scanning method and a projection image projected by the panel method.

Unlike the panel method for forming an image by causing the entire screen to emit light and partially shielding light, the laser scanning method can assign emission and non-emission to each pixel, and therefore, the laser scanning method is generally advantageous in capable of forming an image with a high contrast. It has been found that an HUD of the laser scanning method allows passengers of the vehicle to see information with less attention resources than an HUD of the panel method because the HUD of the laser scanning method improves the visibility by attaining the high contrast.

In particular, with the panel method, light that cannot be completely shielded is projected even on an area that does not have information, and a display frame is projected within the range in which the HUD can display images (this phenomenon is referred to as a postcard). With the laser scanning method, this phenomenon is not observed, and only the content can be projected. In particular, in augmented reality (AR) in which generated images are displayed on top of an existing landscape, the reality is improved.

The augmented reality is a technique for virtually augmenting the world in front of a person by displaying images of non-existent objects on top of existing landscapes. For the augmented reality, an HUD of the panel method may be adopted as long as the HUD can display information in a manner that can be visually recognized with less attention resources (giving less fatigue).

<FIG> is a drawing illustrating an example of configuration of the projection unit <NUM>. The projection unit <NUM> mainly includes a light source unit <NUM>, an optical deflector <NUM>, a mirror <NUM>, a screen <NUM>, and a concave mirror <NUM>. Note that <FIG> only illustrates main constituent elements, and constituent elements other than those illustrated in <FIG> may be provided, or some of the constituent elements illustrated in <FIG> do not have to be provided.

The light source unit <NUM> includes three laser light sources (hereinafter assumed to be laser diodes (LDs)) corresponding to Red(R), Green(G), and Blue(B), a coupling lens, an aperture, a combining element, lenses, and the like. The light source unit <NUM> is configured to combine the laser beams emitted from the three LDs and guides the laser beams toward the reflection surface of the optical deflector <NUM>. The laser beam guided to the reflecting surface of the optical deflector <NUM> is deflected two-dimensionally by the optical deflector <NUM>.

As the optical deflector <NUM>, for example, a micromirror that oscillates with respect to two orthogonal axes, or two micromirrors that each oscillate or rotate with respect to one axis can be used. The optical deflector <NUM> can be, for example, a MEMS (Micro Electro Mechanical Systems) mirror manufactured by a semiconductor process or the like. The optical deflector <NUM> can be driven by, for example, an actuator that uses the deformation force of the piezoelectric element as a driving force. As the optical deflector <NUM>, a galvanometer mirror or a polygon mirror may be used.

The laser beam two-dimensionally deflected by the optical deflector <NUM> enters the mirror <NUM> to be reflected by the mirror <NUM>, and renders a two-dimensional image (intermediate image) on the surface (scanned surface) of the screen <NUM>. For example, a concave mirror can be used as the mirror <NUM>, but a convex mirror or a plane mirror may be used. By deflecting the direction of the laser beam with the optical deflector <NUM> and the mirror <NUM>, the projection unit <NUM> can be downsized or the arrangement of the constituent elements can be flexibly changed.

The screen <NUM> is preferably a microlens array or a micromirror array that has the function of diverging the laser beam at a desired divergence angle, but may also be a laser diffusing plate for diffusing the beam, a transmission plate or a reflecting plate with smooth surface, and the like. Generally, the constituent elements from the light source unit <NUM> to the screen <NUM> are collectively referred to as an HUD apparatus. However, the HUD apparatus may include other components.

The laser beam emitted from the screen <NUM> is reflected by the concave mirror <NUM> and projected onto the front windshield <NUM>. The concave mirror <NUM> has a function similar to that of a lens, and forms an image at a predetermined focal length. Therefore, if the concave mirror <NUM> were be a lens, the image on the screen <NUM> corresponding to the object would form an image at a position of a length R<NUM> according to the focal length of the concave mirror <NUM>. Therefore, when viewed from an occupant such as the driver V of the vehicle <NUM>, the virtual image I is displayed at the position of the length R<NUM>+R<NUM> from the front windshield <NUM>. Where the length from a driver V of the vehicle <NUM> to the front windshield <NUM> is denoted as R3, the virtual image I is displayed (formed) at the position of a length R (=R<NUM>+R<NUM>+R<NUM>) from a viewpoint E of the driver V of the vehicle <NUM> in <FIG>.

At least a part of the light flux to the front windshield <NUM> is reflected toward the viewpoint E of the driver V of the vehicle <NUM>. As a result, the driver V of the vehicle <NUM> can visually recognize, through the front windshield <NUM>, the virtual image I acquired by enlarging the intermediate image on the screen <NUM>. That is, the virtual image I which is an enlarged intermediate image is displayed through the front windshield <NUM> as viewed from the driver V of the vehicle <NUM>.

Normally, the front windshield <NUM> is not flat but is slightly curved. For this reason, a length R is substantially determined by the length R<NUM>+R<NUM> as described above, although the imaging position of the virtual image I is affected not only by the focal length of the concave mirror <NUM> but also by the curved surface of the front windshield <NUM>. In order to form the virtual image I far away so as to reduce the eye movement, the length R<NUM> or R<NUM> is increased. One way to increase the length R<NUM> is to use a mirror to turn the optical path, and another way to increase the length R<NUM> is to adjust the focal length of the concave mirror <NUM>.

It should be noted that due to the front windshield <NUM>, an optical distortion occurs in which the horizontal line of the intermediate image has an upward or downward convex shape. Therefore, it is preferable that at least one of the mirror <NUM> and the concave mirror <NUM> is designed and arranged so as to correct the distortion. Alternatively, it is preferable that the projection image is corrected in consideration of the distortion.

Further, a combiner may be arranged as a transmission and reflection member at the side of the viewpoint E with respect to the front windshield <NUM>. Even where the light from the concave mirror <NUM> is emitted to the combiner, information can be displayed as a virtual image I in a manner similar to the case where the light from the concave mirror <NUM> is emitted onto the front windshield <NUM>.

Next, the in-vehicle system <NUM> in which the display apparatus <NUM> is mounted on the vehicle <NUM> is explained with reference to <FIG> is a block diagram illustrating an example of configuration of the in-vehicle system <NUM> in which the display apparatus <NUM> is mounted on the vehicle <NUM>.

The in-vehicle system <NUM> includes a car navigation system <NUM> that communicates via an in-vehicle network NW such as a CAN (Controller Area Network) bus, an engine ECU (Electronic Control Unit) <NUM>, a display apparatus <NUM>, a brake ECU <NUM>, a steering ECU <NUM>, an image processing apparatus <NUM>, and a detection apparatus <NUM>.

The car navigation system <NUM> has a Global Navigation Satellite System (GNSS) such as a GPS (Global Positioning System), detects the current location of the vehicle <NUM>, and displays the position of the vehicle <NUM> on an electronic map.

The car navigation system <NUM> also receives inputs of the starting point and the destination, searches a route from the starting point to the destination, displays the route on the electronic map, and provides guidance about the direction of travel to the occupant of the vehicle <NUM> by voice, text, animation (displayed on the display), or the like, before making a turn. The car navigation system <NUM> may communicate with a server via a cell network or the like. In this case, the server can send the electronic map to the vehicle <NUM> or perform a route search.

The engine ECU <NUM> determines an ideal fuel injection amount, advances or retards the ignition timing, controls the valve actuation mechanism, and the like according to the information from the sensors and the vehicle situations. Also, the engine ECU <NUM> determines whether to shift the gear by, e.g., referring to a map defining a shifting line for a gear with respect to a relationship between the current vehicle speed and the throttle opening. The engine ECU <NUM> performs the combinations of the above to perform acceleration and deceleration control for adaptive cruise control to follow a preceding ahead. Together with the engine or without using the engine as the power source, an electric motor may be used as a power source.

The brake ECU <NUM> controls the braking force for each of the wheels of the vehicle even when the brake pedal is not operated by the driver V of the vehicle <NUM>. Examples of such controls performed by the brake ECU <NUM> include ABS (Antilock Braking System) control, braking control in adaptive cruise control for following a preceding ahead, pre-crash braking based on TTC (Time To Collision) with an obstacle, hill-start assist control for holding the brake when the vehicle starts on a slope, and the like.

The steering ECU <NUM> detects the steering direction and the steering amount of the steering wheel operated by the driver V of the vehicle <NUM>, and performs power steering control for adding a steering torque in the steering direction. Even when the driver V of the vehicle <NUM> does not perform steering, the steering ECU <NUM> performs steering in a direction to avoid departure from a travelling lane, in a direction to maintain travelling at the center of the travelling lane, or in a direction to avoid approaching an obstacle.

The detection apparatus <NUM> is equipped with various sensors such as a forward camera <NUM>, and can perform various detections such as capturing forward images of the vehicle <NUM>. In this case, the position and the inclination of the forward camera <NUM> is adjusted and attached to the vehicle <NUM> so that the forward image corresponds to the forward view seen through the front windshield <NUM> by the driver V.

An image processing apparatus <NUM> executes image processing on the image of the forward view from the vehicle <NUM> captured by the forward camera <NUM>, obtains information used for generating the projection image, and outputs the information to the display apparatus <NUM>.

Next, the configuration of the detection apparatus <NUM> is explained with reference to <FIG> is a block diagram illustrating an example of configuration of the detection apparatus <NUM>. The detection apparatus <NUM> includes a vehicle speed sensor <NUM> for detecting the vehicle speed to be displayed by the display apparatus <NUM>, a vehicle information sensor <NUM> configured to acquire vehicle information to be displayed by the display apparatus <NUM>, a radar sensor <NUM> and a forward camera <NUM> for detecting obstacles, an occupant state information sensor <NUM> for acquiring occupant state information which is information about occupants, a VICS (Vehicle Information and Communication System Center, registered trademark) reception apparatus <NUM> for receiving traffic jam information from outside, and an external communication apparatus <NUM> for connection with the Internet and the like.

It should be noted that the sensors included in the detection apparatus <NUM> do not have to be provided in a concentrated manner in the detection apparatus <NUM>, but may be provided in any manner as long as these sensors are mounted on the vehicle.

The vehicle speed sensor <NUM> detects a magnet that rotates with the rotation of the shaft of the drive train system with a sensor unit fixed to the vehicle body, and generates a pulse wave proportional to the rotation speed. The vehicle speed can be detected by the number of pulses per unit time.

The vehicle information sensor <NUM> has one or more sensors that detect vehicle information other than the vehicle speed sensor <NUM>. Examples of such sensors include a fuel gauge sensor, a shift lever position sensor, an odometer, a trip meter, a turn signal sensor, a water temperature sensor, and the like. These sensors may acquire pieces of vehicle information with a generally-available structure. The fuel gauge sensor detects the amount of fuel remaining.

The shift lever position sensor detects the shift lever position operated by the driver V of the vehicle <NUM>. The odometer accumulates the travelling distances of the vehicle <NUM>, and provides the total travelling distance. The trip meter provides the distance traveled since the driver V of the vehicle <NUM> last initialized the trip meter.

The turn signal sensor detects the direction of the turn signal operated by the driver V of the vehicle <NUM>. The water temperature sensor detects the temperature of the engine coolant. These are merely examples of information that can be acquired from the vehicle <NUM>, and other information that can be acquired from the vehicle can be adopted as the vehicle information. For example, in an electric vehicle or a hybrid vehicle, the remaining amount of battery, the amount of regenerative power, or the amount of power consumption can also be acquired.

The forward camera <NUM> is an image-capturing device that captures images in front of the vehicle <NUM> (i.e., a direction corresponding to the positive Y direction in <FIG>). The forward camera <NUM> is attached to the back of a rearview mirror or a portion in proximity thereto.

The forward camera <NUM> may be a monocular camera or a stereo camera. In the case of a monocular camera or a stereo camera that can acquire distance information, the radar sensor <NUM> may be omitted. However, when the radar sensor <NUM> is provided in addition to the forward camera <NUM> that can acquire distance information, the distance information acquired by the forward camera <NUM> and the distance information acquired by the radar sensor <NUM> may be combined (integrated), and highly accurate distance information can be acquired by complementing the disadvantages of each other. In addition to the radar sensor <NUM> and the forward camera <NUM>, a sonic sensor (ultrasonic sensor) may be provided.

The radar sensor <NUM> transmits radio waves to the surroundings of the vehicle <NUM>, such as the front, sides, and rear of the vehicle <NUM>, and receives the radio waves reflected and returned by the object. The radar sensor <NUM> may be arranged at such locations that the radar sensor <NUM> can detect obstacles around the vehicle <NUM>. The radar sensor <NUM> may use a TOF (Time of Flight) method to detect the distance to an object according to the time from transmission to reception, and detect the direction of the object according to the emission direction of the radar.

A LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) device is known as a radar sensor of the TOF method. In addition, the FMCW method (Frequency Modulation Continuous Wave) generates a mixed wave of the reception wave and the transmission wave while continuously increasing the frequency of the transmission wave, and converts the beat frequency of the mixed wave generated from slightly different frequencies into a distance. In the FMCW method, the direction of the object is estimated by detecting the phase shift of the reception waves with multiple reception antennas.

The occupant state information sensor <NUM> is a sensor that detects the occupant state information detected directly or indirectly from the occupant of the vehicle <NUM>. A typical example of the occupant state information sensor <NUM> is a face camera. The face camera captures images of the face of the occupant of the vehicle <NUM> and performs face authentication to identify or distinguish the occupant of the vehicle <NUM>. The face direction and the gaze direction can be detected from the face image.

In addition, the occupant state information sensor <NUM> may be, for example, an electrocardiograph sensor, a heart rate sensor, a blood pressure sensor, a body temperature sensor, a pulse sensor, a respiratory sensor, a sweat sensor, a blink sensor, a pupil sensor, a brain wave sensor, or a myoelectric potential sensor. The occupant state information sensor <NUM> may be, for example, in such a form that the occupant of the vehicle wears a wristwatch-type wearable terminal (a smart watch).

The VICS reception apparatus <NUM> receives radio waves distributed by VICS. The VICS is a system that transmits road traffic information about, e.g., traffic congestion and traffic regulations, to the in-vehicle apparatus in real time using FM multiplex broadcasting and beacons. The external communication apparatus <NUM> is connected to the Internet via a network such as <NUM>, <NUM>, <NUM>, LTE, or wireless LAN, to receive various kinds of information. For example, the external communication apparatus <NUM> can receive weather information such as rainfall, snowfall, and fog.

Also, the external communication apparatus <NUM> may receive news, music, video, and like. Further, the external communication apparatus <NUM> can acquire, for example, the state information about a traffic light and the time it takes for the traffic light to change. As described above, the VICS reception apparatus <NUM> and the external communication apparatus <NUM> may perform road-to-vehicle communication. In addition, the external communication apparatus <NUM> may acquire information detected by other vehicles <NUM> through inter-vehicle communication.

The advanced driver-assistance systems (ADAS) not only displays and warns the information, but also controls the vehicle <NUM>. In this case, the ECU performs various driving support in cooperation with the engine ECU <NUM>, the brake ECU <NUM>, and the steering ECU <NUM> based on the distance information about an obstacle detected by at least one of the radar sensor <NUM> and the forward camera <NUM>. For example, the ECU performs acceleration and braking control in adaptive cruise control for following a preceding ahead, pre-crash braking, avoidance of departure from a travelling lane, lane keep cruising, and hill-start assist control for holding the brake when the vehicle starts on a slope, steering for avoiding an obstacle, and the like. For such control, the ECU recognizes road signs and road paints such as white lines from an image captured by the forward camera <NUM>.

In acceleration and deceleration control during adaptive cruise control, the ECU controls the driving power and the braking force to maintain the target distance according to vehicle speed. In the pre-crash braking, depending on the TTC, a warning and a display prompting the driver to press the brake pedal are displayed, and, if the possibility of collision is high, the seat belt is tightened and collision avoidance braking is activated. In avoidance of departure from a travelling lane, the driving assistance ECU <NUM> recognizes white lines (i.e., separation lines between neighboring lanes) from a captured image, and adds a steering torque in a direction opposite to the direction of departing from a travelling lane.

In lane keeping cruising, the center of a travelling lane is set as a target travelling line, and a steering torque proportional to a departure from the target travelling line is applied in the direction opposite to the departure. In the obstacle avoidance steering, in a case where it is determined that a collision cannot be avoided by braking, a travelling line for avoiding an obstacle is determined, and a steering torque for travelling along the travelling line is applied.

Also, for example, in a case where, when it is attempted to make a lane change, the radar sensor <NUM> or the forward camera <NUM> detects a vehicle cruising in an area (i.e., a blind spot) of a neighboring lane that cannot be seen with the door mirror, a warning is given to the occupant. Such a support is referred to as a blind spot monitor.

Next, a functional configuration of the image processing apparatus <NUM> is explained. The image processing apparatus <NUM> executes processing to calculate and obtain position information about the road, traffic signs, and the like in front of the vehicle <NUM> and to output the position information to the display apparatus <NUM>.

<FIG> is a block diagram illustrating an example of a functional configuration of the image processing apparatus <NUM>. As illustrated in <FIG>, the image processing apparatus <NUM> includes a sign area extraction unit <NUM>, a road area extraction unit <NUM>, a sign position obtaining unit <NUM>, and a road position obtaining unit <NUM>.

The sign area extraction unit <NUM> extracts, as a sign image, an image area including a traffic sign from the image of the forward view from the vehicle <NUM>, and outputs the sign image to the sign position obtaining unit <NUM>. The road area extraction unit <NUM> extracts, as a road image, an image area including a road from the image of the forward view from the vehicle <NUM>, and outputs the road image to the road position obtaining unit <NUM>. Image processing, such as pattern matching and edge detection, can be applied to the extraction of the sign image and the road image. When multiple roads and multiple signs are included in the forward image, a sign image is extracted for each of the multiple signs, and a road image is extracted for each of the multiple roads. Note that the sign image contains coordinate information that represents the position of the sign image in the forward image, and the road image contains coordinate information that represents the position of the road image in the forward image.

One of the methods by which the sign area extraction unit <NUM> and the road area extraction unit <NUM> extracts image areas is an image recognition method using machine learning. The machine learning is a technology that allows a computer to acquire learning ability like a human, and allows the computer to autonomously generate an algorithm used for determination, such as data identification, from learning data given in advance and to apply the learned algorithm to perform prediction with regard to new data.

The learning method for machine learning may be any one of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, and may be a learning method acquired by combining these learning methods, and the learning method for machine learning is not limited. The methods of machine learning include a perceptron, deep learning, support vector machine, logistic regression, naive Bayes, decision tree, and random forest.

The sign position obtaining unit <NUM> obtains, as coordinate information, the position information about the traffic sign from the image of the forward view from the vehicle <NUM> on the basis of the sign image. The road position obtaining unit <NUM> obtains, as coordinate information, the position information about the road from the image of the forward view from the vehicle <NUM> on the basis of the road image. When multiple roads and multiple signs are included in the forward image, position information is obtained for each of the multiple roads and the multiple signs.

The image processing apparatus <NUM> outputs, to the display apparatus <NUM>, the sign image extracted by the sign area extraction unit <NUM>, the position information about the traffic sign obtained by the sign position obtaining unit <NUM>, and the position information about the road obtained by the road position obtaining unit <NUM>.

When any traffic sign is not included in the forward image, the image processing apparatus <NUM> outputs sign-absence information indicating the absence of a traffic sign, such as coordinates (<NUM>, <NUM>), as information of the sign position. Also, the image processing apparatus <NUM> may output the road-absence information when the road is no longer included in the forward image.

Next, a hardware configuration of the control unit <NUM> is explained with reference to <FIG> is a drawing illustrating an example of a hardware configuration of the control unit <NUM>. As illustrated in <FIG>, the control unit <NUM> includes a field-programmable gate array (FPGA) <NUM>, a central processing unit (CPU) <NUM>, a read only memory (ROM) <NUM>, a random access memory (RAM) <NUM>, an interface (I/F) <NUM>, a bus line <NUM>, a laser diode (LD) driver <NUM>, an MEMS controller <NUM>, and a hard disk drive (HDD) <NUM>. The FPGA <NUM>, the CPU <NUM>, the ROM <NUM>, the RAM <NUM>, the interface <NUM>, and the HDD <NUM> are electrically connected with each other via the bus line <NUM>.

The CPU <NUM> controls each of the functions of the control unit <NUM>. The ROM <NUM> stores a program 203p that the CPU <NUM> executes to control the functions of the control unit <NUM>. The program 203p is stored in the RAM <NUM>, and the CPU <NUM> uses the RAM <NUM> as a work area for executing the program 203p. The RAM <NUM> has an image memory <NUM>.

The image memory <NUM> is used to generate an image projected as the virtual image I. The interface <NUM> is an interface for communicating with the image processing apparatus <NUM> and the detection apparatus <NUM>, and is connected to a CAN (Controller Area Network) bus, an Ethernet (registered trademark), or the like of the vehicle <NUM>, for example.

The HDD <NUM> is non-volatile memory of a large capacity that stores programs with which processing is executed by the control unit <NUM> and various data such as priority levels. Alternatively, the HDD <NUM> may be an SSD (Solid State Drive) or the like.

The FPGA <NUM> controls the LD driver <NUM> based on the image generated by the CPU <NUM>. The LD driver <NUM> controls the light emission of the LD according to the image by driving the LD of the light source unit <NUM> of the projection unit <NUM>. The FPGA <NUM> operates the optical deflector <NUM> of the projection unit <NUM> via the MEMS controller <NUM> so as to deflect the laser beam in the direction corresponding to the pixel position of the image.

Next, the functional configuration of the control unit <NUM> according to the first embodiment is explained with reference to <FIG> is a block diagram illustrating an example of a functional configuration of the control unit <NUM>. As illustrated in <FIG>, the control unit <NUM> includes an image storage unit <NUM>, an identification unit <NUM>, a direction target determination unit <NUM>, a direction table storage unit <NUM>, a direction indication unit <NUM>, an image generation unit <NUM>, and an output unit <NUM>.

Among the above units of the control unit <NUM>, the functions of the identification unit <NUM>, the direction target determination unit <NUM>, the direction indication unit <NUM>, and the image generation unit <NUM> are implemented by causing the CPU <NUM> to execute a predetermined program. The functions of the image storage unit <NUM> and the direction table storage unit <NUM> are implemented by the HDD <NUM> and the like. The functions of the output unit <NUM> are implemented by the interface <NUM> and the like.

The image storage unit <NUM> is a database for storing traffic sign icon images corresponding to traffic signs. In addition, to each of the stored images of traffic signs, identification information for identifying the traffic sign is assigned. The traffic sign icon image may be an image of a small size, as long as, when the traffic sign icon image is placed in the projection image and the driver V sees the virtual image of the projection image, the driver V can visually recognize the traffic sign icon image.

The image storage unit <NUM> preferably stores images of all the traffic signs used in the road traffic at that time, but the embodiment is not limited thereto, and the image storage unit <NUM> may store only some of the traffic signs such as frequently used traffic signs. In addition, it is preferable that the stored images are updated on a regular basis in order to reflect the changes of the existing traffic signs and to reflect the addition of new traffic signs.

The stored images may be updated by downloading the latest data from the Internet or the like via the external communication apparatus <NUM>. The update may be performed automatically on a regular basis, or the driver V of the vehicle <NUM> may operate the in-vehicle system <NUM> to manually update the stored images.

The identification unit <NUM> obtains a traffic sign icon image and identification information corresponding to a traffic sign indicated by a sign image by referring to the image storage unit <NUM> based on the sign image received from the image processing apparatus <NUM>. As an example, a received sign image and an image of each of the traffic signs stored in advance in the image storage unit <NUM> are compared by pattern match processing, and an image with a similarity equal to or more than a predetermined threshold value is obtained as a traffic sign icon image. Also, the identification information attached to the obtained traffic sign icon image can be obtained.

The identification unit <NUM> outputs the obtained traffic sign icon image to the image generation unit <NUM>, and outputs the obtained identification information to the direction target determination unit <NUM>.

The direction target determination unit <NUM> determines, as an indication direction displayed to the driver V, either a direction in which a target road is present or a direction in which a traffic sign is present, on the basis of the identification information of the traffic sign received from the identification unit <NUM>. In other words, either the target road or the traffic sign is determined as the target type of the indication direction. Then, the direction target determination unit <NUM> outputs information indicating the target type of the indication direction to the direction indication unit <NUM>.

Specifically, the direction target determination unit <NUM> makes the above determination by referring to the direction table storage unit <NUM> storing a correspondence table indicating a correspondence between the identification information and the indication directions defined in advance, on the basis of the identification information of the traffic signs received from the identification unit <NUM>.

For example, in cases of regulatory traffic signs indicating regulations on roads such as "Do Not Enter", "Stop", and the like, the target road is the target type of the indication direction. In cases of auxiliary traffic signs for calling for attention according to an ambient environment of a road such as a presence of a school in close proximity, a pedestrian crossing in close proximity, and the like, and complementing traffic regulations, the traffic sign is the target type of the indication direction. In a case of a traffic sign of a speed limit and the like, any of the target road and the traffic sign may be adopted as the target type of the indication direction, and a rule for adopting which of the target road and the traffic sign as the target road or the traffic sign is preferably defined in advance according to a setting configured by the manufacturer of the display apparatus <NUM>, the manufacturer of the vehicle <NUM>, the user of the display apparatus <NUM>, or the like.

The direction indication unit <NUM> displays, to the driver V, an indication direction, i.e. either a direction in which the target road, to which the traffic sign is applied, is present or a direction in which the traffic sign is present, on the basis of the position of the traffic sign and the position of the road in front of the vehicle <NUM>.

Specifically, the direction indication unit <NUM> receives the position information about each of the traffic sign and the road from the image processing apparatus <NUM>, and receives information about the target type of the indication direction from the direction target determination unit <NUM>.

In a case where the target type of the indication direction is the target road, the direction indication unit <NUM> determines, as the target road, the road to which the traffic sign is applied, on the basis of the position of the traffic sign and the position of the road in front of the vehicle <NUM>. Then, the direction indication unit <NUM> uses a reference position defined in advance and the coordinate information of the target road to calculate and obtain, as an indication direction θ, a direction in which the target road is present with respect to the reference position.

In contrast, in a case where the target type of the indication direction is the traffic sign, the direction indication unit <NUM> uses the coordinate information about the reference position and the traffic sign to calculate and obtain, as an indication direction θ, a direction of the traffic sign with respect to the reference position.

In this case, the viewpoint (i.e., the eye) of the driver V or a position corresponding to the viewpoint can be used as the reference position. In this case, the direction in which the target road or the traffic sign is present as seen from the driver V corresponds to the indication direction.

The direction indication unit <NUM> outputs the indication direction θ to the image generation unit <NUM> to cause the image generation unit <NUM> to generate a projection image including the indication direction θ, so that the indication direction θ is displayed to the driver V.

The image generation unit <NUM> generates a projection image including the traffic sign icon image arranged therein. Specifically, the image generation unit <NUM> generates the projection image of a predetermined image size with the reference position being adjusted with respect to the forward image such that a traffic sign icon image is arranged in the direction of the indication direction θ with respect to the reference position. The predetermined image size is preferably such an image size that the size of the virtual image of the projection image is smaller than the forward view so as to reduce the angle of view of projection to the front windshield <NUM> by the projection unit <NUM>.

The arrangement of the traffic sign icon image in the projection image is executed by replacing an image area in the direction of the indication direction θ with respect to the reference position with the traffic sign icon image. The image generation unit <NUM> can arrange other information such as a vehicle speed indication in the projection image. Further, in a case where any traffic sign is not included in the forward image, the image generation unit <NUM> generates a projection image not including any traffic sign icon image.

The output unit <NUM> outputs image data of the projection image generated by the image generation unit <NUM> to the projection unit <NUM>. Specifically, the output unit <NUM> controls the LD driver <NUM> and the MEMS controller <NUM> to cause the projection unit <NUM> to project a projection image generated by the image generation unit <NUM> to display the projection image as the virtual image.

Next, a flow of processing by the control unit <NUM> is explained with reference to <FIG> is a flowchart illustrating an example of processing by the control unit <NUM>.

First, in step S81, the identification unit <NUM> receives a sign image from the image processing apparatus <NUM>.

Subsequently, in step S82, the identification unit <NUM> obtains the traffic sign icon image and the identification information corresponding to the traffic sign indicated by the sign image by referring to the image storage unit <NUM> on the basis of the sign image of the traffic sign. Then, the identification unit <NUM> outputs the obtained traffic sign icon image to the image generation unit <NUM>, and outputs the obtained identification information to the direction target determination unit <NUM>.

Subsequently, in step S83, the direction target determination unit <NUM> determines, as the target type of the indication direction displayed to the driver V, either the target road or the traffic sign on the basis of the identification information about the traffic sign received from the identification unit <NUM>. Then, the direction target determination unit <NUM> outputs information indicating the determined target type of the indication direction to the direction indication unit <NUM>.

Subsequently, in step S84, the direction indication unit <NUM> receives the position information about each of the traffic sign and the road from the image processing apparatus <NUM>. This step S84 may be performed at any point in time between steps S81 to S83 explained above.

Subsequently, in step S85, the direction indication unit <NUM> obtains either an indication direction in which the target road, to which the traffic sign is applied, is present or an indication direction in which the traffic sign is present. In a case where the target type of the indication direction is the target road, the direction indication unit <NUM> determines, as the target road, the road to which the traffic sign is applied, on the basis of the position of the traffic sign and the position of the road in front of the vehicle <NUM>. Then, the direction indication unit <NUM> uses the reference position defined in advance and the coordinate information about the target road to calculate and obtain, as the indication direction θ, the direction in which the target road is present with respect to the reference position.

In contrast, in a case where the target type of the indication direction is the traffic sign, the direction indication unit <NUM> uses the coordinate information about the reference position and the traffic sign to calculate and obtain, as the indication direction θ, the direction of the traffic sign with respect to the reference position.

Therefore, the direction indication unit <NUM> outputs the indication direction θ to the image generation unit <NUM>.

Subsequently, in step S86, the image generation unit <NUM> generates a projection image in which the traffic sign icon image is arranged in the direction of the indication direction θ with respect to the reference position.

Subsequently, in step S87, the output unit <NUM> outputs the image data of the projection image generated by the image generation unit <NUM> to the projection unit <NUM>.

In this manner, the display apparatus <NUM> can generate the projection image for displaying, to the driver V, either an indication direction in which the target road, to which the traffic sign is applied, is present or an indication direction in which the traffic sign is present. In addition, the virtual image of the projection image generated by the projection unit <NUM> can be displayed.

The projection image is updated from time to time according to the forward image that changes as the vehicle <NUM> moves, and the virtual image of the updated projection image is superimposed on the forward view. In a case where any traffic sign is no longer included in the forward image, the virtual image of the projection image not including the traffic sign icon image is superimposed on the forward view.

Next, examples of applications of the display apparatus <NUM> are explained with reference to specific examples of forward images, projection images, and forward sceneries in which projection images are superimposed.

<FIG> is a drawing illustrating an example of a forward image <NUM> captured by the forward camera <NUM>. The forward image <NUM> includes a preceding ahead <NUM>, traffic signs <NUM> and <NUM>, and roads <NUM> and <NUM>. The traffic sign <NUM> is a road sign applied to the road <NUM>. The traffic sign <NUM> is a road sign applied to the road <NUM>.

The sign area extraction unit <NUM> in the image processing apparatus <NUM> extracts a sign image 82a including the traffic sign <NUM> and a sign image 83a including the traffic sign <NUM> from the forward image <NUM>, and outputs the sign image 82a and the sign image 83a to the sign position obtaining unit <NUM>. The road area extraction unit <NUM> extracts a road image 84a including the road <NUM> and a road image 85a including the road <NUM> from the forward image <NUM>, and outputs the road image 84a and the road image 85a to the road position obtaining unit <NUM>.

The sign position obtaining unit <NUM> obtains the coordinate information about the sign position 82b, i.e., the position of the traffic sign <NUM>, from the sign image 82a. The sign position obtaining unit <NUM> obtains the coordinate information about the sign position 83b, i.e., the position of the traffic sign <NUM>, from the sign image 83a. Such a sign position is, for example, the position of a pixel on the most positive side in the X direction and on the most negative side in the Y direction from among the pixels constituting the traffic sign in the sign image. The position of which pixel is adopted as the sign position is defined in advance.

Likewise, the road position obtaining unit <NUM> obtains the coordinate information about the road position 84b, i.e., the position of the road <NUM> in the road image 84a. Also, the road position obtaining unit <NUM> obtains the coordinate information about the road position 85b, i.e., the position of the road <NUM> in the road image 85a. Such a road position is, for example, a position corresponding to the center of gravity of the road image. Which pixel is to be adopted as the road position is defined in advance.

The identification unit <NUM> appropriately corrects the sign image 82a and the sign image 83a by applying preprocessing such as image rotation processing so as to allow the pattern matching processing to be easily executed. Thereafter, the identification unit <NUM> obtains the traffic sign icon image and the identification information by referring to the image storage unit <NUM> on the basis of the sign images 82a and 83a.

In the example of <FIG>, the traffic signs <NUM> and <NUM> are traffic signs indicating one-way traffic, and accordingly, the identification unit <NUM> obtains a traffic sign icon image of one-way traffic as traffic sign icon images corresponding to the traffic signs <NUM> and <NUM>. Alternatively, the identification unit <NUM> may obtain, as the traffic sign icon image corresponding to one-way traffic, a traffic sign icon image of "Do Not Enter" indicating that the vehicle cannot enter the road due to one-way traffic. A rule for obtaining such a traffic sign icon image is defined in advance.

The one-way traffic sign is a traffic regulation applied to the road, and accordingly, the direction target determination unit <NUM> determines that the target types of the indication directions of the traffic signs <NUM> and <NUM> are "target roads", on the basis of the identification information of the traffic signs <NUM> and <NUM> received from the identification unit <NUM>.

In the forward image <NUM>, the direction indication unit <NUM> determines, as the target road of the traffic sign <NUM>, the road <NUM> that is at the shortest distance from the sign position 82b, and determines, as the target road of the traffic sign <NUM>, the road <NUM> that is at the shortest distance from the sign position 83b. In the example of <FIG>, the distance from the sign position 82b to the road <NUM> and the distance from the sign position 82b to the road <NUM> are about the same, but the road <NUM> is adopted as the target road of the traffic sign <NUM> in order to avoid overlappingly choosing the road <NUM> that is adopted as the target road of the traffic sign <NUM>.

A reference position VC in <FIG> is a position corresponding to the viewpoint (i.e., the eyes) of the driver V. Where coordinates of the road position 84b are defined as (X<NUM> , Y<NUM>), coordinates of the road position 85b are denoted as (X<NUM>, Y<NUM>), and coordinates of the reference position VC are denoted as (XC, YC), the direction indication unit <NUM> calculates an indication direction θ<NUM> of the road <NUM> and an indication direction θ<NUM> of the road <NUM> according to the following expressions (<NUM>) and (<NUM>). <MAT><MAT>.

<FIG> is a drawing illustrating an example of a projection image <NUM> generated by the image generation unit <NUM>. The projection image <NUM> includes a traffic sign icon image <NUM> arranged at the indication direction θ<NUM> of the road <NUM> to which the traffic sign <NUM> is applied, a figure image <NUM>, and a vehicle speed image <NUM> arranged at a predetermined position. In this case, the traffic sign icon image <NUM> is an example of a predetermined image.

The traffic sign icon image <NUM> is to virtually display a road sign of "Do Not Enter" indicating that the vehicle cannot enter due to one-way traffic. The figure image <NUM> is a figure image displaying the indication direction θ<NUM>. The vehicle speed image <NUM> is a character image indicating the vehicle speed of the vehicle <NUM> detected by the vehicle speed sensor <NUM>. The vehicle speed image <NUM> is fixed at the predetermined position, and even when the vehicle <NUM> moves and the forward image changes, the position of the vehicle speed image <NUM> does not change. The vehicle speed image <NUM> is an example of a fixed image.

In the present embodiment, although the figure image <NUM> indicates a direction by displaying an arrow and a three-dimensional conical-shaped figure, the direction may be indicated in a manner different from the figure image <NUM>, as long as the direction can be indicated. For example, a character image may be displayed to indicate the direction by displaying language such as "right", "front left", and the like. In this case, for example, instead of the arrow and the three-dimensional conical-shaped figure, or together with the arrow and the three-dimensional conical-shaped figure, the display apparatus <NUM> may display a character image including characters or language such as "right front", "right direction at angle of θ<NUM> degrees with respect to the horizontal direction". The character image <NUM> displaying characters, "RIGHT FRONT", in <FIG> is an example of such a character image.

Next, <FIG> is a drawing illustrating an example of the forward view <NUM> of the vehicle <NUM> according to the present embodiment. The forward view <NUM> is an image that the driver V visually recognizes by seeing a virtual image <NUM>' of the projection image <NUM> projected and superimposed by the display apparatus <NUM> onto the front windshield <NUM>. As illustrated in <FIG>, the size of the virtual image <NUM>' is smaller than the forward view <NUM>.

A traffic sign icon virtual image <NUM>', i.e., a virtual image of the traffic sign icon image <NUM>, and a figure virtual image <NUM>', i.e., a virtual image of the figure image <NUM>, are displayed in a direction along the indication direction θ<NUM> of the road <NUM>, i.e., the target road to which the traffic sign <NUM> is applied. Also, a vehicle speed virtual image <NUM>', i.e., a virtual image of the vehicle speed image <NUM>, is displayed at the predetermined position.

The traffic sign icon virtual image <NUM>' corresponding to the traffic sign <NUM> is displayed in the direction along the indication direction θ<NUM>, so that the driver V can recognize the traffic sign <NUM> and the road <NUM> to which the traffic sign <NUM> is applied, without moving his or her eyes.

However, the traffic sign icon virtual image <NUM>' does not have to be strictly displayed in the direction along the indication direction θ<NUM>, and as long as the driver V can recognize the direction in which the target road and the traffic sign are present, the traffic sign icon virtual image <NUM>' may be displayed in a direction somewhat shifted from the indication direction θ<NUM>.

In this case, for example, when it is attempted to superimpose the traffic sign icon virtual image <NUM>' or the figure virtual image <NUM>' on the road <NUM> in the forward view <NUM>, a virtual image larger than the virtual image <NUM>' is desired, and the angle of view of projection by the projection unit <NUM> is required to be increased.

In contrast, in the embodiment, the indication direction θ<NUM> of the road <NUM> is displayed, so that the driver V can recognize the road <NUM> to which the traffic sign <NUM> is applied, even though the traffic sign icon virtual image <NUM>' or the figure virtual image <NUM>' is not superimposed at the position where the road <NUM> is visible in the forward view <NUM>.

<FIG> is a drawing illustrating an example of a forward view <NUM> that has changed from the state of <FIG> as a result of movement of the vehicle <NUM>. According to the movement of the vehicle <NUM>, the forward view <NUM> changes, and the indication direction of the road <NUM> as seen from the driver V has also changed from θ<NUM> to θ<NUM>. According to the change in the indication direction, the position where the traffic sign icon virtual image <NUM>' is displayed has also changed.

According to the change in the indication direction, the position and the shape of the figure virtual image <NUM>' have changed. The three-dimensional conical shape of the figure virtual image <NUM>' has changed, so that, as compared with the case where the direction is simply indicated by an arrow and the like, the direction can be displayed with a higher degree of reality. Regardless of the movement of the vehicle <NUM>, the position of the vehicle speed virtual image <NUM>' is fixed, and only the numerical value indicating the vehicle speed is changed.

The traffic sign icon virtual image <NUM>' and the figure virtual image <NUM>' are arranged around the vehicle speed virtual image <NUM>' on a line connecting the reference position Vc and the road position 84b, and according to the indication direction, the positions of the traffic sign icon virtual image <NUM>' and the figure virtual image <NUM>' change along the circumferences around the vehicle speed virtual image <NUM>'.

<FIG> and <FIG> illustrate an example for displaying only the indication direction θ<NUM> of the road <NUM> to which the traffic sign <NUM> is applied. However, the indication direction θ<NUM> of the road <NUM> to which the traffic sign <NUM> is applied can also be displayed in a similar manner.

Next, the advantageous effects of the display apparatus <NUM> are explained.

There is a display apparatus such as a head up display for recognizing traffic signs such as road signs, road surface marking, and the like in front of the vehicle on the basis of an image of the forward view from the vehicle obtained by a camera attached to the vehicle and superimposing virtual images of traffic sign icon images at positions of the road that are visible in the forward view.

However, with such a display apparatus, in a case where a path such as a road, to which the traffic sign is applied, or the traffic sign are visible in proximity to an edge portion and the like of the forward view, the display apparatus is required to project a wide-angle view in order to superimpose virtual images of predetermined images such as traffic sign icon images at positions of traffic signs in the forward view. For example, a virtual image with a size larger than the virtual image <NUM>' of the projection image in <FIG> is required to be superimposed.

In this case, <FIG> is a drawing for explaining an increase in the angle of view of projection by the projection unit <NUM>. As illustrated in <FIG>, when the driver V sees a virtual image I' that is of a larger size than the virtual image I, an angle ϕ' of view of projection that is larger than the angle ϕ of view of projection for the virtual image I is required. When the angle of view of projection is increased, the size of the display apparatus may increase, and complex and expensive projection lenses may be required to ensure the quality of the virtual image.

In contrast, in the present embodiment, an indication direction in which either the target road or the traffic sign is present is displayed to the driver V. For example, the traffic sign icon image is arranged along the indication direction in the virtual image, and the image generation unit <NUM> generates a projection image including a figure indicating the indication direction to cause the virtual image of the generated projection image to be displayed, so that the indication direction is displayed to the driver V.

In the embodiment, the direction is displayed to the driver V, and accordingly, it is not necessary to superimpose a virtual image of a predetermined image such as a traffic sign icon image at the position of the target road that are visible in the forward view, and it is not necessary to increase the angle of view of projection. Therefore, even with a small angle of view of projection such as the angle ϕ of view of projection in <FIG>, either the target road or the traffic sign can be displayed.

Traffic signs include regulatory traffic signs indicating regulations on roads such as "Do Not Enter", "Stop", and the like, and auxiliary traffic signs for calling for attention according to an ambient environment of a road such as a presence of a school in close proximity, a pedestrian crossing in close proximity, and the like. In the regulatory traffic signs indicating regulations on roads, it is preferable to allow the driver V to recognize the target road, but in a case of the auxiliary traffic signs, it is preferable to allow the driver V to recognize the traffic sign itself rather than the target road.

In contrast, in the present embodiment, the display apparatus <NUM> includes a direction target determination unit configured to determine, as the target type of the indication direction, either the target road or the traffic sign on the basis of the traffic sign. Therefore, when the target type of the indication direction is the target road, the indication direction of the target road can be displayed, and in the case of the auxiliary traffic signs, the indication direction of the traffic sign can be displayed, so that information appropriate for the traffic sign can be provided to the driver V.

In the present embodiment, the indication direction that changes according to the movement of the vehicle <NUM> can be displayed. Therefore, according to a change in the situation caused by the movement of the vehicle <NUM>, appropriate information can be provided to the driver V.

In the present embodiment, the indication direction can be displayed by arranging the traffic sign icon image <NUM> along the indication direction around the fixed image such as the vehicle speed image <NUM> arranged in a fixed manner in the projection image. The fixed image is frequently seen by the driver V, and in many cases, the fixed image is fixed at a position where the driver V can easily see the fixed image. The traffic sign icon image <NUM> is arranged around such a fixed image, so that the indication direction can be displayed to the driver V to allow the driver V to easily see the indication direction.

Although, in the present embodiment, both of the traffic sign icon image <NUM> and the figure image <NUM> are displayed together as the predetermined image, either the traffic sign icon image <NUM> or the figure image <NUM> may be displayed as the predetermined image. Also, the figure image <NUM> may be a simple figure image of an arrow or a character image indicating the indication direction with an indication of an angle and the like. Although, in the present embodiment, the position of the traffic sign icon image <NUM> is changed along the indication direction in order to display the indication direction, the indication direction may be displayed by rotating the traffic sign icon image <NUM> without changing its position.

In the present embodiment, the image processing apparatus <NUM> obtains the position information about the traffic sign and the road on the basis of images captured by the forward camera <NUM>, but the present embodiment is not limited thereto. The image processing apparatus <NUM> obtains the road in front of the vehicle <NUM> and the position information about the traffic sign by using information received from the car navigation system <NUM>. Also, the vehicle <NUM> may be provided with a gyro sensor for detecting the orientation of the vehicle <NUM>, and the detection value of the gyro sensor may be used.

In the present embodiment, the image storage unit <NUM> stores the traffic sign icon image, but the present embodiment is not limited thereto. The image storage unit <NUM> may store character images including characters expressing information indicated by traffic signs. The image storage unit <NUM> may store figure images displaying traffic signs in a simplified manner.

In the present embodiment, the identification unit <NUM> obtains the traffic sign icon image and the identification information by referring to the image storage unit <NUM>, and the direction target determination unit <NUM> determines the target type of the indication direction by referring to the direction table storage unit <NUM>, but the present embodiment is not limited thereto. The identification unit <NUM> and the direction target determination unit <NUM> may achieve such functions by using the machine learning explained above.

In the present embodiment, the indication direction θ is obtained by two-dimensional image processing using an image captured by the monocular forward camera <NUM>, but the present embodiment is not limited thereto. The indication direction θ may be obtained by three-dimensional image processing based on a projection relationship by using distance information of a distance to either the target road or the traffic sign detected by a stereo camera and the like serving as the forward camera <NUM> and a known relationship in position between the vehicle <NUM> and the driver V. With such three-dimensional image processing, the driver V can visually recognize, with a higher degree of reality, the virtual image of the predetermined image to find the indication direction.

Next, the display apparatus 1a according the second embodiment is explained.

When the vehicle <NUM> moves, the traffic sign may be out of the forward view of the vehicle <NUM>, and accordingly, the forward camera <NUM> may no longer be able to obtain information about the traffic sign. In this case, the traffic sign icon image corresponding to the traffic sign of which information is no longer obtained may be hidden, but if the traffic sign is a traffic sign that is applicable to a certain section of a road, such as a speed limit, it may be preferable to continue showing the virtual image of the traffic sign icon image to the driver V even after the information is no longer obtained.

In the present embodiment, when the forward camera <NUM> is no longer able to obtain information about the traffic sign due to the movement of the vehicle <NUM>, a projection image including a traffic sign icon image fixed at a predetermined position is generated. Therefore, even when the information is no longer obtained, the display apparatus 1a according to the second embodiment allows the virtual image of the traffic sign icon image to remain within the view of the driver V.

Whether to generate a projection image including a traffic sign icon image fixed at a predetermined position or a projection image not including a traffic sign icon image is selected according to the traffic sign. Therefore, when information is no longer obtained, whether to display the virtual image of the traffic sign icon image or hide the traffic sign icon image can be selected according to the traffic sign.

<FIG> is a block diagram illustrating an example of a functional configuration of a control unit 20a of the display apparatus 1a. As illustrated in <FIG>, the control unit 20a includes a continuation table storage unit <NUM>, an information holding unit <NUM>, a direction indication unit 25a, and an image generation unit 26a. The direction indication unit 25a includes a determination unit <NUM> and a selection unit <NUM>.

Among them, the functions of the continuation table storage unit <NUM> are achieved by the HDD <NUM> and the like, and the functions of the information holding unit <NUM> are achieved by the RAM <NUM> and the like.

The continuation table storage unit <NUM> stores a correspondence table indicating the correspondence between identification information and display continuation information defined in advance. In this case, the display continuation information is information indicating that the traffic sign icon image is to be either fixed at a predetermined position or is to be hidden. For example, when the traffic sign icon image is to be fixed, the display continuation information is set to "<NUM>", and when the traffic sign icon image is to be hidden, the display continuation information is set to "<NUM>".

The information holding unit <NUM> temporarily holds the identification information about the traffic sign before a movement of the vehicle <NUM>. Because the vehicle <NUM> is constantly moving while the vehicle <NUM> is cruising, it is preferable that identification information in a cycle previous to a predetermined cycle in the image capturing sequence of the forward camera <NUM> is adopted as the identification information about the traffic sign before the movement of the vehicle <NUM>, so that the identification information immediately before the movement can be obtained.

The direction indication unit 25a receives identification information from the direction target determination unit <NUM>. Then, when information about a traffic sign is obtained, the obtained identification information is output to and held in the information holding unit <NUM>. When information about a traffic sign is no longer obtained, the display continuation information is output to the image generation unit 26a. The direction indication unit 25a has not only the above functions but also functions similar to the functions of the direction indication unit <NUM>.

The determination unit <NUM> of the direction indication unit 25a determines whether information about the traffic sign is obtained on the basis of the information about the sign position received from the image processing apparatus <NUM>, and outputs the determination result to the selection unit <NUM>.

Specifically, when the traffic sign is no longer included in the forward view of the vehicle <NUM>, and the forward camera <NUM> no longer obtains the information about the traffic sign, the image processing apparatus <NUM> outputs missing sign information as explained with reference to <FIG>. When the determination unit <NUM> receives the missing sign information, the determination unit <NUM> determines that the information about the traffic sign has not been obtained.

When the information about the traffic sign has not been obtained, the selection unit <NUM> selects, on the basis of the traffic sign, whether to either fix the traffic sign icon image at the predetermined position or hide the traffic sign icon image, and outputs the selection result to the image generation unit <NUM>.

Specifically, when the information about the traffic sign is determined not to have been obtained, the selection unit <NUM> accesses the information holding unit <NUM> to obtain the identification information about the traffic sign temporarily held by the information holding unit <NUM>. Then, the display continuation information is obtained by referring to the continuation table storage unit <NUM> on the basis of this identification information. This display continuation information corresponds to the above selection result.

When the image generation unit 26a receives display continuation information (for example, "<NUM>") indicating that the traffic sign icon image is to be fixed at the predetermined position, the image generation unit 26a generates a projection image including the traffic sign icon image fixed at the predetermined position. When the image generation unit 26a receives display continuation information (for example, "<NUM>") indicating that the traffic sign icon image is to be hidden, the image generation unit 26a generates a projection image not including a traffic sign icon image. When the information about the traffic sign is obtained (i.e., when the information about the sign position is not the missing sign information), the image generation unit 26a generates a projection image in which the traffic sign icon image is arranged in the direction of the indication direction θ with respect to the reference position.

Next, a flow of processing by the control unit 20a is explained with reference to <FIG> is a flowchart of an example of processing of the control unit 20a. The processing from steps S151 to S154 in <FIG> is similar to the processing from steps S81 to S84 in <FIG>, and accordingly, repeated explanation thereabout is omitted. With respect to steps other than steps S151 to S154, duplicate explanations similar the explanations of <FIG> are omitted as necessary.

In step S155, the determination unit <NUM> determines whether the information about the traffic sign has been obtained on the basis of the information about the sign position received from the image processing apparatus <NUM>.

When the information about the sign position is determined to have been obtained in step S155 (Yes in step S155), the direction indication unit 25a outputs the identification information about the traffic sign to the information holding unit <NUM> to cause the information holding unit <NUM> to hold the identification information about the traffic sign in step S156.

Subsequently, in step S157, the direction indication unit 25a obtains an indication direction θ in which either the target road or the traffic sign is present, and outputs the indication direction θ to the image generation unit 26a.

Subsequently, in step S158, the image generation unit 26a generates a projection image in which the traffic sign icon image is arranged in the direction of the indication direction θ with respect to the reference position. Thereafter, the processing in step S163 is performed.

When the information about the sign position is determined not to have been obtained in step S155 (No in step S155), the selection unit <NUM> accesses the information holding unit <NUM> to obtain the identification information about the traffic sign temporarily held by the information holding unit <NUM> in step S159.

Subsequently, in step S160, the selection unit <NUM> obtains the display continuation information by referring to the continuation table storage unit <NUM> on the basis of the identification information.

Subsequently, in step S161, the image generation unit 26a determines whether the traffic sign is to be continuously displayed on the basis of the display continuation information.

When the traffic sign is determined to be continuously displayed in step S161 (Yes in step S161), the image generation unit 26a generates a projection image including the traffic sign icon image fixed at the predetermined position in step S162. When the traffic sign is determined not to be continuously displayed in step S161 (No in step S161), the image generation unit 26a generates a projection image not including the traffic sign icon image in step S163.

Subsequently, in step S164, the output unit <NUM> outputs the projection image generated by the image generation unit 26a to the projection unit <NUM>.

In this manner, when the forward camera <NUM> no longer obtains information about the traffic sign as a result of movement of the vehicle <NUM>, the display apparatus 1a can generate, according to the traffic sign, either a projection image including the traffic sign icon image fixed at the predetermined position or a projection image not including the traffic sign icon image. Then, the virtual image of the projection image by the projection unit <NUM> can be displayed.

It should be noted that the traffic sign icon image fixed at the predetermined position may be hidden by the image generation unit 26a at a point in time when information about a new traffic sign is obtained from the forward image as a result of movement of the vehicle <NUM>.

Next, examples of applications of the display apparatus 1a are explained with reference to specific examples of forward images, projection images, and forward sceneries in which projection images are superimposed.

First, <FIG> is a drawing illustrating an example of a forward view <NUM> of the vehicle <NUM> according to the present embodiment. The forward view <NUM> is an image that the driver V visually recognizes by seeing a virtual image <NUM> of the projection image projected and superimposed by the display apparatus 1a onto the front windshield <NUM>. The forward view <NUM> includes not only the virtual image <NUM> but also a traffic sign <NUM>, a traffic sign icon virtual image <NUM>, a figure virtual image <NUM>, and a vehicle speed virtual image <NUM>.

In the example of <FIG>, the traffic sign is determined to be the target type of the indication direction, and the traffic sign icon virtual image <NUM> and the figure virtual image <NUM> are displayed along an indication direction θ<NUM> of the traffic sign <NUM>. The vehicle speed virtual image <NUM> is displayed in a fixed manner at a predetermined position.

Next, <FIG> is a drawing illustrating an example of the forward view <NUM> that has changed from the state of <FIG> as a result of movement of the vehicle <NUM>. According to the movement of the vehicle <NUM>, the forward view <NUM> changes and no longer includes the traffic sign <NUM> illustrated in <FIG>. In other words, in the state of <FIG>, the information about the traffic sign is not obtained.

In the example as illustrated in <FIG>, the traffic sign is a road sign of the speed limit, and is applied to a certain section of a road. Therefore, even after the information about the traffic sign is no longer obtained, it is preferable to continuously display the road sign. Therefore, the correspondence table in the continuation table storage unit <NUM> defines the road sign of the speed limit to be continuously displayed. The display apparatus 1a continuously displays the traffic sign icon virtual image <NUM> fixed besides the vehicle speed virtual image <NUM> on the basis of this correspondence table. It is preferable that the traffic sign icon virtual image <NUM> is displayed besides the vehicle speed virtual image <NUM>, because the driver V can easily compare the vehicle speed of the vehicle <NUM> and the speed limit.

In <FIG>, for example, the traffic sign icon virtual image <NUM> is displayed in a fixed manner besides the vehicle speed virtual image <NUM>. However, the display position of the traffic sign icon virtual image <NUM> is not limited thereto, and the traffic sign icon virtual image <NUM> may be displayed at any position. The traffic sign icon virtual image <NUM> is preferably displayed at such a position that the driver V can easily see the traffic sign icon virtual image <NUM>.

In a manner as explained above, in the present embodiment, when the information about the traffic sign is no longer obtained as a result of movement of the vehicle <NUM>, a projection image including the traffic sign icon image fixed at the predetermined position is generated. Therefore, even when the information about the traffic sign is no longer obtained, the display apparatus 1a allows the driver V to continue to visually recognize the virtual image of the traffic sign icon image.

In addition, in the present embodiment, whether to either fix the traffic sign icon image at the predetermined position or hide the traffic sign icon image is selected according to the traffic sign. Therefore, when the information about the traffic sign is no longer obtained, whether to display, for the driver V, the virtual image of the traffic sign icon image fixed at the predetermined position or to hide the traffic sign icon image can be selected according to the traffic sign. In addition, information appropriate for the traffic sign can be provided to the driver V.

The embodiment for carrying out the present invention has been described above with reference to specific examples, but the present invention is not limited to the particulars of the described examples and various modifications and substitutions can be made without departing from the scope of the claimed subject matter.

In the above embodiment, some or all of the functions provided in the image processing apparatus <NUM> may be provided in the control unit <NUM>. Some of the functions provided in the control unit <NUM> may be provided in other constituent elements such as the image processing apparatus <NUM>, the engine ECU <NUM>, the detection apparatus <NUM>, and the like. Information stored in the image storage unit <NUM>, the direction table storage unit <NUM>, and the like may be stored in an external server, and the stored information may be referred to by accessing the external server via a network.

In the embodiments, the road is an example of a path, the vehicle is an example of a moving body, and the HUD is an example of a display apparatus, but the embodiments are not limited thereto. The embodiments can be applied to paths other than roads, moving bodies other than vehicles, and display apparatuses other than the HUD.

The embodiments also include a display method. For example, the display method causes a display apparatus to display a virtual image of a projection image, and the display method includes obtaining a predetermined image obtained based on a traffic sign;causing an image generation unit (<NUM>) to generate the projection image including the predetermined image; and causing a direction indication unit (<NUM>) to output either an indication direction (Θ) in which a target path, to which the traffic sign is applied, is present or an indication direction (Θ) in which the traffic sign is present, the indication direction (Θ) being determined based on a position of the traffic sign and a position of a path in front of the moving body (<NUM>). The method further includes causing the image generation unit (<NUM>) to generate the projection image including the indication direction (Θ), the indication direction (Θ) indicating a direction in which the target path or the traffic sign is present with respect to a reference position (Vc), wherein the projection image of a predetermined image size is generated with the reference position (Vc) being adjusted with respect to a forward image such that a traffic sign icon image is arranged in the direction of the indication direction (Θ) with respect to the reference position (Vc). According to the display method, the effects similar to the effects achieved by the display apparatus can be obtained.

The embodiment also includes a program. For example, the program causes a computer to execute the display method disclosed above. According to the program, the effects similar to the effects achieved by the display apparatus can be obtained.

Claim 1:
A display apparatus (<NUM>) for being mounted on a moving body (<NUM>) and for displaying a virtual image of a projection image to a driver (V) of the moving body (<NUM>), comprising:
an image processing apparatus (<NUM>) configured to execute image processing on an image of a forward view from the moving body (<NUM>) and to output position information;
an identification unit (<NUM>) configured to obtain a traffic sign icon image based on a sign image received from the image processing apparatus (<NUM>);
an image generation unit (<NUM>) configured to generate the projection image including a predetermined image obtained based on a traffic sign; characterized in that the display apparatus further comprises
a direction indication unit (<NUM>) configured to output either an indication direction (θ) in which a target path, to which the traffic sign is applied, is present or an indication direction (θ) in which the traffic sign is present, based on a position of the traffic sign and a position of a path in front of the moving body (<NUM>), and to cause the image generation unit (<NUM>) to generate the projection image including the indication direction (θ), the indication direction (θ) indicating a direction in which the target path or the traffic sign is present with respect to a reference position (Vc), wherein
the image generation unit (<NUM>) is configured to generate the projection image of a predetermined image size with the reference position (Vc) being adjusted with respect to a forward image such that a traffic sign icon image is arranged in the direction of the indication direction (θ) with respect to the reference position (Vc).