Patent Description:
Patent literature (PTL) <NUM> discloses technology for displaying a predicted trajectory in overlay on an image of the area behind a vehicle when the vehicle is in reverse. The predicted trajectory is the trajectory traversed by the left and right wheels of the vehicle. This technology overlays the predicted trajectory in a ladder shape at constant intervals on the image of the area behind the vehicle, enabling the driver to recognize distance. <CIT> discloses a vehicle periphery monitoring system that, if the detected distance from a vehicle to a detected obstacle is equal to or greater than a preset distance, performs an image processing process on a captured image of the periphery of the vehicle to acquire the height of the vehicle. If the acquired height of the obstacle is less than a preset height, then the obstacle is displayed as a safe obstacle. If the acquired height of the obstacle is equal to or greater than the preset height, then the obstacle is displayed as a dangerous obstacle. The vehicle periphery monitoring system is able to indicate to the vehicle driver whether the detected obstacle is dangerous or safe. The vehicle periphery monitoring system is not required to perform the image processing process on the entire area of the captured image, and does not require a CPU having a high processing capability. <CIT> discloses a vehicle rear view monitoring device that improves the degree of positional precision of the overlay of the obstacle mark on the obstacle displayed on a captured image, because the vehicle rear view monitoring device changes the detection threshold level of each of distance sensors to change horizontal and vertical detection areas and a bearing resolution, determines the display position of an obstacle mark from the distances to an obstacle detected by the plurality of distance sensors, and amplitude information about the amplitude of a reflected wave from the obstacle, and overlays the obstacle mark on the obstacle by using an overlay unit.

According to the present invention, an image processing apparatus according to claim <NUM>, a camera according to claim <NUM>, a movable body according to claim <NUM>, and an image processing method according to claim <NUM> are provided.

An image processing apparatus, a camera, a moveable body, an image processing method, and the like that enable a user to recognize whether there is a possibility of colliding in the height direction with an obstacle present in the travel direction of a moveable body are disclosed below.

Embodiments of the present disclosure are described below through examples with reference to the drawings. Identical reference signs in the drawings indicate identical or equivalent constituent elements.

<FIG> illustrates an example configuration of an image processing apparatus <NUM> according to an embodiment of the present disclosure. As illustrated in <FIG>, the image processing apparatus <NUM> according to the present embodiment is mounted in a moveable body <NUM>. The image processing apparatus <NUM> overlays a display indicating a course trajectory of a specific portion of the moveable body <NUM> in the travel direction of the moveable body <NUM> on a surrounding image yielded by capturing an image of the surrounding area of the moveable body <NUM>.

The moveable body <NUM> is, for example, a vehicle. The vehicle may be an automobile or an industrial vehicle but is not limited to these. Other examples may include railway vehicles, vehicles for daily life, and fixed-wing aircraft that run on a runway. Examples of automobiles include, but are not limited to, passenger vehicles, trucks, buses, motorcycles, and trolley buses, and may include other vehicles that travel on the road. Industrial vehicles include industrial vehicles for agriculture and for construction. Industrial vehicles include, but are not limited to, forklifts and golf carts. Industrial vehicles for agriculture include, but are not limited to, tractors, cultivators, transplanters, binders, combines, and lawnmowers. Industrial vehicles for construction include, but are not limited to, bulldozers, scrapers, backhoes, cranes, dump cars, and road rollers. The vehicle may also be a human-powered vehicle. The vehicle is not limited to the above-listed types. For example, automobiles may include industrial vehicles that can drive on the road, and the same vehicle may be included in multiple categories.

The image processing apparatus <NUM> illustrated in <FIG> includes an interface <NUM>, a processor <NUM>, and a memory <NUM>. An imaging unit <NUM> that acquires surrounding images of the moveable body <NUM> is mounted in the moveable body <NUM>. The image processing apparatus <NUM> and the imaging unit <NUM> form a camera 1A mounted in the moveable body <NUM>.

First, the imaging unit <NUM> is described.

The imaging unit <NUM> is an on-vehicle camera mounted in the moveable body <NUM>. The imaging unit <NUM> acquires surrounding images yielded by capturing images of the surrounding area of the moveable body <NUM>. A plurality of imaging units <NUM> may be mounted in the moveable body <NUM>. For example, when four on-vehicle cameras are mounted in the moveable body <NUM>, one imaging unit <NUM> may be arranged at each of a position capable of imaging the surrounding area in front of the moveable body <NUM> and at least a portion of the front surface of the moveable body <NUM>, a position capable of imaging the surrounding area behind the moveable body <NUM> and at least a portion of the back surface of the moveable body <NUM>, a position capable of imaging the surrounding area on the left side of the moveable body <NUM> and at least a portion of the left surface of the moveable body <NUM>, and a position capable of imaging the surrounding area on the right side of the moveable body <NUM> and at least a portion of the right surface of the moveable body <NUM>. With this arrangement, images are captured of the surrounding area in four directions from the moveable body <NUM>.

The imaging unit <NUM> includes at least an imaging optical system 11a and an image sensor 11b.

For example, the imaging optical system 11a includes one or more lenses, apertures, and the like. The lens included in the imaging optical system 11a is a wide-angle lens such as a fisheye lens, for example. The imaging optical system 11a forms an image of a subject on an optical detection surface of the image sensor 11b. The image sensor 11b includes a charge coupled device (CCD) image sensor, a complementary metal-oxide semiconductor (CMOS) image sensor, or the like, for example. A plurality of pixels are arrayed on the optical detection surface of the image sensor 11b. The image sensor 11b generates a captured image by capturing the image of the subject formed on the optical detection surface. The imaging unit <NUM> may output the surrounding image to external apparatuses, such as an electronic control unit (ECU), a display, and/or a navigation apparatus mounted in the moveable body <NUM>. The imaging unit <NUM> may include a function to perform predetermined image processing on the surrounding image, such as white balance adjustment, exposure adjustment, or gamma correction.

Next, the components of the image processing apparatus <NUM> are described.

The interface <NUM> communicates with the components of the moveable body <NUM> in a wired or wireless manner. For example, the interface <NUM> acquires a surrounding image captured by the imaging unit <NUM> and outputs the surrounding image to the processor <NUM>.

Examples of the processor <NUM> include a dedicated processor such as a digital signal processor (DSP) and a general-purpose processor such as a central processing unit (CPU). The processor <NUM> controls overall operations of the image processing apparatus <NUM>. For example, the processor <NUM> overlays a display indicating the course trajectory of a specific portion of the moveable body <NUM> in the travel direction of the moveable body <NUM> on the surrounding image acquired by the interface <NUM> at a position corresponding to the height of the specific portion from the road surface. Upon detecting an obstacle included in the surrounding image and present in the travel direction of the moveable body <NUM>, the processor <NUM> changes the display indicating the course trajectory when the obstacle and the course trajectory of the specific portion of the moveable body <NUM> are in contact.

The processor <NUM> displays the surrounding image that has the display indicating the course trajectory of the specific portion of the moveable body <NUM> overlaid thereon on a display <NUM>, for example, included in the moveable body <NUM>. When the obstacle and the course trajectory of the specific portion of the moveable body <NUM> are in contact, the processor <NUM> may output a signal indicating danger of contact to an external destination. For example, when the obstacle and the course trajectory of the specific portion of the moveable body <NUM> are in contact, the processor <NUM> may cause a buzzer mounted in the moveable body <NUM> to output a warning sound. When the moveable body <NUM> includes a sonar configured to detect surrounding obstacles with sound waves, the processor <NUM> may be configured not to output the warning sound or the like even if the sonar detects a possibility of contact with an obstacle as long as the obstacle and the course trajectory of the specific portion of the moveable body <NUM> are not in contact.

The specific portion of the moveable body <NUM> is a portion restricted in the height direction with respect to movement of the moveable body <NUM>. For example, the specific portion is the bottom surface or top surface of the vehicle body of the moveable body <NUM>. Examples of the specific portion when the moveable body <NUM> is a vehicle include the bottom surface of the bumper of the vehicle, the roof of the vehicle, and an object installed on the roof of the vehicle.

The memory <NUM> includes a primary memory device, a secondary memory device, and the like, for example. The memory <NUM> stores various information and programs necessary for operation of the image processing apparatus <NUM>.

The operations of the image processing apparatus <NUM> are now described.

<FIG> is a diagram illustrating an example of a display, indicating a course trajectory of a specific portion of the moveable body <NUM> in the travel direction of the moveable body <NUM>, overlaid by the processor <NUM> on a surrounding image. An example is described below for the case of overlaying a display indicating a course trajectory of a specific portion of the moveable body <NUM> on a surrounding image of the area behind the moveable body <NUM> when the moveable body <NUM> is moving in reverse. The specific portion of the moveable body <NUM> in <FIG> is the bottom surface of the bumper <NUM>.

As illustrated in <FIG>, the processor <NUM> overlays a display <NUM> indicating the course trajectory of the bottom surface of the bumper <NUM> on the surrounding image. The processor <NUM> overlays the display <NUM> indicating the course trajectory of the bottom surface of the bumper <NUM> (the display indicating the course trajectory of the bottom surface of the vehicle body of the moveable body <NUM>) on the surrounding image at a position corresponding to the height of the bottom surface of the bumper <NUM> from the road surface <NUM>.

<FIG> is a diagram illustrating the overlay of a display indicating a course trajectory on a surrounding image by the processor <NUM> when the moveable body <NUM> is viewed from the side. In the following explanation, the display indicating a course trajectory overlaid on a surrounding image is depicted virtually in the side views of the moveable body <NUM> viewed from the side.

As illustrated in <FIG>, the processor <NUM> overlays the display <NUM> indicating the course trajectory of the bottom surface 2a of the bumper <NUM> on the surrounding image. The course trajectory is flat and extends from the bottom surface 2a of the bumper <NUM> in the travel direction of the moveable body <NUM>. The processor <NUM> overlays the display indicating the course trajectory on the surrounding image as a flat surface of a predetermined color, for example. The processor <NUM> may overlay the flat surface on the surrounding image as a semitransparent or mesh-patterned flat surface.

The processor <NUM> detects whether an obstacle is present in the travel direction of the moveable body <NUM>. The method for detecting whether an obstacle is present may, for example, be a method for detecting an obstacle by image analysis of the surrounding image captured in the travel direction of the moveable body <NUM>. This example is not limiting, however, and any appropriate method may be used. It is assumed below that an obstacle <NUM>, such as a parking block, is present on the road surface <NUM>, as illustrated in <FIG>.

Upon detecting the obstacle <NUM> present in the travel direction of the moveable body <NUM>, the processor <NUM> judges whether the image of the obstacle <NUM> and the display <NUM> indicating the course trajectory are in contact in the surrounding image. The display <NUM> indicating the course trajectory is overlaid at a position corresponding to the height of the bottom surface 2a of the bumper <NUM> from the road surface <NUM>. Accordingly, the processor <NUM> judges whether the image of the obstacle <NUM> and the display <NUM> indicating the course trajectory are in contact by whether the image of the obstacle <NUM> is higher than the display <NUM> indicating the course trajectory of the specific portion of the moveable body <NUM>. In other words, the processor <NUM> judges whether the obstacle <NUM> and the specific portion of the moveable body <NUM> (bottom surface 2a of the bumper <NUM>) are in contact.

Upon judging that the obstacle <NUM> and the specific portion of the moveable body <NUM> are in contact, the processor <NUM> changes a portion of the display <NUM> indicating the course trajectory, as illustrated in <FIG>. Specifically, the processor <NUM> changes the portion, within the display <NUM> indicating the course trajectory, that is in contact with the image of the obstacle <NUM> included in the surrounding image. In <FIG>, the processor <NUM> changes the display <NUM> indicating the course trajectory of the portion in contact with the image of the obstacle <NUM> included in the surrounding image to appear as if embedded in the obstacle <NUM>, as illustrated by the dashed ovals.

When the height of the obstacle <NUM> is less than the bottom surface 2a of the bumper <NUM> of the moveable body <NUM>, the obstacle <NUM> and the bottom surface 2a of the bumper <NUM> of the moveable body <NUM> do not come in contact. If the predicted trajectory alone were overlaid on the surrounding image, the user would not be able to recognize the possibility of contact in the height direction between the obstacle <NUM> present on the road surface <NUM> and the moveable body <NUM>. In the present disclosure, however, the processor <NUM> overlays the display <NUM> indicating the course trajectory of the specific portion of the moveable body <NUM> on the surrounding image at a position corresponding to the height of the specific portion. The processor <NUM> then changes the display <NUM> indicating the course trajectory when the obstacle <NUM> and the course trajectory of the specific portion are in contact. In this way, the user can easily recognize whether there is a possibility of contact in the height direction between the obstacle <NUM> present on the road surface <NUM> and the moveable body <NUM>.

<FIG> is a diagram illustrating another example of a display, indicating a course trajectory of a specific portion of the moveable body <NUM>, overlaid by the processor <NUM> on a surrounding image. In <FIG>, the specific portion of the moveable body <NUM> is a rear spoiler <NUM> installed at the back of the roof of the vehicle. An obstacle <NUM> is present above the road surface <NUM> in <FIG>.

As illustrated in <FIG>, the processor <NUM> overlays a display <NUM> indicating the course trajectory of the bottom surface of the bumper <NUM> on the surrounding image. The processor <NUM> also overlays a display <NUM> indicating the course trajectory of the rear spoiler <NUM> (display indicating the course trajectory of the upper surface of the vehicle body of the moveable body <NUM>) on the surrounding image at a position corresponding to the height of the upper surface of the rear spoiler <NUM> from the road surface <NUM>.

Upon detecting the obstacle <NUM>, the processor <NUM> judges whether the image of the obstacle <NUM> and the display <NUM> indicating the course trajectory are in contact in the surrounding image. The display <NUM> indicating the course trajectory is overlaid at a position corresponding to the height of the rear spoiler <NUM> from the road surface <NUM>. Accordingly, the processor <NUM> judges whether the obstacle <NUM> and the display <NUM> indicating the course trajectory are in contact by whether the image of the obstacle <NUM> is higher than the display <NUM> indicating the course trajectory of the specific portion of the moveable body <NUM>. In other words, the processor <NUM> judges whether the obstacle <NUM> and the specific portion of the moveable body <NUM> (rear spoiler <NUM>) are in contact.

Upon judging that the obstacle <NUM> and the specific portion of the moveable body <NUM> are in contact, the processor <NUM> changes a portion of the display <NUM> indicating the course trajectory, as illustrated in <FIG>. Specifically, the processor <NUM> changes the portion, within the display <NUM> indicating the course trajectory, that is in contact with the image of the obstacle <NUM> included in the surrounding image. In <FIG>, the processor <NUM> changes the display <NUM> indicating the course trajectory so that the portion, within the display <NUM> indicating the course trajectory, that is in contact with the image of the obstacle <NUM> is crushed, as illustrated by the dashed circles.

When the height of the obstacle <NUM> is greater than the rear spoiler <NUM> of the moveable body <NUM>, the obstacle <NUM> and the rear spoiler <NUM> do not come in contact. If the predicted trajectory were simply overlaid on the surrounding image, the user would not be able to recognize the possibility of contact in the height direction between the obstacle <NUM> present above the road surface <NUM> and the moveable body <NUM>. In the present disclosure, however, the processor <NUM> overlays the display <NUM> indicating the course trajectory of the specific portion of the moveable body <NUM> on the surrounding image at a position corresponding to the height of the specific portion. The processor <NUM> then changes the display <NUM> indicating the course trajectory when the obstacle <NUM> and the course trajectory of the specific portion are in contact. In this way, the user can recognize whether there is a possibility of contact in the height direction between the obstacle <NUM> and the moveable body <NUM>.

When the moveable body <NUM> and the obstacle are in contact, the processor <NUM> may overlay a display indicating danger of contact on the surrounding image and/or output a signal indicating danger of contact to an external destination. For example, the processor <NUM> may output a signal indicating danger of contact via the interface <NUM> to a buzzer mounted in the moveable body <NUM> and cause the buzzer to sound.

When the moveable body <NUM> and the obstacle are not in contact, the processor <NUM> may calculate the difference in height between the specific portion the moveable body <NUM> and the obstacle, and as illustrated in <FIG>, may overlay a display <NUM> indicating the calculated difference on the surrounding image. <FIG> illustrates an example of the display <NUM>, indicating the difference in height between the rear spoiler <NUM> and the obstacle <NUM>, overlaid on the surrounding image.

Examples of the road surface <NUM> being flat have been described thus far, but the road surface <NUM> may have a height difference. The overlay of a display indicating the course trajectory on the surrounding image when the road surface <NUM> has a height difference is described below.

<FIG> illustrates an example of the moveable body <NUM> seen from the side. In <FIG>, the road surface <NUM> is, for example, formed by a horizontal road surface 3a, a horizontal road surface 3b at a lower position than the road surface 3a, and a downward-sloping road surface 3c that connects the road surface 3a and the road surface 3b. In <FIG>, the moveable body <NUM> is moving from the road surface 3a towards the road surface 3b across the road surface 3c. An obstacle <NUM> is present on the road surface 3b. The height of the obstacle <NUM> is greater than the height of the bottom surface 2a of the bumper <NUM> from the road surface <NUM>. In other words, the obstacle <NUM> and the bottom surface 2a of the bumper <NUM> are in contact at the road surface 3b. While the moveable body <NUM> is on the road surface 3a, the upper end of the obstacle <NUM> is at a lower position than the bottom surface 2a of the bumper <NUM>.

When the moveable body <NUM> is on the road surface 3a, one method could be to overlay a display <NUM> indicating the course trajectory on the surrounding image as a flat shape extending in the travel direction of the moveable body <NUM> at a position corresponding to the height of the bottom surface 2a of the bumper <NUM> from the road surface 3a, as illustrated in <FIG>. While the moveable body <NUM> is on the road surface 3a, the upper end of the obstacle <NUM> is at a lower position than the bottom surface 2a of the bumper <NUM>, as described above. Therefore, while the moveable body <NUM> is on the road surface 3a, the display <NUM> indicating the course trajectory and the obstacle <NUM> are not in contact, as illustrated in <FIG>. In other words, from the user's perspective, the moveable body <NUM> and the obstacle <NUM> appear not to come into contact. As described above, however, the obstacle <NUM> and the bottom surface 2a of the bumper <NUM> are in contact at the road surface 3b. Such a method therefore cannot properly enable the user to recognize the possibility of contact between the moveable body <NUM> and the obstacle <NUM>.

The processor <NUM> corrects the display indicating the course trajectory in accordance with the height difference of the road surface <NUM>. Specifically, the processor <NUM> corrects the vertical position of the display indicating the course trajectory in accordance with the height difference of the road surface <NUM>. The correction of the display indicating the course trajectory is described below with reference to <FIG> and <FIG>.

As illustrated in <FIG>, the height from the road surface <NUM> of the imaging unit <NUM> provided in the moveable body <NUM> is designated H. The height H of the imaging unit <NUM> from the road surface <NUM> is a unique, known value. The height of the imaging unit <NUM> from the road surface 3b is designated h2. The correction value in the height direction of the display indicating the course trajectory is designated x. The correction value x corresponds to the height of the road surface 3c. The distance from the imaging unit <NUM> to the obstacle <NUM> is designated d. The distance d is the distance between the imaging unit <NUM> and the contact point between the obstacle <NUM> and the road surface 3b as viewed from the imaging unit <NUM>. The angle between the horizontal direction and the direction from the imaging unit <NUM> to the obstacle <NUM> is designated β.

Upon detecting the obstacle <NUM>, the processor <NUM> calculates the distance d and the angle β. Details of the method for calculating the distance d and the angle β are provided below. The processor <NUM> calculates the height h2 of the imaging unit <NUM> from the road surface 3b in accordance with Expression (<NUM>) below based on the calculated distance d and angle β.

Upon calculating the height h2 of the imaging unit <NUM> from the road surface 3b, the processor <NUM> calculates the correction value x in accordance with Expression (<NUM>) below.

The processor <NUM> corrects the display <NUM> indicating the course trajectory of the bottom surface 2a of the bumper <NUM> based on the calculated correction value x. Specifically, the processor <NUM> overlays a display 100a indicating the course trajectory on the surrounding image as a flat shape extending in the travel direction of the moveable body <NUM> at a position corresponding to the height of the bottom surface 2a of the bumper <NUM> from the road surface 3a, as illustrated in <FIG> and <FIG>. The processor <NUM> also overlays a display 100b indicating the course trajectory on the surrounding image as a flat shape extending towards the obstacle <NUM> from ahead of the obstacle <NUM> at a position lower, by the correction value x, than the display 100a indicating the course trajectory. The processor <NUM> also overlays a display 100c indicating the course trajectory that connects the display 100a indicating the course trajectory and the display 100b indicating the course trajectory.

When the display 100b indicating the course trajectory and the obstacle <NUM> are in contact as illustrated in <FIG>, the processor <NUM> changes the display 100b indicating the course trajectory at the portion where the display 100b indicating the course trajectory and the obstacle <NUM> are in contact, as illustrated in <FIG>. In this way, the user can properly recognize whether there is a possibility of contact in the height direction between the obstacle <NUM> and the moveable body <NUM> even when the road surface <NUM> has a height difference.

In <FIG>, an example of the road surface <NUM> sloping downward in the travel direction of the moveable body <NUM> has been described, but the processor <NUM> also calculates the correction value x by a similar method when the road surface <NUM> slopes upward in the travel direction of the moveable body <NUM>, as illustrated in <FIG>.

In <FIG>, the road surface <NUM> is, for example, formed by a horizontal road surface 3a, a horizontal road surface 3b at a higher position than the road surface 3a, and an upward-sloping road surface 3c that connects the road surface 3a and the road surface 3b.

As illustrated in <FIG>, the processor <NUM> also calculates the correction value x by the above-described Expressions (<NUM>), (<NUM>) when the road surface <NUM> slopes upward. The processor <NUM> then corrects the display <NUM> indicating the course trajectory based on the calculated correction value x. Specifically, the processor <NUM> overlays a display 100a indicating the course trajectory on the surrounding image as a flat shape extending in the travel direction of the moveable body <NUM> at a position corresponding to the height of the bottom surface 2a of the bumper <NUM> from the road surface 3a. The processor <NUM> also overlays a display 100b indicating the course trajectory on the surrounding image as a flat shape extending towards the obstacle <NUM> from ahead of the obstacle <NUM> at a position higher, by the correction value x, than the display 100a indicating the course trajectory. The processor <NUM> also overlays a display 100c indicating the course trajectory that connects the display 100a indicating the course trajectory and the display 100b indicating the course trajectory. In this way, the processor <NUM> enables the user to properly recognize whether there is a possibility of contact in the height direction between the obstacle <NUM> and the moveable body <NUM> even when the road surface <NUM> slopes upward.

Details of the method for calculating the above-described distance d and angle β are now provided.

The processor <NUM> detects the distance d and the angle β using a stereo camera, a Light Detection and Ranging (Lidar) apparatus, an infrared sensor, or the like. A Lidar apparatus is an apparatus for emitting pulsed laser light and measuring scattered light from the laser light to detect the distance and the like to an object. The processor <NUM> may use the calculation method described with reference to <FIG> to calculate the distance d and the angle β.

As illustrated in <FIG>, a three-dimensional coordinate system (XYZ coordinate system) with the origin O at the optical center of the lens included in the imaging unit <NUM> is considered. In <FIG>, the optical axis of the imaging unit <NUM> is assumed to match the z-axis. The focal length, in the z-axis direction, of the lens provided in the imaging unit <NUM> is designated f. It is assumed that, along with movement of the moveable body <NUM>, a feature point of the obstacle moves from the perspective of the imaging unit <NUM> in a plane, which is a distance Zo from the origin O along the z-axis, from position P<NUM>(X<NUM>, Y<NUM>, Z<NUM>) to position P<NUM>(X<NUM>, Y<NUM>, Z<NUM>) by a movement amount ΔD in a direction rotated by angle θ relative to the straight direction. The angle θ and the movement amount ΔD are known from the movement amount of the moveable body <NUM>. In this case, the feature point of the obstacle moves in the z-axis direction from position pi(ui, v<NUM>) to position p<NUM>(u<NUM>, v<NUM>) in the U-V plane, which is at a distance equal to the focal length f from the origin O.

In the coordinate system illustrated in <FIG>, the processor <NUM> calculates Z<NUM> in accordance with Expression (<NUM>) below.

Next, the processor <NUM> calculates the coordinates of position P<NUM>(X<NUM>, Y<NUM>, Z<NUM>) in accordance with Expressions (<NUM>) to (<NUM>) below. <MAT> <MAT> <MAT>.

The processor <NUM> then calculates the distance d and the angle β based on the calculated coordinates of position P<NUM>(X<NUM>, Y<NUM>, Z<NUM>).

In <FIG>, the correction of the display <NUM> indicating the course trajectory of the bottom surface (bottom surface 2a of the bumper <NUM>) of the vehicle body of the moveable body <NUM> when the road surface <NUM> has a height difference has been described. When the road surface <NUM> has a height difference, the processor <NUM> may also correct the display <NUM> indicating the course trajectory of the upper surface of the vehicle body of the moveable body <NUM> based on the correction value x. The correction of the display <NUM> indicating the course trajectory of the upper surface of the vehicle body of the moveable body <NUM> is described below with reference to <FIG>.

In <FIG>, the road surface <NUM> is formed by a horizontal road surface 3a, a horizontal road surface 3b at a lower position than the road surface 3a, and a downward-sloping road surface 3c that connects the road surface 3a and the road surface 3b. An obstacle <NUM> is present above the road surface 3b. While the moveable body <NUM> is on the road surface 3a, a roof <NUM> of the moveable body <NUM> is assumed to be higher than the bottom surface of the obstacle <NUM>. In other words, when the obstacle <NUM> is viewed from the moveable body <NUM> while the moveable body <NUM> is on the road surface 3a, the roof <NUM> of the moveable body <NUM> appears to be in contact with the obstacle <NUM>.

The processor <NUM> calculates the correction value x using a similar method as the method described with reference to <FIG>. In <FIG>, however, the processor <NUM> calculates the distance between the imaging unit <NUM> and a point 3b' on the road surface 3b directly below the edge 201a of the obstacle <NUM> closest to the moveable body <NUM> as the distance d. The processor <NUM> calculates the angle between the horizontal direction and the direction from the imaging unit <NUM> towards the point 3b' as the angle β. The point 3b' on the road surface 3b directly below the edge 201a of the obstacle <NUM> is, for example, calculated based on the surrounding image captured by the imaging unit <NUM>.

Next, the processor <NUM> corrects the display <NUM> indicating the course trajectory of the upper surface (roof <NUM>) of the vehicle body of the moveable body <NUM>. Specifically, the processor <NUM> overlays a display 101a indicating the course trajectory on the surrounding image as a flat shape extending in the travel direction of the moveable body <NUM> at a position corresponding to the height of a rear edge 6a of the roof <NUM> from the road surface 3a. The processor <NUM> also overlays a display 101b indicating the course trajectory on the surrounding image as a flat shape extending towards the obstacle <NUM> from ahead of the obstacle <NUM> at a position lower, by the correction value x, than the display 101a indicating the course trajectory. The processor <NUM> also overlays a display 101c indicating the course trajectory that connects the display 101a indicating the course trajectory and the display 101b indicating the course trajectory. In this way, the display indicating the course trajectory of the upper surface of the vehicle body of the moveable body <NUM> is also corrected in accordance with the height difference of the road surface <NUM>.

An example of the obstacle <NUM> and the display 101b indicating the course trajectory not coming in contact has been illustrated in <FIG>, but the processor <NUM> may change a portion of the display 101b indicating the course trajectory when the obstacle <NUM> and the display 101b indicating the course trajectory are in contact.

The judgment of whether the obstacle <NUM> and the display 101b indicating the course trajectory are in contact may be made as follows, for example. As illustrated in <FIG>, the height from the imaging unit <NUM> to the obstacle <NUM> is designated H', the distance from the imaging unit <NUM> to the edge 201a of the obstacle <NUM> is designated D, and the angle between the horizontal direction and the direction from the imaging unit <NUM> towards the edge 201a of the obstacle <NUM> is designated α. The height of the roof <NUM> of the moveable body <NUM> from the road surface <NUM> is designated h3.

The processor <NUM> calculates the distance D and the angle α. The distance D and the angle α are calculated by the method described with reference to <FIG>. The processor <NUM> calculates the height H' from the imaging unit <NUM> to the obstacle <NUM> in accordance with Expression (<NUM>) below.

As illustrated in <FIG>, the height of the bottom surface of the obstacle <NUM> is represented as H + H'. By comparing the height (H + H') of the obstacle <NUM> and the height h3 of the roof <NUM>, the processor <NUM> judges whether the obstacle <NUM> and the display 101b indicating the course trajectory are in contact.

<FIG> illustrates an example of the edge 6a of the roof <NUM> of the moveable body <NUM> being directly above or near the contact point between the road surface <NUM> and a tire <NUM>. In some moveable bodies <NUM>, however, the edge 6a of the roof <NUM> of the moveable body <NUM> may be provided farther back than the contact point between the road surface <NUM> and the tire <NUM>. With reference to <FIG>, the overlay of a display <NUM> indicating the course trajectory on a surrounding image is described for a moveable body <NUM> in which the edge 6a of the roof <NUM> is provided farther back than the contact point between the road surface <NUM> and the tire <NUM>. In <FIG>, the roof <NUM> of the moveable body <NUM> is higher than the bottom surface of the obstacle <NUM> while the moveable body <NUM> is on the road surface 3a, as in <FIG>. In other words, when the obstacle <NUM> is viewed from the moveable body <NUM> while the moveable body <NUM> is on the road surface 3a, the roof <NUM> of the moveable body <NUM> appears to be in contact with the obstacle <NUM>.

In <FIG>, the edge 6a of the roof <NUM> of the moveable body <NUM> is provided farther back by a distance of L from the contact point between the tire <NUM> and the road surface <NUM>. The distance L is a unique, known value.

First, the processor <NUM> calculates the correction value x. Next, the processor <NUM> judges whether the height of the point that is the distance L, towards the moveable body <NUM>, from the point 3b' on the road surface 3b directly below the edge 201a of the obstacle <NUM> and the height of the contact point of the tire <NUM> are the same.

While the moveable body <NUM> is on the road surface 3a, the roof <NUM> of the moveable body <NUM> is higher than the bottom surface of the obstacle <NUM> in <FIG>, as described above. Therefore, when a point that is the distance L towards the moveable body <NUM> from the point 3b' on the road surface 3b lies on the road surface 3a, the edge 6a of the roof <NUM> of the moveable body <NUM> and the obstacle <NUM> come into contact upon the moveable body <NUM> moving to the point. In this case, the processor <NUM> overlays the display <NUM> indicating the course trajectory extending from the edge 6a of the roof <NUM> to the obstacle <NUM> on the surrounding image at a position corresponding to the height of the roof <NUM> from the road surface <NUM>, without correcting the display <NUM> indicating the course trajectory using the correction value x. In this way, the user can recognize that there is a possibility of contact with the obstacle <NUM>.

Next, an image processing method used in the image processing apparatus <NUM> is described with reference to the flowchart in <FIG>. The image processing apparatus <NUM> repeats the flow illustrated in <FIG> at predetermined time intervals, for example.

The interface <NUM> acquires a surrounding image captured by the imaging unit <NUM> (step S11).

The processor <NUM> overlays a display indicating the course trajectory of a specific portion of the moveable body <NUM> in the travel direction of the moveable body <NUM> on the acquired surrounding image at a position corresponding to the height of the specific portion from the road surface <NUM> (step S12).

Next, the processor <NUM> judges whether an obstacle is present in the travel direction of the moveable body <NUM> (step S13).

When it is judged that an obstacle is not present in the travel direction of the moveable body <NUM> (step S13: No), the processor <NUM> ends the processing without changing the display indicating the course trajectory overlaid on the surrounding image.

When it is judged that an obstacle is present in the travel direction of the moveable body <NUM> (step S13: Yes), the processor <NUM> judges whether the obstacle and the course trajectory of the specific portion of the moveable body <NUM> are in contact (step S14).

When it is judged that the obstacle and the course trajectory are not in contact (step S14: No), the processor <NUM> ends the processing without changing the display indicating the course trajectory overlaid on the surrounding image.

When it is judged that the obstacle and the course trajectory are in contact (step S14: Yes), the processor <NUM> changes the display indicating the course trajectory. Here, the processor <NUM> changes the portion, within the display indicating the course trajectory, where the image of the obstacle included in the surrounding image is in contact with the display of the course trajectory of the specific portion.

The image processing apparatus <NUM> thus includes the interface <NUM> configured to acquire a surrounding image of the moveable body <NUM> and the processor <NUM> configured to overlay a display indicating a course trajectory of a specific portion of the moveable body <NUM> in a travel direction of the moveable body <NUM> on the surrounding image at a position corresponding to the height of the specific portion from a road surface. The processor <NUM> is configured to change the display indicating the course trajectory when an obstacle, included in the surrounding image and present in the travel direction of the moveable body <NUM>, and the course trajectory of the specific portion are in contact.

Claim 1:
An image processing apparatus (<NUM>) comprising:
an interface (<NUM>) configured to acquire a surrounding image of a moveable body (<NUM>); and
a processor (<NUM>) configured to overlay a display (100a, 100b, 100c, 101a, 101b, 101c) indicating a course trajectory of a specific portion of the moveable body (<NUM>) in a travel direction of the moveable body (<NUM>) on the surrounding image at a position corresponding to a height of the specific portion from a road surface (<NUM>; 3a, 3b, 3c), wherein
the processor (<NUM>) is configured to correct a vertical position of the display (100a, 100b, 100c, 101a, 101b, 101c) indicating the course trajectory in accordance with a height difference of the road surface (<NUM>; 3a, 3b, 3c), and
to change the display (100a, 100b, 100c, 101a, 101b, 101c) indicating the course trajectory, when an obstacle (<NUM>, <NUM>), included in the surrounding image and present in the travel direction of the moveable body (<NUM>), and the course trajectory of the specific portion are in contact, wherein the processor (<NUM>) is configured to change a portion, within the display (100a, 100b, 100c, 101a, 101b, 101c) indicating the course trajectory, where an image of the obstacle (<NUM>, <NUM>) included in the surrounding image and the display (100a, 100b, 100c, 101a, 101b, 101c) of the course trajectory of the specific portion are in contact.