Source: https://patents.google.com/patent/US8593454B2/en
Timestamp: 2020-07-09 12:23:03
Document Index: 319864470

Matched Legal Cases: ['Application No. 2010', 'art 11', 'art 10', 'art 10', 'art 10', 'art 10', 'art 10', 'art 10', 'art 10', 'art 10', 'art 10', 'art 10', 'art 10', 'art 10', 'art 11', 'art 11', 'art 11', 'art 10', 'art 10', 'art 10', 'art 1', 'art 11', 'art 11', 'art 11']

US8593454B2 - Image generation device and operation support system - Google Patents
US8593454B2
US8593454B2 US13/649,369 US201213649369A US8593454B2 US 8593454 B2 US8593454 B2 US 8593454B2 US 201213649369 A US201213649369 A US 201213649369A US 8593454 B2 US8593454 B2 US 8593454B2
US13/649,369
US20130033494A1 (en
2010-04-12 Priority to JP2010-091657 priority Critical
2010-04-12 Priority to JP2010091657A priority patent/JP5550970B2/en
2011-04-08 Priority to PCT/JP2011/058898 priority patent/WO2011129276A1/en
2013-02-07 Publication of US20130033494A1 publication Critical patent/US20130033494A1/en
2013-11-26 Publication of US8593454B2 publication Critical patent/US8593454B2/en
This is a continuation application filed under 35 U.S.C. 111(a) claiming benefit under 35 U.S.C. 120 and 365(c) of International Application PCT/JP2011/058898, filed on Apr. 8, 2011, designating the U.S., which claims priority to Japanese Patent Application No. 2010-091657. The entire contents of the foregoing applications are incorporated herein by reference.
The image generation device disclosed in Patent Document 1 projects an image taken by a camera mounted on a vehicle onto a three-dimensional space model configured by a plurality of plane surfaces or curved surfaces that surround the vehicle. The image generation device generates a visual point conversion image using the image projected onto the space model, and displays the produced visual point conversion image to a driver. The visual point conversion image is an image of a combination of a road surface image, which virtually reflects a state of a road taken from directly above, and a horizontal image, which reflects a horizontal direction image. Thereby, the image generation device can relate, when the driver driving the vehicle looks at the visual point conversion image, an object in the visual point conversion image to an object actually existing outside the vehicle without giving an uncomfortable feeling.
The image generation device disclosed in Patent Document 1 generates a space model based on a feature point on a road surface that is image-taken by a camera, and corrects the space model in accordance with a movement of a vehicle (a change in a positional relationship between the vehicle and the feature point). A problem not considered is that, when generating or correcting a space model, an object in an overlapping area between image-taking ranges of two cameras disappears when projecting that object on a space model. Therefore, because the space model is not corrected in order to prevent such an image disappearance, a view point conversion image, which includes an object in an overlapping range, cannot be generated properly.
It is an object of the present invention to provide an image generation device, which generates an output image using a space model that prevents disappearance of an object from an image, the object existing in an area where image-taking ranges of two cameras overlap with each other, and an operation support system using the device.
In order to achieve the above-mentioned objects, there is provided according to an aspect of the present invention an image generation device that generates an output image based on a plurality of input images image-taken by a plurality of image-taking parts mounted to a body to be operated, the image generation device including: a coordinates correspondence part configured to cause coordinates on a columnar space model, which is arranged to surround the body to be operated and having a center axis and a side surface, to correspond to coordinates on a plurality of input image planes on which the plurality of input images are positioned, respectively; and an output image generation part configured to generate the output image by causing values of the coordinates on the plurality of input image planes to correspond to values of the coordinates on an output image plane on which the output image is positioned through coordinates on the columnar space model, wherein a distance between the center axis and the side surface of the columnar space model is determined in accordance with installation positions of the image-taking parts.
There is provided according to another aspect of the present invention an operation support system that supports a movement or an operation of a body to be operated, including: the above-mentioned image generation device; and a display part configured to display the output image generated by the image generation device.
According to the present invention, it is possible to provide an image generation device, which generates an output image using a space model that prevents disappearance of an object from an image, the object existing in an area where image-taking ranges of two cameras overlap with each other, and an operation support system using the device.
FIG. 16A is a view illustrating a positional relationship between three cameras and a space model MD when viewing a shovel from above.
FIG. 17 is a view illustrating processing-target images generated by the image generation device based on the input images taken by the cameras installed as illustrated in FIG. 16A.
FIG. 18 is a view illustrating processing-target images generated by the image generation device based on the input images taken by the cameras 2 installed as illustrated in FIG. 16B.
FIGS. 19A and 19B are views for explaining an example of a procedure of determining a range of a value which the radius of the space model MD can take.
FIG. 20 is a view illustrating a positional relationship between image-taking range of the cameras and a space model.
FIG. 2 is a side view of an excavator 60 to which the image generation device is mounted. The excavator 60 includes a lower-part running body 61 of a crawler type, a turning mechanism 62 and an upper-part turning body 63. The upper-part turning body 63 is mounted on the lower-part running body 61 via the turning mechanism 62 so as to be turnable about a tuning axis PV.
The “processing-target image” is an image, which is generated based on an input image and to be subjected to an image conversion process (for example, a scale conversion, an affine conversion, a distortion conversion, a viewpoint conversion processing). For example, an input image, which is an input image taken by a camera that takes an image of a ground surface from above and contains an image (for example, an empty part) in a horizontal direction according to a wide view angle, is used in an image conversion process. In such a case, the input image is projected onto a predetermined space model so that a horizontal image thereof is not displayed unnaturally (for example, is not handled as an empty part on a ground surface). Then, an image suitable for the image conversion process can be obtained by re-projecting a projection image projected on the space model onto a different two-dimensional plane. It should be noted that the processing-target image may be used as an output image as it is without applying an image conversion process.
The “space model” is a target object of an input image, and includes at least a plane surface or a curved surface (for example, a plane surface parallel to the processing-target image plane or a plane surface or curved surface that forms an angle with the processing-target image plane) other than a processing-target image plane, which is a plane surface on which the processing-target image is positioned.
It should be noted that the image generation device 100 may generates generate an output image by applying an image conversion process to a projection image projected onto the space model without generating a processing-target image. Moreover, the projection image may be used as an output image as it is without being subjected to an image conversion process.
FIG. 4 is a view illustrating an example of a relationship between the space model MD and the processing-target image plane. In FIG. 4, the processing-target image plane R3 is a plane containing the plane surface area R1 of the space model MD. It should be noted that although the space model MD is illustrated as a cylindrical form, which is different from the half-cylindrical form as illustrated in FIG. 3, for the purpose of clarification in FIG. 4, the space model MD may be either of the half-cylindrical form or the cylindrical form. The same applies in figures mentioned below. Additionally, the processing-target image plane R3 may be a circular area, which contains the plane surface area R1 of the space model MD, or may be an annular area, which does not contain the plane surface area R1 of the space model MD.
It should be noted that the output image generation part 11 may generate the output image by changing a scale of the processing-target image without using a concept of a virtual camera.
If there are a plurality of cameras 2, each of the cameras 2 has an individual UVW coordinates system. Thereby, the coordinates correspondence part 10 translates and rotates the XYZ coordinates system with respect to each of the plurality of UVW coordinates systems.
It should be noted that when i, j and k are imaginary number units, the quaternion is a hypercomplex number satisfying the following condition.
X=Q z XQ* z [Formula 7]
Thereafter, the coordinates correspondence part 10 decomposes the image height h to a U-component and a V-component on the UV coordinates system according to an argument φ, and divides them by a numerical value corresponding to a pixel size per one pixel of the input image plane R4. Thereby, the coordinates correspondence part 10 can cause the coordinates P (P′) on the space model MD and the coordinates on the input image plane R4.
Because the coordinates correspondence part 10 operates the conversion of coordinates by using the quaternion, the coordinates correspondence part 10 provides an advantage in that a gimbal lock is not generated unlike a case where a conversion of coordinates is operated using an Euler angle. However, the coordinates correspondence part 10 is not limited to one performing an operation of conversion of coordinates using a quaternion, and the conversion of coordinates may be operated using an Euler angle.
If it is possible to cause a correspondence to coordinates on a plurality of input image planes R4, the coordinates correspondence part 10 may cause the coordinates P (P′) to correspond to the coordinates on the input image plane R4 with respect to a camera of which an incident angle is smallest, or may cause the coordinates P (P′) to correspond to the coordinates on the input image plane R4 selected by an operator.
FIGS. 6A and 6B are views for explaining correspondence between coordinates according to the coordinates correspondence part 10. FIG. 6A is a view illustrating a correspondence relationship between the coordinates on the input mage plane R4 of the camera 2 using a normal projection (h=f tan α) and the coordinates on the space model MD. The coordinates correspondence part 10 causes both coordinates to correspond to each other by causing each of the line segments, which connect coordinates on the input image plane R4 of the camera 2 and the coordinates on the space model MD corresponding to the coordinates on the input image plane R4, to pass the optical center C of the camera 2.
Specifically, the coordinates correspondence part 10 causes the coordinates on the input image plane to correspond to the coordinates on the space model MD based on a predetermined function (for example, an orthogonal projection (h=f sin α), a stereographic projection (h=2 f tan(α/2)), an equisolid angle projection (h=f sin(α/2)), an equidistant projection (h=fα), etc.). In this case, the line segment K1-L1 and the line segment K2-L2 do not pass the optical center C of the camera 2.
FIG. 6B is a view illustrating a correspondence relationship between the coordinates on the curved surface area R2 of the space model MD and the coordinates on the processing-target image plane R3. The coordinates correspondence part 10 introduces a group of parallel lines PL, which are a group of parallel lines PL positioned on the XZ-plane and form an angle p between the processing-target image plane R3, and causes both coordinates to correspond to each other so that both the coordinates on the curved surface area R2 of the space model MD and the coordinates on the processing-target image plane R3 corresponding to the coordinates on the curved surface area R2 are positioned on one of the parallel lines.
FIG. 6C is a view illustrating a correspondence relationship between the coordinates on the processing-target image plane R3 and the coordinates on the output image plane R5 of the virtual camera 2V using, as an example, a normal projection (h=f tan α). The coordinates correspondence part 10 causes both coordinates to correspond to each other so that each of the line segments connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing-target image plane R3 corresponding to the coordinates on the output image plane R5 passes the optical center CV of the virtual camera 2V.
Specifically, the output image generation part 11 causes the coordinates on the output image plane R5 to correspond to the coordinates on the processing-target image plane R3 based on a predetermined function (for example, an orthogonal projection (h=f sin α), a stereographic projection (h=2 f tan(α/2)), an equisolid angle projection (h=f sin(α/2)), an equidistant projection (h=fα), etc.). In this case, the line segment M1-N1 and the line segment M2-N2 do not pass the optical center CV of the virtual camera 2V.
As mentioned above, the output image generation part 11 causes the coordinates on the output image plane R5 to correspond to the coordinates on the processing-target image plane R3, and stores the coordinates on the output image plane R5 and the coordinates on the processing-target image R3 in the processing-target image—output image correspondence relation map 42 by relating them to each other. Then, the output image generation part 11 generates the output image to relate a value of each pixel in the output image to a value of each pixel in the input image while referring to the input image—space model correspondence relation map 40 and the space model—processing-target image correspondence relation map 41 stored in the coordinates correspondence part 10.
It should be noted that FIG. 6D is a view of a combination of FIG. 6A through FIG. 6C, and illustrates a mutual positional relationship between the camera 2, the virtual camera 2V, the plane surface area R1 and the curved surface area R2 of the space model MD, and the processing-target image plane R3.
As illustrated in FIGS. 8A and 8B, the intervals of the coordinates Ma through Md on the processing-target image plane R3 decease nonlinearly as the distance (height) between the start point of the group of auxiliary lines AL and the original point O increases. That is, a degree of decrease of each of the intervals increases as the distance between the curved surface area R2 of the space model MD and each of the coordinates Ma through Md increases. On the other hand, in the example illustrated in FIGS. 8A and 8B, because a conversion to the group of coordinates on the processing-target image plane R3 is not performed, the intervals of the group of coordinates on the plane surface area R1 of the space model MD do not change.
It should be noted that when an output image is generated directly based on the image projected on the space model MD, the image portion on the output image plane R5 corresponding to the image projected on the curved surface area R2 alone cannot be enlarged or reduced because the plane surface area R1 and the curved surface area R2 cannot be handled separately (because they cannot be separate objects to be enlarged or reduced).
Specifically, the coordinates correspondence part 10 acquires the coordinate point of the optical center C of the camera 2 using a normal projection (h=f tan α), and computes a point at which a line segment extending from a coordinate point on the space model MD, which, is a line segment passing the optical center C, intersects with the input image plane R4. Then, the coordinates corresponding part 10 derives a coordinate point on the input image plane R4 corresponding to the computed point as a coordinate point on the input image plane R4 corresponding to the coordinate point on the space model MD, and stores a correspondence relationship therebbetween in the input image—space model map 40.
On the other hand, if it is determined that all of the coordinate points are caused to correspond (YES of step S3), the control part 1 causes the processing-target image generation process to end and, thereafter, causes the output image generation process to start. Thereby, the output image generation part 11 causes the coordinates on the processing-target image plane R3 to the coordinates on the output image plane R5 to correspond (step S4).
Alternatively, when generating the output image using the virtual camera 2V using a normal projection (h=f tan α), the output image generation part 11 may compute, after acquiring the coordinate point of the optical center CV of the virtual camera 2V, a point at which a line segment extending from a coordinate point on the output image plane R5, which line segment passes the optical center CV, intersects with the processing-target image plane R3. Then, the output image generation part 11 may derive the coordinates on the processing-target image plane R3 corresponding to the computed point as a coordinate point on the processing-target image plane R3 corresponding to the coordinate point on the output image plane R5, and may store a correspondence relationship therebetween in the processing-target image—output image correspondence relation map 42.
Similarly, when changing a view point of the output image, the image generation device 100 is capable of generating an output image (view point conversion image) which is viewed from a desired view point by merely rewriting the processing-target image—output image correspondence relation map 42 by changing values of various parameters of the virtual camera 2V without rewriting the contents of the input image—space model correspondence relation map 40 and the space model—processing-target image correspondence relation map 41.
In FIG. 13B, similar to the positional relationship illustrated in FIG. 12B, the optical axis G1 of the backside camera 2B and the optical axis G2 of the right side camera 2R intersect with the plane surface area R1 of the space model MD and the plane (XY-plane) on which the processing-target image plane R3 is positioned, respectively. Moreover, the optical axis G1 and the optical axis G2 intersect with each other at a point J1 on the cylinder center axis (re-projection axis). It should be noted that the optical axis G1 and the optical axis G2 may be in a twisted positional relationship if the components, when it is projected on a plane parallel to the XY-plane, intersect with each other at a point on the cylinder center axis (re-projection axis).
The output image is trimmed to be in a circular shape so that the image when the excavator 60 performs a turning operation can be displayed without an uncomfortable feel. That is, the output image is displayed so that the center CTR of the circle is at the cylinder center axis of the space model, and also on the turning axis PV of the excavator 60, and the output image rotates about the center CTR thereof in response to the turning operation of the excavator 60. In this case, the cylinder center axis of the space model MD may be coincident with or not coincident with the re-projection axis.
A distance between a turning axis and the object OBJ3 is smaller than a radius of the space model MD. A distance between the turning radius and the object OBJ4 is larger than the radius of the space model MD.
In FIG. 16A, the image-taking range of the backside camera 2B is represented by a single dotted chain line of a sector shape, and the image-taking range of the right side camera 2R is represented by a dashed line of a sector shape. The object OBJ3 is in an area where the image-taking range of the backside camera 2B and the image-taking range of the right side camera 2R overlaps each other (hereinafter, referred to as “image-taking range overlapping area”), and is contained in a space inside the curved surface area R2 of the space model MD. The object OBJ4 is in the image-taking range overlapping area, and is contained outside the curved surface area R2 of the space model MD.
A part of coordinates of an input image portion corresponding to an image-taking range overlapping area in the input image taken by the backside camera 2B is associated with coordinates in an area DG1 (refer to FIG. 16A) of the plane surface area R1 and an area DY1 (refer to FIG. 16B) of the curved surface area R2 in the space model MD. Other parts (under ordinary circumstances, parts that can be associated with coordinates in an area DG2 (refer to FIG. 16A) of the plane surface area R1 and in an area DY2 (refer to FIG. 16B) of the curved surface area R2 in the space model MD) are not associated with coordinates on the space model MD. This is because the coordinates in a part of the input image portion corresponding to the image-taking range overlapping area in the input image taken by the right side camera 2R are associated with coordinates in the areas DG2 and DY2.
Similarly, a part of coordinates of an input image portion corresponding to the image-taking range overlapping area in the input image taken by the right side camera 2R is associated with coordinates in the area DG2 (refer to FIG. 16A) of the plane surface area R1 and the area DY2 (refer to FIG. 16B) of the curved surface area R2 in the space model MD. Other parts (under ordinary circumstances, parts that can be associated with coordinates in the area DG1 (refer to FIG. 16A) of the plane surface area R1 and in the area DY1 (refer to FIG. 16B) of the curved surface area R2 in the space model MD) are not associated with coordinates on the space model MD. This is because the coordinates on a part of the input image portion corresponding to the image-taking range overlapping area in the input image taken by the backside camera 2B are associated with coordinates in the areas DG1 and DY1.
It should be noted that as illustrated in FIG. 17 or FIG. 18, the areas DY1 and DY2 of the curved surface area R2 in the space model MD are associated with areas DW1 and DW2 on the processing-target plane R3
According to the above-mentioned association, the input image portions taken by the backside camera R2 and the right side camera 2R are projected on the areas DG1 and DG2 of the plane surface area R1 in the space model MD, respectively, the input image portions representing the object OBJ3 contained in a space inside the curved surface area R2 of the space model MD within the image-taking range overlapping area. In this projected image, a small part contacting the road surface remains as a road surface pattern, and the rest of a large part (an image portion viewing the object OBJ3 from a horizontal direction, and a part which is not handled as a road surface pattern) extending from the road surface disappears (FIG. 16C).
On the other hand, according to the above-mentioned association, the input image portions taken by the backside camera 2B and the right side camera 2R are projected on the areas DY1 and DY2 in the curved surface area R2 of the space model MD, respectively, the input image portions representing the object OBJ4 contained in a space outside the curved surface area R2 of the space model MD within the image-taking range overlapping area. Further, the object OBJ4 is projected on each of the areas DW1 and DW2 of the processing-target image plane R3, and appears as two objects OBJ4-1 and OBJ4-2 in the areas DW1 and DW2, respectively (refer to FIG. 16C).
The description of the positional relationship between the image-taking range of the camera 2 (backside camera 2B and right side camera 2R) and the space model MD is continued with reference to FIG. 17 and FIG. 18.
FIG. 17 and FIG. 18 are views, similar to FIG. 16C, illustrating processing-target images generated by the image generation device 100 based on the input images taken by the cameras 2 installed as illustrated in FIG. 16A and FIG. 16B. The examples illustrated in FIGS. 17 and 18 differ in the arrangement (a radius of the space model MD) of the curved surface area R2 of the space model MD, but the same in other points.
The space model MD illustrated in FIG. 17 is formed so that a distance between the cylinder center axis of the space model MD and a point at which the overlap of the image-taking range of the backside camera 2B and the image-taking range of the right side camera 2R begins is coincident with the radius of the space model MD. The areas DG1 and DG2 do not exist in the plane surface area R1 of the space model MD, and, as a result, disappearance of the object OBJ3 does not occur.
Specifically, The object OBJ3 existing in the space inside the curved surface area R2 in the space model MD in FIG. 16A exists in the apace outside the curved surface area R2 in FIG. 17. As illustrated in FIG. 17, the input image portions taken by the backside camera 2B and the right side camera 2B, respectively, are projected on the areas DY1 and DY2 in the curved area R2 of the space model, respectively, the input image portions representing the object OBJ3 contained in the space outside the curved surface area R2 of the space model MD within the image-taking range overlapping area. Further, the object OBJ3 is projected on each of the areas DW1 and DW2 of the processing-target image plane R3, and appears in each of the areas DW1 and DW2 by being divided into two objects OBJ3-1 and OBJ3-2. Therefore, the object OBJ3 is prevented from disappearing.
On the other hand, the space model MD illustrated in FIG. 18 is formed by increasing the radius of the space model so that the object OBJ4 existing in the space outside the curved surface area R2 in the space model MD in FIG. 16A is contained in the space inside the curved surface area R2. As a result, the plane surface area R1 (an area displayed as a road surface image) can be enlarged, but the object OBJ4 is also caused to disappear as well as the object OBJ3 in the areas DG1 and DG2 in the enlarged plane surface area R1.
As mentioned above, the image generation device 100 causes objects existing at positions farther from the excavator 60 in the image-taking range overlapping area to disappear as the radius of the space model MD is larger. On the other hand, there is a tendency of displaying the objects position closer to the excavator 60 in the image-taking range overlapping area by dividing it into two pieces as the radius of the space model MD is smaller.
The image generation device 100 derives an optimum radius of the space model MD while considering the above-mentioned tendency.
For example, the camera 2 attached to the upper-part turning body 63 of the excavator 60 is installed at a position (for example, height of 2 meters) higher than a standard body height of operators who work in a surrounding area of the excavator 60. In this case, an image of looking down an operator within a predetermined distance range from the excavator 60 is acquired, and, thus, the image generation device 100 can handle the image of the operator as not a three-dimensional object but a road surface pattern. Here, if the camera is installed at a lower position, the operator is viewed from the horizontal direction, and, thus, the operator must be handled as a three-dimensional object in order to display the image of the operator without an uncomfortable feel. As compared to such a case, when the camera 2 is installed at a position higher than the body height of the operator, disappearance of the image of the operator can be suppressed even if the operator is included in the space inside the curved surface area R2 by increasing the radius of the space model MD.
As mentioned above, the image generation device 100 is capable of generating an output image without giving an uncomfortable feel while suppressing disappearance of an object existing in the image-taking range overlapping area of the two cameras by determining a radius of the space model MD in response to the installation position of the cameras (for example, installation height).
FIGS. 19A and 19B are views for explaining an example of a procedure of determining a range of a value which the radius of the space model MD can take. FIG. 19A is a plan view of the excavator 60 when viewing from above, and FIG. 19B is a side view of the excavator 60 when viewing from a transverse direction.
FIGS. 19A and 19B illustrate that each of the optical axis G1 of the backside camera 2B and the optical axis G2 of the right side camera 2R intersects with a road surface and the both optical axes intersect with each other at a point J1 on the cylinder center axis of the space model MD. The backside camera 2B and the right side camera 2R are attached to the vehicle body in an obliquely downward direction so that they do not protrude out of the vehicle body of the excavator 60 and an image of the road surface in the vicinity of the vehicle body can be taken, and, thus, they take an image by including a part of the vehicle body in the image.
A description will be given first of a procedure of determining a minimum value of values which the radius of the space model MD can take.
Each of the backside camera 2B and the right side camera 2R has an image-taking range indicated by hatching as illustrated in FIG. 19A. Coarse hatching corresponds to the image-taking range of the backside camera 2B, and fine hatching corresponds to the image-taking range of the right side camera 2R. Each of the image-taking ranges includes an area of which an image cannot be taken as it is hidden behind the vehicle body of the excavator 60 as illustrated in FIG. 19B. In FIGS. 19A and 19B, this area is a black marked area, and an area DS1 corresponds to the backside camera 2R and an area DS2 corresponds to the right side camera 2R. It should be noted that when the areas DS1 and DS2 are displayed in the output image, the areas DS1 and DS2 may be marked out in a warning color (for example, black) so that an operator can recognize that it is an area which is hidden behind the vehicle body of the excavator 60 and an image thereof cannot be taken.
Each of the areas DS1 and DS2 can be set by drawing boundary lines BL1 and BL2 on the road surface at positions away from the excavator 60 by a predetermined distance as illustrated in FIG. 19A, the boundary lines BL1 and BL2 delimiting a range, which is not hidden behind the vehicle body and an image thereof can be taken, and a range, which is hidden behind the vehicle body and an image thereof cannot be taken. The predetermined distance is a distance determined by an installation position of each of the backside camera 2B and the right side camera 2R.
The boundary lines BL1 and BL2 intersect with each other at an intersection point PT1 within the image-taking range overlapping area (an area where two kinds of hatching overlap). A distance between the intersection point PT1 and the cylinder center axis is used as a minimum value of the values which the space model MD takes. Thereby, a curved surface area R2 min can be provided, which is a curved surface area as a projection target which can prevent disappearance of an image of an operator existing in a surrounding area of the excavator 60, and which has a smallest radius from among the curved surface areas R2 of the space model MD.
Next, a description is given of a procedure of determining a maximum value of the values which the radius of the space model MD can take.
A person PSN is a cylindrical virtual object used to derive a maximum value of the values which the radius of the space model MD can take. The person PSN has a predetermined size (for example, an outer diameter when viewing from above is 600 millimeters), and is arranged adjacent to an outer side of the intersection point PT1 in the image-taking range overlapping area (outside in the radius direction when viewing from the cylinder center axis).
The backside camera 2B and the right side camera 2R draw auxiliary lines EL1 and EL2, respectively, as illustrated in FIG. 19, the auxiliary lines EL1 and EL2 pass the respective optical axes and are tangential lines of the circle indicating the person PSN.
The auxiliary lines EL1 and EL intersect with each other at an intersection point PT2 in the image-taking range overlapping area. A distance between the intersection point PT2 and the cylinder center is used as a maximum value of the values which the radius of the space model MD can take. Thereby, a curved surface area R2max having the maximum radius from among the curved surface areas R2 of the space model MD can be provided, the curved surface area R2max being a curved surface area as a projection target which can prevent disappearance of an image of an operator existing in a surrounding area of the excavator 60.
The curved surface area R2 of the space model MD actually used is set larger than the curved surface area R2 min and smaller than the curved surface area R2max by determining the range of the values which the radius of the space model can take as mentioned above. In this case, the image of the operator existing in the surrounding area of the excavator 60 is projected on the curved surface area R2 by at least a part of the body (typically, a heat part thereof) being divided into two portions, the image using the person PSN illustrated in FIGS. 19A and 19B and being simulated with a condition which can most easily cause the person PSN to disappear. Therefore, the image of the person PSN can escape from being disappeared completely.
Moreover, in a case where the excavator 60 is traveled in a direction of the camera 2 taking an image, the operator of the excavator 60 can see that the direction of traveling can be displayed with easy recognition in the output image as compared to a case where the excavator 60 is turned. Therefore, it is desirable that the image generation device 100 displays an output image which allows an easy recognition of the distance between the excavator 60 and an object existing in a traveling direction of the excavator 60 by increasing the area displayed as a road surface image by increasing the radius of the space model MD.
On the other hand, when turning the excavator 60, the driver of the excavator 60 can see that the direction of turning is more easily recognized in the output image as compared to a case where the excavator 60 is traveled in the image-taking direction of the camera 2. Therefore, it is desirable that the image generation device 100 displays an output image that can prevent disappearance of an object in an area surrounding the excavator 60 in the image-taking range overlapping area by reducing the radius of the space model MD. For example, the image generation device 100 displays an object in an area surrounding the excavator 60 in the image-taking range overlapping area by dividing the object into two pieces.
In order to properly use two processing-target images generated using two space models having different radii, the image generation device 100 may be configured so that the driver can selectively display one of an output image to be displayed when traveling the excavator 60 (hereinafter, referred to as “traveling output image”) and an output image to be displayed when turning the excavator 60 (hereinafter, referred to as “turning output image”. Alternatively, the turning output image may be automatically switched to the traveling output image when the excavator 60 starts to travel, and the traveling output image may be automatically switched to the turning output image when the shovel is stopped.
As mentioned-above, the image generation device 100 is capable of generating an output image having no uncomfortable feeling, which is suitable for an operation of the excavator 60, by determining the radius of the space model MD in response to operating conditions of the excavator 60.
In this case, the image generation device 100 may use, for example, a distance to a nearest object existing in the image-taking range overlapping area as the radius of the space model MD. Alternatively, the image generation device 100 may use as the radius of the space model MD a distance to a nearest object, which exists in the image-taking range overlapping area and has a distance to the cylinder center axis longer than a predetermined value (a value determined in accordance with an installation position (for example, an installation height) of the camera). This is because an image of an object of which the distance is shorter than the predetermined value is handled as a road surface pattern, thereby escaping from being disappeared.
Next, a description is given, with reference to FIG. 20, of a function of the image generation device 100 to display distinguishably an input image corresponding to the image-taking range overlapping area from input images corresponding to other areas. It should be noted that, similar to FIG. 18, FIG. 20 is a view illustrating an example of display of the processing-target image. In FIG. 20, arrangement of the camera 2, arrangement of the space model MD, and the arrangement of the objects OBJ3 and OBJ4 are the same as the example illustrated in FIG. 17.
In FIG. 20, the image generation device 100 emphatically displays an image portion in the area DW1 on the processing-target image plane R3, which is associated with an image portion corresponding to the image-taking range overlapping area of which an image is taken by the backside camera 2B, so that the image portion in the area DW1 is distinguishable from other image portions.
In this case, the image generation device 100 does not emphatically display an image portion in the area DW2 on the processing-target image overlapping area, which is associated with an image portion corresponding to the image-taking range overlapping area of which an image is taken by the right side camera 2R.
The objects OBJ3-1 and OBJ3-2 appearing in the areas DW1 and DW2 do not represent that the two objects actually exist, but merely represent the single object OBJ3 being divided into two pieces on an image thereof and doubly displayed. Thus, the image generation device 100 is capable of causing a driver to recognize that one of the areas DW1 and DW2 corresponds to the image-taking range overlapping area by emphatically displaying only the one of the areas DW1 and DW2. Further, the image generation device 100 is capable of causing the driver to recognize that both the object OBJ3-1 appearing in the area DW1 and the object OBJ3-2 appearing in the area DW2 are derived from the single object OBJ3. The same is applicable to the objects OBJ4-1 and OBJ4-2.
It should be noted that the image generation device 100 may emphatically display the image portions in both the areas DW1 and DW2 on the processing-target image plane R3.
Moreover, the image generation device 100 may emphatically displays both the areas DW1 and DW2 or one of them on the processing-target image plane R3 by increasing or decreasing brightness thereof. Alternatively, the image generation device 100 may emphatically display both the areas DW1 and DW2 or one of them on the processing-target image plane R3 by changing a color image into a monochrome image (conversion into a gray scale).
Moreover, the image generation device 100 may display three image portions distinguishable from each other, the three image portions include the image portion in the area DW1 on the processing-target image plane R3, the image portion in the area DW2 on the processing-target image plane 3, and other image portions. The image portion in the area DW1 on the processing-target image plane R3 is a portion associated with the image portion corresponding to the image-taking range overlapping area of which image is taken by the backside camera 2B. Moreover, the image portion in the area DW2 on the processing-target image plane R3 is a portion associated with the image portion corresponding to the image-taking range overlapping area of which image is taken by the right side camera 2R.
It should be noted that the positional relationship between the image-taking range of the camera 2 (right side camera 2R and backside camera 2B) and the space model MD illustrated in FIG. 16A through FIG. 20 and the action and effect thereof are obtained when the image generation device 100 generates the processing-target image. However, the same action and effect can be obtained even if the image generation device 100 does not generate the processing-target image (in a case where the processing-target image plane R3 does not exist). In this case, the processing-target images in FIG. 16, FIG. 17, FIG. 18 and FIG. 20 may be alternatively read as an output image generated using an image projected on the space model MD.
Although the image generation device 100 uses the cylindrical space model MD as a space model in the above-mentioned embodiments, the image generation device may use a space model having other columnar shapes such as a polygonal column, etc., or may use a space model constituted by two planes including a bottom surface and a side surface. Alternatively, the image generation device 100 may be a space model having only a side surface.
1. An image generation device that generates an output image based on a plurality of input images image-taken by a plurality of image-taking parts mounted to a body to be operated, the image generation device comprising:
a coordinates correspondence part configured to cause coordinates on a columnar space model, which is arranged to surround said body to be operated and having a center axis and a side surface, to correspond to coordinates on a plurality of input image planes on which the plurality of input images are positioned, respectively; and
an output image generation part configured to generate said output image by causing values of the coordinates on said plurality of input image planes to correspond to values of the coordinates on an output image plane on which said output image is positioned through coordinates on said columnar space model,
wherein a distance between the center axis and the side surface of said columnar space model is determined in accordance with installation positions of said image-taking parts, and
wherein said body to be operated is a machine, which is capable of traveling and turning, and the distance between the center axis and the side surface of said columnar space model when turning is set smaller than the distance when traveling.
US13/649,369 2010-04-12 2012-10-11 Image generation device and operation support system Active US8593454B2 (en)
JP2010-091657 2010-04-12
JP2010091657A JP5550970B2 (en) 2010-04-12 2010-04-12 Image generating apparatus and operation support system
PCT/JP2011/058898 WO2011129276A1 (en) 2010-04-12 2011-04-08 Image generation device and operation support system
PCT/JP2011/058898 Continuation WO2011129276A1 (en) 2010-04-12 2011-04-08 Image generation device and operation support system
US20130033494A1 US20130033494A1 (en) 2013-02-07
US8593454B2 true US8593454B2 (en) 2013-11-26
ID=44798652
US13/649,369 Active US8593454B2 (en) 2010-04-12 2012-10-11 Image generation device and operation support system
US (1) US8593454B2 (en)
EP (1) EP2560140A4 (en)
JP (1) JP5550970B2 (en)
KR (1) KR101475583B1 (en)
CN (1) CN102792334B (en)
WO (1) WO2011129276A1 (en)
US10134119B2 (en) 2014-01-21 2018-11-20 Sumitomo Heavy Industries, Ltd. Image generation device and operation support system
JP5805574B2 (en) * 2012-03-29 2015-11-04 住友建機株式会社 Perimeter monitoring equipment for work machines
JP5960007B2 (en) * 2012-09-20 2016-08-02 住友建機株式会社 Perimeter monitoring equipment for work machines
DE102012223588A1 (en) * 2012-12-18 2014-06-18 Robert Bosch Gmbh Device for the extended representation of a surrounding area of a vehicle
JP6386213B2 (en) * 2013-05-16 2018-09-05 住友建機株式会社 Excavator
US20020018047A1 (en) 2000-07-07 2002-02-14 Matsushita Electric Industrial Co., Ltd. Picture composing apparatus and method
JP2004021307A (en) 2002-06-12 2004-01-22 Matsushita Electric Ind Co Ltd Driving support device
JP2007041791A (en) 2005-08-02 2007-02-15 Nissan Motor Co Ltd Bird's-eye image display system and bird's-eye image display method
US20080309784A1 (en) 2007-06-15 2008-12-18 Sanyo Electric Co., Ltd. Camera System And Mechanical Apparatus
JP5369465B2 (en) * 2008-03-25 2013-12-18 富士通株式会社 Vehicle image processing device, vehicle image processing method, and vehicle image processing program
DE102008035428B4 (en) * 2008-07-30 2010-11-18 Daimler Ag Method and device for monitoring an environment of a vehicle
JP2010091657A (en) 2008-10-06 2010-04-22 Ricoh Co Ltd Cleaning device, process unit, and image forming apparatus
CN101661625B (en) * 2009-09-22 2012-07-18 北京理工大学 Method for generating sample images under all vision angles on basis of standard image
2010-04-12 JP JP2010091657A patent/JP5550970B2/en active Active
2011-04-08 WO PCT/JP2011/058898 patent/WO2011129276A1/en active Application Filing
2011-04-08 EP EP11768797.0A patent/EP2560140A4/en active Pending
2011-04-08 CN CN201180013445.8A patent/CN102792334B/en active IP Right Grant
2011-04-08 KR KR1020127029553A patent/KR101475583B1/en active IP Right Grant
2012-10-11 US US13/649,369 patent/US8593454B2/en active Active
International Search Report mailed on May 10, 2011.
EP2560140A4 (en) 2017-03-15
US20130033494A1 (en) 2013-02-07
CN102792334B (en) 2015-09-23
JP5550970B2 (en) 2014-07-16
WO2011129276A1 (en) 2011-10-20
CN102792334A (en) 2012-11-21
JP2011221865A (en) 2011-11-04
EP2560140A1 (en) 2013-02-20
KR20130008606A (en) 2013-01-22
KR101475583B1 (en) 2014-12-30
JPWO2011118125A1 (en) 2013-07-04 Device for supporting driving of a vehicle
JP5479956B2 (en) 2014-04-23 Ambient monitoring device for vehicles
KR20100107378A (en) 2010-10-05 Image processing method and image processing apparatus
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:KIYOTA, YOSHIHISA;REEL/FRAME:029113/0028