Source: https://patents.justia.com/patent/9233643
Timestamp: 2019-10-16 15:13:56
Document Index: 405521692

Matched Legal Cases: ['art 10', 'art 10', 'art 10', 'art 10', 'art 10', 'art 11', 'art 11', 'art 10', 'art 10', 'art 11', 'art 11']

US Patent for Image generation device and operation support system Patent (Patent # 9,233,643 issued January 12, 2016) - Justia Patents Search
Justia Patents With Plural Optical AxesUS Patent for Image generation device and operation support system Patent (Patent # 9,233,643)
Oct 11, 2012 - SUMITOMO HEAVY INDUSTRIES, LTD.
θ = cos - 1 ⁡ ( A · B  A  ⁢  B  ) [ Formula ⁢ ⁢ 1 ]
N = A × B  A  ⁢  B  ⁢ sin ⁢ ⁢ θ [ Formula ⁢ ⁢ 2 ]
D = ( 0 ; ex , ey , ez ) = QSQ * ⁢ ⁢ where , S = ( 0 ; sx , sy , sz ) , ⁢ Q = ( cos ⁢ θ 2 ; l ⁢ ⁢ sin ⁢ θ 2 , m ⁢ ⁢ sin ⁢ θ 2 , n ⁢ ⁢ sin ⁢ θ 2 ) [ Formula ⁢ ⁢ 6 ]
Here, in the present embodiment, when the quaternion expressing a rotation, which causes the Z-axis to be coincident with the −W-axis, is Qz, the point X on the X-axis in the XYZ coordinates system is moved to a point X′. Therefore, the point X′ is expressed as follows.
X′=QzXQz* [Formula 7]
R=QxQz [Formula 8]
In an optical system of a camera, normally, an image height h is a function of an incident angle α and a focal distance f. Accordingly, the coordinate correspondence part 10 computes the image height h by selecting an appropriate projection system such as a normal projection (h=f tan α), an orthogonal projection (h=f sin α), a stereographic projection (h=2 f tan(α/2)), an equisolid angle projection (h=f sin(α/2)), an equidistant projection (h=fα), etc.
u = h ⁢ ⁢ cos ⁢ ⁢ φ a U [ Formula ⁢ ⁢ 10 ] v = h ⁢ ⁢ sin ⁢ ⁢ φ a v [ Formula ⁢ ⁢ 11 ]
FIGS. 6A and 6B are views for explaining correspondence between coordinates according to the coordinates correspondence part 10. FIG. 6A is a view illustrating a correspondence relationship between the coordinates on the input mage plane R4 of the camera 2 using a normal projection (h=f tan α) and the coordinates on the space model MD. The coordinates correspondence part 10 causes both coordinates to correspond to each other by causing each of line segments, which connect coordinates on the input image plane R4 of the camera 2 and the coordinates on the space model MD corresponding to the coordinates on the input image plane R4, passes the optical center C of the camera 2.
Specifically, the coordinates correspondence part 10 causes the coordinates on the input image plane to correspond to the coordinates on the space model MD based on a predetermined function (for example, an orthogonal projection (h=f sin α), a stereographic projection (h=2f tan(α/2)), an equisolid angle projection (h=f sin(α/2)), an equidistant projection (h=fα), etc.). In this case, the line segment K1-L1 and the line segment K2-L2 do not pass the optical center C of the camera 2.
FIG. 6C is a view illustrating a correspondence relationship between the coordinates on the processing-target image plane R3 and the coordinates on the output image plane R5 of the virtual camera 2V using, as an example, a normal projection (h=f tan α). The coordinates correspondence part 10 causes both coordinates to correspond to each other so that each of line segments connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing-target image plane R3 corresponding to the coordinates on the output image plane R5 passes the optical center CV of the virtual camera 2V.
Specifically, the output image generation part 11 causes the coordinates on the output image plane R5 to correspond to the coordinates on the processing-target image plane R3 based on a predetermined function (for example, an orthogonal projection (h=f sin α), a stereographic projection (h=2f tan(α/2)), an equisolid angle projection (h=f sin(α/2)), an equidistant projection (h=fα), etc.). In this case, the line segment M1-N1 and the line segment M2-N2 do not pass the optical center CV of the virtual camera 2V.
FIG. 9A is a view of a case where an angle β is ft/wed between the group of parallel lines PL positioned on the XZ-plane and the processing-target image plane R3. FIG. 9B is a view of a case where an angle β2 (β2>β) is formed between the group of parallel lines PL positioned on the XZ-plane and the processing-target image plane R3. It is assumed that the plane surface area R1 and the curved surface area R2 of the space model MD, the processing-target image plane R3, the output image plane R5 and the optical center CV of the virtual camera 2V using a normal projection (h=f tan α) in FIG. 9A are common to those of FIG. 9B, respectively.
FIG. 10 is a view of a case where an angle β is formed between the group of parallel lines PL positioned on the XZ-plane and the processing-target image plane R3. FIG. 10 illustrates a state where the optical center CV of the virtual camera 2V using a normal projection (h=f tan α) is moved outside the space model MD (a state where a value of the X-coordinate of the optical center CV is set larger than a radius of the plane surface area R1).
It should be noted that a description was given, with reference to FIGS. 9A and 9B and FIG. 10, of the virtual camera 2V using a normal projection, and the same applies to a virtual camera 2V using projection system (for example, an orthogonal projection, a stereographic projection, an equisolid angle projection, an equidistant projection, etc.) other than the normal projection. In such a case, the output image generation part 11 causes the coordinates on the output image plane R5 to correspond to the coordinates on the processing-target image plane R3 in accordance with a function (for example, an orthogonal projection (h=f sin α), a stereographic projection (h=2 f tan(α/2)), an equisolid angle projection (h=f sin(α/2)), an equidistant projection (h=fα), etc.) corresponding to the respective projection system instead of causing the coordinate point M1 on the processing-target image plane R3 (the plane surface area R1) to correspond to the coordinate point N1 on the output image plane R5 (according to the function h=f tan α) so that the line segment M1-N1 passes the optical center CV. In this case, the line segment M1-N1 does not pass the optical center CV of the virtual camera 2V.
Specifically, the coordinates correspondence part 10 acquires the coordinate point of the optical center C of the camera 2 using a normal projection (h=f tan α), and computes a point at which a line segment extending from a coordinate point on the space model MD, which is a line segment passing the optical center C, intersects with the input image plane R4. Then, the coordinates corresponding part 10 derives a coordinate point on the input image plane R4 corresponding to the computed point as a coordinate point on the input image plane R4 corresponding to the coordinate point on the space model MD, and stores a correspondence relationship therebetween in the input image-space model map 40.
Alternatively, when generating the output image using the virtual camera 2V using a normal projection (h=f tan α), the output image generation part 11 may compute, after acquiring the coordinate point of the optical center CV of the virtual camera 2V, a point at which a line segment extending from a coordinate point on the output image plane R5, which line segment passes the optical center CV, intersects with the processing-target image plane R3. Then, the output image generation part 11 may derive the coordinated on the processing-target image plane R3 corresponding to the computed point as a coordinate point on the processing-target image plane R3 corresponding to the coordinate point on the output image plane R5, and may store a correspondence relationship therebetween in the processing-target image-output image correspondence relation map 42.
The first space model part MD1 is arranged so that the optical axis G1 of the backside camera 2B and the optical axis G2 of the right side camera 2R intersect with each other at a point J1 on a first cylinder center axis (first re-projection axis). Alternatively, the first space model part MD1 is arranged so that a perpendicular line drawn from the optical center of the backside camera 2B to the first cylinder center axis (first re-projection axis) and a perpendicular line drawn from the optical center of the right side camera 2B to the first cylinder center axis (first re-projection axis) orthogonally intersect at a point J2 on the first cylinder axis (first re-projection axis). The first space model part MD1 includes a plane surface area R1a and a curved surface area R2a. In FIGS. 17A and 17B, the curved surface area R2a is represented as a dotted line arc having a center at points J1 and J2.
Similarly, the second space model part MD2 is arranged so that the optical axis G1 of the backside camera 2B and the optical axis G3 of the left side camera 2L intersect with each other at a point J3 on a second cylinder center axis (second re-projection axis). Alternatively, the second space model part MD2 is arranged so that a perpendicular line drawn from the optical center of the backside camera 2B to the second cylinder center axis (second re-projection axis) and a perpendicular line drawn from the optical center of the left side camera 2L to the second cylinder center axis (second re-projection axis) orthogonally intersect at a point J4 on the second cylinder axis (second re-projection axis). The second space model part MD2 includes a plane surface area R1b and a curved surface area R2b. In FIGS. 17A and 17B, the curved surface area R2b is represented as a solid line arc having a center at points J3 and J4.
The space model MD is formed by combining by smoothly connecting the dotted line arc representing the curved surface area R2a of the first space model part MD1 and the solid line arc representing the curved surface area R2b of the second space model part MD2 on the optical axis G1 of the backside camera 2B.
It should be noted that a processing-target image plane R3a corresponding to the first space model part MD1 and a processing-target image plane R3b corresponding to the second space model part MD2 are set in the same manner as the above-mentioned embodiment. The processing-target image plane R3a and R3b may be an arc area having a center at the points J1 and J2 and an arc area having a center at the points J3 and J4, or an area shared by the first space model part MD1 and the second space model part MD2.
A second space model part MD2A is arranged so that the optical axis G1 of the backside camera and the optical axis G3A of the left side camera 2L intersect with each other at a point J3A on a second cylinder center axis (second re-projection axis). In the present embodiment, the point G3A overlaps with an installation position of the backside camera 2B. A second space model part MD2A includes a plane surface area R1c and a curved surface area R2c. In FIG. 18, the curved surface area R2c is represented as a solid line arc having a center at the point 3A.
In FIG. 19, the first space model part MD1 is arranged so that the optical axis G5 of the second backside camera 2B-2 and the optical axis G2 of the right side camera 2R intersect with each other at a point J5 on the first cylinder center axis (first re-projection axis). The first space model part MD1 includes a plane surface area R1d and a curved surface area R2d. In FIG. 19, the curved surface area R2d is represented as a dotted line arc having a center at the point J5.
The second space model part MD2 is arranged so that the optical axis G4 of the first backside camera 2B-1 and the optical axis G5 of the second backside camera 2B-2 intersect with each other at a point J6 on the second cylinder center axis (second re-projection axis). The second space model part MD2 includes a plane surface area R1e and a curved surface area R2e. In FIG. 19, the curved surface area R2e is represented as a dashed line arc having a center at the point J6.
The third space model part MD3 is arranged so that the optical axis G4 of the first backside camera 2B-1 and the optical axis G3 of the left side camera 2L intersect with each other at a point J7 on the third cylinder center axis (third re-projection axis). The third space model part MD3 includes a plane surface area R1f and a curved surface area R2f. In FIG. 19, the curved surface area R2f is represented as a solid line arc having a center at the point J7.
Moreover, in FIG. 20, the first space model part MD1 is arranged so that the optical axis G5A of the second backside camera 2B-2 and the optical axis G2 of the right side camera 2R intersect with each other at a point J5A on the first cylinder center axis (first re-projection axis). The first space model part MD1 includes a plane surface area R1g and a curved surface area R2g. In FIG. 20, the curved surface area R2g is represented as a dotted line arc having a center at the point J5A.
The second space model part MD2 is arranged so that the optical axis G4 of the first backside camera 2B-1 and the optical axis G5A of the second backside camera 2B-2 intersect with each other at a point J6A on the second cylinder center axis (second re-projection axis). The second space model part MD2 includes a plane surface area R1h and a curved surface area R2h. In FIG. 20, the curved surface area R2h is represented as a dashed line arc having a center at the point J6A.
Similar to the example illustrated in FIG. 19, the third space model part MD3 is arranged so that the optical axis G4 of the first backside camera 2B-1 and the optical axis G3 of the left side camera 2L intersect with each other at a point J7 on the third cylinder center axis (third re-projection axis). The third space model part MD3 includes a plane surface area Rlf and a curved surface area R2f, and a top plan view of the curved surface area R2f is represented as a solid line arc having a center at the point J7.
It should be noted that a processing-target image plane R3d (R3g) corresponding to the first space model part MD1, a processing-target image plane R3e (R3h) corresponding to the second space model part MD2, and a processing-target image plane R3f corresponding to the third space model part MD3 are set in the same manner as the above-mentioned embodiment. That is, the processing-target image plane R3d (R3g), the processing-target image plane R3e (R3h), and the processing-target image plane R3f may be an arc area having a center at the point J5, an arc area having a center at the point J6, and an arc area having a center at the point J7, respectively, or may be an area shared by the first space model part MD1, the second space model part MD2, and the third space model part MD3.
1. An image generation device that generates an output image based on at least three input images obtained by taking images by at least three image-taking parts mounted to a body to be operated, which includes a turning body rotatable around a turning axis, and having different optical axis directions, the image generation device comprising:
a coordinates correspondence part implemented by a hardware control part and configured to cause coordinates of an XYZ coordinate system on a columnar space model arranged to surround said body to be operated to correspond to coordinates on at least three input image planes on which said at least three input images are positioned, respectively; and
an output image generation part implemented by the hardware control part and configured to generate said output image by causing values of the coordinates on said at least three input image planes to correspond to values of the coordinates on an output image plane on which said output image is positioned through coordinates on said columnar space model,
wherein said columnar space model is a combination of a plurality of cylindrical space model parts each having a reference axis,
each of said plurality of cylindrical space model parts corresponds to one of pairs of adjacent image-taking parts from among said at least three image-taking parts,
each reference axis of said plurality of cylindrical space model parts is arranged at a single point where components of optical axes of the image-taking parts included in the one of the pairs of image-taking parts intersect, the components being obtained by projecting the optical axes on an XY coordinate plane of the XYZ coordinate system, the XY coordinate plane being perpendicular to the turning axis, and
each of said plurality of cylindrical space model parts has a corresponding predetermined radius on the XY coordinate plane around the corresponding reference axis,
wherein the reference axes respectively of said plurality of cylindrical space model parts are located at different positions on the XY coordinate plane,
wherein each predetermined radius is different.
2. The image generation device as claimed in claim 1,
wherein each of said plurality of cylindrical space model parts is arranged so that perpendicular lines drawn from optical centers of the corresponding pair of image-taking parts to said reference axis are perpendicular to each other.
3. The image generation device as claimed in claim 1,
wherein said plurality of cylindrical space model parts are arranged so that said plurality of cylindrical space model parts are connected on the optical axis of the image-taking part, which corresponds in common to said connected plurality of cylindrical space model parts, and connection ends respectively of the connected plurality of cylindrical space model parts are positioned at a same location on the XY coordinate plane.
4. The image generation device as claimed in claim 1,
wherein said coordinates correspondence part causes coordinates on a processing-target image plane on which a processing-target image to be subjected to an image conversion process is positioned to correspond to coordinates on said columnar space model, and said output image generation part generates said output image by mapping pixel values of coordinates on said input image plane to the pixel values of coordinates on said output image plane through the coordinates on said processing-target image plane and the coordinates on said columnar space model.
7. An image generation device that generates an output image based on at least three input images obtained by taking images by at least three image-taking parts mounted to a body to be operated, which includes a turning body rotatable around a turning axis, and having different optical axis directions, the image generation device comprising:
each of said cylindrical space model parts corresponds to one of pairs of adjacent image-taking parts from among said at least three image-taking parts,
perpendicular lines drawn from optical centers of the one of pairs of image-taking parts to said reference axis of a corresponding one of said cylindrical space model parts are arranged in perpendicular to each other, and
each of said plurality of cylindrical space model parts has a corresponding predetermined radius on an XY coordinate plane of the XYZ coordinate system around the corresponding reference axis, the XY coordinate plane being perpendicular to the turning axis,
wherein each reference axis is the reference axes respectively of said plurality of cylindrical space model parts are located at different positions on the XY coordinate plane,
said output image generation part generates said output image by mapping pixel values of coordinates on said input image plane to the pixel values of coordinates on said output image plane through the coordinates on said processing-target image plane and the coordinates on said columnar space model.
11. An operation support system that supports a movement or an operation of a body to be operated, which includes a turning body rotatable around a turning axis, comprising:
13. An operation support system that supports a movement or an operation of a body to be operated, which includes a turning body rotatable around a turning axis, comprising:
15. The image generation device as claimed in claim 2,
7307655 December 11, 2007 Okamoto et al.
8379054 February 19, 2013 Katayama et al.
20060238536 October 26, 2006 Katayama et al.
20070097206 May 3, 2007 Houvener et al.
20070120660 May 31, 2007 Yamada et al.
20080309784 December 18, 2008 Asari et al.
20090058988 March 5, 2009 Strzempko et al.
20100245573 September 30, 2010 Gomi
20110032357 February 10, 2011 Kitaura et al.
20110234801 September 29, 2011 Yamada
20120026333 February 2, 2012 Okuyama
20120069188 March 22, 2012 Ohno
20120327238 December 27, 2012 Satoh
H07309577 November 1995 JP
3286306 May 2002 JP
2008-083786 April 2008 JP
WO 2009/144994 December 2009 WO
International Search Report mailed on Jul. 5, 2011.
Patent number: 9233643
Patent Publication Number: 20130033493
Inventor: Yoshihisa Kiyota (Kanagawa)
Application Number: 13/649,338
Current U.S. Class: With Plural Optical Axes (359/403)
International Classification: G06T 7/00 (20060101); B60R 1/00 (20060101); B66C 15/00 (20060101); E02F 9/26 (20060101); H04N 7/18 (20060101);