Stereoscopic image pick-up apparatus

Abstract: An image pick-up apparatus for capturing parallax images for the left eye and right eye which meet a fusion condition. A stereo camera image-captures a subject and displays captured parallax images on a stereoscopic display. An observer looks at the screen of the display. The lines of sight of both eyes of the observer are then detected, and a distance from the view point of the observer to the subject is measured based on the line-of-sight data. Instead of the observer, the apparatus determines whether the subject distance satisfies the fusion condition. The apparatus thus determines an image-fusible range, based on a (known) convergence angle, a (known) base line distance, a (known) size of the display, and the distance of distinct vision of the observer.

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pick-up apparatus for stereoscope for picking up parallax images for stereoscopic viewing. More particularly, the present invention relates to an image pick-up apparatus for picking up, without user intervention, parallax images that satisfy binocular fusion conditions and an image pick-up apparatus that lets a user know whether the binocular fusion conditions are satisfied. 2. Description of the Related Art Some known stereoscopic image pick-up apparatuses capture parallax images from a plurality of view points using a plurality of cameras. In such a stereoscopic image pick-up apparatus, a plurality of cameras are mounted on tripod heads, and a user manually adjusts a spacing and a convergence angle between camera axes, based on his or her operational experience, depending on a scene to be captured, and image-captures the scene while actually observing the stereoscopic view on a monitor. Such an experience-based operation is not efficient, and no good result is expected when there is no one available who has a good knowledge of picking up stereoscopic images. Head-mounted displays and eyeglass-type displays have been developed today, and these displays let an image for the right eye to be selectively presented on the right eye and an image for the left eye to be selectively presented on the left eye. A stereoscopic image producing a depth perception is thus observed. A liquid-crystal display is combined with a lenticular sheet having a predetermined pitch or a mask with apertures and non-apertures formed in a predetermined pattern to impart a directivity to a light ray from the liquid-crystal display, and by making the directivity match the pattern of the image presented on the liquid-crystal display, the observer watches the image for the right eye on the right eye and the image for the left eye on the left eye. The observer thus enjoys the image presenting depth. Conventionally, the image presented is typically acquired through a binocular-type camera having two lens sets. An apparatus, disclosed in Japanese Examined Patent Publication No. JP-A-8-27499 (telescopic television image pick-up apparatus), does not require two lens sets. This apparatus includes two liquid-crystal shutters, a total reflection mirror and a half mirror and picks up alternately left and right parallax images through a single lens set. Regardless of the above binocular system or the system picking up alternately left and right images in a time division manner, the adjustment of a parallax between left and right images, namely, the adjustment of convergence is required during image-capturing. The convergence adjustment is typically manually performed. The above binocular-type camera employs two lens sets, one for forming the image for the right eye and the other for forming the image for the left eye, and the user feels fatigue when observing a stereoscopic image or cannot to attain image fusion at all, if performance differences due to manufacturing error (for example, in magnification, deviation in optical axis, tint, brightness, distortion, field tilt and the like) are present between the two lenses. To make the two lens sets match in performance, the accuracy of components needs to be heightened. Adjustment is further required if the improved accuracy of components is still not sufficient enough. To compensate for performance difference, special means is used, for example, images are electronically corrected. When a zoom lens is used, the zoom variator operations of the left and right lens sets must be interlocked in a state with these performances factored in. This arrangement is costly and time consuming to manufacture, and is inadequate for mass production. To observe the images picked up by the binocular-type camera, two monitors are typically required. The use of the two monitors is not practical. To record the two images, a special recording device is needed to synchronize the one image to the other. Without using it, the conversion of two image signals into one image signal is contemplated. To convert the two image signals into one image signal, however, a special converter is required to alternately display and record the left and right parallax images. Compared with a single lens camera, the binocular-type camera itself is bulky. The entire system of the image pick-up apparatus, in need of the above-described special devices, fails to achieve widespread commercial acceptance among users because of its bulkier, costlier and less mobile design. The time-division camera proposed in the above-cited Japanese Examined Patent Publication No. JP-A-8-27499 combines the optical paths for left and right parallax images at a half mirror into one optical path to guide images to a single lens. When images are transmitted through or reflected from the half mirror before entering the lens, the quantity of light is reduced to half or smaller. The arrangement disclosed in the above-cited Japanese Examined Patent Publication No. JP-A-8-27499 theoretically presents a difference between the lengths of the optical paths of the left and right parallax images, suffering a magnification difference between the left and right images. This becomes a cause of the fatigue of the user when he or she observes the images picked-up, and as a result, the user cannot observe the images fused and cannot observe them in a stereoscopic view. Moreover, document EP-A-0 641 132 discloses a stereoscopic image pick-up and display apparatus, wherein two camera bodies each having an objective lens are operated and the output signals are further evaluated. A convergence control is performed so that the optical axes of the cameras converge at an optimum point to ensure that the parallax of the captured images comes within the binocular fusing range of a viewer. Specifically, a parallax calculator is provided for calculating the parallax of an image and the parallax of the nearest object is used for calculating the depth-wise position of an object to be reproduced at the nearest point when a captured image is displayed by the stereoscopic image display apparatus. The fusing range is defined on the depth display position calculator, and the converging angle is controlled in a manner that the object at the nearest point is positioned within the fusional range of the viewer. Furthermore, a line-of-sight detecting means is used for detecting the point on which the viewer's eye is focused. Furthermore, document JP-A-7 067 023 discloses a multi-eye image pick-up apparatus with immediate image pick-up wherein a first and a second video camera portion are provided for picking up a stereoscopic image. When both camera portions are operated, the optical axis thereof is controlled and the control means sets the optical axes of the first and the second video camera portions at predetermined optical axis angles. Furthermore, the apparatus comprises means for visual axis detection, thereby determining the position of the eyeball and, upon data evaluation, the viewing direction of the user. SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image pick-up apparatus through which the user or viewer easily learns during image-picking up whether a main subject falls within a range of image fusion. This object is accomplished according to the present invention by an image pick-up apparatus and an image pick-up method as set out in the appended claims. The present invention provides an image pick-up apparatus for capturing images for stereoscopic viewing, which comprises image pick-up means for picking up left and right parallax images of a main subject respectively for the left and right eyes, display means for displaying the left and right parallax images, picked up by the image pick-up means, line-of-sight detection means for detecting the lines of sight of the left and right eyes, looking the respective images displayed by the display means, and determining means for determining, based on the output of the line-of-sight detection means, whether the main subject falls within an image-fusible range. The apparatus detects the lines of sight of an observer who observes the display means on which the left parallax image and right parallax image are presented, and, instead of the observer, determines, based on the lines of sight, whether the subject falls within the image-fusible range. In a preferred embodiment of the preset invention, the determination result provided by the determining means is reported to the user. In a preferred embodiment of the present invention, an image-fusible range is defined by the inter-pupillary distance and the distance of distinct vision of a user. In a preferred embodiment of the present invention, the image date of the parallax images is stored in a memory in response to the output of the determining means. Since the detected lines of sight of the user are the ones that look toward the display parallax images, the data about the lines of sight need to be converted into those in the coordinate systems of the image pick-up means. In a preferred embodiment of the present invention, the line-of-sight detection means further comprises converting means for converting the left and right lines of sight of a user into left and right direction vectors that are respectively expressed in left and right coordinate systems of the image pick-up means, and coordinates calculator means for calculating the coordinate values of the crossing point of the left and right direction vectors in the world coordinate system. In a preferred embodiment of the present invention, the image-fusible range is expressed according to a farthest position and a nearest position from the view point of the user in the direction of depth. In a preferred embodiment, the farthest position of the image-fusible range is set to be a point so that the horizontal distance between two second points on the left and right image planes of the image pick-up means corresponding to a first point (A) of the farthest position is substantially equal to the inter-pupillary distance of the user. The nearest position of the image-fusible range is set to be a point so that a position (C') where two points on the left and right image planes of the image pick-up means corresponding to a first point (C) of the nearest position, through perspective transformation based on the left and right view points of the user, look standing out to the user, is approximately equal to the point at the distance of distinct vision of the user. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing the construction of a first embodiment of the present invention; FIG. 2 is an explanatory diagram showing the position of the eyes of an observer relative to an optical system of an image pick-up apparatus of the first embodiment of the present invention; FIG. 3 is a flow diagram showing a control process for measuring a distance to a subject based on the line-of-sight data of the observer in the first embodiment of the present invention; FIG. 4 is an explanatory diagram showing the operation executed in step S8 through step S10 in the control process shown in FIG. 3; FIG. 5 is an explanatory diagram illustrating the principle of determining an image-fusible range; FIG. 6 is an explanatory diagram illustrating the principle of determining the farthest position of the image-fusible range; FIG. 7 is an explanatory diagram illustrating the principle of determining the nearest position of the image-fusible range; FIG. 8 is an explanatory diagram illustrating the principle of determining the nearest position of the image- fusible range; FIG. 9 is a flow diagram showing the control process for determining the image-fusible range. DETAILED DESCRIPTION OF THE INVENTION Referring to the drawings, the preferred embodiment of the present invention is now discussed. <First embodiment> ··· hardware structure FIG. 1 is a block diagram showing the construction of a "stereoscopic image pick-up apparatus" according to a first embodiment of the present invention. Referring to FIG. 1, there is shown a stereo camera 1 for capturing a plurality of parallax images, and in the first embodiment, the stereo camera 1 comprises a right optical system 101 for capturing a parallax image for the right eye and CCD 103 of a photoelectric converter and a left optical system 102 for capturing a parallax image for the left eye and CCD 104 of a photoelectric converter. The pair of optical system 101 and CCD 103 is identical in optical specification to the pair of optical system 102 and CCD 104. The optical system 101 and CCD 103 and the optical system 102 and CCD 104 are arranged so that the spacing between the two pairs (hereinafter referred to as a base line distance) and the angle made between the two pairs (hereinafter referred to as a convergence angle) are variably set with an unshown mechanism. The base line distance and the convergence angle are adjusted by a known convergence/base line distance adjuster 11. A convergence angle/base line distance detector 10 detects the convergence angle and base line distance with its unshown encoder. FIG. 2 shows the positions of the optical systems 101 and 102 relative to a main subject. As shown, let  b represent the convergence angle between the optical systems 101 and 102, and 2k represent the base line distance. As already described, the adjuster 11 adjusts the convergence angle  b and the base line distance 2k, and an adjusted angle is detected by the detector 10, and is then sent to a calculating unit 3 (FIG. 1) to be described later. An image display 2 includes a right display unit 201 for the right eye, a left display unit 202 for the left eye, a right line-of-sight detector 203, and a left line-of-sight detector 204. The right display unit 201 and left display unit 202 are identical in specification, and are the so-called retinal display that provides an image through the after image effect by irradiating and scanning the retina with a light beam from a liquid crystal, CRT, LED or laser, each equipped with an observation optical system. In the first embodiment, the display units 201 and 202 respectively scan the left and right retinas by a light beam from an LED circuit 12 to give left and right stereoscopic images to the user. A right line-of-sight detector 203 and left line-of-sight detector 204. in image display 2 employ the method of detecting the line of sight from the so-called corneal reflection light. To convert the line-of-sight information, determined using the corneal reflection light, into a coordinate system of an image, the image display 2 includes liquid-crystal displays LCD205 and LCD206. This method is disclosed in detail in Japanese Unexamined Patent Publication No. JP-A-5-68188, and the discussion of the method is omitted here. The line-of-sight information about the eyeball obtained by the line-of-sight detectors 203 and 204 is sent to the calculating unit 3 to be described later, and serves as a basis for the calculation of the direction vector of the line of sight. The calculating unit 3 of the image pick-up apparatus calculates an image-pickable range of the stereo camera 1 that assures that the left and right parallax images picked up by the stereo camera 1 causes an image fusion on the user (such a range is hereinafter referred to as "image-fusible range"). The calculating unit 3, substantially realized by computer software, is shown in a functional block diagram in FIG. 1, and comprises a right line-of-sight range calculator 301, a left line-of-sight range calculator 302, a line-of-sight vector calculator 303, a crossing point calculator 307, a distance calculator 305, an image-pickable range calculator 304, and a comparator 306. The two line-of-sight range calculators 301 and 302 are respectively connected to the right line-of-sight detector 203 and the left line-of-sight detector 204, and calculate the line-of-sight ranges for the left and right eyes of the user based on the outputs from the two detectors 203 and 204. The left and right line-of-sight ranges calculated are input to the line-of-sight vector calculator 303, which calculates the direction vectors of the left and right lines of sight ( d R and d L to be described later). The crossing point calculator 307 determines the coordinates of the crossing point of the lines of sight, based on the detected direction vectors d R and d L of the lines of sight. The distance calculator 305 calculates a distance 1 from the view point of the user to the subject, based on the detected direction vectors of the lines of sight. The image-pickable range calculator 304 calculates the image-pickable range (namely, between the distal limit A of the range and the proximal limit B of the range in the direction of depth). The comparator 306 compares the distance 1 to the subject, calculated by the distance calculator 305, to the limit values A and B, calculated by the image-pickable range calculator 304, to determine whether the subject meets an image fusion condition. Specifically, if the distance to the subject, calculated by the distance calculator 305, meets the following equation, B ≤ ℓ ≤ A the subject is within the image-pickable range, and is determined to meet the image fusion condition. An image controller 4 controls the units in the image pick-up apparatus. Specifically, the image controller 4 controls right and left camera processors 5 and 6, for the right and left eyes, which converts image signals from CCDs 103 and 104 into a predetermined image format. These processors 5 and 6, under the control of the image controller 4, convert the left and right parallax images picked up by the stereo camera 1 into the predetermined format and store them in an image memory 9. To present stereoscopic images to the user, the images in the image memory 9 are read and converted into video signals by the image controller 4, and the video signals are then sent to the display units 201 and 202 via drivers 7 and 8, respectively. The image controller 4 exchanges data with the image memory 9. The image controller 4 also sends to the LED circuit 12 a signal that controls the lighting and extinction of an unshown LED installed in the image display 2. <Operation>···First embodiment A series of operations of the system are now discussed. FIG. 3 shows the operation starting at the line-of-sight detectors 203 and 204 and ending with the calculating unit 3, particularly, to the process for determining the direction vectors of the lines of sight based on the data detected by the line-of-sight detectors 203 and 204. To detect the image-fusible range in the first embodiment, the subject is picked-up using the stereo camera, and the parallax images in stereoscopic view are presented to the user using the display. Specifically, observing the left and right parallax images presented on the display, the observer indirectly looks at the subject. The image pick-up apparatus then detects the lines of sight of the observer's eyes. Step S1 corresponds to the line-of-sight detectors 203 and 204 shown in FIG. 1, steps S2 and S3 correspond to the line-of-sight range detectors 301 and 302 shown in FIG. 1, step S4 corresponds to the line-of-sight vector calculator 303, steps S5, S6, S8, S9 and S10 correspond to the crossing point calculator 307, and step S7 corresponds to the distance calculator 305. In step S1, the lines of sight, namely, the directions of rotation of the eyeballs are detected, based on the corneal reflection phenomenon. As described above, each of the line-of-sight detectors 203 and 204 has its own LCD and CCD. LCD 205 and LCD 206 irradiate the eyeballs of the observer by infrared light beams having a known pattern. The patterned beams are reflected by the corneas, and are then captured by CCD 207 and CCD 208 as images. Since LCD 205 and LCD 206 are beforehand respectively aligned with CCD 207 and CCD 208 in their respective coordinate systems, the pixel positions on CCD 207 and CCD 208 are considered to correspond to the respective spot positions on LCD 205 and LCD 206. In step S2, the coordinate values of each beam spot in each LCD coordinate system are determined. The coordinate values of the beam spot in each LCD coordinate system represent the angle of rotation of the respective eyeball. In step S3, the coordinate values of the beam spot in each LCD coordinate system obtained in step S2 are converted into the coordinate values in the coordinate systems of the stereo camera 1 (namely, the coordinate systems of CCD 103 and CCD 104). Step S4 determines direction vectors d R and d L of straight lines that connect the coordinates of the left beat spot and right beam spot, expressed in the coordinate systems of the stereo camera 1, to reference points of the optical systems 101 and 102. The direction vectors are lines of sight expressed in the world coordinate system. The line of sight expressed in the world coordinate system is hereinafter referred to as "line-of-sight vector" or simply "direction vector". The right direction vector d R and left direction vector d L are not necessarily present in the same plane. It is thus determined in step S5 whether the right direction vector d R and left direction vector d L are present the same plane. When it is determined that the two vectors are present in the same plane, the controller 4 goes to step S6 to calculate the crossing point (designated X) of the right direction vector d R and left direction vector d L in the world coordinate system. When the two vectors are not present in the same plane, the controller 4 goes to step S8, where plane P in which the right direction vector d R and the reference point of the right optical system 101 lie is determined as shown in FIG. 4. In step S9, an orthogonal projection vector d L ' of the left direction vector d L with respect to the plane P is determined. In step S10, the coordinates of the crossing point (designated X') of the right direction vector d R and the left direction vector d L ' is calculated in the world coordinate system. Step S7 determines the distance 1 between the predetermined reference point of the stereo camera 1 (the center point between the reference point of the right optical system 101 and the reference point of the left optical system 102) and the crossing point determined in either step S6 or step S10 (X or X'). Discussed next is the calculation of the range 1 of the distance to the subject satisfying the image fusion condition, namely, the image-pickable range of distance to the subject. A range within which humans can fuse left and right parallax images presented on display screens as a stereoscopic image (an image-fusible range) is different depending on the distal side (beyond the display screens) or on the proximal side (on this side of the display screens). The image-fusible range depends on the characteristics of the human eyes. Considering that the inter-pupillary distance of the human is about 65 mm wide, and that the distance of distinct vision of the human is about 200 mm, this system determines a distal limit A based on the human inter-pupillary distance (about 65 mm) and a proximal limit C based on the distance of distinct vision. Referring to FIG. 5, the image-fusible range is discussed. As shown, the y axis represents the direction of depth, and the z axis represents the vertical direction of the apparatus. The x axis is perpendicular to the direction of depth. FIG. 5 is drawn at a constant in the z axis. Referring to FIG. 5, there are shown a reference point O L of the left optical system 102 (the lens center when the optical system is considered as a single thin lens) and a reference point O R of the right optical system 101 (the lens center when the optical system is considered as a single thin lens). When the optical systems 101 and 102 are set up, the reference points O L and O R are fixed points, and the crossing point B of the optical axes of the optical systems 101 and 102 also becomes fixed. Let A represent a distal limit point of the image-fusible range with respect to the point B and let C represent a proximal limit point of the image-fusible range with respect to the point B. The point B then coincides with the center of the image screen of the left and right optical systems 102 and 101. Since the point B has zero parallax, the point B appears as a point on the display screen to the observer looking at the stereoscopic image displayed. To the same observer, the point A appears as point present beyond the display screen and the point C appears as a point on this side of the display screen. With the origin of the x-y coordinates positioned at the center of the base line distance between the points O L and O R , these points are A(0,A), B(0,B), C(0,C), O L (-k,0), and O R (k,0). Furthermore,  b ,  a , and  c represent the following angles. ∠BO L O = ∠BO R O =  b ∠BO L A = ∠BO R A =  a ∠BO L C = ∠BO R C =  c Since the point B is the crossing point of the optical axes of the two optical systems 101 and 102 as already described, it is possible to make an image pick-up plane coincide with a plane which is perpendicular to each optical axis and in which the point B lies. Referring to FIG. 6, a virtual image pick-up plane of the left optical system 102 is set up to be aligned on the point B. The distal point A and the proximal point C respectively become a point A L and a point C L on the virtual image pick-up plane. Let 2 w represent the lens view angle of each of the optical systems 101 and 102. The display units 201 and 202 presenting the parallax images each have a horizontal width of 2W s , and are positioned in front of the observer with a predetermined distance d s allowed therebetween. In this setup, the length of the virtual image pick-up plane shown in FIG. 6 is preferably equal to the horizontal dimension of each of the display units 201 and 202. With this arrangement, the size of the virtual image pick-up plane is 2W s . As shown in FIG. 6, the points A and C are present within an angle range of 2 w . Referring to FIG. 6, the left eye of the observer sees the point A as the point A L ' on the image pick-up plane, though the point A is actually present straight ahead of the observer. The reason why there is a distance BA' L between the point B and the point A' L is that the two eyes of human are spaced having a parallax. The distance BA' L is the quantity that may be called a parallax of the subject, and is expressed in a deviation on the image. Let D A represent the deviation of the point A and D B represent the deviation of the point C, and the following equation holds. D A tan  a = D C tan  c = W S tan  W D A and D B are thus expressed by the following equations. D A = W S × tan a tan W D C = W S × tan  c tan  W In determining the fusion condition in the image pick-up system, the left and right parallax images are obtained by picking up a subject 200 on the stereo camera 1 of FIG. 1 (with its base line distance 2k and convergence angle  b ) and the parallax images are displayed on the display units 201 and 202 (with their horizontal dimensions of 2W s ) placed at a position in front of the observer with the predetermined distance d s allowed. A determination is made of whether the left and right parallax images are displayed in a fused state to the eyes of the observer. Specifically, the determination is made depending on whether the distance 1 from the stereo camera 1 to the subject satisfies the above parameters (namely, the fusion condition, to be described later, determined by the base line distance of 2k, the convergence angle of  b , the distance d s to the display units and the size 2W s of the display units). Let 2d h represent the inter-pupillary distance of the observer, and as can be seen from FIG. 6, when the deviation of the point A, namely, the distance BA' L (=2D A ) is smaller than 2d h , in other words, 2D A ≦2d h , the subject on this side of the point A must be fused. The following equation thus holds. d h ≥ D A where tan( a +  b ) = A / k ... tan  b = B / k The following equation is derived from EQ1, EQ3 through EQ5. A ≤ k × d h · tan  W + W S · tan  b W S - d h · tan  b · tan  W The right side of EQ 6 signifies the image-fusible range beyond the screen with the point B aligned with the center of the screen. Specifically, if the subject 200 is beyond the point B but on this side of the point A defined by the following equation, the subject 200 is within the fusible range. k × d h · tan  W + W S · tan  b W S - d h · tan  b · tan  W As already described, k and  b are obtained by the convergence angle/base line distance detector 10.  w is a value known from lens data, and W S is also a known value in display condition. In this embodiment, d h is a predetermined value (65 mm/2 = 32.5 mm). However, the value of d h is not limited to this. Referring to FIG. 7, the point C is now considered. FIG. 7 shows not only the virtual image pick-up plane 250L but also the virtual image pick-up plane 250R. Specifically, images A L and C L , corresponding to the points A and C, are obtained in the right image pick-up plane 250L, and images A R and C R , corresponding to the points A and C, are obtained in the left image pick-up plane 250R. If the observer looks at the image pick-up plane with both eyes, the left and right fields of view are combined on a single virtual plane 400. On the virtual plane 400, points A' L and A' R are formed for the point A, and points B' L and B' R are formed for the point B. To the observer, the point A appears to be present at a crossing point A' where a straight line 300L connecting the left eye to the point A' L intersects a straight line 300R connecting the right eye to the point A' R . And the point C appears to be present at a crossing point C' where a straight line 350L connecting the left eye to the point C' L intersects a straight line 350R connecting the right eye to the point C' R . In other words, the point A appears to stand back by a distance d B , and the point C appears to be projected by a distance d P as shown in FIG. 7. FIG. 8 shows the relationship of the projection distance d P , the distance d S to the display unit and the deviation D C . As can be seen from FIG. 8, the following equation holds. D C d P = d h (d S - d p ) From EQ7, D C = d p · d h d S - d p Since the following equation holds, tan  C = k · (B - C) k 2 + B × C C = {d P · W S · tan  b - d h · (d S - d P ) · tan  W } × {d P · W S + tan  b · d h · (d S - d P ) · tan  W } The right side of EQ10 signifies the proximal limit value of the image-fusible range with the screen center aligned with the point B. Specifically, the image-fusible range on this side of the point B is beyond the point C. k and  b are obtained by the convergence angle/base line distance detector 10.  w is a value known from lens data, and W S and d S are values known from the display condition. In this embodiment, d h and d p are predetermined values (d h =65 mm, d p =200 mm), though the present invention is not limited to these values. From the above discussion, the distance 1 to the subject 200 that satisfies the fusion condition is expressed by the following equation. k × d h · tan  W + W S · tan  b W S - d h · tan  b - tan  W ≤ ℓ ≤ {d P · W S · tan  b - d h · (d S - d P ) · tan  W } × {d P · W S + tan  b · d h · (d S - d P ) · tan  W } The principle of the determination of whether the subject meets the fusion condition in the image pick-up system has been discussed. The image pick-up system has a function of determining whether the subject being captured meets the fusion condition, namely, falls within the image-pickable range (C ≤ ℓ ≤ A), and informing the user of the determination result. FIG. 9 is a flow diagram showing the control process that incorporates the user informing function. In step S11, the convergence angle  b and the base line distance 2k, set by the user, are read into the optical systems 101 and 102. In step S12, the known size W s of the image pick-up apparatus, the distance d s to the display units, the inter-pupillary distance d h , and the projection distance d P are also read. In step S13, the distal fusion limit point A is determined according to EQ6. In step S14, the proximal fusion limit point C is determined according to EQ10. In step S15, the image-pickable range is determined. The comparator 306 compares the distance to the subject which the user looks at, calculated by the distance calculator 305, namely, the distance 1 to the subject which the user is now image-picking up, with the distal fusion limit point A and the proximal fusion limit point C. The comparator 306 then compares the distance to the object of the view point determined in step S11 with the image-pickable range determined in step S15. Specifically, the comparator 306 determines whether the object of the view point falls within or out of the image-pickable range. The comparison result is sent to the image controller 4. Either the display unit 201 or display unit 202 or both are provided with LEDs (not shown), and the display units 201 and 202 light or flash the LEDs to indicate whether the subject is within or out of the range. The LEDs may be lit to indicate that the subject is out of the range and may be extinguished to indicate that the subject is within the range. Alternatively, the LEDs may be flashed to indicate that the subject is out of the range and may be extinguished to indicate that the subject is within the range. The image display 2 may present the result of the above determination. As described above, when image-picking up, the user easily knows whether the main subject is within the image-fusible range (image-pickable range). Since the user easily knows, when image-capturing, whether the main subject is within the image-fusible range according to the first embodiment, the userpicks up a stereoscopic image easy to see and imposing less fatigue. <Modification of the first embodiment and unclaimed examples> A diversity of modifications of the embodiment are contemplated. In the embodiment, the observer looks at the image of the subject displayed on the display units and the lines of sight of the observer are then detected. An unclaimed example also works to detect the lines of sight of the observer (user) who directly looks at the subject. This is because the distance 1 to the subject is also determined from the lines of sight of the user who directly looks at the subject. No particular optical specifications are set for the optical system 101 and CCD 103. Any systems are perfectly acceptable as long as the optical system 101 and CCD 103 in pairs are respectively identical, in optical specification, to the optical system 102 and CCD 104 in pairs. In the embodiment, image picking is performed in a cross method in which the camera optical axes cross each other. According to an example, image picking is performed in a parallel method in which the camera optical axes run in parallel. The line-of-sight detection is not limited to the method in which the angle of rotation of the eyeball is detected. Alternatively, any other method may be used, including the EPG method for making use of a voltage difference between the voltages of the eyeballs, the method of using a difference in reflectance between the white and the pupil in the sclera of the eye, and the search coil method in which a coil-embedded contact lens is put on under the presence of a magnetic field to measure the movement of the eyeballs. The image display 2 in the embodiment is a fixed type. The embodiment also works in any other appropriate display means such as a head-mounted display (HMD). The medium of the image memory 9 in the embodiment is a magnetic tape. The embodiment is not limited to this. Alternatively, the image memory 9 may be an IC memory, a magneto-optical disc, a DVD disc, a compact disc, or a PD disc.

Claims: An image pick-up apparatus for capturing images for stereoscopic viewing, comprising: image pick-up means (1) for picking up left and right parallax images including a main subject respectively for the left and right eyes; display means (2) for displaying the left and right parallax images, picked up by said image pick-up means; line-of-sight detection means (203, 204) for detecting the lines of sight of the left and right eyes looking toward the respective images displayed by said display means (2) in the coordinate system of said display means; distance calculation means (303, 307, 305) for calculating a crossing point of the lines of sight of the left and right eyes detected by said line-of-sight detection means, determining the main subject on the basis of the calculated crossing point of the lines of sight in the coordinate system of said image pick-up means and calculating a distance from the image pick-up means to the determined main subject; image-pickable range calculation means (304) for calculating an image-fusible range based on angles of the respective image pick-up means, a distance between.the image pick-up means, a distance from the left and right eyes to said display means, human-pupillary distance limit, and a proximal limit based on distance of distinct vision of human eyes; and determining means (306) for determining whether the distance from said image pick-up means to said main subject calculated by said distance calculation means falls within image-fusible range calculated by said image-pickable range calculation means. An image pick-up apparatus according to claim 1, further comprising informing means for informing a user of a determination result provided by said determining means (306). An image pick-up apparatus according to claim 2, wherein said informing means displays said determination result. An image pick-up apparatus according to claim 1, wherein the output of said image pick-up means (1) is stored in a memory in response to the output of said determining means. An image pick-up apparatus according to claim 1, wherein said line-of-sight detection means (203, 204) further comprises: converting means (303, 304) for converting the left and right lines of sight of a user into left and right direction vectors that are respectively expressed in left and right coordinate systems of said image pick-up means; and coordinates calculator means (307) for calculating the coordinate values of the crossing point of the left and right direction vectors in the world coordinate system. An image pick-up apparatus according to claim 1, wherein said image-pickable range calculation means (304) expresses said image-fusible range according to a farthest position and a nearest position from said image pick-up means in the direction of depth. An image pick-up apparatus according to claim 6, wherein said image-pickable range calculation means (304) sets the farthest position of said image-fusible range to be a point so that the horizontal distance between two second points on the left and right image planes of said image pick-up means (1) corresponding to a first point (A) of said farthest position is substantially equal to the inter-pupillary distance of the user. An image pick-up apparatus according to claim 6, wherein said image-pickable range calculation means (304) sets the nearest position of said image-fusible range to be a point so that a position (C') where two points on the left and right image planes of said image pick-up means corresponding to a first point (C) of the nearest position, through perspective transformation based on the left and right view points of the user, look standing out to the user, is approximately equal to the point at the distance of distinct vision of the user. An image pick-up apparatus according to claim 1, wherein said image display means (2) and said line-of-sight detection means (203, 204) are built into a head-mounted display. An image pick-up apparatus according to claim 9, wherein said head-mounted display comprises a light-emitting diode. An image pick-up method for capturing images for stereoscopic viewing, comprising: an image pick-up step of picking up left and right parallax images including a main subject respectively for the left and right eyes by an image pick-up means; a display step of displaying the picked left and right parallax images by a display means; a line-of-sight detection step (S1) of detecting the lines of sight of the left and right eyes looking toward the respective displayed images in the coordinate system of said display means; a distance calculation step of calculating a crossing point of the lines of sight of the left and right eyes detected by said line-of-sight detection step, determining the main subject on the basis of the calculated crossing point of the lines of sight in the coordinate system of said image pick-up means and calculating a distance from the image pick-up means to the determined main subject; an image-pickable range calculation step of calculating an image-fusible range based on angles of the respective image pick-up means, a distance between the image pick-up means, a distance from the left and right eyes to said display means, human-pupillary distance limit, and a proximal limit based on distance of distinct vision of human eyes; and a determining step of determining whether the distance from said image pick-up means to said main subject calculated by said distance calculation step falls within image-fusible range calculated by said image-pickable range calculation step. An image pick-up method according to claim 11, further comprising the step of informing a user of the determination result of said determining step. An image pick-up method according to claim 12, wherein said informing step displays said determination result. An image pick-up method according to claim 11, wherein said determining step defines said image-fusible range according to the inter-pupillary distance and the distance of distinct vision of a user. An image pick-up method according to claim 11, further comprising the step of storing the output of said image pick-up step in a memory in response to the output of said determining step. An image pick-up method according to claim 11, wherein said step of detecting the lines of sight comprises the step of converting the left and right lines of sight of a user into left and right direction vectors that are respectively expressed in left and right coordinate systems in said image pick-up step; and the step of calculating the coordinate values of a crossing point of the left and right direction vectors in the world coordinate system. An image pick-up method according to claim 16, wherein said determining step determines whether the calculated coordinate values of the crossing point in the world coordinate system fall within said image-fusible range. An image pick-up method according to claim 11, wherein said image-pickable range calculating step expresses said image-fusible range according to a farthest position and a nearest position from the image pick-up means in the direction of depth. An image pick-up method according to claim 18, wherein said image-pickable range calculating step sets the farthest position of said image-fusible range to be a point so that the horizontal distance between two second points in left and right image planes of said image pick-up step corresponding to a first point (A) of said farthest position is substantially equal to the inter-pupillary distance of the user. An image pick-up method according to claim 18, wherein said image-pickable range calculating step sets the nearest position of said image-fusible range to be a point so that a position (C') where two points on the left and right image planes of said image pick-up step corresponding to a first point (C) of the nearest position, through perspective transformation based on the left and right view points of the user, look standing out to the user, is approximately equal to the point at the distance of distinct vision of the user. An image pick-up apparatus according to claim 1, further comprising: recording means for recording, in accordance with the output of said determining means, the left parallax image signal and the right parallax image signal from said image pick-up means (1). An image pick-up method according to claim 11, further comprising: the recording step of recording, in accordance with the output of said determining step, the left parallax image signal and the right parallax image signal obtained through said image pick-up step. An image pick-up apparatus according to claim 1, wherein said image pick-up means (1) comprises adjusting means for adjusting either the base line distance between said first and second optical systems or the convergence angle between said first and second optical systems, or for adjusting both the base line distance and the convergence angle. A computer readable recording medium for storing a program that is executed by a computer, wherein said medium stores a program that executes the method according to claim 11.

Inventors: OGINO SHIGERU, OGINO, SHIGERU, Ogino, Shigeru c/o Canon Kabushiki Kaisha
Applicants: CANON KK, CANON KABUSHIKI KAISHA
Application year: 1998
Main classification: H04N13
IPC codes: H04N13