Abstract:
An image-display control apparatus includes a detection unit for detecting a predetermined object from a photographed image, a position calculation unit for obtaining a positional relationship between the predetermined object in the photographed image detected by the detection unit and an object in a virtual image, an image removing unit for removing a predetermined partial image in the object in the virtual image by referring to a result of calculation of the position calculation unit, a synthesis unit for synthesizing the virtual image and the photographed image so that the predetermined object in the photographed image is present at a position of the partial image in the virtual image removed by the image removing unit, and a display control unit for displaying an image obtained as a result of synthesis of the synthesis unit on a display device.

Description:
BACKGROUND OF THE INVENTION  
         [0001]    1. Field of the Invention  
           [0002]    The present invention relates to an image generation system for generating a three-dimensional mixed spatial image by fusing a real-space image and a virtual-space image, and the like.  
           [0003]    2. Description of the Related Art  
           [0004]    Recently, in the field of various types of image processing, a technique of synthesizing a virtual-space image, such as CG (computer graphics) or the like, with a real-space image (photographed image) by fusing a real space and a virtual space is being used.  
           [0005]    When intending to fuse a real space and a virtual space, image synthesis of an object in the real space and an object in the virtual space causes problems. As shown in FIG. 9, when intending to simply paste an object in a virtual space  901  making a real space  902  a background, the object in the virtual space  901  is always displayed in front of objects in the real space  902  (see a fused space  903 ).  
           [0006]    However, an object in a virtual space is sometimes positioned behind an object in a real space. In such a case, it is necessary to display the object in the virtual space behind the object in the real space.  
           [0007]    A system disclosed in Japanese Patent Application Laid-Open (Kokai) No. 11-331874 (1999), serving as a conventional technique for knowing the position of a moving object in a real space in real time, obtaining a fore and aft relationship with an object in a virtual space, and displaying an image which is consistent in the fore and aft relationship, includes a position sensor for time serially inputting standing-point-position information, a stereoscopic camera where a plurality of images are consecutively input time serially, and an image processing apparatus.  
           [0008]    The image processing apparatus detects depth images from stereoscopic images, which are input in consecutive time series, in consecutive time series, assumes the observer&#39;s standing point in a future time at which the observer is provided with a three-dimensional image, based on a change in the standing point in the past which has been input from the position sensor, and consecutively changes consecutively obtained depth images into images from the assumed standing point in the future.  
           [0009]    However, in the system disclosed in the above-described patent application, since the standing point is assumed only based on information relating to a change in the standing point in the past which has been time serially input from the position sensor, accuracy in assumption is insufficient from the viewpoint of providing a more precise composite real space.  
         SUMMARY OF THE INVENTION  
         [0010]    The present invention has been made in consideration of the above-described problems.  
           [0011]    It is an object of the present invention to provide a composite-reality providing system in which, when fusing a real space and a virtual space, by exactly knowing the positional relationship between an object in the real space and an object in the virtual space, a very precise three-dimensionally consistent mixed real space can be provided.  
           [0012]    According to one aspect, the present invention which achieves the above-described object relates to an image-display control apparatus including detection means for detecting a predetermined object from a photographed image, position calculation means for obtaining a positional relationship between the predetermined object in the photographed image detected by the detection means and an object in a virtual image, image removing means for removing a predetermined partial image in the object in the virtual image by referring to a result of calculation of the position calculation means, synthesis means for synthesizing the virtual image and the photographed image so that the predetermined object in the photographed image is present at a position of the partial image in the virtual image removed by the image removing means, and display control means for displaying an image obtained as a result of synthesis of the synthesis means on display means.  
           [0013]    According to another aspect, the present invention which achieves the above-described object relates to an image-display control apparatus including detection means for detecting a predetermined object from each of first and second photographed images obtained by photographing a real space, position calculation means for obtaining a positional relationship between the predetermined objects in the first and second photographed images detected by the detection means and an object in a virtual image, image removing means for removing predetermined partial images in the object in the virtual image by referring to a result of calculation of the position calculation means, synthesis means for forming a first synthesized image by synthesizing the virtual image and the first photographed image and forming a second synthesized image by synthesizing the virtual image and the second photographed image so that the predetermined objects in the first and second photographed images are present at respective positions of the partial images in the virtual image removed by the image removing means, and display control means for displaying the first synthesized image obtained by the synthesis means on display means for a left eye, and displaying the second synthesized image on display means for a right eye.  
           [0014]    According to still another aspect, the present invention which achieves the above-described object relates to an image processing system including a camera for photographing a real space, a position/direction sensor for detecting a position and a direction of an object in the real space, a computer for synthesizing the photographed image photographed by the camera and a virtually formed virtual image, and display means for displaying an image obtained as a result of the synthesis by the computer. The computer includes image input means for inputting the photographed image photographed by the camera, detection means for detecting a predetermined object from the photographed image, position calculation means for obtaining a positional relationship between the predetermined object in the photographed image detected by the detection means and an object in a virtual image, image removing means for removing a predetermined partial image in the object in the virtual image by referring to a result of the calculation of the position calculation means, synthesis means for synthesizing the virtual image and the photographed image so that the predetermined object in the photographed image is present at a position of the partial image in the virtual image removed by the image removing means, and display control means for displaying an image obtained as a result of the synthesis of the synthesis means on the display means.  
           [0015]    According to yet another aspect, the present invention which achieves the above-described object relates to an image generation system including first and second cameras for photographing the same real space, a position/direction sensor for detecting a position and a direction of an object in the real space, a computer for outputting first and second synthesized images by synthesizing first and second photographed images photographed by the first and second cameras, respectively, and a virtually formed virtual image, and display means for a right eye and display means for a left eye for displaying the first and second synthesized images, respectively. The computer includes image input means for inputting the first and second photographed images, detection means for detecting respective predetermined objects from the first and second photographed images, position calculation means for obtaining a positional relationship between the predetermined objects in the first and second photographed images detected by the detection means and an object in a virtual image, image removing means for removing predetermined partial images in the object in the virtual image by referring to a result of calculation of the position calculation means, synthesis means for forming a first synthesized image by synthesizing the virtual image and the first photographed image and forming a second synthesized image by synthesizing the virtual image and the second photographed image so that the predetermined objects in the first and second photographed images are present at respective positions of the partial images of the virtual image removed by the image removing means, and display control means for displaying the first synthesized image obtained by the synthesis means on display means for a left eye, and displaying the second synthesized image on display means for a right eye.  
           [0016]    According to still another aspect, the present invention which achieves the above-described object relates to an image-display control method including a detection step of detecting a predetermined object from a photographed image, a position calculation step of obtaining a positional relationship between the predetermined object in the photographed image detected in the detection step and an object in a virtual image, an image removing step of removing a predetermined partial image in the object in the virtual image by referring to a result of the calculation in the position calculation step, a synthesis step of synthesizing the virtual image and the photographed image so that the predetermined object in the photographed image is present at a position of the partial image in the virtual image removed in the image removing step, and a display control step of displaying an image obtained as a result of synthesis in the synthesis step on display means.  
           [0017]    According to still another aspect, the present invention which achieves the above-described object relates to an image-display control method including a detection step of detecting a predetermined object from each of first and second photographed images obtained by photographing the same real space, a position calculation step of obtaining a positional relationship between the predetermined objects in the first and second photographed images detected in the detection step and an object in a virtual image, an image removing step of removing predetermined partial images in the object in the virtual image by referring to a result of calculation in the position calculation step, a synthesis step of forming a first synthesized image by synthesizing the virtual image and the first photographed image, and forming a second synthesized image by synthesizing the virtual image and the second photographed image so that the predetermined objects in the first and second photographed images are present at respective positions of the partial images of the virtual image removed in the image removing step, and a display control step of displaying the first synthesized image obtained in the synthesis step on display means for a left eye, and displaying the second synthesized image on display means for a right eye.  
           [0018]    The foregoing and other objects, advantages and features of the present invention will become more apparent from the following detailed description of the preferred embodiments taken in conjunction with the accompanying drawings. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0019]    [0019]FIG. 1 is a diagram illustrating an appearance and an outline of a composite-reality providing system according to an embodiment of the present invention;  
         [0020]    [0020]FIGS. 2A and 2B are diagrams, each illustrating an appearance and the structure of a head-mount display  100  shown in FIG. 1;  
         [0021]    [0021]FIG. 3 is a block diagram illustrating the configuration of the composite-reality providing system shown in FIG. 1;  
         [0022]    [0022]FIG. 4 is a diagram illustrating an image of image synthesis processing by the system shown in FIG. 1;  
         [0023]    [0023]FIG. 5 is a flowchart illustrating the processing of a computer  300  shown in FIG. 3 from input of an image from a camera to synthesis of a real space and a virtual space;  
         [0024]    FIGS.  6 A- 6 D are diagrams, each illustrating a pose of a player which is necessary for pre-processing performed before play;  
         [0025]    [0025]FIGS. 7A and 7B are diagrams, each illustrating an image of a method for assuming the position and the shape of an arm using position sensors  120  and  121  shown in FIG. 1;  
         [0026]    [0026]FIG. 8 is a diagram illustrating an image of a method for forming an external mask by combining position information from the position sensors  120  and  121  and position information from a photographed image; and  
         [0027]    [0027]FIG. 9 is a diagram illustrating an image of conventional synthesis processing as an image. 
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0028]    A preferred embodiment of the present invention will now be described with reference to the drawings. FIG. 1 is a diagram illustrating an appearance and an outline of a composite-reality providing system according to the preferred embodiment.  
         [0029]    As shown in FIG. 1, in this composite reality providing system, the player wears a video see-through-type head-mount display  100 . A HMD (head-mount display)-position sensor  120  is mounted on the head-mount display  100 , and is connected to a three-dimensional position sensor main body  200  via a cable.  
         [0030]    A palm-position sensor  121  is mounted on a palm of the player, and is also connected to the three-dimensional position sensor main body  200 .  
         [0031]    The thee-dimensional position sensor main body  200  is connected to a three-dimensional-position-sensor fixed station  210  for reading the positions of the moving position sensors  120  and  121 , and a computer  300  via cables.  
         [0032]    The principle of these position sensors will now be described. That is, the position sensors  120  and  121  detect the movement (the position and the direction) of the player by changes in magnetic lines of force provided by the magnetic field generated by the three-dimensional-position-sensor fixed station  210 , and transmit obtained information to the three-dimensional position sensor main body  200 . After converting the received information into information relating to the position and the direction of the player, the three-dimensional position sensor main body  200  supplies the computer  300  with the resultant information.  
         [0033]    The player enters a game field  400 , and feels a three-dimensional image provided by the composite-reality providing system. For example, within the game field  400 , when assuming a case in which an organism or the like by a virtual image moves around a hand where the palm-position sensor  121  is mounted, the organism performs various movements behind and in front of the hand.  
         [0034]    [0034]FIGS. 2A and 2B are diagrams, each illustrating an appearance and the structure of the head-mount display  100  shown in FIG. 1: FIG. 2A illustrates the front side; and FIG. 2B illustrates the back side having LCD (liquid-crystal display) surfaces.  
         [0035]    The head-mount display  100  is a video see-through-type display in which images photographed by a camera are displayed on display units, each comprising a LCD or the like, and includes a right-eye camera  110 , a left-eye camera  111 , a right-eye LCD  130 , a left-eye LCD  131 , and the HMD-position sensor  120 .  
         [0036]    The right-eye camera  110  and the left-eye camera  111  correspond to the standing point of the player&#39;s right eye and the standing point of the player&#39;s left eye, respectively. An image photographed by the right-eye camera  110  is first input to the computer  300 , and is then displayed on the right-eye LCD  130  by being fused with a virtual space (a virtual image). Similarly, an image photographed by the left-eye camera  111  is first input to the computer  300 , and is then displayed on the left-eye LCD  131  by being fused with the virtual space.  
         [0037]    [0037]FIG. 3 is a block diagram illustrating the configuration of the composite-reality providing system of the embodiment. In FIG. 3, the computer  300  includes a CPU (central processing unit)  301  and a memory  302  as main components. The CPU  301  and the memory  302  are connected to a serial I/O (input/output) unit  310 , a video capture card (1)  320 , a video capture card (2)  321 , a video card (1)  330 , and a video card (2)  331  via a PCI (Peripheral Component Interconnect) bridge  303 .  
         [0038]    The serial I/O unit  310  is connected to the three-dimensional position sensor main body  200 . The video capture card (1)  320 , the video capture card (2)  321 , the video card (1)  330  and the video card (2)  331  are connected to the right-eye camera  110 , left-eye camera  111 , the right-eye LCD  130  and the left-eye LCD  131 , respectively.  
         [0039]    Next, a description will be provided of an outline of image synthesis processing by the system of the embodiment having the above-described configuration, i.e., processing of superposing an image in a real space (a photographed image) and an image in a virtual space (a virtual image), with reference to FIG. 4.  
         [0040]    In a case in which a shark  10  and a rock  11  are present in the virtual space, a man&#39;s hand  12  is present in the real space, and composite reality is provided such that, as shown in a fused space  605  in FIG. 4, the shark  10  in the virtual space is present in front of the man&#39;s hand  12  in the real space, and the rock  11  in the virtual space is present behind the man&#39;s hand  12  in the real space, if, as shown in FIG. 9, the object in the virtual image  901  is simply superposed on the object in the real space  902 , the shark  10  and the rock  11  are positioned in front of the man&#39;s hand  12  in the fused space  903 . As a result, fusion of the virtual space and the real space does not succeed.  
         [0041]    Accordingly, in this system, a portion to be hidden in the real space is removed from the virtual space using a mask having the shape of the man&#39;s hand.  
         [0042]    That is, as shown in FIG. 4, a mask  602  having the shape of the man&#39;s hand is covered in a virtual space  601 . At that time, masking processing is performed based on a priority order for each object. The first, second and third priority orders are given in this embodiment to the shark  10 , the man&#39;s hand  12  and the rock  11 , respectively, from the front side. If masking processing is performed with these priority orders, the shark  10  is displayed as it is because it is present in front of the man&#39;s hand  12 , and a portion of the rock  11  overlapping with the mask  602  having the shape of the man&#39;s hand is removed because the rock  11  is present behind the man&#39;s hand  12  (see reference numeral  11   a  in FIG. 4).  
         [0043]    The image in the virtual space after the masking processing is as represented by reference numeral  603  shown in FIG. 4. The fused space  605  is obtained by synthesizing the image  603  and a real space  604 . In the fused space  605 , the shark  10  is positioned in front of the man&#39;s hand  12  and the rock  11  is positioned behind the man&#39;s hand  12 .  
         [0044]    As described above, in this system, when fusing a real space and a virtual space, it is always possible to provide a composite real space which is three-dimensionally consistent, whether an object in the real space is at the front side or an object in the virtual space is at the front side.  
         [0045]    The details of the operation of this system will now be described with reference to FIGS.  5 - 8 . FIG. 5 is a flowchart illustrating the processing of the computer  300  from input of an image from the camera to synthesis of a real space and a virtual space. FIGS.  6 A- 6 D are diagrams, each illustrating a pose of a player which is necessary for pre-processing performed before the play.  
         [0046]    This system is a stereoscopic-camera system using two cameras, i.e., the right-eye camera  110  and the left-eye camera  111 . Since processing performed in each of the right-eye system and the left-eye system is entirely the same, only processing in the right-eye system will be described. By storing a program for the flowchart shown in FIG. 5 in a storage device, such as the memory  302  within the computer  300 , or the like, and causing the program to operate, the following control method can be realized.  
         [0047]    First, the player takes predetermined poses before playing, and the positions of the position sensors  120  and  121  at each of the predetermined poses are obtained. In this embodiment, four poses, for example, a state in which an arm is straightly lowered (see FIG. 6A), a state in which the arm is horizontally stretched (see FIG. 6B), a state in which the arm is horizontally stretched and then the elbow is bent (see FIG. 6C), and a state in which the arm is horizontally stretched in the forward direction and then the elbow is bent (see FIG. 6D), are provided. By analyzing the position and the posture of the player based on these four poses, more accurate estimation of the position can be realized.  
         [0048]    When the system has been started, then, in step S 501 , an image signal is transmitted from the right-eye camera  110  to the video capture board  320 . The computer  300  captures an image from the right-eye camera  110  via the video capture card  320 .  
         [0049]    Then, in step S 502 , the three-dimensional-position-sensor fixed  10  station  210  detects the positions of the HMD-position sensor  120  and the palm sensor  121 , and assumes the position and the shape of the arm of the player (the object) from the positions detected by the two sensors  120  and  121 .  
         [0050]    A method for assuming the position and the shape of the arm will be described later with reference to FIGS. 7A and 7B.  
         [0051]    In step S 503 , the portion of the object (arm) is more precisely obtained from the image captured in step S 501 , based on position information assumed from the position sensors  120  and  121  in step S 502 .  
         [0052]    Then, in step S 504 , a virtual space is formed. Then, in step S 505 , the positional relationship between the object in the real space assumed in step S 503  and the object in the virtual space formed in step S 504  is obtained.  
         [0053]    Then, in step S 506 , an external mask corresponding to the shape of the object in the real space which is to cover the virtual space is formed. A method for forming the external mask will be described later with reference to FIG. 8.  
         [0054]    Then, in step S 507 , a portion not to be displayed in the virtual space ( 603  shown in FIG. 4) is removed by superposing the mask ( 602  shown in FIG. 4) on the virtual space ( 601  shown in FIG. 4).  
         [0055]    Then, in step S 508 , the object ( 603  shown in FIG. 4) and the object in the real space ( 604  shown in FIG. 4) are superposed. The resultant synthesized image ( 605  shown in FIG. 4) is an image based on the depth in the three-dimensional space.  
         [0056]    Then, in step S 509 , the synthesized image is displayed. The process then returns to step S 501 , and the same processing is performed.  
         [0057]    [0057]FIGS. 7A and 7B are diagrams, each illustrating an image of a method for assuming the position and the shape of the arm from the position sensors  120  and  121 : FIG. 7A illustrates an example of the shape of the arm; and FIG. 7B illustrates another example of the shape of the arm.  
         [0058]    As shown in FIGS. 7A and 7B, first, the position of each of the position sensors  120  and  121  is detected. Then, a frame model  122  is formed by performing assumption based on the detected position and direction of each of the sensors  120  and  121 . Finally, a human model is formed by putting flesh on the frame  122 .  
         [0059]    In FIGS. 7A and 7B, the sensors  120  have the same position, and the sensors  121  have different positions and directions. However, as described above, the flow of basic processing is entirely the same.  
         [0060]    [0060]FIG. 8 is a diagram illustrating an image of a method for forming an external mask by combining position information from the position sensors  120  and  121  and position information from the photographed image.  
         [0061]    First, in step  801 , sensor-position information is received from the position sensors  120  and  121 . Then, in step  802 , the position and the posture of the player are assumed from the sensor-position information obtained in the above-described step  802 . When the position and the posture of the player are known, the position and the direction of the camera can also be assumed simultaneously.  
         [0062]    In step  803 , an image to be photographed by the cameras  110  and  111  is assumed and constructed from the position and the posture of the player obtained in step  802 , and the position and the direction of each of the cameras  110  and  111 .  
         [0063]    In parallel to the above-described steps  801 - 803 , in step  804 , an input image from each of the cameras is captured. Then, in step  805 , the outer shape of the arm is extracted based on the assumed image constructed in step  803 , from the photographed images obtained in step  804 . The outer shape thus extracted is used as a mask (step  806 ).  
         [0064]    The present invention is not limited to the apparatus of the above-described embodiment, but may also be applied to a system comprising a plurality of apparatuses, or to an apparatus comprising a single unit. The present invention may, of course, be realized by supplying a system or an apparatus with a storage medium storing program codes of software for realizing the functions of the above-described embodiment, and reading and executing the program codes stored in the storage medium by means of a computer (or a CPU or an MPU (microprocessor unit)) of the system or the apparatus.  
         [0065]    In such a case, the program codes themselves read from the storage medium realize the functions of the above-described embodiment, so that the storage medium storing the program codes constitutes the present invention. For example, a floppy disk, a hard disk, an optical disk, a magnetooptical disk, a CD(compact disc)-ROM (read-only memory), a CD-R (recordable), a magnetic tape, a nonvolatile memory card or a ROM may be used as the storage medium for supplying the program codes. The present invention may, of course, be applied not only to a case in which the functions of the above-described embodiment are realized by executing program codes read by a computer, but also to a case in which an OS (operating system) or the like operating in a computer executes a part or the entirety of actual processing, and the functions of the above-described embodiment are realized by the processing.  
         [0066]    The present invention may, of course, be applied to a case in which, after writing program codes read from a storage medium into a memory provided in a function expanding board inserted into a computer or in a function expanding unit connected to the computer, a CPU or the like provided in the function expanding board or the function expanding unit performs a part or the entirety of actual processing, and the functions of the above-described embodiment are realized by the processing.  
         [0067]    As described above in detail, when fusing a real space and a virtual space, it is possible to exactly know the fore and aft relationship between an object in the real space and an object in the virtual space, and always provide a very precise composite real space which is three-dimensionally consistent.  
         [0068]    The individual components shown in outline or designated by blocks in the drawings are all well known in the image-display control apparatus arts and their specific construction and operation are not critical to the operation or the best mode for carrying out the invention.  
         [0069]    While the present invention has been described with respect to what are presently considered to be the preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiment. To the contrary, the present invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.