Abstract:
An information processing apparatus generates and displays a mixed reality image in which a plurality of subjects have virtual objects by their hands. The apparatus includes a unit for measuring a position of the hand of the subject, a unit for measuring a view position and a view direction of the subject, and a unit for generating the virtual object and synthesizing the virtual object with the real space image. Also included is an upcast action detecting unit for calculating a shifting speed of the hand in an upward direction and for detecting an upcast action, a unit for judging that the hand is stopped and a determining unit for determining a partner subject to whom the virtual object is upcast.

Description:
BACKGROUD OF THE INVENTION 
   1. Field of the Invention 
   The present invention relates to a technique for controlling a virtual object in a compound real-like space image by an action of a hand of a subject. 
   2. Related Background Art 
   As user interfaces in compound real-like systems, there have been proposed a method for recognizing a gesture from a position of a hand obtained by a position sensor and a method for recognizing a direction along which a hand is moved. 
   However, regarding the casting of a virtual object, it was difficult to judge a moment when the virtual object is left from the hand by an upcast action and starts to move. Further, it was possible to identify a partner who receives based on a direction along which the hand is moved, but there were problems that collision accident may occur in a narrow area and control of a delicate direction is difficult. 
   Further, in order to generate an event in which objects are exchanged with each other simultaneously, since it was necessary to align timings for casting completely, it was difficult to generate such an event easily. 
   SUMMARY OF THE INVENTION 
   The present invention is made in consideration of the above-mentioned problems, and an object of the invention according to one aspect is to realize the casting of a virtual object to a partner who is desired by a subject by a simple and safe method in an apparatus for generating a compound real-like space image by synthesizing a real space image and a virtual space image. 
   Further, an object of the invention according to another aspect is to easily realize simultaneous exchanging. 
   In order to achieve the above objects, the present invention according to one aspect under the above objects relates to an information processing apparatus for generating and displaying a compound real space image in which a plurality of subjects have virtual objects by their hands respectively, by synthesizing a real space image and a virtual space image representing a virtual object, comprising means for measuring a position posture of the hand of the subject, means for measuring a view position and a view direction of the subject, and means for generating the virtual object in dependence upon the view position, the view direction and the position posture of the hand, and synthesizing the virtual object image with the real space image, and further comprising upcast action detecting means for calculating a shifting speed of the hand in an upward direction and for detecting an upcast action, means for judging that the hand is stopped if the shifting speed of the hand in the upward direction becomes smaller than a predetermined value after the upcast action was detected by the detecting means, and determining means for determining a partner subject to whom the virtual object is upcast on the basis of the view position and the view direction of the subject when it is judged that the hand is stopped. 
   The present invention according to another aspect under the above object comprises judging means for judging whether or not the partner subject is effecting the upcast action when the judged upcast action to the partner subject is decided, measuring means for measuring the view direction of the partner subject, and control means for controlling a timing of processing for casting and receiving the virtual object between the subject and the partner subject in accordance with a judged result of the judging means and a measured result of the measuring means. 
   The present invention according to a further aspect under the above objects relates to an information processing method in which a position posture of a hand of a subject is measured, a view position and a view direction of the subject are measured, a virtual space image is generated in accordance with the view position, the view direction and the position posture of the hand, and a compound real-like space image is generated by synthesizing the virtual space image with a real space image, and wherein, when a shifting speed of the hand of the subject is detected and a condition that the shifting speed is not less than a predetermined value continues for a time not less than a predetermined time, the virtual space image is moved to another subject, and the another subject to whom the virtual space image is to be shifted is judged on the basis of the view position and the view direction when the shifting speed of the hand becomes smaller than a predetermined value after the condition that the shifting speed of the hand continues for the time not less than the predetermined time. 
   The present invention according to a further aspect under the above objects is characterized in that, if a condition that a shifting speed of a hand of another subject is not less than a predetermined value continues when the shifting speed of the hand becomes smaller than a predetermined value after a condition that the shifting speed of the hand is not less than the predetermined value continues for the time not less than the predetermined time, the respective virtual space images are moved from each other simultaneously between the subject and the another subject. 
   The other objects, advantages and features of the present invention will be apparent from the following detailed explanation of the invention. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
       FIG. 1  is a system constructural view showing a schematic construction of a compound real-like system; 
       FIG. 2  is a constructural view showing a construction of an HMD; 
       FIG. 3  is a flow chart showing production processing for a compound real space image; 
       FIG. 4  is a perspective view of the HMD, looked at from a direction of a photo-taking portion; 
       FIG. 5  is a perspective view of the HMD, looked at from a displaying portion; 
       FIG. 6  is a block diagram showing a hardware construction for one observer in the system of  FIG. 1 ; 
       FIG. 7  is a flow chart showing processing for a casting event; and 
       FIG. 8  is a flow chart showing processing for a simultaneous exchanging event. 
   

   DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
   The present invention will now be explained in connection with embodiments thereof with reference to the accompanying drawings. 
   First Embodiment 
     FIG. 1  is a system constructural view showing a schematic construction of a compound real-like system to which a first embodiment of the present invention is applied. 
   A first observer  100  wears an HMD (Head Mount Display)  110  on his head and a glove  120  on his hand. 
   As shown in  FIG. 2 , the HMD  110  is constituted by a video camera  111 , an LCD  112 , a position direction measuring device receiver (position sensor for eyes)  113 , and optical prisms  114 ,  115 . The video camera  111  photo-takes a real space image in a view position and a view direction of the observer directed by the optical prism  115 . The position sensor  113  for eyes is utilized to measure the view position and the view direction of the observer. The LCD  112  serves to display a compound real space image and the image is directed to the observer by the optical prism  114 . 
   The glove  120  incorporates therein a position sensor  121  for a hand and a speaker (not shown). The position sensor  121  for hand is utilized as a sensor for measuring a position and a direction of the hand of the observer. The speaker emits a sound corresponding to an event generated at the hand position. As such a sound, for example, a sound generated when a virtual space object is touched or struck by the hand or a sound generated when a state of the virtual space object displayed in sync with the hand position is changed can be considered. 
   There are provided a position direction measuring device transmitter (position sensor transmitter)  130  and a position direction measuring device main body (position sensor main body)  131 . The position sensor  113  for eyes, the position sensor  121  for hand and the position sensor transmitter  130  are connected to the position sensor main body  131 . The position sensor transmitter  130  emits magnetism, and the magnetism is received by the position sensor  113  for eyes and the position sensor  121  for hand. The position sensor main body  131  calculates positions and directions of the eyes and the hand on the basis of received intensity signals from the position sensor  113  for eyes and the position sensor  121  for hand. Here, as the position direction measuring device, FASTRAK manufactured by Polhemus Inc. in USA can be used. 
   A processing apparatus  140  serves to produce the compound real space image for one observer and to display such an image on the HMD  110 . For example, the processing apparatus  140  is constituted by a personal computer, and a video capture card, a video card having a CG drawing function or a sound card. The HMD  110 , the speaker and the position sensor main body  131  are connected to the processing apparatus  140 . 
   A virtual character  170  is compounded as if it is rested on the hand of the first observer  100 . A view line of the first observer is denoted by the reference numeral  180 . The view line can be measured by the position sensor  113  for eyes, the position sensor transmitter  130  and the position sensor main body  131 . A second observer  101  also has the same construction as the first observer. 
   Next, a procedure for producing the compound real space image in the processing apparatus  140  will be explained with reference to a flow chart shown in  FIG. 3 . 
   First of all, the processing apparatus  140  takes the view position and the view direction, and the position and direction of the hand sent from the position sensor main body  131  (step S 301 ). Incidentally, in the step S 301 , thread S 311  for periodically acquiring the view position and the view direction, and the position and direction of the hand sent from the position sensor main body  131  is utilized. 
   Then, the time of the virtual space is renewed and a virtual space state (kind, position and condition of the virtual space object) is renewed (step S 302 ). In this case, if there is a virtual space object in which a position direction thereof is changed in sync with the position direction of the real space object, such an object is also renewed together. For example, when it is pretended to be a condition that the character of the virtual always rides on the hand, in this step S 302 , the position direction of the glove is renewed. 
   Then, a relationship between the position direction (position of hand and view position) of the real space object and the position direction of the virtual space object is checked, and, if it is judged that a pre-defined event is generated, the condition of the virtual space is renewed in accordance with such an event (step S 303 ). For example, it is considered that, if the virtual space object is touched by the hand, the virtual space object is exploded. 
   Then, the real space image in the view position and the view direction of the observer obtained from the video camera  111  is taken in (step S 304 ). In this step S 304 , thread S 314  for periodically acquiring the virtual space image obtained by the video camera  111  from the video capture card is utilized. 
   And, the virtual space image from the view position and the view direction of the observer acquired in the step S 301  is produced in accordance with the condition of the virtual space renewed in the steps S 302  and S 303  (step S 305 ). 
   Lastly, the virtual space image produced in the step S 305  is synthesized with the real space image taken-in in the step S 304 , and the synthesized image is outputted to the LCD  112  of the HMD  110  (step S 306 ). The above-mentioned processing operations are carried out repeatedly until any end action is effected (step S 307 ). 
     FIGS. 4 and 5  are perspective views of the HMD  110 , where  FIG. 4  is a perspective view looked at from a direction of a photo-taking portion and  FIG. 5  is a perspective view looked at from a displaying portion. 
   The HMD includes an HMD displaying portion  201 . The HMD displaying portion  201  has two displaying portions, i.e. a displaying portion  201 R for right eye and a displaying portion  201 L for left eye, and each displaying portion includes a color liquid crystal and a prism and serves to display the compound real space image corresponding to the view position and the view direction of the observer. 
   There are provided constructural members  204  to  208  for head mounting. When the HMD  301  is mounted on the head, first of all, the HMD is worn on the head with a condition that a length adjusting portion  206  loosened by an adjuster  205 . Then, after a forehead mounting portion  208  is closely contacted with the forehead, the length adjusting portion  206  may be tightened by the adjuster  205  so that a side head mounting portion  204  and a rear head mounting portion  207  are closely contacted with the side head portion and the rear head portion, respectively. 
   There is provided an HMD photo-taking portion  203  for photo-taking the real space image in the view position and the view direction of the observer. The HMD photo-taking portion  203  has two photo-taking portions, i.e. a photo-taking portion  203 R for right eye and a photo-taking portion  203 L for left eye, and each photo-taking portion is constituted by a small-sized camera of NTSC type. The photo-taken real space image is overlapped with the virtual space image to generate the compound real space image. 
   A receiver  302  is utilized to receive magnetism emitted from the transmitter as information for measuring the view position and the view direction of the subject. As attachment portions for attaching the receiver  302  to the HMD  301 , there are three portions, i.e. receiver joint portions  200 R,  200 L and  200 C, so that the receiver can be detachably attached to any of these receiver joint portions  200 R,  200 L and  200 C. That is to say, in  FIGS. 4 and 5 , although the receiver  302  is attached to the receiver joint portion  200 R at the right in the advancing direction of the view line of the observer, it can be attached to the receiver joint portion  200 L at the left in the advancing direction of the observer or to the receiver joint portion  200 C on the middle line of the observer. 
   In the illustrated embodiment, although the receiver joint portions  200 R,  200 L and  200 C each has a construction having an insertion opening into which the receiver  302  is fixedly inserted, other detachable joint (attachment) systems may be used. 
   A receiver signal line  210  is exposed externally of the HMD  301  from the vicinity of the receiver joint portion  200 C. The receiver signal line  210  has an adequate length so that the receiver  302  can be attached to either of the receiver joint portions  200 R,  200 L and  200 C. 
   There is provided bundled wires  209  obtained by bundling signal lines and power supply wires for the HMD displaying portion  201  and the HMD photo-taking portion  203  and various wires such as the receiver signal line  210  and the like, and the bundled wires are attached to the rear head mounting portion  207 . The signal lines and the power supply wires, among the bundled wires  209 , for the left and right HMD displaying portions  201 R,  201 L, HMD photo-taking portions  203 R,  203 L and the like are passed through the left and right side head mounting portions  204 . 
     FIG. 6  is a block diagram showing a hardware construction for one observer in the system of  FIG. 1 . A processing apparatus (computer)  307  includes a video capture board  350  for right eye, a video capture board  351  for left eye, a graphic board  352  for right eye, a graphic board  353  for left eye, an I/O interface  354  and a network interface  359 , and these elements are connected to a CPU  356 , an HDD  355  and a memory  357 . 
   The video capture boards  351 ,  350  for left and right eyes are connected to video cameras  203 L,  203 R for left and right eyes, respectively, so that the actually taken images photo-taken by the video cameras  203 L,  203 R can be converted to forms which can be synthesized with the virtual space image in the processing apparatus  307 . Further, the graphic boards  353 ,  352  for left and right eyes are connected to the displaying portions (display devices)  201 L,  201 R for left and right eyes, respectively, thereby to effect display control for these displaying portions  201 L,  201 R for left and right eyes. 
   Further, the I/O interface  354  is connected to a position direction measuring device main body  306  and the network interface  359  is connected to a network  330 . 
   In the illustrated embodiment, by using the above-mentioned system, a water surface of CG is generated on a palm of the hand, and, further, the compound real space image for synthesizing the virtual character is provided there. 
   In the illustrated embodiment, the operation of the virtual character is controlled by using the action of the hand. Hereinbelow, a method for effecting the casting of the virtual character will be concretely explained with reference to  FIGS. 7 and 8 . 
   In a step S 2001 , the upcast action is detected from the measured result of the position sensor for hand. For example, the fact that the first observer  100  is moving his hand upwardly at a speed not less (greater) than a predetermined speed (value) for a time period not less (greater) than a predetermined time (value) is detected. That is, the detection of the upcast action is executed by comparing the speed and the time of the upcast action with the predetermined value. The hand motion can be detected by analyzing the progress of the hand position detected by the position sensor  121  for hand. 
   In a step S 2002 , from the measured result of the position sensor for hand, by detecting the time when the ascending speed (value) of the hand becomes smaller (not more) than a predetermined speed (value) after the upcast action was detected in the step S 2001 , the stopping of the upcast action is detected. That is, the stopping of the upcast action is detected by comparing the speed of the upcast action with the predetermined value. 
   In a step S 2003 , the partner who receives the virtual character is judged. In  FIG. 1 , although only two observers are drawn and other observers are omitted, in the illustrated embodiment, it is assumed that at least one observer is present other than the drawn observers. The partner who receives is determined from the view line  180  of the first observer  100  at the moment when the stopping of the upcast action is detected in the step S 2002 . 
   As shown in  FIG. 1 , since each observer has the position sensor, the position of each observer can be grasped. In the step S 2002 , the observer who exists at a position nearest to the view direction of the observer who is effecting the upcast action is detected, and the detected observer is determined as the partner who receives. Now, it is assumed that the second observer  101  is the partner who receives and explanation continues. 
   In a step S 2004 , the virtual character (virtual object) is moved in the compound real space to jump the virtual character toward the hand of the second observer determined in the step S 2003 . This can be realized by controlling the drawing of the virtual space image in the step S 305  of  FIG. 3  and synthesis in the step S 306 . 
   Through the above-mentioned processing operations, by the simple upcast action using the hand, the upcast action of the virtual object to the second observer can be realized. 
   According to the illustrated embodiment, the determination of the moment when the movement is started from the position sensor for hand can be judged simply and properly, and the partner can be determined easily and properly from the view direction determined from the sensor for eye. Thus, according to the illustrated embodiment, the upcast action of the virtual object to any observer can be determined easily and properly. 
   Next, a method for effecting simultaneous exchanging of the virtual characters will be explained with reference to  FIG. 8 . 
   Procedures until the partner who receives is determined in the step S 2003  are the same as those shown in  FIG. 7 . In a step S 2005 , the state of the partner who receives (second observer) is checked, and, if the second observer is effecting the upcast action, in a step S 2006 , the view direction of the second observer is judged, and, if the second observer is looking at the first observer, in a step S 2007 , the event for effecting the simultaneous exchanging is generated. As such, according to the illustrated embodiment, the timing for casting the virtual object can easily be fitted. 
   If the steps S 2005  and S 2006  are not established, in the step S 2004 , the virtual character is merely cast from the first observer to the second observer. 
   According to the illustrated embodiment, in the apparatus for producing and displaying the compound real-like space image by synthesizing the real space image with the virtual space image, the upcasting of the virtual object to the partner who is desired by the subject can be effected by the convenient and safe method, and the simultaneous exchanging can also be realized. 
   Other Embodiments 
   It should be noted that the present invention includes a concept that a program code of software for realizing the functions of the above-mentioned embodiment is supplied to a computer in an apparatus or a system connected to various devices to operate such devices thereby to realize the functions shown in  FIGS. 7 and 8 , and the various devices are operated in accordance with a program stored in the computer (CPU or MPU) of the apparatus or the system. 
   In this case, the program code of the software itself realizes the functions of the above-mentioned embodiment, and, thus, the program code itself and means for supplying the program code to the computer, for example, a storing medium storing such a program code constitute the present invention. 
   As the storing medium for storing such a program code, for example, a floppy disk, a hard disk, an optical disk, a photo-magnetic disk, CD-ROM, a magnetic tape, a non-volatile memory card or ROM can be used. 
   Further, it should be noted that, when not only the functions of the above-mentioned embodiment are realized by executing the supplied program code by means of the computer, but also the function of the above-mentioned embodiment are realized in cooperation with OS (operating system) in which the program code is worked in the computer or other application software, the program code constitutes the present invention. 
   Further, it should be noted that the present invention includes a concept that, after the supplied program code is stored in a memory of function expansion board inserted into the computer or of a function expansion unit connected to the computer, a CPU of the function expansion board or of the function expansion unit executes the actual processing partially or totally on the basis of instruction of the program code to realize the functions of the above-mentioned embodiment. 
   Although the present invention was explained in connection with preferred embodiments thereof, any person skilled in the art can recognize that various modifications and alterations can be made without departing from the sprit and scope of the invention.