Augmented reality game system using identification information to display a virtual object in association with a position of a real object

An image processing technology for displaying a real object and a virtual object associated with each other is provided. An image analysis apparatus according to the present invention changes a motion pattern of the virtual object depending on the actual movement of the real object. The change detector 110 detects temporal state change in the motion pattern of the real object captured by an imaging apparatus and the display controller 120 reads a motion pattern from the motion pattern storage 122 based on the event generated by the state change of an image. The display pattern controller 122 controls display mode of the virtual object using the read motion pattern.

This application is a National Phase Application of International Application No. PCT/JP2005/009547, filed May 25, 2005, which claims the benefit under 35 U.S.C. 119 (a-e) of Japanese Application No. 2004-254886 filed Sep. 1, 2004, and Japanese Application No. 2004-254887 filed Sep. 1, 2004, which is herein incorporated by reference.

FIELD OF THE INVENTION

The present invention relates to a technology for processing an image, and more specifically, to a technology for displaying an image where a real object and a virtual object are associated.

BACKGROUND TECHNOLOGY

Recently, the technology is widely available for capturing an image of a two-dimensional code by a camera for recognition and allowing a predetermined process associated with the code pattern to be performed. As one of those techniques, an image analysis technique is proposed which captures two-dimensional barcode using a video camera and displays a three-dimensional image corresponding to a two-dimensional barcode on a displaying device (e.g. Japanese Laid-Open Publication No. 2000-322602). According to the Japanese Laid-Open Publication No. 2000-322602, a spatial coordinate where a three-dimensional object is displayed is determined based on the coordinate of captured two-dimensional barcode and a focal distance of a CCD video camera and a three-dimensional image is superimposed on the two-dimensional barcode.

DISCLOSURE OF THE INVENTION

Problem to be Solved by the Invention

The technique disclosed in Japanese Laid-Open Publication No, 2000-322602 allows realizing excellent visual effects by displaying a real object and a virtual object combined. However, the Japanese Laid-Open Publication No. 2000-322602 only discloses a displaying technique of a virtual object when a two-dimensional barcode remains at rest. Thus the document does not show awareness of display processing of a virtual object when a two-dimensional barcode is moved. The present inventor has focused on display processing when a two-dimensional barcode is moved and found out the possibility of realizing a still more excellent visual effect by devising display processing of a virtual object and applying the display processing to a field of moving images, for example, a game.

The document only discloses a displaying technique of a virtual object associated with a single two-dimensional barcode. Thus it does not show awareness of display processing with a plurality of two-dimensional barcodes. The present inventor has focused attention on display processing when a plurality of real objects, such as two-dimensional barcodes, exist in real space, and found out the possibility of realizing a still more excellent visual effect by devising display processing of a virtual object and applying the display processing to, for example, a game.

A general purpose of the present invention is to provide a technique for establishing association and displaying a real object and a virtual object, especially to provide a technique for controlling a virtual object when displayed in relation to the movement of a real object, in a field of moving images, for example, a game.

Another general purpose of the present invention is to provide a technique for establishing association and displaying a real object and a virtual object, especially to provide a technique for controlling a virtual object when displayed based on the positional relation of a plurality of real objects.

Means to Solve the Problem

In this background, an image processing apparatus according to at least one embodiment of the present invention comprises: a storage which stores identification information for identifying a real object and three-dimensional image data of a virtual object associated with each other; a reader which reads three-dimensional image data of a virtual object associated with identification information on an image of the real object included in a frame image captured by an imaging apparatus from the storage; a display controller which displays the virtual object in association with a displayed position of the real object using read three-dimensional image data; and a change detector which detects a temporal state change of an image of a real object captured by the imaging apparatus. In the image processing apparatus, the display controller controls the virtual object as displayed based on a state change detected by the change detector. The term “a real object” means an object existing in real space as tangible goods, and the term “a virtual object” means a non-existing object in real space, which is represented by data in virtual space.

The image processing apparatus provides a new visual effect of a virtual object associated with the motion of a real object since the displayed virtual object is controlled based on the state change of a captured real object image (i.e., an actual motion of the real object).

A game apparatus according to at least one embodiment of the present invention comprises: a storage which stores identification information for identifying a real object and three-dimensional image data of a game character associated with each other; a reader which reads three-dimensional image data of a game character associated with identification information on an image of a real object included in a frame image captured by an imaging apparatus from the storage; a display controller which displays the game character in association with a displayed position of the real object using read three-dimensional image data; and a change detector which detects a temporal state change of an image of a real object captured by the imaging apparatus. In the game apparatus, the display controller controls the game character as displayed based on the state change detected by the change detector.

This game apparatus provides a new visual effect of a game character associated with the motion of a real object since the displayed game character is controlled based on the state change of a captured real object image (i.e., the motion of the real object).

An image processing method according to at least one embodiment of the present invention comprises: reading three-dimensional image data of a virtual object associated with identification information on an image of a real object included in a frame image captured by an imaging apparatus from a storage which stores identification information for identifying a real object and three-dimensional image data of a virtual object associated with each other; displaying a virtual object, establishing association with a displayed position of the real object, using read three-dimensional image data; detecting a temporal state change of an image of a real object captured by the imaging apparatus; and controlling the virtual object as displayed based on the detected state change.

A computer program product according to at least one embodiment of the present invention comprises; a reading module which reads three-dimensional image data of a virtual object associated with identification information on an image of a real object included in a frame image captured by an imaging apparatus from a storage which stores identification information for identifying a real object and three-dimensional image data of a virtual object associated with each other; a displaying module which displays a virtual object, establishing association with a displayed position of the real object, using read three-dimensional image data; a detecting module which detects a temporal state change of an image of a real object captured by the imaging apparatus; and a controlling module which controls the virtual object as displayed based on the detected state change.

An image processing apparatus according to at least one embodiment of the present invention comprises: a storage which stores identification information for identifying a real object and three-dimensional image data of a virtual object associated with each other; a positional relation detector which detects a positional relation among a plurality of real object images included in a frame image captured by an imaging apparatus; a condition evaluator which determines whether the detected positional relation among at least two of real object images fulfills a predefined condition; a reader which reads three-dimensional image data of a plurality of virtual object images associated with identification information on a plurality of real objects included in a frame image captured by the imaging apparatus; and a display controller which, in case the condition evaluator determines that the predefined condition is fulfilled, determines a displaying pattern of a virtual object based on identification information on at least two real object images which fulfills the predefined condition and performs display processing of a virtual object according to the determined display pattern, using a plurality of pieces of read three-dimensional image data. The term “a real object” means an object existing in real space as tangible goods, and the term “a virtual object” means a non-existing object in real space, which is represented by data in virtual space.

According to the image processing apparatus, new visual effect of a virtual object is provided by placing a plurality of real objects in a pre-determined positional relation since the image processing apparatus controls the virtual object as displayed based on the positional relation among a plurality of captured real object images.

A game apparatus according to at least one embodiment of the present invention comprises: a storage which stores identification information for identifying a real object and three-dimensional image data of a game character associated with each other; a positional relation detector which detects a positional relation among a plurality of real object images included in a frame image captured by an imaging apparatus; a condition evaluator which determines whether the detected positional relation among at least two real object images fulfills a predefined condition; a reader which reads three-dimensional image data of a plurality of game characters associated with identification information on a plurality of real objects included in a frame image captured by an imaging apparatus; and a display controller which, in case the condition evaluator determines that the predefined condition is fulfilled, determines a displaying pattern of a game character based on identification information on at least two real object images which fulfill a predefined condition and performs display processing of a game character according to the determined display pattern, using a plurality of pieces of read three-dimensional image data.

According to the game apparatus, a new visual effect of a game character is provided by placing a plurality real objects in a pre-determined positional relation since the game apparatus controls the game character as displayed based on the positional relation among a plurality of captured real object images.

An image processing method according to at least one embodiment of the present invention comprises: detecting a positional relation among a plurality of real object images included in a frame image captured by an imaging apparatus; determining whether the detected positional relation among at least two of real object images fulfills a predefined condition; reading three-dimensional image data of a plurality of virtual objects associated with identification information on a plurality of real objects included in a frame image; determining a displaying pattern of a virtual object based on identification information on at least two real object images which fulfill the predefined condition, in case the predefined condition is determined to be fulfilled; and performing display processing of a virtual object according to the determined display pattern, using a plurality of pieces of read three-dimensional image data.

A computer program product according to at least one embodiment of the present invention comprises: a detecting module which detects a positional relation among a plurality of real object images included in a frame image captured by an imaging apparatus; a determining module which determines whether the detected positional relation among at least two of real object images fulfills a predefined condition; a reading module which reads three-dimensional image data of a plurality of virtual objects associated with identification information on a plurality of real object images included in a frame image from a storage which stores identification information for identifying a real object and three-dimensional image data of a virtual object associated with each other; a determining module which determines displaying pattern of a virtual object based on identification information on at least two real object images which fulfill a predefined condition, in case the predefined condition is determined to be fulfilled; and a performing module which performs display processing of a virtual object according to the determined display pattern, using a plurality of pieces of three-dimensional image data read from the storage.

Optional combinations of the aforementioned constituting elements, and implementations of the invention in the form of methods, apparatuses, systems, recording mediums and computer programs may also be practiced as additional modes of the present invention.

The present invention enables to provide a technique for controlling a virtual object relating to a real object.

The present invention enables to provide a technique for controlling a virtual object based on positional relation of a plurality of real objects.

DESCRIPTION OF THE REFERENCE NUMERALS

BEST MODE FOR CARRYING OUT THE INVENTION

First Embodiment

The first embodiment of the present invention provides a technique wherein a temporal state change of a real object image captured by an imaging apparatus is detected and a mode, for example an appearance, of displaying virtual object is controlled based on the detected change. A real object may be a one-dimensional object, a two-dimensional object or a three-dimensional object. It is favorable that a real object be provided with a distinctive part that identifies the real object. For example, a real object may be a two-dimensional object, such as a card that is provided with two-dimensionally represented coded information as a distinctive part, or a three-dimensional object with uniquely shaped three-dimensional part as a distinctive part. A two-dimensional shape of a two-dimensional object may constitute a unique distinctive part or distinctive coded information may be affixed on a three-dimensional object. A virtual object may be, so to say, a character, such as a person, an animal or material goods that is represented three-dimensionally in virtual space. The first embodiment described below relates to an image processing technology in a game application and adopts a game character presented in a game application as a virtual object

FIG. 1shows the structure of a game system1according to the first embodiment. The game system1is provided with an imaging apparatus2, an image processing apparatus10and an output apparatus6. The image processing apparatus10is provided with an image analysis apparatus20and a game apparatus30. The image analysis apparatus20and the game apparatus30may be separate apparatuses or may be integrally combined. The imaging apparatus2is embodied by a video camera comprising a charge coupled device (CCD) imaging element, a metal oxide semiconductor (MOS) imaging element or the like. The imaging apparatus2captures an image of real space periodically so as to generate a frame image in each period. An imaging area5represents a range captured by the imaging apparatus2. The position and size of the imaging area5are adjusted by adjusting the height and orientation of the imaging apparatus2. A game player manipulates a game card4, a real object, in the imaging area5. The game card4is provided with a distinctive part that uniquely identifies the card.

The output apparatus6is provided with a display7. The output apparatus6may also be provided with a speaker (not shown). The image processing apparatus10causes the display7to display a frame image captured by the imaging apparatus2. In this process, the image processing apparatus10controls a character, a virtual object, to be superimposed on the game card4when displayed. The player can easily recognize whether the game card4is located within the imaging area5by watching the display7. If the game card4is not located within the imaging area5, the player may allow the imaging apparatus2to capture the image of the game card4by shifting the position of the game card4or by adjusting the orientation of the imaging apparatus2.

In the game application according to the first embodiment, the player moves a character by manipulating the game card4. As called for by the nature of the game application, it is favorable that the player feel the sense of unity between the game card4and the character. For this purpose, the image of the character is superimposed on the game card4. As the player moves the game card4slowly in the imaging area5, the character tracks the movement of the game card4, remaining placed on the game card4.

The character's motion is controlled by the image processing apparatus10. First, the image analysis apparatus20extracts image information for the game card4from the frame image captured by the imaging apparatus2. The image analysis apparatus20further extracts the unique distinctive part to identify the game card4from the image information on the game card4. In this process, the image analysis apparatus20determines attitude information, orientation information and distance information on the game card4in space, by referring to the image information on the game card4.

FIG. 2shows an exemplary two-dimensional code printed on the surface of a game card4. An orientation indicator11and an identification indicator12are printed on the surface of the game card4. The orientation indicator11is provided to indicate the front side of the game card4and the identification indicator12is provided to represent a distinctive field to identify the card uniquely. An identification indicator12is coded information made by a plurality of blocks printed on a predetermined field. Of the plurality of blocks, four corner blocks are given to a plurality of game cards4commonly. Thus actually, a distinctive part is comprised of blocks other than the four corner blocks. The four corner blocks are used to measure a distance from an imaging apparatus2.

Referring back toFIG. 1, the image processing apparatus20determines the distance between the imaging apparatus2and the game card4from the length between the four corner blocks in the image of the game card4identified in the frame image. The image processing apparatus20further determines the orientation of the game card4by using the orientation indicator11. In this case, the orientation indicator defines the front side and the character is controlled so that the character faces forward when displayed on the game card4. The image analysis apparatus20also acquires identification information on the game card4by referring to an array of blocks other than the four corner blocks.

WhileFIG. 1shows a state in which the game card4is put on a table3, the game card4may be inclined with respect to the table3, or may be elevated from the table3. The image analysis apparatus20has the function of recognizing the inclined position of the game card4or variation in the height of the game card4with respect to the table3, through image analysis. The result of image analysis by the image analysis apparatus20is sent to the game apparatus30. The frame image captured by the imaging apparatus2may be sent to the game apparatus30for image analysis by the game apparatus30. In this case, the image processing apparatus10may be formed only of the game apparatus30.

The game apparatus30controls the character to be displayed on the game card4on the screen of the display7based on the result of image analysis by the image analysis apparatus20. A character may be assigned to each game scene for a game card4as appropriate. In this case, when game scenes are switched, displayed characters are also changed. In the first embodiment, the game apparatus30detects a change over time of the captured game card image through imaging analysis results. Based on the state change, the game apparatus30controls display mode, for example an appearance, of a character.

FIG. 3shows the structure of the image analysis apparatus. The image analysis apparatus20is provided with a frame image acquirer40, a real object extractor42, a state determiner44, an identification information acquirer46, an identification information storage48and a transferring unit50. The identification information storage48stores information on a distinctive field for identifying a real object and identification information for identifying the real object with each other. To be more specific, the identification information storage48stores pattern information on the identification indicator12and identification information in a one-to-one relationship. Identification information is used to allot a character in the game apparatus30. Especially, in a game application that allows a plurality of game cards4to exist, associating respective game card4with identification information allows to recognize each game cards4. The state determiner44determines the state of the real object in the defined coordinate system. More specifically, the state determiner44is provided with an attitude determiner52which determines the attitude of a card, an orientation determiner54which determines the orientation of a card and a distance determiner56which determines the focal distance from the imaging apparatus2.

The frame image acquirer40acquires a frame image of real space captured by the imaging apparatus2. Given that one game card is placed in the imaging area5here, as shown inFIG. 1. The imaging apparatus2captures a frame image at regular intervals. Preferably, the imaging apparatus2generates frame images at intervals of 1/60 second.

The real object extractor42extracts a real object image, i.e., an image of the game card4, from the frame image. This process is performed by translating image information into binary bit representation and extracting the image of the game card4from binary bit representation (i.e. dot processing). Extracting an image may be performed by detecting ons and offs of a bit. This process may also be performed by a known image matching technology. In this case, the real object extractor42registers image information on a real object to be used in a memory (not shown) beforehand. Matching registered image information and captured image information allows cutting out an image of a game card4from the frame image.

The attitude determiner52determines the attitude of the real object image. More specifically, the attitude determiner52determines the coordinate of the center point of the real object image, the inclination of the real object image with respect to the table3, height of the real object image from the table3and the like. For that purpose, the state determiner44detects geometry information of the table3on which the game card4is placed and moved beforehand. The state determiner44further defines the surface of the table3as a reference plane and records geometry information on the attitude of the game card4placed on the reference plane as the initial attitude of the game card4. This geometry information may be formed with reference to the imaging apparatus2. The state determiner44maintains the position, the attitude and the like of the table3as coordinate data in imaging space from the geometry information acquired of table3. The attitude determiner52determines the inclination and height of the game card4with respect to the reference plane as a difference in relative quantity of state from the attitude of the game card4recorded as the initial state (i.e., difference of coordinate values in imaging space). In case a player picks up or inclines the game card4, change in quantity of state in reference to the initial attitude occurs and the height and the inclination of the real object image with respect to the table3change.

The orientation determiner54determines the orientation of the real object image. The orientation determiner54may detect the orientation indicator11shown inFIG. 2from the real object image and determine the orientation of the real object. The orientation determiner54may also determines the orientation of the real object as the orientation of inclination in case the inclination of the real object is recognized by the altitude determiner52.

The distance determiner determines the distance between the imaging apparatus2and the game card4from the length among the four corners of the identification indicator12in the image of the game card4.

The identification information acquirer46extracts a distinctive feature from the real object image and acquires a corresponding identification information from the identification information storage48. WhileFIG. 1shows one game card4, the game system1according to the first embodiment is compatible with a plurality of game cards4. For example, in case five game cards4are allowed to be used at the same time, identification information1to5may be allotted to respective game card.

Attitude information, orientation information, distance information determined in the state determiner44and identification information acquired by the identification acquirer46are associated with each other and transmitted to the game apparatus30from the transmitting unit50. If a plurality of game cards4exist within the imaging area5, altitude information, orientation information, distance information and identification information on each game card4are associated with each other before being transmitted to the game apparatus30from the transmitting unit50. Since the frame image captured by the imaging apparatus2is displayed on the display7, the frame image itself is also transmitted to the game apparatus30from the transmitting unit50according to the first embodiment.

FIG. 4shows the structure of a game system30. The game system30is provided with an analysis information acquirer100, a game progress processor102, a character determiner104, a character storage106, a change detector110, a display controller120, a motion pattern storage122. The change detector110is provided with a movement quantity monitoring unit112, a rotation detector114and an existence recognizer116. The change detector110detects temporal state change of the real object image captured by the imaging apparatus2.

Processing function of the game apparatus30according to the first embodiment is implemented by a CPU, a memory, a program loaded into the memory, etc.FIG. 4depicts structure implemented by the cooperation of the elements. The program may be built in the game apparatus30or supplied from an external source in the form of a recording medium. Therefore, it will be obvious to those skilled in the art that the functional blocks may be implemented by a variety of manners including hardware only, software only or a combination of both. In the illustrated example, the CPU of the game apparatus30has the functions of the analysis information acquirer100, the game progress processor102, the character determiner104, the change detector110, the display controller120.

The analysis information acquirer100receives an analysis result from the image analysis apparatus20. This analysis result includes attitude information, orientation information, distance information and identification information on the game card4, a real object. The analysis information acquirer100delivers the received analysis result to the game progress processor102. The analysis information acquirer100may receive frame image data from the imaging apparatus2directly. In this case, game apparatus30has the functions of image analysis apparatus20and performs the same process as described above in relation to the image analysis apparatus20.

The game progress processor102controls the whole process of the game application. In the game application according to the first embodiment, the game progress comprises a plurality of stages and a different game scene is set to each game stage. A player clears the terminating condition of each stage stepwise and the game finishes when he clears the final stage. The game progress processor102controls the progress of the game and reports the name of a game stage to be started next and identification information sent from the analysis information acquirer100to the character determiner104at the time of starting the game or changing stages.

The character storage106stores identification information on the game card4and three-dimensional image data of a character associated with each other for each stage. Based on the game stage name and identification information, the character determiner104reads three-dimensional image data of a character associated with identification information from the character storage106and provides the game progress processor102with the data. The read three dimensional image data may be provided to the display controller120directly. The game progress processor102provides the display controller120with three-dimensional image data and attitude information, orientation information and distance information on the game card4. The display controller120displays the character on the display7in association with the displayed position (displayed region or displayed area) of the game card4.

More specifically, the display controller120receives the frame image sent from the image analysis apparatus20and displays it on display7. The display controller120recognizes the attitude, orientation and the distance of the game card4from the attitude information, the orientation information and the distance information on the game card4and determines the attitude, the orientation and the size of the character to be displayed on a display7using three-dimensional image data. For example, the character may be displayed inclined along the normal to the card in case the game card4is inclined against the table3. The display controller120may locate the character at any position as far as it is superimposed on the game card4. The displayed position of the character is set to be above a center of the game card4in ordinary display mode. A character may have an inner parameter that represents, for example, emotion or condition depending on the player's operating history.

The motion pattern storage122stores a motion pattern of a character in ordinary operating state. More specifically, the motion pattern storage122sets a motion pattern associated with a character, a game stage and an inner parameter. Thus, based on the character name, game stage being played and inner parameter of a character, the display controller120chooses a motion pattern from the motion pattern storage122and controls the character on display7.

As the player moves the game card4slowly, the image analysis apparatus20transmits the analysis results of a frame image to the analysis information acquirer100successively. Manipulating a game card4slowly or not moving the card at all will be referred to as an ordinary manipulating state compared to a state change of a game card4described below. The display controller120receives three-dimensional image data of a character and the attitude information, the orientation information and the distance information on the game card4from the game progress processor102. The controller makes a character be superimposed on a displayed position of a game card4and follow the game card4. Thus a character is displayed consistently on a game card4on the display7, which makes a player feel a sense of togetherness of a character and a game card4. As described, the display controller120superimposes a character on a game card4at an ordinary manipulating state of a game card4.

In case a game card4is set at a predetermined state for imaging through player's manipulation, the display controller120does not make a character simply follow the game card4but controls the display mode of the character and makes variations in the motion pattern of the character. Player's action on a game card4works as a trigger to change the motion pattern of a character, which gives a player a pleasure different from the pleasure derived from an ordinary manipulation using, for example, a game controller. The game progress processor102delivers attitude information, orientation information and distance information on the game card4to the change detector110to detect whether a game card4is manipulated in an ordinary manner. The change detector110detects a temporal state change in an image of a game card4in a frame image.

The movement quantity monitoring unit112monitors the quantity of movement of a game card4captured by the imaging apparatus2. More specifically, the movement quantity monitoring unit112determines the velocity of the game card4based on the central coordinate and distance information included in attitude information on the game card4. The movement quantity monitoring unit112stores the central coordinate and distance information on the game card4for each frame in a memory (not shown) and calculates a movement vector using change in distance and a difference in the central coordinate among a predetermined number of frames. Thus, movement velocity is calculated. In case the central coordinate is represented as a three-dimensional coordinate, difference among central coordinate values simply determines a movement vector. The movement quantity monitoring unit112monitors the quantity of movement of a game card4captured by the imaging apparatus2. The movement quantity monitoring unit112may monitor a quantity of movement in determined virtual space or may monitor an actual quantity of movement.

On determining that the movement velocity of a game card4exceeds a predetermined reference speed, the movement quantity monitoring unit112reports the results to the game progress processor102. The movement velocity of a game card4may be the movement velocity of the captured game card4in virtual space or an actual movement speed. On receiving the determination results, the game progress processor102recognizes that the game card4is moved quickly by a player. This event is referred to as a “slide event”. The game progress processor102reports the name of the character and the occurrence of the slide event to the display controller120. On receiving the report on the occurrence of the slide event, the display controller120searches the motion pattern storage122for the motion pattern of the character corresponding to the slide event.

The motion pattern storage122stores not only the motion pattern in an ordinary manipulating state described above, but also the motion pattern of character at an occurrence of an event. The motion pattern storage122defines a motion pattern associated with a name of an event as well as a character, a game stage and an inner parameter. Thus based on the name of the character, the game stage being played now, the inner parameter of the character and the name of the event, the display controller120chooses the motion pattern from the motion patter storage122, displays and controls a character on the display7.

Being informed of the occurrence of the slide event, the display controller120reads the motion pattern which dose not make the character follow the movement of a game card4but make the character fall down on the spot from the motion pattern storage122and performs it. By moving a game card4quickly, a player feels as if the character is not able to follow the movement of the game card4and is left behind. Choosing a motion pattern which embodies the feeling and presenting it on the screen of the display7allow image processing that fits a player's sense.

FIGS. 5A-5Dshow the motion of the character represented on the display when the slide event occurs.FIG. 5Ashows a state wherein the character is superimposed on the game card4. This state corresponds to an ordinary manipulating state wherein the character is displayed and controlled in accordance with the motion pattern based on, for example, an inner parameter.

InFIG. 5B, the game card4is moved left on the screen by the player. A player's finger manipulating the game card4is also displayed on the display7(not shown). In case the movement quantity monitoring unit112determines that the velocity of the game card4exceeds a predetermined reference velocity, the game progress processor102reports an occurrence of a slide event to the display controller120. The display controller120makes the game card4move left on the display7based on a frame image sent periodically from the image analysis apparatus20.

At the time of the detection of the occurrence of the slide event, the display controller120does not make the character follow the movement of the game card4but make the character fall down as shown inFIG. 5C. I.e., when the slide event occurs, the display controller120stops the movement of the character on the spot so that the character is not superimposed on the displayed position of the game card4. This makes the displayed position of the game card4and the displayed position of the character apart momentarily.

As shown inFIG. 5D, the display controller120makes the character get up at a predetermined point of time and move to the displayed position of the game card4. For example, the action may be timed to occur when the quick movement of the game card4is ended or when a predetermined span of time elapsed since the occurrence of the slide event. The display controller120plays back the motion of the character moving back to the central coordinate of the game card4as a target on the display7. A series of movement of the character shown inFIGS. 5A-5Dare determined by a motion pattern chosen by the display controller120.FIG. 5shows the character being back to the displayed position of the game card4. With the game application according to the first embodiment, a player enjoys the series of movement of the three-dimensional character by moving the game card4. In the game system1, the player's attempts to manipulate the game card4in various ways induce new movements of the character on the display, which raises the excitement of playing the game application.

The movement quantity monitoring unit112monitors not only the movement velocity, but also a moving direction of the game card4captured by the imaging apparatus2. The movement quantity monitoring unit112stores the central coordinate of a game card4on each frame in a memory (not shown) and calculates a movement vector from difference in central coordinate of a game card4among frames. Thus a direction of the movement vector can be detected.

The movement quantity monitoring unit112compares a direction of a movement vector with that of another which precedes in time. On detecting a state in which the angle made by movement vectors is substantially 180 degree a plurality of times within a fixed span of time, the movement quantity monitoring unit112reports the detected result to the game progress processor102. Three occurrences of reversal in the direction of the movement vector in two seconds may be set as a condition to report. On receiving the detection results, the game progress processor102′ recognizes that the game card4shuttles to and fro. This event is referred to as a “shuttle event”. The game progress processor102reports the name of a character and the occurrence of the shuttle event to the display controller120. Being informed of the occurrence of the shuttle event, the display controller120searches the motion pattern storage122for the motion pattern of the character corresponding to the shuttle event.

FIG. 6A-6Cshow a motion of the character represented on the display when the shuttle event occurs.FIG. 6Ashows a state wherein a player scolds the character by shuttling the game card4to and fro. When the movement quantity monitoring unit112determines that the game card4is moved to and fro such that a predetermined condition is fulfilled, the game progress processor102reports the occurrence of the shuttle event to the display controller120. The display controller120displays the shuttling movement of the game card4and changes motion patterns of the character based on the motion pattern retrieved from the motion pattern storage122. In this example, the shuttling movement of the game card4works as a trigger to perform the motion pattern wherein the character is scolded. And the character shows a shocked expression because of being scolded. During the shuttling movement of the game card4, the character may not follow the shuttling movement of the game card4but the displayed position of the character may be fixed as far as it is within the movement range of the game card4. In case the amplitude of the shuttling movement is large enough to leave the character apart from the game card4, it is favorable to make the character follow the game card4.

FIG. 6Bshows a character having grown huge by shuttling a game card4to and fro. In this way, the motion pattern storage122may store a plurality of motion patterns in correspondence with a shuttling movement of a game card4. The motion pattern storage122may store a motion pattern in relation with a game stage and may further store a motion pattern in relation with an inner parameter of a character as described above.

Referring back toFIG. 4, the rotation detector114detects a rotating movement of a game card4. More specifically, the rotation detector114detects a rotating movement of a game card4based on a center coordinate and orientation information included in attitude information on a game card4. The rotation detector114stores center coordinate and attitude information on a game card4for each frame in a memory (not shown). In case the orientation of a game card4defined by orientation information changes with respect to time on a substantial plane and the center coordinate of a game card4does not shift during the changing of orientation, the rotation detector114determines that the game card4is rotated. A condition to detect rotation may be that the orientation of a game card4is changed more than 360 degree to the same rotational direction.

On determining that a game card4is rotating, the rotation detector114reports the judgment to the game progress processor102. On receiving the determination results, the game progress processor102recognizes that the game card4is being rotated. This event is referred to as a “rotation event”. The game progress processor102reports the name of a character and the occurrence of the rotation event to the display controller120. On receiving information on the occurrence of the rotation event, the display controller120searches for a motion pattern corresponding to the rotation event defined for the character.

The display controller120chooses a motion pattern defined for a character in the game stage being played. Using the chosen motion pattern, the display controller120changes motion patterns of a character. More specifically, the display controller120reads and performs a motion pattern in which a character feeling dizzy.

FIG. 7shows a state in which a character feels faint by a rotary motion of a game card. The state in which a character feels faint returns to an ordinary state after a lapse of predefined time. It is easy to grasp through intuition that the rotary motion of the game card4makes a character feel dizzy. It is favorable that the motion pattern of a character corresponding to the manipulation of a card link to the manipulation of the card itself since a game controller is not used in the first embodiment. In this way, associating a manipulation of a card and a motion pattern of a character with each other enables a player to manipulate easily. Determining a three-dimensional character's motion pattern by a manipulation of a card makes it possible to realize a new game application, which gives a player a new experience and sensation.

Referring back to aFIG. 4, the existence recognizer116checks whether a game card4exists within the imaging field5. Existence of a game card4within the imaging field5is determined by whether information on a game card4is analyzed in the image analysis apparatus20. In case a game card4is hidden by a player the image analysis apparatus20is not able to recognize an image of a game card4, thus image analysis results of a game card4is not sent to the game apparatus30.

In case a real object is not recognized in a series of predefined number of frame images, the existence recognizer116determines that a real object is not captured by the imaging apparatus2. Conversely, in case a number of consecutive frame images in which a real object is not recognized is less than predefined number, the existence recognizer116determines the real object is captured by the imaging apparatus2. A real object not being recognized in a predefined numbers of consecutive frame images is set as the condition since it is necessary to neglect a frame in which a game card4is not detected by chance, for example, by an influence of a lighting.

On determining that a game card4is not captured, the existence recognizer116reports the determination results to the game progress processor102. On receiving the determination results, the game progress processor102recognizes that a game card4does not exist in an imaging field5. This event is referred to as a “hiding event”. The game progress processor102reports a name of a character and an occurrence of the hiding event to the display controller120. A player can generate a hiding event by, for example, hiding a game card4by his hand or moving a game card4out of the imaging field5. On receiving information on an occurrence of the hiding event, the display controller120searches the motion pattern storage122for the motion pattern corresponding to the hiding event set for the character.

In this case, the display controller120makes the character disappear from a screen of the display7using the chosen motion pattern. This motion pattern is also easy to understand for a player. Thus, a player is able to manipulate a character with intuition without understanding how to play the game sufficiently.

The existence recognizer116may determine that a state change between a state wherein a game card4is captured and a state wherein a game card4is not captured is repeated. A player can disable imaging of a game card4by holding his hand over the game card4. And moving out his hand enables imaging a game card4. If a switching between a captured and not captured state is repeated predefined times in predefined time span, the existence recognizer116detects the change in image capturing state and reports the detected results to the game progressing processor102. On receiving the determination results, the game progress processor102recognizes that switching between a state where a game card4is captured and a state where a game card4is not captured by the imaging apparatus2occurs. This event is referred to as a “switching event”. The game progress processor102reports the name of a character and an occurrence of the switching event to the display controller120. On receiving information on the occurrence of the switching event, the display controller120searches the motion pattern storage122for a motion pattern of the character corresponding to the switching event.

In this case, the display controller120displays a new virtual object on display7using the chosen motion pattern. This new object is not displayed in an ordinary manipulating state. An occurrence of the switching event works as a trigger to display the entirety of the virtual object newly. This is an appearance of, so to say, a hidden character in the game industry. An appearance of a new character makes it possible to bring a change to game progression.

With the game system1, a player does not have to remember a motion pattern allotted to a manipulation of a card necessarily. A player may manipulate a game card4in variety of ways and try to move a character. The game application according to the first embodiment provides a player a new way to enjoy a game.

FIG. 8shows a flowchart for an image processing according to the first embodiment. In the game apparatus30, the analysis information acquirer100acquires identification information on a game card4from the image analysis apparatus20(S10). On receiving identification information, the character determiner104reads three-dimensional image data of the character corresponding to identification information and the stage being played from the character storage106. The display controller120superimposes the read three-dimensional image data of the character on the displayed position of the game card4on the display7.

The change detector110monitors a state change of a game card4with respect to time (S16). On detecting a predefined state change (Y in S16), the display controller120reads a motion pattern corresponding to the state change from the motion pattern storage122(S18), displays and controls a character according to the motion pattern (S20). If the stage continues (N in S22), the display controller120returns character's display mode to the ordinary state and superimposes the character on the game card4. If a predefined state change is not detected (N in S16) and there is not a switching between stages, a superposing display mode is maintained. In case stages are changed (Y in S22), the present flow ends. When a subsequent stage begins, three-dimensional image data of a character corresponding to the stage is read out and the flow described above is performed.

The first embodiment is explained above. This embodiment is only illustrative in nature and it will be obvious to those skilled in the art that variations in constituting elements and processes are possible and that those variations are within the scope of the present invention. While an example in which a motion pattern of a character is changed is explained according to the first embodiment, it is also possible, for example, to present additional virtual object other than the main character and the new virtual object moved in the opposite direction so that it goes apart from the main character when displayed.

As an example of variations, the display controller120may display another virtual object together with a character and detection of a state change of the game card4by the change detector110may work as a trigger to move the virtual object in the direction determined by the orientation determiner54in the image analysis apparatus20, so that the virtual object moves apart from the character. Another virtual object may be an item used for game progress (e.g., a virtual object like a ball thrown by a character).

FIG. 9Ashows an exemplary displaying in which a character throws a ball and plays bowls. As described above, the orientation determiner54determines the orientation of a game card4, based on the position of the orientation indicator11on the game card4in real space. In case a displayed position of virtual bowling pins on the display7is fixed, a player moves the game card4and adjusts position and direction for the character to throws the ball, while watching the display7. Bowling pins may be virtual objects displayed on another game card4. The character determiner104reads three-dimensional image data of the ball from the character storage106and provides it to the game progress processor102on condition bowling pins are displayed on the other game card4. The display controller120receives three-dimensional image data of the ball from the game progress processor102and controls the character to hold the ball when displayed. When the character is displayed at a desired position as the player moves the card, the player manipulates the game card4and generates an event that is set as a trigger to throw the ball. It is favorable that this event be announced to the player on the screen or through a speaker. On receiving information on occurrence of the event, the display controller120rolls the ball in the direction determined by the orientation determiner45and calculates the number of bowling pins which fall down based on the direction by a predetermined computation. In this case, the display controller120unifies coordinates of bowling pins and the character into the same coordinate system and determines whether the ball, as moving object, and bowling pins make contact, which makes this displaying process possible. Playing bowls is given as one example above. Launching a virtual object from a character allows developing a new game story using an object other than a character.

While the orientation determiner54may determine the direction using the direction indicator11printed on the game card4, in case the game card4is inclined, it may adopt a vector along a slope as the direction of the game card4.

FIG. 9Bshows another exemplary display in which a character throws a ball and plays bowls. The orientation determiner54determines a direction in which the game card4is inclined in real space. This direction of inclination is defined as a direction on the table3perpendicular to the side of the game card4that makes contact with the table3. Differing from an example ofFIG. 9A, in this case, direction to throw the ball is determined based on a line where the game card4and the table3make contact with each other. The player places the game card4at a desired position and makes it inclined. In this process, setting the game card4inclined may be set as a trigger to throw a ball. Detecting from attitude information that the game card4is inclined, the game progress processor102reports it to the display controller120. The display controller120reads out a motion pattern and rolls the ball in the direction determined by the orientation determiner54.

In the first embodiment and an example of variation described above, display mode of the character is controlled based on a state change of the game card4. To make the game application more interesting and exciting, not only a display mode of the character but also, for example, voice may be used for presentation effect. In this case, when a state change of the game card4is detected, by the change detector110, the game progress processor102may reports to a voice controller (not shown), and the voice controller may direct an auditory presentation effect of the character through the speaker. In this case the game apparatus30functions not only as an image processing apparatus but also as a voice processing apparatus. Thus the game apparatus30may be referred to as a processor which is able to control both image and voice. The game apparatus30may control only voice depending on a state change of the game card4.

Second Embodiment

The second embodiment of the present invention provides a technique for detecting a positional relation among a plurality of real object images captured by the imaging apparatus and controlling a display mode of a virtual object based on the detected relation. A real object may be a one-dimensional object, a two-dimensional object or a three-dimensional object. It is favorable that a real object be provided with a distinctive part that identifies the real object. For example, a real object may be a two-dimensional object, such as a card that is provided with two-dimensionally represented coded information as a distinctive part, or a three-dimensional object with a uniquely shaped three-dimensional part as a distinctive part. A two-dimensional shape of a two-dimensional object may constitute a unique distinctive part or distinctive coded information may be affixed on a three-dimensional object. A virtual object may be, so to say, a character, such as a person, an animal or material goods that is represented three-dimensionally in virtual space. The second embodiment described below relates to an image processing technology in a game application and adopts a game character presented in a game application as a virtual object. The second embodiment depicts a game application in which the player's manipulation for making real objects contact with each other leads to an occurrence of an event corresponding to the contact and performing an event one by one makes the game progress.

FIG. 10shows the structure of a game system201according to the second embodiment. The game system201is provided with an imaging apparatus2, an image processing apparatus210and an output apparatus6. The image processing apparatus210is provided with an image analysis apparatus220and a game apparatus230. The image analysis apparatus220and the game apparatus230may be separate apparatuses or may be integrally combined. The imaging apparatus2is embodied by a video camera comprising a charge coupled device (CCD) imaging element, a metal oxide semiconductor (MOS) imaging element or the like. The imaging apparatus2captures an image of real space periodically so as to generate a frame image in each period. An imaging area5represents a range captured by the imaging apparatus2. The position and size of the imaging area5are adjusted by adjusting the height and orientation of the imaging apparatus2. A game player manipulates a game card4, a real object, in the imaging area5. The game card4is provided with a distinctive part that uniquely identifies the card.

The output apparatus6is provided with a display7. The output apparatus6may also be provided with a speaker (not shown). The image processing apparatus210causes the display7to display a frame image captured by the imaging apparatus2. In this process, the image processing apparatus210controls a character, a virtual object, to be superimposed on the game card4when displayed. In the illustrated example ofFIG. 10, two game cards exist in the imaging area5and characters are superimposed on each game card4on display7. The player can easily recognize whether the game card4is located within the imaging area5by watching the display7. If the game card4is not located within the imaging area5, the player may allow the imaging apparatus2to capture the image of the game card4by shifting the position of the game card4or by adjusting the orientation of the imaging apparatus2.

In the game application according to the second embodiment, the player moves a character by manipulating the game card4. As called for by the nature of the game application, it is favorable that the player feel the sense of unity between the game card4and the character. For this purpose, the image of the character is superimposed on the game card4.

The character's motion is controlled by the image processing apparatus210. First, the image analysis apparatus220extracts image information for the game card4from the frame image captured by the imaging apparatus2. The image analysis apparatus220further extracts the unique distinctive part to identify the game card4from image information on the game card4. In this process, the image analysis apparatus220determines position information, orientation information and distance information on the game card4in space, by referring to image information on the game card4. As described above inFIG. 2, an orientation indicator11and an identification indicator12are printed on a surface of a game card4.

As described in regard toFIG. 2, The orientation indicator11is provided to indicate the front side of the game card4and the identification indicator12is provided to represent a distinctive field to identify the card uniquely. An identification indicator12is coded information made by a plurality of blocks printed on a predetermined field. Of the plurality of blocks, four corner blocks are given to a plurality of game cards4commonly. Thus actually, a distinctive part is comprised of blocks other than the four corner blocks. The four corner blocks are used to measure a distance from an imaging apparatus2.

Referring back toFIG. 10, the image processing apparatus220determines the distance between the imaging apparatus2and the game card4from the length between the four corner blocks in the image of the game card4identified in the frame image. The image processing apparatus220further determines the orientation of the game card4by using the orientation indicator11. In this case, the orientation indicator defines the front side and the character is controlled so that the character faces forward when displayed on the game card4. The image analysis apparatus220also acquires identification information on the game card4by referring to an array of blocks other than the four corner blocks.

The result of image analysis by the image analysis apparatus220is sent to the game apparatus230. The frame image captured by the imaging apparatus2may be sent to the game apparatus230and the game apparatus30may analyze the image. In this case, the image processing apparatus210may be formed only of the game apparatus230.

The game apparatus230controls the character to be displayed on the game card4on the screen of the display7based on the result of image analysis by the image analysis apparatus220. A character may be assigned to each game scene for a game card4as appropriate. In this case, when game scenes are switched, displayed characters are also changed. In the second embodiment, the game apparatus230detects a positional relation among a plurality of real object images. On judging that the positional relation among a plurality of real object images fulfills a predefined condition, the game apparatus230controls the display mode of the character.

FIG. 11shows the structure of the image analysis apparatus220. The image analysis apparatus220is provided with a frame image acquirer240, a real object extractor242, a state determiner244, identification information acquirer246, identification information storage248and a transmitting unit250. The identification information storage248stores information on the distinctive field for identifying the real object and identification information for identifying the real object with each other. To be more specific, the identification information storage48stores pattern information on the identification indicator12and identification information in a one-to-one relationship. Identification information is used to allot a character in the game apparatus230. For example, in case of the game system201which allows using five game cards4simultaneously, number1to5may be allotted to respective game card4as identification information. Relating respective game card4to identification information allows recognizing each game card4. The state determiner244determines the state of the real object in the defined coordinate system. More specifically, the state determiner244is provided with an attitude determiner252which determines the attitude of a card, an orientation determiner254which determines the orientation of a card and a distance determiner256which determines the focal distance from the imaging apparatus2.

The frame image acquirer240acquires a frame image of real space captured by the imaging apparatus2. It is given that a plurality of game cards are placed in the imaging area5here, as shown inFIG. 10. The imaging apparatus2captures a frame image at regular intervals. Preferably, the imaging apparatus2generates frame images at intervals of 1/60 second.

The real object extractor242extracts a plurality of real object images, i.e., a plurality of game card4images, from the frame image. This process is performed by translating image information into a binary bit representation and extracting the image of the game card4from the binary bit representation (i.e. dot processing). Extracting an image may be performed by detecting ons and offs of a bit. This process may also be performed by a known image matching technology. In this case, the real object extractor242registers image information on a real object to be used in a memory (not shown) in advance. Matching registered image information and captured image information allows cutting out images of multiple game cards4from the frame image.

The state determiner244detects geometry information of the table3on which the game card4is placed and moved beforehand. The state determiner244further defines the surface of the table3as a reference plane and records geometry information on the attitude of the game card4placed on the reference plane as the initial attitude of the game card4. This geometry information may be formed with reference to the imaging apparatus2. The state determiner44maintains the position, the attitude and the like of the table3as coordinate data in imaging space from the geometry information acquired of table3. The position determiner252determines the position of the real object image. More specifically, the position determiner252determines coordinates of the center point of the real object image in the frame image. In addition to the position of the game card4, the attitude determiner252may determine the inclination and height of the game card4with respect to the reference plane as a difference in relative quantity of state from the attitude of the game card4recorded as the initial state (i.e., difference of coordinate values in imaging space). The orientation determiner254determines the orientation of the real object image. The orientation determiner254may detect the orientation indicator11shown inFIG. 2from the real object image and determine the orientation of the real object. The distance determiner256determines the distance between the imaging apparatus2and the game card4from the length among the four corners of identification indicator12in the game card4image. The identification information acquirer246extracts a distinctive feature from the real object image and acquires corresponding identification information from identification information storage248.

Position information, orientation information and distance information determined by the state determiner244and identification information acquired by the identification acquirer246are associated with each other and transmitted to the game apparatus30from the transmitting unit250. Associating is performed for each game card4. To display the frame image captured by the imaging apparatus2on the display7, the frame image itself is also transmitted to the game apparatus230from the transmitting unit250.

FIG. 12shows the structure of the game system230. The game system230is provided with the analysis information acquirer300, a game progress processor302, a character determiner304, a character storage306, a positional relation detector310, a condition evaluator312, a display controller320and a display pattern storage322.

Processing functions of the game apparatus230according to the second embodiment are implemented by a CPU, a memory, a program loaded into the memory, etc.FIG. 12depicts function blocks implemented by the cooperation of the elements. The program may be built in the game apparatus230or supplied from an external source in the form of a recording medium. Therefore, it will be obvious to those skilled in the art that the functional blocks may be implemented by a variety of manners including hardware only, software only or a combination of both. In the illustrated example, the CPU of the game apparatus230has the functions of the analysis information acquirer300, the game progress processor302, the character determiner304, the positional relation detector310, the condition evaluator312, a display controller320.

The analysis information acquirer300receives an analysis result from the image analysis apparatus320. This analysis result includes position information, orientation information, distance information and identification information on the game card4, a real object. The analysis information acquirer300delivers the received analysis result to the game progress processor302. The analysis information acquirer300may receive frame image data from the imaging apparatus2directly. In this case, the game apparatus230has the functions of image analysis apparatus220and perform the same process as described above in relation to the image analysis apparatus220.

The game progress processor302controls the whole process of the game application. In the game application according to the second embodiment, the game progress comprises a plurality of stages and a different game scene is set to each game stage. A player clears the terminating condition of each stage stepwise and the game finishes when he clears the final stage. The game progress processor302controls the progress of the game and reports the name of a game stage to be started next and identification information sent from analysis information acquirer300to the character determiner304at the time of starting the game or changing stages. The character storage306stores identification information of the game card4and the three-dimensional data of a character associated with each other for each stage.

FIG. 13shows the contents stored in the character storage306. The game system201according to the second embodiment allows five game cards4to be used. The character storage306stores three-dimensional data of a character corresponding to each of five game cards4in relation with a game stage. At stage1, a character “a man”, a character “a woman”, a character “a drum”, a character “a restaurant building” and a character “a post office building” are allotted to the game card of identification information1, the game card of identification information2, the game card of identification information3, the game card of identification information4and the game card of identification information5, respectively. At stage2, a character “a man”, a character “a woman”, a character “a door of a restaurant”, a character “a waiter” and a character “a table and chairs” are allotted to the game card of identification information1, the game card of identification information2, the game card of identification information3, the game card of identification information4and the game card of identification information5, respectively. Characters allotted to identification information3,4and5are different between stage1and stage2.

Based on the name of a game stage and a plurality of pieces of identification information, the character determiner304reads three-dimensional image data of a plurality of characters associated with identification information and provides the game progress processor302with data. The read three dimensional image data may be provided to the display controller320directly. The game progress processor302provides the display controller320with three-dimensional image data and position information, orientation information and distance information on the game cards4. The display controller320displays a character on the display7in association with the displayed position of the game card.

More specifically, the display controller320receives the frame image sent from the image analysis apparatus220and displays it on the display7. The display controller320recognizes the position, orientation and the distance of the game card4from position information, orientation information and distance information on the game card4and determines the position, the orientation and the size of the character to be displayed on the display7using three-dimensional image data. The display controller320may locate the character at any position as far as it is superimposed on the game card4. However, in ordinary display mode, the displayed position of the character is set to be above a center of the game card4.

As a player moves a game card4, the analysis result of a frame image is sent from the image analysis apparatus220to the analysis information acquirer300successively. The display controller320receives three-dimensional image data of a character and position information, orientation information and distance information on the game card4from the game progress processor302and makes a character follow the game card4so that the image of the character is superimposed on a displayed position of the game card4. Thus a character is displayed consistently on the game card4on the display7, which makes a player feel a sense of unity between the game card4and the character.

FIG. 14shows an example of displaying on the display in the stage1. In this case, a game card4a, a game card4b, a game card4c, a game card4dand a game card4eindicate the game card of identification information1, the game card of identification information2, the game card of identification information3, the game card of identification information4and the game card of identification information5, respectively.

A character “man”, a character “a drum”, a character “a restaurant building” and a character “a post office building” are superimposed on the game card4a, the game card4c, the game card4dand the game card4erespectively. On the game card4csticks for striking a drum are also displayed with the drum. In this process, A woman is not yet displayed on the game card4B.

In the game application according to the second embodiment, an event is generated when an arranged position of a plurality of game cards4conforms to a predefined positional relation. By performing the event, game story progresses. The Player's manipulation on a plurality of game cards4allows changing, for example, a motion pattern of a character, which gives a player a pleasure different from that derived from ordinary manipulation using a game controller or the like. In illustrated example ofFIG. 14, the character “woman” appears on the game card4bby performing a predefined event in stage1.

A positional relation detector310detects a positional relation among images of a plurality of game card4images included in a frame image captured by the imaging apparatus2. More specifically, the game progress processor302delivers positional information, orientation information and distance information on a plurality of game cards4to the positional relation detector310in the first place. The positional relation detector310detects position relation among game cards based on position information and distance information on a plurality of game cards4. In this case, it is favorable to detect positional relations among game cards for all the combinations of no less than two game cards4. For example, the positional relation detector310may compute distance between central coordinates based on the central coordinate of each game card4.

Based on the positional relation detected by the positional relation detector310, the condition evaluator312determines whether the positional relation among game cards4fulfills a predefined condition. In this case, whether a detected positional relation among no less than two game card4images fulfills the predefined condition is determined. As an example of condition determination, the condition evaluator312determines whether images of game card4are in contact with each other. The condition evaluator312may determine a contact between game card images simply if a distance between central coordinates of two game card images are within a predefined range.

In determining a contact between game card images, the condition evaluator312takes orientations of game card images into consideration. Arrangement of game card images in space is determined, based on central coordinates and orientation of game card images. This enables the condition evaluator312to learn the arrangement of game card images in space and determine whether they have contact with each other. In this process, the condition evaluator312can also learn on which side of game card images they have contact with each other by taking the orientation into consideration. Since the orientation of game card images is determined by the orientation determiner254, the condition evaluator312can determine which side of a rectangular game card has contact, front side or left side, based on determined orientation information. Since the condition evaluator312recognizes the orientation of game card images when judging contact, it is possible to generate a different event, depending on orientation of game cards that have contact. On determining that game card images have contact with each other, the condition evaluator312reports determination results, identification information on contacting game card images and orientation of game card image to the game progress processor302. The game progress processor302transfers the information to the display controller320. The processing of the positional relation detector310and the condition evaluator312described above may be performed simultaneously.

Furthermore, the condition evaluator312may define a virtual viewing angle to, for example, one game card image, and may determine whether another game card image exists in the viewing angle. This determination on condition is performed based on the positional relation detected by the positional relation detector310, and is used for confirmation on whether another character exists within the viewing angle of the character in the game. In determining that another game card image exists within the viewing angle, the condition evaluator312reports determination results, identification information on the game card image on which a viewing angle is defined and identification of a game card image which exists within the viewing angle to the game progress processor302. This information is transferred to the display controller320.

In case the condition evaluator312determines that the predefined condition is fulfilled, the display controller320determines the display pattern of the character based on identification information on no less than two game card images which fulfill the condition. The display controller320receives determination results sent from the game progress processor302, identification information on game card images which fulfill the condition and orientation of game card images, refers to the display pattern storage322and determines the display pattern.

The display pattern storage322stores the motion pattern of a virtual object. The motion pattern may be, for example, a motion pattern among characters corresponding to identification information on no less than two game card images which fulfill condition. This motion pattern is stored in relation with identification information on no less than two game card images and orientation of those game card images. More specifically, when for example, the game card4awith identification information1and the game card4cwith identification information3come into contact with each other on their front sides, the display controller320is able to read a predefined motion pattern from the display pattern storage322. I.e., that the image of the game card4band the image of the game card4care in contact and that they have contact on their front sides is defined as a condition for reading the display pattern from the display pattern storage322. Thus, in case a game card4aand a game card4care in contact on the front side and the left side respectively, the display controller320is not able to read a motion pattern.

The display controller320performs display process of a character on the frame image sent from the image analysis apparatus220, using three-dimensional image data of a plurality of characters according to the determined motion pattern.

FIG. 15shows the state which follows the state shown inFIG. 14and in which two cards are placed so that they are in contact with each other. In this case, a game card4band a game card4care in contact with each other. The game card4aand the game card4care in contact on their front sides. Since the man faces forward on the game card4aand the drum is set facing the forward on a game card4c, the man is able to strike the drum. Conversely, the man is not able to strike the drum when he stands left or right side of the drum. Identification information on game cards4which are in contact and orientation at the time of contacting are defined as a condition for reading a motion pattern from the display pattern storage322, as described above. Thus by contacting front sides of game cards, the display controller320reads out a predefined motion pattern and performs display process in which the man takes sticks which are put on the game card4cand strikes the drum. This series of motion is determined by a display pattern which is read out.

In this game application the motion of striking the drum is set as the condition to present the character, woman. On recognizing that the drum is struck by the man, the game progress processor302presents the woman on the game card4b. Subsequently, the player moves the characters man and woman in front of the restaurant.

FIG. 16shows the state which follows the state ofFIG. 15and in which two cards are placed so that they are in contact with another card. The left side of the game card4dcontacts the front side of the game card4aand the front side of the game card4b. Identification information and orientation of the contacting game card are set as, a condition to read a display pattern as described above. In stage1, moving characters the man and the woman in front of the restaurant, as shown inFIG. 16, is set as the condition to end stage1and proceed to stage2. Being informed of the positional relation among the game card4a,4b, and4cas shown inFIG. 16from the positional relation detector310, the condition evaluator312determines that the front side of the game card4aand4bare in contact with the left side of the game card4dand reports it to the game progress processor302. On receiving the report, the game progress processor302recognizes that a closing condition for stage1is fulfilled and performs a switching process of game stages. The condition evaluator312may determine the fulfillment of the closing condition. The game progress processor302reports a subsequent stage name (stage2), to the character determiner304.

On receiving the name of the stage, the character determiner304reads three-dimensional image data of identification information allotted for stage2, referring to a corresponding relation shown inFIG. 13. In this way, by changing corresponding relations depending on stages to which the character determiner304refers, a new character is able to be presented for each stage, that makes a game story beefed up. In stage2, a character “restaurant door”, a character “waiter”, a character, “a table and chairs” are allotted to the game card4of identification information3, the game card4of identification information4and the game card4of identification information5, respectively as shown inFIG. 13. The display mode of characters shown inFIG. 16are replaced.

In the game system201according to the second embodiment, a variety of image processing technologies are realized other than the one described above by using a positional relation of game card images.

FIG. 17illustrate a process in which one character is allotted to a plurality of game card images.FIG. 17Ashows a state in which a building is allotted for each three game card. Although, three buildings allotted to card4a,4band4care identical, a character allotted to a card is not a subject of interest in this process.FIG. 17Bshows a state in which three game cards are in contact with each other. In this case, one big building is allotted to three game cards.

The display pattern storage322stores a display pattern in which one virtual object is allotted to no less than two game card images which fulfill a condition. In this example, the display pattern storage322stores identification information on three game card images (identification information1,2and3) and the display pattern associated with orientation of the contacting part of the game card images. In this process, the condition to read out the display pattern is that orientation of contacting parts is left or right, that is, each game card image is in contact with other game card image on the left side or the right side.

When the condition evaluator312determines that game card4a,4band4chave contact on their left or right side, the determination result, identification information on game card images and orientation of images are provided to display controller320via the game progress processor302. Based on the information, the display controller320reads a display pattern from the display pattern storage322. Based on the display pattern, the display controller320performs display processing as shown inFIG. 17B. Through this process, a character can be displayed huge. Thus, a new visual effect can be realized in the game story. Player's manipulation to place game cards in contact with each other leads to an appearance of an unexpected character, which improves amusement of the game.

FIG.18AB explain a process in which orientation of a character is changed.FIG. 18Aindicates a positional relation between the character “man” on the game card4aand the character “woman” on the game card4B. Two dashed lines301extending from the “man” indicate the virtual viewing angle of the man. The condition evaluator312determines whether the image of the game card4bis within the virtual viewing angle of the game card4abased on position information and orientation information on the game card4aand position information on the game card4b. Position information and orientation information are delivered from the positional relation detector310. In determining that the character “woman” is within a virtual viewing angle, the determination result and identification information on game card4aand4care transferred to the game progress processor302. It is assumed that the game card4ais moved along a path shown as an arrow by subsequent player's manipulation.

FIG. 18Bshows a state after the card is moved along the arrow shown inFIG. 18B. Although the character on the game card4ais set to face forward originally, he faces toward the character on the game card4bin this example. The display controller320receives identification information on the game card4aand the4band information indicating that the game card4bis within the viewing angle of the game card4afrom the game progress processor302. The display pattern storage322stores the motion pattern in which the character on the game card4aturns so that he continues to look at the character on game card4b. This display pattern is set to be readable when the condition related to the game cards and the condition related to the viewing angle are established. The display controller320reads out the motion pattern and performs display processing as shown inFIG. 18B. That is, the display controller320changes the orientation of the character on the game card4a, depending on the position of the character on the game card4b. Through this process, the player recognizes that the character on the game card4bmay have an important influence on game progress for a character on the game card4a. A display mode like this gives a player a chance to generate an event by placing the game card4ain contact with the game card4b.

FIG.19AB illustrate a process in which a virtual object expands and contracts when displayed.FIG. 19Ashows a state in which the game card4aand the game card4bare in contact with each other. By making two game cards in contact with each other, a virtual object303appears between characters. This virtual object303is represented as if it expands, by moving game cards apart as shown inFIG. 19B. On the contrary, a virtual object303is represented as if it contracts when game cards are moved close to each other from the state shown inFIG. 19B. This makes it possible to realize a new visual effect. The display pattern storage322stores the display pattern in which the virtual object that is extendable depending on positional relation between characters is presented between the characters by making the front sides of the game card4aand the game card4bcome into contact with each other.

When the positional relation detector310detects that the positional relation between game card images changes from the first state to the second state, the display controller320reads out a display pattern in which the virtual object connecting characters associated with respective identification information is displayed as if it extends or contracts from the display pattern storage322and determines the motion pattern. The display controller320performs display as shown inFIGS. 19A and 19B, using the display pattern.

FIG. 20shows a state in which a task is presented to a player to make the game more amusing. The task is, for example, to move two game cards4on the table3along the arrow301indicated on the display7. Two game cards4should maintain contact with each other during the movement. The positional relation detector310calculates respective movement vectors of the game card4aand4band defines the average of two vectors as the moving direction of the two game cards. The movement vector is computed by storing central coordinates and distance information of the game card images for each frame into a memory (not shown) and calculating distance change and difference in central coordinates of the game card4in consecutive frames. The condition evaluator312determines whether the task is achieved from the movement vector. More specifically, the condition evaluator312determines whether the movement vector is along the arrow305. If the direction of the vector and the arrow205are substantially identical, the task is cleared. Player may receive an optional merit in the game by clearing the task.

FIG. 21shows a flowchart for an image processing according to the second embodiment. In the game apparatus230, the positional relation detector310detects the positional relation among a plurality of game card4images included in a frame image captured by the imaging apparatus2(S110). Based on the positional relation detected by the positional relation detector310, the condition evaluator312determines whether the positional relation among game cards4fulfills a predefined condition (S12). If the predefined condition is fulfilled, (Y in S112) the display controller320determines the display pattern associated with the fulfilled condition and reads the display pattern from the display pattern storage322(S114). The display controller320performs display processing of a character using the determined display pattern (S116).

While the predefined condition is not fulfilled (N in S112) and the stage continues (N in S118), the positional relation detector310continues to detect the positional relation among game cards4. In case the stages are changed (Y in S118), the present flow ends. When the subsequent stage begins, three-dimensional image data of a character corresponding to the stage is read out and the flow described above is performed.

The present invention is explained above according to the second embodiment. The second embodiment is only illustrative in nature and it will be obvious to those skilled in the art that variations in constituting elements and processes are possible and that those variations are within the scope of the present invention.

In the second embodiment described above, a character is controlled when displayed, based on the positional relation among game cards4. To make the game application more interesting and exciting, not only a display mode of the character but also, for example, voice may be used for presentation effect. In this case, if the condition evaluator312determines that positional relation among game cards4fulfills a predefined condition, the game progress processor302may reports to a voice controller (not shown), and the voice controller may direct an auditory presentation effect of the character through a speaker. In this case the game apparatus230functions not only as an image processing apparatus but also as a voice processing apparatus. Thus the game apparatus230may be referred to as a processor which is able to control both image and voice. The game apparatus230may control only voice depending on positional relation among game cards4.

The present invention is explained above according to a plurality of embodiments. These exemplary embodiments is only illustrative in nature and it will be obvious to those skilled in the art that variations in constituting elements and processes are possible and that those variations are within the scope of the present invention. While the first and the second embodiments of the present invention are described above, combination of respective contents of the embodiments enables to control a display mode of a virtual object more effectively.

INDUSTRIAL APPLICABILITY

The present invention is applicable to a field of image processing.