Abstract:
An image processor is provided which utilizes images taken by an imaging device, e.g., a digital camera, as an input interface to enter commands, etc. Such image processor includes a memory operable to store an image from a series of real time images of a location captured by an imaging device over time, the stored image at least partially including a target that is subject to movement from one point in the time to another point in the time. The image processor further includes a detector operable to detect the target and a movement component thereof by detecting features of the captured images at different points in the time, and includes an image generator operable to generate an object image representing a predetermined object so that a color of a predetermined portion of the object image varies according to the detected movement component of the target. Such image generator is further operable to generate a combined image from the object image and the stored image and to output in real time a signal representing the combined image, to permit the combined image to be displayed to the location imaged by the imaging device.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS  
       [0001]    This application is a continuation of U.S. patent application Ser. No. 09/971,962 filed Oct. 5, 2001, the disclosure of which is hereby incorporated by reference herein. That application is based upon and claims the benefit of priority from the prior Japanese Patent Applications Nos. 2000-307574, filed Oct. 6, 2000, and 2001-295098 filed Sep. 26, 2001, the entire contents of both of which are incorporated herein by reference. 
     
    
     
       BACKGROUND OF THE INVENTION  
         [0002]    The present invention relates to an image processing technology for using an image taken by an image pickup apparatus such as a video camera as an interface for inputting commands, etc.  
           [0003]    A keyboard, mouse, controller, etc. are input devices often used for a computer, video game machine, etc. The operator inputs desired commands by operating these input devices to render a computer, etc. to execute processing according to the commands entered. Then, the operator sees images and listens to sound, etc. obtained as the processing results from a display device and speaker.  
           [0004]    The operator enters commands by operating many buttons provided on the input device while watching a cursor shown on the display device.  
           [0005]    Such operations greatly depend on operating experiences of the operator. For example, for a person who never touched the keyboard before, entering desired commands using the keyboard is quite troublesome and time-consuming, and prone to input errors due to mistyping from the keyboard. For this reason, there is a demand for a man-machine interface that will provide the operator with an easy way to operate.  
           [0006]    On the other hand, with the progress of multimedia technologies, people in general households can now readily enjoy capturing images using a video camera into a computer, etc., editing and displaying the images on a display device. Such technologies are also used for personal authentication by analyzing images of a physical body such as a face, extracting characteristic parts thereof to identify individuals.  
           [0007]    Conventionally, these images are used as information to be processed by a computer such as editing or analysis. However, images taken have not been used so far for a purpose such as entering commands to a computer, for example.  
         SUMMARY OF THE INVENTION  
         [0008]    It is an object of the present invention to provide an image processing technology to use images taken by an image pickup apparatus, etc. as an input interface to enter commands, etc.  
           [0009]    According to an aspect of the invention, an image processor is provided which utilizes images taken by an imaging device, e.g., a digital camera, as an input interface to enter commands, etc. Such image processor includes a memory operable to store an image from a series of real time images of a location captured by an imaging device over time, the stored image at least partially including a target that is subject to movement from one point in the time to another point in the time. The image processor further includes a detector operable to detect the target and a movement component thereof by detecting features of the captured images at different points in the time, and includes an image generator operable to generate an object image representing a predetermined object so that a color of a predetermined portion of the object image varies according to the detected movement component of the target. Such image generator is further operable to generate a combined image from the object image and the stored image and to output in real time a signal representing the combined image, to permit the combined image to be displayed to the location imaged by the imaging device.  
           [0010]    The “target” refers to a remarked part of an object (person or matter, etc.) whose image is taken by an image pickup apparatus that supplies the image to an image processor. An image processor is provided which utilizes images taken by an imaging device, e.g., a digital camera, as an input interface to enter commands, etc.  
           [0011]    Another image processor provided according to another aspect of the invention includes a memory operable to store an image from a series of real time images of a location captured by an imaging device over time, the stored image at least partially including a target that is subject to movement from one point in the time to another point in the time. The image processor further includes a detector operable to detect the target and a movement component thereof by detecting features of the captured images at different points in the time, and includes an image generator operable to generate an object image representing a predetermined object so that the object image follows a movement of the detected target and includes an image representing a trace of the movement. Such image generator is further operable to generate a combined image from the object image and the stored image and to output in real time a signal representing the combined image, to permit the combined image to be displayed to the location imaged by the imaging device.  
           [0012]    These image processors generate object images according to the movements of targets included in the mirrored moving image. That is, the movement, color, shape of the object image to be displayed on the display device and if there is a plurality of object images, which object image should be displayed, etc. are determined by the movement of the target. For example, if the target is the operator, the object is determined according to the movement of the operator. Thus, the mirrored moving image is available as a kind of input interface.  
           [0013]    It is also possible to comprise means for making preparations for executing required processing based on the generated object image according to the movement component of the target.  
           [0014]    It is also possible to further comprise means for comparing a combined image obtained by combining the object image generated by the image generating means and the mirrored moving image at the actual time point, with a template image which is the image of part of the target included in the immediately preceding mirrored moving image, detecting the part of the combined image whose image feature is most resembling the template image and making preparations for executing required processing based on this object image when the image of the part of the detected combined image includes the object image.  
           [0015]    By associating the object image with predetermined processing and further comprising means for executing the processing linked to the object image when the movement component of the target detected by the detecting means satisfies predetermined conditions, it is possible to execute processing using the movement of the target as an input.  
           [0016]    It is also possible to construct the image processor so that the mirrored moving image includes a plurality of targets, construct the detecting means to detect the movement components of the plurality of targets and detect one target based on the respective movement components of the detected plurality of targets, construct the image generating means to change the object image according to the movement component of the one target detected by the detecting means.  
           [0017]    According to another aspect of the invention, an imaging processing method is provided having steps including storing an image from a series of real time images of a location captured by an imaging device over time, the stored image at least partially including a target that is subject to movement from one point in the time to another point in the time; detecting the target and a movement component thereof by detecting features of the captured images at different points in the time; generating an object image representing a predetermined object so that a color of a predetermined portion of the object image varies according to the detected movement component of the target; generating a combined image from the object image and the stored image; outputting in real time a signal representing the combined image, to permit the combined image to be displayed to the location imaged by the imaging device.  
           [0018]    According to still other aspects of the invention, a recording medium is provided having instructions recorded thereon for performing a method such as described above, and a semiconductor device is provided which has functions to perform such method. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0019]    These objects and other objects and advantages of the present invention will become more apparent upon reading of the following detailed description and the accompanying drawings in which:  
         [0020]    [0020]FIG. 1 is an overall configuration diagram of an image processing system applying the present invention;  
         [0021]    [0021]FIG. 2 is a configuration diagram of an image processor according to an embodiment of the present invention;  
         [0022]    [0022]FIG. 3 is a functional block diagram of the image processor according to the embodiment of the present invention;  
         [0023]    [0023]FIG. 4 is a flow chart showing a processing procedure of Embodiment 1;  
         [0024]    [0024]FIG. 5 is a flow chart showing a processing procedure of Embodiment 1;  
         [0025]    [0025]FIG. 6 illustrates a combined image according to Embodiment 1;  
         [0026]    [0026]FIG. 7 illustrates a menu image;  
         [0027]    [0027]FIG. 8 is a flow chart showing a processing procedure of Embodiment 2;  
         [0028]    [0028]FIG. 9 illustrates a combined image according to Embodiment 2;  
         [0029]    [0029]FIG. 10 is a view illustrating a drawing using a recursive texture;  
         [0030]    [0030]FIG. 11 is a flow chart showing a processing procedure of Embodiment 3;  
         [0031]    [0031]FIG. 12 is a flow chart showing a processing procedure of Embodiment 3; and  
         [0032]    [0032]FIG. 13 illustrates a combined image according to Embodiment 3. 
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT  
       [0033]    An embodiment of the present invention will be specifically described with reference to the drawings accompanying herewith.  
         [0034]    [0034]FIG. 1 is a configuration example of an image processing system applying the present invention.  
         [0035]    This image processing system takes pictures of the operator who sits in front of a display device  3  using an analog or digital video camera  1 . In this way, the image processing system captures moving images into an image processor  2  consecutively in a time series to generate mirrored moving images. Of these mirrored moving images, the image processing system combines object images expressing objects such as a menu and cursor at positions where remarked objects such as the eyes and hands of the operator (hereinafter the remarked objects are referred to as “targets”) to generate a combined image (this, too, becomes a moving image) and displays this combined image on the display device  3  in real time.  
         [0036]    A mirrored moving image can be generated by subjecting the moving image captured from the video camera  1  to mirroring (right/left inversion of image) by the image processor  2 , but it is also possible to place a mirror in front of the video camera  1  and take pictures of a moving image on the mirror surface reflecting the operator by the video camera  1  to form a mirrored moving image. In any case, a combined image whose display mode changes in real time according to the movement of the target is displayed on the display device  3 .  
         [0037]    The image processor  2  is implemented by a computer that forms the required functions using a computer program.  
         [0038]    The computer according to this embodiment whose hardware configuration is shown by way of example in FIG. 2 has two buses; a main bus B 1  and sub bus B 2  to which a plurality of semiconductor devices each having specific functions is connected. These buses B 1  and B 2  are mutually connected or disconnected via a bus interface INT.  
         [0039]    The main bus B 1  is connected to a main CPU  10  which is a main semiconductor device, a main memory  11  made up of a RAM, a main DMAC (Direct Memory Access Controller)  12 , an MPEG (Moving Picture Experts Group) decoder (MDEC)  13  and a graphic processing unit (hereinafter referred to as “GPU”)  14  incorporating a frame memory  15  which serves as a drawing memory. The GPU  14  is connected with a CRTC (CRT controller)  16  for generating a video signal so as to display the data drawn in the frame memory  15  on the display device  3 .  
         [0040]    The CPU  10  loads a start program from the ROM  23  on the sub bus B 2  at the startup via the bus interface INT, executes the start program and operates an operating system. The CPU  10  also controls the media drive  27 , reads an application program or data from the medium  28  mounted in this media drive  27  and stores this in the main memory  11 . The CPU  10  further applies geometry processing (coordinate value calculation processing) to various data read from the medium  28 , for example, three-dimensional object data (coordinate values of vertices (typical points) of a polygon, etc.) made up of a plurality of basic graphics (polygons) and generates a display list containing geometry-processed polygon definition information (specifications of shape of the polygon used, its drawing position, type, color or texture, etc. of components of the polygon).  
         [0041]    The GPU  14  is a semiconductor device having the functions of storing drawing context (drawing data including polygon components), carrying out rendering processing (drawing processing) by reading necessary drawing context according to the display list notified from the main CPU  10  and drawing polygons in the frame memory  15 . The frame memory  15  can also be used as a texture memory. Thus, a pixel image in the frame memory can be pasted as texture to a polygon to be drawn.  
         [0042]    The main DMAC  12  is a semiconductor device that carries out DMA transfer control over the circuits connected to the main bus B 1  and also carries out DMA transfer control over the circuits connected to the sub bus B 2  according to the condition of the bus interface INT. The MDEC  13  is a semiconductor device that operates in parallel with the CPU  10  and has the function of expanding data compressed in MPEG (Moving Picture Experts Group) or JPEG (Joint Photographic Experts Group) systems, etc.  
         [0043]    The sub bus B 2  is connected to a sub CPU  20  made up of a microprocessor, etc., a sub memory  21  made up of a RAM, a sub DMAC  22 , a ROM  23  that records a control program such as operating system, a sound processing semiconductor device (SPU: Sound Processing Unit)  24  that reads sound data stored in the sound memory  25  and outputs as audio output, a communication control section (ATM)  26  that transmits/receives information to/from an external apparatus via a network (not shown), a media drive  27  for setting a medium  28  such as CD-ROM and DVD-ROM and an input device  31 .  
         [0044]    The sub CPU  20  carries out various operations according to the control program stored in the ROM  23 . The sub DMAC  22  is a semiconductor device that carries out control such as a DMA transfer over the circuits connected to the sub bus B 2  only when the bus interface INT separates the main bus B 1  from sub bus B 2 . The input unit  31  is provided with a connection terminal  32  through which an input signal from an operating device  35  is input, a connection terminal  33  through which an image signal from a video camera  1  is input and a connection terminal  34  through which a sound signal from the video camera  1  is input.  
         [0045]    This Specification will only explain about images and omit explanations of sound for convenience.  
         [0046]    In the computer constructed as shown above, and with reference to FIG. 3, the main CPU  10 , sub CPU  20  and GPU  14  read and execute a predetermined computer program from the recording medium such as the ROM  23  and medium  28 , and thereby form a functional block necessary for operating as the image processor  2 , that is, an image input device  101 , an image inverter  102 , an object data storage device  103 , an object data input device  104 , an object controller  105 , a superimposing image generator  106 , a difference value detector  107  and a display controller  108 .  
         [0047]    In the relationship with the hardware shown in FIG. 1, the image input device  101  is formed by the input device  31  and the sub CPU  20  that controls the operation thereof, the image inverter  102 , the object data input device  104 , the object controller  105  and the difference value detector  107  are formed by the CPU  10  and the superimposing image generator  106  are formed by the GPU  104 , and the display controller  108  is formed by the GPU  14  and CRTC  16  cooperating with each other. The object data storage device  103  is formed in a memory area accessible to the main CPU  10 , for example, the main memory  11 .  
         [0048]    The image input device  101  incorporates images taken by the video camera  1  via the connection terminal  33  of the input device  31 . In the case where the image entered is a digital image, the image input device  101  incorporates the image as is. In the case where the image taken and entered is an analog image, the image input device  101  incorporates the image after converting it from analog to digital.  
         [0049]    The image inverter  102  subjects the image incorporated by the image input device  101  to mirroring, that is, right/left inversion to form a mirrored moving image.  
         [0050]    The object data storage device  103  stores object data to express objects such as a menu (including a submenu), matchstick, and cursor together with identification data thereof.  
         [0051]    The object data input device  104  incorporates necessary object data from the object data storage device  103  and sends the object data to the object controller  105 . The object data to be incorporated is instructed by the object controller  105 .  
         [0052]    The object controller  105  generates an object image based on the object data incorporated from the object data input section  104  according to the instruction content. Especially, the object controller  105  determines the object display condition based on a difference value sent from the difference value detector  107  and generates an object image to realize the display condition. The difference value will be described later.  
         [0053]    The superimposing image generator  106  draws a combined image obtained by superimposing the mirrored moving image output from the image inverter  102  on the object image generated by the object controller  105  in the frame memory  15 .  
         [0054]    By the way, in addition to generating a combined image by superimposing the object image, it is also possible to display the object image on the mirrored moving image using publicly known imposing processing.  
         [0055]    The difference value detector  107  compares the image features of the mirrored moving image of the combined image generated by the superimposing image generator  106  frame by frame and derives the difference value of the image features between the mirrored moving images of the preceding and following frames. Furthermore, the difference value detector  107  generates a difference image between the mirrored moving images of the preceding and following frames as required.  
         [0056]    The difference value in the image features is a value quantitatively expressing a variation per frame of the movement component of the target included in the mirrored moving image. For example, the difference value indicates a distance that the target has moved in the mirrored moving image or an area between the area after the movement and the area before the movement.  
         [0057]    When a plurality of targets is included within one mirrored moving image, a difference value in the image features expresses a variation in the movement of each target, and therefore it is possible to quantitatively calculate the variation in the movement of each target by calculating this difference value.  
         [0058]    The difference image is an image expressing a variation in the movement per frame of each target included in the mirrored moving image at every point in time. For example, when a target moves between two mirrored moving images, the difference image is an image made up of the image of the target before the movement and the image of the target after the movement.  
         [0059]    In order to derive the difference value and difference image, the difference value detector  107  stores a certain mirrored moving image as a “reference image” relative to mirrored moving images of other frames in the main memory  11 . The mirrored moving image to be stored may be a full one-frame worth mirrored moving image or may be a mirrored moving image that is only part of the target because all that is required is to make it possible to derive a difference value in the image features.  
         [0060]    In the following explanations, whenever a distinction should be made between an image of part of a target and an image of the rest of the target, such an image is called “template image”.  
         [0061]    The difference value detected by the difference value detector  107  is sent to the object controller  105  and used to control movements of object images.  
         [0062]    The display controller  108  converts the combined image generated by superimposing image generator  106  to a video signal and outputs the video signal to the display device  3 . The display device  3  displays the combined image (moving image) on a screen using this video signal.  
         [0063]    &lt;Image Processing Method&gt; 
         [0064]    An embodiment of the image processing method carried out using the above-described image processing system will now be explained.  
       Embodiment 1  
       [0065]    On the display device  3 , as shown in FIG. 6, suppose the image processor  2  displays a combined image consisting of the mirrored moving image of the operator taken by the video camera  1  and subjected to mirroring with a menu image as an example of an object image superimposed.  
         [0066]    As a target, it is possible to select various objects such as the eyes, mouth, hands, etc. of the operator. Here, a case will be described where the operator&#39;s hand is the target and instructions are entered to the menu image by detecting the amount of movement of the hand in the area in which the menu image is displayed.  
         [0067]    The menu image has a hierarchic structure as shown in FIG. 7. When the operator selects “menu” at the top layer, a pull-down image highlighting one of “select1”, “select2” or “select3” at the lower layer is displayed and when one item is selected from the pull-down image, the process determining image (for example, “process 21”, “process 22”, “process 23”, “process 24”) of the menu at the lower layer of the selected pull-down image are displayed.  
         [0068]    The process determining image is stored in the object data storage device  103  linked to the program to render the main CPU  10  to execute the determined process (event) and when a certain process determining image is selected, the program linked thereto starts to execute the corresponding process (event).  
         [0069]    [0069]FIG. 4 and FIG. 5 show the procedure for processing by the image processor  2  to enable such an operation.  
         [0070]    First, with reference to FIG. 4, the difference value detector  107  updates the mirrored moving image to that of the next frame and when the combined image generated by the superimposing image generator  106  is thereby updated (step S 101 ), image features of the mirrored moving image included in the preceding and following combined images to be updated are compared and the difference value is calculated (step S 102 ). The difference value calculated here is a value expressing one movement of the operator&#39;s hand in the area in which the menu image is displayed. The difference values calculated are recorded in the main memory  11  and cumulatively added for a certain period of time (step S 103 ). The reason that difference values are cumulatively added is that the operator&#39;s will about the operation instruction is detected by the image processor  2  based on a plurality of movements of the operator&#39;s hand. If the operator&#39;s will about the operation instruction can be checked according to the amount of one time movement of the hand, cumulative addition need not always be performed.  
         [0071]    The difference value detector  107  sends the difference value (cumulative sum) to the object controller  105 .  
         [0072]    The object controller  105  determines the color of the menu image according to the difference value (cumulative sum) received from the difference value detector  107  (step S 104 ). For example, a plurality of colors of the menu image is provided and the color is changed one by one every time a movement of the hand is detected. It is also possible to change the color from transparent to semitransparent, opaque, etc. Or the actual difference value (cumulative sum) is compared with a predetermined threshold (step S 105 ) and if the cumulative sum is smaller than the threshold (step S 105 : N), the routine goes back to step S 101  assuming that it is not sufficient to determine that “menu” of the menu screen has been selected.  
         [0073]    When the cumulative sum exceeds the threshold (step S 105 : Y), the object controller  105  determines that “menu” of the menu screen has been selected, shows a pull-down image and reports it to the difference value detector  107  (step S 106 ).  
         [0074]    Thus, when the cumulative sum of the movement of the operator&#39;s hand detected in the area in which the menu image is displayed exceeds the threshold, the object controller  105  detects that “menu” of the menu image has been selected and shows the pull-down image. The color of the menu image changes according to the cumulative sum of the amount of movement of the hand, and therefore the operator can know a rough amount of additional movement of the hand required to select “menu”.  
         [0075]    Furthermore, since the display device  3  shows a mirrored moving image, the operator can perform the above-described operation in much the same way the operator looks in a mirror, providing a man-machine interface easy-to-operate for the operator.  
         [0076]    Thus, according to FIG. 5, when it is detected that “menu” on the menu screen has been selected, that is, the difference value (cumulative sum) has exceeded the threshold, the difference value detector  107  stores the image of the operator&#39;s hand (target) at that time as a template image (step S 107 ).  
         [0077]    When the frame is updated and the menu image is thereby replaced by the pull-down image in its subordinate layer and a combined image is shown (step S 108 ), a search is started for the location of the image of the operator&#39;s hand in the new combined image. That is, the difference value detector  107  searches for an image that matches the template image from the combined image (step S 109 ).  
         [0078]    More specifically, the difference value detector  107  divides the combined image into areas in the same size as that of the template image and searches for the image most resembling the template image from among the images in the respective areas after the division. The image most resembling the template image in the area is, for example, when the sum total of absolute values (or squares) of differences between pixels of the images compared can be expressed as distances, an image whose distance from the template image is a minimum.  
         [0079]    When a matched image is found (step S 110 : Y), it is determined whether the matched image is a pull-down image or not (step S 111 ). If the matched image is a pull-down image (step S 111 : Y), the area of the pull-down image is detected from “select1”, “select2” or “select3” (step S 112 ). The detected pull-down image becomes the pull-down image indicated and selected by the operator. Information on the selected pull-down image is reported from the difference value detector  107  to the object controller  105 .  
         [0080]    The object controller  105  reads a process-determining image accompanying the selected pull-down image from the object data storage device  103  and generates an object image to which this process-determining image is attached (step S 113 ).  
         [0081]    In this way, the display device  3  shows how the menus are selected one after another by the operator.  
         [0082]    In the example in FIG. 7, the pull-down image of “select2” is selected from the menu image at the top layer and the process determining images (“process 21”, “process 22”, “process 23” and “process 24”) accompanying the pull-down image of “select2” are displayed.  
         [0083]    The template image is replaced by a new one for every frame.  
         [0084]    That is, the difference value detector  107  discards the template image used for the preceding frame and stores the above-described matched image (image of the operator&#39;s hand used to select the pull-down image) as a new template image (step S 114 ). Then, the routine returns to step S 108  to specify any one of the process determining images (“process 21”, “process 22”, “process 23” and “process 24”) as shown above.  
         [0085]    In step S 111 , when the matched image is outside the area of the pull-down image but is any one of the process determining images within the process determining image area (step S 111 : N, S 115 : Y), the process determining image is assumed to have been selected and the content of the process linked thereto is determined, that is, the program is made executable and the process using the menu image is finished (step S 118 ).  
         [0086]    When the matched image is outside the areas of the pull-down image and the process determining image but within the menu image area (step S 111 : N, S 115 : N, S 116 : Y), this means that the operator attempts to select another pull-down image, and therefore the routine discards the template image, stores the matched image as a new template image and returns to step S 108  (step S 117 ).  
         [0087]    In step S 110 , when no matched image to be compared is found (step S 110 : N) or when a matched image is found but is an image outside the area of the menu image, the process by the menu image is finished at that time (step S 111 : N, S 115 : N, S 116 : N).  
         [0088]    By carrying out processing according to the menu image in the above procedure, the operator can easily select the process with a desired content while watching the own mirrored moving image shown on the screen of the display device  3 . Furthermore, the operator can enter instructions while checking the own behavior on the screen at any time, which prevents the operator from averting his/her eyes from the display device  3  as in the case of using an input device such as a keyboard.  
       Embodiment 2  
       [0089]    The image processing system according to this embodiment links an object image to a program that causes the main CPU  10  to execute an event to be subjected to image processing so that processing of the relevant event is executed according to the action of the operator within the mirrored moving image on the object image.  
         [0090]    As an example of an object image to be superimposed on the mirrored moving image, this embodiment shows a case of using an image of a matchstick and an image of a flame expressing that the matchstick ignites and burns.  
         [0091]    As a premise, the image of the matchstick, which is the object image, is linked beforehand to a program to display an ignition animation indicating that the matchstick has ignited on the display device  3 . Then, when the operator in the mirrored moving image behaves as if he/she struck the image of the match within the combined image, the ignition animation is designed to appear in the ignition part of the image of the matchstick. The image of the flame is displayed when the operator strikes the image of the matchstick.  
         [0092]    The image of the flame can be generated using a technique of, for example, recursive texture drawing.  
         [0093]    The “recursive texture drawing” refers to a drawing technique of referencing an image of an object rendered by texture mapping as texture of another image and carrying out texture mapping recursively. “Texture mapping” is a technique of rendering an image of an object to enhance the texture of the image by pasting bitmap data of the texture to the surface of the object and can be implemented by also using the frame memory  15  as a texture memory. When carrying out such recursive texture drawing, gouraud shading is applied to a polygon on which the texture is drawn, that is, the brightness at vertices of the polygon is calculated and the brightness inside the polygon is calculated by interpolating the brightness of each vertex (this technique is called “gouraud shading drawing”).  
         [0094]    To express the flame image, the positions of vertices of a mesh which is the source of the flame image are shifted using random numbers as shown in FIG. 10 and the positions of new vertices are determined. The brightness at the vertices is also determined based on random numbers. The positions of the vertices and brightness at the vertices are determined every time the frame is updated. Every unit of the mesh which is the source of the flame image becomes a polygon.  
         [0095]    On each polygon, the image that becomes the basis of the flame drawn in the frame memory  15  is formed through the above-described recursive texture drawing and the above-described gouraud shading is applied based on the brightness at each vertex of the polygon. This makes it possible to express a rising air current caused by the flame, shimmering, attenuation of the flame in a more realistic way.  
         [0096]    Suppose the image processor  2  shows a combined image with the image of a matchstick superimposed on the mirrored moving image of the operator on the display device  3 . Here, suppose the target is the operator&#39;s hand. By detecting the amount of movement of the hand in the area in which the image of the matchstick is displayed, the program linked to the image of the matchstick is executed and the ignition animation is displayed on the display device  3 .  
         [0097]    [0097]FIG. 8 shows the processing procedure using the image processor  2  to realize such an operation.  
         [0098]    When the mirrored moving image is updated to the image of the next frame and the combined image generated by the superimposing image generator  106  is thereby updated (step S 301 ), the difference value detector  107  compares image features of the mirrored moving images included in the combined images before and after the updating, calculates a difference value of the image in the ignition section of the image of the matchstick and generates a difference image of the ignition section of the image of the matchstick (step S 202 ). The difference value calculated here is a value that quantitatively expresses the movement of the hand in the ignition section of the image of the matchstick. The difference value generated is an image made up of the images of the hand before and after moving the hand in the ignition section of the image of the matchstick.  
         [0099]    The calculated difference value is recorded in the main memory  11  and cumulatively added for a certain period of time (step S 203 )  
         [0100]    The difference value detector  107  sends the cumulative sum, which is the cumulative sum of the difference images and difference values to the object controller  105 .  
         [0101]    The object controller  105  determines the color of the difference image according to the cumulative sum received from the difference value detector  107  and generates a flame image based on this difference image (step S 204 ). The flame image is generated, for example, by dividing the difference image into meshes and using the aforementioned recursive texture based on these meshes. The color of the flame image is determined according to the color of the difference image. The flame image generated is superimposed on the ignition section of the image of the matchstick.  
         [0102]    In this way, the flame image with the color according to the amount of movement of the hand added is displayed in the area showing the movement of the hand in the ignition section of the image of the matchstick.  
         [0103]    Determining the color of the flame image according to the cumulative sum of difference values makes it possible, for example, to express how the color of the flame image displayed in the ignition section of the matchstick gradually changes according to the amount of movement of the hand.  
         [0104]    Then, the object controller  105  compares the value indicating the color of the flame image with a predetermined threshold (step S 205 ). For example, if the color of the flame image is expressed by R, G and B values, the sum of their respective values can be used.  
         [0105]    When the value indicating the color is equal to or greater than the threshold (step S 205 : Y), the object controller  105  determines to execute the program that displays the ignition animation indicating that the match has ignited (step S 206 ).  
         [0106]    That is, whether or not to start the ignition animation is determined according to the color of the flame image. For example, when the color of the flame image changes from red to yellow according to the amount of movement of the hand, the ignition animation starts when the flame images turns yellow. The operator can know a rough amount of additional movement of the hand required to start the ignition animation.  
         [0107]    The superimposing image generator  106  generates a combined image superimposing the ignition animation on the object image including the matchstick image and flame image, on the mirrored moving image obtained from the video camera  1  (step S 207 ). The ignition animation is displayed in the ignition section of the matchstick image.  
         [0108]    When the value indicating the color is smaller than the threshold (step S 205 : N), the object controller  105  sends the object image superimposing the flame image on the matchstick image to the superimposing image generator  106 . The superimposing image generator  106  generates a combined image by superimposing this object image on the mirrored moving image obtained from the video camera  1  (step S 208 ).  
         [0109]    Then, if, for example, an instruction for finishing the processing is received from the operation device  35 , the processing is finished (step S 209 : Y). If no instruction for finishing the processing is received (step S 209 : N), the routine returns to step S 201  and the display controller  108  displays the combined image generated in step S 207  or step S 208  on the display device  3 .  
         [0110]    As shown above, the system executes the process of determining whether or not to execute the program for displaying the ignition animation linked to the matchstick image according to how much the operator moves his/her hand in the ignition section of the matchstick image.  
         [0111]    Since the operator can perform operations for executing various events while watching the mirrored moving image, it is possible to perform input operations for executing processes more easily than conventional operations using input devices such as a keyboard and mouse.  
       Embodiment 3  
       [0112]    Another embodiment will now be explained. As a premise, suppose the image processor  2  shows a combined image with a cursor (pointer) image, which is an example of an object image, superimposed on the mirrored moving image of the operator on the display device  3  as shown in FIG. 13( a ). Also suppose a plurality of targets such as the hand, eyes, mouth of the operator are included in the mirrored moving image.  
         [0113]    Here, a case will be explained whereby focusing on the movement of the operator&#39;s hand from the plurality of these targets, the cursor image is expressed in such a way as to follow this movement of the hand.  
         [0114]    As shown in FIG. 13( a ), the cursor image is an image like a face with an emphasis put on the eyes, which allows the eyes to be oriented toward the target. Furthermore, the cursor image moves following the movement of the target. That is, when the cursor image is distant from the target, the cursor image moves toward the target and when the cursor image catches the target, the cursor image follows the movement of the target.  
         [0115]    [0115]FIG. 11 and FIG. 12 show the processing procedure using the image processor  2  to realize such an operation.  
         [0116]    According to FIG. 11, when the mirrored moving image is updated to the image of the next frame and the combined image generated by the superimposing image generator  106  is thereby updated (step S 301 ), the difference value detector  107  compares image features of the mirrored moving image included in the combined images before and after the updating and calculates the difference value thereof (step S 302 ) The difference value calculated here is a value quantifying the movements of the hands, eyes, mouth, etc. of the operator, which become candidates of the target in the mirrored moving image.  
         [0117]    The difference value detector  107  sends the difference value of each target to the object controller  105 .  
         [0118]    The object controller  105  detects one target based on the difference value of each target sent from the difference value detector  107  (step S 303 ). For example, the object controller  105  detects a target whose difference value reaches a maximum. In this example, suppose the operator&#39;s hand is detected as the target.  
         [0119]    Upon detecting the target, the object controller  105  determines how the cursor image is displayed according to the target.  
         [0120]    First, the object controller  105  determines whether the target in the combined image updated in step S 310  is outside the cursor image or not (step S 304 ). If the target is within the cursor image (step S 304 : N), the object controller  105  determines that the cursor image has caught the target (step S 308 ).  
         [0121]    If the target is outside the cursor image (step S 304 : Y), the object controller  105  determines that the cursor image has not caught the target and carries out processing for determining how the cursor image is displayed. That is, the object controller  105  generates a cursor image so that the eyes in the cursor image are oriented toward the target.  
         [0122]    Furthermore, the object controller  105  determines the speed at which the cursor image moves toward the target according to the distance between the cursor image and target (step S 306 ). This speed is adjusted to increase as the cursor image moves away from the target. This makes it possible to obtain an image in which the cursor moves toward the target faster as the cursor image stays farther from the target.  
         [0123]    The superimposing image generator  106  superimposes such a cursor image on the mirrored moving image of the next frame and thereby generates a combined image as shown in FIG. 13( a ) (step S 307 ). Then, the routine goes back to step S 301  and performs the same operation for the combined image generated.  
         [0124]    The routine carries out the operations of step S 301  to S 307  until the cursor image catches the target, that is, until it is determined in step S 304  that the target is within the cursor image.  
         [0125]    Such operations can provide an image as shown in FIG. 13( a ) in which the eyes in the cursor image are oriented toward the target (hand) and the cursor image chases after the target.  
         [0126]    Then, according to FIG. 12, when the cursor image catches the target, the difference value detector  107  stores the image of the target at that time as a template image (step S 309 ). For example, the difference value detector  107  stores the section of the mirrored moving image that overlaps with the cursor image as the template image.  
         [0127]    Then, the difference value detector  107  acquires the mirrored moving image of the next frame from the image inverter  102  (step S 310 ) The difference value detector  107  searches for the position of an image that matches the stored template image from among the acquired mirrored moving images (step S 311 ).  
         [0128]    More specifically, the difference value detector  107  divides the acquired mirrored moving image into areas of the same size as the template image and searches for an image in the area most resembling the template image from among the images in the respective divided areas. Upon detecting the matched image as a result of the search, the difference value detector  107  reports the position of the detected image to the object controller  105 .  
         [0129]    The object controller  105  determines the position reported from the difference value detector  107  as the position of the cursor image for the next combined image (step S 312 ).  
         [0130]    The superimposing image generator  106  superimposes the cursor image at the position determined in step S 312  by the object controller  105  on the same mirrored moving image as the mirrored moving image acquired in step S 310  by the difference value detector  107  and thereby generates a combined image as shown in FIG. 13( b ) (step S 313 ). Then, the frame is updated and the display controller  108  displays the combined image generated on the display device  3  (step S 314 ).  
         [0131]    Repeating the above-described operations after the target is caught (step S 309  to step S 314 ) obtains an image in which the cursor image follows the target. That is, when the cursor image catches the target (hand) as shown in FIG. 13( b ), the cursor image is displayed thereafter following the target wherever the target moves. Even when the operator extends the hand as shown in FIG. 13( b ) to FIG. 13( c ), the cursor image is displayed at the tip of the extended hand of the operator together with the movement of the hand recognized as the target.  
         [0132]    Use of the cursor image allows the operator to know at a glance which position of the part of the own body is functioning as the cursor when selecting a process from the menu image as shown in Embodiment 1, for example.  
         [0133]    Furthermore, if, for example, the trace of the movement of the cursor image is set to be kept and displayed, it is possible to show the trace of the movement of the target on the display device  3 . This makes it possible to show, for example, pictures and characters, etc. drawn in the space on the display device  3 .  
         [0134]    As is clear from the foregoing explanations, when the operator needs to enter data, etc. the present invention allows the operator to enter or select the data easily using the mirrored moving image while watching the combined image displayed on the display device, and can thereby provide a user-friendly input interface without the need to get accustomed thereto.  
         [0135]    Various embodiments and changes may be made there unto without departing from the broad spirit and scope of the invention. The above-described embodiment intended to illustrate the present invention, not to limit the scope of the present invention. The scope of the present invention is shown by the attached claims rather than the embodiment. Various modifications made within the meaning of an equivalent of the claims of the invention and within the claims are to be regarded to be in the scope of the present invention.