Image processing apparatus, image processing method, and image communication system

In an image processing apparatus, an image pickup unit takes images of an object including the face of a person wearing the glasses by which to observe a stereoscopic image that contains a first parallax image and a second parallax image obtained when the object in a three-dimensional (3D) space is viewed from different viewpoints. A glasses identifying unit identifies the glasses included in the image of the object taken by the image pickup unit. A face detector detects a facial region the face of the person included in the image of the object taken by the image pickup unit, based on the glasses identified by the glasses identifying unit. An augmented-reality special rendering unit adds a virtual feature to the facial region of the face of the person detected by the face detector.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, an image processing method, and a image communication system.

2. Description of the Related Art

With the ongoing sophistication of consumer television of recent years, three-dimensional (3D) television capable of offering stereoscopic vision is gaining in popularity. Although there are a variety of methods for realizing 3D television, some of the methods require a user to wear dedicated eyeglasses for observing the stereoscopic images.

In a scheme where the dedicated glasses are required to observe stereoscopic images, the user naturally must wear the dedicated glasses. The inventor of the present inventions directed his attentions to the fact that the user must wear the dedicated glasses, and has reached a realization that not only the eyeglasses can be used to observe the stereoscopic images but also new field of application for the eyeglasses can be sought.

SUMMARY OF THE INVENTION

The present invention has been made in view of the circumstances, and a purpose thereof is to provide a new field of use of eyeglasses that are used to observe stereoscopic images.

In order to resolve the above-described problems, one embodiment of the present invention provides an image processing apparatus. The image processing apparatus includes: an image pickup unit configured to take an image of an object, which includes a face of a person wearing glasses by which to observe a stereoscopic image that contains a first parallax image and a second parallax image obtained when the object in a three-dimensional (3D) space is viewed from different viewpoints; a glasses identifying unit configured to identify the glasses included in the image of the object taken by the image pickup unit; a face detector configured to detect a facial region of the face of the person included in the image of the object taken by the image pickup unit, based on the glasses identified by the glasses identifying unit; and an augmented-reality special rendering unit configured to add a virtual feature to the facial region detected by the face detector.

Another embodiment of the present invention relates to an image communication system. The system includes at least two of the above-described image processing apparatuses, and the at least two image processing apparatuses are connected in a manner that permits mutual communication via a communication line.

Still another embodiment of the present invention relates to an image processing method executed by a processor. The method includes: capturing an image of an object, which includes a face of a person wearing glasses by which to observe a stereoscopic image that contains a first parallax image and a second parallax image obtained when the object in a three-dimensional (3D) space is viewed from different viewpoints; identifying the glasses from the captured image of the object; detecting a facial region of the face of the person from the captured image of the object, based on the identified glasses; and adding a virtual feature to the detected facial region.

Optional combinations of the aforementioned constituting elements, and implementations of the invention in the form of methods, apparatuses, systems, recording media, computer programs, and so forth may also be effective as additional modes of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

A description will be given of an outline of preferred embodiments. In the preferred embodiments, images including an image of the face of a person who wears the eyeglasses with which to observe stereoscopic images are acquired and then a facial region of the face of the person is detected using the glasses as a landmark. The thus acquired images are subjected to a special rendering of augmented reality, by which a virtual feature is added, around the detected facial region.

FIG. 1is an illustration giving an overview of an image processing system100according to an embodiment. The image processing system100includes a stereo camera200, an image processing apparatus300, a three-dimensional (3D) television400, 3D glasses500used to observe the stereoscopic images displayed by the 3D television400.

The stereo camera200includes a first camera202and a second camera204for taking images of a user, who is an object to be captured, from different points of view. Here, the images of an object as seen from different points of view in a three-dimensional (3D) space are called “parallax images”. Since the left eye and the right eye of a human are about 6 cm situated apart from each other, there occurs a parallax (disparity) between the image seen by the left eye and the image seen by the right eye. And it is considered that the human brain recognizes the depth of objects using the parallax images sensed through the left and right eyes. Accordingly, if parallax images sensed through the left eye and the right eye are projected onto the respective eyes, the brain will recognize the parallax images as an image having depths, or a perspective image. In the following, the images of an object, including left-eye parallax images and right-eye parallax images, as seen from different points of view in the 3D space will be simply referred to as “stereoscopic image(s)”. The stereo camera200may be realized by use of solid-state image pickup devices such as CCD (Charge-Coupled Device) sensors and CMOS (Complementary Metal Oxide) sensors.

The image processing apparatus300processes the images (video images) of an object taken by the stereo camera200. The detail of the image processing apparatus300will be discussed later. The 3D television400displays three-dimensional images generated by the image processing apparatus300. Through the 3D glasses500, the user can recognize the images displayed by the 3D television as stereoscopic images having depths.

There are a variety of 3D television systems for showing perspective images by use of parallax images to human viewers. However, in the present embodiment a description is given of a 3D television, as an example, using a system where left-eye parallax images and right-eye parallax images are displayed alternately in time division, namely in a time sharing manner.

The 3D television400presents the left-eye parallax images and the right-eye parallax images, generated by the image processing apparatus300, alternately in time division. The image processing apparatus300transmits the display timing of parallax images on the 3D television400to the 3D glasses500as a synchronization signal. The 3D glasses500operates the shutter on the left lens or the right lens according to the synchronization signal received. The shutter may be implemented by use of known liquid crystal shutter technology, for instance.

More specifically, when the 3D television400displays a parallax image for the left eye, the 3D glasses500shields the images entering the right eye by closing the shutter for the right-eye lens. Thus, when the 3D television400displays a parallax image for the left eye, the parallax image for the left eye is projected onto the left eye of the user only. On the other hand, when the 3D television400displays a parallax image for the right eye, the 3D glasses500closes the shutter for the left-eye lens with the result that the parallax image for the right eye is projected onto the right eye of the user only.

FIG. 1illustrates a case where the image processing apparatus300and the 3D television400are different from those described above. For example, the image processing apparatus300may be a stationary game device. Also, all or part of functions of the image processing apparatus300may be incorporated into the 3D television as a part thereof.

FIG. 2is a diagram showing a relationship between the shutter timing of the 3D glasses500and the display timing of parallax images in the 3D television400.FIG. 2indicates that at time2tthe right-eye shutter of the 3D glasses500is open for a predetermined time duration (e.g., 10 milliseconds) and the backlight of the display panel of the 3D television400simultaneously lights up for the same time duration. Also, at time4tthe left-eye shutter of the 3D glasses500is open and the backlight of the display panel of the 3D television400simultaneously lights up for the same time duration. At other times than time2tand time t4, both the right-eye shutter and the left-eye shutter of the 3D glasses500are closed and, at the same time, the backlight of the display panel of the 3D television400turns off.

At time2tthe 3D television400displays right-eye parallax images to present the right-eye parallax images to the right eye of the user. And at time4tthe 3D television400displays left-eye parallax images to present the left-eye parallax images to the left eye of the user. This can present perspective 3D images having a sense of depth to the user.

FIG. 3illustrates a functional structure of an image processing apparatus300according to an embodiment. The image processing apparatus300includes a left-eye image generator302, a right-eye image generator304, a glasses identifying unit306, a face detector308, a feature point detector310, a 3D model generator312, an augmented-reality special rendering unit314, a stereoscopic image generator316, and an output unit318.

The left-eye image generator302visualizes the information acquired from the first camera202so as to generate left-eye parallax images. The right-eye image generator304visualizes the information acquired from the second camera204so as to generate right-eye parallax images.

The glasses identifying unit306identifies the 3D glasses500from the images of an object that are captured by the stereo camera200and then visualized by the left-eye image generator302and the right-eye image generator304. As described earlier, implemented in the present embodiment is the shutter glasses where the shutter on the left lens or the right lens is operated according to the synchronization signal received from the image processing apparatus300. Accordingly, the glasses identifying unit306includes a shutter region identifying unit320and a frame identifying unit322.

The 3D glasses500alternately closes the left lens and the right lens in time division, thereby blocking alternately the images projected onto the left eye and the right eye, respectively. This means that, in the captured images of the face of the user wearing the 3D glasses500, each of the user's eyes looking through the left and right lens of the 3D glasses500is alternately blocked and the image of the blocked lens is not captured. Thus, the shutter region identifying unit320identifies the 3D glasses500in a manner such that a region, where the passage of the image of the object is blocked from the images of the object including the image of the face of user wearing the 3D glasses500, is detected as a lens region.

With the lens region identified by the shutter region identifying unit320as a starting point, the frame identifying unit322tracks the glasses frame of the 3D glasses500so as to identify the 3D glasses500. The face detector308detects the face of the user with a glasses region identified by the glasses identifying unit306.

Thus, the user watching the 3D television400of a type where the dedicated glasses are worn is required to wear the 3D glasses500. It is therefore possible to start identifying the glasses region. Where, in particular, the shutter-type 3D glasses are used, it is possible to identify the lens region of the 3D glasses500as a landmark. The lens region is a somewhat large region as compared with the face of a human and therefore the lens region can be detected stably and quickly. For example, as compared with a case where the glasses frame is to be detected, the lens has a two-dimensional extensity and therefore it can be detected stably and quickly.

FIGS. 4A to 4Dshow the images of the face of the user wearing the 3D glasses and the facial regions of the face of the user extracted based on the shutter region.FIG. 4Aillustrates an image obtained when the shutters of the left and right lenses of the 3D glasses500are closed.FIG. 4Billustrated an image when the shutter of the right-eye lens is closed and the shutter of the left-eye lens is open.FIG. 4Cillustrated an image when the shutter of the left-eye lens is closed and the shutter of the right-eye lens is open.FIG. 4Dis a diagram showing a result obtained when the facial region of the user's face is extracted with the lens region of the 3D glasses500as a starting point.

The shutter region identifying unit320calculates a difference between an image obtained when the shutters of the left and right lens of the 3D glasses500are closed as shown inFIG. 4Aand an image when the shutter of the right-eye lens of the 3D glasses500is closed and the shutter of the left-eye lens thereof is open as shown inFIG. 4B, for instance. Since there is a large difference in the left-eye lens region therebetween, the difference value of pixels in this region will be large. Thus, a region where the difference value is large is identified as the left-eye lens region by the shutter region identifying unit320. Also, the shutter region identifying unit320calculates a difference between the image obtained when the shutters of the left and right lens of the 3D glasses500are closed as shown inFIG. 4Aand an image when the shutter of the left-eye lens of the 3D glasses500is closed and the shutter of the right-eye lens thereof is open as shown inFIG. 4C. Thereby, the shutter region identifying unit320can identify the right-eye lens region.

Once the lens region of the 3D glasses is identified, the frame identifying unit322can identify the frame of the 3D glasses500by tracking an edge connected to the lens region thereof. Also, once the lens region of the 3D glasses500is identified, the both eyes of the user can be identified and therefore the approximate size of the face of the user can be estimated based on the distance between the eyes. The face detector308detects a flesh-color region and the edge with the lens region of the 3D glasses as the starting point and thereby can identify a facial region of the user's face.

FIGS. 5A and 5Billustrate an expression area of the person and features points in the expression area.FIG. 5Aillustrates the expression area near the eyes and mouth where the facial expression is more likely to appear.FIG. 5Ashows an expression area334anear the eyes and an expression area334bnear the mouth. Note that the expression area334aand the expression area334bwill be generically referred to as “expression area334” or “expression areas334”. The expression area334is an area where the emotion (e.g., anger, confusion, and laughter) of the person is more likely to appear and will be used to render a special effect of augmented reality (AR) discussed later.

FIG. 5Billustrate feature points in the expression areas334wherein these feature points are collectively denoted by the reference numeral(s)334. The feature points are the ends of eyebrows (denoted by the reference numerals336aand336d), the inner ends of eyebrows (336band336c), the tails of eyes (336eand336g), the inner corners of eyes (336fand336h), the corners of mouth (336iand336k), the center of upper lip (336j), the center of lower lip (336l), chin (336m), and the centers of pupils, for instance. These feature points are used to calculate the depth information on the face of the user based on the rendering of augmented reality (described later), the orientation of the user's face, the analysis of the expression, and the principle of triangulation.

Now refer back toFIG. 3. The feature point detector310detects the feature points as shown inFIG. 5B, based on the facial region detected by the face detector308. This can be achieved by use of a general-purpose technique such as an edge detection method. The 3D model generator312maps the face of the user into a 3D model of a versatile face of the person. This can be accomplished as follows, for example. That is, the feature points detected by the feature point detector310are mapped to the vertices of a polygon using a wire frame model of the face constituted by a plurality of polygons and the like. Then the facial region detected by the face detector308is texture-mapped. Alternatively, a 3D model may be produced by calculating the depth information on the face of the user from the feature points by use of the principle of triangulation. Hereinbelow, the image of the facial region of the face of the user detected by the face detector308will be referred to as “expression image” also, and the feature points of the face of the user detected by the feature point detector310will be referred to as “expression data” also.

The augmented-reality special rendering unit314adds virtual features to the facial region of the face of the person detected by the face detector and its surrounding regions. Here, the “augmented reality” is a collective term for a way of thinking where a 3D model is first projected into a real space, displayed on the 3D television400, which the user wearing the 3D glasses500observes and then various virtual features are added to this real space, and the techniques by which to achieve such a way of thinking.

More to the point, the augmented-reality special rendering unit314adds various augmented realities based on the 3D model of the user's face generated by the 3D model generator312. For that purpose, the augmented-reality special rendering unit314includes a background special rendering unit326, a mirror image generator328, an image pickup position correcting unit330, a face special rendering unit332, and a special rendering control unit324for controlling these components and the operations of thereof.

The background special rendering unit326renders a special effect of augmented reality to a background region. Here, the background region is a region other than the facial region which has been detected by the face detector308and then modeled by the 3D model generator312. As will be discussed later, the image processing system100may be used as a television telephone, for instance, if the image processing system100is connected to other image processing systems100via a network. In such a case, the stereo camera200may generally well be installed within a home of the user. However, there are cases where it is not preferable that what is actually seen in the home is transmitted as it is. To cope with this, the background special rendering unit326replaces the background region with a different image, scumbles the background region or the like. Thus, the present embodiment is advantageous in that undisguisedly transmitting what is actually seen in the home can be prevented.

FIG. 6is a diagram for explaining an operation of the image pickup position correcting unit330according to an embodiment. Since the user normally watches the 3D television400from the front, the stereo camera220cannot be installed in the position where the stereo camera220captures the images of the user from right in front of the user. Instead, the stereo camera220is installed on top of the 3D television (the reference numeral212inFIG. 6) or at the bottom thereof (the reference numeral214inFIG. 6), for instance. A midpoint (the reference numeral216) is between the top and bottom of the 3D television. In such a case, the image of the user captured by the stereo camera200will be either an image looking down at the user or an image looking up at the user.

Since the 3D model generator312generates a 3D model of the face of the user, the image pickup position correcting unit330can produce images obtained when the image of the user is captured from an arbitrary direction. Thus, the image pickup position correcting unit330produces images that would be obtained when the image of the user is captured from the frontal direction, based on the 3D model of the face of the user generated by the 3D model generator312. Thereby, the user can observe the images of his/hers that would be captured from right in front of himself/herself. If the television telephone is to be used, the user can make eye contact with a conversation partner and vice versa. This can reduce a sense of discomfort in making conversation with the conversation partner, as compared with the case where the images taken from the directions other than those taken from right in front of the conversation partner are used in making conversation with each other.

Now refer back toFIG. 3. The face special rendering unit332excludes the 3D glasses500identified by the glasses identifying unit306and generates images of the face where the user is not wearing the 3D glasses500. The face special rendering unit332performs an image processing of applying makeup on the user's face, beautifying the user's skin and the like. The face special rendering unit332renders a special effect of disguise through an image processing where, for example, the user's face is replaced by another person's face or an animal character or the like. In this manner, the face special rendering unit332renders a special effect to the face of a user who is considered an important object. This is advantageous in that an unusual and extraordinary rendering can be presented to the user.

FIG. 7illustrates an exemplary special rendering of disguise by the face special rendering unit332according to an embodiment. In this example, the face special rendering unit332renders a special effect of disguising the user's face to a dog face in a manner such that the feature points detected by the feature point detector310and an image of a dog prepared in advance and its feature points (not shown) are matched with each other.

The mirror image generator328generates a 3D model, where the image of the user is reflected in a mirror, based on the 3D model of the user's face detected by the face detector308and generated by the 3D generator312. The user can observe his/her own face, to which the special effect of augmented reality has been rendered, as the images reflected in the mirror before the transmission of television-telephone signals. Use of the image processing system using the 3D glasses500worn by the user allows the user to check his/her own figure, to which a special effect of augmented reality has been rendered, before entering a cyber-world and thereby allows the user to feel the switch from an ordinary scene to the extraordinary.

FIG. 8illustrates an exemplary mirror image that the mirror image generator328generates based on images excluding the images of 3D glasses500. Although the user is actually wearing the 3D glasses500, the images in the case where no 3D glasses500are worn are presented on a screen of the 2D television400. Also, an actual mirror image of the user is presented on the screen of the 3D television400.

It is required to wear the 3D glasses500in order for the user to observe stereoscopic images. However, the user does not necessarily wish to display on the 3D television400his/her direct images showing that he/she wears the 3D glasses500and to transmit those images to the conversation partner. Rather, there may be cases where the user does not take an active stance toward displaying the images as it is and transmitting them as it is but wishes to render an extraordinary special effect to the images.

In the present embodiment, the 3D model generator312generates the 3D model of the user's face, so that extraordinary special effects can be rendered to the images through various augmented realities. Then the 3D glasses500can be used in the face detection processing as a preprocessing for generating the 3D model. This is because it is guaranteed that the user wears the 3D glasses500.

The special rendering control unit324receives instructions from the user via a user interface such as a not-shown remote controller and then controls the special rendering performed by each component of the augmented-reality special rendering unit314. Though not shown in the Figures, the augmented-reality special rendering unit314may be provided with a function of adding other augmented realities. Here, the other augmented realities include an augmented reality where characters are displayed near the user's face using a “speech balloon” technique, for instance.

The stereoscopic image generator316generates stereoscopic images including the left-eye parallax images and the right-eye parallax images obtained when a 3D model of the user in a virtual 3D space is seen from different points of view, based on the 3D model of the user generated by the 3D model generator312or based on the 3D model of the user to which the augmented-reality special rendering unit314has rendered a special effect. The output unit318outputs the stereoscopic images generated by the stereoscopic image generator316to the 3D television400or transmits the stereoscopic images to other image processing system(s)100via a network such as the Internet.

FIG. 3illustrates a functional structure to realize the image processing apparatus300according to the present embodiment, and other structural components are omitted inFIG. 3. Each element shown inFIG. 3and described as a functional block for performing various processings may be implemented hardwarewise by a CPU, main memory and other LSIs, and softwarewise by image processing programs or the like loaded in the main memory. Therefore, it is understood by those skilled in the art that the functional blocks may be implemented by a variety of manners including hardware only, software only or a combination of both, and are not limited to any particular one.

FIG. 9is a flowchart showing a procedure for processing the augmented reality in the image processing apparatus300according to the present embodiment. In the following flowchart, the procedure of each structural component is shown using S (the capital letter of “Step”), which means a step, and numbers combined. The processing of the flowchart shown inFIG. 9starts when the left-eye image generator302and the right-eye image generator304visualize the outputs of the stereo camera200.

The left-eye image generator302and the right-eye image generator304visualize an object, including the face of the user wearing the 3D glasses500, outputted from the stereo camera200(S10). The glasses identifying unit306identifies the 3D glasses500from the images of the object visualized by the left-eye image generator302and the right-eye image generator304(S12).

Based on the 3D glasses identified by the glasses identifying unit306, the face detector308detects a facial region of the face of the user from the object, including the face of the user, visualized by the left-eye image generator302and the right-eye image generator304(S14). The feature point detector310detects feature points from the facial region of the user's face detected by the face detector308(S16).

The 3D model generator312generates the 3D model of the user's face based on both the facial region of the user's face detected by the face detector308and the feature points detected by the feature point detector310(S18). The augmented-reality special rendering unit314renders a special effect of augmented reality, based on the 3D model of the user's face generated by the 3D model generator312(S20).

The stereoscopic image generator312generates stereoscopic images including the left-eye parallax images and the right-eye parallax images obtained when the 3D model of the user in a virtual 3D space is seen from different points of view, based on the 3D model of the user generated by the 3D model generator312or based on the 3D model of the user to which the augmented-reality special rendering unit314has rendered a special effect (S22). The output unit318outputs the stereoscopic images generated by the stereoscopic image generator316to an external device (S24). As the output unit318has outputted the stereoscopic images, the processing in this flowchart will be terminated.

FIG. 10schematically illustrates a 3D television telephone system700according to an embodiment. The 3D television telephone system700is an image communication system in which at least two image processing systems100are connected in a manner that permits communication with each other via a communication line600. In the example shown inFIG. 10, a first image processing system100a, including a first stereo camera200a, a first image processing apparatus300a, and a first 3D television400a, and a second image processing system100b, including a second stereo camera200b, a second image processing apparatus300b, and a second 3D television400b, are connected in a manner that permits mutual communication via the communication line600such as the Internet.

FIG. 11illustrates a usage example of the 3D television telephone system11according to an embodiment. The first stereo camera200atakes images of an object including the face of a first user800wearing 3D glasses500a. The images taken by the first stereo camera200aare subjected to special effects of various augmented realities such as the exclusion of the glasses and the correction of the image pickup position and then are transmitted to the second 3D television400bvia the communication line600. A second user900wearing 3D glasses500bwatches the second 3D television400b, so that the second user900can watch the stereoscopic images sent from the first image processing apparatus300a.

InFIG. 11, the images taken by the first stereo camera200ahave already been subjected to the special effects of augmented realities, so that the 3D glasses500aare removed in the images of first user800displayed on the second 3D television400beven though the first user800is actually wearing the 3D glasses500a.

Similarly, the images of the second user900wearing the 3D glasses500bare subjected to the special effects of augmented realities and then are sent to the first 3D television400athat the first user800watches. In this manner, by employing the 3D television telephone system700, the users can video chat using the images that have been subjected to the special effects of augmented realities.

FIG. 12shows transmission frequency according to the types of information transmitted in the 3D television telephone system700according to an embodiment. The 3D television telephone system700according to the embodiment converts the stereoscopic images generated at a transmission side into a transmission format of stereoscopic images such as MVC (Multi-view Video Coding) so as to be transmitted.

As described earlier, the 3D glasses serve as the landmark in the present embodiment and therefore the 3D model of the face can be generated stably and with a high degree of accuracy and furthermore the rendering elements such as the background image and the expression images can be separated from each other. Thus, in the present embodiment, the information, such as the position and orientation of the user's face, expression images, and expression data, which requires the real-timeliness is gathered together and combined in units of frame and is transmitted in real time. On the other hand, the 3D model of the face, the special effects using the augmented reality, and the like are transmitted beforehand prior to the communication by the 3D television telephone and therefore these are not transmitted in units of frame.

As shown inFIG. 12, the information requiring a higher level of the real-timeliness is transmitted more frequently than the information requiring a lower level thereof is transmitted. As a result, the 3D television telephone can be achieved at a high quality and a high bit rate. Also, since the background image is separated from the other images, the background image may be replaced by another image, the frame rate when it is to be sent can be reduced, and the compression rate thereof can be raised.

An operation implementing the above-described structure is as follows. The user wears the 3D glasses and uses the image processing system100. The stereo camera200captures the images of an object including the user wearing the 3D glasses. The facial region of the user's face is detected using the 3D glasses500as the landmark, and various special effects of augmented realities are rendered. The images to which a special effect of augmented reality has been rendered are displayed on the 3D television400and are transmitted to another image processing system100.

As described above, the embodiments provide a new usage field where the 3D glasses500are not only used to watch the stereoscopic images but also used as the landmark in rendering special effects of augmented realities.

The present invention has been described based upon illustrative embodiments. The above-described embodiments are intended to be illustrative only and it will be obvious to those skilled in the art that various modifications to the combination of constituting elements and processes could be developed and that such modifications are also within the scope of the present invention.

The description has been given of a case where the shutter glasses are employed but this should not be limited to the 3D glasses500and, for example, polarized glasses may be employed instead. In such a case, a lenticular marker, a light-emitting diode or the like is added to the glasses frame, so that the polarized glasses may be used as the landmark. In particular, the lenticular marker is characterized by the feature that the design or pattern thereof varies when viewed from different angles. Thus, it is advantageous in that the orientation and the angle of the face can be measured by converting a relative angle between the glasses and the camera into a change in the pattern. Also, in order to facilitate the observation of the expression areas334, an under-rim glasses frame that covers the lower half of the lens may be employed.

The description has been given of a case where used are the stereo cameras200including the first camera202and the second camera204that capture the images of the user from different viewpoints. However, the image pickup devices are not limited to the stereo cameras but may be a monocular camera instead. In this case, the feature points detected by the feature point detector310are directly mapped to a 3D model of a general-use face. As compared with the case where the stereo cameras are used, the accuracy of mapping may drop but if the fact that the accuracy is not so much important factor in the augmented reality where the 3D model is used is taken into consideration, this modification will be advantageous in terms of the suppressed cost because only the single unit of camera is used.