Composite image output apparatus and composite image delivery apparatus

A composite image output/delivery apparatus for (10) adding to a picked-up scene image a corresponding animation image and outputting/delivering a resulting composite image without adding a fixed animation image simply to the picked-up scene image. When an object image is extracted from the picked-up scene image received via an image input unit (17), any specified type of character image corresponding to the scene image is selected from an appropriate one of a character 1 “memory (12)e, a character 3 “cat” pose image memory (12f). The selected type of character image replaces the object image in the picked-up scene image or adds to the scene image to thereby provide and display a resulting composite image. That is, by replacing the object image with an animation image similar in pose to the object image or by adding the animation image to the scene image, a very delightful picked-up image with an animation is easily created without adding a fixed animation image simply to an image pickup framerabbit” pose image memory (12d), a character 2 “bear” pose image.

This application is a U.S. National Phase Application under 35 USC 371 of International Application PCT/JP03/06696 filed May 28, 2003.

TECHNICAL FIELD

The present invention relates to a composite image output apparatus for combining a picked-up scene image with another and outputting a resulting composite image, and also relates to a composite image delivery apparatus for delivering a composite image based upon a picked-up scene image.

BACKGROUND ART

Some types of conventional still cameras or video cameras have the function of combining an image picked-up thereby with other images, and displaying a resulting composite image.

For instance, such a camera now in practical use outputs a composite image formed by combining a picked-up scene image with an animation image of a flower or a popular character prepared at a fixed or desired position in a pickup frame.

An image picked up by a digital camera can be displayed on a display screen of a personal computer, combined with a different image and edited. In this case, the different image, for example, can be selectively cut out from another picture file and pasted at a desired position in a picked-up image.

However, the problem with such a conventional image processing apparatus is that a prepared image is simply combined at a fixed or desired position with a picked-up scene image and then outputted, and therefore the composite picture excluding the picked-up image tends to be always flat and tiresome.

DISCLOSURE OF INVENTION

The present invention has been made in view of the above-mentioned problem. It is an object of the present invention to provide an image output apparatus, an image output processing program, and an image output method which makes it possible to not only combine a fixed animation image with a picked-up scene image to thereby output a resulting composite image, but also combine an animation image corresponding to the picked-up scene image with the picked-up scene image.

It is another object of the present invention to provide an image delivery server, an image delivery program, and an image delivery method which makes it possible to not only deliver a fixed image, but also deliver an image.

In order to achieve the above objects, according to one aspect of the present invention, there is provided a composite image output apparatus comprising:

a picked-up scene image acquiring unit (means) which acquires a picked-up scene image representing a scene containing at least one object;

a target image extracting unit (means) which extracts a target image representing any particular one of the at least one object from the picked-up scene image acquired by the acquiring unit;

a corresponding-image obtaining unit (means) which obtains an image corresponding to the target image extracted by the extracting unit; and

an image outputting unit (means) which adds the image obtained by the obtaining unit to the picked-up scene image acquired by the picked-up scene image acquiring unit and outputs a resulting composite image.

In this composite image output apparatus, when a target image representing any particular one of the at least one object is extracted by the target image extracting means from the picked-up scene image acquired by the acquiring means, an image corresponding to the extracted target image is obtained by the corresponding-image obtaining means. Thereupon, the image obtained by the obtaining means is added to the picked-up scene image acquired by the picked-up scene image acquiring means to thereby output a resulting composite image. That is, the user can enjoy creating a delightful composite image.

According to another aspect of the present invention, there is also provided a composite image output apparatus comprising:

a picked-up scene image acquiring unit (means) which acquires a plurality of picked-up scene images, each representing a scene containing at least one object;

a target image extracting unit (means) which extracts a target image representing any particular one of the at least one object from any selected one of the plurality of picked-up scene images acquired by said picked-up scene image acquiring unit;

a corresponding-image obtaining unit (means) which obtains an image corresponding to the target image extracted by said extracting unit; and

an image outputting unit (means) which replaces in the picked-up scene image the extracted target image with the image obtained by the corresponding-image obtaining unit, for adding the corresponding-image to a selected one of the others of the plurality of picked-up scene images from which no target images are extracted by the target image extracting unit, and for outputting a resulting composite image.

In this composite image output apparatus, when a target image representing any particular one of the at least one object is extracted by the target image extracting means from any one of the plurality of picked-up scene images acquired by the picked-up scene image acquiring means, an image corresponding to the extracted target image is obtained by the corresponding-image obtaining means. Thereupon, by the image outputting means the target image extracted by the target image extracting means is replaced with the image obtained by the corresponding-image obtaining means. Also, by the image outputting means, the corresponding-image is then added to a selected one of the others of the plurality of picked-up scene images from which no target images are extracted by the target image extracting means to output a resulting composite image. That is, the user can enjoy creating a delightful image.

According to still another aspect of the present invention, there is also provided a composite image delivery apparatus comprising:

a picked-up scene image receiver (receiving means) which receives from a terminal a picked-up scene image representing a scene containing an object;

a target image extractor (extracting means) which extracts a target image representing the object from the picked-up scene image received by the picked-up scene image receiving unit;

a corresponding-image obtainer (obtaining means) which obtains an image corresponding to the target image extracted by the target image extractor unit;

an image adder (adding means) which adds the corresponding-image obtained by the corresponding-image obtainer to the picked-up scene image to thereby provide a resulting composite image; and

an image deliverer (deliver means) which delivers the composite image provided by the image adder to the terminal.

In the composite image delivery apparatus, when a picked-up scene image representing a scene containing at least one object is received from a terminal, a target image representing the object is extracted from the received picked-up scene image by the target image extracting means. An image corresponding to the extracted target image is obtained by the corresponding-image obtaining means. Thereupon, the obtained image is added by the image adding means to the picked-up scene image to thereby provide a resulting composite image, which is then delivered by the image delivering means to the terminal. That is, the user can enjoy creating a delightful image.

According to a further aspect of the present invention, there is also provided a composite image delivery apparatus that communicates with a terminal, the apparatus extracting a target image contained in a picked-up scene image, sending the terminal pose information on the extracted target image, receiving an image corresponding to the pose information that the terminal sent in response to the sending of the pose information by the composite image delivery apparatus, adding the corresponding image to the picked-up scene image to thereby output a resulting composite image, the apparatus comprising:

a pose information receiver (receiving means) which receives from the terminal the pose information for the target image contained in the picked-up scene image;

a corresponding-image storing memory (storing means) having stored a plurality of images, each corresponding to a respective one of different items of pose information; and

an image deliverer (deliver means) which reads from the corresponding-image storing memory an image corresponding to the pose information received by the pose information receiver, and delivers the read image to the terminal.

In this composite image delivery apparatus, when the terminal extracts the target image contained in the picked-up scene image and sends pose information for the target image to the composite image delivery apparatus, the composite image delivery apparatus receives the pose information in the pose information receiving means thereof. Thereupon, an image corresponding to the received pose information is read by the image delivering means from the corresponding-image storing means, and delivered to the terminal. That is, the terminal can receive from the image delivery apparatus the image corresponding to the target image extracted from the picked-up scene image and add the image to the picked-up scene image to thereby output a resulting interesting composite image. Thus, the user can enjoy creating a delightful image.

BEST MODE FOR CARRYING OUT THE INVENTION

First Embodiment

FIG. 1is a block diagram showing a structure of an electronic circuit of an image output apparatus10according to the first embodiment of the present invention.

This image output apparatus10includes a computer, which reads a program recorded on any of various kinds of recording media or a program received externally and controls its own operation in accordance with the program. The electronic circuit of the computer includes a CPU (central processing unit)11.

The CPU11controls the operation of the components of the circuit in accordance with an apparatus control program previously stored in memory12, an apparatus control program read into the memory12through a recording medium reader14from an external recording medium13such as a CD-ROM, or an apparatus control program read into the memory12through an electric transmission controller15from a Web server21(in this case, a program server) on the Internet N. The apparatus control program stored in the memory12is started up by an input signal issued in response to a user's operation of an input unit16having keys or a touch panel, or by an image signal input through an image input unit17, or by a communication signal used to communicate with each Web server21on the Internet N, which is connected through the electric transmission controller15to the image output apparatus10.

Besides the memory12, the recording medium reader14, the electric transmission controller15, the input unit16and the image input unit17, the CPU11is connected to a distance sensor18, a display unit19including an LED, and an image output unit20.

The image input unit17selectively obtains picked-up scene image data from a digital camera17A, a digital video movie camera17B, or a VTR (video player)17C.

The distance sensor18detects a distance between the distance sensor18and an object in a scene whose object image is input to the image input unit17.

The image output unit20outputs image data processed by this apparatus based upon the picked-up scene image received through the image input unit17. The output image data is selectively output to a memory card20A, a VTR (video recorder)20B or an image printer20C to store, record or print the image.

The memory12has stored a system program in charge of control of the whole operation of the image output apparatus10and a communication program to exchange data with each web server21on the Internet N through the electric transmission controller15. The memory12also has stored an image processing program12ato perform a combining/outputting process for an image corresponding to a target image in the picked-up scene image by the image output apparatus10through the image input unit17and the image output unit20.

Again, the memory12has prepared a model pose image memory12b, a size-adjusted model pose image memory12c, a character1“rabbit” pose image memory12d, a character2“bear” pose image memory12eand a character3“cat” pose image memory12f. Furthermore, it has prepared a picked-up scene image memory12gand an image processing work area12hin the work area thereof.

FIG. 2shows “rabbit”, “bear” and “cat” character images striking various poses (part1) stored in the character1“rabbit” pose image memory12d, the character2“bear” pose image memory12e, and the character3“cat” pose image memory12f, respectively, of the image output apparatus10.

FIG. 3shows further “rabbit”, “bear” and “cat” character images striking various poses (part2) stored in the character1“rabbit” pose image memory12d, the character2“bear” pose image memory12e, and the character3“cat” pose image memory12f, respectively, of the image output apparatus10.

FIG. 4shows still further “rabbit”, “bear” and “cat” character images striking various poses (part3) stored in the character1“rabbit” pose image memory12d, the character2“bear” pose image memory12e, and the character3“cat” pose image memory12f, respectively, of the image output apparatus10.

Each of the different types of character images striking various poses stored in the character “rabbit”, “bear” and “cat” pose image memories12d,12eand12fis used as one imitating a pose of a respective one of the objects (target) images in a picked-up scene image input from the image input unit17. Different pose numbers are stored with different poses in corresponding relationship.

FIG. 5shows images of a model corresponding to various model poses (part1) stored in the model pose memory12bin the image output apparatus10.

FIG. 6shows further images of the model corresponding to various model poses (part2) stored in the model pose memory12bin the image output apparatus10.

FIG. 7shows still further images of the model corresponding to various model poses (part3) stored in the model pose memory12bin the image output apparatus10.

Each of the model images striking various poses stored in the model pose image memory12bis used to obtain a model image similar in pose to an object (target) image in a picked-up scene image input from the image input unit17. Different poses are stored with different pose numbers in corresponding relationship and model images of three variations A, B and C are stored for each pose number.

The respective “rabbit”, “bear” and “cat” character images stored in the character1“rabbit” pose image memory12d, the character2“bear” pose image memory12e, and the character3“cat” pose image memory12fcorresponds to the model images striking the poses indicated by all poses numbers (Nos.1A,1B,1C,2A,2B,2C, . . . nA, nB, nC) stored in the model pose image memory12b.

FIG. 8shows size-adjusted pose images stored in a size-adjusted model pose image memory12cof the image output apparatus, in whichFIG. 8(A)shows, as an example, model pose images (with pose Nos.1A to5A) produced to a standard body-shaped object (target) image shown at the left end ofFIG. 8(A)and stored in the memory12c.FIG. 8(B)shows, as an example, size-adjusted model pose images (with pose Nos.1A to5A) obtained by adjusting the size of the respective standard body-shaped model pose images ofFIG. 8(A)lengthwise and widthwise to a nonstandard body-shaped object (target) image and stored in the memory12c.

A size-adjusted model image to be stored in the size-adjusted-model pose image memory12cis obtained as follows. A standard body-shaped posing model image of a full-length shown by a pose No.2A inFIG. 8(A)is adjusted in size (lengthwise and widthwise) to a nonstandard body-shaped object image shown inFIG. 8(B)to become a size-adjusted image corresponding to the pose No.2A. In accordance with a size-adjust ratio (balance) of the size-adjusted model image for the pose No.2A, all other standard body-shaped model images for the other poses are size adjusted to non-standard body-shaped ones, and then stored in the size-adjusted model pose image memory12c.

The picked-up scene image memory12greceives picked-up scene image data from the image input unit17and stores the picked-up scene image data in units of a picked-up scene image.

The image processing work area12htemporarily stores image data to be processed in accordance with the image process program12awhen the image data is processed.

Operation of the image output apparatus10, structured as described above, will now be outlined.

FIG. 9illustrates addition of a “bear” character image as a follower when a follower add mode is specified by the image output apparatus10.

When an object (target) image H is extracted, as shown inFIG. 9(B), from a picked-up scene image12g1ofFIG. 9(A)and a “bear” character image is specified as a follower, a “walking” character image “bear” of pose No.4(side view) similar in pose to the object image H as shown inFIG. 9(C)is read from the character2“bear” pose image memory12e. Thereupon, a position in a picked-up scene image12g1where the object image H was present a certain period of time ago is recognized from the positions of presence of the object image H on the series of picked-up scene images12g1. The “walking bear” character image (side view) of the pose No.4read from the character2“bear” pose image memory12eis added in the picked-up scene image to the position where the object image H existed the certain period of time ago, as shown inFIG. 9(D).

That is, the “walking bear” character image similar in pose to the object image H in the picked-up scene image12g1is added to the position in the picked-up image where the “bear” character image follows the object image H.

FIG. 10illustrates addition of a “cat” character image in a space area by the image output apparatus10when a space-area character image add mode is specified.

If an object (target) image H1of imitation is specified and extracted from a picked-up scene image12g2ofFIG. 10(A), as shown inFIG. 10(B)and a “cat” character image is specified as being added in a space area, a “cat” character image of a pose No.10(“banzai”) similar in pose to the object (target) image H1is read from the character3“cat” pose image memory12fas shown inFIG. 10(C). Then, this read “cat” character image of the pose No.10(“banzai”) is added to any random position in the space area in the picked-up scene image12g2where none of the objects (target) images H1and H2exists, as shown inFIG. 10(D).

That is, the “cat” character image imitating the pose of the specified object image in the picked-up scene image12g2is added to any random position in the space area of the picked-up scene image12g2.

FIG. 11illustrates addition of a mirror-reversed “rabbit” character image when a mirror-reversed image add mode is specified by the image output apparatus10.

If an object (target) image H is extracted, as shown inFIG. 11(B), from a picked-up scene image12g3including the object (target) image H conscious of dancing with a partner (not shown), for example, as shown inFIG. 11(A), and a “rabbit” character image is specified as being added as a dance partner character image, a “rabbit” character image of a pose No.19(“dance”) similar in pose to the object (target) image H is read from the character1“rabbit” pose image memory12d, as shown inFIG. 11(C). Thereupon, the read “rabbit” character image is mirror reversed and added as a dance partner to a position in the picked-up scene image12g3where the mirror-reversed “rabbit” image links with the object (target) image H for dancing purposes, as shown inFIG. 11(D).

That is, the “rabbit” character image imitating the dancing pose of the object image H in the picked-up scene image12g3is mirror reversed and added as a dance partner to a position in the picked-up scene image where the mirror-reversed “rabbit” links with object image H.

FIG. 12illustrates replacement of an object image by a “rabbit” character image when a multiple image pickup/replace mode is specified by the image output apparatus10.

For example, when the user wishes to produce an scene image in which a “rabbit” character as a replacement image gives a kiss to the user, an object (target) image H1that is desired to strike a kissing pose is extracted, as shown inFIG. 12(B), from a first picked-up scene image12g4ofFIG. 12(A)containing the object image H1striking the kissing pose. When a “rabbit” character image is specified as a replacing character image, a “rabbit” character image with pose No.20(“kiss”) similar in pose to the object (target) image H1, as shown inFIG. 12(C). is read from the character1“rabbit” pose image memory12d.

Then, as shown inFIG. 12(D), a “rabbit” character image with pose No.20(“kiss”) similar in pose to the first object image H1read from the character1“rabbit” pose image memory12dis added to a second picked-up scene image12g4′ containing an object image H2different in pose from the object (target) image H in the first picked-up image, as shown inFIG. 12(E).

That is, the “rabbit” character image imitating the object image H1in the first picked-up scene image12g4is combined with the object image H2in the second picked-up scene image12g4′.

FIG. 13illustrates replacement of an object image by a character “rabbit” image when an object select/replace mode is selected by the image output apparatus10.

If an object image H1is specified and extracted as one to be replaced, as shown inFIG. 13(B), from a picked-up scene image12g5containing three object images H1to H3, for example, as shown as inFIG. 13(A), and a “rabbit” image is specified as a replacing object image, a “rabbit” character image with pose No.1(Say cheese (close-up)) similar in pose to the object image H1is read from the character1“rabbit” pose image memory12d, as shown inFIG. 13(C).

On the other hand, the specified object image H1and other object images H2, H3are cut out from the picked-up scene image12g5, and a background image12g5′ is produced as illustrated byFIGS. 13(B),13(F),13(E) and13(D), respectively.

Thereupon, as shown inFIG. 13(G), the “rabbit” character image similar in pose to the specified object image H1read from the character1“rabbit” pose image memory12d, and the object images H2, H3cut out of the picked-up scene image12g5are combined with the background image12g5′ at respective proper positions in decreasing order of the image pickup distances to the respective objects whose images were picked up.

That is, the “rabbit” character image imitating the pose of the specified object image H1in the picked-up scene image12g5replaces the specified object image H1to thereby provide a resulting composite image.

Then, the image process which will be performed by the image output apparatus10will be described next in more detail.

FIG. 14is a flowchart showing an image process (part1) to be performed by the image output apparatus10.

FIG. 15is a flowchart showing an image process (part2) to be performed by the image output apparatus10, continued from the image process (part1) ofFIG. 14.

The type of a character (“rabbit”, “bear” or “cat”) to be used as an added character or as a replacement in a picked-up scene image12gis specified by manipulation of the input unit16(step S1).

An add/compose mode or a replace/compose mode is selected and specified from among a “1-1: random space-area character image add mode”, a “1-2: follower add mode”, a “1-3: mirror-reversed character image add mode”, and a “2-1: object image select/replace mode”, and a “2-2: multiple image pickup/replace mode. (step S2).

When a scene image12gnpicked up by a digital camera17A, a digital video movie camera17B or a VTR (player)17C begins to be input to the image output apparatus12gn(step S3) and a start operation is executed to add the specified character image to the scene image12gn(step S4), the process proceeds to extracting the object image inFIG. 16(step SA).

FIGS. 16(A) and 16(B)are flowcharts of target image extracting processes (methods1and2), respectively, included in the image process in the image output apparatus10.

In the target image extracting process (method1) ofFIG. 16(A), a series of picked-up scene images12gn, input sequentially at predetermined timings through the image input unit17to the image output apparatus10, are stored in the picked-up scene image memory12gin the work area (step A11).

Thereupon, object images Hn striking different poses corresponding to the moves of each object are extracted as respective blocks from each of the series of picked-up scene images12gnstored in the picked-up scene image memory12g(step A12), and are sorted in the image processing work area12h(step A13).

In the target image extracting process (method2) inFIG. 16(B), a distance between the distance sensor18and each of the objects in a scene whose image is picked up is measured by the distance sensor18(step A21). The image Hn of each object determined as being present in a short-distance depending upon the distance between the background and the object is extracted as a separate block (step A22).

Each extracted object image Hn is then stored as a separate block in the image processing work area12h(step A23).

When each of the object images Hn included in the picked-up scene image12gnis extracted as a block from the picked-up scene image12gnand stored in the image processing work area12h(step SA), the picked-up scene image12gnwith each extracted object image Hn in a broken-lined enclosure M is displayed on the display19, for example, as shown inFIG. 9(A),11(A),12(A) or13(A) (step S5).

Then, it is determined whether or not there are a plurality of blocks of object image Hn in the picked-up scene image12gndisplayed on the display19(step S6).

If it is determined that there are not a plurality of blocks of object image Hn in the picked-up scene image12gn, the only one object image Hn extracted from the picked-up scene image12gnis set as an object image of imitation (step S6→S7a).

If it is determined that there are a plurality of blocks of object image Hn in the picked-up scene image12gn, a target image of imitation is specified from among the plurality of blocks of object image by the user's manipulation of the input unit16(step S6→S7b).

When the object image Hn is specified as the target image of imitation, the process proceeds to a corresponding-animation image acquiring process inFIG. 17(step SB).

FIGS. 17(A) and 17(B)are different flowcharts showing corresponding-animation image acquiring processes (methods1and2), respectively, included in the image process to be performed in the image output apparatus10.

In the corresponding-animation image acquiring process (method1) ofFIG. 17(A), a pose similarity of each of the posing images of the specified character to the object image Hn extracted from the picked-up scene image12gnand set as the target image of imitation is evaluated (step B1C), and a posing character image bearing the highest pose similarity is shown on the display19(step B12).

Here, when a posing character image is selected which bears the highest pose similarity to the target image of imitation displayed on the display19by the user's manipulation of the input unit16, the specified character image (with its pose No.) is registered as an animation image corresponding to the target image of imitation. As long as there are posing character images with different pose similarities, a posing character bearing a next highest pose similarity is displayed each time “next display” is clicked (step B13→B14).

On the other hand, if (or each time) the “next display” is clicked by the user's manipulation of the input unit16in a state where the character image bearing the highest pose similarity to the target image of imitation is displayed on the display19, a character images bearing a second highest pose similarity will appear (step B13→B15).

When a character image bearing a desired pose similarity is displayed and selected by the user's manipulation of the input unit16, the selected character image (specified by its pose No.) is registered as an animation image corresponding to the target image of imitation (step B13→B14).

That is, in the corresponding-animation image acquiring process (method1), a similarity in pose of the respective posing images of the specified character to the object image Hn as the target image of imitation, which has been extracted from the picked-up scene image12gnand set as the target image, is directly evaluated. On the basis of this evaluation, a specified character image is determined, which is used as an added character image or as a replacement striking a pose imitating that of the object image Hn.

In the corresponding animation image acquiring process (method2) inFIG. 17(B), a similarity in pose of each of the posing images of a specified model stored in the model pose image memory12bto the object image Hn extracted from the picked-up scene image12gnand set as the target image of imitation is evaluated (step B2C). A specified character image corresponding to the pose No. of a model image bearing the highest pose similarity is shown on the display19(step B22).

When a character image is selected which corresponds to a posing model image bearing the highest pose similarity to the target image of imitation displayed on the display19, by the user's manipulation of the input unit16, the character image (specified by its pose No.) is registered as an animation image corresponding to the target image of imitation (step B23→B24).

On the other hand, if (or each time) “next display” is clicked by the user's manipulation of the input unit16in a state where a character image is displayed on the display19, which image corresponding to the pose No. of the model image bearing the highest pose similarity to the target image of imitation, a specified character image is displayed corresponding to the pose No. of the model image bearing a second highest pose. As long as there are still posing character images with different pose similarities, a posing character bearing a next highest pose similarity is displayed each time “next display” is clicked (step B23→B25).

Then, when a character image is selected and displayed which corresponds to a model pose image bearing a desired pose similarity and then selected by the user's manipulation of the input unit16, the specified character image (specified by its pose No.) is registered as an animation image corresponding to the target image of imitation (step B23→B24).

As just described above, in the corresponding-animation image acquiring process (method2), a similarity in pose of each of the different posing images of a basic model of a human figure to the object image Hn as the target image of imitation extracted from the picked-up scene image12gnand set as such is evaluated. Then, on the basis of this evaluation, a specified character image is determined, which imitates the pose of the object image Hn, and is used as an added character or as a replacement.

In this case, compared with the corresponding-animation image acquiring process (method1), in which a similarity in pose of each of the various posing images of a specified character to the object image Hn as the target image of imitation is directly evaluated to thereby determine a character image to be used as an added character or as a replacement, the corresponding-animation image acquiring process (method2), in which a similarity in pose of the different posing images of the specified model to the object image Hn extracted from a picked-up scene image12gnand set as a target image of imitation is evaluated, and a character image for use as an added character or as a replacement is determined from a model pose No. selected based upon this evaluation is advantageous in that the use of a human-figure model makes it easy to obtain a model image bearing a higher pose similarity to the object image and obtain a character image bearing a higher pose similarity (imitation).

FIG. 18is a flowchart showing evaluation of a similarity in pose of a character/model image to a target image of imitation in the corresponding-animation image acquiring processes (methods1and2) ofFIG. 17included in the image process to be performed in the image output apparatus10.

When pose similarity evaluation is executed in the corresponding-animation image acquiring process (method1), a similarity in pose of each of the various posing images of a specified character to the target image of imitation is evaluated. When pose similarity evaluation is executed in the corresponding-animation of imitation image acquiring process (method2), a similarity in pose of each of the various posing images of a specified model to a target image of imitation is evaluated.

FIG. 19is a diagram showing an example of evaluation of a similarity in pose of a specified-character image to a target image of imitation included in the pose similarity evaluating process of the corresponding-animation image acquiring process (method1) to be performed in the image output apparatus10.

FIG. 20is a diagram showing an example of evaluation of a similarity in pose (part1) of a model image to a target image of imitation included in the pose similarity evaluate process of the corresponding-animation image acquiring process (method2) to be performed in the image output apparatus10.

FIG. 21is a diagram showing an example of evaluation of a similarity in pose (part2) of a model image to a target image of imitation included in the pose similarity evaluating process of the corresponding-animation image acquiring process (method2).

First of all, in the similarity evaluation process (method1), a specified-character image with a first pose No. is read (step C1), for example, as shown inFIG. 19(A)and the specified-character image and the target image of imitation H1are adjusted so as to coincide in size (lengthwise and widthwise), as shown inFIG. 19(X)(step C2).

Thereupon, the adjusted character image and the target image of imitation H1are superposed on each other and then a similarity in pose (%) between them is evaluated, and stored as a similarity in pose of the character image to the target image of imitation (step C3).

Here, it is then determined whether the character image (with the first pose No.) as the target of similarity evaluation is asymmetric (step C4). If it is determined that the character image is not asymmetric (step C4, “No”), size adjustment and superposition-similarity evaluation between the target image of imitation and each of the posing images of the specified character having respective successive pose Nos. are sequentially performed, and their similarities are stored, as shown inFIGS. 19(B) and 19(C), for example (step C8→C9→C2-C4).

On the other hand, if it is determined that the character image with the pose No. as the target of similarity evaluation is asymmetric, the character image is mirror reversed (step C4→C5), and the reversed character image and the target image of imitation are adjusted to coincide in size (lengthwise and widthwise) (step C6).

Thereupon, the target image of imitation and the reversed character image are superposed on each other, and a similarity in pose (%) between them is evaluated and stored as the similarity in pose of the reversed image to the target image of imitation (step C7).

Then, if there are still further other posing images of the specified character having successive pose Nos. (step C8), they are sequentially read (step C9), and as described above, a similarity evaluation process is performed repeatedly between the respective posing images of the specified character having the successive pose Nos. and the target image of imitation (steps C2-C7).

For example, as shown inFIG. 19, when a similarity in pose of the “rabbit” character image to the object image (target image) H1is evaluated, a “rabbit” character image12d{circle around (1)}A with pose No.1A is evaluated as bearing the highest pose.

Then, when the similarity evaluation process (method2) is performed, as shown, for example, inFIG. 20(A)or21(A), a model image of a first pose No. {circle around (1)}A is read (step C1), and adjusted in size (lengthwise and widthwise) to the target image of imitation H1or H, as shown inFIG. 20(X)orFIG. 21(X)(step C2).

Then, the target image of imitation H1or H and the adjusted model image are superposed and a similarity in pose (%) between them is evaluated and stored as the similarity in pose of the model image to the target image of imitation (step C3).

Now, it is then determined whether or not the model image as the target of similarity evaluation is asymmetric (step4). If it is determined that the model posing image12b(1A) is not asymmetric, for example, as shown inFIG. 20(A)or21(A) (step C4, “No”), a model image with a next pose No. is read (step C8→C9), and in the same manner as described above, a pose similarity evaluation process is repeated between the target image of imitation H1or H and the model image of the next pose (steps C2-C4).

On the other hand, as shown, for example, inFIG. 20(B)or21(B) if it is determined that a model image12b(1B) or12b(4B) with pose No.1B or4B as the target of similarity evaluation in this instance is asymmetric, the model image with pose No. {circle around (1)}B or {circle around (4)}B is mirror reversed (step C4to C5), and as shown inFIG. 20(C)or21(C), this reversed model image12b(1B)R or12b(4B)R is adjusted in size (lengthwise and widthwise) to the target image of imitation H1or H (step C6).

Then, the target image of imitation H1or H and the reversed and size-adjusted model image12b(1B)R or12b(4B)R are superposed on each other, and a similarity in pose (%) between them is evaluated stored as the similarity in pose of the posing reversed model image12b(1B)R or12b(4B)R to the target image of imitation (step C7).

Then, if there are further model images with successive pose Nos. (step C8), these model images are sequentially read (step C9), and in the same manner as described, the pose similarity evaluation process is repeated between the target image of imitation and the respective model images (steps C2-C7).

Consequently, as shown, for example, inFIG. 20, if a pose similarity evaluation is performed between the object image (target image) H1and the respective posing model images, a model image12b(1B) with pose No.1B is evaluated as one bearing the highest pose similarity. As shown inFIG. 21for example, if pose similarity evaluation is carried out between the object image (target image) H and the respective model images, a model image12b(4B)R with pose No. {circle around (4)}BR is evaluated as one bearing the highest pose similarity.

As described, when a specified character image imitating that of the object image Hn as the target image of imitation extracted from the picked-up scene image12gnand set as such is determined in the corresponding-animation image acquiring process (step SB) as described with reference toFIGS. 17-21, it is then determined whether a compose mode for the specified image is an add/compose mode or otherwise a replace/compose mode (step S8).

If it is determined that the compose mode for the specified image is the “add/compose mode” and the “1-1: random space-area character image add mode” (step S8to S9, “1-1”), a “cat” character image12f{circle around (10)} which imitates a pose “banzai” of the object image H1as the target of imitation obtained in the corresponding-animation image acquiring process (seeFIG. 17) is added to and displayed at a random position in a space area other than in an area of the extracted object blocks M in the picked-up scene image12g2′, as shown inFIG. 10(D), for example (step S11a).

At this time, if the manual operation is performed to move the “cat” character image12f{circle around (10)} from its present position where the “cat” character image is added in the picked-up scene image12g2′ to any specified different position (step S16to S17), the “cat” character image is then added again to the different position and a resulting composite image is displayed (step S18).

Thereupon, data on the composite image is output from the image output unit20(step S19), recorded on a memory card20A or a VTR (recorder)20B or otherwise printed by an image printer20C on demand.

If it is determined that the specified image compose mode is the “add/compose mode” and the “1-2: follower add mode” (step S8→S9, “1-2”), as shown, for example, inFIG. 9a reversed “bear” character image12e{circle around (4)}R imitating a “walking” pose of the object image (target image) H in the picked-up scene image12g1, obtained in the corresponding-animation image acquiring process (seeFIG. 17), is added to and displayed at a position in the picked-up scene image12g1′ where the “bear” character image, so to speak, follows the object (target image) H, in other words, at the position of the object image (target image) H in a relevant one of the series of picked-up scene images12g1a certain time ago, as shown inFIG. 9(D)(step S11b).

At this time, if the added and displayed “bear” character image12e{circle around (4)}R is moved by a manual operation from its present position to any specified different position (step S16→S17), the “bear” character image is moved to the different position to thereby display a resulting composite image (step S18).

Thereupon, image data on this composite image is output from the image output unit20(step S19), and recorded on a memory card20A or the VTR (recorder)20B or otherwise printed out by the image printer20C on demand.

If it is determined that the compose mode for the specified image is the “add/compose mode” and the “1-3: mirror-reversed character image add mode (step S8to S9“1-3”), as shown, for example, inFIG. 11the “rabbit” character image12d{circle around (19)}A (FIG. 11(C)) imitating a dancing pose of the object (target) image H in the picked-up scene image12g3, obtained in the corresponding-animation image acquiring process (seeFIG. 17), is mirror reversed, added to a position where the reversed “rabbit” character image links with the object (target) image H striking a dancing pose in the picked-up scene image12g3′, and a resulting composite image is displayed, as shown inFIG. 11(D)(step S11c).

When in this case a process for moving the mirror reversed “rabbit” character image12d{circle around (19)}R displayed at the position in the picked-up scene image12g3in which the mirror reversed “rabbit” character image links with the object (target) image H from its present position to any specified different position is performed manually (step S16→S17), the mirror-reversed “rabbit” image12d{circle around (19)}R is moved to the different position to thereby provide a resulting composite image, which is then displayed (step S18).

Thereupon, data on this composite image is outputted from the image output unit20(step S19), recorded on the memory card20A or VTR (recorder)20B or otherwise printed out by the image printer20C on demand.

If it is determined that the specified image compose mode includes the “replace/compose mode” and the “2-1: object select/replace mode” (step SB8→S10, “2-1”), the specified object (target) image H1as the target image of imitation specified in the picked-up scene image12g5acquired in the corresponding-animation image acquiring process (FIG. 17) is replaced with a “rabbit” character image12d{circle around (1)} imitating the pose of the object image H1(step SB18), and a resulting composite image is then displayed, as shown inFIG. 13(G).

In this case, three object images, that is, the object (target) image H1and other two object images H2and H3are cut out from the picked-up scene image12g5to thereby provide a picked-up background image12g5′. The “rabbit” character image12d{circle around (1)} similar in pose to the object image H1, the remaining two object images H2and H3cut out from the scene image12g5are then added to the respective proper positions in the picked-up background image12g5′ in decreasing order of the image pickup distances to the respective objects concerned sensed by the distance sensor18in the image pickup to thereby provide a resulting composite image.

Without producing the remaining background image12g5′, the “rabbit” character image12d{circle around (1)} similar in pose to the object (target) image H1and the other two images H2and H3may be added to the their proper positions in the picked-up scene image12g5in decreasing order of the image pickup distances to the respective objects concerned sensed by the distance sensor18in the image pickup to thereby provide a resulting composite image.

When the process for moving the “rabbit” character image12d{circle around (1)} added in place of the target image H1in the picked-up scene image12g5″ from its present position to any specified different position is performed (step S16→S17), the “rabbit” character image12d{circle around (1)} is moved to the specified different position in the picked-up scene image12g5″ to thereby provide a resulting composite image, which is then displayed (step S18).

Thereupon, data on the composite image is outputted from the image output unit20(step S19), recorded on the memory card20A or VTR (recorder)20B or otherwise printed by the image printer20C, on demand.

If it is determined that the image compose mode includes the “replace/compose” mode and the “2-2: multiple image pickup/replace mode” (step S8→S10, “2-2”), the position of the object (target) image H1in the first picked-up scene image12g4is stored, for example, as the one where the object image H1is to be replaced with an imitating image, as shown inFIG. 12(step S10→S13).

When a second scene image12g4′ is picked up (step S14), a “rabbit” character image12d(20) imitating the pose of the object image H1acquired in the corresponding-animation image acquiring process (FIG. 17) is added in the second-time picked-up scene image12g4′ to a position corresponding to the stored position of the object (target) image H1in the first picked-up scene image12g4such that the “rabbit” character image12d(20) links with the object image H1and a resulting composite image12g4″ is then displayed (step S15).

When the process for moving the “rabbit” character image12d(20) from its present position to any specified different position is performed manually in the second picked-up scene image12dg4′ (step S16→S17), the “rabbit” character12d(20) is moved to the specified different position and a resulting composite image is displayed (step S18).

Thereupon, image data on this composite image is then outputted from the image output unit20(step S19), recorded on the memory card20A or the video tape recorder20B or otherwise printed out by the image printer20C on demand.

As will be obvious from the above, according to the image processing function of the image output apparatus10, when an object image is extracted from the scene image received through the image input unit17, any particular type of character image similar in pose to the extracted object image is selected from among the character1“rabbit” pose image memory12d, the character2“bear” pose image memory12eand the character3“cat” pose image memory12f, the selected particular type of character image replaces the object image in the picked-up scene image, or is added to the picked-up scene image to thereby provide a resulting composite image. That is, a very delightful scene image which uses the animation image imitating the pose of the target object as an added one or otherwise as one replacing the target object is easily created without creating a scene image to which a fixed animation image is simply added in the image pickup frame.

A function of the image output apparatus10that adjusts the size of a model image will be described next. In the function of adjusting the size of the model image, the posing model images coincident in size to the standard body-shaped object (target) images and stored in the model pose image memory12bare adjusted in size to a non-standard body-shaped object (target) image and stored as size-adjusted model images in the size-adjusted model pose image memory12c. In the corresponding-animation image acquiring process (part2), a size-adjusted model image stored in the adjusted-model pose image memory12cis used to evaluate a similarity in pose of the adjusted model image to an object image as the target image of imitation to thereby obtain a corresponding specified-character image bearing that pose similarity. This allows a character image similar in pose to an object image having any body shape to be obtained easily.

FIG. 22is a flowchart of a posing model size adjust process to be performed by the image output apparatus10.

When a picked-up image representing a scene containing an object including, for example, a non-standard body-shaped full length figure of the user shown inFIG. 8(B)is inputted via the image input unit17(step D1), a image of the user as an object is extracted in the target image extracting process (method1or2) ofFIG. 16(step DA) and compared to a model image12b{circle around (2)}A of a standard body-shaped full length stored in the model pose image memory12b(step D2).

Then, the model image12b{circle around (2)}A is adjusted in size to the object image extracted from the picked-up scene image with respect to the longitudinal proportions of face, body and legs (step D3) and then with respect to the width proportions of face, body and legs (step D4).

Thereupon, all other posing model images stored in the model pose image memory12bare adjusted in size to the size-adjusted model image12b{circle around (2)}A and then registered in the size-adjusted model pose image memory12c(step D5).

As described above, a similarity in pose of any one of the size-adjusted model images stored in the size-adjusted model pose image memory12cto an object image as the target image of imitation is evaluated and a character image bearing a similarity equal in pose to the object image is obtained such that character images similar in pose to object images having all kinds of body shapes can be easily obtained. A character image more similar in pose to the object image as the target object of imitation can be added to the picked-up scene image or replace the object image in the picked-up scene image to thereby provide a resulting composite image.

While in the embodiment a character image similar in pose to the object image extracted from the picked-up scene image is selectively obtained and its animation image is used as an added one or a replacement in the picked-up scene image to thereby output a resulting composite image, face images having various expressions and persons' images actually picked-up may be stored beforehand and used individually as an added one or a replacement imitating the pose of the object image extracted from the picked-up scene image in the picked-up scene image to thereby output a resulting composite image.

The present invention can be implemented well in a digital camera. If the present invention is implemented, especially in a digital camera with a liquid crystal display, an object image can be picked up so that a resulting composite image may be confirmed on the spot.

While in the embodiment the picked-up scene image as a still image has been illustrated, object images changing in time series relative to the background image of dynamic images inputted via the image input unit17, for example, a digital video movie17B may be sequentially extracted from the dynamic images. Character images similar in pose to the sequentially extracted object images, respectively, may be selectively acquired and added to the picked-up dynamic images or replace the object images in the respective dynamic images to thereby provide a resulting series of composite dynamic images.

The image processing function of the embodiment may be used to replace a character image as a target image of imitation specified and extracted from animation images, for example, used in a game with a prepared desired character image or a person's image picked up actually to thereby display a resulting composite image.

The respective processes performed by the image output apparatus10described in the embodiment, that is, any one of the image process shown in the flowcharts ofFIGS. 14 and 15, the object image extracting process included in the image process shown in the flowchart ofFIG. 16, the corresponding-animation image acquiring process included in the image process shown in the flowchart ofFIG. 17, the process for evaluating the similarity in pose of a character (or model) image to the object image shown in the flowchart ofFIG. 18included in the corresponding-animation image acquiring process, and the posing model size-adjusting process shown in the flowchart ofFIG. 22can be stored and distributed as programs, which a computer can execute, on memory cards (ROM and RAM cards, etc.), magnetic discs (floppy discs, hard discs, etc.), optical discs (CD-ROMs, DVDs, etc.), or external recording mediums13such as semiconductor memories, etc. Various computer terminals having a communication function with a communication network (the Internet) N can read the programs stored in the external recording mediums13through the recording medium reader14into the memories12and control its operation in accordance with the read programs to thereby perform the image processing functions described in the embodiment and perform processes similar to the methods mentioned above.

Data included in the programs to perform the respective above-mentioned processes can be transmitted via the communication network (the Internet) N in the form of a program code. The program data can be taken from the computer terminals connected to the communication network (the Internet) N to thereby perform the above-mentioned image processing functions.

Second Embodiment

FIG. 23is a block diagram indicative of the composition of an electronic circuit of an image delivery server110in an image delivery system of the invention.

FIG. 24is a block diagram indicative of the composition of an electronic circuit of a terminal120in the image delivery system according to the second embodiment.

The image delivery system provides image creating/delivering services from the image delivery server110to various terminals120such as a portable telephone120A, a personal digital assistant (PDA)120B, a personal computer (PC)120C, a digital camera120D, etc., connected to a communication network such as the Internet or Bluetooth (R). The image delivery server110delivers to each of the terminals120an image corresponding to a target image in an image picked up by the terminal120(120A-120D) (for example, a character image corresponding to a person image in a picked-up scene image).

InFIG. 23, the image delivery server110includes a computer that reads programs recorded on various recording mediums or programs received externally with the computer operation being controlled by the read programs. The computer includes a CPU (Central Processing Unit)111in its electronic circuit.

The CUP111controls the respective elements of the circuit in accordance with a server control program beforehand stored in a memory112, a server control program read into the memory112via a recording medium reader114from an external recording medium113such as a CD-ROM or a server control program read into the memory112via an electric transmission controller115from another Web server (in this case, a program server) on the Internet N. The server control program stored in the memory112is started up by a communication signal from each terminal120(120A-120C) on the Internet N connected via the transmission controller115, a communication signal from the digital camera120D connected via a short-distance wireless communication unit116or an input signal produced by an operator from an input unit117including keys or a touch panel.

The CPU111is connected to the memory112, recording medium reader114, transmission controller115, short-distance wireless communication unit116, and input unit117as well as a display unit118including an LCD or a CRT.

In addition to the system program in charge of the entire operation of the image delivery server110, a communication program for data transmission with the respective terminals120(120A-120C) on the Internet. N via the transmission controller115, and a short-distance communication program for data communication with digital cameras120D in a communication range of the short-distance wireless communication unit16via the communication unit16, the memory112has stored an image delivery program112ato produce an image corresponding to an object image in the picked-up image obtained from a respective one of the terminals120(120A-120D) communication connected via the transmission controller115and the short-distance wireless communication unit116, add the corresponding image to the picked-up image and deliver a resulting image to the terminal.

The memory112has prepared a model pose image memory112b, a size-adjusted model pose image memory112c, a character1“rabbit” pose image memory112d, a character2“bear” pose image memory112e, a character3“cat” pose image memory112f, and a requester-terminal information memory112gas well as in its work area a picked-up scene image memory112hand an image processing work area112i.

The character “rabbit”, “bear” and “cat” images striking respective poses stored in the characters “rabbit”, “bear” and “cat” pose image memories112d,112eand112f, respectively, of the image delivery server110are identical to those illustrated inFIGS. 2-4of the first embodiment, andFIGS. 2-4are also used to describe the second embodiment.

Each of the various posing images of the characters stored in the memories112d,112eand112fis used as an imitating image similar in pose to an object (target) image in a picked-up scene image inputted at a request of a respective one of the external terminals120(120A-120D) communication connected to the image delivery server110. Each posing character image is stored with its corresponding pose No.

FIGS. 5-7relate to the first embodiment and are also used to illustrate the second embodiment here.FIGS. 5-7show model images striking the respective poses stored in the model pose image memory112bof the image delivery server110.

Each of the various posing model images stored in the memory112bis used to obtain a model image similar in pose to an object (target) image in a picked-up scene image inputted at a request of a respective one of the external terminals120(120A-120D) communication connected to image delivery server110with the different poses being stored with corresponding pose numbers. In addition, three different variations of model images A, B and C are stored for each pose number.

The character images stored in each of the character1“rabbit” pose image memory112d, the character2“bear” pose image memory112e, and the character3“cat” pose image memory112fare the ones striking poses corresponding to all the pose Nos. (Nos.1A,1B,1C,2A,2B,2C, . . . , na, nb, nc) stored in the model pose image memory112b.

A size-adjusted model image of pose No.2A to be stored in a size-adjusted model pose image memory112cis obtained by vertically and horizontally adjusting the size of a standard body-shaped model image of a full length figure striking a pose shown by a pose No.2A inFIG. 4to a nonstandard body-shaped object (target) image. All other standard body-shaped model images striking respective poses are size adjusted in accordance with a vertical-horizontal ratio of the size-adjusted model image with pose No.2A and stored in the size-adjusted model pose image memory112c.

A size-adjusted model image to be stored in the size-adjusted-model pose image memory112cis obtained as follows. A standard body-shaped posing model image of a full-length shown by pose No.2A inFIG. 8(A)is adjusted in size (lengthwise and widthwise) to a nonstandard body-shaped object image shown inFIG. 8(B)to become a size-adjusted image corresponding to pose No.2A. In accordance with a size-adjust ratio (balance) of the size-adjusted model image for pose No.2A, all other standard body-shaped model images for the other poses are size adjusted to the non-standard body-shaped ones, and then stored in the size-adjusted-model pose image memory12c.

A requesters terminal information memory112ghas registered IDs, bank account numbers, and passwords of the terminals120(120A-120D), as their individual mail-order business information about image delivery members who have requested the image delivery server110to register them as such.

The picked-up scene image memory112htemporarily stores, in units of a picked-up scene image, picked-up scene image data inputted from the respective external terminals120(120A-120D) communication connected to the image delivery server110.

The image processing work area112itemporarily stores image data processed in accordance with the image delivery program112adepending upon the image processing.

InFIG. 24, each terminal120includes a computer that reads a program recorded on each type of recording medium or an externally received program with the computer operation being controlled by the program. Its electronic circuit includes a CPU (Central Processing Unit)121.

CUP121controls operation of the respective circuit elements in accordance with a terminal control program read beforehand into the memory122, a terminal control program read into the memory122via a recording medium reader124from an external recording medium123such as a CD-ROM or a ROM card, or a terminal control program read into the memory122via an electric transmission controller125from a Web server (in this case, program server) on the Internet N. The terminal control program stored in the memory122is started up by an input signal produced by a user's operation on an input unit126including keys or a touch panel, an image signal inputted by a digital camera132through an image input unit127a, a communication signal with each Web server such as an image delivery server110on the Internet N connected via the transmission controller125to the terminal120, a communication signal with the image delivery server110connected via a short-distance wireless communication unit128to the terminal120, or a present-position detection signal indicative of the present position of the moving terminal120from a GPS (Global Positioning System)129.

The CPU121is connected to the memory122, recording medium reader124, transmission controller125, input unit126, image input unit127a, short-distance wireless communication unit128, and GPS129as well as a display unit130of LCD and an image printer131through an image output unit127b.

The image input unit127areceives picked-up scene image data from an externally connected or built-in digital camera132. For example, if the terminal120includes a portable telephone120A, a personal digital assistant (PDA)120B or a digital camera120D, the image input unit127receives the picked-up scene image data from the built-in digital camera132. If the terminal120includes a personal computer (PC)120C, the image input unit127receives the picked-up scene image data from the externally connected digital camera132.

The image output unit127boutputs image data processed and delivered by the image delivery server110based upon the picked-up scene25image inputted by the image input unit127aat a request of the terminal. The outputted image data is then delivered to and printed by the image printer131.

In addition to the system program in charge of control of the whole operation of the terminal120(120A-120D), a communication program for data communication with each Web server such as the image delivery server110on the Internet N via the transmission controller125, and a short-distance communication program for data communication with the image delivery server110present in a communication range of the short-distance wireless communication unit128via the wireless communication unit128, the memory122has stored a communication/image processing program122afor requesting the image delivery server110to perform image processing based upon the picked-up scene image inputted from the image input unit127a, for displaying image data delivered by the image delivery server110, and for performing a printing process through the image output unit127b.

The terminal120(120A-120D) includes in the memory122a model pose image memory122band a size-adjusted-model pose image memory122csimilar in specification to those memories112band112cincluded in the memory112of the image delivery server110of the second embodiment.

The memory122includes an image processing work area122dthat temporarily stores image data processed in accordance with the communication/image processing program122adepending upon its processing.

Operation of the image delivery system of the second embodiment will be outlined next.

FIG. 11relates to the first embodiment and is also applied to describe the second embodiment.FIG. 11illustrates addition of a mirror-reversed character image “rabbit” in a picked-up scene image when the terminal120has specified a mirror-reversed character image add mode and requested for image delivery from the image delivery server110.

As shown inFIG. 11(A), the terminal120picks up a scene image12g3containing an object image (target image) H striking a pose, for example, beforehand conscious of dancing with a partner, and requests the image delivery server110to create a composite image in which a mirror-reversed character image dances with the target image by specifying a mirror-reversed character image add mode, and deliver it to the terminal120.

Then, as shown inFIG. 11(B), in the image delivery server110the object (target) image H is extracted from the picked-up scene image12g3sent by the terminal120. When a “rabbit” character image is specified as, an added one, a “rabbit” character image with pose No.19(“dance”) similar in pose to the object (target) image H is read out from the character1“rabbit” pose image memory112d, as shown inFIG. 11(C). Thereupon, this “rabbit” character image is mirror reversed and added in the picked-up scene image12g3′ to a position symmetrical to the position of the object (target) image in respect to a vertical center line, and where the mirror-reversed “rabbit” image links with the object (target) H for dancing purposes, as shown inFIG. 11(D). This produces a composite image which is then delivered to the requester's terminal120and displayed or printed out there.

That is, the image delivery server110mirror reverses the “rabbit” character image imitating the dancing pose of the object (target) image H in the scene image12h1picked up by the terminal120, and adds the mirror-reversed “rabbit” character image as a dance partner to a position in the picked-up scene image12h1where the mirror-reversed “rabbit” character image links with the object (target) image striking the dancing pose for dancing purposes. A resulting composite image is then sent to the terminal120.

FIG. 25illustrates replacement of an object image with a “bear” character image, which will be performed by the image delivery server110when the terminal120of the image delivery system has requested image creation and delivery from the server and specified a multiple image pickup/replace mode.

When the user at the terminal120desires to create a scene image in which the object image and a “bear” character image wave good-by to each other, the user picks up a scene image containing the object (target) image H1striking the same “good-by” pose as the character image is desired to strike, as shown inFIG. 25(A), sends the picked-up scene image to the image delivery server110, and requests the image delivery server10to create a composite scene image based upon the picked up scene image12h2by specifying the multiple image pickup/compose mode and deliver the composite scene image to the terminal.

As shown inFIG. 25(B), the image delivery server110extracts the image (target image) H1from the first picked-up image12h2sent by the user. When a “bear” character image is specified as being a replacing one by the user, a “bear” character image (striking a pose No.11“Good-by”) similar in pose to the object (target) image H1is read out from the character2“bear” pose image memory112e. In this case, when the present time is sensed, for example, as “08:00 p.m.” or it is determined that it is now at night, the “bear” character image striking the pose “Good-by” is read out, for example, as a night character image12e{circle around (11)}N sitting on the moon.

Then, as shown inFIG. 25(D), when a second scene image12h2′ containing a target image H2as a partner corresponding to the object image (target image) H1in the first picked-up scene image12h2and striking another pose is picked up by the terminal120, the read “bear” character image striking the pose No.11(“Good-by”) for use at night is added to the second picked-up scene image12h2′, to thereby provide a resulting composite image shown as12h2″ inFIG. 25(E). The resulting composite image is then delivered to the requester's terminal120and displayed or printed out there.

That is, in the image delivery server110the character image imitating the good-by pose of the object (target) image H1in the first scene image12h2picked up by the terminal120is added to a proper position relative to the position of the object (target) image H2in a second scene image12h2′ picked up by the terminal120to thereby provide a resultant composite image, which is then delivered to the terminal120.

WhileFIG. 13was used for explanation of the object replacing process of the image output apparatus110of the first embodiment, it is also applied for explanation of a similar object replacing process of the image delivery server of the second embodiment.

FIG. 13illustrates replacement of an object image with a “rabbit” character image by the image delivery server100when the terminal120requested image creation and delivery from the image delivery server and specified an object image select/replace mode.

As shown inFIG. 13(A), the terminal120picks up a scene image12g5containing, for example, three person (object) image H1-H3, sends the scene image along with notice of a specified replacing character image, for example, of “rabbit” to the image delivery server110, and requests the image delivery server110to create a composite image based upon the picked-up scene image by specifying an object image select/replace mode, and then to deliver it to the terminal120.

Thereupon, as shown inFIG. 13(B), the image delivery server110extracts from the picked-up scene image12g5sent by the terminal120an object (or target) image H1specified as being replaced by the terminal120. When a replacing “rabbit” character image is specified, a “rabbit” character image similar in pose (pose No.1“Say cheese” (close-up)) to the object (target) image H1is read out from the character1“rabbit” pose image memory112d, as shown inFIG. 13(C).

As shown inFIGS. 13(B), (D), (E) and (F), the specified object (target) image H1and other object images H2and H3are cut out from the scene image12g5and the background image12g5′ remains.

Thereupon, as shown inFIG. 13(G), the “rabbit” character image similar in pose to the specified object image (target image) H1read out from the character1“rabbit” pose image memory112dand the other two object images H2and H3cut out from the scene image12g5are added to proper positions in the background image12g5′ in decreasing order of the image pickup distances to the respective objects concerned to thereby provide a resulting composite image12g5″. The composite image12g5″ is then delivered to the requester's terminal120and displayed or printed out there.

That is, in the image delivery server110the “rabbit” character image imitating the pose of the specified object (target) image H1in the scene image12g5picked up by the terminal120replaces the object (target) image H1in the scene image12g5picked up by the terminal120, and a resulting composite image is delivered to the terminal120.

In the image delivery system, the image process modes include a follower add mode and a space-area character image add mode whose illustrative descriptions using the drawings are omitted in the outline of the operation of the image delivery server, in addition to the reversed-character image add mode, multiple image pickup/replace mode and object image select/replace mode.

In the follower add mode, when the terminal120picks up a scene image12hncontaining an object (target) image H walking or striking a walking pose and requests the image delivery server100to create a composite scene image based upon the scene image12hnby specifying the follower add mode and a character image as the follower imitating the pose of the object image H and to deliver the composite scene image to the terminal120, the image delivery server110adds the specified character image in the scene image12hnto a position where the specified character image as the follower follows the object (target) image H to thereby deliver a resulting composite image to the terminal120.

When the terminal120picks up a scene image12hncontaining any object (target) image H, sends this scene image12hnand a specified character image imitating the pose of the specified object (target) image H to the image delivery server110, and requests the delivery server110to create a composite scene image based upon the scene image and the specified character image H in the picked-up scene image12hnby specifying the space-area character image add mode and to deliver the composite image to the terminal, the image delivery server110adds the specified character image to a random position in a space area of the scene image12hnto thereby deliver a resulting composite image to the terminal120.

The details of the image processing and delivering operation by the image delivery server110and the terminal20will be described next.

FIG. 26is a flowchart of an image process (part1) to be performed by the terminal120.

FIG. 27is a flowchart of an image process (part2) continued from the image process (part1) ofFIG. 26.

FIG. 28is a flowchart of a server process (part1) to be performed by the image delivery server110.

FIG. 29is a flowchart of a server process (part2) continued from the server process (part1) ofFIG. 28.

First, the terminal120(120A,120B or120C) responds to the manipulation of its input unit126to connect via the transmission controller125or short-distance wireless communication unit128to the image delivery server110on the Internet N or in a short-distance wireless communication range and to send the user's ID and password thereof to the image delivery server110(step TA1).

When the image delivery server110receives the ID and password from the terminal120(120A,120B or120C) via the transmission unit115through the Internet N or from the terminal (digital camera)120D via the short-distance wireless communication unit116(step SB1), it determines whether or not the terminal is a full member depending on whether the terminal ID and password can find their matches among the members' ones registered in the requester terminal information memory112gof the memory112(step SB2).

When it determines that the ID and password of the terminal120can not find their matches among the registered members' IDs and passwords and that the terminal is not a full member, the connection of the terminal120to the image delivery server110is interrupted and an error process is performed (step SB2, “No.”).

When the connection of the terminal120to the image delivery server110is not interrupted over a predetermined time after the terminal120has sent its ID and password to the image delivery server110(step TA1), data representing the present position of the terminal120(in the case of the portable telephone120A, data representing the position of its base station concerned) sensed by GPS129is sent from the terminal120to the image delivery server110(step TA2).

When the image delivery server110receives from the terminal120data representing the present position of the terminal (step SB3), it determines whether the terminal120is at present in a predetermined particular place (for example, a souvenir picture pickup place in a theme park or tourist spot) or whether the present time is in a specified time zone (for example, at night) (step SB4).

If the image delivery server110determines that the terminal120is at present in the predetermined particular place or that the present time is in the specified time zone (step SB4), a particular character corresponding to the particular place (for example, in the case of a theme park, a popular character there) or a particular character corresponding to the specified time zone (for example, in the case of at night, a “bear” character sitting on a horned moon) is determined as the specified character corresponding to the target object and the name of this character is then sent to the terminal120(step SB5).

When the image delivery server110determines that the terminal120under connection is not in the predetermined place, or that the present time is not in the specified time zone (step SB4), the image delivery server110waits for reception of the type of a specified character from the terminal120(step SB6).

If the terminal120receives the name of the particular character determined and sent by the image delivery server110because the terminal120is in the particular place or the present time is in the specified time zone (step TA3), the particular character name is delivered to and displayed by the display unit130(step TA4).

When the terminal120does not receive the name of the particular character from the image delivery server110because the terminal120is not in the particular place or the present time is not in the specified time zone (step TA3), the user of the terminal120manipulates the input unit126to thereby specify the type of a character (“rabbit”, “bear” or “cat”) image to be used as an added one, or a replacement, in the picked-up scene image and sends the specified type of a character image to the image delivery server110(step TA5).

Then, the terminal120selects the add or replace/compose mode from among the “1-1: random space-area character image add mode”, “1-2: follower add mode”, “1-3: mirror-reversed character image add mode, “2-1: object image select/replace mode” and “2-2: multiple image pickup/replace mode”, and then sends a mode No. representing the selected mode to the image delivery server110(step TA6).

The digital camera132then starts to input a series of picked-up scene images through the image input unit127ato the terminal120, which then sends the series of picked-up scene images to the image delivery server110(step TA7).

When the image delivery server110receives the mode No. representing the selected mode from the terminal120(step SB7) and the series of picked-up scene images from the terminal120(step SB8), it performs a process for extracting the target image ofFIG. 30included in the server process (step SBC).

In this target image extracting process, the series of picked-up scene images12hnreceived from the terminal120at predetermined timings are stored in the pickup scene memory112hof the work area (step SC1).

Thereupon, a series of successive posing object images Hn indicating the series of successively changing acts of each object are extracted from the series of picked-up scene images12hnstored in the picked-up scene image memory112h(step SC2) and stored in the image processing work area12i(step SC3).

That is, when the respective object images Hn contained in the picked-up scene image12hnare extracted as separate blocks from the picked-up scene image12hnand stored in the image processing work area12i(step SBC), a picked-up scene image12hnwhich contains the respective extracted character images Hn in broken-lined enclosures M is displayed on the display unit118, for example, as shown inFIG. 11(A),12(A) or13(A) (step SB9).

Thereupon, it is determined whether there are a plurality of blocks of object image Hn in the picked-up scene image12hndisplayed on the display unit118(step SB10).

If the image delivery server110determines that there are no plurality of blocks of object image Hn in the picked-up scene image12hn, a single object image Hn extracted from the picked-up scene image12hnis set as a target image of imitation (step SB10→SB11)

When the image delivery server110determines that there are a plurality of blocks of object image Hn in the picked-up scene image12hn, the image delivery server110sends the terminal120a request that the user specify an object image block to be imitated (step SB10→SB12).

When the terminal120receives this request from the server110and displays it (step TA8), the user manipulates the input device126to specify a target image of imitation from among the plurality of blocks of object image Hn and gives notice of the specified image to the image delivery server110(step TA9).

Then, the image delivery server110receives this notice from the terminal (step SB13).

The image delivery server110then extracts the specified object image Hn from the picked-up scene image12hnreceived from the terminal120and sets it as a target image of imitation. Then, the image delivery server110shifts its process to a corresponding-animation image acquiring process included in the server process ofFIG. 31(step SBD).

In this corresponding-animation image acquiring process, the image delivery server110first evaluates a similarity in pose of each of the model images striking different poses stored in the model pose image memory112bto the object image Hn extracted from the picked-up scene image12hnand set as the target image of imitation (step SDC). The image delivery server110then sends the terminal120a specified character image A corresponding to a pose No. of the model image bearing the highest pose similarity to the object image (step SD1).

When the terminal120receives that specified character image A from the image delivery system110and displays it (step TA10), the user determines whether the displayed specified posing character image is “OK” or “next display” (display of a next candidate) should be clicked and then sends notice of this determination to the image delivery server110(step TA11).

When the image delivery server110receives the notice of the selection from the terminal120and determines that the terminal120has selected “the displayed specified posing character image is OK based upon the present evaluated similarity”, the image delivery server110registers the specified character image (identified by its pose No.) as an animation image corresponding to the target image of imitation (step SD2→SD3).

When the image delivery server110receives the notice of the selection from the terminal120and determines that the “next display” for the posing image of the received specified character is selected based upon the present similarity evaluation, the image delivery server110sends the terminal120another specified character image A corresponding to a pose No. of a model image bearing a second highest pose similarity. This is repeated each time the user's selection of the “next display” is received from the terminal120as long as there are posing character images corresponding to the posing model images (step SD2→SD4).

When the terminal120receives a posing character image corresponding to a model image bearing a desired pose similarity received from the image delivery server110and displays the posing character image, and the user manipulates the input unit126to send “OK” for the displayed character image to the image delivery server110(steps TA10-TA12), the image delivery server110registers the OK'd character image (identified by its pose No.) as an animation image corresponding to the target image of imitation (step SD2→SD3).

As described above, in the corresponding-animation image acquiring process by the image delivery server110, a similarity in pose of a model image representing each pose of a basic model of a person to the object image Hn as the target image of imitation extracted from the picked-up scene image12hnby the terminal120is evaluated, and a specified character image to be used as an added one or replacement imitating the pose of the object image Hn is determined.

In this case, compared with the case in which a similarity in pose of each of the various posing images of a specified character to the object image Hn as the target image of imitation is directly evaluated to thereby determine a character image to be used as an added one or as a replacement, the corresponding-animation image acquiring process in which a similarity in pose of each model image to the object image Hn as the target image of imitation is evaluated and a character image to be used as an added one or a replacement is determined from the pose No. of the selected model image is more advantageous in that the use of a human-figure model makes it easy to obtain a higher similarity in pose to the object image and hence a character image bearing a higher pose similarity (imitation).

While the similarity evaluating process ofFIG. 18has been performed by the image output apparatus of the first embodiment, a similar similarity evaluating process is performed by the image delivery server in the second embodiment. Thus, in the second embodiment the similarity evaluating process will be described by applyingFIG. 18.

FIG. 18is a flowchart of a process for evaluating a similarity in pose of an animation image to a target image of imitation in the corresponding-animation image acquiring process included in the server process to be performed by the image delivery server110of the image delivery system.

FIG. 32illustrates examples of evaluating a similarity in pose of a model character image to a target image of imitation in the corresponding animation image acquiring process to be performed by the image delivery server110.

As shown inFIG. 32(A), a model image, for example, with a first pose No. is read out (step C1) and adjusted in size to the target image of imitation H1ofFIG. 32(X)(step C2).

Then, the target image H1and the size-adjusted model image are superposed on each other, and a similarity in pose (%) between them is evaluated and stored as the similarity in pose of the model image to the target image H1(step C3).

It is then determined whether the model image is asymmetrical (step C4). When it is determined that the model image is not asymmetrical, for example, as shown by12b(1)A inFIG. 32(A)(step C4, “No.”), a next pose No. model image is read out (step C8→C9) and a similarity in pose of this model image to the target image of imitation H1is evaluated. As long as a further next model image is not asymmetrical, such similarity evaluating process is repeated (steps C2-C4).

If it is determined that a model image under similarity evaluation is asymmetrical for example, as shown by a pose No.19model image12b19ofFIG. 32(B), the model image is mirror reversed (step C4→C5). Then, as shown inFIG. 32(C),1the mirror-reversed model image12b(19)R is adjusted in size (lengthwise and widthwise) to the target image of imitation (step C6).

Then, the target image H1and the mirror-reversed model image12b(19) R are superimposed over each other. A similarity in pose (%) between these images is evaluated and stored as the similarity in pose of the mirror-reversed model image to the target image H1(step C7).

Then, as long as there are still model image with respective pose Nos. (step C8), they are sequentially read out (step C9), and a similarity in pose of a respective one of the model images to the target image of imitation is evaluated repeatedly (step C2-C7).

In this example, when, for example as shown inFIG. 32, the similarity in pose of the model image to the object (target) image H1is evaluated, the model image12b(19) with pose No.19is evaluated as bearing the highest similarity in pose to the target image H1.

As described above, when a specified character image imitating the object image Hn is determined as the target image of imitation extracted from the picked-up scene image12hnin the corresponding-animation image acquiring process (step SBD) described with respect toFIGS. 31,18and32, it is determined whether the image compose mode specified by and received from the terminal120is the “add/compose mode” or otherwise the “replace/compose mode” (step SB14).

If the image compose mode specified by the terminal120is the “add/compose mode” and the “1-1: random space-area character image add mode” (step SB14→SB15, “1-1”), the specified character image imitating the pose of the target image H1as the target image of imitation acquired in the corresponding-animation image acquiring process (FIG. 31) is added in the picked-up scene image12hnto a random position in its space-area except in a place from which the object image block is extracted, for example, as inFIG. 10(step SB17a). A resulting composite image is then sent to the terminal120(step SB22).

The terminal120then receives the composite image from the image delivery server110and displays it (step TA13). When the user at the terminal120performs a process for moving the character image from its resent position to any specified different position in the composite picked-up scene image12hn, information on its movement is then sent to the image delivery server110(step TA14).

When the image delivery server110receives this information from the terminal120(step SB23→SB24), the image delivery server110moves the character image to the different position based upon the information in the space-area of the picked-up scene image12hn(step SB25) and then sends a resulting composite image to the terminal120(step SB26).

Thereupon, the terminal120receives and displays this composite image from the image delivery server110(step TA15→TA16).

That is, the image delivery server110sends the terminal120data on the composite image in which the specified character image imitating the pose of the specified object image H1and created in the random space-area character image add mode is added to a random position in the space area of the picked-up scene image12hn.

When the image delivery server110determines that the image compose mode specified by the terminal120is the “add/compose mode” and the “1-2: follower add mode” (step SB14→SB15, “1-2”), it adds a specified character image imitating a waling pose of the object image (target image) in the picked-up scene image12hnacquired in the corresponding-animation image acquiring process (FIG. 31) at a position in the picked-up scene image12hnwhere the specified character image follows the walking object image (target image) H at a given distance from this object image, or at a position in the picked-up scene images12hnwhere the object (target) image assumed a certain time ago, as inFIG. 9(step SB17b). This image delivery server110then sends a resulting composite image to the terminal120(step SB22).

When the terminal120receives the composite image from the image delivery server110and displays it there (step TA13), and the user of the terminal120manually performs a process for moving the specified character image as the follower in the composite image from its present position to any specified different position, information on its movement is sent to the image delivery server110(step TA14).

When the image delivery server110receives this information from the terminal120(step SB23→SB24), the image delivery server110again adds the specified character image as the follower to the different position in the picked-up scene image12hn(step SB25) and sends a resulting composite image to the terminal120(step SB26).

Then, the terminal120receives this composite image from the image delivery server110and displays it (step TA15→TA16).

That is, the image delivery server110sends the terminal120data on a composite image in which the specified character image imitating the pose of the specified object image H and created in the follower add mode is added to the follower's position after the object image H.

When the image delivery server110determines that the image compose mode specified by the terminal120includes the “add/compose mode” and the “1-3: mirror-reversed character image add mode” (step SB14→SB15, “1-3”), the image delivery server110mirror reverses a “rabbit” character image12d{circle around (19)} imitating the dancing pose of an object image (target image) H in the scene image12g3, adds to a position in the scene image12g3where the mirror-reversed “rabbit” character image links with the target image H (step SB17c), and then sends a resulting composite scene image12g3′ to the terminal120(step SB22).

When the terminal120receives the composite image12g3′ from the scene image delivery server110and displays it (step TA13), and the user manually performs a process for moving the reversed “rabbit” character image12d{circle around (19)}R (R means reversal) in the picked-up scene image12g3′ from its present position to any specified different position, information on its movement is delivered to the image delivery server110(step TA14).

When the image delivery server110receives the movement information from the terminal120(step SB23→SB24), the image delivery server110again adds the mirror-reversed “rabbit” character image to the different position based upon the movement information in the scene image12g3′ (step SB25), and a resulting composite scene image is then sent to the terminal120(step SB26).

Then, the terminal120receives this composite scene image from the image delivery server110and displays it (step TA15→TA16).

That is, the image delivery server110sends the terminal120data on a scene image12g3in which the mirror-reversed “rabbit” character image imitating the pose of the dancing object image H and created in the mirror-reversed character image add mode is added to the position where the mirror-reversed “rabbit” character image links with the object image H, for example, as inFIG. 11(D).

If the image delivery server110determines that the image compose mode specified by the terminal120includes the “replace/compose mode” and the “2-1: object image select/replace mode” (step SB14→SB16, “2-1”), the image delivery server110replaces the specified object image (target image) H1with a “rabbit” character image12d{circle around (1)} imitating the pose of the object image H1as the target image of imitation specified in the picked-up scene image12g5acquired in the corresponding-animation image acquiring process (FIG. 31) (step SB18), and then sends a resulting composite image to the terminal120, for example, as inFIG. 13(step SB22).

In this case, three object images, that is, the target object image H1specified in the picked-up scene image12g5, and other two object images H2and H3are cut out from the picked-up scene image12g5, and a picked-up background image12g5′ is produced. The “rabbit” character image12d{circle around (1)} similar in pose to the specified target object image H1, the remaining two object images H2and H3cut out from the scene image12g5are then added to the respective relevant positions in the picked-up background image12g5′ in decreasing order of the sensed image pickup distances to the respective objects concerned to thereby provide a resulting composite image.

Without producing the remaining background image12g5′, the “rabbit” character image12d{circle around (1)} similar in pose to the target image H1and the other two images H2and H3may be added to the their relevant positions in the picked-up scene image12g5in decreasing order of the sensed image pickup distances to the respective objects concerned.

When the terminal120receives the composite image from the image delivery server110and displays it (step TA13), and manually performs the process for moving the “rabbit” character image12d{circle around (1)} added in place of the target image H1in the picked-up scene image12g5″ from its present position to any specified different position, information on its movement is then sent to the image delivery server110(step TA14).

When the image delivery server110receives this information (step SB23→SB24), it reads the “rabbit” character image12d{circle around (1)} to the different position based upon the movement information in the picked-up scene image12g5″ in place of the target character image H1(step SB25), and sends a resulting composite image to the terminal120(step SB26).

Thereupon, the terminal120receives this composite image from the image delivery server110and displays it (step TA15→TA16).

That is, the image delivery server110sends the terminal120data on a composite image created in the object image select/replace mode and in which the object image H1is replaced with the “rabbit” character image imitating the pose “Say cheese” of the object image H1, as inFIG. 13(G).

If the image delivery server110determines that the image compose mode specified by the terminal120includes the “replace/compose” mode and the “2-2: the multiple image pick-up/replace mode” (step SB14→SB16, “2-2”), the position of the target object image H1in a first picked-up scene image12h2is stored as the position where the imitating image replaces the target object image H1, for example, as shown inFIG. 25. (step SB16→SB19).

When the image delivery server110acquires a second picked-up scene image12h2′ (step SB20), it adds a “bear” character image12e{circle around (11)}N (in this case, a special “night bear” character image) imitating the pose of the object image H1acquired in the corresponding-animation image acquiring process (FIG. 31) to a position in the second picked-up scene image12h2′ corresponding to the stored position of the object image (target image) H1in the first picked-up scene image12h2such that the “bear” character image12e{circle around (11)}N waves good-by to the object image H2(step SB21) and a resulting composite image12h2″ is then sent to the terminal120(step SB22).

When the terminal120receives and displays the composite image12h2″ from the image delivery server110(step TA13) and manually performs the process for moving the “night bear” character image12e{circle around (11)}n in the composite image12h2″ from its present position to any specified different position in the composite image12h2″, information on its movement is sent to the image delivery server110(step TA14).

When the image delivery server110receives this information (step SB23→SB24), it reads the “bear” character image12e{circle around (11)}N′ to the different position in the second picked-up scene image12h2′ based upon the received movement information (step SB25), and a resulting composite image is then sent to the terminal120(step SB26).

Then, the terminal120receives this composite image from the image delivery server110and displays it (step TA15→TA16).

That is, the image delivery server110sends the terminal120the composite image12h2″ created in the multiple image pickup/replace mode in which the “bear” character image (particular “night” character image) imitating the “Good-by waving” pose of the object image H1in the first picked-up scene image12h2is combined with the object image H2in the second picked-up scene image12h2′. A resulting composite image is then delivered to the terminal120.

The terminal120receives and displays this composite image. When the user of the terminal120desires to send the composite image displayed on the terminal120to a specified different terminal, the user inputs an address of the different terminal via the input unit126and then sends the composite image to the image delivery server110(step TA17→TA18).

When the image delivery server110receives the address of the different terminal from the terminal120(step SB27), the image delivery server110sends the composite image to the different terminal of that address (step SB28), calculates a charge for the composite image-creating and sending services provided this time to the requester user and the specified different terminal and sends the requester terminal120the contents of the charge (step SB29).

The requester terminal120then receives the contents of the charge from the image delivery server110and displays it on the display130(step TA19).

When the terminal120inputs and sends no address of the specified different terminal (step TA17→No), the image delivery server110calculates a charge for only the composite image creating and delivering services provided to the requester terminal this time and sends the requesting terminal120the contents of the charge (step S27→SB30).

Thereupon, the requester terminal120receives and displays the contents of the charge from the image delivery server110and displays it on the display30(step TA20).

The image delivery server110sends a corresponding bill to a bank account of the requester terminal registered in the requester terminal information memory112g(step SB31).

Therefore, according to the image delivering function of the image delivery system of the second embodiment, when the scene image picked-up by the terminal120is sent to the image delivery server110, the image delivery server110extracts an object image from the received scene image, acquires any specified type of character image similar in pose to the extracted object image from among the character1“rabbit” pose image memory12d, the character2“bear” pose image memory12e, and the character3“cat” pose image memory12f, replaces the object image in the picked-up scene image with the acquired specified type of character image in the picked-up scene image or adds the required character image to the scene image to thereby provide a resulting composite image, and sends this composite image to the terminal120. Thus, a very delightful composite scene image in which an animation image imitating the pose of the target object in the original scene image is used as an added one or otherwise as one replacing the target object is created and delivered to the terminal without creating and delivering a scene image to which a fixed animation image is simply added in the image pickup frame.

By using the image delivery server110in the image delivery system of the second embodiment, various services are provided which include delivery of a mail with a composite scene image including an added or replacing character image imitating the pose of an object image in the picked-up scene image, pay composition and delivery of a composite scene image in which a character image imitating the pose of a target image in a picked-up original scene image is used as an added or replacing one in the original scene image to a respective one of image-pickup type mobile terminals120B,120C,120D such as a cellular phone terminal120A with a digital camera, pay or free creation and delivery of a composite scene image in which a particular character image imitating the pose of the object image in a picked-up scene image is added to, or otherwise replaces the object image, for publicizing the particular character on a particular home page, and pay creation and delivery of a picked-up scene image in which a popular character image imitating the pose of an object image in a picked-up scene image is added to a digital camera terminal120D connected via a short-distance wireless communication unit116only in a specified place such as an amusement spot.

As described above, in the image delivery system of the second embodiment, an image picked up by the terminal120is sent to the image delivery server110. In the image delivery server110, an object image is extracted from the picked-up scene image, a character image is acquired which is similar in pose to the extracted object image, the character image is added to the scene image or replaces the object image in the scene image. A resulting composite image is then delivered to the terminal120and a bill for this service is sent to the terminal120. In contract, as will be described in the next third embodiment, the terminal120extracts an object image from the picked-up scene image, determines a pose No. of a model image similar in pose to the extracted object image, and requests the image delivery server110to deliver a specified character image with the determined pose No. The image delivery server110only delivers to the terminal120the character image with the determined pose No., which was requested by the terminal120, and bills the terminal for this service. The terminal120further adds the delivered character image to the scene image or otherwise replaces an object image in the scene image with the delivered character image to thereby provide a resulting composite image.

Third Embodiment

The electronic circuits of the image delivery server110and terminals120(120A-120D) of the image delivery system as the third embodiment of the present invention are substantially similar to those of the second embodiment described with reference toFIGS. 23 and 24, excluding the following points:

In the image delivering system of the third embodiment, the model image memory112band the size-adjusted model pose image memory112care not provided in the image delivery server110, but in the terminal120.

FIG. 33is a flowchart of an image process (part1) to be performed by the terminal120in the image delivery system of the third embodiment.

FIG. 34is a flowchart of an image process (part2) continued from the image process (part1) ofFIG. 33.

FIG. 35is a flowchart of a server process to be performed by the image delivery server110of the third embodiment.

First, the terminal120(120A,120B or120C) responds to the manipulation of the input unit126to connect via the transmission controller125or short-distance wireless communication unit128to the image delivery server110on the Internet N or in a short-distance wireless communication range and then to send the user's ID and password to the image delivery server110(step TF1).

When the image delivery server110receives the ID and password from the terminal120(120A,120B or120C) via the transmission unit115through the Internet N or the terminal (digital camera)120D via the short-distance wireless communication unit116(step SJ1), it determines whether the terminal120is a full member or not depending on whether the terminal ID and password can find their matches among the registered IDs and passwords of the member terminals in the requester terminal information memory112gof the memory112(step SJ2).

When it is determined that the ID and password of the terminal120can not find their matches among the registered IDs and passwords of the member terminals and that the terminal is not a full member, the connection of the terminal120to the image delivery server110is interrupted and an error process is performed (step SJ2, “No.”).

When the connection of the terminal120to the image delivery server110is not interrupted over a predetermined time after the terminal120has sent its ID and password to the image delivery server110(step TF1), data representing the present position of the terminal120(in the case of the cellular phone120A, data representing the position of its base station concerned) sensed by GPS129is sent from the terminal120to the image delivery server110(step TF2).

When the image delivery server110receives from the terminal120data representing its present position (step SJ3), it determines whether the terminal120is at present in a predetermined particular place (for example, a souvenir picture pickup place in a theme park or tourist spot) or whether the present time is in a specified time zone (for example, at night) (step SJ4).

If the image delivery server110determines that the terminal120under connection is at present in the predetermined particular place or that the present time is in the specified time zone (step SJ4), a special character corresponding to the particular place (for example, in the case of a theme park, a popular character there) or a special character corresponding to the specified time zone (for example, in the case of at night, a “bear” character sitting on the sickle moon) is determined as the special character corresponding to the target object, and the name of this character is then sent to the terminal120(step SJ5).

When the image delivery server110determines that the terminal120under connection is not in the predetermined place, or that the present time is not in the specified time zone (step SJ4), the image delivery server110waits for reception of a specified type of character from the terminal120(step SJ6).

If the terminal120receives the name of the special character determined and sent by the image delivery server110because the terminal120is in the particular place or the present time is in the specified time zone (step TF3), the special character name is delivered to and displayed by the display unit130(step TF4).

When the terminal120does not receive the name of the special character from the image delivery server110because the terminal120is not in the particular place or the present time is not in the specified time zone (step TF3), the user of the terminal120manipulates the input unit126to thereby specify the type of a character (“rabbit”, “bear” or “cat”) to be used as an added one or a replacement in the picked-up scene image and sends the specified type of a character to the image delivery server110(step TF5).

Thereupon, the picked-up scene images start to be input from the digital camera132via the image input unit127ato the terminal120(step TF6), and the process proceeds to the target image extracting process (TFG) ofFIG. 36included in the image process by the terminal120.

In this target image extracting process, the series of picked-up scene images inputted through the image input unit127ainto the terminal120at predetermined timings are stored in the work area122dof the memory122(step TG1).

Thereupon, a series of successive object images Hn indicating the respective acts of each object are extracted as separate blocks from the series of picked-up scene images stored in the image processing work area122d(step TG2) and stored as the respective blocks in the work area122d(step TG3).

That is, when the respective object images Hn contained in the picked-up scene image are extracted as separate blocks from the picked-up scene images and stored in the image processing work area122d(step TFG), a picked-up scene image which contains the respective extracted character images Hn in their broken-lined enclosures M is displayed on the display unit130, for example, as inFIG. 11(A),12(A) or13(A) (step TF7).

Thereupon, it is then determined whether there are a plurality of blocks of object image Hn in the picked-up scene image displayed on the display unit130(step TF8).

If it is determined that there are no plurality of blocks of object image Hn, a single object image Hn extracted from the picked-up scene image is set as a target image of imitation (step TF8→TF9)

If it is determined that there are a plurality of blocks of object image Hn, an object image Hn is specified as a target image of imitation from among those blocks of object image Hn by the user's manipulation of the input unit126(step TF8→TF10)

When the object image Hn is extracted from the picked-up scene image and specified as the target image of imitation Hn, the process proceeds to a pose No. determining process ofFIG. 37(step TFI).

In this pose No. determining process, a similarity in pose of each of the differently posing model images stored in the model pose image memory122bto the object image Hn extracted from the picked-up scene image and set as the target image of imitation is evaluated (step TIK), and a posing model image bearing the highest pose similarity is displayed on the display unit130(step TI1).

When this displayed model image is selected by the user at the input unit126, the pose No. of this selected model image is registered as the one corresponding to the target image of imitation (step TI2→TI3).

When the “next display” is clicked by the user's manipulation on the input unit126in a state where a model image corresponding to the pose No. representing the highest similarity in pose to the target image of imitation is displayed, a model image with a pose No. representing a second highest similarity in pose to the target image is displayed. As long as there are still a plurality of model images having different similarities, a model image bearing a next lower pose similarity is displayed each time the “next display” is clicked. (step TI2→TI4).

When a model image with a pose No. representing a desired pose similarity is displayed and selected by the user's manipulation on the input unit126, the pose No. of this model image is registered as that corresponding to the target image of imitation (step TI2→TI3).

As described above, in the pose No. determining process, the pose similarity of each model image (striking a respective pose) representing a basic model of a person to the object image Hn as the target image of imitation extracted from the picked-up scene image is evaluated, and the pose No. of a model image imitating the pose of the object image Hn is determined based upon the evaluation of the similarity.

FIG. 38is a flowchart of evaluating a similarity in pose of a model image to the target image of imitation in the pose No. determining process included in the image process to be performed by the terminal120in the image delivering system of the third embodiment.

For example, as shown inFIG. 32(A), a model image with a first pose No. is read out (step TK1) and adjusted in size to the target image of imitation H1ofFIG. 32(X)(step TK2).

Then, the target image H1and the size-adjusted model image are superposed on each other, and a similarity in pose (%) between them is evaluated and stored as the similarity in pose of the model image to the target image H1(step TK3).

It is then determined whether the model image with the pose No. of interest is asymmetrical (step TK4). When it is determined that the model image is not asymmetrical, for example, as shown by12bA(1) inFIG. 32(A)(step TK4, “No.”), a next pose No. model image is read out (step TK8→TK9) and a similarity in pose of this model image to the target image of imitation H1is evaluated likewise. As long as a further next model image is not asymmetrical, such similarity evaluating process is repeated (steps TK2-TK4).

If it is determined that the model image of similarity evaluation is asymmetrical, for example, as shown by a pose No.19model image12b19ofFIG. 32(B), the model image is mirror reversed (step TK4→TK5). Then, as shown inFIG. 32(C),1the mirror-reversed model image12b(19)R is adjusted in size (lengthwise and widthwise) to the target image of imitation (step TK6).

Then, the target image H1and the mirror-reversed model image12b(19)R are superposed on each other. A similarity in pose between these images (%) is evaluated and stored as the pose similarity of the mirror-reversed model image to the target image H1(step TK7).

Then, as long as there are further model image of different pose Nos. (step TK8), they are sequentially read out (step TK9), and a similarity in pose of a respective one of the model images to the target image of imitation is repeatedly evaluated (step TK2-TK7).

In this example, when for example, as shown inFIG. 32, the similarity in pose of the model image to the object (target) image H1is evaluated, the model image12b(19) with pose No.19is evaluated as bearing the highest similarity in pose to the target image H1.

As described above, when a pose No. of a model image imitating the pose of the object image Hn as the target image of imitation extracted from the picked-up scene image is determined in the pose No. determining process (step TFI) described with respect toFIGS. 37,38and32, a request for delivery of a specified character image striking a pose for the determined pose No. is then sent to the image delivery server110(step TF11).

When the image delivery server110receives the request for delivery of a character image from the terminal120(step SJ7), the image delivery server110selectively reads a character image with the determined pose No. from an appropriate one of the character1“rabbit” pose image memory112d, the character2“bear” pose image memory112e, and the character3“cat” pose image memory112fincluded in the memory112and delivers the read character image to the requester terminal120(step SJ8).

Thereupon, the image delivery server110calculates the charge for the delivery service provided to the requester user and sends notice of the contents of the charge to the requester terminal120(step SJ9), and a corresponding bill is sent to a bank account of the requester terminal registered in the requester terminal information memory112g(step SJ10).

When the requester terminal120receives the character image with the determined pose No. from the image delivery server110(step TF12), either the add/compose mode that adds the received character image to the picked-up scene image or the replace/compose mode that replaces the target image H in the picked-up scene image with the received character image is selected from among the “1-1: random space-area character image add mode”, “1-2: follower add mode”, “1-3: mirror-reversed character image add mode”, “2-1: object image select/replace mode” or “2-2: multiple image pickup/replace mode” (step TF13).

In the flowchart of the image process performed by the terminal120inFIGS. 33 and 34, routines corresponding to the replace/compose mode “2-1: object image select/replace mode” and “2-2: multiple image pickup/replace mode” are omitted. The omitted routines include the same process as are expressed by the steps SB16, SB18-SB21ofFIGS. 28 and 29and are selectively performed along with the process of steps TF14and TF15a-TF15cin the add/compose mode which will be described later.

Therefore, only the process to be performed when the add/compose mode is selected will be described herein and the process to be performed when the replace/compose mode is selected will be omitted.

If it is determined in step TF13that the specified-image compose mode is the “add/compose mode” and the “1-1: random space-area character image add mode” (step F14, “1-1”), the specified character image imitating the pose of the target image Hn as the target image of imitation delivered by the image delivery server110is added in the picked-up scene image12hnto a random position in its space-area except in the place from which the object image block is extracted. A resulting composite image is then displayed (steps TF15a, TF16).

Then, when the process for moving the character image in the composite image from its present position to any specified different position is performed in the composite image (step TF17→step TF18), the character image is moved to the different position in the composite image, and another resulting composite image is then displayed (TF19-TF20).

When it is determined that the specified-image compose mode includes the “add/compose mode” and the “1-2: follower add mode” (step TF14, “1-2”), a specified character image delivered by the image delivery server110and imitating a walking pose of the object image (target image) in the picked-up scene image is added to a position in the latest scene image corresponding to the position on a relevant one of the series of picked-up scene images12hnwhere the walking object image (target image) H assumed a certain time ago to thereby provide a resulting composite image, as inFIG. 9(D)(steps TH15b, TF16).

When the process for moving the specified character image as the follower in the composite image from its present position to any specified different position in the composite image is performed manually (step TF17to TF18), the specified character image is moved to the different position in the composite image and another resulting composite image is then displayed (steps TF19, TF20).

When it is determined that the specified-image compose mode includes the “add/compose mode” and the “1-3: reversed-character image add mode” (step F14, “1-3”), a “rabbit” character image12d{circle around (19)} imitating the dancing pose of an object image (target image) H in the scene image12g3is mirror reversed, the mirror-reversed “rabbit” character image is added at a position in the scene image12g3where the mirror reversed “rabbit” character image links with the target image H for dancing purposes, and then a resulting composite image12g3′ is displayed, as shown inFIG. 11(D)(step TF15c→TF16).

When the process for moving the reversed “rabbit” character image12d{circle around (19)}R in the composite image12g3′ from its present position to any specified different position is manually performed (step TF17to step TF18), the mirror reversed “rabbit” character image is moved to the different position in the composite image12g3′, and another resulting composite image is then displayed (steps TF19, TF20).

Then, the terminal120receives notice of the contents of the charge for the use of the server service by the user from the image delivery server110and displays it on the display unit130(step TF21).

Therefore, according to the image delivering function of the image delivery system of the third embodiment, when the terminal120picks up an image of a scene containing objects, an object image is extracted from the picked-up scene image. A pose No. is determined from a model image similar in pose to the extracted object image. The terminal120requests delivery of a posing character image with the determined pose No. and a specified type of the character image from the image delivery server110. Thereupon, the image delivery server110reads a character image with the specified pose No. and type from an appropriate one of the character1“rabbit” pose image memory112d, the character2“bear” pose image memory112e, and the character3“cat” pose image memory112f, and delivers the read character image to the terminal120, which replaces the object image in the picked up scene image with the received character image similar in pose to the object or adds the received character image to the scene image, and displays a resulting composite image. That is, the user at the terminal120does not simply add a fixed animation image to the pickup frame for delighting purposes, but acquires an animation image similar in pose to the object image in the scene image from the image delivery server110and replaces the object image in the picked-up scene image with the animation image or adds the animation image to the picked-up scene image to thereby produce a very delightful composite image containing the animation image.

The image delivery server110in the image delivery system of the third embodiment is capable of providing image-pickup type mobile terminals120B,120C,120D such as a cellular phone terminal120A, which includes a digital camera, with pay delivery of a character image imitating a pose of an object image in each picked-up scene image, pay or free delivery of a particular character image imitating a pose of an object image in a picked-up scene image for publicizing the particular character on a particular home page, and pay delivery of a popular character image imitating a pose of an object image in a picked-up scene image to a digital camera terminal120D via a short-distance wireless communication unit116only in a specified place such as an amusement spot, etc.

While in the respective embodiments a character image similar in pose to the object image extracted from the picked-up scene image is selectively obtained and used as an added one or a replacement in the picked-up scene image to thereby provide a resulting composite image, face images having various expressions and person images actually picked-up may be stored beforehand and used as an added one or a replacement imitating a pose of an object image extracted from the picked-up scene image to thereby output a resulting composite image.

In the respective above embodiments the picked-up scene image as a still image has been illustrated. On the other hand, when the picked-up scene images include a series of animation images inputted, object images changing in time series relative to the background image of the series of dynamic images inputted may be sequentially extracted from the series of animation images. Character images (FIG. 39) similar in pose to the sequentially extracted object images may then be selectively acquired, delivered, or added to the picked-up dynamic images or replace the object images in the respective animation images to thereby provide a series of resulting composite animation images, which will be then delivered.

FIG. 39illustrates animation character images striking various poses stored in the character “rabbit” animation pose image memory112dM in the image delivery server110.

By utilizing the image processing function of the image delivery server110or terminal120in the respective embodiments, a prepared desired character image similar in pose to a target image of imitation that includes an appearing character specified and extracted, for example, from an animation image in a game, or otherwise an actually picked-up person image may be acquired and delivered to the requester terminal or replace the character appearing in the animation image to thereby deliver a resulting composite image to the requester terminal.

As will be described in a next fourth embodiment, object images striking various poses picked up by the user, or character images striking various poses appearing in a game downloaded from game servers (game HP) on the Internet N may be recorded in memories similar to the character1“rabbit” pose image memory112d, the character2“bear” pose image memory112e, and the character3“cat” pose image memory112fof the memory112in the image delivery server110. When a character image similar in pose to the object image extracted from the picked-up scene mage in the second or third embodiment is acquired from the image delivery server110, it may be selected from among the originally registered “rabbit”, “bear” and “cat” character images, and the person images and game character images registered by the user.

Fourth Embodiment

FIG. 40is a block diagram indicative of a whole composition of an image delivery system in the fourth embodiment.

This embodiment handles the user's pickup of object images striking various poses, using a cellular phone220A with a digital camera, and registration of the object images as the user's original character images in a memory212of an image delivery server210, or otherwise registration of game characters striking various poses downloaded from a memory212A of a game server210A as the user's original characters in the memory212of the delivery server210.

FIGS. 41(A) and 41(B)are flowcharts of the user's original character image registering processes to be performed in the cellular phone220A and the image delivery server210, respectively, of the image delivery system in the fourth embodiment.

First, when the user's original character images are downloaded by and registered on the cellular phone220A with the digital camera from the game server210A on the Internet N, a homepage (HP) of the game server210A from which the character images are acquired is specified, and connected to the cellular phone220A (step TS1→TS2).

When the user specifies a game character to be set as his or her original character image on the homepage (HP) of the game server210A (step TS3), the various posing images of a specified game character stored in the memory212A of the game server210A are downloaded (step TS4).

Thereupon, the user ID (=123) of the cellular phone220A and the respective posing character images downloaded from the game server210A are delivered to the image delivery server210(step TS5).

When the user picks up and registers the user's original character images using the cellular phone220A without downloading similar character images from the Internet N, the digital camera232picks up a scene image containing object images striking their respective poses (step TS1→TS7).

Thereupon, the cellular phone220A extracts the character images as object images excluding the background image in the picked-up scene image (step TS8) and stores them as corresponding-character images taking respective poses (step TS9).

Then, the cellular phone220A then sends its terminal user's ID (=123) and the stored corresponding-images striking the respective poses to the image delivery server210(step TS5).

When the image delivery server210receives the terminal user's ID and the character images striking their respective poses from the cellular phone220A (step SR1), it registers the character images as the ones striking respective poses for the user's ID, apart from the respective “rabbit”, “bear” and “cat” character images (112d-112f), for example, ofFIG. 23in the memory212(step SR2).

When the image delivery server210selects and deliver a character image similar in pose to the object image extracted from the picked-up scene image in the second or third embodiment, the image delivery server210can select a character image from the registered “rabbit”, “bear” and “cat” character images, and the character images picked up and registered by the user, and the game characters.

The various processes performed by the image delivery system described with reference to the respective embodiments, that is, the image process shown by the flowchart ofFIGS. 26 and 27performed by the terminal120of the image delivery system of the second embodiment, the server process performed by the image delivery server110shown by the flowcharts ofFIGS. 28 and 29in the image delivery system of the second embodiment, the object image extracting process included in the server process in the second embodiment shown by the flowchart ofFIG. 30, the corresponding-animation image acquiring process included in the server process of the second embodiment shown by the flowchart ofFIG. 31, the process of evaluating the similarity in pose of the character/model image to the target image of imitation included in the corresponding-animation image acquiring process in the second embodiment shown by the flowchart ofFIG. 18, the image process performed by the terminal120in the image delivery system of the third embodiment shown by the flowcharts ofFIGS. 33 and 34, the server process of the image delivery server110in the image delivery system of the third embodiment shown by the flowchart ofFIG. 35, the target image extracting process included in the image process of the third embodiment shown by the flowchart ofFIG. 36, the pose number determining process included in the image process of the third embodiment shown by the flowchart ofFIG. 37, the process of evaluating the similarity in pose of the model pose image to the target image of imitation included in the pose number determining process in the third embodiment shown by the flowchart ofFIG. 38, the user's character image registering process performed in the cellular phone220A in the image delivery system of the fourth embodiment shown by the flowchart ofFIG. 41, and the user's character image registering process performed by the image delivery server210may be stored and distributed as computer-executable programs on memory cards (ROM cards, RAM cards, etc.), magnetic discs (floppy discs, hard discs, etc.), optical discs (CD-ROMs, DVDs, etc.), and external recording mediums such as semiconductor memories113(123,213,223). Various computer terminals having communication functions with the communication networks (the Internet) N can read the programs stored on the external recording mediums113(123,213,223) into the memories112(122,212,222) using the recording medium readers114(124,214,224). Thus, operation of the computers may be controlled by the these programs to perform the image delivering function described with reference to the respective embodiments to thereby perform the processes similar to those mentioned above.

The program data to realize the respective processes can be transmitted through the communication network (the Internet) N in the form of a program code. The program data can be taken from the computer terminals connected to the communication network (the Internet) N to realize the image delivering function.

The present invention is not limited to the ones implemented by the respective embodiments. Various changes and modification could be easily made by those skilled in the art without departing from the spirit and scope of the invention.