Source: http://www.google.com/patents/US20020031262?dq=5072412
Timestamp: 2017-11-24 06:18:52
Document Index: 346145519

Matched Legal Cases: ['art 14', 'art 17', 'art 14', 'art 17', 'art 18', 'art 19', 'art 20', 'art 18', 'art 19', 'art 21', 'art 15', 'art 1', 'art 2', 'art 3', 'art 4', 'art 5', 'art 6', 'art 7', 'art 8', 'art 9', 'art 10', 'art 11', 'art 12', 'art 13', 'art 5', 'art 2', 'art 4', 'art 5', 'art 17', 'art 17', 'art 18', 'art 18', 'art 18', 'art 19', 'art 21', 'art 20', 'art 21', 'art 20', 'art 17', 'art 18', 'art 14', 'art 15', 'art 17', 'art 22', 'art 23', 'art 23', 'art 10', 'art 22', 'art 23', 'art 22', 'art 17', 'art 22', 'art 17', 'art 22', 'art 17', 'art 22', 'art 23', 'art 22', 'art 17', 'art 22', 'art 22', 'art 14', 'art 15', 'art 17', 'art 18', 'art 26', 'art 24', 'art 17', 'art 18', 'art 26', 'art 24', 'art 24', 'art 26', 'art 24', 'art 26', 'art 26', 'art 724', 'art 724', 'art 724', 'art 724', 'art 724']

Patent US20020031262 - Method and device for media editing - Google Patents
The present media editing device generates media including messages in an easy manner in a communication terminal such as a mobile terminal. Therein, a moving image data storage part 14 stores moving image data recorded by a user. A region extraction part 17 extracts any region including the user from...http://www.google.com/patents/US20020031262?utm_source=gb-gplus-sharePatent US20020031262 - Method and device for media editing
Publication number US20020031262 A1
Application number US 09/950,085
Also published as CN1200537C, CN1344084A, US6961446
Publication number 09950085, 950085, US 2002/0031262 A1, US 2002/031262 A1, US 20020031262 A1, US 20020031262A1, US 2002031262 A1, US 2002031262A1, US-A1-20020031262, US-A1-2002031262, US2002/0031262A1, US2002/031262A1, US20020031262 A1, US20020031262A1, US2002031262 A1, US2002031262A1
Inventors Kazuyuki Imagawa, Yuji Takata, Hideaki Matsuo, Katsuhiro Iwasa, Tetsuya Yoshimura
Original Assignee Kazuyuki Imagawa, Yuji Takata, Hideaki Matsuo, Katsuhiro Iwasa, Tetsuya Yoshimura
Patent Citations (24), Referenced by (163), Classifications (38), Legal Events (6)
Method and device for media editing
US 20020031262 A1
The present media editing device generates media including messages in an easy manner in a communication terminal such as a mobile terminal. Therein, a moving image data storage part 14 stores moving image data recorded by a user. A region extraction part 17 extracts any region including the user from the moving image data. A front determination part 18 detects whether or not the user in the extracted region is facing the front. A sound detection part 19 detects the presence or absence of a sound signal of a predetermined level or higher. A frame selection part 20 determines starting and ending frames based on the results outputted from the front determination part 18 and the sound detection part 19. An editing part 21 performs, for example, an image conversion process by clipping out the media based on thus determined starting and ending frames. A transmission data storage part 15 stores the resultantly edited media as transmission data.
1. A media editing method for editing media including an image sequence comprised of a plurality of images showing a user partially or entirely as a subject, said method comprising the steps of:
extracting a region from said images including the user partially or entirely;
determining whether or not the user included in the region extracted in said extracting step is facing a predesignated direction;
selecting a part of the image sequence between time points determined as the user facing the predesignated direction in said determining step by scanning said image sequence from a start point to an end point, and from the end point to the start point; and
editing the media including the image sequence selected in said selecting step.
2. The media editing method according to claim 1, wherein said determining step determines whether or not the user included in the region extracted in said extracting step is facing the front.
3. The media editing method according to claim 1, further comprising the step of detecting a sound included in said media, and
said selecting step selects, by scanning the image sequence from the start point to the end point, and from the end point to the start point, the part of said image sequence satisfying as being between the time points determined in said determining step as the user facing the predesignated direction, and between time points at which a sound is each detected.
4. The media editing method according to claim 1, wherein said editing step specifies the image sequence selected in said selecting step by description in a meta-deta format.
5. The media editing method according to claim 1, wherein said editing step clips out the image sequence selected in said selecting step from said media.
6. The media editing method according to claim 1, wherein said editing step selects the first image in the image sequence selected in said selecting step as an initial display image.
7. The media editing method according to claim 1, wherein said editing step calculates a partial region corresponding to said image sequence based on a position and size of the region extracted in said extracting step, and performs editing by using said partial region.
8. The media editing method according to claim 6, wherein said editing step specifies said partial region by description in a meta data format.
9. The media editing method according to claim 1, wherein said editing step arranges a text included in said media onto an arrangement region which is so set as not to overlap at all the region extracted in said extracting step, or to overlap as little as possible if overlaps.
said extracting step extracts a face region including the user's face, and
said editing step arranges a text included in said media onto a speech bubble region which is so set as not to overlap at all the region extracted in said extracting step, or to overlap as little as possible if overlaps.
11. The media editing method according to claim 1, wherein said editing step performs an image conversion process for one or more of said images.
said editing step refers to a face characteristic calculated based on the face region extracted in said extracting step, and from a character database storing a plurality of character images and the face characteristic each corresponding thereto, selecting one or more of said character images.
13. The media editing method according to claim 11, wherein said editing step calculates said face characteristic based on character data inputted by the user.
14. The media editing method according to claim 11, wherein said editing step calculates said face characteristic based on a length-to-width ratio or a partial characteristic of a face in the face region extracted in said extracting step.
15. A media editing device for editing media including an image sequence comprised of a plurality of images showing a user partially or entirely as a subject, said device comprising:
a region extraction part for extracting a region from said images including the user partially or entirely;
a frame selection part for selecting a part of the image sequence between time points determined as the user facing the predesignated direction by said predesignated face orientation determination part by scanning said image sequence from a start point to an end point, and from the end point to the start point; and
16. The media editing device according to claim 15, further comprising a sound detection part for detecting a sound included in said media, and
said frame selection part selects, by scanning the image sequence from the start point to the end point, and from the end point to the start point, the part of said image sequence satisfying as being between the time points determined by said predesignated face orientation determination part as the user facing the predesignated direction, and between time points at which a sound is each detected.
17. The media editing device according to claim 15, wherein said editing part selects the first image in the image sequence selected by said frame selection part as an initial display image.
18. The media editing device according to claim 15, wherein said editing part calculates a partial region corresponding to said image sequence based on a position and size of the region extracted by said region extraction part, and performs editing by using said partial region.
19. The media editing device according to claim 15, wherein said editing part arranges a text included in said media onto an arrangement region which is so set as not to overlap at all the region extracted by said region extraction part, or to overlap as little as possible if overlaps.
20. The media editing device according to claim 20, wherein
said region extraction part extracts a face region including the user's face, and
said editing part refers to a face characteristic calculated based on the face region extracted by said region extraction part, and from a character database storing a plurality of character images and the face characteristic each corresponding thereto, selecting one or more of said character images.
21. A computer-readable recording medium on which a program is recorded to be carried out by a device for editing media including an image sequence comprised of a plurality of images showing a user partially or entirely as a subject, said program comprising the steps of:
There have been proposed a number of devices for once recording a sequence of events occurring at meetings, seminars, and interviews, communication over phones and videophones, images from televisions and monitor cameras, for later reproduction, by means of digital disks, digital still cameras, video tapes, or semiconductor memories, for example. The devices for such recording and reproduction have become popular as they are more reliable, than hand writing, for recording sound and image information.
With broadband communications that is recently widely available, information devices exemplarily including videophones, doorphones, and camera-equipped mobile terminals are now popularly used for person-to-person communication with sound and image information. For example, e-mails conventionally exchanged by text are now being replaced by videomails using sound and moving images. Also, with the widespread use of visualphones, messages left in answering machines so far recorded only by sound are now often accompanying video information. As such, simultaneous use of sound and moving images is now prevalent for the recent communication.
(1) Press a recording button provided on a recording device.
(2) Record whatever message.
(3) Lastly, press an end button.
In the following embodiments of the present invention, media denotes any message (or message data) for communication using still and moving images, for example.
(4) Determine which portion of the stored message to send, then clip out that portion for sending.
In the case that the stored message is a videomail to a friend, for example, the following step may be taken:
(5) Perform media editing to the message, including wallpapering, cartoon-like-character arranging, image cutting-out, and the like.
Among those steps, in step (4), when determining which portion of the message (that is, determining start and end points for clipping), the user has to playback and listen to the stored message. However, the user may find it difficult or impossible to do such clipping when using a camera-equipped mobile terminal, an answering machine, and the like.
Further, a predesignated face orientation determining step may determine whether or not the user is facing the front. A sound detection step may be also included to detect a sound included in the media. Moreover, the frame selecting step may select, by scanning the image sequence from the start point to the end point, and from the end point to the start point, the part of the image sequence satisfying as being between the time points determined in the determining step as the user facing the predesignated direction, and between time points at which a sound is each detected.
Further, the editing step may arrange a text included in the media onto an arrangement region or a speech bubble region which is so set as not to overlap at all the region extracted in the frame extracting step, or to overlap as little as possible if overlaps.
[0030]FIG. 1 is a block diagram showing the hardware structure of a media editing terminal capable of image communications realizing a media editing method of the present invention;
[0031]FIG. 2 is a block diagram showing the information flow and procedure of the processing at the time of media editing of the present invention;
[0032]FIG. 3 is a block diagram showing the functional structure of a media editing device according to a first embodiment;
[0033]FIG. 4 is a diagram for illustrating a clipping process applied to certain moving image data;
[0034]FIG. 5 is a diagram exemplarily showing meta data having index information of FIG. 4 described based on MPEG-7 standards;
[0035]FIG. 6 is a diagram showing an exemplary screen display of a terminal receiving a videomail which includes moving image data, and information (e.g., addresser, title);
[0036]FIG. 7 is a block diagram showing the functional structure of a media editing device according to a second embodiment;
[0037]FIG. 8 shows an exemplary trimming process and the resultant display screen;
[0038]FIG. 9 is a diagram showing exemplary meta-data Description for a partial region;
[0039]FIG. 10 shows an exemplary display screen showing only moving images with no space left for a title and a main text;
[0040]FIG. 11 shows an exemplary display screen where a title is arranged in a region not overlapping an image region including the user;
[0041]FIG. 12 shows an exemplary display screen where a main text is arranged in a region barely overlapping an image region including the user;
[0042]FIG. 13 is a diagram showing exemplary Description of meta data about a layout process of writing a text into moving images;
[0043]FIG. 14 shows an exemplary display image of a videomail on the receiver end having a character added;
[0044]FIG. 15 is a block diagram showing the functional structure of a media editing device according to a fourth embodiment;
[0045]FIG. 16 is a diagram exemplarily showing face characteristic values specifically focusing on the hair;
[0046]FIG. 17 is a diagram showing an exemplary editing screen for selecting which character to use;
[0047]FIG. 18 is a diagram showing an exemplary screen on the receiver end receiving a character mail;
[0048]FIG. 19 is a diagram showing another exemplary screen on the receiver end receiving a character mail; and
[0049]FIG. 20 is a block diagram showing the structure of a distributed-type media editing device (system).
[0053]FIG. 1 is a block diagram showing the hardware structure of a media editing terminal where image communications is carried out in such a manner as to realize the media editing method of the present invention. In FIG. 1, the present media editing terminal includes an input part 1, an image capturing part 2, an image display part 3, a sound input part 4, and a sound output part 5, all of which receive/provide information from/to the user. Further, included are an image-capturing control part 6, a sound input/output control part 7, a display control part 8, a communications part 9, a recording part 10, a recording control part 11, a signal processing part 12, and a control part 13, all of which process the information received/provided by the user. These constituents are interconnected via a system bus, an external bus, and the like. Here, the above structure is identical or similar to that of a general-type computer.
The sound output part 5 is composed of a speaker, and the like, and outputs, to the user, his/her recorded voice, received sound, and warning sound and beep operationally necessary, for example.
Here, the present media editing terminal may be of an integrated-type including every constituent mentioned above in one housing, or of a distributed-type performing data exchange among the constituents over a network or signal lines. For example, a camera-equipped mobile phone terminal is of the integrated-type carrying every constituent in a single housing. On the other hand, a doorphone is regarded as of the distributed-type because, at least, the image capturing part 2, the sound input part 4, and the sound output part 5 are externally located in the vicinity of the door, and the remains are placed in another housing located in the living room, for example. This is for establishing an interface with visitors. Alternatively, such a distributed-type device may have a character database (later described) located outside.
Generally, once the user creates a message in the form of videomail by his/her mobile terminal, he/she may have an itch to immediately send out the message. With the convenient interface provided, the user's such needs are thus met with a videomail created with a simple operation (e.g., one button operation). What is better, the resultant videomail layout is comprehensible to its addressee, having the message clipped at the beginning and end, the image trimmed to have the user centered, wallpaper and speech bubbles arranged as appropriate, for example. Herein, not all of the above processes are necessarily applied in the following embodiments, and combining any process needed for each different application will do. In the below, the embodiments of the present invention are individually described in detail.
A. Start data recording
B. Start first message
C. End first message
D. Pause before continuing message
E. Start second message
F. No sound (e.g., breathing)
G. End second message
H. End data recording
Here, as for the graph in the drawing, the lateral axis indicates a lapse of time, while the longitudinal axis an inputted sound level. Alphabetical letters A to H each indicate a predetermined time. Each of the cartoon sketches above the graph represents a scene in the image data, which is recorded simultaneously with the sound substantially at the predetermined time point (A to H). The cartoon sketches are exemplary of the user's behavior in the course of message recording through the doorphone before leaving the place.
There have been various methods for extracting regions including the subject. For example, disclosed in Japanese Patent Laid-Open Publication No. 5-91407 (1993-91407) is a method for first defining any part where movement change is small as a background, and extracting other regions as “subject regions”. Here, the movement change is determined based on relative comparison between video signals in any two adjoining frames of the moving images. Another method for extracting subject regions is disclosed in Japanese Patent Laid-Open Publication No. 5-161131 (1993-161131). In this method, any image showing only the background is retained in advance to use for finding and computing any difference from each frame of the moving images on a pixel basis. Herein, whatever region not so different from the background is regarded as a background region, and if the difference is conspicuous, the region is extracted as the subject region. As another method, to extract any specific part of the subject such as a head or a face, images are searched for any ellipse region. Such method is described in “Human Head Tracking using Adaptive Appearance Models with a Fixed-Viewpoint Pan-Tilt-Zoom Camera” by Yachi et al., MIRU2000, Meeting on Image Recognition and Understanding, pp. 9-14. There are various other known methods for detecting a face image based on color information, focusing on a specific part of the face such as eyes, a mouth, or the like, and a method based on template matching. Under these conventional methods, the region extraction part 17 can easily extract subject regions.
Next, as for the regions extracted by the region extraction part 17, the front determination part 18 detects whether or not the user therein is facing the front. For such detection, there have been various methods. As an example, a front image is previously prepared as a template for template matching. As another example, there is a method for identifying the face orientation in images by applying SVM (Support Vector Machine) which is a statistical feature-recognition technique. The method is described in “Head Classifier: A Real-time Face Classification System” by Baba et al., the 7th Symposium on Sensing via Image Information, pp. 411-416. Under such conventional methods, the front determination part 18 can determine whether or not the person in the image is facing the front. Also, with a designated face orientation determination part provided in place of the front determination part 18, determined is whether the user in the image region is facing a predetermined direction (e.g., 45 degrees to the right). With such structure, the user's best face orientation can be designated in advance for image selection.
Further, since the present media editing device performs both front determination and sound detection, clipping can be done with reliability to a part recorded as a message. Specifically, even if the user is facing the camera but deep in thought, clipping never miss a time point when he/she starts speaking. Here, the present media editing device can achieve almost the same effects without sound detection. This is because the user normally faces toward the camera to start message recording, and thus front determination sufficiently serves the purpose. Also, if the user utters in spite of his/her intention before starting message recording, sound detection may not be considered effective. Therefore, the sound detection part 19 may be omissible.
Next, the editing part 21 performs media (moving image data) clipping on the basis of the starting and ending frames determined by the frame selection part 20. Here, the resultant moving image data generated by the editing part 21 may include only the clipped portion and remains are all deleted, or the resultant data may be meta data including the clipped portion as an index. If the resultant data is meta data, no moving image data has been deleted, and thus any portion not clipped but important can be saved for later use. Exemplified below is a case where the meta data format is MPEG-7.
[0104]FIG. 5 is a diagram exemplarily showing meta data having index information of FIG. 4 described based on MPEG-7 standards. In the meta data of FIG. 5, “VideoSegmentDS” is used to interrelate points B and G to each corresponding image frame. Here, such interrelation to the actual image frames is established by “MediaTimePoint” of “MediaTime”, and resultantly described is the time of the corresponding VideoSegment. For example, the description of “T13:20:01:1F15” found in FIG. 5 means “the first frame at 13:20, 01 second (note that, 15 frames per second, from frames 0 to 14)”. With such description of meta data, reproduction control for selecting only a specific portion of the moving image data becomes possible.
As such, when the resultant data is meta data including a clipping portion as an index with no moving image data deleted, editing can be done without restraint if the data needs to be corrected after automatic clipping. This is because, unlike the case where the resultant data is moving image data including only the clipped portion, there needs to re-edit only the meta data.
In the above, the starting and ending frames provided by the frame selection part 20 are utilized for automatic clipping. Here, the starting frame may be defined as being an image appearing first on a terminal screen on the receiver end. In this sense, the clipping technique of the present media editing device is considered effective even better. To be more specific, assuming a case where the user first sees a still image (e.g., a preview image, thumbnail image) showing what moving images are coming or already in storage. Here, such a still image is now referred to as an initial display image. In the example of FIG. 4, the first frame image is the one at point A. However, the image at A shows the user not facing towards the camera, and it is not considered suitable for the initial display image such as a preview or a thumbnail image. Accordingly, by using the meta data as illustrated in FIG. 5, the starting frame is defined as the initial display image. As a result, the frame image at point B showing the user facing the front is suitably displayed as the initial display image. The present media editing device thus has no need to newly transmit a still image as the initial display image to the receiver end. If newly transmitting, the media editing device uses the region extraction part 17 and the front determination part 18 to scan the data from the start point to the end point. Point B is resultantly detected, and the frame image corresponding thereto is transmitted as the initial display image. In this manner, the image showing the user facing the front appropriately goes to the receiver end.
Described first is an assumable case in the present embodiment. Generally, any media to be transmitted in the form of videomail includes, not only moving image data, information about who has sent the moving images with what title, for example. FIG. 6 is a diagram showing an exemplary screen display of a terminal receiving such a videomail. As shown in FIG. 6, on a display image 100, displayed are a moving image section 104, a header section 101 exemplarily indicating who has sent the videomail to whom with what title, a text section 102, and a decoration section 103 having decorations appropriately laid out.
[0116]FIG. 7 is a block diagram showing the functional structure of the media editing device of the second embodiment. In FIG. 7, the present media editing device includes the moving image data storage part 14, the transmission data storage part 15, the region extraction part 17, a layout part 22, and a basic data storage part 23. These constituents perform partially or entirely the aforementioned layout process of FIG. 2.
In FIG. 7, the basic data storage part 23 corresponds to the recording part 10 of FIG. 1, and stored therein are such a text shown in FIG. 6, and basic data exemplified by image data for decoration. The layout part 22 reads, as appropriate, the basic data from the basic data storage part 23 by the user's operation, and performs the layout process including the trimming process. The details are left for later description.
[0119]FIG. 8 shows an exemplary trimming process and the resultant display screen. In FIG. 8, shown in the upper is the moving image section 104 received from the same addresser of FIG. 6. Due to the reasons described in the above, the section contains a high proportion of background region behind the user's image. Thus, only the user region is trimmed in the following manner for laying out.
[0124]FIG. 9 is a diagram showing exemplary meta-data Description for such a partial region. Description in FIG. 9 is, as is in the first embodiment, in MPEG-7 format. In this Description, “VideoSegmentDS” described in the first embodiment is applied to each frame, and the frames are each set by a partial region using “Still Region DS”. As for partial region information, “ContourShape” is used to describe the partial region in rectangular (the number of peaks is 4 in the drawing) and the coordinates thereof (not shown).
When the meta data is used as such, unlike newly generating moving image data by cutting out a partial region therefrom, the amount of the moving image data is not reduced. The user on the receiver end, however, can freely change the layout according to the size of the terminal screen or his/her preference. For example, the user can relocate the partial region on the image to suit his/her preference, or make settings to display any other partial region. In such cases also, settings as the partial region set by the layout part 22 initially appearing on the screen is considered convenient. This is because the region indicating who has sent the message is displayed first.
In MPEG-7, not only the method for setting “StillRegionDS” on a frame basis as shown in FIG. 9, “MovingRegionDS” being information about any moving region, or “AudioVisualRegionDS” being information about region with sound may be used. As a comprehensive basic definition thereof, there is “SegmentDS” indicating a part of the multimedia contents. With any DS based on this definition, Description equivalent to that of FIG. 9 can be done with less amount.
Described first is an assumable case in the present embodiment, specifically a case where the display image 100 of FIG. 6 is trimmed in such a manner that the moving image section 104 occupies a larger space as much as possible for display on a small screen (of mobile phone, for example). Here, presumably, information to be displayed on such a small screen is, at least, a “title”, a “text”, and moving images. Actually, the small screen is fully occupied only by the moving images, and there is no space left for the title and text. FIG. 10 shows an exemplary display screen showing only the moving images.
Here, the present media editing device is similar in structure to that of the second embodiment. To display such text information, however, the region extraction part 17 and the layout part 22 in the present media editing device are changed in their operations In detail, onto the image region including the user (the user's image region) that has been detected by the region extraction part 17, the layout part 22 arranges the text information (e.g., title, text) so as not to overlap at all, or to overlap as little as possible if overlaps. This operation is described in detail below.
First, the region extraction part 17 detects the user's image region in the moving image data, and calculates the position and size thereof. Then, the layout part 22 receives thus calculated position and size of the region, and the basic data (e.g., title, text) stored in the basic data storage part 23. The layout part 22 sets a region for arranging the basis data in the range not overlapping the user's image region at all (or overlapping as little as possible). FIG. 11 shows an exemplary display screen where a text title is arranged in a space not overlapping the user's image region. As shown in FIG. 11, the text title is arranged in a space above the user's head with no overlap. With such arrangement, the resultant layout can contain any needed text together with moving images occupying a sizable proportion.
The shape of the speech bubble region shown in FIG. 12 has, as quite familiar in cartoons, a sharp protrusion in the vicinity of the user's mouth. The position of the protrusion is calculated by an image recognition process. Specifically, the region extraction part 17 extracts a mouth region from the user's image region, and calculates its position. The layout part 22 arranges the protrusion onto thus calculated position (or proximal position considered appropriate), and then sets the speech bubble region in the range not overlapping the user's image region at all (or overlapping as little as possible) in consideration of the number of letters of the text.
The resultant layout image is preferably displayed on the screen as the initial image (aforementioned initial display image) on the receiver end. That is, when opening incoming mails, the addressee first sees the image of FIG. 11 or 12, and checks only the title or the main text therewith. If the main text does not fit in one page, a scrolling process may be applied, for example. As such, the receiver checks a main text, for example, only in the first display image but not while the moving images are reproduced. This is surely not restrictive, and the main text or the title may be superimposed and displayed during when the moving images are reproduced so that the receiver can read the text while hearing and seeing the message in the form of the moving images.
Next, the layout part 22 preferably generates meta data which is the deciding factor for what layout in the similar manner in the first and second embodiments. This is done to perform the layout process, that is, the process for writing a text into moving images.
[0139]FIG. 13 is a diagram showing exemplary Description of meta data about such a layout process. Description of FIG. 13 is, similar to the first and second embodiments, in MPEG-7 format. Based on a value of “MediaDuration”, that is, the length indicated by a predetermined point of the media, any sentence between “Text” tags is superimposed for display. As such, with Description of meta data, text display is enabled without the process for embedding text in moving images.
[0144]FIG. 15 is a block diagram showing the functional structure of the media editing device of the fourth embodiment. In FIG. 15, the present media editing device includes the moving image data storage part 14, the transmission data storage part 15, the region extraction part 17, the front determination part 18, an editing part 26, a character selection part 24, and a character database 25. These constituents carry out the layout process of FIG. 2 partially or entirely.
Described next is the operation of the media editing device of the present embodiment. The region extraction part 17 and the front determination part 18 operate in the similar manner to the first embodiment, and determine whether or not the user in the moving images is facing the front. The result is forwarded to the editing part 26, from which any image determined as being the front image is provided to the character selection part 24. Based on thus received image(s), the character selection part 24 selects one or more of potential characters from the character database 25, where various many characters are stored as a database. Then, a character ID each corresponding to thus selected character(s) are inputted into the editing part 26.
In order to select any potential character registered in the character database 25 with reference to the extracted face characteristic values, used may be the aforementioned characteristic representations, or correlation values calculated with respect to the registered face characteristic values. Here, if the correlation value exceeds a threshold value set for the potential character images considered suitable, the corresponding character image is extracted as a potential. The character selection part 24 then notifies the character ID corresponding to thus extracted potential character to the editing part 26.
[0155]FIG. 18 is a diagram showing an exemplary screen on the receiver end receiving the transmission data generated as such. As shown in FIG. 18, in the lower left of the screen, displayed is a character selected by the user (addresser), and in the lower right, a message in the form of moving images is displayed.
[0156]FIG. 19 is a diagram showing another exemplary screen on the receiver end receiving the transmission data. As shown in FIG. 19, displayed in the lower part of the screen is a character selected by the user (addresser). Here, during when the message in the form of moving images is reproduced, the character may not be displayed, and in the meantime, the moving images may take over its display position. Such a layout may be generated by the editing part 26, or set on the receiver end.
Here, the number of potential character to be selected may be one, and if this is the case, mail creation becomes easier without selecting any potential character.
In FIG. 20, such a distributed-type media editing device includes a character mail editing terminal 501, a character selection part 724, and a character database 725, which are interconnected over a network 600. Here, the character mail editing terminal 501 has the functions, partially or entirely, of the media editing devices of the first to third embodiments, and the character selection part 724 is located separately therefrom. Since this distributed-type media editing device is similar in structure and operation to the integrated-type, the same effects are to be achieved. Further, in the distributed-type media editing device of FIG. 20, in addition to the character mail editing terminal 501, the character selection part 724 and the character database 725 may be used also by a character mail reception terminal 502, or the like, where incoming mails are received and edited. If so, when receiving a character ID in an character mail, the character mail reception terminal 502 only needs to receive the corresponding character image from the character database 725. In such a structure, terminals do not have to carry data large in amount. Moreover, in the case that the character mail reception terminal 502 operates as the media editing device when returning mails, the character selection part 724 and the character database 725 can be shared.
As such, in the distributed-type media editing device, the character selection part 724 and the character database 725 can be shared by a plurality of users. Therefore, terminals have no need to include those constituents, and can use databases storing various many characters.
As is known from the above, in the present media editing device, the user can easily create a character mail with any preferred character added thereto by narrowing down various many characters based on front images extracted from moving images. Further, with such a character mail, person-to-person communication can be smooth and active.
US6661448 * Jun 26, 2001 Dec 9, 2003 Nokia Mobile Phones Ltd. Method and system for providing and transmitting alternative video data during interruptions in video transmissions
US6724417 * Nov 27, 2001 Apr 20, 2004 Applied Minds, Inc. Method and apparatus maintaining eye contact in video delivery systems using view morphing
US7440013 Aug 12, 2004 Oct 21, 2008 Fujifilm Corporation Image pickup device with facial region detector and method of synthesizing image including facial region
US7864198 Aug 4, 2006 Jan 4, 2011 Vodafone Group Plc. Image processing method, image processing device and mobile communication terminal
US8049915 * Aug 6, 2007 Nov 1, 2011 Canon Kabushiki Kaisha Image processing apparatus, image processing method and storage medium
US8467509 Sep 18, 2009 Jun 18, 2013 Canon Kabushiki Kaisha Video displaying apparatus, video displaying system and video displaying method
US8532462 Nov 16, 2010 Sep 10, 2013 Twentieth Century Fox Film Corporation Non-destructive file based mastering for multiple languages and versions
US8712219 * Dec 28, 2004 Apr 29, 2014 Core Wireless Licensing S.A.R.L. Method for editing a media clip in a mobile terminal device, a terminal device utilizing the method and program means for implementing the method
US9007633 * Mar 15, 2013 Apr 14, 2015 Fuji Xerox Co., Ltd. Systems and methods for efficiently capturing high-quality scans of multi-page documents with hand-held devices
US9230298 * Nov 11, 2010 Jan 5, 2016 Sony Corporation Information processing device, and information processing system
US9514512 Dec 18, 2013 Dec 6, 2016 Samsung Electronics Co., Ltd. Method and apparatus for laying out image using image recognition
US9723303 * Jun 28, 2011 Aug 1, 2017 Tektronix, Inc. System for generating text defined test patterns
US20050021625 * Jun 23, 2004 Jan 27, 2005 Matsushita Elec. Ind. Co.Ltd. Communication apparatus
US20050212822 * Jun 3, 2003 Sep 29, 2005 Yoshinori Honma Display device, display method and advertisement method using display device
US20080152188 * Aug 6, 2007 Jun 26, 2008 Canon Kabushiki Kaisha Image processing apparatus, image processing method and storage medium
US20110116764 * Nov 16, 2010 May 19, 2011 Twentieth Century Fox Film Corporation Non-destructive file based mastering for multiple languages and versions
US20120050542 * Jun 28, 2011 Mar 1, 2012 Tektronix, Inc. System for generating text defined test patterns
US20120287152 * Nov 11, 2010 Nov 15, 2012 Sony Computer Entertainment Inc. Information processing device, and information processing system
US20140268247 * Mar 15, 2013 Sep 18, 2014 Fuji Xerox Co., Ltd. Systems and Methods for Efficiently Capturing High-Quality Scans of Multi-Page Documents with Hand-Held Devices
CN103914280A * Jan 7, 2014 Jul 9, 2014 三星电子株式会社 Method and apparatus for laying out image using image recognition
EP1349407A2 * Mar 27, 2003 Oct 1, 2003 Casio Computer Co., Ltd. Data communication apparatus, data communication system, and data communication method
EP1349407A3 * Mar 27, 2003 Dec 6, 2006 Casio Computer Co., Ltd. Data communication apparatus, data communication system, and data communication method
EP1424839A1 Nov 19, 2003 Jun 2, 2004 Nec Corporation Method for generating graphics animations from still pictures in a cellular telephone and associated cellular telephone
EP1507386A1 Aug 11, 2004 Feb 16, 2005 Fuji Photo Film Co., Ltd. Image pickup device and image synthesizing method
EP1635569A2 * Sep 6, 2005 Mar 15, 2006 Nec Corporation Television telephone system, communication terminal device, character information transmission method used therefor, and program
EP1635569A3 * Sep 6, 2005 Sep 16, 2009 Nec Corporation Television telephone system, communication terminal device, character information transmission method used therefor, and program
EP1695551A1 * Oct 12, 2004 Aug 30, 2006 Nokia Corporation Transform-domain video editing
EP1695551A4 * Oct 12, 2004 Jun 13, 2007 Nokia Corp Transform-domain video editing
EP1713030A1 * Feb 7, 2005 Oct 18, 2006 Vodafone K.K. Image processing method, image processing apparatus, and mobile communication terminal apparatus
EP1713030A4 * Feb 7, 2005 May 2, 2007 Vodafone Kk Image processing method, image processing apparatus, and mobile communication terminal apparatus
EP1906405A2 Sep 21, 2007 Apr 2, 2008 Samsung Electronics Co., Ltd. Apparatus and method for managing multimedia content in mobile terminal
EP1906405A3 * Sep 21, 2007 Sep 3, 2008 Samsung Electronics Co., Ltd. Apparatus and method for managing multimedia content in mobile terminal
EP2753065A3 * Jan 2, 2014 Jan 7, 2015 Samsung Electronics Co., Ltd Method and apparatus for laying out image using image recognition
WO2005009044A1 * Jul 23, 2004 Jan 27, 2005 Casio Computer Co., Ltd. A method and device for configuring a multimedia message for presentation
WO2010109274A1 * Aug 25, 2009 Sep 30, 2010 Sony Ericsson Mobile Communications Ab Voice-controlled image editing
WO2012030965A2 * Aug 31, 2011 Mar 8, 2012 Fox Entertainment Group, Inc. Localized media content editing
WO2012030965A3 * Aug 31, 2011 Jul 5, 2012 Fox Entertainment Group, Inc. Localized media content editing
WO2016014273A1 * Jul 13, 2015 Jan 28, 2016 Google Inc. Management and presentation of notification content
U.S. Classification 382/190, 348/E07.078, 382/309
International Classification H04M3/53, H04M3/56, G06T5/00, H04N7/14, H04N1/387, G10L13/00, G06T7/60, H04M11/00, G06T11/60, H04N5/91, G06T3/00, G10L11/02
Cooperative Classification H04N21/4223, H04M3/5307, H04M2201/38, H04N21/4788, H04M3/567, H04W4/12, G11B27/28, H04N21/8543, H04N21/41407, H04N21/47205, H04W4/18, H04N2007/145, H04W88/02, H04N7/141, H04N21/4331
European Classification H04N21/8543, H04N21/4788, H04N21/433C, H04N21/414M, H04N21/4223, H04N21/472E, H04N7/14A, G11B27/28
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:IMAGAWA, KAZUYUKI;TAKATA, YUJI;MATSUO, HIDEAKI;AND OTHERS;REEL/FRAME:012164/0602