Source: http://www.google.com/patents/US20040207720?ie=ISO-8859-1
Timestamp: 2014-12-29 13:27:56
Document Index: 687899117

Matched Legal Cases: ['art 106', 'art 110', 'art 101', 'art 101', 'art 101', 'art 102', 'art 102', 'art 101', 'art 102', 'art 101', 'art 101', 'art 102', 'art 105', 'art 105', 'art 102', 'art 105', 'art 107', 'art 103', 'art 103', 'art 104', 'art 104', 'art 103', 'art 104', 'art 109', 'art 109', 'art 104', 'art 109', 'art 107', 'art 106', 'art 107', 'art 107', 'art 107', 'art 102', 'art 105', 'art 107', 'art 104', 'art 107', 'art 107', 'art 107', 'art 107', 'art 107', 'art 107', 'art 110', 'art 110', 'art 110', 'art 107', 'art 106', 'art 107', 'art 107', 'art 108', 'art 108', 'art 107', 'art 101', 'art 101', 'art 102', 'art 102', 'art 101', 'art 102', 'art 105', 'art 103', 'art 103', 'art 104', 'art 104', 'art 103', 'art 104', 'art 109', 'art 105', 'art 102', 'art 105', 'art 107', 'art 109', 'art 104', 'art 109', 'art 104', 'art 109', 'art 107', 'art 107', 'art 107', 'art 107', 'art 107', 'art 107', 'art 107', 'art 106', 'art 107', 'art 107', 'art 108', 'art 108', 'art 107', 'art 110', 'art 101', 'art 102', 'art 103', 'art 104', 'art 105', 'art 106', 'art 107', 'art 108', 'art 109']

Patent US20040207720 - Face information transmission system - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inAdvanced Patent SearchPatentsAn object is to provide a face information transmission system enabling the generation of images with a high possibility of reflecting such elements as the feelings and intentions of a user. A mobile phone, as such a face information transmission system, comprises an image acquisition part to acquire...http://www.google.com/patents/US20040207720?utm_source=gb-gplus-sharePatent US20040207720 - Face information transmission systemAdvanced Patent SearchPublication numberUS20040207720 A1Publication typeApplicationApplication numberUS 10/748,248Publication dateOct 21, 2004Filing dateDec 31, 2003Priority dateJan 31, 2003Also published asCN1237799C, CN1527602A, DE602004001887D1, DE602004001887T2, EP1443465A1, EP1443465B1, US7224367Publication number10748248, 748248, US 2004/0207720 A1, US 2004/207720 A1, US 20040207720 A1, US 20040207720A1, US 2004207720 A1, US 2004207720A1, US-A1-20040207720, US-A1-2004207720, US2004/0207720A1, US2004/207720A1, US20040207720 A1, US20040207720A1, US2004207720 A1, US2004207720A1InventorsSaori Miyahara, Akira Matsuki, Yukiko TozawaOriginal AssigneeNtt Docomo, Inc.Export CitationBiBTeX, EndNote, RefManReferenced by (4), Classifications (25), Legal Events (2) External Links: USPTO, USPTO Assignment, EspacenetFace information transmission systemUS 20040207720 A1Abstract An object is to provide a face information transmission system enabling the generation of images with a high possibility of reflecting such elements as the feelings and intentions of a user. A mobile phone, as such a face information transmission system, comprises an image acquisition part to acquire images of a face of a subject; a first generation part to generate first image information relating to the subject and including the positions of characteristic points of the face based on the acquired images; a second generation part to generate a character image according to the facial expression of the face of the subject based on the generated first image information; and a transmission part to transmit the generated character image to another mobile phone. Images(8) Claims(7)
BRIEF DESCRIPTION OF THE DRAWINGS [0015] The present invention may be more readily described with reference to the accompanying drawings, in which: [0016]FIG. 1 is a drawing to explain the face information transmission system which is an aspect of this invention; [0017]FIG. 2A shows an example of a face image acquired by a face information transmission system which is an aspect of this invention; [0018]FIG. 2B shows an example of a face image acquired by a face information transmission system which is an aspect of this invention; [0019]FIG. 3A shows an example of a character image generated by a face information transmission system which is an aspect of this invention; [0020]FIG. 3B shows an example of a character image generated by a face information transmission system which is an aspect of this invention; [0021]FIG. 4 shows an example of information stored in the character information storage part of FIG. 1; [0022]FIG. 5 is a flowchart showing the method of transmission of character images used by a face information transmission system which is an aspect of this invention; [0023]FIG. 6 is a flowchart showing the method of transmission of character images used by a face information transmission system which is an aspect of this invention; and, [0024]FIG. 7 is a drawing used to explain a face information transmission program which is an aspect of this invention. DESCRIPTION OF THE PREFERRED EMBODIMENTS [0025] Information of this invention can easily be understood through consideration of the following detailed description while referring to the attached drawings, which are provided only for illustration. Aspects of this invention are then explained, referring to the attached drawings. Where possible, the same components are assigned the same symbols, and redundant explanations are omitted. [0026] An explanation is given, using FIG. 1, for a mobile phone (face information transmission system) which is an aspect of this invention. FIG. 1 is a drawing to explain the mobile phone 10. The mobile phone 10 is configured to enable mutual communication of information with another mobile phone (communication terminal) 30 via a network 20. [0027] Next, the mobile phone 10 is explained. The mobile phone 10 physically is configured as a mobile phone capable of information communication and comprising a CPU (central processing unit), memory, input devices such as buttons and a microphone, a display device such as a liquid crystal display, an image acquisition device such as a camera, and similar. [0028] The mobile phone 10 comprises, as functional components, an image acquisition part (image acquisition means) 101; first generation part (first generation means) 102; utterance acquisition part (utterance acquisition means) 103; phoneme identification part (phoneme identification means) 104; image judgment part (image judgment means) 105; interrupt input part 106; second generation part (second generation means) 107; transmission part (transmission means) 108; phoneme judgment part (phoneme judgment means) 109; and character information storage part 110. Next, each of the components is explained in detail. [0029] The image acquisition part 101 is a part which acquires images of the face of the user, as the subject, of the mobile phone 10. The images of a face acquired by the image acquisition part 101 may be instantaneous (a static image), or may be along a time series (moving images or video). The image acquisition part 101 outputs acquired images to the first generation part 102. [0030] The first generation part 102 generates first image information identifying the positions of characteristic points of the face, based on an image acquired and output by the image acquisition part 101. More specifically, as shown in FIG. 2A, the first generation part 102 identifies characteristic points 401 identifying the eyes and eyebrows, and characteristic points 402 identifying the mouth and nose, in a face image 403 of the subject contained in an image 40 output by the image acquisition part 101, and generates the face image 403 and characteristic points 401, 402 as first image information of a still image. When images output by the image acquisition part 101 are moving images, the image 40 shown in FIG. 2A and an image 40 a after a prescribed time has elapsed are received. As shown in FIG. 2B, the image 40 a includes the face image 403 a having moved during the lapse of a prescribed length of time, and the characteristic points 401 a and 402 a are identified in the face image 403 a. Hence first image information in the case of moving images includes the face image 403 and characteristic points 401, 402, and the face image 403 a and characteristic points 401 a, 402 a. The first generation part 102 outputs the first image information thus generated to the image judgment part 105. [0031] The image judgment part 105 is a part which judges whether the first image information output by the first generation part 102 satisfies prescribed conditions. The prescribed conditions may be set appropriately using such factors as the circumstances and desires of the user using the mobile phone 10, or may be set appropriately according to requirements of the hardware of the mobile phone 10. As prescribed conditions, for example, inability to acquire a majority of the characteristic points contained in the first image information, or substantial deviation of the distribution of characteristic points in the face image, may be set as conditions. The image judgment part 105 outputs the result of judgment as to whether the first image information satisfies the prescribed conditions, together with the first image information, to the second generation part 107. [0032] The utterance acquisition part 103 is a part which acquires utterances issued by the user, as the subject, of the mobile phone 10. The utterance acquisition part 103 outputs acquired utterances to the phoneme identification part 104. [0033] The phoneme identification part 104 is a part which identifies phonemes corresponding to utterances acquired and output by the utterance acquisition part 103. A phoneme is the smallest unit of sound which may affect the meaning [of an utterance]. For example, if an output utterance is �konnnichiwa� (�hello�)), then phonemes are identified as �ko (h)�, �n (ε)�, �ni (l)�, �chi (o)�, �wa (u)�. The phoneme identification part 104 outputs an utterance and identified phonemes to the phoneme judgment part 109. [0034] The phoneme judgment part 109 is a part which judges whether a phoneme output by the phoneme identification part 104 satisfies prescribed conditions. The prescribed conditions may be set appropriately using such factors as the circumstances and desires of the user using the mobile phone 10, or may be set appropriately according to requirements of the hardware of the mobile phone 10. As prescribed conditions, for example, whether a phoneme can be identified or not may be set as a condition. The phoneme judgment part 109 outputs the results of judgment as to whether a phoneme satisfies the prescribed conditions, together with the utterance, to the second generation part 107. [0035] The interrupt input part 106 receives an interrupt instruction input by the user of the mobile phone 10, and outputs the interrupt instruction to the second generation part 107. More specifically, when the user presses buttons to which are assigned instructions relating to such facial expressions as �laugh�, �cry�, or �be surprised�, the corresponding instruction is input, and is output to the second generation part 107. [0036] The second generation part 107 is a part which generates a character image (second image information) according to the facial expression of the face of the subject, based on the first image information generated by the first generation part 102. When the image judgment part 105 judges that the first image information satisfies the prescribed conditions, the second generation part 107 generates the character image according to the facial expression of the face of the subject based on, at least, the first image information; and when the first image information does not satisfy the prescribed conditions, but a phoneme identified by the phoneme identification part 104 satsfies prescribed conditions, [the second generation part 107] generates the character image according to the facial expression of the subject based on that phoneme. [0037] When both the first image information and the phoneme satisfy the respective prescribed conditions, the second generation part 107 generates the character image based on the first image information and the phoneme. For example, in a case in which in the first image information only the inclination of the face can be acquired, when a phoneme has been identified, the two are used in a complementary manner to generate the character image. When neither the first image information nor the phoneme satisfy the respective prescribed conditions, the second generation part 107 generates a character image based on whether or not there is an utterance. For example, when an utterance exceeds a prescribed threshold, it is assumed that the user is speaking, and so an image of a speaking character is generated. Further, when neither the first image information nor the phoneme satisfy the respective prescribed conditions, and an utterance has not been acquired, image information stored in advance may be used as the character image. [0038] When the second generation part 107 generates a character image based on first image information, when for example [image generation] is based on the face image 403 and characteristic points 401, 402 shown in FIG. 2A, the positions of the characteristic points 401, 402 relative to the face image 403 are identified as a distribution. Based on this distribution, the second generation part 107 determines the positions of the characteristic points 501, 502 relative to the character face image 503 as shown in FIG. 3A, and generates a still character image 50. When the first image information corresponds to a moving image, that is, when [image generation] is based on the face image 403 and characteristic points 401, 402 shown in FIG. 2A and on the face image 403 a and characteristic points 401 a, 402 a after the lapse of a prescribed length of time as shown in FIG. 2B, in addition to the character image 50 shown in FIG. 3A, a character image 50 a is generated based on the character face image 503 a and characteristic points 501 a, 502 a shown in FIG. 3B. [0039] When the second generation part 107 generates a character image based on a phoneme, information stored in the character information storage part 110 is utilized. FIG. 4 shows an example of information stored in the character information storage part 110. According to the example shown in FIG. 4, a �phoneme�, �characteristic point data�, and a �character image� are stored in association in the character information storage part 110. The second generation part 107 extracts the �characteristic point data� and �character image� corresponding to each �phoneme� and generates a character image as a still image or as a moving image. In the example shown in FIG. 4, an image in the mouth area is displayed; but the correspondence may be with an image of the entire face. Further, images incorporating peculiarities of the user may be stored as �character images�. A �character image� incorporating peculiarities of the user are not limited to those based on �phonemes�, and may be applied to cases in which character images are generated based on �first image information� or on �utterances� as well. [0040] When an interrupt instruction is output from the interrupt input part 106, the second generation part 107 changes the character image according to the interrupt instruction. For example, when a �laugh� interrupt instruction is output, the generated character image is changed so as to assume a laughing facial expression. The second generation part 107 outputs the generated or changed character image to the transmission part 108. [0041] The transmission part 108 transmits the character image generated by the second generation part 107 to the mobile phone 30. [0042] Next, a method of transmission of a character image using the mobile phone 10 is explained, using the flowcharts of FIG. 5 and FIG. 6. The flowcharts of FIG. 5 and FIG. 6 form a series. The user inputs to the mobile phone 10 an instruction to create and transmit a character image. (step S01). In response to this input instruction, the operations of the steps S02, S03 and the steps S04, S05 explained next are performed in parallel. [0043] The image acquisition part 101 acquires an image of the face of the user, as the subject, of the mobile phone 10 (step S02). The image acquisition part 101 outputs the acquired image to the first generation part 102. The first generation part 102 generates first image information, identifying the positions of characteristic points of the face, based on the image acquired and output by the image acquisition part 101 (step S03). The first generation part 102 outputs the generated first image information to the image judgment part 105. [0044] The utterance acquisition part 103 is a part which acquires utterances issued by the user, as the subject, of the mobile phone 10 (step S04). The utterance acquisition part 103 outputs an acquired utterance to the phoneme identification part 104. The phoneme identification part 104 identifies phonemes corresponding to an utterance acquired and output by the utterance acquisition part 103 (step S05). The phoneme identification part 104 outputs the utterance and identified phonemes to the phoneme judgment part 109. [0045] The image judgment part 105 judges whether first image information output by the first generation part 102 satisfies prescribed conditions (step S06). The image judgment part 105 outputs to the second generation part 107 the judgement result as to whether the first image information satisfies the prescribed conditions, together with the first image information. [0046] The phoneme judgment part 109 judges whether a phoneme output by the phoneme identification part 104 satisfies prescribed conditions (steps S07, S08). Also, the phoneme judgment part 109 judges whether an utterance output by the phoneme identification part 104 is the state of a substantial utterance exceeding a prescribed threshold (step S09). The phoneme judgment part 109 outputs to the second generation part 107 the result of judgment as to whether the phoneme satisfies prescribed condition, the result of judgment as to whether the utterance exceeds the prescribed threshold, the utterance, and the phoneme. [0047] When the first image information satisfies the prescribed conditions, and the phoneme also satisfies the prescribed conditions (from step S06 to step S07), the second generation part 107 generates a character image according to the facial expression of the face of the subject, based on the first image information and the phoneme (step S10). [0048] When the first image information satisfies the prescribed conditions, and the phoneme does not satisfy the prescribed conditions (from step S06 to step S07), the second generation part 107 generates a character image according to the facial expression of the face of the subject, based on the first image information (step S11). [0049] When the first image information does not satisfy the prescribed conditions, but the phoneme satisfies the prescribed conditions (from step S06 to step S08), the second generation part 107 generates a character image according to the facial expression of the face of the subject, based on the phoneme (step S12). [0050] When the first image information does not satisfy the prescribed conditions, and the phoneme also does not satisfy the prescribed conditions, but the utterance exceeds the prescribed threshold (step S06, and from step S08 to step S09), the second generation part 107 generates a character image according to the facial expression of the face of the subject, based on the utterance (step S13). [0051] When the first image information does not satisfy the prescribed conditions, and the phoneme also does not satisfy the prescribed conditions, and the utterance does not exceed the prescribed threshold (step S06, and from step S08 to step S09), the second generation part 107 generates a character image based on default information stored in advance (step S14). [0052] The second generation part 107 judges whether an interrupt instruction has been output from the interrupt input part 106 (step S15). When an interrupt instruction has been output, the second generation part 107 changes the character image according to the interrupt instruction (step S16). The second generation part 107 outputs the generated or changed character image to the transmission part 108. The transmission part 108 transmits the character image generated by the second generation part 107 to the mobile phone 30 (step S17). [0053] Next, a face information transmission program 92, which causes a computer capable of information communication and comprising a CPU (central processing unit), memory, input devices such as buttons and a microphone, a display device such as a liquid crystal display, an image acquisition device such as a camera, and similar to function as the mobile phone 10 of this aspect, as well as computer-readable recording media 9 on which [the program] is recorded, are explained. FIG. 7 shows the configuration of recording media 9 on which is recorded a face information transmission program 92. The recording media 9 may be, for example, a magnetic disk, optical disc, CD-ROM, or memory incorporated in a computer. [0054] As shown in FIG. 7, the recording media 9 comprises a program area 91 which records the program, and a data area 93 which records data. In the data area 93 is stored a character information database 931 similar to the character information storage part 110 explained using FIG. 1. [0055] The face information transmission program 92 is recorded in the program area 91. The face information transmission program 92 comprises a main module 921 which supervises processing; an image acquisition module 922; a first generation module 923; an utterance acquisition module 924; a phoneme identification module 925; an image judgment module 926; an interrupt input module 927; a second generation module 928; a transmission module 929; and a phoneme judgment module 930. Here, the functions realized through operation of the image acquisition module 922, first generation module 923, utterance acquisition module 924, phoneme identification module 925, image judgment module 926, interrupt input module 927, second generation module 928, transmission module 929, and phoneme judgment module 930 are similar to the respective functions of the image acquisition part 101, first generation part 102, utterance acquisition part 103, phoneme identification part 104, image judgment part 105, interrupt input part 106, second generation part 107, transmission part 108, and phoneme judgment part 109 of the above mobile phone 10. [0056] In this aspect, a character image is generated according to the facial expression of the face of the subject, based on first image information generated based on the positions of characteristic points; hence image information which captures the facial expression of the face of the subject can be generated. A facial expression often reflects such elements as the feelings and intentions of the subject, and a character image can be generated as an image reflecting such elements and transmitted to the mobile phone 30. [0057] In this aspect, when first image information does not satisfy prescribed conditions, a character image is generated according to the facial expression of the face of the subject based on a phoneme, so that even if for some reason measurement of the positions of characteristic points of the face of the subject is incomplete, a character image can still be generated. [0058] In this aspect, first image information includes information identifying the distribution of characteristic points in the face of the subject; hence the relative positional relationships of characteristic points can be grasped, and a more appropriate character image can be generated. [0059] In this aspect, first image information is generated based on displacements in the positions of characteristic points measured along a time series, so that changes in facial expression of the face of the subject can be grasped as changes in the positions of characteristic points. Hence character images can be generated according to changes in facial expression. [0060] In this aspect, first image information includes information identifying the movement of characteristic points relative to the face of the subject, so that movement of characteristic points relative to the face can be identified along a time series, and changes in facial expression of the face of the subject can be grasped more accurately. Referenced byCiting PatentFiling datePublication dateApplicantTitleUS7280041 *Jun 17, 2005Oct 9, 2007Lg Electronics Inc.Method of communicating and disclosing feelings of mobile terminal user and communication system thereofUS7693306Feb 23, 2006Apr 6, 2010Konami Gaming, Inc.System and method for capturing images from mobile devices for use with patron tracking systemUS8306824Aug 26, 2009Nov 6, 2012Samsung Electronics Co., Ltd.Method and apparatus for creating face character based on voiceUS8565392Nov 9, 2011Oct 22, 2013WOOW Inc. LimitedApparatus and method for physical interaction over a distance using a telecommunication device* Cited by examinerClassifications U.S. Classification348/14.02, 375/E07.025, 455/566, 704/E21.02International ClassificationG06T13/80, G06T13/00, H04N21/439, H04N21/442, G10L15/00, G10L17/26, H04N21/431, G10L15/10, H04N7/14, G06T1/00, G06K9/00, G06T7/20, H04N1/387, H04B1/40Cooperative ClassificationH04N21/8106, G10L2021/105, H04N21/41407, G06K9/00268European ClassificationH04N21/81A, H04N21/414M, G06K9/00F2Legal EventsDateCodeEventDescriptionOct 28, 2010FPAYFee paymentYear of fee payment: 4Jun 24, 2004ASAssignmentOwner name: NTT DOCOMO, INC., JAPANFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MIYAHARA, SAORI;MATSUKI, AKIRA;TOZAWA, YUKIKO;REEL/FRAME:015505/0616;SIGNING DATES FROM 20031215 TO 20031222RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services©2012 Google