Patent Publication Number: US-2022237857-A1

Title: Producing a digital image representation of a body

Description:
RELATED APPLICATIONS 
     The present application is a continuation-in-part of commonly owned U.S. patent application Ser. No. 17/231,325 filed Apr. 15, 2021, entitled “Automatic Creation of a Photorealistic Customized Animated Garmented Avatar” (attorney docket SPRE 9716 US), which claims the priority benefit of U.S. provisional patent application 63/142,294 filed on Jan. 27, 2021, entitled “Automatic Creation of a Photorealistic Customized Animated Garmented Avatar for a Person” (attorney docket SPRE 9716 PR); both of said previous patent applications are hereby incorporated in their entireties into the present patent application. 
    
    
     TECHNICAL FIELD 
     The present invention pertains to the field of using a digitized avatar and measurements of a user to produce a photorealistic digital image representation of the user, e.g., a human that may be adorned in garments, shoes, accessories, jewelry, hats, eyeglasses, sunglasses, belts, purses, handbags, backpacks, etc. 
     BACKGROUND ART 
     Previous attempts to integrate avatars with digitized representations of users have too often been clumsy, and have introduced unwanted artifacts into the final composite image. The present invention solves many of the problems associated with the prior art. 
     DISCLOSURE OF INVENTION 
     Methods and apparati for using an avatar ( 2 ) to help produce a digital image representation ( 17 ) of a body ( 1 ). A method embodiment of the present invention comprises the steps of capturing ( 21 ) measurements of the body ( 1 ) and combining the measurements ( 1 ) with an avatar ( 2 ) to produce a set of avatar images ( 11 ); invoking ( 22 ) a neural network ( 3 ) to generate a set of photorealistic face and hair images ( 12 ); combining ( 26 ) the avatar images ( 11 ) and the face and hair images ( 12 ) to produce a set of composite images ( 16 ); and converting ( 27 ) the set of composite images ( 16 ) into a set of final images ( 17 ) representing the body ( 1 ). 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       These and other more detailed and specific objects and features of the present invention are more fully disclosed in the following specification, reference being had to the accompanying drawings, in which: 
         FIG. 1  is a flowchart diagram illustrating the overall method of the present invention. 
         FIG. 2  is a block diagram corresponding to the  FIG. 1  embodiment. 
         FIG. 3  is a diagrammatic representation of a typical computer that is capable of executing the method steps of the present invention. 
         FIG. 4  is a block diagram illustrating the conversion of user  40  body measurements  1  into the set of avatar images  11  that are shown in  FIGS. 1 and 2 . 
         FIG. 5  is a block diagram illustrating the conversion of face/hair measurements  51  into the set of face/hair images  12  that are shown in  FIGS. 1 and 2 . 
         FIG. 6  is a block diagram showing the conversion of background scene  15  into the set of background images  13  that are shown in  FIGS. 1 and 2 . 
         FIG. 7  is a block diagram showing the conversion of foreground scene  4  into the set of foreground images  14  that are shown in  FIGS. 1 and 2 . 
     
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
     The following detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show illustrations in accordance with example embodiments. These example embodiments, which are also referred to herein as “examples,” are described in enough detail to enable those skilled in the art to practice the present subject matter. The embodiments can be combined, other embodiments can be utilized, or structural, logical and electrical changes can be made without departing from the scope of what is claimed. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope is defined by the appended claims and their equivalents. 
     In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one. In this document, the term “or” is used to refer to a nonexclusive “or,” such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. Furthermore, all publications, patents, and patent documents referred to in this document are incorporated by reference herein in their entirety, as though individually incorporated by reference. In the event of inconsistent usages between this document and those documents so incorporated by reference, the usage in the incorporated reference(s) should be considered supplementary to that of this document; for irreconcilable inconsistencies, the usage in this document controls. 
     The invention described herein enables a user  40 , normally a human user, to integrate his or her body measurements  1 , and detailed hair and face features, with one of a pre-existing set of avatars  2  maintained by the administrator of the system. In alternative embodiments, every frame of the avatar  2  is resized and every frame of the user&#39;s garments is resized, based upon the dimensions of the garments. 
     The administrator may be an employee or agent of a company that sells clothing and maintains a Website where the user  40  can interface with the system. The elements of the system that are managed by the administrator are maintained on one or more computers under the control of the administrator. In integrating the user body measurements  1 , including details of the user&#39;s face and hair, with the digitized avatar  2 , the user  40  can visually see how the company&#39;s clothing would look on the user  40 . The end product  17  of this invention, which is normally a video but can also be a set of still images, can be sold or licensed. Also, this invention can advantageously help the company sell its line of clothing to the user  40  and to other users not depicted herein. 
     As will be seen from the following description, the inputs to the inventive system are usually two-dimensional (2D) images or 2D images derived from 3D scenes, but the inputs may be 3D images. These input images are normally videos, such as scans of the user&#39;s face and hair, but may be one or more still images, such as 2D photos capturing the user&#39;s measurements  1 . Similarly, the final output product of the system is usually a set of two-dimensional (2D) images  17 , but they may be 3D images  17 . Images  17  normally comprise a video having multiple frames, but in certain embodiments, images  17  can simply be one or more still images. 
     As used in the following description, user  40  is deemed to be a female, but of course user  40  can also be a male. 
     As shown in  FIG. 4 , user  40  takes a selfie  41  of herself using a cell phone or other camera, or otherwise inserts a photo  41  of herself into her cell phone or as an input to the Website maintained by the administrator. Photo  41  is converted by a conventional photo analyzing extractor  42  into a set of body measurements  1  of user  40 . The set of body measurements  1  may include a digitized representation of the user&#39;s face, hair, body, garments, shoes, accessories, jewelry, hats, eyeglasses, sunglasses, belts, purses, handbags, backpacks, etc. Extractor  42  does not have to be a neural network, but it can be a neural network. Alternatively to capturing measurements  1  by photographic means, the measurements  1  can be manually input to MSD (Measurement Space Deformation) module  62 . This alternative embodiment is not illustrated in  FIG. 4 . Six measurements  1  give excellent results, with the most relevant measurements  1  normally being height, bra size, and dress size. 
     Independently of this user  40  input, a pre-existing avatar catalog  61  managed by the administrator of the system is accessed to select an avatar  2  that is suitable for use by user  40 . The particular avatar  2  may be selected by the user  40  or suggested by the administrator. The set of avatars  61  is more fully described in U.S. priority patent application Ser. No. 17/231,325 filed Apr. 15, 2021. The avatar images  2  may include at least one of garments, shoes, accessories, jewelry, hats, eyeglasses, sunglasses, belts, purses, handbags, and backpacks. The digitized avatar  2  may be a video representing successive positions of an avatar head during the playing of the avatar video. The avatar video  2  may represent a sequence representing desired hair motion during the playing of the final video  17 . The avatar  2  can be portrayed in any situation at all. For example, the avatar  2  may be in a vehicle such as a car, at a public event such as a basketball game, etc. The body measurements  1  and the selected avatar  2  are combined by MSD module  62  into alembic file  65  at method step  21  (see  FIG. 1 ). Alembic file  65  includes representations of garments appropriate for the user&#39;s size and measurements. MSD module  62  is not a neural network, but a neural network can be used in lieu of MSD module  62  in certain embodiments. 
     All of files  1 ,  2 , and  65  can be three-dimensional (3D). When alembic file  65  is 3D, it is usually converted into two dimensions by renderer  66  to produce a set of two-dimensional avatar images  11  as shown in  FIGS. 1 and 2 . In some embodiments, renderer  66  outputs avatar images  11  having all three of the input dimensions. Renderer  66  can be a neural network, but is typically not a neural network. Rather, renderer  66  is typically an off-the-shelf game engine as commonly used by the video game industry, such as an Epic Games Unreal engine. The fact that items  42 ,  62 , and  66  don&#39;t have to be neural networks is desirable, because neural networks have to be trained before being used, making them resource-intensive. 
     Avatar  2  is typically a brief (e.g., 15 second) video showing an individual in some action sequence (e.g., a ballerina on a dance floor, or a passenger on a yacht) that user  40  wishes to emulate as she pictures herself in clothing sold by the company, third party clothing (e.g., clothing of an influencer or name brand clothing), or fantasy clothing that is not being sold by anyone. The user  40  can select the video  2  from a pre-established catalog  61  of videos that are computer generated, real life videos, or a combination of computer generated and real life videos  2 . The invention thus enables users  40  to see themselves in scenes of their choice and to visualize their dreams. 
     With reference to  FIG. 5 , as part of selfie  41  or, preferably, as a different photo  51 , user  40  takes a selfie  51  of just her face and hair, or otherwise enters a detailed photo  51  of her face and hair into the Website of the company, or into her cell phone based mobile software application. The face and hair images  51  may include at least one of jewelry, hats, eyeglasses, and sunglasses. Photo  51  is processed by a neural network  52  to produce an identity vector  56 . Neural network  52  may be, for example, similar or identical to neural network  4 ,  34 ,  304 , or  404  described in commonly owned U.S. patent application Ser. No. 17/559,437 filed Dec. 22, 2021, which patent application is hereby incorporated by reference in its entirety into the present patent application. 
     Identity vector  56  is then preferably combined with a destination vector  113  representing the illumination of the user&#39;s face and hair. This combination is performed by neural network  53  at step  22  (see  FIG. 1 ) to produce the set of two-dimensional face and hair images  12  shown in  FIGS. 1 and 2 . In those embodiments where the user&#39;s face and hair is processed as part of the user&#39;s overall body  41 , the set of images  12  includes two-dimensional representations of the entirety of the user&#39;s body  11 , including any garments, shoes, accessories, jewelry, hats, eyeglasses, sunglasses, belts, purses, handbags, backpacks, etc., and not just the user&#39;s face and hair  51 . 
     In some embodiments of the present invention, the functions of neural networks  52  and  53  can be performed by a single neural network. 
     Destination vector  113  is typically produced by starting with an image showing the desired illumination  110  that will appear in the final set of images  17 . This starting point  110  may be similar or identical to driver illumination sphere  412  described in the aforesaid U.S. patent application Ser. No. 17/559,437 filed Dec. 22, 2021. Illumination sphere  110  may include a representation of an image sequence and matte representing a boundary between head and neck, thereby facilitating accurate blending of image  11 - 14  inputs in the final converting step  27 . This desired illumination background  110  is usually processed by an illumination renderer  111 , such as an instance of the aforesaid Unreal engine, which produces a gray sphere  112  as an intermediate step prior to the formation of the destination vector  113 . The percentage of gray in gray sphere  112  is typically around 18%, but other percentages of gray can be used as desired. Vector  113  is now in the proper format to serve as an input to neural network  53 . 
     In certain embodiments, the skin tone of the user&#39;s face is separately extracted by neural network  52  to form a skin tone vector  57 , which is fed as an additional input to previously described renderer  66 . This technique gives a more realistic portrayal of the user&#39;s skin tone when the set of avatar images  11  is produced. 
     It is normally but not always desirable to have a set of background images  13  (such as a background video) as part of the set of final images  17  produced by the present invention. In these embodiments, it is preferable to separately pre-process a 3D background scene  15 . As shown in  FIG. 6 , the background scene  15  is usually processed by renderer  71 , such as an instance of the previously described Unreal engine, to produce the set of two-dimensional background images  13  that is shown in  FIGS. 1 and 2 . 
     In some embodiments, there is an optional three-dimensional foreground scene  4  that can also be integrated into the final product  17 . The foreground scene  4  represents images that will appear, in the final product  17 , to be in front of the avatar images  11 . In these embodiments, and as shown in  FIG. 7 , the three-dimensional foreground scene  4  is usually processed by renderer  72 , such as an instance of the previously described Unreal engine, which produces the set of two-dimensional foreground images  14  that is shown in  FIGS. 1 and 2 . 
     At method step  26  (see  FIG. 1 ), the avatar images  11 , face/hair/body images  12 , optional background images  13 , and optional foreground images  14  are combined to form a set of two-dimensional or three-dimensional composite images  16 . These images  16 , which will ultimately become the set of final output images  17 , can be edited during step  26 . For example, the several images  11 - 14  can be combined using fade-outs, fade-ins, or other known cinematographic techniques. Step  26  advantageously does not require the use of a neural network, but a neural network can be fruitfully employed in some embodiments. During step  26 , the sets of images  11 - 14  are made to have the same size and resolution as each other, and are aligned with each other. 
     Finally, in step  27 , the composite images  16  undergo final processing to produce the end product: the set of final two-dimensional images  17 . Images  17  may be one or more still images, but are normally an output video comprising a plurality of frames. During this final conversion step  27 , an audio track and/or special effects can be added to the video, and the frames can be tailored to the particular computer to be employed by user  40 . The tailoring can be accomplished by, for example, cropping frames of the video  17 , and/or by adjusting the resolution to accommodate the user&#39;s computer monitor. 
       FIG. 3  shows a diagrammatic representation of a computing device or a machine in, for example, the electronic form of a computer system  900 , within which a set of instructions for causing the machine to perform any one or more of the methods discussed herein can be executed. In some embodiments, the machine operates as a standalone device, while in other embodiments it can be connected (e.g., networked) to other machines. In a networked deployment, the machine can operate in the capacity of a server, a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be or comprise a personal computer (PC), tablet PC, cellular telephone, web appliance, network router, switch, bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that separately or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein. 
     In an illustrative embodiment, the computer system  900  comprises at least one processor  905  (e.g., a central processing unit, a GPU, and so on, singly or in combination), and a memory, which in this example comprises a main memory  910  and a static memory  915 . The computer system  900  may further comprise a video display  925 , a signal generation device  940  (e.g., a speaker), and a network interface device  945 . In addition, the computer system  900  may comprise at least one input device  930 , such as an alphanumeric input device (e.g., a keyboard), a cursor control device (e.g., a mouse), a microphone, a digital camera, and so forth. Communication among the components may be accomplished via a bus  920 . Computer system  900  can be implemented in a variety of ways—for example, a speech processing system can be used to input the user&#39;s dimensions; and a holographic display, HD television monitor, or billboard display can be used in lieu of or in addition to video display unit  925 . 
     A drive unit  935  includes a computer-readable medium  950 , which stores one or more sets of instructions  955  and data embodying or utilized by any one or more of the methods or functions described herein. The instructions  955  can also reside, completely or at least partially, within the main memory  910  and/or within the processors  905  during execution thereof by the computer system  900 . The main memory  910  and the processors  905  also constitute machine-readable media. All the modules of the present invention can be embodied in an embedded system in an FPGA or SOC (system on a chip), with everything embedded in a single processor. 
     The instructions  955  can further be transmitted or received over a communications network  960  via the network interface device  945  utilizing any one of a number of well-known transfer protocols (e.g., Hyper Text Transfer Protocol (HTTP), Controller Area Network, Serial, and Modbus). The communications network  960  may include the Internet, local intranet, PAN, LAN, WAN, Metropolitan Area Network, VPN, a cellular network, Bluetooth radio, or an IEEE 802.9-based radio frequency network, and the like. 
     While the computer-readable medium  950  is shown in an example embodiment to be a single medium, the term “computer-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “computer-readable medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the machine and that causes the machine to perform any one or more of the methods of the present application, or that is capable of storing, encoding, or carrying data utilized by or associated with such a set of instructions. The term “computer-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media. Such media can also include, without limitation, hard disks, floppy disks, flash memory cards, digital video disks, random access memory, read only memory, and the like. 
     The example embodiments described herein can be implemented in an operating environment comprising computer-executable instructions installed on a computer, in software, in hardware, or in a combination of software and hardware. The computer can be a mobile (cell) phone. The computer-executable instructions can be written in a computer programming language or can be embodied in firmware logic. If written in a programming language conforming to a recognized standard, such instructions can be executed on a variety of hardware platforms and for interfaces to a variety of operating systems. Although not limited thereto, computer software programs for implementing the present method can be written utilizing any number of suitable programming languages such as, for example, HyperText Markup Language (HTML), Dynamic HTML, Extensible Markup Language, Extensible Stylesheet Language, Document Style Semantics and Specification Language, Cascading Style Sheets, Synchronized Multimedia Integration Language, Wireless Markup Language, Java™ Jini™, C, C++, C#, Go, .NET, Adobe Flash, Perl, UNIX Shell, Visual Basic or Visual Basic Script, Virtual Reality Markup Language, ColdFusion™ Objective-C, Scala, Clojure, Python, JavaScript, HTML5 or other compilers, assemblers, interpreters, or other computer languages or platforms, as one of ordinary skill in the art will recognize. 
     The above description is included to illustrate the operation of preferred embodiments, and is not meant to limit the scope of the invention. The scope of the invention is to be limited only by the following claims. From the above discussion, many variations will be apparent to one skilled in the art that would yet be encompassed by the spirit and scope of the present invention.