Patent ID: 12254561

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The following detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show illustrations in accordance with example embodiments. These example embodiments, which are also referred to herein as “examples,” are described in enough detail to enable those skilled in the art to practice the present subject matter. The embodiments can be combined, other embodiments can be utilized, or structural, logical and electrical changes can be made without departing from the scope of what is claimed. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope is defined by the appended claims and their equivalents.

In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one. In this document, the term “or” is used to refer to a nonexclusive “or,” such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. Furthermore, all publications, patents, and patent documents referred to in this document are incorporated by reference herein in their entirety, as though individually incorporated by reference. In the event of inconsistent usages between this document and those documents so incorporated by reference, the usage in the incorporated reference(s) should be considered supplementary to that of this document; for irreconcilable inconsistencies, the usage in this document controls.

The invention described herein enables a user40, normally a human user, to integrate his or her body measurements1, and detailed hair and face features, with one of a pre-existing set of avatars2maintained by the administrator of the system. In alternative embodiments, every frame of the avatar2is resized and every frame of the user's garments is resized, based upon the dimensions of the garments.

The administrator may be an employee or agent of a company that sells clothing and maintains a Website where the user40can interface with the system. The elements of the system that are managed by the administrator are maintained on one or more computers under the control of the administrator. In integrating the user body measurements1, including details of the user's face and hair, with the digitized avatar2, the user40can visually see how the company's clothing would look on the user40. The end product17of this invention, which is normally a video but can also be a set of still images, can be sold or licensed. Also, this invention can advantageously help the company sell its line of clothing to the user40and to other users not depicted herein.

As will be seen from the following description, the inputs to the inventive system are usually two-dimensional (2D) images or 2D images derived from 3D scenes, but the inputs may be 3D images. These input images are normally videos, such as scans of the user's face and hair, but may be one or more still images, such as 2D photos capturing the user's measurements1. Similarly, the final output product of the system is usually a set of two-dimensional (2D) images17, but they may be 3D images17. Images17normally comprise a video having multiple frames, but in certain embodiments, images17can simply be one or more still images.

As used in the following description, user40is deemed to be a female, but of course user40can also be a male.

As shown inFIG.4, user40takes a selfie41of herself using a cell phone or other camera, or otherwise inserts a photo41of herself into her cell phone or as an input to the Website maintained by the administrator. Photo41is converted by a conventional photo analyzing extractor42into a set of body measurements1of user40. The set of body measurements1may include a digitized representation of the user's face, hair, body, garments, shoes, accessories, jewelry, hats, eyeglasses, sunglasses, belts, purses, handbags, backpacks, etc. Extractor42does not have to be a neural network, but it can be a neural network. Alternatively to capturing measurements1by photographic means, the measurements1can be manually input to MSD (Measurement Space Deformation) module62. This alternative embodiment is not illustrated inFIG.4. Six measurements1give excellent results, with the most relevant measurements1normally being height, bra size, and dress size.

Independently of this user40input, a pre-existing avatar catalog61managed by the administrator of the system is accessed to select an avatar2that is suitable for use by user40. The particular avatar2may be selected by the user40or suggested by the administrator. The set of avatars61is more fully described in U.S. priority patent application Ser. No. 17/231,325 filed Apr. 15, 2021. The avatar images2may include at least one of garments, shoes, accessories, jewelry, hats, eyeglasses, sunglasses, belts, purses, handbags, and backpacks. The digitized avatar2may be a video representing successive positions of an avatar head during the playing of the avatar video. The avatar video2may represent a sequence representing desired hair motion during the playing of the final video17. The avatar2can be portrayed in any situation at all. For example, the avatar2may be in a vehicle such as a car, at a public event such as a basketball game, etc. The body measurements1and the selected avatar2are combined by MSD module62into alembic file65at method step21(seeFIG.1). Alembic file65includes representations of garments appropriate for the user's size and measurements. MSD module62is not a neural network, but a neural network can be used in lieu of MSD module62in certain embodiments.

All of files1,2, and65can be three-dimensional (3D). When alembic file65is 3D, it is usually converted into two dimensions by renderer66to produce a set of two-dimensional avatar images11as shown inFIGS.1and2. In some embodiments, renderer66outputs avatar images11having all three of the input dimensions. Renderer66can be a neural network, but is typically not a neural network. Rather, renderer66is typically an off-the-shelf game engine as commonly used by the video game industry, such as an Epic Games Unreal engine. The fact that items42,62, and66don't have to be neural networks is desirable, because neural networks have to be trained before being used, making them resource-intensive.

Avatar2is typically a brief (e.g., 15 second) video showing an individual in some action sequence (e.g., a ballerina on a dance floor, or a passenger on a yacht) that user40wishes to emulate as she pictures herself in clothing sold by the company, third party clothing (e.g., clothing of an influencer or name brand clothing), or fantasy clothing that is not being sold by anyone. The user40can select the video2from a pre-established catalog61of videos that are computer generated, real life videos, or a combination of computer generated and real life videos2. The invention thus enables users40to see themselves in scenes of their choice and to visualize their dreams.

With reference toFIG.5, as part of selfie41or, preferably, as a different photo51, user40takes a selfie51of just her face and hair, or otherwise enters a detailed photo51of her face and hair into the Website of the company, or into her cell phone based mobile software application. The face and hair images51may include at least one of jewelry, hats, eyeglasses, and sunglasses. Photo51is processed by a neural network52to produce an identity vector56. Neural network52may be, for example, similar or identical to neural network4,34,304, or404described in commonly owned U.S. patent application Ser. No. 17/559,437 filed Dec. 22, 2021, which patent application is hereby incorporated by reference in its entirety into the present patent application.

Identity vector56is then preferably combined with a destination vector113representing the illumination of the user's face and hair. This combination is performed by neural network53at step22(seeFIG.1) to produce the set of two-dimensional face and hair images12shown inFIGS.1and2. In those embodiments where the user's face and hair is processed as part of the user's overall body41, the set of images12includes two-dimensional representations of the entirety of the user's body11, including any garments, shoes, accessories, jewelry, hats, eyeglasses, sunglasses, belts, purses, handbags, backpacks, etc., and not just the user's face and hair51.

In some embodiments of the present invention, the functions of neural networks52and53can be performed by a single neural network.

Destination vector113is typically produced by starting with an image showing the desired illumination110that will appear in the final set of images17. This starting point110may be similar or identical to driver illumination sphere412described in the aforesaid U.S. patent application Ser. No. 17/559,437 filed Dec. 22, 2021. Illumination sphere110may include a representation of an image sequence and matte representing a boundary between head and neck, thereby facilitating accurate blending of image11-14inputs in the final converting step27. This desired illumination background110is usually processed by an illumination renderer111, such as an instance of the aforesaid Unreal engine, which produces a gray sphere112as an intermediate step prior to the formation of the destination vector113. The percentage of gray in gray sphere112is typically around 18%, but other percentages of gray can be used as desired. Vector113is now in the proper format to serve as an input to neural network53.

In certain embodiments, the skin tone of the user's face is separately extracted by neural network52to form a skin tone vector57, which is fed as an additional input to previously described renderer66. This technique gives a more realistic portrayal of the user's skin tone when the set of avatar images11is produced.

It is normally but not always desirable to have a set of background images13(such as a background video) as part of the set of final images17produced by the present invention. In these embodiments, it is preferable to separately pre-process a 3D background scene15. As shown inFIG.6, the background scene15is usually processed by renderer71, such as an instance of the previously described Unreal engine, to produce the set of two-dimensional background images13that is shown inFIGS.1and2.

In some embodiments, there is an optional three-dimensional foreground scene4that can also be integrated into the final product17. The foreground scene4represents images that will appear, in the final product17, to be in front of the avatar images11. In these embodiments, and as shown inFIG.7, the three-dimensional foreground scene4is usually processed by renderer72, such as an instance of the previously described Unreal engine, which produces the set of two-dimensional foreground images14that is shown inFIGS.1and2.

At method step26(seeFIG.1), the avatar images11, face/hair/body images12, optional background images13, and optional foreground images14are combined to form a set of two-dimensional or three-dimensional composite images16. These images16, which will ultimately become the set of final output images17, can be edited during step26. For example, the several images11-14can be combined using fade-outs, fade-ins, or other known cinematographic techniques. Step26advantageously does not require the use of a neural network, but a neural network can be fruitfully employed in some embodiments. During step26, the sets of images11-14are made to have the same size and resolution as each other, and are aligned with each other.

Finally, in step27, the composite images16undergo final processing to produce the end product: the set of final two-dimensional images17. Images17may be one or more still images, but are normally an output video comprising a plurality of frames. During this final conversion step27, an audio track and/or special effects can be added to the video, and the frames can be tailored to the particular computer to be employed by user40. The tailoring can be accomplished by, for example, cropping frames of the video17, and/or by adjusting the resolution to accommodate the user's computer monitor.

FIG.3shows a diagrammatic representation of a computing device or a machine in, for example, the electronic form of a computer system900, within which a set of instructions for causing the machine to perform any one or more of the methods discussed herein can be executed. In some embodiments, the machine operates as a standalone device, while in other embodiments it can be connected (e.g., networked) to other machines. In a networked deployment, the machine can operate in the capacity of a server, a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be or comprise a personal computer (PC), tablet PC, cellular telephone, web appliance, network router, switch, bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that separately or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

In an illustrative embodiment, the computer system900comprises at least one processor905(e.g., a central processing unit, a GPU, and so on, singly or in combination), and a memory, which in this example comprises a main memory910and a static memory915. The computer system900may further comprise a video display925, a signal generation device940(e.g., a speaker), and a network interface device945. In addition, the computer system900may comprise at least one input device930, such as an alphanumeric input device (e.g., a keyboard), a cursor control device (e.g., a mouse), a microphone, a digital camera, and so forth. Communication among the components may be accomplished via a bus920. Computer system900can be implemented in a variety of ways—for example, a speech processing system can be used to input the user's dimensions; and a holographic display, HD television monitor, or billboard display can be used in lieu of or in addition to video display unit925.

A drive unit935includes a computer-readable medium950, which stores one or more sets of instructions955and data embodying or utilized by any one or more of the methods or functions described herein. The instructions955can also reside, completely or at least partially, within the main memory910and/or within the processors905during execution thereof by the computer system900. The main memory910and the processors905also constitute machine-readable media. All the modules of the present invention can be embodied in an embedded system in an FPGA or SOC (system on a chip), with everything embedded in a single processor.

The instructions955can further be transmitted or received over a communications network960via the network interface device945utilizing any one of a number of well-known transfer protocols (e.g., Hyper Text Transfer Protocol (HTTP), Controller Area Network, Serial, and Modbus). The communications network960may include the Internet, local intranet, PAN, LAN, WAN, Metropolitan Area Network, VPN, a cellular network, Bluetooth radio, or an IEEE 802.9-based radio frequency network, and the like.

While the computer-readable medium950is shown in an example embodiment to be a single medium, the term “computer-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “computer-readable medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the machine and that causes the machine to perform any one or more of the methods of the present application, or that is capable of storing, encoding, or carrying data utilized by or associated with such a set of instructions. The term “computer-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media. Such media can also include, without limitation, hard disks, floppy disks, flash memory cards, digital video disks, random access memory, read only memory, and the like.

The example embodiments described herein can be implemented in an operating environment comprising computer-executable instructions installed on a computer, in software, in hardware, or in a combination of software and hardware. The computer can be a mobile (cell) phone. The computer-executable instructions can be written in a computer programming language or can be embodied in firmware logic. If written in a programming language conforming to a recognized standard, such instructions can be executed on a variety of hardware platforms and for interfaces to a variety of operating systems. Although not limited thereto, computer software programs for implementing the present method can be written utilizing any number of suitable programming languages such as, for example, HyperText Markup Language (HTML), Dynamic HTML, Extensible Markup Language, Extensible Stylesheet Language, Document Style Semantics and Specification Language, Cascading Style Sheets, Synchronized Multimedia Integration Language, Wireless Markup Language, Java™ Jini™, C, C++, C#, Go, .NET, Adobe Flash, Perl, UNIX Shell, Visual Basic or Visual Basic Script, Virtual Reality Markup Language, ColdFusion™ Objective-C, Scala, Clojure, Python, JavaScript, HTML5 or other compilers, assemblers, interpreters, or other computer languages or platforms, as one of ordinary skill in the art will recognize.

The above description is included to illustrate the operation of preferred embodiments, and is not meant to limit the scope of the invention. The scope of the invention is to be limited only by the following claims. From the above discussion, many variations will be apparent to one skilled in the art that would yet be encompassed by the spirit and scope of the present invention.