Patent Description:
There are two main implementations of virtual fittings at present: 3D modeling-based virtual fittings and 2D image-based virtual fittings. The 3D virtual fittings require acquiring 3D information of human bodies and clothing, and then performing patch synthesis according to target people, which can realize displays at any angles, but generally the devices are expensive and the effects are different from that of daily images. Acquiring 3D information of the human bodies and the clothing is generally costly and cumbersome. The 2D image-based virtual fittings, emerging in recent years, use pure images to synthesize the target images. However, such technology is now commonly used for experiments at small image resolutions and is not practical temporarily. At present, the generation technology of the 2D image-based virtual fittings generally performs poorly in detail features such as human hands. <NPL> proposes an image-based Virtual Try-On Network, where a desired clothing item is transferred onto the corresponding region of a person using a coarse-to-fine strategy. A coarse sample is first generated with a multi-task encoder-decoded conditioned on a detailed clothing-agnostic person representation. The coarse results are further enhanced with a refinement network which learns the optimal composition. <NPL> proposes a virtual Try-On network, M2E-Try On Ney, where clothes are transferred from a model image to a person image without needing a clean product image. Three components are proposed to solve the challenges of: <NUM>) non-rigid nature of clothes; <NUM>) richness in textures of fashion items; and <NUM>) variation of identity appearances. These components are: a pose alignment network (PAN), a texture refinement network (TRN), and a fitting network (FIN).

Embodiments of the present disclosure provide a method and apparatus for visual fitting.

According to the method and apparatus for a virtual fitting provided in the embodiments of the present disclosure, compared with 3D modeling-based virtual fittings, 2D image-based virtual fittings has the advantages such as low cost, concise processes, broad coverage on clothing categories, and the generated image is more like a daily photo; compared with present 2D image-based virtual fittings, has the advantages such as supporting any image resolution and better expression ability on fine body details.

By reading the detailed description of non-limiting embodiments with reference to the following accompanying drawings, other features, objects and advantages of the present disclosure will become more apparent.

Embodiments of the present disclosure will be further described below in detail in combination with the accompanying drawings. It should be appreciated that detailed embodiments described herein are merely used for explaining the relevant invention, rather than limiting the invention. In addition, it should be noted that, for the ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

It should be noted that embodiments in the present disclosure and the features in embodiments may be combined with each other on a non-conflict basis. Embodiments of the present disclosure will be described below in detail with reference to the accompanying drawings.

<FIG> shows an example architecture <NUM> to which a method or apparatus for a visual fitting of the present disclosure may be applied.

As shown in <FIG>, the system architecture <NUM> may include terminal device(s) <NUM>, <NUM>, <NUM>, a network <NUM> and a server <NUM>. The network <NUM> serves as a medium for providing a communication link between the terminal device(s) <NUM>, <NUM>, <NUM> and the server <NUM>. The network <NUM> may include various types of connections, such as wired or wireless communication links, or optical fiber cables.

A user may use the terminal device(s) <NUM>, <NUM>, <NUM> to interact with the server <NUM> through the network <NUM> to receive or send messages. Various communication client applications, such as visual fitting applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, may be installed on the terminal device(s) <NUM>, <NUM>, <NUM>.

The terminal device(s) <NUM>, <NUM>, <NUM> may be various electronic devices having a display screen and supporting web browsing, including but not limited to, a smart phone, a tablet computer, an electronic book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop portable computer and a desktop computer.

The server <NUM> may be a server providing various services, such as a background server of a visual fitting providing support for a visual fitting webpage displayed on the terminal device(s) <NUM>, <NUM>, <NUM>. The background server of the visual fitting may perform analysis processing on received data, such as a fitting request, and feedback a processing result (such as an effect diagram of a visual fitting) to the terminal devices.

It should be noted that the method for a virtual fitting provided by the embodiments of the present disclosure is generally executed by the server <NUM>. Correspondingly, the apparatus for a virtual fitting is generally provided in the server <NUM>.

It should be appreciated that the number of the terminal devices, the network and the server in <FIG> is merely illustrative. Any number of terminal devices, networks and servers may be provided according to actual requirements.

Further referring to <FIG> shows a flow <NUM> of a method for virtual fitting according to an embodiment of the present disclosure. The method for a virtual fitting includes steps <NUM> to <NUM>.

Step <NUM>, receiving a fitting request including a model picture and a user image.

In this embodiment, an execution body of the method for virtual fitting (such as the server <NUM> shown in <FIG>) may receive, through a wired or wireless connection, the fitting request from a terminal using which a user performs visual fitting. The fitting request includes the model picture and the user image. The user image (or a video, the video will be processed frame by frame according to a process of an image) is selected as Image1, and the model picture corresponding to a piece of clothing that the user wants to try on is selected as Image2.

Step <NUM>, performing human body positioning analysis and surface coordinate analysis on the model picture and the user image respectively.

In this embodiment, the human body positioning analysis and the surface coordinate analysis are performed on the pictures in step <NUM> respectively, so that to obtain the human body positioning result and the surface coordinates of the model picture and the human body positioning result and the surface coordinates of the user image respectively. The body posture of the user, such the right arm being bent to pinch the waist and the left arm being drooped, can be determined through the human body positioning result. Herein, a densepose algorithm is used, which divides a human body into <NUM> parts, each part having a corresponding UV coordinate, and this coordinate information may be used to transfer surface texture as shown in <FIG>. Alternatively, other algorithms may be used for performing the human body positioning analysis and the surface coordinate analysis. Alternatively, a human body may be divided into more parts as required, thereby reflecting more details.

Step <NUM>, performing clothing segmentation on the model picture and the user image respectively.

In this embodiment, the clothing segmentation may be performed on the model whose clothing interests the user in the model picture, and a person in the user image who is going to try the clothing on respectively, by using the CE2P algorithm. To accommodate the fitting task, we divided a two-dimensional human body image into <NUM> categories: a hat, a pair of gloves, a pair of glasses, a blouse, a dress, a coat, a pair of socks, a pair of pants, a scarf, a skirt, a left shoe, a right shoe, hair, a face, a neck, a left hand, a right hand, a left leg, and a right leg. Other pixels are all classified into a background category. The algorithm classifies clothes into a number of clothing categories to facilitate selection of a different clothing category for transfer, and also includes exposed parts of a human skin, and segmenting pixels of these human parts helps restore details of the human body in the user images.

Step <NUM>, covering, based on results of the clothing segmentation and results of the surface coordinate analysis, pixels corresponding to a piece of clothing in the model picture to corresponding positions in the user image to obtain a synthesized image and to-be-completed information.

In this embodiment, during selecting a piece of clothing for texture transfer, a segmentation result (obtained by the CE2P algorithm) corresponding to the piece of clothing in the model picture is selected, and in combination with the surface coordinate information obtained by the densepose algorithm, the pixels corresponding to the piece of clothing are covered to the corresponding positions in the user image. At the same time, since the task postures in the model picture and the user image are usually different, the original clothes in the original model picture and the user image may be not the same (for example, one is a short sleeve and the other is a long sleeve), and the exposed parts of the model and person in the two 2D images are also different, the clothing texture cannot be completely and correspondingly transferred. In order to make the final synthesized image as true as possible, before this step of transferring the clothing texture, we need to erase pixels corresponding to a piece of clothing in the user image, and at the same time, erase parts that may be covered by the new piece of clothing (for example, when a blouse is changed, an exposed arm of the person in the user image will be erased). If the texture transferred from the model picture is not enough to cover all the erased pixels, a defect area will appear, thereby obtaining the to-be-completed information.

In some alternative implementations of this embodiment, the fitting request further includes a type of a target piece of clothing, and before the covering pixels corresponding to a piece of clothing in the model picture to the corresponding positions in the user image, the method further includes: determining, based on the type of the target piece of clothing and a result of the human body positioning analysis on the user image, a to-be-erased area in the user image; and erasing, based on the to-be-erased area, a part in the user image, the part being probably covered by the target piece of clothing. For example, the to-be-erased area is the upper part of the person in the user image when a blouse is to-be-changed, and after the upper part of the person in the user image is erased, the texture of the blouse in the model picture is transferred to the upper part of the person in the user image. Since the image after the transfer may be not complete, defect pixels in the user image may be determined based on information such as the positioning information and the erased area. These defect pixels constitute "mask", which is used to mark which position of the user image needs to be completed.

Step <NUM>, inputting the synthesized image, a result of the human body positioning analysis on the user image and the to-be-completed information into a pre-trained image completion network to obtain a completed image.

In this embodiment, the image after the texture transfer in step <NUM> is not complete, and in order to maintain consistency, some parts in the original user image are erased while the texture transfer fails to completely fill these parts, and the image obtained in this way is generally as shown in <FIG>. The incomplete image and the corresponding mask are input to a pre-trained image completion network to obtain the completed image. The image completion network may employ an existing generative adversarial network, such as "inpainting".

In some alternative implementations of this embodiment, the method further includes: fusing a result of a clothing analysis on the user image with the completed image to obtain a fused image. Since the ability of the generator of the image completion network is limited, it is difficult to restore the complicated human body parts such as the face and the fingers at the same time as the clothing and the limbs are completed. Therefore, when finally the fusion is performed, the result of the segmentation of the CE2P algorithm will be took into account, and the details such as the face, the hair, and the fingers are extracted from the original image and fused into the image, so that a vivid image can be obtained.

In some alternative implementations of this embodiment, the method further includes: embedding, in combination with information of the human body positioning analysis on the user image, the fused image in step <NUM> into the original user image for outputting.

<FIG> are schematic diagrams of an application scenario of the method for a virtual fitting according to the present disclosure. As shown in <FIG>, after the positioning, the surface coordinate analysis (as shown in <FIG>), and the clothing segmentation are respectively performed on the model picture and the user image, the clothing texture in the model picture is transferred to the user image, and then the effect diagram of the fitting is obtained after the image is completed and fused.

According to the method provided by the above embodiment of the present disclosure, by combining the most advanced densepose algorithm and the CE2P algorithm the image-level virtual fitting is achieved based on the 2D image, and has characteristics of low cost and wild application range. At the same time, in combination with the self-developed completion algorithm for a human body, an image of any resolution can be processed. This solution can be applied to a virtual fitting room of an e-commerce website, which can be experienced by the user without pre-collecting clothing data. Alternatively, this solution can be applied to photo editing software or only used as an entertainment, and has a strong practical value.

Further referring to <FIG> shows a flow <NUM> of a method for training an image completion network, and the flow <NUM> of the method for training an image completion network include steps <NUM> to <NUM>.

Step <NUM>, acquiring a sample image set.

In this embodiment, in order to train the image completion network for a human body image, model images from a large number of e-commerce websites may be collected as the sample image set. The image completion network generally follows the framework of GAN (Generative Adversarial Networks). The structures of the generation network and the discriminant network are as shown in <FIG>, are both fully convolution structures, and support any resolution. At the same time, in order to support batch training, all sample images are randomly cut into square inputs of 384x384 pixels during the training.

Step <NUM>,for a sample image in the sample image set, erasing a part in the sample image randomly, the part being probably covered by a piece of clothing, to obtain to-be-completed information of the sample image.

In this embodiment, a part in the sample image that may be probably covered by a piece of clothing is randomly erased to generate "mask".

Step <NUM>, for the sample image in the sample image set, performing human body positioning analysis on the sample image to obtain posture information in the sample image.

In this embodiment, the human body positioning analysis is performed on the sample image by using, for example, the densepose algorithm in step <NUM>, to obtain the posture information in the sample image, and then the analyzed posture of the human body is used as an input to guide the generation of the human body.

Step <NUM>, using an image obtained after the erasing, the to-be-completed information of the sample image, and the posture information in the sample image in the sample image set as inputs, and using the sample image before the erasing as an output, to perform a machine learning training to obtain the image completion network.

In this embodiment, the image completion network may adopt the structure as shown in <FIG>. The training process is as follows: the sample images are selected from the sample image set, and then the training steps are executed:.

In some alternative implementations of this embodiment, the image completion network includes a generation network and a discriminant network, the generation network includes a gated convolution layer, a residual block, a self-attention layer, and a transposed convolution layer, and the discriminant network includes a spectral normalization convolution layer.

Further referring to <FIG>, as an implementation of the method shown in each of the above figures, an embodiment of the present disclosure provides an apparatus for a visual fitting. The embodiment of the apparatus corresponds to the embodiment of the method shown in <FIG>, and the apparatus is particularly applicable to various electronic devices.

As shown in <FIG>, the apparatus <NUM> for a visual fitting provided by this embodiment includes: a receiving unit <NUM>, a positioning unit <NUM>, a segmenting unit <NUM>, a synthesizing unit <NUM> and a completing unit <NUM>. The receiving unit <NUM> is configured to receive a fitting request including a model picture and a user image; the positioning unit <NUM> is configured to perform human body positioning analysis and surface coordinate analysis on the model picture and the user image respectively; the segmenting unit <NUM> is configured to perform clothing segmentation on the model picture and the user image respectively; the synthesizing unit <NUM> is configured to cover, based on results of the clothing segmentation and results of the surface coordinate analysis, pixels corresponding to a piece of clothing in the model picture to corresponding positions in the user image to obtain a synthesized image and to-be-completed information; and the completing unit <NUM> is configured to input the synthesized image, a result of the human body positioning analysis on the user image and the to-be-completed information into a pre-trained image completion network to obtain a completed image.

In this embodiment, the specific processing of the receiving unit <NUM>, the positioning unit <NUM>, the segmenting unit <NUM>, the synthesizing unit <NUM> and the completing unit <NUM> of the apparatus <NUM> for a visual fitting may be described with reference to steps <NUM> to <NUM> in the embodiment corresponding to <FIG> respectively.

In some optional implementations of this embodiment, the apparatus <NUM> further includes a fusing unit (not shown in the accompanying drawings), configured to fuse a result of a clothing analysis on the user image with the completed image to obtain a fused image.

In some optional implementations of this embodiment, the apparatus <NUM> further includes an output unit (not shown in the accompanying drawings), configured to embed, based on a result of the human body positioning analysis on the user image, the fused image into the original user image for outputting.

In some optional implementations of this embodiment, the fitting request further includes a type of a target piece of clothing, and the apparatus <NUM> further includes an erasing unit (not shown in the accompanying drawings), configured to: determine, based on the type of the target piece of clothing and a result of the human body positioning analysis on the user image, a to-be-erased area in the user image, before covering the pixels corresponding to the piece of clothing in the model picture to the corresponding position in the user image; and erase, based on the to-be-erased area, a part in the user image, the part being probably covered by the target piece of clothing.

In some optional implementations of this embodiment, the apparatus <NUM> further includes a training unit (not shown in the accompanying drawings), configured to: acquire a sample image set; for a sample image in the sample image set, erase a part in the sample image randomly, the part being probably covered by a piece of clothing, to obtain to-be-completed information of the sample image; for the sample image in the sample image set, perform the human body positioning analysis on the sample image to obtain posture information in the sample image; and use an image obtained after the erasing, the to-be-completed information in the sample image, and the posture information of the sample image in the sample image set as inputs, and use the sample image before the erasing as an output, to perform a machine learning training to obtain the image completion network.

In some optional implementations of this embodiment, the image completion network includes: a generation network and a discriminant network, the generation network includes a gated convolution layer, a residual block, a self-attention layer, and a transposed convolution layer, and the discriminant network includes a spectral normalization convolution layer.

Referring to <FIG> shows a schematic structural diagram of an electronic device <NUM> (such as the server in <FIG>) adapted to implement embodiments of the present disclosure. The server shown in <FIG> is merely an example and should not be construed as limiting the functionality and usage scope of embodiments of the present disclosure.

As shown in <FIG>, the electronic device <NUM> may include a processing apparatus <NUM> (such as a central processing unit and a graphic processor), which may execute various appropriate actions and processes in accordance with a program stored in a read-only memory (ROM) <NUM> or a program loaded into a random access memory (RAM) <NUM> from a storage apparatus <NUM>. The RAM <NUM> also stores various programs and data required by operations of the electronic device <NUM>. The processing apparatus <NUM>, the ROM <NUM> and the RAM <NUM> are connected to each other through a bus <NUM>. An input/output (I/O) interface <NUM> is also connected to the bus <NUM>.

Generally, the following apparatuses are connected to the I/O interface <NUM>: an input apparatus <NUM> including a touch screen, a touchpad, a keyboard, a mouse a camera, a microphone, an accelerometer, a gyroscope and the like; an output apparatus <NUM> including a liquid crystal display (LCD), a speaker, a vibrator and the like; a storage apparatus <NUM> including a magnetic tap, a hard disk and the like; and a communication apparatus <NUM>. The communication apparatus <NUM> may allow the electronic device <NUM> to perform wireless or wired communication with other devices to exchange data. Although <FIG> shows the electronic device <NUM> having various apparatuses, it should be appreciated that it is not required to implement or provide all the shown apparatuses, and it may alternatively be implemented or provided with more or fewer apparatuses. Each block shown in <FIG> may represent one apparatus or multiple apparatuses according to requirements.

In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer readable medium. The computer program includes program codes for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication apparatus <NUM>, or may be installed from the storage apparatus <NUM>, or may be installed from the ROM <NUM>. The computer program, when executed by the processing apparatus <NUM>, implements the above functionalities as defined by the method of some embodiments of the present disclosure. It should be noted that the computer readable medium described by some embodiments of the present disclosure may be computer readable signal medium or computer readable storage medium or any combination of the above two. The computer readable storage medium may be, but is not limited to: an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, an apparatus, an element, or any combination of the above. A more specific example of the computer readable storage medium may include but is not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a fibre, a portable compact disk read only memory (CD-ROM), an optical memory, a magnet memory or any suitable combination of the above. In some embodiments of the present disclosure, the computer readable storage medium may be any physical medium containing or storing programs which can be used by or in combination with an instruction execution system, an apparatus or an element. In some embodiments of the present disclosure, the computer readable signal medium may include a data signal in the base band or propagating as a part of a carrier, in which computer readable program codes are carried. The propagating signal may be various forms, including but not limited to: an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium may be any computer readable medium except for the computer readable storage medium. The computer readable signal medium is capable of transmitting, propagating or transferring programs for use by or in combination with an instruction execution system, an apparatus or an element. The program codes contained on the computer readable medium may be transmitted with any suitable medium including but not limited to: a wire, an optical cable, RF (Radio Frequency), or any suitable combination of the above.

The above computer readable medium may be included in the electronic device; or may alternatively be present alone and not assembled into the electronic device. The computer readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: receive a fitting request including a model picture and a user image; perform human body positioning analysis and surface coordinate analysis on the model picture and the user image respectively; perform clothing segmentation on the model picture and the user image respectively; cover, based on results of the clothing segmentation and results of the surface coordinate analysis, pixels corresponding to a piece of clothing in the model picture to corresponding positions in the user image to obtain a synthesized image and to-be-completed information; and input the synthesized image, a result of the human body positioning analysis on the user image and the to-be-completed information into a pre-trained image completion network to obtain a completed image.

A computer program code for executing operations of some embodiments of the present disclosure may be written in one or more programming languages or a combination thereof. The programming languages include object-oriented programming languages, such as Java, Smalltalk or C++, and also include conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be completely executed on a user computer, partially executed on a user computer, executed as a separate software package, partially executed on a user computer and partially executed on a remote computer, or completely executed on a remote computer or server. In a case involving a remote computer, the remote computer may be connected to a user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, connected through Internet using an Internet service provider).

The flowcharts and block diagrams in the accompanying drawings show architectures, functions and operations that may be implemented according to the systems, methods and computer program products of the various embodiments of the present disclosure. In this regard, each of the blocks in the flowcharts or block diagrams may represent a module, a program segment, or a code portion, the module, program segment, or code portion including one or more executable instructions for implementing specified logic functions. It should also be noted that, in some alternative implementations, the functions denoted by the blocks may occur in a sequence different from the sequences shown in the figures. For example, any two blocks presented in succession may be executed, substantially in parallel, or they may sometimes be in a reverse sequence, depending on the function involved. It should also be noted that each block in the block diagrams and/or flowcharts as well as a combination of blocks in the block diagrams and/or flowcharts may be implemented using a dedicated hardware-based system executing specified functions or operations, or by a combination of a dedicated hardware and computer instructions.

Claim 1:
A computer-implemented method for virtual fitting, comprising:
receiving (<NUM>) a fitting request comprising a model picture and a user image;
performing (<NUM>) human body positioning analysis and surface coordinate analysis on the model picture and the user image respectively;
performing (<NUM>) clothing segmentation on the model picture and the user image respectively;
covering (<NUM>), based on results of the clothing segmentation and results of the surface coordinate analysis, corresponding positions in the user image with pixels corresponding to a piece of clothing in the model image to obtain a synthesized image and to-be-completed information, wherein the to-be-completed information refers to defect pixels that need to be completed; and
inputting (<NUM>) the synthesized image, a result of the human body positioning analysis on the user image and the to-be-completed information into a pre-trained image completion network to obtain a completed image;
wherein the image completion network is trained through the following steps of:
acquiring (<NUM>) a sample image set;
for a sample image in the sample image set, erasing (<NUM>) a part in the sample image randomly, the part being probably covered by a piece of clothing, to obtain to-be-completed information of the sample image;
for the sample image in the sample image set, performing (<NUM>) the human body positioning analysis on the sample image to obtain posture information in the sample image; and
using (<NUM>) an image obtained after the erasing, the to-be-completed information of the sample image, and the posture information in the sample image in the sample image set as inputs, and using the sample image before the erasing as an output, to perform a machine learning training to obtain the image completion network; and
wherein the image completion network comprises: a generation network and a discriminant network, the generation network comprises a gated convolution layer, a residual block, a self-attention layer, and a transposed convolution layer, and the discriminant network comprises a spectral normalization convolution layer.