Patent ID: 12254597

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and description are not intended to be restrictive. The words “exemplary” or “example” are used herein to mean “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” or “example” is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

Certain embodiments described herein address the limitations of conventional image editing systems by providing a machine learning model to perform morphing operations on an input image to generate a morphed image, wherein the morphed image can be used for modifying online computing environments or other systems. An image editing system is typically a network-based computing system including network-connected servers configured to offer a service (e.g. via a website, mobile application, or other means) allowing end users (e.g., consumers) to interact with the servers using network-connected computing devices (e.g. personal computers and mobile devices) to select or otherwise provide an input image and instructions for editing the input image. For example, the instructions include an input text that specifies target characteristics toward which to morph the input image into a blended image.

Conventional image editing systems may train a neural network to begin an image optimization process from a mean latent code or other predefined latent code representing a generic image comprising a set of characteristics and perform an iterative morphing process to blend the predefined initial latent code with a latent code of the input image to generate a blended image that is semantically similar to the text input. However, in practice, utilizing such a predefined initial latent code may bias the morphing process so that the image morphing process is inaccurate with regard to images having characteristics that do not resemble the predefined latent code. Therefore, the output images of such a conventional text-guided image manipulation model may not be visually realistic and also may result in controversial racial or gender inaccuracies as well as exclusion of applicability of image morphing processes to particular types of images. For example, a predefined initial latent code that corresponds to an image of a white woman may generate an inaccurate output image if the input image is of a man or of a non-white person. Further, while the use of a predefined initial latent code in conventional text-guided image manipulation models may, in some cases, provide accurate results, a number of iterations of a morphing operation required to reach target characteristics specified by an input text may increase as a similarity between the predefined initial latent code and the target characteristics decreases, resulting in increased bandwidth usage by conventional image editing systems. Certain embodiments described herein improve the performance of conventional image manipulation models used in image editing systems by, for example, selecting, from a cache of initial latent codes, an initial latent code that corresponding to characteristics specified in the input text, which can improve an accuracy of an image output by the text-guided image manipulation model and which can reduce a number of iterations of the image manipulation model necessary to generate the output image.

The following non-limiting example is provided to introduce certain embodiments. In this example, an image modification system that implements an image modification model receives an input image (e.g. an image of a young man) as well as an input text that specifies target features for performing an image modification process on the input image (e.g. an input text that says “Santa Claus”). The input image and the input text may be received from a client system (e.g. an image blending service system). The image modification system generates a blended image. For example, the image modification system generates a blended image that resembles the input image (the image of the young man) as well as target features specified in the input text (features of Santa Claus) by applying the image modification model to the input image and the input text. The image modification system selects an initial latent code, based on the input text, from a set of initial latent codes. For example, the initial latent code could be associated with an image of Santa Claus, which was specified in the input text. The image modification system generates a blended latent code by blending the initial latent code with a latent code determined from the input image. The image modification system generates a blended image from the blended latent code.

Continuing with this example, the image modification system transmits the blended image (output by the image modification model) to the system from which the request for the blended image was received. In this example, the system that requested the blended image is an image blending service system. In some embodiments, the image blending service system, or the system to which the blended image is transmitted, modifies features of an online computing environment based on the blended image. In some instances, modifying the features of the online computing environment include presenting the blended image to a user accessing the image blending service system via a user computing device (e.g. via an application or web browser operating on the user computing device), or otherwise performing an action with respect to the user request for the blended image.

In certain embodiments, during a training phase, the image modification model is trained to generate a blended image based on an input image and an input text. In an example, an initial latent code library is constructed from a set of images. The latent code library includes a set of initial latent codes, each of which can be used in a latent code blending process. For example, to construct the latent code library, the image modification system102generates, using a CLIP model, a CLIP code for each of the set of images and, using latent code encoder, an initial latent code for each of the set of images. The image modification model is trained using one or more loss functions during the training phase. For example, a CLIP loss is a loss between an input text CLIP code and a CLIP code determined from a blended image output by the image modification model. At each iteration of the image modification model, one or more parameters of the image modification model may be modified based on the determined CLIP loss. For example, the CLIP loss represents how text features derived from a blended image output by the image modification model correspond to features specified in the input text. Other loss functions may also be used. For example, an identity loss or perceptual loss can represent how features in the blended image output by the image modification model corresponds to features of the input image. For example, a face recognition network can compare features between the blended image and the input image and determine the identity loss. In some examples, the image modification model is trained using a combined loss function determined from one or more loss functions (e.g. a combined loss function based on the CLIP loss function and the identity loss function).

The image modification model that generates a blended image based on an input image and an input text, described herein, provides several improvements and benefits over conventional techniques. For example, the conventional techniques discussed above use, in conjunction with an image blending process, a predefined initial latent code with which to blend an input image latent code determined from the input image. The image modification model described herein can reduce a number of iterations of the image modification model necessary to generate a blended image having features corresponding to features identified in the input text. Therefore, the blended image output provided by the image modification model described herein decrease a usage of computing resources and increase a speed at which an output blended image is generated. Further, the image modification model described herein can increase an accuracy of output blended image with respect to target features specified in the input text through selection of an initial latent code for the image blending process that is semantically similar to the target features specified in the input text.

As used herein, the term “CLIP code” is used to refer to an output of a Contrastive Language-Image Pretraining (“CLIP”) model that is trained on a variety of image-text pairs. Accordingly, the CLIP model can generate a CLIP code for an input image. In certain examples, the image modification model can access a latent code library that includes a set of starting latent codes, each of the starting latent codes associated with a respective clip code which is an output of a CLIP model applied to a respective image generated from the respective starting latent code. In certain examples, an input text CLIP code can be generated from an input text by applying a CLIP model to the input text. In these examples, the CLIP model generates an input text CLIP code that is similar to a format and content to CLIP codes generated when the CLIP model is applied to images. For example, the input text could be “A young boy with blonde hair” and the input text CLIP code could be a matrix or array [B, 512] of B rows and 512 columns, for all text inputs, where B (rows) is the number of text inputs in each batch. In certain examples, an input text CLIP code is compared against a set of CLIP codes in the latent code library to select a CLIP code that has a greatest similarity to the input text CLIP code. In certain examples, the image modification system determines a CLIP loss based on the input text CLIP code and a blended image CLIP code determined from a blended image output by the image modification model.

As used herein, the term “latent code” is used to refer to a vector representation of an image in a latent space (e.g. a StyleGAN space or other latent space). In certain examples, a latent code is a vector (e.g. an 18×512 vector) that represents an image. In certain examples, the latent code includes a set of vectors for a set of layers (e.g. scales). For example, the set of vectors comprises 18 layer-specific 1×512 vectors. For example, in the context of a facial image, higher layers may represent course features (e.g. pose and hairstyle), and lower layers may represent features such as color scheme and details. In certain examples, the latent code is a low-dimensional representation of an image that when, passed through a generative adversarial network (e.g. a StyleGAN network), outputs the image. In certain examples, a latent code is generated from an image by applying a latent code encoder (e.g. a StyleGAN encoder or other encoder) to the image. The latent code encoder can also generate an image from a latent code. In certain examples, an initial latent code is selected from a set of initial latent codes based on similarity of CLIP codes associated with each of the set of latent codes to a CLIP code generated from the input text. For example, the image modification system stores a set of initial latent codes, each of the set of initial latent codes associated with a respective CLIP code. In certain examples, a blended latent code is generated by mixing the selected initial latent code with the input image latent code. A blended image can be generated from the blended latent code.

Example Operating Environment for Determining a Contrast Group from a Set of Recommendable Items

Referring now to the drawings,FIG.1depicts an example of a computing environment100for generating a blended image by modifying an input image using an image modification model that is guided by an input text. The computing environment100includes an image modification system102, which can include one or more processing devices that execute an image modification subsystem104and a model training subsystem106. In certain embodiments, the computing environment100includes a contrastive language-image pre-training (“CLIP”) module103, an initial latent code selection module105, a latent code blending module107, a latent code generator module108, and a request module111.

In certain embodiments, the image modification system102is a network server or other computing device connected to a network140. The image modification system102applies an image modification model109to an input image120and an input text122received from a client system130to generate a blended image124. For example, applying the image modification model109includes selecting, from a latent code library117, a starting latent code118from a set of starting latent codes118. Each of the set of starting latent codes118has a respective associated CLIP code119and the selected starting latent code118has an associated CLIP code119that has a greatest similarity to a CLIP code119determined from the input text122. The one or more processing devices of the image modification system102can further execute a model training subsystem106for training the image modification model109that is used for generating the blended image124. The item recommendation system102transmits the blended image to the client system130via the network140and the client system130stores the blended image124in the data store135. The computing environment100further includes a data store110for storing data used in the generation of the blended image125, such as the training data set114, which includes the latent code library117.

In certain examples, the item recommendation system102generates the latent code library117from a set of training images and includes, for each of the set of images, a set of N initial latent codes (“ILCs”)118(e.g. ILC118-0, ILC118-1, . . . ILC118-N), where each of the set of initial latent codes118is associated with a respective CLIP code (“CC”) of a set of CCs119(e.g. CC119-0, CC119-1, . . . CC119-N). For example, ILC118-0is associated with CC119-0, ILC118-1is associated with CC119-1, etc. In certain embodiments, the set of ILCs118is generated by applying an encoder to each of the set of images and the set of CCs119is generated by applying a CLIP model to each of the set of images. For example, ILC118-0is generated by applying an encoder to an image and CC119-0is associated with the ILC118-0is generated by applying a CLIP model to the image. In certain embodiments, the latent code generator module108applies the encoder to the set of images to generate the CCs119and the CLIP module103accesses the CLIP model and applies the CLIP model to each of the set of images to generate the ILCs118.

The image modification subsystem104, the model training subsystem106, the CLIP module103, the initial latent code selection module105, the latent code blending module107, the latent code generator module108, and the request module111may be implemented using software (e.g., code, instructions, program) executed by one or more processing devices (e.g., processors, cores), hardware, or combinations thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory component). The computing environment100depicted inFIG.1is merely an example and is not intended to unduly limit the scope of claimed embodiments. One of the ordinary skill in the art would recognize many possible variations, alternatives, and modifications. For example, in some implementations, the image modification system102can be implemented using more or fewer systems or subsystems than those shown inFIG.1, may combine two or more subsystems, or may have a different configuration or arrangement of the systems or subsystems.

The image modification subsystem104is configured to receive or otherwise access an input image120and an input text122. In certain embodiments, the request module111receives or accesses the input image120and the input text122. The input text122specifies target features toward which to modify (e.g. morph) the input image120. For example, the input image120is a facial image of the user (a young man with no beard) and the input text122reads “Santa Claus,” who is commonly depicted as an older man with a beard. In this example, the user is interested in modifying (e.g. morphing) the input image120to have features that resemble Santa Claus, as indicated by the input text122. In some instances, the input image120and the input text122are provided to the image modification subsystem104by a client system130(e.g. an image modification service system, a user computing device executing an image modification application, etc.). In certain examples, a user uploads the input image120and enters the input text122and the client system130receives the input image120and the input text122and transmits, via the network140, the input image120and the input text122to the image modification subsystem104. In certain examples, the client system130is a user computing device and the user uploads or otherwise selects the input image120and inputs the input text122via a user interface of the user computing device. In some instances, the client system130includes an image modification application131, which receives and communicates the selection of the input image120and input text122to the image modification subsystem104via the network140. In some instances, the image modification system102provides, for download by the client system130, the image modification application131. In some instances, the image modification application131displays one or more images and a request to select an image, which could read “Please upload/select an image.” The image modification application131receives a selection of the input image120. In some instances, the image modification displays a request for an input text122, which could read “Who/what would you like the selected image look like?” The image modification application131, in some instances, provides a text field or other means of receiving the input text122(e.g. voice to text, selection of one or more words from a list, etc.)

In some instances, the image modification subsystem104receives a request to modify the input image120in accordance with the input text122. For example, the user inputs the input text122(e.g. “Santa Claus”), uploads or otherwise selects the input image120(e.g. the image of a face of the user, a young man with no facial hair), and selects a user interface object that reads “Modify this image.” In some instances, the image modification subsystem104receives the request (to modify the input image120in accordance with the input text122) from the client system130via the network140.

To generate a blended image124, which is a modified input image120which includes features that resemble the input text122, the item recommendation subsystem104employs an image modification model109. Additional details about the image modification model109are provided below with respect toFIG.2. Additional details about generating the blended image124by applying a trained image modification model109are provided below with respect toFIG.3. Additional details about training the image modification model109are provided below with respect toFIG.4. An illustration of additional details about selecting a starting latent code (a feature of the image modification model109) are provided below with respect toFIG.5. An illustration of additional details about blending a starting latent code with an input image latent code to generate a blended image latent code (a feature of the image modification model109) are provided below with respect toFIG.6. In certain examples, the image modification system102includes a request module111configured to receive an input image120and an input text122and a request to modify the input image120in accordance with the input text122. In some examples, the image modification system102includes a request module111, a latent code generator module108, a Contrastive Language-Image Pretraining (“CLIP”) code generator module103, an initial latent code selection module105, and a latent code blending module107. In some instances, the request module111receives, from a client device130or other client system, an input image120, an input text122and a request to generate a blended image124. In some instances, the latent code generator module108generates an input image latent code from the input image120. The CLIP code generator module103generates an input text CLIP code from the input text122(e.g. by applying a CLIP model to the input text122). In some instances, the initial latent code selection module105selects, from the latent code library117including a set of CLIP codes119associated with a set of initial latent codes118, an initial latent code118-N having an associated CLIP code119-N with a greatest similarity to the input text CLIP code118-N. In some instances, the latent code blending module107generates a blended image latent code based on the input image latent code and the selected initial latent code118-N. In some instances, the latent code generator module108generates a blended image124from the blended image latent code. In some instances, the request module111transmits, to the client device130or to the other client system responsive to the request to generate the blended image124, the generated blended image124. In some instances, the CLIP code generator module103generatesna blended image CLIP code based on the blended image124.

The image modification system102determines and trains the image modification model109using the model training subsystem106. The model training subsystem106builds and trains the image modification model109. The model training subsystem106, using the training data set114, trains the image modification model109. Additional details of training an image modification model109are described inFIG.4. In various examples, the model training subsystem106can be implemented as one or more of program code, program code executed by processing hardware (e.g., a programmable logic array, a field-programmable gate array, etc.), firmware, or some combination thereof.

Example of an Image Modification Model

FIG.2depicts an illustration of an image modification model109for use in certain embodiments described herein, for example as describedFIG.1andFIGS.3-6. As depicted inFIG.2, the image modification model109includes a latent code generator model210, a CLIP code generator model220, an initial latent code selection model230, and a latent code blending model240. An example process for applying the image modification model109to an input image120and an input text122is described inFIG.3and an example process for training the image modification model109is described inFIG.4. In certain examples, as depicted inFIG.2, the image modification model109receives an input image120and an input text122, for example, associated with a user request to modify the input image120in accordance with the input text122. The image modification model109generates a blended image124, which can be displayed to the user.

The latent code generator model210is configured to generate an input image latent code201for the input image120. The latent code generator model210, in some instances, is a StyleGAN encoder configured to project the input image120into StyleGAN latent space. The input image latent code201is a vector representation of the input image120for a set of-specific vectors (e.g. 18 layer-specific 1×512 vectors). In certain examples, the latent code generator model210is a mapping network (e.g. StyleGAN mapping network) configured to generate the set of layer-specific vectors. In some instances, particular layers correspond to specific features of an image. For example, in the context of a facial image, higher layers may represent course features (e.g. pose and hairstyle), and lower layers may represent features such as color scheme and details.

The CLIP code generator model220is configured to generate an input text CLIP code202for the input text122. For example, the input text122specifies target features toward which to modify the input image120. In some instances, the input text CLIP code202is text embedding generated from the input text122. In some instances, semantically-related but different input texts122could yield a common input text CLIP code202. The input text CLIP202is a matrix or array [B, 512] of B rows and 512 columns, for all text inputs, where B (rows) is the number of text inputs in each batch

The initial latent code selection model230is configured to select, from a latent code library117, an initial latent code203with which to perform, by the latent code blending model240, a latent code blending process of the initial latent code203with the input image latent code201. The latent code library117includes a set of N initial latent codes (“ILCs”)118(e.g. ILC118-0, ILC118-1, . . . ILC118-N as depicted inFIG.1), where each of the set of initial latent codes118is associated with a respective CLIP code (“CC”) of a set of CCs119(e.g. CC119-0, CC119-1, . . . CC119-N as depicted inFIG.1). The initial latent code selection model230is configured to determine, for each of the set of CCs119, a similarity of the CC119to the input text CLIP code202. The initial latent code selection model230is configured to select an initial latent code203, from the set of ILCs118, that is associated with a CC119having the greatest determined similarity to the input text clip code202. Accordingly, the latent code selection model230, in some instances, selects the initial latent code203, from the set of ILCs118, that is the most semantically-similar to the input text122.

The latent code blending model240is configured to blend the input image latent code201generated by the latent code generator model210with the initial latent code203selected from the latent code library117by the initial latent code selection model230. The latent code blending model240, in some embodiments, is a StyleGAN synthesis network. In certain embodiments, the blending of the input image latent code201and the initial latent code203to generate the blended image latent code204is performed on a layer specific basis. For example, the input image latent code201and the initial latent code203, in some instances, each comprise 18 vectors of size 1×512 and a blending operation is performed for each of the 18 vectors. An illustration of a layer specific blending process to generate a portion of a blended image latent code204is illustrated inFIG.6.

As depicted inFIG.2, the latent code generator model210is configured to generate a blended image from the blended image latent code124generated by the latent code blending model240. For example the latent code generator model210can generate a latent code for an image and can generate an image from a latent code. As depicted inFIG.2, the CLIP code generator model220is configured to generate, for the blended image124generated by the latent code generator model210from the blended image latent code204, a blended image CLIP code205. As depicted inFIG.2, the image modification model109is configured to determine a CLIP loss206based on the input text CLIP code202and the blended image CLIP code205, which were determined by the CLIP code generator model220. In some instances, determining the CLIP loss206includes applying a loss function to the input text CLIP code202and the blended image CLIP code205. In certain embodiments, during a training phase, one or more parameters of the image modification model109can be modified based on the determined CLIP loss206. Additional details about training the image modification model109are described inFIG.4.

Examples of Computer-Implemented Operations for Generating a Blended Image Based on an Input Image and an Input Text

FIG.3depicts an example of a process for generating a blended image124by applying an image modification model109to an input image120and an input text122. One or more computing devices (e.g., the image modification system102or the individual modules contained therein) implement operations depicted inFIG.3. For illustrative purposes, the process200is described with reference to certain examples depicted in the figures. Other implementations, however, are possible.

At block310, the method300involves receiving an input image120, an input text122, and a request to modify the input image120based on the input text122. In an embodiment, the client system130(e.g. a user computing device) transmits the input image120and the input text122via the network140. In certain embodiments, the image modification subsystem104receives the input image120using a receipt module. For example, the user selects the input image120and inputs the input text122via an image modification application131(or web browser application) on the user computing device, which communicates with the image modification system102via the network140. The image modification application131(or web browser application) is configured to transmit a request to modify an input image120according to an input text122responsive to receiving inputs from the user and to display a blended image124generated by the image modification system102. In some instances, the input image120is a human facial image of the user or other human facial image. In other instances, the input image120is a non-human facial image, an image of an object, a landscape, or other type of image. In some instances, the input text122is a set of words provided by the user that specify how the user desires for the input image120to be modified. For example, the input text122includes one or more target features (e.g. a hairstyle, a hair color, a hair length, a skin color, an eye color, an age, a gender, a profession, a facial expression, or other feature or combination of features), a target identity (e.g. a celebrity or other person's name). For example, the user desires an image of the user with features that look like George Washington. In this example, the user provides an input text122of “George Washington” (a historical figure with characteristic facial features and wears a grey wig) and an input image120of the user's face (e.g. a young man with a red hair). In certain examples, the request module of item recommendation system102is configured to receive the input image120, the input text122, and a request to modify the input image120in accordance with the input text122. For example, the user desires to receive an image including features both of the user (from the input image120) and of George Washington (as specified by the input text122).

At block320, the method300involves applying an image modification model109to the input image120and the input text122to generate a blended image124. In some embodiments, block320includes implementing blocks321,323,325,327, and329.

At block321, the method300involves generating, using a latent code generator module, an input image latent code201. In certain examples, the latent code generator module applies the latent code generator model210to the input image120. In some instances, the latent code generator model210is a StyleGAN encoder configured to project the input image120into StyleGAN latent space. The input image latent code201is a vector representation of the input image120for a set of layer-specific vectors (e.g. 18 layer-specific 1×512 vectors). In certain examples, the latent code generator model210is a mapping network (e.g. StyleGAN mapping network) configured to generate the set of layer-specific vectors. Continuing with the previous example, the input image120is an image of the user and the image modification system102generates an input image latent code201representing the image of the user's face.

At block323, the method300involves generating, using a clip code generator module, an input text clip code202from the input text122. In certain examples, the clip code generator module generates the input text clip code202by applying the clip code generator model220to the input text122. For example, the input text122specifies target features toward which to modify the input image120. In some instances, the input text CLIP code202is text embedding generated from the input text122. In some instances, semantically-related but different input texts122could yield a common input text CLIP code202. For example, continuing with the example described previously, the input text CLIP code202generated for input text122“George Washington” is a matrix or array [B, 512] of B rows and 512 columns, for all text inputs, where B (rows) is the number of text inputs in each batch.

At block325, the method300involves selecting, using an initial latent code selection module, an initial latent code203from a library of latent codes having a CLIP code with a greatest similarity to the input text CLIP code202. In certain examples, the initial latent code selection module applies the initial latent code selection model230to the input text CLIP code202to select an initial latent code118from the library. The latent code library117includes a set of N initial latent codes (“ILCs”)118(e.g. ILC118-0, ILC118-1, . . . ILC118-N as depicted inFIG.1), where each of the set of initial latent codes118is associated with a respective CLIP code (“CC”) of a set of CCs119(e.g. CC119-0, CC119-1, . . . CC119-N as depicted inFIG.1). The initial latent code selection model230is configured to determine, for each of the set of CCs119, a similarity of the CC119to the input text CLIP code202. The initial latent code selection model230is configured to select an initial latent code203, from the set of ILCs118, that is associated with a CC119having the greatest determined similarity to the input text clip code202. Accordingly, the latent code selection model230, in some instances, selects the initial latent code203, from the set of ILCs118, that is the most semantically-similar to the input text122. Continuing with the previous example, an example first CLIP code associated with a first initial latent code in the latent code library representing an image of a young woman's face has a lesser similarity to the input text clip code202(determined from the input text122“George Washington”) than a second CLIP code associated with a second initial latent code representing an image of an older man's face with white hair.

At block327, the method300involves generating, using a latent code blending module, a blended latent code204representing a blended image125by blending the input image latent code201with the selected initial latent code203. The latent code blending module, in some embodiments, uses the latent code blending model240(e.g. a StyleGAN synthesis network) to blend the input image latent code201with the selected initial latent code203. In certain embodiments, the blending of the input image latent code201and the initial latent code203to generate the blended image latent code204is performed on a layer specific basis. For example, the input image latent code201and the initial latent code203, in some instances, each comprise 18 vectors of size 1×512 and a blending operation is performed for each of the 18 vectors. An illustration of a layer specific blending process to generate a portion of a blended image latent code204is illustrated inFIG.6. Continuing with the previous example, the selected initial latent code203(of a face of an older man with white hair) is blended with the input image latent code202(determined from the image of the face of the user, a young man with red hair).

In certain embodiments, the latent code blending module uses the latent code blending model240to further blend the blended latent code204with the input image latent code201to generate a subsequent blended latent code204. In other embodiments, the latent blending module does not generate the subsequent blended latent code204.

At block329, the method300involves generating, by the latent code generator module, the blended image125based on the blended latent code204. In some instances, the latent code generator module generates the blended image125based on the subsequent blended latent code204, which is a blend of the blended latent code204and the input image latent code201. The latent code generator module can, using the latent code generator model210, both generate a latent code from an image and generate an image from a latent code. Continuing with the previous example, the blended image125includes features of the user (determined from the input image latent code) as well as George Washington (indicated by the input text122). For example, the blended image125may include the young man's facial structure, but with a gray wig similar to George Washington. In another example, the blended image125may include George Washington's facial structure, but with a reddish gray wig (in between the user's hair color and the grey color of George Washington's wig).

At block330, the method300involves transmitting, by the image modification system102and responsive to the request, the blended image124. For example, the image modification subsystem104transmits blended image124to the system from which the request to modify the input image120was received (e.g. the client system130, which, in some instances, is a user computing device). In some embodiments, the image modification system102stores the blended image124in the data store110, which is accessible to the client system130, and the client system130accesses the blended image124via the network140. In some embodiments, the image modification subsystem104, or the client system130to which the blended image124is transmitted, modifies features of an online computing environment based on the blended image124generated by the image modification model109for the input image120and the input text122. In some instances, modifying the features of the online computing environment include presenting, in a user interface (e.g. via an image modification website hosted by the client system130) the blended image124.

FIG.4depicts an example of a process400for training an image modification model109, according to certain embodiments disclosed herein. One or more computing devices (e.g., the image modification system102or the model training subsystem106) implement operations depicted inFIG.4. For illustrative purposes, the process400is described with reference to certain examples depicted in the figures. Other implementations, however, are possible.

At block410, the method400involves constructing an image modification model109and an initial latent code library117having a set of clip codes119and an associated set of initial latent codes118. The image modification model109, in some instances, includes a latent code generator model210, a CLIP code generator model220, an initial latent code selection model230, and a latent code blending model240. Details of an example image modification model109are described inFIG.2herein.

An example initial latent code library117is depicted inFIG.1. In some instances, the latent code library117is generated from a set of training images and includes, for each of the set of images, a set of N initial latent codes (“ILCs”)118(e.g. ILC118-0, ILC118-1, . . . ILC118-N), where each of the set of initial latent codes118is associated with a respective CLIP code (“CC”) of a set of CCs119(e.g. CC119-0, CC119-1, . . . CC119-N). For example, ILC118-0is associated with CC119-0, ILC118-1is associated with CC119-1, etc. In certain embodiments, the set of ILCs118is generated by applying an encoder to each of the set of images and the set of CCs119is generated by applying a CLIP model to each of the set of images. For example, ILC118-0is generated by applying an encoder to an image and CC119-0is associated with the ILC118-0is generated by applying a CLIP model to the image.

In certain embodiments, blocks420,430,440, and450ofFIG.4substantially correspond to blocks321,323,325, and327ofFIG.3, respectively.

At block420, the method400involves generating, by a latent code generator module, an input image latent code201based on an input image120. In some instances, generating the input image latent code201includes applying the latent code generator model210to the input image120. The input image latent code201is a vector representation of the input image120for a set of layer-specific vectors (e.g. 18 layer-specific 1×512 vectors). In certain examples, the latent code generator model210is a mapping network (e.g. StyleGAN mapping network) configured to generate the set of layer-specific vectors.

At block430, the method400involves generating, by a clip code generator module, an input text CLIP code202based on the input text222. In some instances, generating the input text CLIP code202includes applying the clip code generator model220to the input text222. For example, the input text122specifies target features toward which to modify the input image120. In some instances, generating the input text clip code202includes applying the CLIP code generator model220to the input text222. In some instances, the input text CLIP code202is text embedding generated from the input text122. In some instances, semantically-related but different input texts122could yield a common input text CLIP code202.

At block440, the method400involves selecting, using an initial latent code selection module, an initial latent code203, from the latent code library117, associated with a CLIP code having a greatest similarity to the input text CLIP code202. In some instances, selecting the initial latent code203from the latent code library117includes applying the initial latent code selection model230to the input text CLIP code202. The latent code library117includes a set of N initial latent codes (“ILCs”)118(e.g. ILC118-0, ILC118-1, . . . ILC118-N as depicted inFIG.1), where each of the set of initial latent codes118is associated with a respective CLIP code (“CC”) of a set of CCs119(e.g. CC119-0, CC119-1, . . . CC119-N as depicted inFIG.1). The initial latent code selection model230is configured to determine, for each of the set of CCs119, a similarity of the CC119to the input text CLIP code202. The initial latent code selection model230is configured to select an initial latent code203, from the set of ILCs118, that is associated with a CC119having the greatest determined similarity to the input text clip code202. Accordingly, the latent code selection model230, in some instances, selects the initial latent code203, from the set of ILCs118, that is the most semantically-similar to the input text122.

At block450, the method400involves generating, using a latent code blending module, a blended latent code204representing a blended image125by blending the input image latent code201with the selected initial latent code203. In some instances, generating the blended latent code204includes applying the latent code blending model240to the input image latent code201and the selected initial latent code203. The latent code blending module, in some embodiments, uses a StyleGAN synthesis network to blend the input image latent code201with the selected initial latent code203. In certain embodiments, the blending of the input image latent code201and the initial latent code203to generate the blended image latent code204is performed on a layer specific basis. For example, the input image latent code201and the initial latent code203, in some instances, each comprise 18 vectors of size 1×512 and a blending operation is performed for each of the 18 vectors. An illustration of a layer specific blending process to generate a portion of a blended image latent code204is illustrated inFIG.6.

At block460, the method400involves generating, using the CLIP code generator module, a blended image CLIP code205from a blended image124generated from the blended image latent code204. For example, the CLIP code generator module applies the CLIP code generator model220to the blended image124to generate the blended image CLIP code205. In some instances, the input text CLIP code202is text embedding generated from the input text122. In some instances, semantically-related but different input texts122could yield a common input text CLIP code202.

At block470, the method400involves determining a CLIP loss206based on the blended image CLIP code205and the input text CLIP code202. The CLIP loss206, in some instances, is an inverse cosine similarity between the blended image CLIP code205and the input text CLIP code202. The inverse cosine similarity may be a cosine distance in CLIP latent space. A CLIP loss206,CLIP(w), can be represented as:
CLIP(w)=DCLIP(G(w+Mt(w)),t)  Equation (1),
where G represents the latent code blending model240(e.g. a StyleGAN generator model), w represents the selected initial latent code203, Mt(w) represents a manipulation to the selected initial latent code203, t represents the input text CLIP code202, and DCLIP( ) represents determining a cosine distance in CLIP latent space.

In certain embodiments, an identity loss,ID(w), is determined based on the following equation:
ID(w)=1−R(G(ws)),R(G(w))Equation (2),
where G represents the latent code blending model240, wsrepresents the input image latent code201, w represents the initial latent code203, R is a pretrained network for facial recognition (e.g. an ArcFace network), andcomputes a cosine similarity between its arguments. In certain embodiments, a combined loss,(w), is determined based on the losses determined in Equation (1) and Equation (2). In certain embodiments, a weighted combined loss is determined, for example, using the following equation:
(w)=CLIP(w)+λL2∥Mt(w)∥2+ΔIDID(w)  Equation (3),
where λL2and is λIDare parameter values. In some instances, these λL2and is λIDparameters can be modified based on a nature of a desired image modification operation. For example, if the input text122indicates a change, for the input image, to another identity (e.g. input text of “George Washington”), the λIDcan be set to a lower value (or λL2can be set to a higher value) than for input text122that does not indicate a change to another identity (e.g. input text of “blond hair”). In some instances, the image modification system102may determine whether the input text122corresponds to a change in identity and modify one or more of the parameters λL2and is λIDbased on the determination.

At block480, the method400involves modifying parameters of the image modification model109based on the determined CLIP loss206. For example, the model training subsystem106adjusts parameters of one or more of the latent code generator model210, the CLIP code generator model220, the initial latent code selection model230, and/or the latent code blending model240based on the determined CLIP loss206. In some instances the model training subsystem106adjusts parameters of one or more of the submodels of the image modification model109based on a determined combined loss (e.g. as in Equation 3), where the combined loss is determined based on the CLIP loss206and an identity loss.

In some embodiments, blocks450-480can be repeated for a number of iterations to optimize an output of the image modification model109by minimizing the determined CLIP loss206. In certain examples, the number of training iterations is predefined. For example, after the blended image125is generated and one or more parameters of one or more of the submodels210,220,230, and240of the model109is modified, the blended image latent code206takes the place of the initial latent code204with respect to blocks450-480.

In certain embodiments, blocks410-480are repeated, iteratively, in a gradient descent optimization process to minimize the CLIP loss206. In some instances, an optimization method, for example, a Broyden, Fletcher, Goldfarb, and Shanno (“BFGS”) algorithm, or a limited memory BFGS (“L-BFGS”) algorithm, may be used to minimize the CLIP loss206.

FIG.5depicts an example illustration500of selecting, by an image modification model, an initial latent code from which to generate a blended image, according to certain embodiments disclosed herein. For illustrative purposes, the illustration500is described with reference to certain examples depicted in the figures. Other implementations, however, are possible. As shownFIG.5, the image modification system102receives, from a client system130(a user computing device operated by a user), an input text122that reads “A young boy with blonde hair.” The image modification system102receives the input text122along with an input image120(e.g. an image of the user's face). The image modification system102, using the CLIP code generator model220, generates an input text CLIP code202. The image modification system102selects, from the latent code library117of CLIP codes (CCs)119and associated initial latent codes (ILCs)118, a CLIP code119that is most similar to the input text CLIP code202. In the example depicted inFIG.5, the latent code library117includes N CLIP codes119(e.g. CC119-0, CC119-1, CC119-2. . . CC119-N, as depicted) and an associated N initial latent codes118(ILC118-0, ILC118-1, ILC118-2, . . . ILC118-N, as depicted). The image modification system102may use the initial latent code selection model230to select the initial latent code118associated with the CLIP code119in the latent code library117that is most similar to the input text CLIP code202. In the example ofFIG.5, CC119-2is most similar to the input text CLIP code202and, therefore, the image modification system102selects ILC119-2, which is associated with CC119-2, as the initial latent code119for which to conduct an image modification process using the latent code blending model240.

FIG.6depicts an example illustration of blending, by an image modification model, an initial latent code with an input image latent code, according to certain embodiments disclosed herein.FIG.6illustrates blending, by a latent code blending model240, a portion of an initial latent code203with a corresponding portion of an input image latent code201. As shown inFIG.6, both the portion of the initial latent code203and the portion of the input image latent code201each includes 9 layers, each layer being a 1×512 vector. In certain embodiments, a full initial latent code includes 18 layers, where each layer is a 1×512 vector. As shown inFIG.6, each layer of the portion of the initial latent code203includes an associated blending ratio601, and each layer of the portion of the input image latent code201includes an associated blending ratio602. As shown inFIG.6, the blending ratio determines how much of the respective layer will be considered by the latent code blending model240in a blending process. The ratios601and602depicted inFIG.6are either 0.0 or 1.0, however, other values between 0 and 1 may be used, and the sum of the ratios601and602for each layer is equal to one (1). For example, a layer of the initial latent code203may be assigned a ratio601of 0.75 and a corresponding layer of the input image latent code201may be assigned a ratio602of 0.25. In the example inFIG.6, the top three layers are assigned a ratio601of 0.0 and a ratio602of 1.0, and the bottom six layers are assigned a ratio601of 1.0 and a ratio602of 0.0. Accordingly, in the example ofFIG.6, in the final portion of blended image latent code204, the top three layers are the corresponding layers of the input image latent code201and the bottom six layers are the corresponding layers of the initial latent code203.

Examples of Computing Environments for Implementing Certain Embodiments

Any suitable computer system or group of computer systems can be used for performing the operations described herein. For example,FIG.7depicts an example of a computer system700. The depicted example of the computer system700includes a processing device702communicatively coupled to one or more memory components704. The processing device702executes computer-executable program code stored in a memory components704, accesses information stored in the memory component704, or both. Execution of the computer-executable program code causes the processing device to perform the operations described herein. Examples of the processing device702include a microprocessor, an application-specific integrated circuit (“ASIC”), a field-programmable gate array (“FPGA”), or any other suitable processing device. The processing device702can include any number of processing devices, including a single processing device.

The memory components704includes any suitable non-transitory computer-readable medium for storing program code706, program data708, or both. A computer-readable medium can include any electronic, optical, magnetic, or other storage device capable of providing a processing device with computer-readable instructions or other program code. Non-limiting examples of a computer-readable medium include a magnetic disk, a memory chip, a ROM, a RAM, an ASIC, optical storage, magnetic tape or other magnetic storage, or any other medium from which a processing device can read instructions. The instructions may include processor-specific instructions generated by a compiler or an interpreter from code written in any suitable computer-programming language, including, for example, C, C++, C #, Visual Basic, Java, Python, Perl, JavaScript, and ActionScript. In various examples, the memory components404can be volatile memory, non-volatile memory, or a combination thereof.

The computer system700executes program code706that configures the processing device702to perform one or more of the operations described herein. Examples of the program code706include, in various embodiments, the item recommendation system102(including the recommender subsystem104as well as the request module, the unbiased conversion rate prediction module, the biased conversion rate prediction module, the item selection module, and/or other modules of the item recommendation system120and the model training subsystem106described herein) ofFIG.1, which may include any other suitable systems or subsystems that perform one or more operations described herein (e.g., one or more neural networks, encoders, attention propagation subsystem and segmentation subsystem). The program code706may be resident in the memory components704or any suitable computer-readable medium and may be executed by the processing device702or any other suitable processor.

The processing device702is an integrated circuit device that can execute the program code706. The program code706can be for executing an operating system, an application system or subsystem, or both. When executed by the processing device702, the instructions cause the processing device702to perform operations of the program code706. When being executed by the processing device702, the instructions are stored in a system memory, possibly along with data being operated on by the instructions. The system memory can be a volatile memory storage type, such as a Random Access Memory (RAM) type. The system memory is sometimes referred to as Dynamic RAM (DRAM) though need not be implemented using a DRAM-based technology. Additionally, the system memory can be implemented using non-volatile memory types, such as flash memory.

In some embodiments, one or more memory components704store the program data708that includes one or more datasets described herein. In some embodiments, one or more of data sets are stored in the same memory component (e.g., one of the memory components704). In additional or alternative embodiments, one or more of the programs, data sets, models, and functions described herein are stored in different memory components704accessible via a data network. One or more buses710are also included in the computer system700. The buses710communicatively couple one or more components of a respective one of the computer system700.

In some embodiments, the computer system700also includes a network interface device712. The network interface device712includes any device or group of devices suitable for establishing a wired or wireless data connection to one or more data networks. Non-limiting examples of the network interface device712include an Ethernet network adapter, a modem, and/or the like. The computer system700is able to communicate with one or more other computing devices via a data network using the network interface device912.

The computer system700may also include a number of external or internal devices, an input device714, a presentation device716, or other input or output devices. For example, the computer system700is shown with one or more input/output (“I/O”) interfaces718. An I/O interface718can receive input from input devices or provide output to output devices. An input device714can include any device or group of devices suitable for receiving visual, auditory, or other suitable input that controls or affects the operations of the processing device702. Non-limiting examples of the input device714include a touchscreen, a mouse, a keyboard, a microphone, a separate mobile computing device, etc. A presentation device716can include any device or group of devices suitable for providing visual, auditory, or other suitable sensory output. Non-limiting examples of the presentation device716include a touchscreen, a monitor, a speaker, a separate mobile computing device, etc.

AlthoughFIG.7depicts the input device714and the presentation device716as being local to the computer system7700, other implementations are possible. For instance, in some embodiments, one or more of the input device714and the presentation device716can include a remote client-computing device that communicates with computing system700via the network interface device712using one or more data networks described herein.

Embodiments may comprise a computer program that embodies the functions described and illustrated herein, wherein the computer program is implemented in a computer system that comprises instructions stored in a machine-readable medium and a processing device that executes the instructions to perform applicable operations. However, it should be apparent that there could be many different ways of implementing embodiments in computer programming, and the embodiments should not be construed as limited to any one set of computer program instructions. Further, a skilled programmer would be able to write such a computer program to implement an embodiment of the disclosed embodiments based on the appended flow charts and associated description in the application text. Therefore, disclosure of a particular set of program code instructions is not considered necessary for an adequate understanding of how to make and use embodiments. Further, those skilled in the art will appreciate that one or more aspects of embodiments described herein may be performed by hardware, software, or a combination thereof, as may be embodied in one or more computer systems. Moreover, any reference to an act being performed by a computer should not be construed as being performed by a single computer as more than one computer may perform the act.

The example embodiments described herein can be used with computer hardware and software that perform the methods and processing functions described previously. The systems, methods, and procedures described herein can be embodied in a programmable computer, computer-executable software, or digital circuitry. The software can be stored on computer-readable media. For example, computer-readable media can include a floppy disk, RAM, ROM, hard disk, removable media, flash memory, memory stick, optical media, magneto-optical media, CD-ROM, etc. Digital circuitry can include integrated circuits, gate arrays, building block logic, field programmable gate arrays (FPGA), etc.

In some embodiments, the functionality provided by computer system700may be offered as cloud services by a cloud service provider. For example,FIG.8depicts an example of a cloud computer system800offering a service for selecting, for a set of recommendable items120, a contrast group124that includes a recommended item122and contrast items123, that can be used by a number of user subscribers using user devices804A,804B, and804C across a data network806. In the example, the service for selecting, for a set of recommendable items120, a contrast group124that includes a recommended item122and contrast items123may be offered under a Software as a Service (SaaS) model. One or more users may subscribe to the service for selecting, for a set of recommendable items120, a contrast group124that includes a recommended item122and contrast items123, and the cloud computer system800performs the processing to provide the service for selecting, for a set of recommendable items120, a contrast group124that includes a recommended item122and contrast items123to subscribers. The cloud computer system800may include one or more remote server computers808.

The remote server computers808include any suitable non-transitory computer-readable medium for storing program code810(e.g., the recommender subsystem104and the model training subsystem106ofFIG.1) and program data812, or both, which is used by the cloud computer system800for providing the cloud services. A computer-readable medium can include any electronic, optical, magnetic, or other storage device capable of providing a processing device with computer-readable instructions or other program code. Non-limiting examples of a computer-readable medium include a magnetic disk, a memory chip, a ROM, a RAM, an ASIC, optical storage, magnetic tape or other magnetic storage, or any other medium from which a processing device can read instructions. The instructions may include processor-specific instructions generated by a compiler or an interpreter from code written in any suitable computer-programming language, including, for example, C, C++, C #, Visual Basic, Java, Python, Perl, JavaScript, and ActionScript. In various examples, the server computers508can include volatile memory, non-volatile memory, or a combination thereof.

One or more of the server computers808execute the program code810that configures one or more processing devices of the server computers808to perform one or more of the operations that provide recommended item122and contrast group124selection services. As depicted in the embodiment inFIG.8, the one or more servers providing the services for selecting, for a set of recommendable items120, a contrast group124that includes a recommended item122and contrast items123may implement the recommender subsystem104and the model training subsystem106. Any other suitable systems or subsystems that perform one or more operations described herein (e.g., one or more development systems for configuring an interactive user interface) can also be implemented by the cloud computer system800.

In certain embodiments, the cloud computer system800may implement the services by executing program code and/or using program data812, which may be resident in a memory component of the server computers808or any suitable computer-readable medium and may be executed by the processing devices of the server computers808or any other suitable processing device.

In some embodiments, the program data812includes one or more datasets and models described herein. In some embodiments, one or more of data sets, models, and functions are stored in the same memory component. In additional or alternative embodiments, one or more of the programs, data sets, models, and functions described herein are stored in different memory components accessible via the data network806.

The cloud computer system800also includes a network interface device814that enable communications to and from cloud computer system800. In certain embodiments, the network interface device814includes any device or group of devices suitable for establishing a wired or wireless data connection to the data networks806. Non-limiting examples of the network interface device814include an Ethernet network adapter, a modem, and/or the like. The recommended item122and contrast group124selection service is able to communicate with the user devices804A,804B, and804C via the data network806using the network interface device814.

The example systems, methods, and acts described in the embodiments presented previously are illustrative, and, in alternative embodiments, certain acts can be performed in a different order, in parallel with one another, omitted entirely, and/or combined between different example embodiments, and/or certain additional acts can be performed, without departing from the scope and spirit of various embodiments. Accordingly, such alternative embodiments are included within the scope of claimed embodiments.

Although specific embodiments have been described above in detail, the description is merely for purposes of illustration. It should be appreciated, therefore, that many aspects described above are not intended as required or essential elements unless explicitly stated otherwise. Modifications of, and equivalent components or acts corresponding to, the disclosed aspects of the example embodiments, in addition to those described above, can be made by a person of ordinary skill in the art, having the benefit of the present disclosure, without departing from the spirit and scope of embodiments defined in the following claims, the scope of which is to be accorded the broadest interpretation so as to encompass such modifications and equivalent structures.

General Considerations

Numerous specific details are set forth herein to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter may be practiced without these specific details. In other instances, methods, apparatuses, or systems that would be known by one of ordinary skill have not been described in detail so as not to obscure claimed subject matter.

Unless specifically stated otherwise, it is appreciated that throughout this specification discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining,” and “identifying” or the like refer to actions or processes of a computing device, such as one or more computers or a similar electronic computing device or devices, that manipulate or transform data represented as physical electronic or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the computing platform.

The system or systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device can include any suitable arrangement of components that provide a result conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems accessing stored software that programs or configures the computer system from a general purpose computing apparatus to a specialized computing apparatus implementing one or more embodiments of the present subject matter. Any suitable programming, scripting, or other type of language or combinations of languages may be used to implement the teachings contained herein in software to be used in programming or configuring a computing device.

Embodiments of the methods disclosed herein may be performed in the operation of such computing devices. The order of the blocks presented in the examples above can be varied— for example, blocks can be re-ordered, combined, and/or broken into sub-blocks. Certain blocks or processes can be performed in parallel.

The use of “adapted to” or “configured to” herein is meant as an open and inclusive language that does not foreclose devices adapted to or configured to perform additional tasks or steps. Where devices, systems, components or modules are described as being configured to perform certain operations or functions, such configuration can be accomplished, for example, by designing electronic circuits to perform the operation, by programming programmable electronic circuits (such as microprocessors) to perform the operation such as by executing computer instructions or code, or processors or cores programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes can communicate using a variety of techniques including but not limited to conventional techniques for inter-process communications, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

Additionally, the use of “based on” is meant to be open and inclusive, in that, a process, step, calculation, or other action “based on” one or more recited conditions or values may, in practice, be based on additional conditions or values beyond those recited. Headings, lists, and numbering included herein are for ease of explanation only and are not meant to be limiting.

While the present subject matter has been described in detail with respect to specific embodiments thereof, it will be appreciated that those skilled in the art, upon attaining an understanding of the foregoing, may readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, it should be understood that the present disclosure has been presented for purposes of example rather than limitation, and does not preclude the inclusion of such modifications, variations, and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art.