Patent ID: 12197496

DETAILED DESCRIPTION

Overview

A digital image repository includes many digital images (e.g., millions of digital images) which are available (e.g., via a network) to a user of the repository for inclusion in digital content being created or edited by the user. In order to search for digital images included the digital image repository (e.g., to use in the digital content), conventional systems are limited to performing natural language searches or image-based searches. The image-based searches require an input digital image in order to search for the digital images which may or may not be available. The natural language searches match words and phrases of a search input specified by the user with keywords and phrases (e.g., tags) described by metadata of the digital images included in the digital image repository. However, the natural language searches often return result digital images which are irrelevant and fail to capture a semantic intent of the search input specified by the user which is a limitation of conventional systems.

In order to overcome this limitation, techniques and systems for searching for images using generated images are described. In an example, a computing device implements a search system to receive a text search query (e.g., a natural language search query) for digital images included in a digital image repository. For example, the search system generates a set of prompts for a first machine learning model by processing the natural language search query using a second machine learning model.

In one example, the second machine learning model is trained on training data to generate prompts for machine learning models based on text queries. In this example, the second machine learning model includes a natural language model such as a bidirectional encoder representations from transformers model. In some examples, the training data used to train the second machine learning model includes examples of training prompts having semantic intents which cause the first machine learning model to generate high-quality digital images that depict visual features which correspond to the semantic intents of the training prompts. Accordingly, in these examples, the second machine learning model learns (e.g., as part of training on the training data) to generate prompts which cause the first machine learning model to generate digital images which depict diverse visual features that correspond to semantic intents of natural language search queries.

In an example, the search system implements the first machine learning model to generate a set of digital images based on the set of prompts. For example, the first machine learning model is trained on training data to generate digital images based on prompts or natural language inputs. In some examples, the first machine learning model is a generative machine learning model such as a diffusion model. The search system performs an image-based search for digital images included in the digital image repository using the set of digital images. For instance, the search system receives a search result based on the image-based search that includes result digital images.

In some examples, the search system groups the result digital images into clusters based on perceptual similarities computed for the result digital images. In these examples, the search system computes the perceptual similarities using a learned perceptual image patch similarity loss. For instance, a first result digital image is perceptually similar to a second result digital image if the first and second result digital images are included in a same one of the clusters. Conversely, the first result digital image is not perceptually similar to the second result digital image if the first and second result digital images are included in different ones of the clusters. The search system generates an indication of the search result for display in a user interface which arranges the result digital images in an order based on the clusters. For example, digital images included in a largest one of the clusters are display first, the order displays result digital images interleaved from the largest one of the clusters and a next largest one of the clusters, etc.

Consider an example in which the search system leverages a characteristic of the digital content being created or edited by the user to determine the order for displaying the result digital images in the user interface. In this example, the search system leverages an aspect ratio of an example digital image included in the digital content and aspect ratios of the result digital images in order to display particular result digital images having aspect ratios similar to the aspect ratio of the example digital image first in the order. In another example, the search system leverages a color distribution of the digital content and color distributions of the result digital images in order to display specific result digital images having color distributions similar to the color distribution of the digital content first in the order.

By leveraging the second machine learning model to generate the set of prompts based on the natural language search input and using the set of digital images generated by the first machine learning model based on the set prompts to perform the image-based search, the search system is capable of displaying result digital images from the digital image repository that depict diverse visual features which correspond to a semantic intent of the natural language search input. This is not possible in conventional systems which are limited to displaying result digital images that are not relevant to the semantic intent of the natural language search input. In addition to reflecting the semantic intent of the natural language search input, result digital images (e.g., from the digital image repository) displayed using the described systems for searching for images using generated images are of a higher quality (e.g., created by a professional) than the generated digital images which include artifacts.

In the following discussion, an example environment is first described that employs examples of techniques described herein. Example procedures are also described which are performable in the example environment and other environments. Consequently, performance of the example procedures is not limited to the example environment and the example environment is not limited to performance of the example procedures.

Example Environment

FIG.1is an illustration of an environment100in an example implementation that is operable to employ digital systems and techniques as described herein. The illustrated environment100includes a computing device102connected to a network104. The computing device102is configurable as a desktop computer, a laptop computer, a mobile device (e.g., assuming a handheld configuration such as a tablet or mobile phone), and so forth. Thus, the computing device102is capable of ranging from a full resource device with substantial memory and processor resources (e.g., personal computers, game consoles) to a low-resource device with limited memory and/or processing resources (e.g., mobile devices). In some examples, the computing device102is representative of a plurality of different devices such as multiple servers utilized to perform operations “over the cloud.”

The illustrated environment100also includes a display device106that is communicatively coupled to the computing device102via a wired or a wireless connection. A variety of device configurations are usable to implement the computing device102and/or the display device106. For example, the computing device102includes a storage device108and a search module110. Although the search module110is illustrated to be included in the computing device102, it is to be appreciated that, in some examples, the search module110is included in a remote computing device such as a virtual computing device which exposes functionality of the search module110as a service via the network104. In one example, some functionality of the search module110is exposed as the service via the network104while other functionality of the search module110is included in modules of the computing device102. The storage device108is illustrated to include digital content112such as digital images, digital templates, digital artwork, digital videos, etc.

The search module110is illustrated as having, receiving, and/or transmitting input data114describing a natural language search input116of “a stairway surrounded by mist.” For example, a user interacts with an input device (e.g., a mouse, a keyboard, a microphone, a stylus, a touchscreen, etc.) to generate the input data114by specifying the natural language search input116for digital images included in a digital image repository available via the network104. In this example, the digital image repository includes millions of digital images such as “stock” photographs, graphic art, illustrations, vector objects, icons, etc.

The digital images included in the digital image repository are “tagged” with keywords and phrases which describe the digital images, e.g., the digital images include metadata describing the keywords and phrases. For example, the keywords and phrases describe objects depicted in the digital images, themes or intents of the digital images, visual features of the digital images (e.g., colors depicted, filters applied, modifications/edits applied, etc.), and so forth. Consider an example in which it is possible to identify particular digital images included in the digital image repository by matching (e.g., semantically matching) words and phrases included in the natural language search input116with keywords and phrases described by metadata of the particular digital images.

However, searching for digital images included in the digital image repository by matching terms of the natural language search input116with tagged keywords of the digital images produces a search result that only includes nine particular digital images (of the millions of digital images included in the digital image repository) which do not match a semantic intent of the natural language search input116. For instance, only four of the nine particular digital images depict stairs and only one of the particular digital images depicts mist. In order to generate search results including many digital images that match the semantic intent of the natural language search input116, the search module110receives and processes the input data114to generate prompts for a first machine learning model. In an example, the search module110generates the prompts for the first machine learning model by processing the natural language search input116using a second machine learning model.

As used herein, the term “machine learning model” refers to a computer representation that is tunable (e.g., trainable) based on inputs to approximate unknown functions. By way of example, the term “machine learning model” includes a model that utilizes algorithms to learn from, and make predictions on, known data by analyzing the known data to learn to generate outputs that reflect patterns and attributes of the known data. According to various implementations, such a machine learning model uses supervised learning, semi-supervised learning, unsupervised learning, reinforcement learning, and/or transfer learning. For example, the machine learning model is capable of including, but is not limited to, clustering, decision trees, support vector machines, linear regression, logistic regression, Bayesian networks, random forest learning, dimensionality reduction algorithms, boosting algorithms, transformers, artificial neural networks (e.g., fully-connected neural networks, deep convolutional neural networks, or recurrent neural networks), deep learning, autoregressive models, etc. By way of example, a machine learning model makes high-level abstractions in data by generating data-driven predictions or decisions from the known input data.

Notably, the first machine learning model and the second machine learning model are included in or available to the search module110(e.g., via the network104. In one example, the second machine learning model is included in the search module110and the first machine learning model is included in a remote computing device such as a virtual computing device which is available to the search module110via the network104. In another example, the first machine learning model is included in the search module110and the second machine learning model is included in the remote computing device such as the virtual computing device available to the search module110via the network104.

In an example, the second machine learning model is trained on training data to generate prompts for machine learning models based on natural language search inputs. For example, the second machine learning model includes a bidirectional encoder representations from transformers model (e.g., a BERT model). The first machine learning model is a generative machine learning model trained on large amounts of training data to generate digital images based on natural language descriptions of the digital images. In one example, the first machine learning model includes a diffusion model.

Consider an example in which the training data used to train the second machine learning model includes example inputs to the first machine learning model that cause the first machine learning model to generate high quality digital images that correspond to semantic intents of the example inputs. In this example, the second machine learning model learns to generate prompts which cause the first machine learning model to generate digital images that correspond to semantic intents of the prompts as part of training the second machine learning model on the training data. In an example, the search module110implements the second machine learning model to process the input data114in order to generate a prompt for the first machine learning model such as “stairs leading up to mist at the top of the stairs, shot from below, fog at the top of the stairs.”

For instance, the search module110then implements the first machine learning model to process the generated prompt in order to generate generated data118. As shown, the generated data118describes a digital image120which was generated by the first machine learning model based on the prompt that was generated by the second machine learning model. The digital image120depicts a stairway covered in mist which matches the semantic intent of the natural language search input116.

In some examples, the search module110implements the second machine learning model to generate additional prompts based on the search input116such as “flight of stairs, photograph, mist around and above the stairs.” In these examples, the search module110implements the first machine learning model to generate additional digital images based on processing the additional prompts. The search module110uses the digital image120and the additional digital images to perform image-based searches for digital images included in the digital image repository. In an example, the image-based searches compare visual features of the digital image120and visual features of the additional digital images with visual features of the digital images included in the digital image repository in order return search results.

For example, the search module110receives a search result of the image-based search performed using the digital image120and search results of the image-based searches performed using the additional digital images. The search results include result digital images, and the search module110groups the result digital images into groups or clusters. To do so in one example, the search module110generates a latent representation of each of the result digital images and groups the latent representations into the clusters based on perceptual similarities or distances. For example, first and second latent representations corresponding to first and second result digital images are included in a same one of the clusters if the first and second result digital images are perceptually similar and the first and second latent representations are each included in a different one of the clusters if the first and second result digital images are not perceptually similar.

In an example, the search module110leverages the clusters in order to generate an indication of a search result for display in a user interface122of the display device106. As shown, the indication includes result digital images124-130that are included in the digital image repository and correspond to the semantic intent of the of the natural language search input116. For instance, result digital image124depicts a curved staircase that is shrouded in mist; result digital image126depicts a flight of stairs in a misty forest; result digital image128depicts a pier of stairs extending over water and below mist or fog; and result digital image130depicts a stairway extending up into a distant mist.

In some examples, the search module110generates the indication of the search result by arranging the result digital images124-130in an order to be displayed in the user interface122based on the clusters. In a first example, the search module110identifies a cluster of the clusters which includes a greatest number of the latent representations. In the first example, the result digital images124-130each have a corresponding latent representation that is included in the identified cluster, and the search module110displays additional result digital images having corresponding latent representations in the identified cluster before displaying other result digital images having corresponding latent representations included in other clusters of the clusters.

In a second example, result digital images124-130each have a corresponding latent representation that is included in one of the other clusters. In the second example, the result digital image124has a first latent representation that is included in a first cluster; the result digital image126has a second latent representation that is included in a second cluster; the result digital image128has a third latent representation that is included in a third cluster; and the result digital image130has a fourth latent representation that is included in a fourth cluster. For example, the search module110includes the result digital images124-130as part of the indication of the search result because each of the result digital images124-130is representative of one of the first, second, third, and fourth clusters.

Consider an example in which the search module110generates the indication of the search result by arranging the result digital images124-130in an order to be displayed in the user interface122based on a characteristic of the digital content112. For instance, the digital content112includes a digital template which the user is editing via interactions with the input device. If the digital template includes an example digital image having a particular aspect ratio, then the search module110arranges the result digital images124-130first in the order because the result digital images124-130also have the particular aspect ratio (or approximately the particular aspect ratio). If the digital template has a particular color distribution, then the search module110arranges the result digital images124-130first in the order because the result digital images124-130also have the particular color distribution (or approximately the particular color distribution).

By leveraging the second machine learning model to generate prompts for the first machine learning model based on the natural language search input116, and by performing image-based searches of digital images included in the digital image repository using digital images generated by the first machine learning model, the search module110is capable of identifying and displaying result digital images that match the semantic intent of the natural language search input116such as the result digital images124-130. This is not possible in conventional systems that are limited to searching for digital images based on keywords and phrases (e.g., tags) described by metadata of the digital images. Conventional systems are also not capable of arranging the result digital images124-130in the order based on the characteristic of the digital content112which is a further limitation of the conventional systems relative to the described systems for searching for images using generated images.

FIG.2depicts a system200in an example implementation showing operation of a search module110. The search module110is illustrated to include a prompt module202, a generation module204, an image search module206, and a display module208. The search module110receives the digital content112and the input data114. For example, the search module110also receives repository data210describing digital images included in a digital image repository. In an example, the prompt module202receives and processes the input data114in order to generate prompt data212.

FIG.3illustrates a representation300of a natural language search input and prompts generated for a machine learning model. The representation300includes a natural language search input302of “green trees forest with snow.” For example, the prompt module202receives the input data114describing the natural language search input302. In an example, the prompt module202includes or has access to the second machine learning model that is trained on training data to generate prompts for machine learning models based on natural language search inputs. For instance, the second machine learning model is included in the computing device102or the second machine learning model is included in a remote computing device that is available to the computing device102via the network104. The second machine learning model includes a BERT model in some examples.

The prompt module202implements the second machine learning model to process the input data114in order to generate prompts304-312for the first machine learning model. In some examples, the prompt module202represents the natural language search input302as an input query Q (0) and the prompt module202represents each of the prompts304-312as an output query Q(i) such that a set of the prompts304-312is representable as:
Qs={Q1,Q2, . . . ,Qn}
where: Qsdenotes a set of prompts generated for the first machine learning model.

In one example, the prompt module202generates the prompt data212as describing the prompts304-312. As shown, prompt304is “a forest filled with lots of trees covered in snow, a digital rendering, green alley, merging with tree in a forest, exotic trees;” prompt306is “a forest filled with lots of trees covered in snow, environmental art, monochromatic green;” prompt308is “a forest filled with lots of trees covered in snow, tonalism, green charts, creative commons attribution, enchanted, path into lush forest;” prompt310is “a snow covered forest filled with lots of trees, an ambient occlusion render, environment art, shot from behind blades of grass, green rain;” and prompt312is “a group of trees that are covered in snow, gradient green, black, tall pine trees.”

The generation module204receives the prompt data212describing the prompts304-312for the first machine learning model. In an example, the first machine learning model is included in or available to the generation module204. In this example, the first machine learning model includes the generative machine learning model trained on training data to generate digital images based on natural language descriptions of the digital images. For example, the first machine learning model is included in the computing device102or the first machine learning model is included in a remote computing device that is available to the computing device102via the network104. Examples of generative machine learning models included in the first machine learning model include a diffusion model, a Generative Pre-Trained Transformer 4 model (GPT-4), a Hierarchical Text-Conditional Image Generation with CLIP Latents model (DALL⋅E 2), etc. In some examples, the first machine learning model includes systems of generative machine learning models.

For instance, the generation module204implements the first machine learning model to process the prompt data212in order to generate digital images314-322. In an example, a set of the digital images314-322is representable as:
Gs={G1,G2, . . . ,Gn}
where: Gsdenotes a set of digital images generated based on the generated set of prompts Qs.

The generation module204generates digital image314by processing the prompt304using the first machine learning model. Similarly, the generation module204implements the first machine learning model to generate digital image316based on the prompt306; digital image318based on the prompt308; digital image320based on the prompt310; and digital image322based on the prompt312.

For example, the generation module204generates the generated data118as describing the digital images314-322. In this example, the image search module206receives and processes the generated data118and the repository data210in order to generate result data214.FIG.4illustrates a representation400of search results based on performing image-based searches for digital images included in a digital image repository.

As shown inFIG.4, the representation400includes search results402-410. For instance, the image search module206performs image-based searches of digital images included in the digital image repository as described by the repository data210using the digital images314-322generated by the first machine learning model based on processing the prompts304-312. In one example, the image search module206performs an image-based search of the digital images included in the digital image repository using the digital image314in order to identify search result402which includes a first set of result digital images.

For example, the image search module206identifies search result404which includes a second set of result digital images by performing an image-based search of the digital images included in the digital image repository using the digital image316. Similarly, the image search module206identifies search result406which includes a third set of result digital images by performing an image-based search of the digital images included in the digital image repository using the digital image818; the image search module206identifies search result408which includes a fourth set of result digital images by performing an image-based search of the digital images included in the digital image repository using the digital image320; and the image search module206identifies search result410which includes a fifth set of result digital images by performing an image-based search of the digital images included in the digital image repository using the digital image322. In an example, the image search module206generates the result data214as describing the search results402-410.

FIG.5illustrates a representation500of grouping result digital images into clusters. The display module208receives the result data214and the digital content112, and the display module208processes the result data214in order to group result digital images included in the first, second, third, fourth, and fifth sets of result digital images into clusters502-510. To do so in one example, the display module208generates latent representations (e.g., embeddings) for each of the digital images314-322, and assigns one of the latent representations of the digital images314-322to each of the clusters502-510. Accordingly, in this example, a number of the clusters502-510is equal to a number of the digital images314-322. However, it is to be appreciated that in other examples, the number of the clusters502-510is greater than or less than the number of the digital images314-322.

For example, the display module208generates latent representations (e.g., embeddings) for each of the result digital images included in the first, second, third, fourth, and fifth sets of result digital images, and then groups these latent representations into the clusters502-510(e.g., using k-means clustering). In this example, the display module208groups the latent representations of the result digital images into the clusters502-510based on perceptual similarities computed for the result digital images as described by Zhang et al., The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, arXiv: 1801.03924v2 [cs.CV] (Apr. 10, 2018). In an example, the display module208computes the perceptual similarities using a learned perceptual image patch similarity loss.

As shown, cluster502includes a greatest number of the result digital images (e.g., a greatest number of latent representations of the result digital images). As further shown, result digital images having latent representations included in the cluster502are perceptually similar to the digital image320. For example, the cluster502includes some of the result digital images included in the first, second, third, fourth, and fifth sets of result digital images. In another example, the cluster502includes some result digital images included in the fourth set of result digital images from the search result408and the cluster502also includes some result digital images included in the fifth set of result digital images from the search result410.

In the illustrated example, cluster504includes a second greatest number of the result digital images (e.g., a second greatest number of latent representations of the result digital images). For instance, result digital images having latent representations included in the cluster504are perceptually similar to the digital image322. Cluster506includes a third greatest number of the result digital images (e.g., a third greatest number of latent representations of the result digital images) and result digital images having latent representations included in the cluster506are perceptually similar to the digital image314. Cluster508includes a fourth greatest number of the result digital images (e.g., a fourth greatest number of latent representations of the result digital images). As shown, result digital images having latent representations included in the cluster508are perceptually similar to the digital image318. Finally, cluster510includes a fifth greatest number of the result digital images (e.g., a lowest number of latent representations of the result digital images), and result digital images having latent representations included in the cluster510are perceptually similar to the digital image316.

Consider an example in which the display module208leverages the clusters502-510and an optional diversity input dn described by the input data114to interleave the search results402-410for display in the user interface122. In this example, in response to receiving the input data114describing a diversity input dn=1, the display module208displays result digital images having latent representations in the cluster502(which has the greatest number of latent representations of the result digital images) first and then displays result digital images having latent representations in the cluster504, the cluster506, the cluster508, and the cluster510. For example, in response to receiving the input data114describing a diversity input dn=2, the display module208interleaves result digital images having latent representations in the cluster502and result digital images having latent representations in the cluster504for display in the user interface122. In a similar example, in response to receiving the input data114describing a diversity input dn≤5, the display module208interleaves result digital images having latent representations in dn largest ones of the clusters502-510for display in the user interface122first and then displays result digital images having latent representations in remaining ones of the clusters502-510(if any). Although examples are described relative to an example in which a number of the clusters502-510is equal to five, it is to be appreciated that the described examples are scalable to any number of clusters.

FIG.6illustrates a representation600of an indication of a search result generated for display in a user interface. For example, the display module208generates an indication602of a search result based on the natural language search input302. As shown, the indication602includes result digital images604-618arranged in an order based on the clusters502-510and a diversity input dn. In one example, if the display module208receives the input data114describing a diversity input dn=1, then the result digital images604-618have latent representations included in the cluster502. In another example, if the display module208receives the input data114describing a diversity input dn=2, then some of the result digital images604-618have latent representations included in the cluster502and other ones of the result digital images604-618have latent representations included in the cluster504.

FIGS.7A and7Billustrate examples of indications of search results generated based on characteristics of digital content112.FIG.7Aillustrates a representation700of a search result generated based on an aspect ratio of a digital image to be included in digital content112.FIG.7Billustrates a representation702of a search result generated based on a color distribution of digital content112.

With reference toFIG.7A, the display module208receives the digital content112as including a digital template704. For example, a user interacts with an input device (e.g., a mouse, a keyboard, a touchscreen, a stylus, etc.) to modify the digital template704by searching for digital images included in the digital image repository to replace an example digital image706included in the digital template704. In this example, the display module208defines an aspect ratio of the example digital image706as Rorigand defines an aspect ratio of an output digital image included in a search result as Rirsi. The display module208then computes an aspect ratio penalty as:

Aspect⁢Ratio⁢Penalty=Ri⁢r⁢s⁢iRo⁢r⁢i⁢g⁢if⁢Ri⁢r⁢s⁢i>Ro⁢r⁢g⁢else=Ro⁢r⁢i⁢gRi⁢r⁢s⁢i

With reference toFIG.7B, the display module208receives the digital content112as including a digital template708. In one example, the display module208defines a color distribution of the digital template708as CTorigand defines a color distribution of an output digital image included in a search result as CTirsi. For example, the display module208computes a Color Harmoney Penalty as being equal to an angle of rotation between a primary axis of CTorigand CTirsiin a Hue Saturation Value (HSV) color space.

Consider an example in which the display module208computes a Normalized Contextual Penalty as:

Normalized Contextual Penalty

Normalized⁢Contextual⁢Penalty=α*Aspect⁢Ratio⁢Penalty+β*Color⁢Harmony⁢Penaltyα+β
where: α=2 and β=1 to prioritize aspect ratio over color distribution.

Continuing the above example, the display module208computes a final ranking score for ordering result digital images in an indication of a search result as:

Final⁢Ranking⁢Score=k⁢0ϵ+Normalized⁢Contextual⁢Penalty
where: k0 is a constant value and ∈ is a small constant value for numerical stability.

Accordingly, by leveraging the second machine learning model to generate the prompts304-312for the first machine learning model based on the natural language search input302, and by performing image-based searches of digital images included in the digital image repository using the digital images314-322generated by the first machine learning model, the search module110is capable of identifying and displaying result digital images that match the semantic intent of the natural language search input302such as the result digital images604-618. This is not possible in conventional systems that are limited to searching for digital images based on keywords and phrases (e.g., tags) described by metadata of the digital images. Conventional systems are also not capable of arranging the result digital images604-618in the order based on the characteristic of the digital content112which is a further limitation of the conventional systems relative to the described systems for searching for images using generated images. Moreover, by displaying the result digital images604-618based on the Final Ranking Score, an amount of modification of one of the result digital images604-618to replace the example digital image706is minimized and ones of the result digital images604-618having similar color distributions to the color distribution of the digital template708are displayed first in a displayed order of the result digital images604-618.

In general, functionality, features, and concepts described in relation to the examples above and below are employed in the context of the example procedures described in this section. Further, functionality, features, and concepts described in relation to different figures and examples in this document are interchangeable among one another and are not limited to implementation in the context of a particular figure or procedure. Moreover, blocks associated with different representative procedures and corresponding figures herein are applicable individually, together, and/or combined in different ways. Thus, individual functionality, features, and concepts described in relation to different example environments, devices, components, figures, and procedures herein are usable in any suitable combinations and are not limited to the particular combinations represented by the enumerated examples in this description.

Example Procedures

The following discussion describes techniques which are implementable utilizing the previously described systems and devices. Aspects of each of the procedures are implementable in hardware, firmware, software, or a combination thereof. The procedures are shown as a set of blocks that specify operations performed by one or more devices and are not necessarily limited to the orders shown for performing the operations by the respective blocks. In portions of the following discussion, reference is made toFIGS.1-6.FIG.8is a flow diagram depicting a procedure800in an example implementation in which a set of digital images is generated using a machine learning model based on a natural language search query.

A natural language search query for digital images included in a digital image repository is received (block802). For example, the computing device102implements the search module110to receive the natural language search query. A set of digital images is generated using a machine learning model based on the natural language search query (block804), the machine learning model is trained on training data to generate digital images based on natural language inputs. In an example, the search module110generates the set of digital images using the machine learning model.

An image-based search for digital images included in the digital image repository is performed using the set of digital images (block806). In one example, the computing device102implements the search module110to perform the image-based search. An indication of a search result is generated for display in a user interface based on performing the image-based search (block808). The search module110generates the indication of the search result in some examples.

FIG.9is a flow diagram depicting a procedure900in an example implementation in which a set of prompts for a first machine learning model is generated by processing a natural language search query using a second machine learning model. A natural language search query for digital images included in a digital image repository is received (block902). For example, the search module110receives the natural language search query. A set of prompts is generated for a first machine learning model by processing the natural language search query using a second machine learning model (block904). In some examples, the computing device102implements the search module110to generate the set of prompts for the first machine learning model.

A set of digital images is generated by processing the set of prompts using the first machine learning model (block906). In one example, the search module110generates the set of digital images by processing the set of prompts using the first machine learning model. A search result is received based on performing an image-based search for digital images included in the digital image repository using the set of digital images (block908). For example, the search module110receives the search result.

FIGS.10A,10B, and10Cillustrate examples of searching for images using generated images.FIG.10Aillustrates a representation1000of a first example of result digital images identified based on a first natural language search input.FIG.10Billustrates a representation1002of a second example of result digital images identified based on a second natural language search input.FIG.10Cillustrates a representation1004of a third example of result digital images identified based on a third natural language search input.

With reference toFIG.10A, the representation1000includes result digital images1006identified by searching the digital images included in the digital image repository using conventional systems as well as result digital images1008identified by searching the digital images included in the digital image repository using the described systems for searching for images using generated images based on a natural language search input of “isolated iron lighthouse sunset.” The representation1002illustrated inFIG.10Bincludes result digital images1010identified by searching the digital images included in the digital image repository using conventional systems and also result digital images1012identified by searching the digital images included in the digital image repository using the described systems for searching for images using generated images based on a natural language search input of “a group of people sitting having dinner under lights.” With reference toFIG.10C, the representation1004includes result digital images1014identified by searching the digital images included in the digital image repository using conventional systems as well as result digital images1016identified by searching the digital images included in the digital image repository using the described systems for searching for images using generated images based on a natural language search input of “blue flower on the edge of the brown mountain.”

FIG.11shows an example of a pixel diffusion model1100according to implementations of the present disclosure. The example shown includes pixel diffusion model1100, image1105, pixel space1110, forward diffusion process1115, noisy image1120, reverse diffusion process1125, final output image1130, text prompt1135, text encoder1140, guidance features1145, and guidance space1150.

Diffusion models are a class of generative neural networks which can be trained to generate new data with features similar to features found in training data. In particular, diffusion models can be used to generate novel images. Diffusion models can be used for various image generation tasks including image super-resolution, generation of images with perceptual metrics, conditional generation (e.g., generation based on text guidance), image inpainting, and image manipulation.

Types of diffusion models include DDPMs and DDIMs. In DDPMs, the generative process includes reversing a stochastic Markov diffusion process. DDIMs, on the other hand, use a deterministic process so that the same input results in the same output. Diffusion models may also be characterized by whether the noise is added to the image itself, or to image features generated by an encoder (i.e., latent diffusion).

Diffusion models work by iteratively adding noise to the data during a forward process and then learning to recover the data by denoising the data during a reverse process. For example, during training, pixel diffusion model1100may take an original image1105in a pixel space1110as input and apply forward diffusion process1115to gradually add noise to the original image1105to obtain noisy images1120at various noise levels.

Next, a reverse diffusion process1125(e.g., a U-Net ANN) gradually removes the noise from the noisy images1120at the various noise levels to obtain an output image1130. In some cases, an output image1130is created from each of the various noise levels. The output image1130can be compared to the original image1105to train the reverse diffusion process1125.

The reverse diffusion process1125can also be guided based on a text prompt1135, or another guidance prompt, such as an image, a layout, a segmentation map, etc. The text prompt1135can be encoded using a text encoder1140(e.g., a multi-modal encoder) to obtain guidance features1145in guidance space1150. The guidance features1145can be combined with the noisy images1120at one or more layers of the reverse diffusion process1125to ensure that the output image1130includes content described by the text prompt1135. For example, guidance features1145can be combined with the noisy features using a cross-attention block within the reverse diffusion process1125.

In the machine learning field, attention mechanism is a method of placing differing levels of importance on different elements of an input. Calculating attention may involve three basic steps. First, a similarity between query and key vectors obtained from the input is computed to generate attention weights. Similarity functions used for this process can include dot product, splice, detector, and the like. Next, a softmax function is used to normalize the attention weights. Finally, the attention weights are weighed together with their corresponding values.

Forward diffusion process1115is an example of, or includes aspects of, the corresponding element described with reference toFIG.14. Noisy image1120is an example of, or includes aspects of, the corresponding element described with reference toFIG.14. Reverse diffusion process1125is an example of, or includes aspects of, the corresponding element described with reference toFIG.14.

FIG.12shows an example of U-net1200architecture according to implementations of the present disclosure. The example shown includes U-Net1200, input features1205, initial neural network layer1210, intermediate features1215, down-sampling layer1220, down-sampled features1225, up-sampling layer1230, up-sampled features1235, skip connection1240, final neural network layer1245, and output features1250. The U-Net1200depicted inFIG.12is an example of, or includes aspects of, the architecture used within the reverse diffusion process described with reference toFIG.11.

In some examples, diffusion models are based on a neural network architecture known as a U-Net. The U-Net1200takes input features1205having an initial resolution and an initial number of channels and processes the input features1205using an initial neural network layer1210(e.g., a convolutional network layer) to produce intermediate features1215. The intermediate features1215are then down-sampled using a down-sampling layer1220such that down-sampled features1225features have a resolution less than the initial resolution and a number of channels greater than the initial number of channels.

This process is repeated multiple times, and then the process is reversed. That is, the down-sampled features1225are up-sampled using up-sampling process1230to obtain up-sampled features1235. The up-sampled features1235can be combined with intermediate features1215having a same resolution and number of channels via a skip connection1240. These inputs are processed using a final neural network layer1245to produce output features1250. In some cases, the output features1250have the same resolution as the initial resolution and the same number of channels as the initial number of channels.

In some cases, U-Net1200takes additional input features to produce conditionally generated output. For example, the additional input features could include a vector representation of an input prompt. The additional input features can be combined with the intermediate features1215within the neural network at one or more layers. For example, a cross-attention module can be used to combine the additional input features and the intermediate features1215.

FIG.13shows an example of reverse diffusion according to implementations of the present disclosure. The example shown includes diffusion process1300, forward diffusion process1305, reverse diffusion process1310, noisy image1315, first intermediate image1320, second intermediate image1325, and original image1330. Forward diffusion process1305is an example of, or includes aspects of, the corresponding element described with reference toFIG.11. Reverse diffusion process1310is an example of, or includes aspects of, the corresponding element described with reference toFIG.11. Noisy image1315is an example of, or includes aspects of, the corresponding element described with reference toFIG.11.

As described above with reference toFIG.11, a diffusion model includes a forward diffusion process1305for adding noise to an image (or features in a latent space) and a reverse diffusion process1310for denoising the images (or features) to obtain a denoised image. The forward diffusion process1305can be represented as q(xt|xt-1), and the reverse diffusion process1310can be represented as p(xt-1|xt). In some cases, the forward diffusion process1305is used during training to generate images with successively greater noise, and a neural network is trained to perform the reverse diffusion process1310(i.e., to successively remove the noise).

In an example forward process for a latent diffusion model, the model maps an observed variable x0(either in a pixel space or a latent space) intermediate variables x1, . . . , xTusing a Markov chain. The Markov chain gradually adds Gaussian noise to the data to obtain the approximate posterior q (x1:T|x0) as the latent variables are passed through a neural network such as a U-Net, where x1, . . . , xThave the same dimensionality as x0.

The neural network may be trained to perform the reverse process. During the reverse diffusion process1310, the model begins with noisy data xT, such as a noisy image1315and denoises the data to obtain the p(xt-1|xt). At each step t−1, the reverse diffusion process1310takes xt, such as first intermediate image1320, and t as input. Here, t represents a step in the sequence of transitions associated with different noise levels, The reverse diffusion process1310outputs xt-1, such as second intermediate image1125iteratively until xTis reverted back to x0, the original image1130. The reverse process can be represented as:
pθ(xt-1|xt):=N(xt-1;μθ(xt,t),τθ(xt,t).  (1)

The joint probability of a sequence of samples in the Markov chain can be written as a product of conditionals and the marginal probability:

xT:pθ(x0:T):=p⁡(xT)⁢∏t=1T⁢pθ(xt-1|xt),(2)
where p(xt)=N(xT; 0, I) is the pure noise distribution as the reverse process takes the outcome of the forward process, a sample of pure noise, as input and Πt=1Tpθ(xt-1|xt) represents a sequence of Gaussian transitions corresponding to a sequence of addition of Gaussian noise to the sample.

At interference time, observed data x0in a pixel space can be mapped into a latent space as input and a generated data {tilde over (x)} is mapped back into the pixel space from the latent space as output. In some examples, x0represents an original input image with low image quality, latent variables x1, . . . , xTrepresent noisy images, and {tilde over (x)} represents the generated image with high image quality.

FIG.14shows an example of a procedure1400for training a diffusion model according to embodiments of the present disclosure. The procedure1400represents an example for training a reverse diffusion process as described above with reference toFIG.13.

Additionally or alternatively, certain processes of procedure1400may be performed using special-purpose hardware. Generally, these operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various substeps, or are performed in conjunction with other operations.

At operation1405, the system initializes an untrained model. Initialization can include defining the architecture of the model and establishing initial values for the model parameters. In some cases, the initialization can include defining hyper-parameters such as the number of layers, the resolution and channels of each layer blocks, the location of skip connections, and the like.

At operation1410, the system adds noise to a training image using a forward diffusion process in N stages. In some cases, the forward diffusion process is a fixed process where Gaussian noise is successively added to an image. In latent diffusion models, the Gaussian noise may be successively added to features in a latent space.

At operation1415, the system at each stage n, starting with stage N, a reverse diffusion process is used to predict the image or image features at stage n−1. For example, the reverse diffusion process can predict the noise that was added by the forward diffusion process, and the predicted noise can be removed from the image to obtain the predicted image.

At operation1420, the system compares predicted image (or image features) at stage n−1 to an actual image (or image features), such as the image at stage n−1 or the original input image. For example, given observed data x, the diffusion model may be trained to minimize the variational upper bound of the negative log-likelihood−log pθ(x) of the training data.

At operation1425, the system updates parameters of the model based on the comparison. For example, parameters of a U-Net may be updated using gradient descent. Time-dependent parameters of the Gaussian transitions can also be learned.

Example System and Device

FIG.15illustrates an example system1500that includes an example computing device that is representative of one or more computing systems and/or devices that are usable to implement the various techniques described herein. This is illustrated through inclusion of the search module150. The computing device1502includes, for example, a server of a service provider, a device associated with a client (e.g., a client device), an on-chip system, and/or any other suitable computing device or computing system.

The example computing device1502as illustrated includes a processing system1504, one or more computer-readable media1506, and one or more I/O interfaces1508that are communicatively coupled, one to another. Although not shown, the computing device1502further includes a system bus or other data and command transfer system that couples the various components, one to another. For example, a system bus includes any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or local bus that utilizes any of a variety of bus architectures. A variety of other examples are also contemplated, such as control and data lines.

The processing system1504is representative of functionality to perform one or more operations using hardware. Accordingly, the processing system1504is illustrated as including hardware elements1510that are configured as processors, functional blocks, and so forth. This includes example implementations in hardware as an application specific integrated circuit or other logic device formed using one or more semiconductors. The hardware elements1510are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, processors are comprised of semiconductor(s) and/or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor-executable instructions are, for example, electronically-executable instructions.

The computer-readable media1506is illustrated as including memory/storage1512. The memory/storage1512represents memory/storage capacity associated with one or more computer-readable media. In one example, the memory/storage1512includes volatile media (such as random access memory (RAM)) and/or nonvolatile media (such as read only memory (ROM), Flash memory, optical disks, magnetic disks, and so forth). In another example, the memory/storage1512includes fixed media (e.g., RAM, ROM, a fixed hard drive, and so on) as well as removable media (e.g., Flash memory, a removable hard drive, an optical disc, and so forth). The computer-readable media1506is configurable in a variety of other ways as further described below.

Input/output interface(s)1508are representative of functionality to allow a user to enter commands and information to computing device1502, and also allow information to be presented to the user and/or other components or devices using various input/output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone, a scanner, touch functionality (e.g., capacitive or other sensors that are configured to detect physical touch), a camera (e.g., which employs visible or non-visible wavelengths such as infrared frequencies to recognize movement as gestures that do not involve touch), and so forth. Examples of output devices include a display device (e.g., a monitor or projector), speakers, a printer, a network card, tactile-response device, and so forth. Thus, the computing device1502is configurable in a variety of ways as further described below to support user interaction.

Various techniques are described herein in the general context of software, hardware elements, or program modules. Generally, such modules include routines, programs, objects, elements, components, data structures, and so forth that perform particular tasks or implement particular abstract data types. The terms “module,” “functionality,” and “component” as used herein generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that the techniques are implementable on a variety of commercial computing platforms having a variety of processors.

Implementations of the described modules and techniques are storable on or transmitted across some form of computer-readable media. For example, the computer-readable media includes a variety of media that is accessible to the computing device1502. By way of example, and not limitation, computer-readable media includes “computer-readable storage media” and “computer-readable signal media.”

“Computer-readable storage media” refers to media and/or devices that enable persistent and/or non-transitory storage of information in contrast to mere signal transmission, carrier waves, or signals per se. Thus, computer-readable storage media refers to non-signal bearing media. The computer-readable storage media includes hardware such as volatile and non-volatile, removable and non-removable media and/or storage devices implemented in a method or technology suitable for storage of information such as computer readable instructions, data structures, program modules, logic elements/circuits, or other data. Examples of computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, hard disks, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other storage device, tangible media, or article of manufacture suitable to store the desired information and which are accessible to a computer.

“Computer-readable signal media” refers to a signal-bearing medium that is configured to transmit instructions to the hardware of the computing device1502, such as via a network. Signal media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as carrier waves, data signals, or other transport mechanism. Signal media also include any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

As previously described, hardware elements1510and computer-readable media1506are representative of modules, programmable device logic and/or fixed device logic implemented in a hardware form that is employable in some implementations to implement at least some aspects of the techniques described herein, such as to perform one or more instructions. Hardware includes components of an integrated circuit or on-chip system, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), and other implementations in silicon or other hardware. In this context, hardware operates as a processing device that performs program tasks defined by instructions and/or logic embodied by the hardware as well as a hardware utilized to store instructions for execution, e.g., the computer-readable storage media described previously.

Combinations of the foregoing are also employable to implement various techniques described herein. Accordingly, software, hardware, or executable modules are implementable as one or more instructions and/or logic embodied on some form of computer-readable storage media and/or by one or more hardware elements1510. For example, the computing device1502is configured to implement particular instructions and/or functions corresponding to the software and/or hardware modules. Accordingly, implementation of a module that is executable by the computing device1502as software is achieved at least partially in hardware, e.g., through use of computer-readable storage media and/or hardware elements1510of the processing system1504. The instructions and/or functions are executable/operable by one or more articles of manufacture (for example, one or more computing devices1502and/or processing systems1504) to implement techniques, modules, and examples described herein.

The techniques described herein are supportable by various configurations of the computing device1502and are not limited to the specific examples of the techniques described herein. This functionality is also implementable entirely or partially through use of a distributed system, such as over a “cloud”1514as described below.

The cloud1514includes and/or is representative of a platform1516for resources1518. The platform1516abstracts underlying functionality of hardware (e.g., servers) and software resources of the cloud1514. For example, the resources1518include applications and/or data that are utilized while computer processing is executed on servers that are remote from the computing device1502. In some examples, the resources1518also include services provided over the Internet and/or through a subscriber network, such as a cellular or Wi-Fi network.

The platform1516abstracts the resources1518and functions to connect the computing device1502with other computing devices. In some examples, the platform1516also serves to abstract scaling of resources to provide a corresponding level of scale to encountered demand for the resources that are implemented via the platform. Accordingly, in an interconnected device implementation, implementation of functionality described herein is distributable throughout the system1500. For example, the functionality is implementable in part on the computing device1502as well as via the platform1516that abstracts the functionality of the cloud1514.