GENERATING ARTISTIC CONTENT FROM A TEXT PROMPT OR A STYLE IMAGE UTILIZING A NEURAL NETWORK MODEL

The present disclosure relates to systems, methods, and non-transitory computer readable media that utilize an iterative neural network framework for generating artistic visual content. For instance, in one or more embodiments, the disclosed systems receive style parameters in the form a style image and/or a text prompt. In some cases, the disclosed systems further receive a content image having content to include in the artistic visual content. Accordingly, in one or more embodiments, the disclosed systems utilize a neural network to generate the artistic visual content by iteratively generating an image, comparing the image to the style parameters, and updating parameters for generating the next image based on the comparison. In some instances, the disclosed systems incorporate a superzoom network into the neural network for increasing the resolution of the final image and adding art details that are associated with a physical art medium (e.g., brush strokes).

BACKGROUND

Recent years have seen significant advancement in hardware and software platforms for creating digital visual content. In particular, many conventional systems provide various tools that can be implemented for digitally creating and/or editing artistic visual content. For instance, many existing systems provide tools for creating artistic visual content based on an artistic style derived from a style prompt, such as a digital image.

Despite these advances, however, conventional content creation systems suffer from several technological shortcomings that result in inflexible and inaccurate operation. For instance, many conventional systems are inflexible in that they are typically limited to generating artistic visual content based on a digital image style prompt. Indeed, such systems are often incapable of incorporating artistic styles provided by style prompts of other forms into the creation process. While there do exist some systems that allow for the generation or manipulation of visual content utilizing other forms of style prompts, such as text prompts, these systems are often limited to creating photorealistic visual content rather than artistic visual content. Additionally, many conventional systems are limited to creating visual content from a single domain of content (e.g., faces, churches, cars, etc.).

In addition to flexibility concerns, conventional content creation systems often operate inaccurately. In particular, conventional systems often fail to accurately capture the artistic style provided by the style prompt within the generated visual content. For example, some conventional systems generate visual content by manipulating a content image. These systems, however, often fail to alter the structure of the content image in accordance with the style prompt. Indeed, many systems merely manipulate the content image to create minor variations in its features, such as variations in its color, texture, or background. Accordingly, these systems generate visual content that does not accurately reflect the prompted style.

These, along with additional problems and issues exist with regard to conventional content creation systems.

SUMMARY

One or more embodiments described herein provide benefits and/or solve one or more of the foregoing or other problems in the art with systems, methods, and non-transitory computer-readable media that flexibly generates artistic visual content based on a style image and/or a text prompt utilizing a neural network framework. In particular, in one or more embodiments, the disclosed systems receive style parameters as input (e.g., via a list of style images, a list of text prompts, or a combination of style images and text prompts) and generate a wide range of artistic content, with varying degrees of detail, style and structure with a boost in generation speed. For instance, in some embodiments, the disclosed systems utilize a neural network framework that generates an artistic image in relation to the style parameters via an iterative optimization method. In some cases, the disclosed systems further receive a content image and create the artistic content by manipulating the content image based on the style parameters via the iterative optimization method. For example, in at least one implementation, the disclosed systems utilize the neural network framework to iteratively modify the structure of the content image and incorporate additional artistic details, such as painter-specific patterns or brush marks. Moreover, in one or more embodiments, the disclosed systems further enhance results by utilizing an artistic superzoom framework in the generative pipeline (e.g., to bring additional details such as patterns specific to painters, slight brush marks, etc.). In this manner, the disclosed systems flexibly generate artistic visual content from a variety of style prompt forms utilizing a neural network framework that accurately captures the prompted style in its output.

Additional features and advantages of one or more embodiments of the present disclosure are outlined in the following description.

DETAILED DESCRIPTION

One or more embodiments described herein include an artistic content generation system that flexibly and accurately generates artistic content from various style prompt utilizing a neural network framework. Indeed, in one or more embodiments, the artistic content generation system utilizes an artistic image neural network to generate artistic visual content by stylizing a content image with a list of text prompts, a list of image prompts, or a combination of both. Accordingly, in some embodiments, the artistic content generation system implements text-guided image generation or manipulation to create the artistic visual content. In one or more embodiments, the artistic content generation generates the artistic visual content utilizing a neural network framework having a generative adversarial network (GAN) architecture that incorporates artistic superzoom to increase the resolution of the content and create special artistic effects (e.g., painting effects). In some cases, the neural network framework provides an iterative optimization process that incorporates fractal noise with an augmentation chain to facilitate incorporation of the artistic style associated with the style prompt(s).

To provide an illustration, in one or more embodiments, the artistic content generation system generates, utilizing an artistic generative neural network, an initialized artistic digital image based on a learnable tensor. Further, the artistic content generation system determines, utilizing a multi-domain style encoder of the artistic generative neural network, one or more style encodings for one or more style parameters. The artistic content generation system updates parameters of the learnable tensor by comparing the initialized artistic digital image to the one or more style encodings. Based on the learnable tensor with the updated parameters, the artistic content generation system generates an artistic digital image utilizing the artistic generative neural network.

As mentioned above, in one or more embodiments, the artistic content generation system utilizes a neural network framework for generating an artistic digital image. In particular, in some cases, the artistic content generation system utilizes an artistic image neural network to generate the artistic digital image. In some implementations, the artistic image neural network includes various components, such as an artistic generative neural network, a learnable tensor, at least one artistic superzoom neural network, and one or more additional encoders.

In one or more embodiments, the artistic content generation system utilizes the artistic image neural network to generate an artistic digital image from one or more style parameters. In some cases, the artistic content generation system receives the one or more style parameter by receiving a style digital image (or list of style digital images), a style text prompt (or list of style text prompts), or a combination of both. In some cases, the artistic image neural network utilizes the artistic image neural network to encode the one or more style parameters (e.g., encode the list of style digital images and/or the list of style text prompts) into a common, multi-domain encoding space.

In some embodiments, the artistic content generation system generates an artistic digital image by generating an initialized artistic digital image from the learnable tensor utilizing the artistic image neural network. Moreover, in one or more embodiments, the artistic content generation system further compares the initialized artistic digital image to the received style parameter(s), such as by encoding the initialized artistic digital image into the multi-domain encoding space and comparing the encodings. Based on the comparison, the artistic content generation system updates the learnable tensor. The artistic content generation system utilizes the artistic image neural network to generate the artistic digital image from the updated learnable tensor.

In some cases, the artistic content generation system generates the artistic digital image via an iterative process. Indeed, in some implementations, the artistic content generation system utilizes the artistic image neural network to iteratively generate an intermediate artistic digital image from the learnable tensor, compare the intermediate artistic digital image to the style parameters, and update the learnable tensor based on the comparison. Accordingly, in some instances, the artistic image neural network outputs the artistic digital image after a set number of iterations. In some cases, the artistic content generation system implements a scale hierarchy through different resolutions via the iteration process. Moreover, in some implementations, the artistic image neural network utilizes fractal noise and/or an augmentation chain to increase the speed of convergence.

In one or more embodiments, the artistic content generation system utilizes an artistic superzoom neural network of the artistic image neural network to add additional details to the artistic digital image. For example, the artistic content generation system utilizes the artistic superzoom neural network to increase the resolution of the artistic digital image and/or incorporate art details associated with a physical visual medium (e.g., painting effects, such as brush strokes or other painter-specific artifacts).

Further, in one or more embodiments, the artistic content generation system generates the artistic digital image from another digital image (e.g., a digital image that is separate from the style digital images). In particular, the artistic content generation system utilizes content from the other digital image to generate the artistic digital image. For example, in some implementations, the artistic content generation system utilizes the other digital image to initialize the learnable tensor used in generating the artistic digital image.

The artistic content generation system provides several advantages over conventional systems. For example, the artistic content generation system improves the flexibility of implementing computing devices when compared to conventional systems. To illustrate, by generating an artistic digital image from a style digital image (or list of style digital images) and/or a style text prompt (or list of style text prompts), the artistic content generation system flexibly generates artistic visual content from a variety of style prompt forms. Additionally, the artistic content generation system can flexibly generate artistic visual content using content from a variety of domains, rather than being limited to a single domain as is typical under many conventional systems.

Additionally, the artistic content generation system can improve the accuracy of implementing computing devices when compared to conventional systems. In particular, the artistic content generation system can generate artistic visual content that accurately captures the artistic style provided via one or more style prompts. To illustrate, in some cases, the artistic content generation system utilizes an artistic image neural network to alter the structure of the content displayed in another digital image to generate an artistic digital image. In particular, the artistic image neural network alters the structure of the content in accordance with the one or more style prompts. Thus, the artistic content generation system more accurately aligns the artistic style of the generated visual content with the artistic style provided by the style prompt(s).

In addition, the artistic content generation system can also improve efficiency of implementing computing devices. In particular, by utilizing the proposed neural network framework, in some implementations. the artistic content generation system provides a boost in image generation speed and a corresponding reduction in computer resource requirements. For instance, as described in greater detail below, by utilizing fractal noise followed by stylization while using an augmentation chain, the artistic content generation system can significantly reduce the time and computing resources needed for convergence.

As illustrated by the foregoing discussion, the present disclosure utilizes a variety of terms to describe features and benefits of the artistic content generation system. Additional detail is now provided regarding the meaning of these terms. For example, as used herein, the term “artistic digital image” refers to modified digital visual content (e.g., that includes one or more modifications to add artistic features). In particular, in some embodiments, an artistic digital image refers to a modified digital image that features one or more artistic styles. To illustrate, in some cases, an artistic digital image includes content generated to include one or more artistic styles. In some implementations, an artistic digital image includes a non-photographic digital image. Relatedly, as used herein, the term “initialized artistic digital image” refers to an initial digital image generated in a process (e.g., an iterative process) for generating an artistic digital image. Similarly, as used herein, the term “intermediate artistic digital image” refers to an artistic digital image that is generated in a process (e.g., an iterative process) for generating an artistic digital image but is not the output of the process (e.g., not the final artistic digital image). Accordingly, in some cases, an initialized artistic digital image is an intermediate artistic digital image.

As used herein, the term “artistic encoding” refers to an encoding that corresponds to an artistic digital image. In particular, in some embodiments, an artistic encoding refers to an encoded value or an encoded set of values that represents at least a portion of an artistic digital image. To illustrate, in some cases, an artistic encoding includes an encoding generated from an artistic digital image.

Additionally, as used herein, the term “style parameter” refers to a parameter for creating an artistic digital image. In particular, in some embodiments, a style parameter refers to a patent or latent feature corresponding to an artistic digital image/text prompt (e.g., a feature that is to be included in an artistic digital image). To illustrate, in some cases, a style parameter includes a patent or latent feature associated with content or an artistic style to be incorporated within an artistic digital image. For instance, in some implementations, a style parameter includes an object, a color, a color scheme, a geometry, a landscape, or a theme or concept to be incorporated into an artistic digital image. Relatedly, as used herein, the term “style digital image” refers to a digital image that is associated with (e.g., includes) one or more style parameters to be incorporated into an artistic digital image. Further, as used herein, the term “style text prompt” refers to a text (e.g., a word, a sentence, or a paragraph) that includes (e.g., describes) one or more style parameters to be incorporated into an artistic digital image.

Further, as used herein, the term “style encoding” refers to an encoding that corresponds to a style parameter. In particular, in some embodiments, a style encoding refers to an encoded value or an encoded set of values that represents at least a portion of a style parameter. To illustrate, in some cases, a style encoding includes an encoding generated from a style digital image or a style text prompt.

As used herein, the term “multi-domain encoding space” refers to a latent encoding space for encodings associated with multiple domains. In particular, in some embodiments, a multi-domain encoding space refers to an encoding space that contains (or is capable of containing) encodings generated from data that is associated with least one of multiple different domains. For instance, in some cases, a multi-domain encoding space includes an encoding space that contains (or is capable of containing) style encodings generated from digital images (e.g., style digital images) and style encodings generated from text (e.g., style text prompts).

Relatedly, as used herein, the term “multi-domain style encoder” refers to an encoder that generates encodings within a multi-domain encoding space. In particular, in some embodiments, a multi-domain style encoder refers to an encoder that generates style encodings (e.g., from text and/or digital images) within a multi-domain encoding space. In some cases, a multi-domain style encoder includes one or more component neural network encoders. For example, in some cases, a multi-domain style encoder includes a neural network image encoder and a neural network text encoder.

As used herein, the term “neural network” refers to a type of machine learning model, which can be tuned (e.g., trained) based on inputs to approximate unknown functions used for generating the corresponding outputs. In particular, in some embodiments, a neural network refers to a model of interconnected artificial neurons (e.g., organized in layers) that communicate and learn to approximate complex functions and generate outputs based on a plurality of inputs provided to the model. In some instances, a neural network includes one or more machine learning algorithms. Further, in some cases, a neural network includes an algorithm (or set of algorithms) that implements deep learning techniques that utilize a set of algorithms to model high-level abstractions in data. To illustrate, in some embodiments, a neural network includes a convolutional neural network, a recurrent neural network (e.g., a long short-term memory neural network), a generative adversarial neural network, a graph neural network, or a multi-layer perceptron. In some embodiments, a neural network includes a combination of neural networks or neural network components.

Additionally, as used herein, the term “artistic image neural network” refers to a computer-implemented neural network that generates artistic digital images. In particular, in some embodiments, an artistic image neural network refers to a computer-implemented neural network that generates an artistic digital image based on one or more style parameters and/or another digital image that includes content for generating the artistic digital image. To illustrate, in some cases, an artistic image neural network includes a neural network framework that implements an iterative process for generating an artistic digital image in accordance with one or more style parameters.

In some cases, an artistic image neural network includes an artistic generative neural network. As used herein, the term “artistic generative neural network” includes a computer-implemented generative neural network that generates artistic digital images. For example, in some cases, an artistic generative neural network includes a computer-implemented generative neural network that includes an encoder-decoder neural network architecture for generating artistic digital images (e.g., an intermediate artistic digital image and/or a final artistic digital image).

In some implementations, an artistic image neural network includes one or more artistic superzoom neural networks. As used herein, the term “artistic superzoom neural network” refers to a computer-implemented neural network that increases the resolution of a digital image, such as an artistic digital image or a digital image having content for creating an artistic digital image. In some implementations, an artistic superzoom neural network includes a computer-implemented neural network that adds, to an artistic digital image, one or more art details associated with a physical visual medium (e.g., brush strokes or other painter-specific artifacts).

In some cases, an artistic image neural network includes a learnable tensor. As used herein, the term “learnable tensor” refers to a learnable dimensional data structure. In particular, in some embodiments, a learnable tensor includes a dimensional data structure having one or more parameters (e.g., values) that are changeable. In some embodiments, a learnable tensor corresponds to encodings generated by an encoder of an artistic generative neural network or the encoding space in which such encodings are generated.

As used herein, the term “augmentation chain” refers to a computer-implemented process for modifying a digital image, such as an artistic digital image. In particular, in some embodiments, an augmentation chain refers to a sequence of actions that change a digital image. To illustrate, in some implementations, an augmentation chain refers to a sequence of one or more transformation operations applied to a digital image. Relatedly, as used herein, the term “transformation operation” refers to an operation that modifies one or more aspects of a digital image. For instance, in some embodiments, a transformation operation includes, but is not limited to, one of a resize operation, a crop operation, a perspective operation, an image flip operation, or a noise operation.

Additionally, as used herein the term “fractal noise” refers to noise associated with a digital image. In particular, in some embodiments, fractal noise refers to digital data that affects patent or latent characteristics of a digital image. For instance, in some cases, fractal noise includes digital data that is added to a digital image to affect the patent or latent characteristics of the digital image.

Additional detail regarding the artistic content generation system will now be provided with reference to the figures. For example,FIG.1illustrates a schematic diagram of an exemplary system environment (“environment”)100in which an artistic content generation system106operates. As illustrated inFIG.1, the environment100includes a server(s)102, a network108, and client devices110a-110b.

Although the environment100ofFIG.1is depicted as having a particular number of components, the environment100is capable of having any number of additional or alternative components (e.g., any number of servers, client devices, or other components in communication with the artistic content generation system106via the network108). Similarly, althoughFIG.1illustrates a particular arrangement of the server(s)102, the network108, and the client devices110a-110b, various additional arrangements are possible.

The server(s)102, the network108, and the client devices110a-110bare communicatively coupled with each other either directly or indirectly (e.g., through the network108discussed in greater detail below in relation toFIG.13). Moreover, the server(s)102and the client devices110a-110binclude one of a variety of computing devices (including one or more computing devices as discussed in greater detail with relation toFIG.13).

As mentioned above, the environment100includes the server(s)102. In one or more embodiments, the server(s)102generates, stores, receives, and/or transmits data including neural networks, digital images, and texts. In one or more embodiments, the server(s)102comprises a data server. In some implementations, the server(s)102comprises a communication server or a web-hosting server.

In one or more embodiments, the image editing system104provides functionality by which a client device (e.g., a user of one of the client devices110a-110b) generates, edits, manages, and/or stores digital images. For example, in some instances, a client device sends a digital image to the image editing system104hosted on the server(s)102via the network108. The image editing system104then provides many options that the client device. may use to edit the digital image, store the digital image, and subsequently search for, access, and view the digital image. Further, in some cases, the image editing system104provides one or more options that the client device may use to create an artistic digital image utilizing the digital image.

Additionally, the server(s)102include the artistic content generation system106. In one or more embodiments, via the server(s)102, the artistic content generation system106generates an artistic digital image utilizing an artistic image neural network114. For example, in one or more embodiments, the artistic content generation system106, via the server(s)102, implements an iterative process for generating an artistic digital image in accordance with one or more embodiments. In some cases, via the server(s)102, the artistic content generation system106receives a digital image that includes particular content and generates the artistic digital image based on the content of the digital image utilizing the artistic image neural network114. Example components of the artistic content generation system106will be described below with regard toFIG.11.

In one or more embodiments, the client devices110a-110binclude computing devices that can access, edit, modify, store, and/or provide, for display, digital images, such as artistic digital images. For example, the client devices110a-110binclude smartphones, tablets, desktop computers, laptop computers, head-mounted-display devices, or other electronic devices. The client devices110a-110binclude one or more applications (e.g., the client application112) that can access, edit, modify, store, and/or provide, for display, digital images, such as artistic digital images. For example, in some embodiments, the client application112includes a software application installed on the client devices110a-110b. In other cases, however, the client application112includes a web browser or other application that accesses a software application hosted on the server(s)102.

The artistic content generation system106can be implemented in whole, or in part, by the individual elements of the environment100. Indeed, as shown inFIG.1the artistic content generation system106can be implemented with regard to the server(s)102and/or at the client devices110a-110b. In particular embodiments, the artistic content generation system106on the client devices110a-110bcomprises a web application, a native application installed on the client devices110a-110b(e.g., a mobile application, a desktop application, a plug-in application, etc.), or a cloud-based application where part of the functionality is performed by the server(s)102.

In additional or alternative embodiments, the artistic content generation system106on the client devices110a-110brepresents and/or provides the same or similar functionality as described herein in connection with the artistic content generation system106on the server(s)102. In some implementations, the artistic content generation system106on the server(s)102supports the artistic content generation system106on the client devices110a-110b.

For example, in some embodiments, the server(s)102train one or more machine-learning models described herein (e.g., the artistic image neural network114). The artistic content generation system106on the server(s)102provides the one or more trained machine-learning models to the artistic content generation system106on the client devices110a-110bfor implementation. Accordingly, although not illustrated, in one or more embodiments the client devices110a-110butilize the artistic image neural network114to generate artistic digital images.

In some embodiments, the artistic content generation system106includes a web hosting application that allows the client devices110a-110bto interact with content and services hosted on the server (s)102. To illustrate, in one or more implementations, the client devices110a-110baccesses a web page or computing application supported by the server (s)102. The client devices110a-110bprovides input to the server(s)102(e.g., a style prompt and/or an input digital image). In response, the artistic content generation system106on the server(s)102utilizes the artistic image neural network114to generate an artistic digital image. The server(s)102then provides the artistic digital image to the client devices110a-110b.

In some embodiments, though not illustrated inFIG.1, the environment100has a different arrangement of components and/or has a different number or set of components altogether. For example, in certain embodiments, the client devices110a-110bcommunicates directly with the server(s)102, bypassing the network108. As another example, the environment100includes a third-party server comprising a content server and/or a data collection server.

As mentioned above, the artistic content generation system106generates an artistic digital image.FIG.2illustrates an overview diagram of the artistic content generation system106generating an artistic digital image in accordance with one or more embodiments.

As shown inFIG.2, the artistic content generation system106receives style parameters202. In particular, as illustrated, the artistic content generation system106receives the style parameters202by receiving a style digital image204and a style text prompt206. As previously mentioned, however, the artistic content generation system106receives one of a style digital image or a style text prompt (rather than both) in some embodiments. Further, in some implementations, the artistic content generation system106receives multiple style digital images and/or multiple style text prompts as the style parameters202. Thus, the artistic content generation system106receives the style parameters202by receiving various combinations of style digital images and style text prompts in some embodiments.

In some cases, the artistic content generation system106receives the style parameters202from a client device. For example, in some implementations, the artistic content generation system106receives a communication from a client device containing the style digital image204and/or the style text prompt206. In some cases, however, the artistic content generation system106receives an indication of the style parameters202and retrieves the style parameters202based on the indication. For example, in some cases, the artistic content generation system106stores a style digital image locally or at a remote storage location and retrieves the style digital image from storage in response to receiving an indication that the style digital image has been selected.

As further shown inFIG.2, the artistic content generation system106receives a digital image208. In some cases, the digital image208includes content on which an artistic digital image is to be based. For instance, in some implementations, the digital image208includes one or more foreground objects to include in the artistic digital image or a background to include in the artistic digital image. Similar to the style parameters202, the artistic content generation system106receives the digital image208from a client device or retrieves the digital image208from storage in various embodiments.

It should be understood, however, that the digital image208is optional in some embodiments. In other words, in some implementations, the artistic content generation system106generates an artistic digital image without use of a digital image that includes base content. Distinctions between the process for generating an artistic digital image with or without a digital image having base content will be discussed in more detail below.

Additionally, as shown inFIG.2, the artistic content generation system106utilizes an artistic image neural network210to analyze the style parameters202and the digital image208. Based on the analysis of the style parameters202and the digital image208, the artistic content generation system106generates an artistic digital image212. In one or more embodiments, the artistic digital image212includes one or more artistic styles associated with the style parameters202. In particular, the artistic digital image212includes one or more patent or latent artistic features or characteristics provided by the style parameters202. Further, in some cases, the artistic digital image212includes content from the digital image208. For example, in some cases, the artistic content generation system106generates the artistic digital image212to include at least some of the content from the digital image208as modified in accordance with the style parameters202. To illustrate, in some cases, the artistic content generation system106modifies the structure of the content of the digital image208in generating the artistic digital image212.

As previously mentioned, the artistic content generation system106utilizes an artistic image neural network to generate an artistic digital image.FIGS.3-6Cillustrate diagrams for utilizing an artistic image neural network to generate an artistic digital image. In particular,FIGS.3-6Cillustrate various components of and operations performed by an artistic image neural network in accordance with one or more embodiments.

For example,FIG.3illustrates an overview of an architecture of an artistic image neural network300in accordance with one or more embodiments. As shown inFIG.3, the artistic image neural network300includes an artistic generative neural network having an encoder302and a decoder304. The artistic image neural network300further includes a learnable tensor306. Additionally, the artistic image neural network300includes a multi-domain style encoder308that is composed of a neural network image encoder310and a neural network text encoder312. Further, as shown, the artistic image neural network300includes an additional neural network image encoder314. In some cases, the neural network image encoder310and the additional neural network image encoder314are the same network. In other words, in some implementations, the artistic image neural network300utilizes one neural network image encoder for the neural network image encoders310,314.

As illustrated byFIG.3, the artistic content generation system106provides one or more style parameters to the artistic image neural network300. In particular, as shown, the artistic content generation system106provides a style digital image316and a style text prompt318to the multi-domain style encoder308of the artistic image neural network300. The artistic content generation system106utilizes the multi-domain style encoder308to project the style digital image316and the style text prompt318into a multi-domain encoding space. Specifically, the artistic content generation system106utilizes the neural network image encoder310to project the style digital image316into the multi-domain encoding space and utilizes the neural network text encoder312to project the style text prompt318into the multi-domain encoding space. Thus, the artistic content generation system106utilizes the multi-domain style encoder308to determine style encodings320from the style parameters associated with the style digital image316and the style text prompt318.

By utilizing the multi-domain style encoder308to generate style encodings from a style digital image and/or a style text prompt, the artistic content generation system106enables implementing computing devices to operate more flexibly than conventional systems. Indeed, the artistic content generation system106enables an implementing computing device to utilize style parameters associated with a wider variety of style prompts when compared to other systems.

In one or more embodiments, the artistic content generation system106utilizes, as the multi-domain style encoder308, an encoder that includes the cross-lingual-multimodal-embedding model and the image-embedding model described in U.S. patent application Ser. No. 17/075,450 filed on Oct. 20, 2020, entitled GENERATING EMBEDDINGS IN A MULTIMODAL EMBEDDING SPACE FOR CROSS-LINGUAL DIGITAL IMAGE RETRIEVAL, the contents of which are expressly incorporated herein by reference in their entirety. In some cases, the artistic content generation system106utilizes, as the multi-domain style encoder308, the Contrastive Language-Image Pre-training (CLIP) model described by Alec Radford et al.,Learnable Transferable Visual Models from Natural Language Supervision, ICML, 2021, arXiv:2103.00020, which is incorporated herein by reference in its entirety.

As further shown byFIG.3, the artistic content generation system106provides a digital image322having content to include in the resulting artistic digital image to the artistic image neural network300. In particular, the artistic content generation system106provides the digital image322to the encoder302of the artistic generative neural network300. The artistic content generation system106utilizes the encoder302to define the learnable tensor306based on the digital image322. For example, in some cases, the artistic content generation system106utilizes the encoder302to initialize the parameters (e.g., values or encodings) of the learnable tensor306based on the digital image322. In some cases, the artistic content generation system106improves the flexibility of implementing computing devices by facilitating the creation of artistic digital images using content from a wider variety of domains when compared to other systems.

As previously mentioned, in some embodiments, the artistic content generation system106utilizes the artistic image neural network300to generate an artistic digital image without the use of a digital image having content for the artistic digital image. In such cases, the artistic content generation system106initializes the parameters of the learnable tensor306by selecting a point (e.g., a randomized or semi-randomized point) within an encoding space associated with the artistic generative neural network.

As shown inFIG.3, the artistic content generation system106utilizes the decoder304of the artistic generative neural network to generate an initialized artistic digital image324based on the learnable tensor306. In one or more embodiments, the artistic content generation system106utilizes, as the artistic generative neural network, at least one of the generative models described by Patrick Esser et al.,Taming Transformers for High-Resolution Image Synthesis, CVRP, 2020, arXiv:2012.09841; Andrew Brock et al.,Large Scale GAN Training for High Fidelity Natural Image Synthesis,2018, arXiv:1809.11096; Ting-Yun Chang and Chi-Jen Lu,TinyGAN. Distilling BigGAN for Conditional Image Generation, ACC, 2020, arXiv:2009.13829; and Prafulla Dhariwal and Alex Nichol,Diffusion Models Beat GANs on Image Synthesis,2021, arXiv:2105.05233, all of which are incorporated herein by reference in their entirety.

Further, as shown, the artistic content generation system106utilizes the additional neural network image encoder314to project the initialized artistic digital image324into the multi-domain encoding space. In particular, the artistic content generation system106utilizes the additional neural network image encoder314to generate artistic encodings326from the initialized artistic digital image324.

In one or more embodiments, the artistic content generation system106utilizes, as the additional neural network image encoder314, the image-embedding model described in U.S. patent application Ser. No. 17/075,450. In some cases, the artistic content generation system106utilizes, as the additional neural network image encoder314, the neural network image encoder of the CLIP model described by Alec Radford et al.

Further, as illustrated, the artistic content generation system106compares the initialized artistic digital image324to the style parameters associated with the style digital image316and the style text prompt318. For example, as shown, the artistic content generation system106compares the artistic encodings326generated from the initialized artistic digital image324and the style encodings320generated from the style digital image316and the style text prompt318. To illustrate, as shown, the artistic content generation system106compares the artistic encodings326and the style encodings320utilizing a loss function328. In one or more embodiments, the artistic content generation system106utilizes a style loss function defined as follows:

In equation 3, {tilde over (·)} represents the operation of normalizing a vector, where the artistic content generation system106normalizes a vector {tilde over (v)} as follows

Additionally, in equation 1, l represents the set of style digital images, P represents the set of style text prompts, Encodimage(·) represents the function that encodes an image into the multi-domain encoding space (e.g., as implemented by the neural network image encoder310and the additional neural network image encoder314), and Encodtext(·) represents the function that encodes a style text prompt into the multi-domain encoding space (e.g., as implemented by the neural network text encoder312). Further, t represents the learnable tensor306and Decod(·) represents the function that generates an artistic digital image from the learnable tensor306(e.g., as implemented by the decoder304of the artistic generative neural network).

As shown inFIG.3, the artistic content generation system106back propagates the loss determined via the loss function328to the learnable tensor306(as shown by the line330). In particular, the artistic content generation system106determines one or more gradients with respect to the learnable tensor306via back propagation. In one or more embodiments, the artistic content generation system106updates the parameters of the learnable tensor306using the determined gradient(s). For example, in some cases, the artistic content generation system106updates the parameters in relation to the gradient(s) by decreasing the partial derivatives (taken from the gradient) of the parameter on each component.

As indicated byFIG.3, the artistic image neural network300implements an iterative optimization loop332. In particular, the artistic image neural network300iteratively generates an intermediate digital image from the learnable tensor306, projects the intermediate digital image into the multi-domain encoding space, compares the artistic encodings to the style encodings320via the loss function328, and updates the parameters of the learnable tensor306based on the comparison. Thus, at each iteration after the first iteration, the artistic image neural network300generates a new intermediate artistic digital image based on the learnable tensor306with the parameters as updated from the previous iteration. In some embodiments, the artistic image neural network300uses the last iteration to generate the artistic digital image (e.g., the final artistic digital image) from the learnable tensor306with the most recent parameter updates. Thus, in one or more embodiments, the artistic image neural network300utilizes the iterative optimization loop332to iteratively increase the degree to which the artistic digital image generated from the learnable tensor306incorporates the style parameters.

In one or more embodiments, the artistic content generation system106modifies/transforms the intermediate artistic digital images (such as the initialized artistic digital image) generated from the learnable tensor and the style digital image before comparing them.FIG.4illustrates a diagram for transforming an intermediate digital image and a style digital image for comparison in accordance with one or more embodiments.

For example, as shown inFIG.4, the artistic content generation system106crops an intermediate artistic digital image402utilizing crops of variable cropping size and/or variable cropping offset (e.g., the crops404a-404b). Thus, in some cases, the artistic content generation system106generates a set of transformed intermediate artistic digital images from the intermediate artistic digital image402.

Similarly, as shown inFIG.4, the artistic content generation system106crops a style digital image406utilizing crops of variable cropping size and/or variable cropping offset (e.g., the crops408a-408b). Thus, in cases, the artistic content generation system106generates a set of transformed style digital images from the style digital image406.

In one or more embodiments, the artistic content generation system106utilizes different cropping sizes and/or different cropping offsets for cropping the intermediate artistic digital image402and the style digital image406. For example, in some cases, the artistic content generation system106randomizes the selection of the cropping size and/or the cropping offset. In some cases, however, the artistic content generation system106utilizes the same the cropping sizes and/or cropping offsets for cropping the intermediate artistic digital image402and the style digital image406.

Further, in some embodiments, the artistic content generation system106generates crops of the style digital image406once during generation of an artistic digital image. For example, in some cases, the artistic content generation system106creates one set of transformed style digital images and utilizes the same set for every iteration implemented by the artistic image neural network to generate the artistic digital image. In some cases, however, the artistic content generation system106generates a new set of transformed style digital images for every iteration. Likewise, in one or more embodiments, the artistic content generation system106generates a new set of transformed intermediate artistic digital images for every iteration as the artistic image neural network generates a new intermediate artistic digital image at each iteration. In some instances, the artistic content generation system106generates two sets of transformed intermediate artistic digital images for every iteration (as will be shown with reference to equation 2).

As further shown inFIG.4, the artistic content generation system106utilizes a neural network image encoder410(e.g., the neural network image encoder314discussed above with reference toFIG.3) of the artistic image neural network to generate artistic encodings412from the set of transformed intermediate artistic digital images (e.g., the crops of the intermediate artistic digital image402). Further, the artistic content generation system106utilizes a neural network image encoder414(e.g., the neural network image encoder310discussed above with reference toFIG.3) of the multi-domain style encoder of the artistic image neural network to generate style encodings416from the set of transformed style digital images (e.g., the crops of the style digital image406). In one or more embodiments, the artistic content generation system106resizes the crops so that they are of the size corresponding to the input of the respective encoder.

Accordingly, in one or more embodiments, the artistic content generation system106compares the artistic encodings412and the style encodings416using the loss function418(e.g., the loss function328discussed above with reference toFIG.3). For instance, in some implementations, the artistic content generation system106utilizes a style loss function defined as follows:

The style loss of equation 2 differs from the style loss of equation 1 in that it accommodates the cropped digital images projected into the multi-domain encoding space. For example, in equation 2, A and C represent the sets of transformed intermediate artistic digital images. In one or more embodiments, the artistic content generation system106generates the sets represented by A and C independently from one another. Further, in equation 2, Birepresents the set of transformed style digital images. As suggested, in some cases, the set Biremains constant through the process of generating the artistic digital image.

In one or more embodiments, then artistic content generation system106further utilizes one or more additional loss functions to facilitate control of the amount of content to keep in the final artistic digital image (e.g., the content from the digital image provided to the encoder of the artistic generative neural network). In particular, in some cases, the artistic content generation system106utilizes the additional loss function(s) to ensure a one-to-one correspondence to each iteration between the intermediate results and the content of the original digital image. For example, in one or more embodiments, the artistic content generation system106further utilizes a pixel loss function defined as follows:

In equation 3, k[c,i,j]∀c∈1,C, x∈1,D1, y∈1,D2represents the codec at position i,j in the tensor k of dimensions C×D1×D2. Further, C represents the dimensionality of codes in the latent space of the artistic generative neural network encoder, and D1×D2represents the dimensionality of a digital image of W×H pixels in the latent space. In some cases, D1=[W/2m] and D2=[H/2m] where m represents the number of down-sampling blocks. Further, O represents the digital image having the content for creating the artistic digital image (having dimensions W×H), and Encodgen(·) represents the function that encodes a digital image into the encoding space of the learnable tensor (e.g., as implemented by the encoder of the artistic generative neural network). In other words, t=Encodgen(O).

In some embodiments, the artistic content generation system106further utilizes a perceptual loss function defined as follows:

In equation 4, LPIPS(·) refers to the feature extractor utilized in determining the perceptual loss. In some embodiments, the artistic content generation system106utilizes, as the feature extractor, the Visual Geometry Group 19 (VGG19) model described by Karen Simonyan and Andrew Zisserman,Very Deep Convolutional Networks for Large-Scale Image Recognition, CVPR, 2015, arXiv:1409.1556, which is incorporated herein by reference in its entirety.

Thus, in one or more embodiments, the artistic content generation system106combines one or more of the loss functions defined by equation 2 (or equation 1) and equations 3-4 for comparing encodings within the multi-domain encoding space. For example, in some implementations, the artistic content generation system106utilizes the loss function defined as follows:

In equation 5, wpixeland wperceptualrepresent weights to be applied to the pixel loss and the perceptual loss, respectively. In one or more embodiments, wpixeland wperceptualare configurable. In other words, in some cases, the artistic content generation system106determines wpixeland wperceptualbased on inputs (e.g., received via a client device). Further, in some embodiments, where a digital image having content for the artistic digital image is not used, artistic content generation system106sets wpixeland wperceptualequal to zero.

As previously mentioned, in some embodiments, the artistic content generation system106utilizes an artistic image neural network to generate an artistic digital image via an iterative process that implements a hierarchical scaling to different resolutions.FIG.5illustrates an architecture of an artistic image neural network that implements an iterative process using a hierarchical scaling to different resolutions in accordance with one or more embodiments.

As shown inFIG.5, the artistic image neural network500is similar to the artistic image neural network300discussed in reference toFIG.3with some notable differences. In particular, the artistic image neural network500includes a resize block502that is composed of an artistic superzoom neural network504(labeled “superzoom up-sampling”) and down-sampling blocks506a-506b. Further, the artistic image neural network500includes a conditional block508and an additional artistic superzoom neural network510(labeled “artistic superzoom).

In one or more embodiments, the artistic content generation system106utilizes hierarchical scaling to different resolutions to stylize the learnable tensor512to a variable scale of resolutions. For instance, in some cases, the artistic content generation system106defines such a scaling as follows:

Equation 6 indicates that the scale hierarchy S includes n resolutions in which the decoder514of the artistic generative neural network generates a color image of size (r1i×r2i), and firepresents the number of iterations performed until moving to the next resolution with the index i+1. In one or more embodiments, using the scale hierarchy defined by equation 6, the artistic image neural network500implements an optimization process (e.g., the process of generating an artistic digital image) as will now be described.

In one or more embodiments, the artistic image neural network500initializes the learnable tensor512with a digital image516having content for the artistic digital image—having resized the digital image516to the resolution (r1i×r2i) pixels. In particular, in some embodiments, the artistic content generation system106resizes the digital image516using the resize block502. Further, the artistic image neural network500initializes the learnable tensor512by projecting the digital image516into an encoding space using an encoder518of the artistic generative neural network. As mentioned above, where no digital image having content for the artistic digital image is used, the artistic image neural network500initializes the learnable tensor512by selecting a point in the encoding space associated with the artistic generative neural network. In one or more embodiments, for initialization, the learnable tensor512includes a size of

where C represents dimensionality of codes in the encoding space of the artistic generative neural network and m represents the number of down-sampling blocks. In one or more embodiments, after initialization, the artistic image neural network500performs f1iterations of stylization.

In one or more embodiments, the artistic image neural network500performs a super resolution operation at each resizing if fi+1fi∀∈1,n−1to provide additional enhancements of details in the intermediate results (e.g., the intermediate artistic digital images) and implicitly in the final result. Thus, in one or more embodiments, t=Encodgen(SR(Decod(t))) where Encodgen(·) and Decod(·) represents the encoder518and decoder514, respectively, of the artistic generative neural network, and SR(·) represents the super resolution operation (e.g., as implemented by the artistic superzoom neural network504of the resize block502.

In other words, in one or more embodiments, the artistic content generation system106utilizes the artistic image neural network500to initialize the learnable tensor512(e.g., based on the digital image516or by selecting a point in the encoding space associated with the learnable tensor512). The artistic content generation system106further utilizes the artistic image neural network500to perform a first set of optimization iterations (e.g., via the optimization loop526) to generate a first set of intermediate artistic digital images at a first resolution of the scale hierarchy. Additionally, the artistic content generation system106utilizes the artistic image neural network500to resize the intermediate artistic digital image produced by the last iteration of the first resolution to a second resolution via the resize block502. Further, the artistic content generation system106utilizes the artistic image neural network500to perform a second set of optimization iterations (e.g., via the optimization loop526) to generate a second set of intermediate artistic digital images at this second resolution. The artistic image neural network500similarly operates, iterating through all the resolutions of the scale hierarchy.

In one or more embodiments, the artistic image neural network500utilizes the conditional block508to change the resolution after exhausting the number of iterations for the current resolution of the scale hierarchy. In particular, the artistic image neural network500utilizes the conditional block508to send the intermediate artistic digital image produced by the last iteration for the current resolution to the resize block502(as shown by line520).

At the final iteration for the last resolution of the scale hierarchy, the artistic image neural network500generates the artistic digital image524that will be provided as output. In one or more embodiments, at the final iteration for the last resolution, the artistic image neural network500utilizes the conditional block508to send the artistic digital image produced from the last iteration to the additional artistic superzoom neural network510(as shown by the line522).

In one or more embodiments, the number of iterations for each resolution of the scale hierarchy is configurable. In some cases, the number of resolutions used for the scale hierarchy is configurable. Further, in some instances, each resolution used for the scale hierarchy is configurable.

In one or more embodiments, the artistic content generation system106utilizes the additional artistic superzoom neural network510of the artistic image neural network500to increase the resolution of the artistic digital image524and to incorporate art details associated with a physical visual medium (e.g., painting effects, such as brush strokes or other painter-specific artifacts).

In one or more embodiments, the artistic superzoom neural network504of the resize block502and the additional artistic superzoom neural network510include similar architectures, which will be discussed in more detail below with reference toFIGS.6A-6C. In some cases, the artistic superzoom neural network504and the additional artistic superzoom neural network510operate with natural scales rather than rational scales. Accordingly, in one or more embodiments, artistic superzoom neural network504and the additional artistic superzoom neural network510implement the up-sampling operation as follows:

In equation 6, Resize(a×b)(I) represents the operation of resizing the image I to the dimensions (a×b) and SZ×2(I) is the output from the artistic superzoom neural network that increases the resolution of the image I twice.

Thus, in one or more embodiments, the artistic content generation system106utilizes an artistic image neural network to iteratively utilize one or more style parameters to generate an artistic digital image. In particular, the artistic content generation system106utilizes the style encodings generated from the one or more style parameters to generate the artistic digital image. Accordingly, in some embodiments, the algorithm and acts described with reference toFIGS.4-5comprise the corresponding structure for performing a step for iteratively utilizing one or more style encodings to generate an artistic digital image from a digital image. Further, in some embodiments, the neural network architecture described with reference toFIGS.4-5comprises the corresponding structure for performing a step for iteratively utilizing the one or more style encodings to generate an artistic digital image from the digital image.

FIGS.6A-6Cillustrate the architecture of an artistic superzoom neural network incorporated into an artistic image neural network in accordance with one or more embodiments. As mentioned above, in some cases, the artistic content generation system106utilizes multiple artistic superzoom neural networks within an artistic image neural network to increase the resolution of a particular image and/or to add art details associated with a physical visual medium at various points of the process for generating an artistic digital image.

As shown inFIG.6A, an artistic superzoom neural network600includes input blocks602, fixup resnet dilated blocks604, an up-sampling block606, and a convolutional layer608. As indicated byFIG.6A, the artistic superzoom neural network600implements the up-sampling block606with r number of repetitions. In one or more embodiments, r is configurable. In one or more embodiments, r represents the up-sampling scale. Thus, in some embodiments, the artistic content generation system106utilizes the artistic superzoom neural network600to generate an output image612having increased resolution compared to an input image610. In one or more embodiments, the lines616a-616brepresent the losses utilized in training the artistic superzoom neural network600, which will be discussed in more detail below.

FIG.6Billustrates the architecture of a fixup resent dilated block614in accordance with one or more embodiments. In one or more embodiments, each fixup resnet dilated block from the fixup resnet dilated blocks604includes the architecture shown inFIG.6B.FIG.6Cillustrates the architecture of up-sampling block606in accordance with one or more embodiments.

In some embodiments, the artistic superzoom neural network operates more efficiently than models employed by many conventional systems as the artistic superzoom neural network operates without batch normalization. Accordingly, the speed of the artistic superzoom neural network is relatively faster because there are fewer operations to perform, and the model can be trained with small batches on a basic GPU. Further, in some cases, the artistic superzoom neural network includes one or more attention mechanisms that improve the quality of the output, providing more accurate results when compared to many conventional systems. Further, in one or more embodiments, the artistic content generation system106operates without a particular compression rate (while many conventional systems do), allowing the artistic content generation system106to reconstruct details regardless of compression rate and leading to a model that has a substantially reduced size compared to the models of many conventional systems.

In one or more embodiments, the artistic content generation system106trains the artistic superzoom neural network(s) using an image dataset of famous paintings (e.g., paintings from Van Gogh, Monet, Friedrich, etc.). In some cases, the artistic content generation system106extracts patches from the images in the dataset and utilizes the patches to perform the training. Further, in some cases, the artistic content generation system106augments the paintings using random rotation and/or random resizing transformation operations on both input images and synthetic images. In some cases, the artistic content generation system106further applies random intensities of the synthetic images.

In one or more embodiments, artistic content generation system106utilizes Grto represent an artistic superzoom neural network that increases the size of an input image 2rtimes. If I is an image of size W×H×3, then Gr(I) is an image of size 2rW×2rH×3. In some instances, during the training process, the artistic content generation system106subjects each image T from the image dataset of size 2rW×2rH×3 to a Gaussian filter followed by a down-sampling to the size of W×H×3, resulting in the image {circumflex over (T)}. In some cases, the artistic content generation system106provides {circumflex over (T)} as input for training.

In one or more embodiments, the artistic content generation system106utilizes one or more loss functions for training the artistic superzoom neural network. For example, in some cases, the artistic content generation system106utilizes a loss function composed of a combination of loss functions. For instance, in some cases, the artistic content generation system106utilizes a pixel loss function defined as follows:

In equation 7, l[x,y]∀x∈1,W, y∈1,Hrepresents the pixel at position x, y in the image I of dimension W×H pixels. In some instances, the pixel loss of equation 7 represents a mean squared error (MSE) loss between the resulting image and the ground truth. In some cases, the artistic content generation system106utilizes the loss function represented by equation 7 to preserve the overall structure of the original image so that the output is not a degraded version of the input and a similar version of it.

In some cases, the artistic content generation system106further utilizes a perceptual loss function defined as follows:

In equation 8, ϕl(I) represents the feature map after the ReLU function with the number l in the feature-extractor that receives, as input, the image I. In some cases, the artistic content generation system106utilizes, as the feature extractor described by Karen Simonyan, referenced above. In one or more embodiments, ϕl(I) has the dimensions Wl×Hland Clchannels, S={2,4,8,12,16}. In one or more embodiments, the artistic content generation system106utilizes the loss function represented by equation 8 to preserve the content of the input image but also to transfer a part of the training set style to the output.

In some embodiments, the artistic content generation system106further utilizes an adversarial loss function defined as:

In equation 9, the first term represents the discriminator, and the second term represents the generator. Further, σ(x)=1/1+e−xis the sigmoid function, Y is the set of high-resolution painting images in the dataset, X is the set of painting images in the dataset that is subject to the Gaussian filter and down-sampling with reduction rate of 2r. In one or more embodiments, the artistic content generation system106utilizes the generator G(·) to attempt to generate high-resolution images similar to the real ones (e.g., the ground truths) and utilizes the discriminator D(·) to try to distinguish between the resulting fake G(F) image, F∈X and the corresponding real painting image R∈Y. In some cases, the artistic content generation system106utilizes the loss function represented by equation 9 to generate images that are similar to real-world art, enhancing particular art details.

In one or more embodiments, the artistic content generation system106combines the loss functions represented by equations 7-9 into an overall loss function as follows:

Though equation 10 illustrates particular values for each of the weights, it should be understood that the weights vary in various embodiments. For instance, in some implementations, the weights are configurable.

Thus, in the artistic content generation system106utilizes the loss function represented by equation 10 (or one of the loss functions represented by equations 7-9 or a combination of the loss functions represented by equations 7-9) to train an artistic superzoom neural network. In particular, the artistic content generation system106utilizes the loss function(s) to iteratively modify parameters of the superzoom artistic neural network, enabling the superzoom artistic neural network to reduce the error by which it produces outputs.

As mentioned, in one or more embodiments, the artistic content generation system106performs one or more additional operations via the artistic image neural network to speed up convergence. In particular, the artistic image neural network implements the one or more additional operations to increase the degree to which the style parameters are incorporated into the generated images at each iteration.FIGS.7A-7Billustrate diagrams for utilizing one or more additional operations to increase the speed of convergence in accordance with one or more embodiments.FIG.7Cillustrates graphical representations of the effects of these operations in accordance with one or more embodiments.

For example,FIG.7Aillustrates a diagram for utilizing fractal noise during the process for generating an artistic digital image in accordance with one or more embodiments. Indeed, as shown inFIG.7A, the artistic content generation system106adds fractal noise702to the digital image704having content at the beginning of the process for generating an artistic digital image. For example, in some embodiments, the artistic content generation system106adds the fractal noise702to the alpha channel of the digital image704. In one or more embodiments, the artistic content generation system106defines the fractal noise702added to the digital image704as follows:

In equation 11, (w,h) represents the size of the generated noise, (gx, gy) represents the size of the grid used in PN—Perlin Noise (a gradient noise with at least some degree of coherent structure. Further, n=[log2max(w,h)]−3 represents the number of octaves used. In one or more embodiments, the artistic content generation system106determines the degree of detail of the final noise by obtaining the different octaves of the Perlin Noise. In some cases, each octave includes a degree of detail, and the artistic content generation system106utilizes 1, which represents lacunarity, to determine how much detail is added to each octave by controlling the size of the gradient grid in the Perlin Noise. Further, A represents the amplitude, indicating the importance of each octave in the final result.

As shown inFIG.7A, the artistic content generation system106passes the digital image704with the fractal noise702to the artistic image neural network706. As discussed above with reference toFIG.5, the artistic image neural network706utilizes the digital image704to generate a set of intermediate artistic digital images via a set of iterations of an optimization loop708. At the last iteration of the set of iterations (e.g., the last iteration for the current resolution), the artistic image neural network706generates the intermediate artistic digital image710.

As further shown inFIG.7A, the artistic content generation system106adds the fractal noise702to the intermediate artistic digital image710and passes the intermediate artistic digital image710with the fractal noise702back to the artistic image neural network706for further processing. Accordingly, in some implementations, the artistic content generation system106also adds fractal noise to the generated image at each up-sampling or down-sampling in the scale hierarchy. ThoughFIG.7Ashows adding the fractal noise702before an image is passed to the artistic image neural network706, the artistic content generation system106adds the fractal noise702during or after the resizing process in some cases.

FIG.7Billustrates a diagram for utilizing an augmentation chain during the process for generating an artistic digital image in accordance with one or more embodiments. In particular, as shown inFIG.7B, the artistic content generation system106passes an intermediate artistic digital image720that is to be encoded by a neural network image encoder722through an augmentation chain724. The artistic content generation system106utilizes the augmentation chain724to modify the intermediate artistic digital image720with one or more transformation operations.

As shown inFIG.7B, the augmentation chain724includes a resize operation726(e.g., for randomly resizing each dimension independently), a crop operation728, a perspective operation730, an image flip operation732(e.g., to flip the image horizontally), and a noise operation734(e.g., to add random Gaussian noise). The augmentation chain724contains additional or fewer transformation operations in various embodiments. Further, thoughFIG.7Billustrates a particular sequence of transformation operations, the artistic content generation system106utilizes various sequences in different embodiments. In some cases, the artistic content generation system106passes the intermediate artistic digital image720through the augmentation chain724and subsequently generates the set of transformed intermediate artistic digital images as discussed above with reference toFIG.4. In some cases, the artistic content generation system106passes the set of transformed intermediate artistic digital images through the augmentation chain724.

In one or more embodiments, the artistic content generation system106utilizes fractal noise and an augmentation chain to remove regular surfaces from the images used in the generative process, eliminating the problem of vanishing gradients experienced by many conventional systems. Indeed, in some instances, the artistic content generation system106utilizes the fractal noise to generate artifacts and then uses the augmentation chain to transform the artifacts into details, increasing the speed of the optimization.

FIG.7Cillustrates graphical representations reflecting the effects of fractal noise and an augmentation chain on the process of generating an artistic digital image in accordance with one or more embodiments. In particular,FIG.7Cillustrates the artistic digital image generated utilizing an artistic image neural network after ten iterations and twenty iterations. As shown inFIG.7C, the resulting artistic digital image incorporates significantly more stylistic elements when both the fractal noise and the augmentation chain is used. Even those artistic digital images resulting from use of the fractal noise without the augmentation chain or vice versa appear more stylistic than the artistic digital image that results when neither is used. Accordingly, the graphical representations illustrated inFIG.7Cindicate that the artistic image neural network converges (i.e., incorporates the style parameters) more quickly when fractal noise and/or an augmentation chain is used.

As previously mentioned, the artistic content generation system106enables implementing computing devices to more accurately and flexibly incorporate style parameters into artistic digital images when compared to conventional studies. Researchers have conducted studies to determine the accuracy and flexibility of one or more embodiments of the artistic content generation system106.FIGS.8-10illustrate graphical representations reflecting experimental results regarding the effectiveness of the artistic content generation system106in accordance with one or more embodiments.

For example,FIG.8illustrates artistic digital images generated by one or more embodiments of the artistic content generation system106based on a style digital image and a digital image having content for the artistic digital image. The graphical representations ofFIG.8compare the performance of the artistic content generation system106to the performance a system performing image style transfer using convolutional neural networks as described by Leon A Gatys et al.,Image Style Transfer Using Convolutional Neural Networks, CVPR, pp. 2414-2423, arXiv:1508.06576, 2016. The graphical representations further illustrate the performance of a system implementing universal style transfer via feature transforms as described by Yijun Li et al.,Universal Style Transfer via Feature Transforms, NIPS, 2017, arXiv:1705.08086.

As shown inFIG.8, the artistic content generation system106generates artistic digital images that more accurately capture the style parameters provided by the respective style digital images. For instance, the artistic content generation system106alters the structure of the provided content to more closely adhere to the artistic style presented in the style digital image.

FIG.9illustrates artistic digital images generated by one or more embodiments of the artistic content generation system106based on a style text prompt and a digital image having content for the artistic digital image. The graphical representations ofFIG.9compare the performance of the artistic content generation system106to the performance of a system implementing text-guided image manipulation described by Bowen Li et al.,ManiGAN: Text-Guided Image Manipulation, CVPR, 2020, arXiv:1912.06203.

As shown inFIG.9, the artistic content generation system106generates artistic digital images more flexibly than the other tested system. Indeed, the artistic content generation system106utilizes the style text prompt to modify the structure of the provided content and incorporate the corresponding style parameters. Comparatively, the resulting image generated by the other system provides a photorealistic representation of the provided content with minor variations to color, texture, or background.

FIG.10illustrates artistic digital images generated by one or more embodiments of the artistic content generation system106based on a style text prompt. The graphical representations ofFIG.9compare the performance of the artistic content generation system106to the performance of the DALL-E model as described in Open AI Blog,DALL-E. Creating Images from Text,5 Jan. 2021, https://openai.com/blog/dall-e. As withFIG.9, the graphical representations ofFIG.10illustrate the flexibility of the artistic content generation system106as it can generate artistic digital images solely based on text style prompts while the other model merely generates photo-realistic representations.

Turning now toFIG.11, additional detail will now be provided regarding various components and capabilities of the artistic content generation system106. In particular,FIG.11illustrates the artistic content generation system106implemented by the computing device1100(e.g., the server(s)102and/or one of the client devices110a-110bdiscussed above with reference toFIG.1). Additionally, the artistic content generation system106is also part of the image editing system104. As shown inFIG.11, the artistic content generation system106includes, but is not limited to, a neural network training engine1102, a neural network application manager1104, and data storage1106(which includes artistic image neural network1108and training data1110).

As just mentioned, and as illustrated inFIG.11, the artistic content generation system106includes the neural network training engine1102. In one or more embodiments, the neural network training engine1102trains an artistic image neural network to generate artistic digital images from style digital images, style text prompts, and/or digital image providing content for the artistic digital images. In particular, in some implementations, the neural network training engine1102trains one or more artistic superzoom neural networks of an artistic image neural network to increase the resolution of an image input and/or add one or more art details associated with a physical visual medium.

Further, as shown inFIG.11, the artistic content generation system106includes the neural network application manager1104. In one or more embodiments, the neural network application manager1104utilizes the artistic image neural network trained by the neural network training engine1102to generate artistic digital images. For instance, in some cases, the neural network application manager1104utilizes the artistic image neural network to generate an artistic digital image that incorporates style parameters from at least one style digital image and/or at least one style text prompt. Further, in some cases, the neural network application manager1104utilizes the artistic image neural network to generate an artistic digital image having content from another digital image.

Additionally, as shown, the artistic content generation system106includes data storage1106. In particular, data storage1106(implemented by one or more memory devices) includes artistic image neural network1108and training data1110. In one or more embodiments, the artistic image neural network1108stores the artistic image neural network trained by the neural network training engine1102and utilized by the neural network application manager1104. In some cases, training data1110stores the training data utilized by the neural network training engine1102to train an artistic image neural network. For instance, in some implementations, training data1110stores the images of paintings utilized to train the one or more artistic superzoom neural networks of the artistic image neural network. The data storage1106can also include input digital images, style parameters (e.g., style images and/or text prompts), and artistic digital images.

Each of the components1102-1110of the artistic content generation system106can include software, hardware, or both. For example, the components1102-1110can include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices, such as a client device or server device. When executed by the one or more processors, the computer-executable instructions of the artistic content generation system106can cause the computing device(s) to perform the methods described herein. Alternatively, the components1102-1110can include hardware, such as a special-purpose processing device to perform a certain function or group of functions. Alternatively, the components1102-1110of the artistic content generation system106can include a combination of computer-executable instructions and hardware.

Furthermore, the components1102-1110of the artistic content generation system106may, for example, be implemented as one or more operating systems, as one or more stand-alone applications, as one or more modules of an application, as one or more plug-ins, as one or more library functions or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components1102-1110of the artistic content generation system106may be implemented as a stand-alone application, such as a desktop or mobile application. Furthermore, the components1102-1110of the artistic content generation system106may be implemented as one or more web-based applications hosted on a remote server. Alternatively, or additionally, the components1102-1110of the artistic content generation system106may be implemented in a suite of mobile device applications or “apps.” For example, in one or more embodiments, the artistic content generation system106can comprise or operate in connection with digital software applications such as ADOBE® PHOTOSHOP®, ADOBE® AFTER EFFECTS®, or ADOBE® ILLUSTRATOR®. The foregoing are either registered trademarks or trademarks of Adobe Inc. in the United States and/or other countries.

FIGS.1-11, the corresponding text, and the examples provide a number of different methods, systems, devices, and non-transitory computer-readable media of the artistic content generation system106. In addition to the foregoing, one or more embodiments can also be described in terms of flowcharts comprising acts for accomplishing the particular result, as shown inFIG.12.FIG.12may be performed with more or fewer acts. Further, the acts may be performed in different orders. Additionally, the acts described herein may be repeated or performed in parallel with one another or in parallel with different instances of the same or similar acts.

FIG.12illustrates a flowchart of a series of acts1200for generating an artistic digital image utilizing an artistic image neural network in accordance with one or more embodiments.FIG.12illustrates acts according to one embodiment, alternative embodiments may omit, add to, reorder, and/or modify any of the acts shown inFIG.12. In some implementations, the acts ofFIG.12are performed as part of a method. For example, in some embodiments, the acts ofFIG.12are performed, in a digital medium environment for creating digital content, as part of a computer-implemented method for generating digital visual art. Alternatively, a non-transitory computer-readable medium can store instructions thereon that, when executed by at least one processor, cause a computing device to perform operations comprising the acts ofFIG.12. In some embodiments, a system performs the acts ofFIG.12. For example, in one or more embodiments, a system includes one or more memory devices comprising an artistic image neural network that includes an artistic generative neural network, a learnable tensor, and an artistic superzoom neural network. The system further includes one or more server devices configured to cause the system to perform the acts ofFIG.12.

The series of acts1200includes an act1202for generating an initialized artistic digital image based on a learnable tensor. For example, in some embodiments, the act1202involves generating, utilizing an artistic generative neural network of an artistic image neural network, an initialized artistic digital image based on a learnable tensor.

In one or more embodiments, the artistic content generation system106receives a digital image comprising content for creating the artistic digital image and initializes the parameters of the learnable tensor based on the digital image utilizing an encoder of the artistic generative neural network. Accordingly, in some instances, the artistic content generation system106generates, utilizing the artistic generative neural network, the initialized artistic digital image based on the learnable tensor by generating, utilizing a decoder of the artistic generative neural network, the initialized artistic digital image based on the learnable tensor with the initialized parameters. In some implementations, the artistic content generation system106modifies the digital image utilizing fractal noise and initializes the parameters of the learnable tensor based on the digital image utilizing the encoder of the artistic generative neural network by initializing the parameters of the learnable tensor based on the digital image with the fractal noise utilizing the encoder of the artistic generative neural network.

Additionally, the series of acts1200include an act1204for determining style encodings for style parameters. For instance, in some cases, the act1204involves determining, utilizing a multi-domain style encoder of the artistic image neural network, one or more style encodings for one or more style parameters.

In some implementations, the artistic content generation system106receives at least one of a style digital image that includes the one or more style parameters or a style text prompt that includes the one or more style parameters. Accordingly, in some cases, the artistic content generation system106determines the one or more style encodings for the one or more style parameters by generating the one or more style encodings within a multi-domain encoding space from the at least one of the style digital image or the style text prompt utilizing the multi-domain style encoder.

The series of acts1200also includes an act1206of updating the learnable tensor using the initialized artistic digital image and the style encodings. To illustrate, in some instances, the act1206involves updating parameters of the learnable tensor by comparing the initialized artistic digital image to the one or more style encodings.

In one or more embodiments, the artistic content generation system106modifies the initialized artistic digital image utilizing an augmentation chain of transformation operations. Accordingly, in some cases, the artistic content generation system106compares the initialized artistic digital image to the one or more style encodings by comparing the modified initialized artistic digital image to the one or more style encodings.

Further, in some cases, the artistic content generation system106generates artistic encodings within a multi-domain encoding space from the initialized artistic digital image utilizing a neural network image encoder; and compares the initialized artistic digital image to the one or more style encodings by comparing the artistic encodings to the one or more style encodings within the multi-domain encoding space.

Further, the series of acts1200includes an act1208of generating an artistic digital image based on the updated parameters of the learnable tensor. For example, in one or more embodiments, the act1208involves generating, utilizing the artistic generative neural network, an artistic digital image based on the learnable tensor with the updated parameters.

In one or more embodiments, the artistic content generation system106further modifies the parameters of the learnable tensor. For instance, in some cases, the artistic content generation system106modifies the updated parameters of the learnable tensor by utilizing a plurality of iterations to: generate, utilizing the artistic generative neural network, an intermediate artistic digital image based on the learnable tensor with the updated parameters; and modify the updated parameters of the learnable tensor based on comparing the intermediate artistic digital image to the one or more style encodings. Accordingly, in some embodiments, the artistic content generation system106generates the artistic digital image based on the learnable tensor with the updated parameters by generating the artistic digital image based on learnable tensor with the modified parameters.

In some cases, the artistic content generation system106utilizes the plurality of iterations to generate, utilizing the artistic generative neural network, the intermediate artistic digital image by: generating, via a first set of iterations and utilizing the artistic generative neural network, a first set of intermediate artistic digital images corresponding to a first image resolution; and generating, via a second set of iterations and utilizing the artistic generative neural network, a second set of intermediate artistic digital images corresponding to a second image resolution.

In some embodiments, the series of acts1200further includes acts for modifying the artistic digital image. For instance, in some cases, the acts include modifying the artistic digital image to include one or more art details associated with a physical visual medium utilizing an artistic superzoom neural network.

To provide an illustration, in one or more embodiments, the artistic content generation system106receives a set of style parameters for creating an artistic digital image; and generates the artistic digital image utilizing the set of style parameters by iteratively: generating an intermediate artistic digital image based on the learnable tensor utilizing the artistic generative neural network of an artistic image neural network; comparing the intermediate artistic digital image to the set of style parameters; and updating parameters of the learnable tensor based on comparing the intermediate artistic digital image to the set of style parameters. Further, the artistic content generation system106modifies the artistic digital image to include one or more art details associated with a physical visual medium utilizing the artistic superzoom neural network of the artistic image neural network.

In some cases, the artistic content generation system106receives the set of style parameters for creating the artistic digital image by receiving one or more style digital images that include style parameters and one or more style text prompts that include additional style parameters. Additionally, in some embodiments, the artistic content generation system106initializes the parameters of the learnable tensor by selecting a point within an encoding space associated with the artistic generative neural network.

In one or more embodiments, the artistic content generation system106further generates the artistic digital image utilizing the set of style parameters by iteratively: modifying the intermediate artistic digital image utilizing an augmentation chain of transformation operations comprising at least one of a resize operation, a crop operation, a perspective operation, an image flip operation, or a noise operation; and comparing the intermediate artistic digital image to the set of style parameters by comparing the modified intermediate artistic digital image to the set of style parameters.

In some cases, the artistic content generation system106utilizes various sets of iterations in generating the artistic digital image. For instance, in some embodiments, the artistic content generation system106generates the artistic digital image utilizing the set of style parameters by: generating, via a first set of iterations and utilizing the artistic generative neural network, a first set of intermediate artistic digital images corresponding to a first image resolution; and generating, via a second set of iterations and utilizing the artistic generative neural network, a second set of intermediate artistic digital images corresponding to a second image resolution that is higher than the first image resolution.

In some cases, the artistic content generation system106receives a digital image comprising content for creating the artistic digital image. Accordingly, in some implementations, the artistic content generation system106generates the first set of intermediate artistic digital images utilizing the digital image at the first image resolution; and up-samples an intermediate artistic digital image from the first set of intermediate artistic digital images for use in the second set of iterations utilizing an additional artistic superzoom neural network. In some embodiments, the artistic content generation system106modifies the digital image for use in the first set of iterations utilizing fractal noise; and modifies the intermediate artistic digital image from the first set of intermediate artistic digital images for use in the second set of iterations utilizing additional fractal noise.

In one or more embodiments, the artistic content generation system106compares the intermediate artistic digital image to the set of style parameters by comparing the intermediate artistic digital image to the set of style parameters utilizing a style loss and at least one of a pixel loss or a perceptual loss corresponding to a digital image comprising content for creating the artistic digital image.

To provide another illustration, in one or more embodiments, the artistic content generation system106receives, from a computing device, a digital image and one or more style parameters comprising at least one of a style digital image or a style text prompt; determines one or more style encodings for the one or more style parameters; iteratively utilizes the one or more style encodings to generate an artistic digital image from the digital image; and provides the artistic digital image for display via the computing device.

In some implementations, the artistic content generation system106receives the one or more style parameters comprising the at least one of the style digital image or the style text prompt by receiving the style digital image. Accordingly, in some cases, the artistic content generation system106further generates a set of transformed style digital images from the style digital image by cropping the style digital image utilizing at least one of a variable cropping size or a variable cropping offset. Further, in some instances, the artistic content generation system106determining the one or more style encodings for the one or more style parameters by determining a plurality of style encodings from the set of transformed style digital images.

FIG.13illustrates a block diagram of an example computing device1300that may be configured to perform one or more of the processes described above. One will appreciate that one or more computing devices, such as the computing device1300may represent the computing devices described above (e.g., the server(s)102and/or the client devices110a-110b). In one or more embodiments, the computing device1300may be a mobile device (e.g., a mobile telephone, a smartphone, a PDA, a tablet, a laptop, a camera, a tracker, a watch, a wearable device). In some embodiments, the computing device1300may be a non-mobile device (e.g., a desktop computer or another type of client device). Further, the computing device1300may be a server device that includes cloud-based processing and storage capabilities.

As shown inFIG.13, the computing device1300can include one or more processor(s)1302, memory1304, a storage device1306, input/output interfaces1308(or “I/O interfaces1308”), and a communication interface1310, which may be communicatively coupled by way of a communication infrastructure (e.g., bus1312). While the computing device1300is shown inFIG.13, the components illustrated inFIG.13are not intended to be limiting. Additional or alternative components may be used in other embodiments. Furthermore, in certain embodiments, the computing device1300includes fewer components than those shown inFIG.13. Components of the computing device1300shown inFIG.13will now be described in additional detail.

In particular embodiments, the processor(s)1302includes hardware for executing instructions, such as those making up a computer program. As an example, and not by way of limitation, to execute instructions, the processor(s)1302may retrieve (or fetch) the instructions from an internal register, an internal cache, memory1304, or a storage device1306and decode and execute them.

The computing device1300includes memory1304, which is coupled to the processor(s)1302. The memory1304may be used for storing data, metadata, and programs for execution by the processor(s). The memory1304may include one or more of volatile and non-volatile memories, such as Random-Access Memory (“RAM”), Read-Only Memory (“ROM”), a solid-state disk (“SSD”), Flash, Phase Change Memory (“PCM”), or other types of data storage. The memory1304may be internal or distributed memory.

The computing device1300includes a storage device1306including storage for storing data or instructions. As an example, and not by way of limitation, the storage device1306can include a non-transitory storage medium described above. The storage device1306may include a hard disk drive (HDD), flash memory, a Universal Serial Bus (USB) drive or a combination these or other storage devices.

As shown, the computing device1300includes one or more I/O interfaces1308, which are provided to allow a user to provide input to (such as user strokes), receive output from, and otherwise transfer data to and from the computing device1300. These I/O interfaces1308may include a mouse, keypad or a keyboard, a touch screen, camera, optical scanner, network interface, modem, other known I/O devices or a combination of such I/O interfaces1308. The touch screen may be activated with a stylus or a finger.

The computing device1300can further include a communication interface1310. The communication interface1310can include hardware, software, or both. The communication interface1310provides one or more interfaces for communication (such as, for example, packet-based communication) between the computing device and one or more other computing devices or one or more networks. As an example, and not by way of limitation, communication interface1310may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI. The computing device1300can further include a bus1312. The bus1312can include hardware, software, or both that connects components of computing device1300to each other.