Patent Description:
An inpainting system is configured to inpaint a missing region of an input image based on a guide image. The input image and the guide image may be different images, and the guide image may thus provide new and controllable image content for guiding the inpainting of the missing region. The inpainting system includes one or more encoders configured to generate a first latent representation of the input image and a second latent representation of the guide image. The first and second latent representations are combined to generate a combined latent representation by, for example, concatenating the first and second latent representations and/or determining a cross-attention therebetween. The inpainting system also includes a style generative adversarial network (StyleGAN) configured to generate an intermediate output image based on the combined latent representation. In particular, the StyleGAN may be arranged and trained such that the intermediate output image includes inpainted image content for the missing region of the input image, and the inpainted image content may include a combination of visual features of the guide image and the input image. The inpainting system is configured to generate an output image by replacing the missing region of the input image with the inpainted image content.

In a first embodiment according to the invention, a method as defined in claim <NUM> is provided.

In a second embodiment according to the invention, a system as defined in claim <NUM> is provided.

In a third embodiment according to the invention, a non-transitory computer-readable medium as defined in claim <NUM> is provided.

In a fourth example embodiment, a system may include various means for carrying out each of the operations of the first embodiment according to the invention.

These, as well as other embodiments, aspects, advantages, and alternatives, will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference where appropriate to the accompanying drawings. Further, this summary and other descriptions and figures provided herein are intended to illustrate embodiments by way of example only and, as such, that numerous variations are possible.

Example methods, devices, and systems are described herein. It should be understood that the words "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration. " Any embodiment or feature described herein as being an "example," "exemplary," and/or "illustrative" is not necessarily to be construed as preferred or advantageous over other embodiments or features unless stated as such.

Image inpainting may involve generating image content for a missing region (i.e., a region to be inpainted) of an incomplete image based on image content of remaining regions of the incomplete image. However, inpainting performance may be poor when, for example, the missing region is relatively large in relation to the incomplete image and/or parts of the incomplete image include complex patterns. For example, inpainted image content for relatively large missing regions may include undesirable artifacts such as tessellations. Additionally, inpainting the missing region based on the image content of the remaining regions of the incomplete image might not allow for control of the content and/or style of the generated image content. Specifically, it may be difficult to control the extent to which different portions of the remaining regions of the incomplete image influence the inpainted image content, and/or to introduce new content from another image into the inpainted region.

Accordingly, an inpainting system may be configured to use a guide image to assist with the inpainting of a missing region of an input image. The inpainting system may be configured to combine visual features of the guide image with visual features of the remaining regions of the input image. Thus, the inpainting system may be configured to generate inpainted image content that blends visual features of both the guide image and the input image in a manner that is visually-plausible, coherent, realistic, natural, and/or otherwise visually desirable. For example, the inpainted image content may be based on, and thus visually resemble, a color palette of the guide image, geometric features present in the guide image, a frequency content of the guide image, statistical properties of the guide image, and/or a semantic content of the guide image, among other visual features, properties, and/or attributes of the guide image. Thus, the inpainting system may use the guide image to (i) control the visual appearance of the inpainted image content and (ii) provide additional image data beyond that available in the remaining portions of the input image, thereby improving the quality of the inpainted image content. The inpainting system may, therefore, allow effective inpainting of regions that were not previously possible to effectively inpaint.

The inpainting system may include one or more encoders, a latent representation operator, a StyleGAN model, and an output image generator. The one or more encoders may be configured to generate a first latent representation of the input image and a second latent representation of the guide image. The latent representation may include, for example, a vector and/or a feature map, among other possibilities. The latent representation operator may be configured to determine a combined latent representation of the input and guide images by, for example, concatenating the first and second latent representations and/or by determining a cross-attention therebetween.

The combined latent representation may be provided as input to a StyleGAN model, which may be configured to generate an intermediate output image that includes generated image content for the missing region of the input image. Components of the StyleGAN model may be arranged to allow the StyleGAN to inpaint the missing region of the input image by combining the visual features of the input image and the guide image, as represented by the combined latent representation. In particular, the StyleGAN model may be configured to perform at least part of the combination of the visual features in a deep latent and/or feature space, rather than in pixel space, thereby improving the quality of the inpainted image content. The output image generator may be configured to generate an output image by combining the inpainted image content of the intermediate output image with the input image according to an indication of the region to be inpainted (e.g., an inpainting mask) that indicates the missing region of the input image.

The inpainted image content for a given input image may be controlled by selecting a guide image that has a target set of visual features. Thus, for example, a user may be able to cause the inpainting model to generate inpainted image content with a desired visual appearance by providing the inpainting model with a guide image that has and/or approximates the desired visual appearance. In another example, a guide image may be selected from a plurality of candidate guide images based on a similarity between the input image and each of the plurality of guide images. For example, a guide image that is most similar to the input image may be selected, and the resulting inpainted image content may thus visually match the remaining parts of the input image more accurately than if a dissimilar guide image were used instead. Further, multiple different output images may be generated based on the given input image by inpainting the missing region based on multiple different guide images, each of which may include a different set of visual features.

<FIG> illustrates an example computing device <NUM>. Computing device <NUM> is shown in the form factor of a mobile phone. However, computing device <NUM> may be alternatively implemented as a laptop computer, a tablet computer, and/or a wearable computing device, among other possibilities. Computing device <NUM> may include various elements, such as body <NUM>, display <NUM>, and buttons <NUM> and <NUM>. Computing device <NUM> may further include one or more cameras, such as front-facing camera <NUM> and rear-facing camera <NUM>.

Front-facing camera <NUM> may be positioned on a side of body <NUM> typically facing a user while in operation (e.g., on the same side as display <NUM>). Rear-facing camera <NUM> may be positioned on a side of body <NUM> opposite front-facing camera <NUM>. Referring to the cameras as front and rear facing is arbitrary, and computing device <NUM> may include multiple cameras positioned on various sides of body <NUM>.

Display <NUM> could represent a cathode ray tube (CRT) display, a light emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light emitting diode (OLED) display, or any other type of display known in the art. In some examples, display <NUM> may display a digital representation of the current image being captured by front-facing camera <NUM> and/or rear-facing camera <NUM>, an image that could be captured by one or more of these cameras, an image that was recently captured by one or more of these cameras, and/or a modified version of one or more of these images. Thus, display <NUM> may serve as a viewfinder for the cameras. Display <NUM> may also support touchscreen functions that may be able to adjust the settings and/or configuration of one or more aspects of computing device <NUM>.

Front-facing camera <NUM> may include an image sensor and associated optical elements such as lenses. Front-facing camera <NUM> may offer zoom capabilities or could have a fixed focal length. In other examples, interchangeable lenses could be used with front-facing camera <NUM>. Front-facing camera <NUM> may have a variable mechanical aperture and a mechanical and/or electronic shutter. Front-facing camera <NUM> also could be configured to capture still images, video images, or both. Further, front-facing camera <NUM> could represent, for example, a monoscopic, stereoscopic, or multiscopic camera. Rear-facing camera <NUM> may be similarly or differently arranged. Additionally, one or more of front-facing camera <NUM> and/or rear-facing camera <NUM> may be an array of one or more cameras.

One or more of front-facing camera <NUM> and/or rear-facing camera <NUM> may include or be associated with an illumination component that provides a light field to illuminate a target object. For instance, an illumination component could provide flash or constant illumination of the target object. An illumination component could also be configured to provide a light field that includes one or more of structured light, polarized light, and light with specific spectral content. Other types of light fields known and used to recover three-dimensional (3D) models from an object are possible within the context of the examples herein.

Computing device <NUM> may also include an ambient light sensor that may continuously or from time to time determine the ambient brightness of a scene that cameras <NUM> and/or <NUM> can capture. In some implementations, the ambient light sensor can be used to adjust the display brightness of display <NUM>. Additionally, the ambient light sensor may be used to determine an exposure length of one or more of cameras <NUM> or <NUM>, or to help in this determination.

Computing device <NUM> could be configured to use display <NUM> and front-facing camera <NUM> and/or rear-facing camera <NUM> to capture images of a target object. The captured images could be a plurality of still images or a video stream. The image capture could be triggered by activating button <NUM>, pressing a softkey on display <NUM>, or by some other mechanism. Depending upon the implementation, the images could be captured automatically at a specific time interval, for example, upon pressing button <NUM>, upon appropriate lighting conditions of the target object, upon moving computing device <NUM> a predetermined distance, or according to a predetermined capture schedule.

<FIG> is a simplified block diagram showing some of the components of an example computing system <NUM>. By way of example and without limitation, computing system <NUM> may be a cellular mobile telephone (e.g., a smartphone), a computer (such as a desktop, notebook, tablet, server, or handheld computer), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a gaming console, a robotic device, a vehicle, or some other type of device. Computing system <NUM> may represent, for example, aspects of computing device <NUM>.

As shown in <FIG>, computing system <NUM> may include communication interface <NUM>, user interface <NUM>, processor <NUM>, data storage <NUM>, and camera components <NUM>, all of which may be communicatively linked together by a system bus, network, or other connection mechanism <NUM>. Computing system <NUM> may be equipped with at least some image capture and/or image processing capabilities. It should be understood that computing system <NUM> may represent a physical image processing system, a particular physical hardware platform on which an image sensing and/or processing application operates in software, or other combinations of hardware and software that are configured to carry out image capture and/or processing functions.

Communication interface <NUM> may allow computing system <NUM> to communicate, using analog or digital modulation, with other devices, access networks, and/or transport networks. Thus, communication interface <NUM> may facilitate circuit-switched and/or packet-switched communication, such as plain old telephone service (POTS) communication and/or Internet protocol (IP) or other packetized communication. For instance, communication interface <NUM> may include a chipset and antenna arranged for wireless communication with a radio access network or an access point. Also, communication interface <NUM> may take the form of or include a wireline interface, such as an Ethernet, Universal Serial Bus (USB), or High-Definition Multimedia Interface (HDMI) port, among other possibilities. Communication interface <NUM> may also take the form of or include a wireless interface, such as a Wi-Fi, BLUETOOTH®, global positioning system (GPS), or wide-area wireless interface (e.g., WiMAX or 3GPP Long-Term Evolution (LTE)), among other possibilities. However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used over communication interface <NUM>. Furthermore, communication interface <NUM> may comprise multiple physical communication interfaces (e.g., a Wi-Fi interface, a BLUETOOTH® interface, and a wide-area wireless interface).

User interface <NUM> may function to allow computing system <NUM> to interact with a human or non-human user, such as to receive input from a user and to provide output to the user. Thus, user interface <NUM> may include input components such as a keypad, keyboard, touch-sensitive panel, computer mouse, trackball, joystick, microphone, and so on. User interface <NUM> may also include one or more output components such as a display screen, which, for example, may be combined with a touch-sensitive panel. The display screen may be based on CRT, LCD, LED, and/or OLED technologies, or other technologies now known or later developed. User interface <NUM> may also be configured to generate audible output(s), via a speaker, speaker jack, audio output port, audio output device, earphones, and/or other similar devices. User interface <NUM> may also be configured to receive and/or capture audible utterance(s), noise(s), and/or signal(s) by way of a microphone and/or other similar devices.

In some examples, user interface <NUM> may include a display that serves as a viewfinder for still camera and/or video camera functions supported by computing system <NUM>. Additionally, user interface <NUM> may include one or more buttons, switches, knobs, and/or dials that facilitate the configuration and focusing of a camera function and the capturing of images. It may be possible that some or all of these buttons, switches, knobs, and/or dials are implemented by way of a touch-sensitive panel.

Processor <NUM> may comprise one or more general purpose processors - e.g., microprocessors - and/or one or more special purpose processors - e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, or application-specific integrated circuits (ASICs). In some instances, special purpose processors may be capable of image processing, image alignment, and merging images, among other possibilities. Data storage <NUM> may include one or more volatile and/or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and may be integrated in whole or in part with processor <NUM>. Data storage <NUM> may include removable and/or non-removable components.

Processor <NUM> may be capable of executing program instructions <NUM> (e.g., compiled or non-compiled program logic and/or machine code) stored in data storage <NUM> to carry out the various functions described herein. Therefore, data storage <NUM> may include a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by computing system <NUM>, cause computing system <NUM> to carry out any of the methods, processes, or operations disclosed in this specification and/or the accompanying drawings. The execution of program instructions <NUM> by processor <NUM> may result in processor <NUM> using data <NUM>.

By way of example, program instructions <NUM> may include an operating system <NUM> (e.g., an operating system kernel, device driver(s), and/or other modules) and one or more application programs <NUM> (e.g., camera functions, address book, email, web browsing, social networking, audio-to-text functions, text translation functions, and/or gaming applications) installed on computing system <NUM>. Similarly, data <NUM> may include operating system data <NUM> and application data <NUM>. Operating system data <NUM> may be accessible primarily to operating system <NUM>, and application data <NUM> may be accessible primarily to one or more of application programs <NUM>. Application data <NUM> may be arranged in a file system that is visible to or hidden from a user of computing system <NUM>.

Application programs <NUM> may communicate with operating system <NUM> through one or more application programming interfaces (APIs). These APIs may facilitate, for instance, application programs <NUM> reading and/or writing application data <NUM>, transmitting or receiving information via communication interface <NUM>, receiving and/or displaying information on user interface <NUM>, and so on.

In some cases, application programs <NUM> may be referred to as "apps" for short. Additionally, application programs <NUM> may be downloadable to computing system <NUM> through one or more online application stores or application markets. However, application programs can also be installed on computing system <NUM> in other ways, such as via a web browser or through a physical interface (e.g., a USB port) on computing system <NUM>.

Camera components <NUM> may include, but are not limited to, an aperture, shutter, recording surface (e.g., photographic film and/or an image sensor), lens, shutter button, infrared projectors, and/or visible-light projectors. Camera components <NUM> may include components configured for capturing of images in the visible-light spectrum (e.g., electromagnetic radiation having a wavelength of <NUM> - <NUM> nanometers) and/or components configured for capturing of images in the infrared light spectrum (e.g., electromagnetic radiation having a wavelength of <NUM> nanometers - <NUM> millimeter), among other possibilities. Camera components <NUM> may be controlled at least in part by software executed by processor <NUM>.

<FIG> illustrates inpainting system <NUM> that may be configured to inpaint a region of an input image based on a guide image. Specifically, inpainting system <NUM> may include encoder model <NUM>, encoder model <NUM>, latent representation operator <NUM>, StyleGAN model <NUM>, and output image generator <NUM>. Inpainting system <NUM> may be configured to generate output image <NUM> based on input image <NUM>, guide image <NUM>, and inpainting mask <NUM>.

Input image <NUM> may represent any type of image data including, for example, a red-green-blue (RGB) image and/or a grayscale image, among other possibilities. Input image <NUM> may include a region to be inpainted, which may be indicated and/or defined by and/or based on an indication of the region to be inpainted, such as, for example, inpainting mask <NUM>. The region to be inpainted may include one or more pixels of input image <NUM> that are intended to be assigned new pixel values, and may thus indicate one or more portions of input image <NUM> that are to be replaced with new image content. The region to be inpainted may alternatively be referred to as a missing region, an erased region, and/or a removed region, among other possibilities. The region to be inpainted in input image <NUM> may have various shapes and/or sizes.

In one example, inpainting mask <NUM> may include a binary image that uses a first binary value (e.g., <NUM>) to indicate pixels of input image <NUM> to be inpainted and a second binary value (e.g., <NUM>) to indicate pixels of input image <NUM> to be preserved. In another example, inpainting mask <NUM> may include a grayscale image generated based on fitting a statistical distribution (e.g., a Gaussian distribution) to the remaining region of input image <NUM>, such that pixels near the region to be inpainted have higher values than pixels that are further away from the region to be inpainted. Such a grayscale image may be useful in quantifying a similarity of portions of input image <NUM> and guide image <NUM>, since similarity between input image <NUM> and guide image <NUM> may be more important for pixels near the region to be inpainted and may be less important for pixels that are further away from the region to be inpainted. In some implementations, inpainting mask <NUM> may have the same resolution as input image <NUM>.

In some implementations, the indication of the region to be inpainted may include, for example, a list of the pixel coordinates of input image <NUM> that are to be inpainted, and such a list may be arranged into a format other than a mask. Nonetheless, various possible formats of the indication of the region to be inpainted may be convertible to, and thus at least partially equivalent to, inpainting mask <NUM>. Thus, inpainting mask <NUM> is used herein as a representative example of various possible indications of the region to be inpainted in input image <NUM>.

Guide image <NUM> may represent any type of image data including, for example, a red-green-blue (RGB) image and/or a grayscale image, among other possibilities. Guide image <NUM> may be used by inpainting system <NUM> to guide the inpainting of the region to be inpainted in input image <NUM>, as indicated by inpainting mask <NUM>. Specifically, guide image <NUM> may contain image content that is to be incorporated into the region to be inpainted so that, following inpainting, this region at least partly visually resembles guide image <NUM>. Guide image <NUM> may alternatively be referred to as an example image and/or a reference image, among other possibilities.

The inpainted image content generated by StyleGAN model <NUM> may include a combination of visual features of input image <NUM> and visual features of guide image <NUM>. StyleGAN model <NUM> may be configured to incorporate the visual features of guide image <NUM> into the region to be inpainted in input image <NUM> in a manner that is contextually and/or visually consistent with the remaining (i.e., non-inpainted) parts of input image <NUM>, such that output image <NUM> looks realistic, natural, visually-plausible, and/or non-artificial.

Accordingly, guide image <NUM> may allow a user to control the inpainting process carried out by inpainting system <NUM>. Specifically, by selecting different instances of guide image <NUM>, the region to be inpainted in input image <NUM> may be inpainted with different visual features. Additionally, when the region to be inpainted is relatively large, and thus takes up more than a threshold fraction (e.g., <NUM>%, <NUM>%, <NUM>%) of input image <NUM>, it may be difficult to generate plausible image content for the region to be inpainted based on the remaining parts of input image <NUM>. That is, input image <NUM> might not include sufficient remaining image data to allow for satisfactory inpainting, or the remaining image data might not include image data that is useful for the region to be inpainted, for example because the region to be inpainted includes a portion that is relatively distinct from the remaining image data. For example, the region to be inpainted may have previously represented a building, but the remaining parts of input image <NUM> might not represent any buildings. Accordingly, guide image <NUM> may provide additional image content that allows for improved inpainting, and thus results in output image <NUM> looking more realistic, natural, and/or non-artificial than an output image based exclusively on input image <NUM>.

In order to perform such inpainting, encoder model <NUM> may be configured to generate latent representation <NUM> based on guide image <NUM>, and encoder model <NUM> may be configured to generate latent representation <NUM> based on input image <NUM>. Latent representation <NUM> may be a first tensor representing guide image <NUM> in a latent space, and latent representation <NUM> may be a second tensor representing input image <NUM> in the latent space. For example, latent representations <NUM> and <NUM> may each include one or more corresponding feature vectors (e.g., a row or column vector having N elements, such as a 1x512 feature vector), and/or one or more corresponding feature maps (e.g., D feature maps each having a width W and a height H, such as <NUM>8x8 feature maps), among other possibilities.

In some implementations, encoder models <NUM> and <NUM> may share the same architecture and parameters. That is, encoder models <NUM> and <NUM> may be two instances of the same model. In other implementations, encoder models <NUM> and <NUM> may have different architectures and/or parameters. Accordingly, one or more of latent representations <NUM> and/or <NUM> may be normalized to have a shared size and/or value range, among other possibilities. In one example, encoder model <NUM> and/or encoder model <NUM> may utilize a VGG architecture.

Latent representation operator <NUM> may be configured to generate combined latent representation <NUM> based on latent representations <NUM> and <NUM>. In one example, latent representation operator <NUM> may include a concatenation. Thus, combined latent representation <NUM> may include a concatenation of latent representations <NUM> and <NUM>. In another example, latent representation operator <NUM> may include a (first) cross-attention operator. Thus, combined latent representation <NUM> may include cross-attention value(s) between latent representations <NUM> and <NUM>. The cross-attention operator may include various possible variations of the attention operator, including global attention, window attention, random attention, multi-headed attention, soft attention, and/or hard attention, among other possibilities. Accordingly, combined latent representation <NUM> may represent the visual features of both input image <NUM> and guide image <NUM> and/or one or more relationships between these visual features. Specifically, combined latent representation <NUM> may represent input image <NUM> and guide image <NUM> as a subset of a deep latent and/or feature space, rather than using pixel space, and thus allow StyleGAN model <NUM> to better understand and/or combine the visual features of images <NUM> and <NUM>.

StyleGAN model <NUM> may be configured to generate intermediate output image <NUM> based on combined latent representation <NUM>. In some implementations, StyleGAN model <NUM> may be additionally configured to generate intermediate output image <NUM> based on skip-connection <NUM> with one or more intermediate layers of encoder model <NUM>, and/or a skip-connection (not shown) with one or more intermediate layers of encoder model <NUM>.

StyleGAN model <NUM> may represent one or more versions of the Style Generative Adversarial Network, as detailed in, for example, a paper titled "A Style-Based Generator Architecture for Generative Adversarial Networks," authored by Karras et al, and published as arXiv:<NUM> (i.e., StyleGAN1), a paper titled "<NPL>), a paper titled "<NPL>), and/or other variations thereof. Accordingly, StyleGAN model <NUM> may include an adaptive instance normalization component, a weight demodulation component, and/or an alias-free architecture, among other possibilities. Additionally or alternatively, StyleGAN model <NUM> may include architectural components of other types of neural networks configured to generate image data.

Intermediate output image <NUM> may include inpainted image content for at least the region to be inpainted in input image <NUM> and, in some cases, may additionally include new and/or modified image content for other regions of input image <NUM>. Intermediate output image <NUM> may have a same resolution as input image <NUM>, may be a scaled version of input image <NUM>, and/or may represent a subset of input image <NUM> that includes the region to be inpainted, among other possibilities.

Output image generator <NUM> may be configured to generate output image <NUM> based on input image <NUM>, inpainting mask <NUM>, and intermediate output image <NUM>. Specifically, output image generator <NUM> may be configured to replace the region to be inpainted in input image <NUM>, as indicated by inpainting mask <NUM>, with the inpainted image content of a corresponding portion of intermediate output image <NUM>. For example, output image generator <NUM> may implement the function IOUTPUT = IINPUT(<NUM> - M) + IINTERMEDIATEM, where IOUTPUT represents output image <NUM>, IINPUT represents input image <NUM>, IINTERMEDIATE represents intermediate output image <NUM>, M represents inpainting mask <NUM>, and (<NUM> - M) represents the residual of inpainting mask <NUM> (i.e., portions of input image <NUM> that are to remain unmodified). In some implementations, output image generator <NUM> may be configured to apply one or more blending functions to pixels around (e.g., within a threshold pixel distance of) the edges of the region to be inpainted, thereby creating a smooth transition between the inpainted content and the original content of input image <NUM>.

<FIG>, <FIG>, and <FIG> illustrate example architectures of StyleGAN model <NUM> that may be used by inpainting system <NUM>. Specifically, <FIG>, <FIG>, and <FIG> illustrate StyleGAN models 330A, 330B, and 330C, respectively, each of which represents a potential architecture for StyleGAN model <NUM>. Aspects of StyleGAN models 330A, 330B, and 330C may be combined and/or interchanged to achieve a desired quality of intermediate output image <NUM>, model size, model runtime, and/or other performance metrics.

Specifically, as shown in <FIG>, StyleGAN model 330A may include mapping network <NUM>, synthesis network <NUM>, and initialization mapping <NUM>, each of which may be learnable during training of StyleGAN model <NUM> and/or inpainting system <NUM>. In one example, mapping network <NUM> may include a plurality (e.g., <NUM>) fully-connected neural network layers arranged in a sequence. Synthesis network may include style block <NUM> and style blocks <NUM> through <NUM>, which may collectively be referred to as style blocks <NUM> - <NUM>. Each respective style block of style blocks <NUM> - <NUM> may be associated with a corresponding size scale. For example, style blocks <NUM> - <NUM> may include <NUM> style blocks, with style blocks <NUM> and <NUM> corresponding to a 4x4 resolution, the two style blocks subsequent to style block <NUM> corresponding to an 8x8 resolution, and style block <NUM> and its preceding style block corresponding to a 1024x1024 resolution. An output of a given style block may be provided as input to a subsequent style block, thus allowing style blocks <NUM> - <NUM> to progressively increase the resolution of the representation of intermediate output image <NUM>. Intermediate output image <NUM> may be generated by a final block (not shown) of synthesis network <NUM> configured to convert an output of style block <NUM> into image data.

The mathematical operations applied by each of style blocks <NUM> - <NUM> may depend on the StyleGAN version being utilized in a particular implementation, with some architectural commonalities being present regardless of the StyleGAN version. For example, each respective style block may include and/or be associated with a learned affine transformation A, a learned noise broadcast operation B, and a learned convolution (e.g., a 3x3 convolution). Additionally, every other style block may include an upsample operation. In some cases, style blocks <NUM> - <NUM> may implement an adaptive instance normalization operation (as in StyleGAN1), a weighted demodulation operation (as in StyleGAN2), and/or an alias-free architecture (as in StyleGAN3), among other possible StyleGAN variations.

Mapping network <NUM> may be configured to generate intermediate latent representation <NUM> based on noise input <NUM>. Noise input <NUM> may be expressed as z ∈ Z, and may be alternatively referred to as a latent code, latent point, and/or a latent vector, among other possibilities. Noise input <NUM> may be sampled from, based on, and/or using a Gaussian distribution corresponding to latent space Z.

Intermediate latent representation <NUM> may be expressed as w ∈ W, and may include a vector having a plurality of values. Concatenator <NUM> may be configured to concatenate combined latent representation <NUM> with intermediate latent representation <NUM>, thereby generating combined intermediate latent representation <NUM>. Combined intermediate latent representation <NUM> may be provided as input to each of style blocks <NUM> - <NUM>. For example, combined intermediate latent representation <NUM> may be provided as input to a corresponding affine transformation A of each respective style block of style blocks <NUM> - <NUM>. Additionally, noise input <NUM> may be provided as input to each of style blocks <NUM> - <NUM>, and may control the variability of image features at different size scales.

Initialization mapping <NUM> may be configured to generate initialization values <NUM> based on combined latent representation <NUM>. Initialization values <NUM> may be provided as input to style block <NUM> (i.e., the initial style block of style blocks <NUM> - <NUM>) to initialize synthesis network <NUM>. Specifically, initialization values <NUM> may be used in place of the 4x4x512 constant conventionally used to initialize the initial style block of synthesis network <NUM>. Accordingly, initialization mapping <NUM> may transform combined latent representation <NUM> to, for example, a 4x4x512 tensor containing initialization values <NUM>.

<FIG> illustrates aspects of a second possible architecture of StyleGAN model <NUM>, which are according to the invention. Specifically, StyleGAN model 330B may include similarity calculator <NUM>, concatenators <NUM> and <NUM>, modulators <NUM> and <NUM>, multipliers <NUM>, adder <NUM>, convolution <NUM>. StyleGAN model 330B may also include mapping network <NUM> configured to generate intermediate latent representation <NUM>, initialization mapping <NUM> configured to generate initialization values <NUM>, and synthesis network <NUM>, aspects of which are omitted from <FIG> for clarity of illustration. Modulators <NUM> and <NUM> may each represent, for example, a respective neural network model and/or aspects thereof. One or more parameters of modulator <NUM>, modulator <NUM>, similarity calculator <NUM>, and/or convolution <NUM> may be leamable during training of StyleGAN model 330B and/or inpainting system <NUM>.

Similarity calculator <NUM> may be configured to generate similarity metric <NUM> between guide image <NUM> and input image <NUM>. Similarity metric <NUM> may include one or more similarity values configured to quantify a similarity between various regions and/or visual features of guide image <NUM> and input image <NUM>. Similarity calculator <NUM> may be configured to generate similarity metric <NUM> based on guide image <NUM> and input image <NUM>, latent representations <NUM> and <NUM>, and/or inpainting mask <NUM>.

In one example, similarity calculator <NUM> may be configured to determine a cosine distance between latent representation <NUM> and latent representation <NUM>. In another example, similarity calculator <NUM> may be a learnable model configured to generate similarity metric <NUM>. In a further example, similarity calculator <NUM> may use the grayscale version of inpainting mask <NUM> to weight the similarity between input image <NUM> and guide image <NUM>. That is, a similarity between a portion of guide image <NUM> and pixels of input image <NUM> that are near the region to be inpainted may contribute to similarity metric <NUM> more than a similarity between the portion of guide image <NUM> and pixels of input image <NUM> that are further away from the region to be inpainted, since the inpainted image content should blend well with image content of input image <NUM> at the boundaries of the region to be inpainted, but may differ from image content of input image <NUM> that is not near the boundaries of the region to be inpainted.

By quantifying the similarity between different aspects of guide image <NUM> and input image <NUM>, similarity metric <NUM> may allow StyleGAN model 330B to incorporate into intermediate output image <NUM> aspects of guide image <NUM> that are similar to aspects of input image <NUM>. That is, similarity calculator <NUM> may provide a model component configured to explicitly quantify the similarity between input image <NUM> and guide image <NUM>, thereby allowing other parts of StyleGAN model 330B to specialize in performing other aspects of the inpainting process based on similarity metric <NUM>. Accordingly, StyleGAN model 330B may be configured to generate inpainted image content that, when inserted into the region to be inpainted in input image <NUM>, appears contextually and/or visually coherent and/or consistent with the remaining parts of input image <NUM>. For example, inpainted image content at the transitions between the region to be inpainted and the remaining parts of input image <NUM> may appear more natural, realistic, non-artificial, and/or visually-plausible.

Concatenator <NUM> may be configured to generate a concatenation of latent representation <NUM> and intermediate latent representation <NUM>, which may be provided as input to modulator <NUM>. Modulator <NUM> may be configured to generate input similarity feature <NUM> based on a comodulation and/or processing of (i) the concatenation of latent representation <NUM> and intermediate latent representation <NUM> and (ii) an output of style block <NUM>. Modulator <NUM> may be configured to comodulate and/or process input similarity feature <NUM> and latent representation <NUM>. Multiplier <NUM> may be configured to multiply an output of modulator <NUM> by similarity metric <NUM>. Adder <NUM> may be configured to add an output of multiplier <NUM> to input similarity feature <NUM>, thereby generating fused similarity feature <NUM>.

Concatenator <NUM> may be configured to concatenate fused similarity feature <NUM> with input similarity feature <NUM>. Convolution <NUM> may be configured to generate output similarity feature <NUM> based on the concatenation of fused similarity feature <NUM> and input similarity feature <NUM>. Output similarity feature <NUM> may be provided as input to style block <NUM>. For example, output similarity feature <NUM> may be provided as input to an upsampling operator of style block <NUM> and/or a convolution operator of style block <NUM>.

A commensurate set of operations may be performed based on the respective outputs of other style blocks of synthesis network <NUM>. For example, the output of each respective style block of style blocks <NUM> - <NUM> may be processed in the manner illustrated by <FIG> to generate a corresponding instance of input similarity feature <NUM>, fused similarity feature <NUM>, and output similarity feature <NUM>, with the output similarity feature of the respective style block being provided as input to a style block immediately following the respective style block. Thus, in some cases, a respective instance of each of modulators <NUM> and <NUM>, multiplier <NUM>, adder <NUM>, concatenators <NUM> and <NUM>, and convolution <NUM> may be a subset of one or more corresponding style block of style blocks <NUM> - <NUM>.

The computation carried out by StyleGAN model 330B may be expressed as <MAT>, and <MAT>, where N is iterated from <NUM> (corresponding to style block <NUM>) to M - <NUM> (corresponding to style block <NUM>), M represents the number of style blocks <NUM> - <NUM> (e.g., <NUM>), ∥ represents a concatenation operation, FINPUT represents latent representation <NUM>, FGUIDE represents latent representation <NUM>, R represents similarity metric <NUM>, <MAT> represents an output of the Nth style block, w represents intermediate latent representation <NUM>, <MAT> represents a learned function implemented by modulator <NUM> for Nth style block, <MAT> represents a learned function implemented by modulator <NUM> for Nth style block, <MAT> represents input similarity feature <NUM> computed for the Nth style block, <MAT> represents fused similarity feature <NUM> computed for the Nth style block, <MAT> represents output similarity feature <NUM> computed for the Nth style block and provided as input to the (N + <NUM>)th style block.

In some implementations, similarity metric <NUM> may additionally or alternatively be specified by a user. This may allow the user to manually control a ratio of features of input image <NUM> and guide image <NUM>. For example, by increasing similarity metric <NUM>, a contribution of latent representation <NUM> may be increased relative to a contribution of latent representation <NUM>, thus allowing the inpainted image content in output image <NUM> to look more like guide image <NUM>. Specifically, increasing similarity metric <NUM> increases the value of fused similarity feature <NUM>, which is based on both latent representation <NUM> and latent representation <NUM>, relative to a value of input similarity feature, which is based on latent representation <NUM> but not on latent representation <NUM>. Conversely, by decreasing similarity metric <NUM>, a contribution of latent representation <NUM> may be increased relative to a contribution of latent representation <NUM>, thus allowing the inpainted image content in output image <NUM> to look more like input image <NUM>.

<FIG> illustrates aspects of a third possible architecture of StyleGAN model <NUM>. Specifically, StyleGAN model 330C may include (second) cross-attention calculator <NUM> and concatenator <NUM>. StyleGAN model 330C may also include mapping network <NUM> configured to generate intermediate latent representation <NUM>, initialization mapping <NUM> configured to generate initialization values <NUM>, and synthesis network <NUM>, aspects of which are omitted from <FIG> for clarity of illustration. One or more parameters of cross-attention calculator <NUM> may be learnable during training of StyleGAN model 330B and/or inpainting system <NUM>.

Cross-attention calculator <NUM> may be configured to determine cross-attention value(s) based on (i) intermediate encoder state(s) <NUM> of encoder model <NUM> and (ii) one or more intermediate outputs of one or more of style blocks <NUM> - <NUM>. For example, cross-attention calculator <NUM> may be configured to determine cross-attention value(s) <NUM> based on an intermediate output of style block <NUM> and a corresponding intermediate encoder state of intermediate encoder state(s) <NUM>. Specifically, cross-attention calculator <NUM> may be configured to compare a given intermediate style block output to a corresponding intermediate encoder state that has a same and/or matching resolution. For example, an intermediate encoder state having a 16x16 resolution may be compared to an intermediate style block output that also has a 16x16 resolution.

Cross-attention calculator <NUM> may thus search guide image <NUM> for patches and/or features that are similar to input image <NUM>, thereby allowing similar, rather than dissimilar patches and/or features of guide image <NUM> to be used for generating the inpainted image content. Specifically, cross-attention calculator <NUM> may perform this search in latent space, rather than pixel space, and may thus be able to consider various semantics of guide image <NUM> and input image <NUM>.

Concatenator <NUM> may be configured to concatenate intermediate latent representation <NUM> with cross-attention value(s) <NUM>, and thereby generate combined intermediate latent representation <NUM>. Combined intermediate latent representation <NUM> may be provided as input to style block <NUM>. For example, combined intermediate latent representation <NUM> may be provided as input to the affine transformation A of style block <NUM>.

A commensurate set of operations may be performed based on the respective outputs of other style blocks of synthesis network <NUM>. For example, cross-attention calculator <NUM> may be configured to determine corresponding cross-attention value(s) for each of a plurality of intermediate style block outputs of style blocks <NUM> - <NUM> and a plurality of corresponding intermediate encoder states of encoder model <NUM>. Specifically, the output of each respective style block of style blocks <NUM> - <NUM> may be processed as illustrated in <FIG> to generate a corresponding instance of cross-attention value(s) <NUM> and combined intermediate latent representation <NUM>, with combined intermediate latent representation <NUM> being provided as input to a style block immediately following the respective style block. Thus, in some cases, a respective instance of each of cross-attention calculator <NUM> and concatenator <NUM> may be a subset of a corresponding style block of style blocks <NUM> - <NUM>.

<FIG> illustrates an example training system <NUM> that may be used to train one or more components of inpainting system <NUM>. Specifically, training system <NUM> may include inpainting system <NUM>, perceptual loss model <NUM>, perceptual loss function <NUM>, discriminator model <NUM>, adversarial loss function <NUM>, and model parameter adjuster <NUM>. Training system <NUM> may be configured to determine updated model parameters <NUM> based on training input image <NUM>, training guide image <NUM>, and training inpainting mask <NUM>. Training input image <NUM>, training guide image <NUM>, and training inpainting mask <NUM> may be analogous to input image <NUM>, guide image <NUM>, and inpainting mask <NUM>, respectively, but may be processed at training time rather than at inference time.

Inpainting system <NUM> may be configured to generate, based on training input image <NUM>, training guide image <NUM>, and training inpainting mask <NUM>, training output image <NUM>, which may be analogous to output image <NUM>. Training output image <NUM> may include inpainted training image content <NUM>, which may represent the image content synthesized by inpainting system <NUM> for the region to be inpainted in training input image <NUM>.

Specifically, encoder model <NUM> of inpainting system <NUM> may be configured to generate a first training latent representation (analogous to latent representation <NUM>) of training input image <NUM> and encoder model <NUM> may be configured to generate a second training latent representation (analogous to latent representation <NUM>) of training guide image <NUM>. Additionally, latent representation operator <NUM> may be configured to generate a combined training latent representation (analogous to combined latent representation <NUM>) based on the first training latent representation and the second training latent representation. StyleGAN model <NUM> may be configured to generate inpainted training image content <NUM>, which may form part of an intermediate training output image (analogous to intermediate output image <NUM>), based on the combined training latent representation. Output image generator <NUM> may be configured to generate training output image <NUM> based on the intermediate training output image, training input image <NUM>, and training inpainting mask <NUM>.

A quality with which inpainting system <NUM> inpaints the region to be inpainted in training input image <NUM> based on training guide image <NUM> may be quantified using perceptual loss model <NUM>, perceptual loss function <NUM>, discriminator model <NUM>, and/or adversarial loss function <NUM>, among other possibilities. Since inpainted training image content <NUM> is expected to include a combination of visual features of both training guide image <NUM> and training input image <NUM>, directly comparing either of these images to inpainted training image content <NUM> might not accurately quantify the inpainting quality. Accordingly, loss functions <NUM> and <NUM> may instead quantify the inpainting quality using feature and/or latent space and discriminator space.

Specifically, perceptual loss model <NUM> may be configured to generate perceptual feature representation <NUM> based on training guide image <NUM> and perceptual feature representation <NUM> based on inpainted training image content <NUM>. Perceptual feature representations <NUM> and <NUM> may represent, for example, vector embeddings of the corresponding image data that are indicative of various visual features of the corresponding image data. Perceptual loss function <NUM> may be configured to generate perceptual loss value <NUM> based on a comparison of perceptual feature representations <NUM> and <NUM>. For example, perceptual loss function <NUM> may be configured to determine an L-<NUM> and/or L-<NUM> distance between perceptual feature representations <NUM> and <NUM>, thereby quantifying how visually similar inpainted training image content <NUM> is to training guide image <NUM>.

In some implementations, perceptual loss model <NUM> may additionally or alternatively be configured to generate an additional perceptual feature representation of at least part of training input image <NUM>, and perceptual loss function <NUM> may be configured to generate perceptual loss value <NUM> based additionally or alternatively on comparing this additional perceptual feature representation to perceptual feature representation <NUM>. Further, in some implementations, perceptual loss function <NUM> may be configured to determine a training similarity metric (which may be analogous to similarity metric <NUM>) indicative of a similarity between training input image <NUM> and training guide image <NUM>. Perceptual loss value <NUM> may be determined by weighting the comparison of perceptual feature representation <NUM> and perceptual feature representation <NUM> according to the training similarity metric. Specifically, perceptual loss function <NUM> may be configured to reward (i) the presence in inpainted training image content <NUM> of features that are similar between images <NUM> and <NUM> more than (ii) the presence in inpainted training image content <NUM> of features that are dissimilar between images <NUM> and <NUM>, thereby conditioning inpainting system <NUM> to extract from a given guide image visual features that are similar to visual features of an input image to be inpainted.

Discriminator model <NUM> may be configured to generate discriminator output <NUM> based on training output image <NUM>. Specifically, discriminator output <NUM> may indicate whether discriminator model <NUM> estimates that training output image <NUM> is generated by inpainting system <NUM> or is a natural image that has not been generated by inpainting system <NUM>. Thus, inpainting system <NUM> and discriminator model <NUM> may implement an adversarial training architecture. Accordingly, adversarial loss function <NUM> may include, for example, a hinge adversarial loss, and may be configured to generate adversarial loss value <NUM> based on discriminator output <NUM>. Adversarial loss function <NUM> may thus incentivize inpainting system <NUM> to generate inpainted image content that appears natural, realistic, and/or non-artificial.

Model parameter adjuster <NUM> may be configured to determine updated model parameters <NUM> based on perceptual loss value <NUM> and adversarial loss value <NUM>, and possibly other loss values that may be determined by other loss functions of training system <NUM>. Model parameter adjuster <NUM> may be configured to determine a total loss value based on a weighted sum of these loss values, where the relative weight of the corresponding loss values may be an adjustable training parameter. Updated model parameters <NUM> may include one or more updated parameters of any trainable component of inpainting system <NUM>, including, for example, encoder models <NUM> and <NUM>, mapping network <NUM>, initialization mapping <NUM>, synthesis network <NUM>, similarity calculator <NUM>, modulators <NUM> and <NUM>, convolution <NUM>, and/or cross-attention calculator <NUM>, among other possibilities.

Model parameter adjuster <NUM> may be configured to determine updated model parameters <NUM> by, for example, determining a gradient of the total loss function. Based on this gradient and the total loss value, model parameter adjuster <NUM> may be configured to select updated model parameters <NUM> that are expected to reduce the total loss value, and thus improve performance of inpainting system <NUM>. After applying updated model parameters <NUM> to inpainting system <NUM>, the operations discussed above may be repeated to compute another instance of the total loss value and, based thereon, another instance of updated model parameters <NUM> may be determined and applied to inpainting system <NUM> to further improve the performance thereof. Such training of inpainting system <NUM> may be repeated until, for example, the total loss value is reduced to below a target threshold loss value.

<FIG> includes example images that illustrate the performance of inpainting system <NUM>. Specifically, <FIG> includes input image <NUM> as a representative example of input image <NUM>, guide image <NUM> as a representative example of guide image <NUM>, inpainting mask <NUM> as a representative example of inpainting mask <NUM>, and output image <NUM> as a representative example of output image <NUM>.

Input image <NUM> represents a first building having a first architectural style, which may be described as Industrial. Input image <NUM> may include a region to be inpainted, as indicated by inpainting mask <NUM>. Specifically, inpainting mask <NUM> indicates the region to be inpainted in input image <NUM> using a white color, and regions of input image <NUM> that are not intended to be inpainted using a black color. Although input image <NUM> includes the original image content of the region to be inpainted, it is to be understood that, in some cases, the region to be inpainted might not include the original image content (e.g., due to intentional removal thereof and/or accidental loss thereof). Guide image <NUM> represents a second building having a second architectural style, which may be described as Gothic.

Output image <NUM> includes, in the region to be inpainted, inpainted image content having visual features of both input image <NUM> and guide image <NUM>. Specifically, output image <NUM> includes, in the region to be inpainted, a representation of a third building that resembles the architectural style of the second building depicted in guide image <NUM>. Output image <NUM> also includes, in the region to be inpainted, representation of parts of trees, clouds, and the sky, visual aspects of which may have been derived from input image <NUM> and/or guide image <NUM>. Thus, the third building and the surrounding scenery are inpainted by inpainting model <NUM> in a manner that is visually coherent with the original (i.e., non-inpainted) parts of input image <NUM>. Specifically, the inpainted portion of output image <NUM> includes visual features of both input image <NUM> and guide image <NUM> arranged in a natural, realistic, visually-plausible, and/or non-artificial manner.

<FIG> illustrates a flow chart of operations related to inpainting a region of an input image based on a guide image. The operations may be carried out by computing device <NUM>, computing system <NUM>, and/or inpainting system <NUM>, among other possibilities. The embodiments of <FIG> may be simplified by the removal of any one or more of the features shown therein. Further, these embodiments may be combined with features, aspects, and/or implementations of any of the previous figures or otherwise described herein.

Block <NUM> may involve obtaining (i) an input image comprising a region to be inpainted, (ii) an indication of the region to be inpainted in the input image, and (iii) a guide image.

Block <NUM> may involve determining, by an encoder model, (i) a first latent representation of the input image and (ii) a second latent representation of the guide image.

Block <NUM> may involve generating a combined latent representation based on the first latent representation and the second latent representation.

Block <NUM> may involve generating, by a style generative adversarial network (StyleGAN) model and based on the combined latent representation, an intermediate output image comprising inpainted image content for the region to be inpainted in the input image.

Block <NUM> may involve generating, based on the input image, the indication of the region to be inpainted, and the intermediate output image, an output image representing the input image with the region to be inpainted comprising the inpainted image content from the intermediate output image.

In some embodiments, the StyleGAN model may be configured to generate the inpainted image content to include a combination of visual features of the guide image and visual features of the input image.

In some embodiments, the first latent representation may include a first feature vector, and the second latent representation may include a second feature vector.

In some embodiments, the first latent representation may include a first feature map, and the second latent representation may include a second feature map.

In some embodiments, generating the combined latent representation may include concatenating the first latent representation and the second latent representation.

In some embodiments, generating the combined latent representation may include determining a first cross-attention between the first latent representation and the second latent representation.

In some embodiments, the StyleGAN model may include a mapping network and a synthesis network. The synthesis network may include a plurality of style blocks. Generating the intermediate output image may include generating, by the mapping network and based on a noise input, an intermediate latent representation, and generating a combined intermediate latent representation based on the intermediate latent representation and the combined latent representation. Generating the intermediate output image may also include generating, by the synthesis network, the intermediate output image by (i) initializing an initial style block of the plurality of style blocks based on the combined latent representation and (ii) providing the combined intermediate latent representation as input to each respective style block of the plurality of style blocks.

In some embodiments, the StyleGAN model may include a mapping network and a synthesis network. The synthesis network may include a plurality of style blocks. Generating the intermediate output image may include generating, by the mapping network, an intermediate latent representation, and determining a similarity metric indicative of a similarity between the input image and the guide image. Generating the intermediate output image may also include generating, for a respective style block of the plurality of style blocks, an input similarity feature based on a comodulation of (i) an output of a preceding style block of the plurality of style blocks, (ii) the intermediate latent representation, and (iii) the first latent representation. Generating the intermediate output image may additionally include generating, for the respective style block, a fused similarity feature based on a sum of (i) the input similarity feature and (ii) a product of the similarity metric and a comodulation of the input similarity feature and the second latent representation. Generating the intermediate output image may further include generating, for the respective style block, an output similarity feature based on a convolution of (i) the input similarity feature and (ii) the fused similarity feature, and providing the output similarity feature as input to the respective style block.

In some embodiments, determining the similarity metric may include one or more of: determining a distance metric between the first latent representation and the second latent representation, generating the similarity metric by a similarity model based on the input image, the guide image, and the indication of the region to be inpainted, or comparing the input image to the guide image by weighting pixels of the input image according to a Gaussian distribution defined based on the indication of the region to be inpainted.

In some embodiments, the StyleGAN model may include a mapping network and a synthesis network. The synthesis network may include a plurality of style blocks. Generating the intermediate output image may include generating, by the mapping network, an intermediate latent representation, and determining a second cross-attention between (i) an intermediate encoder state of the encoder model based on the guide image and (ii) an intermediate style block state of a corresponding style block of the plurality of style blocks. A resolution of the intermediate encoder state may match a resolution of the intermediate style block state. Generating the intermediate output image may also include generating a concatenation of the second cross-attention and the intermediate latent representation, and providing the concatenation as input to a subsequent style block of the plurality of style blocks (e.g., a style block immediately following the corresponding style block).

In some embodiments, the intermediate style block state of the corresponding style block may include an output of the corresponding style block.

In some embodiments, generating the output image may include determining a sum of (i) a first product of the input image and an inverse of the indication of the region to be inpainted and (ii) a second product of the intermediate output image and the indication of the region to be inpainted.

In some embodiments, the StyleGAN model may be and/or may have been trained by a training process that includes obtaining (i) a training input image that includes a training region to be inpainted, (ii) a training indication of the region to be inpainted in the training input image, and (iii) a training guide image. The training process may also include determining, by the encoder model, (i) a first training latent representation of the training input image and (ii) a second training latent representation of the training guide image, and generating a combined training latent representation based on the first training latent representation and the second training latent representation. The training process may additionally include generating, by the StyleGAN model and based on the combined training latent representation, an intermediate training output image that includes inpainted training image content for the training region to be inpainted in the training input image. The training process may further include determining, by a perceptual loss model, (i) a first perceptual feature representation of the training guide image and (ii) a second perceptual feature representation of the inpainted training image content. The training process may yet further include determining a perceptual loss value based on a comparison of the first perceptual feature representation and the second perceptual feature representation, and adjusting one or more parameters of the StyleGAN model based on the perceptual loss value.

In some embodiments, the training process may also include generating, based on the training input image, the training indication of the region to be inpainted, and the intermediate training output image, a training output image representing the training input image with the region to be inpainted including the inpainted training image content from the intermediate training output image. The training process may further include determining an adversarial loss value based on processing of the training output image by a discriminator model, and adjusting the one or more parameters of the StyleGAN model further based on the adversarial loss value.

In some embodiments, the training process may further include determining a training similarity metric indicative of a similarity between the training input image and the training guide image, and determining the perceptual loss value by weighting the comparison of the first perceptual feature representation and the second perceptual feature representation according to the training similarity metric.

In some embodiments, a second guide image that is different from the guide image may be obtained, and a third latent representation of the second guide image may be determined by the encoder model. A second combined latent representation may be generated based on the first latent representation and the third latent representation. A second intermediate output image comprising second inpainted image content for the region to be inpainted in the input image may be generated by the StyleGAN model based on the second combined latent representation. The second inpainted image content may be different from the inpainted image content of the intermediate output image. A second output image representing the input image with the region to be inpainted including the second inpainted image content from the second intermediate output image may be generated based on the input image, the indication of the region to be inpainted, and the second intermediate output image.

In some embodiments, the StyleGAN model may include one or more of (i) an adaptive instance normalization component (e.g., as implemented by StyleGAN1), (ii) a weight demodulation component (e.g., as implemented by StyleGAN2), or (iii) an alias-free architecture (e.g., as implemented by StyleGAN3).

In some embodiments, the indication of the region to be inpainted may include a mask.

The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise.

With respect to any or all of the message flow diagrams, scenarios, and flow charts in the figures and as discussed herein, each step, block, and/or communication can represent a processing of information and/or a transmission of information in accordance with example embodiments.

A step or block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical operations or actions in the method or technique. The program code and/or related data may be stored on any type of computer readable medium such as a storage device including random access memory (RAM), a disk drive, a solid state drive, or another storage medium.

The computer readable medium may also include non-transitory computer readable media such as computer readable media that store data for short periods of time like register memory, processor cache, and RAM. The computer readable media may also include non-transitory computer readable media that store program code and/or data for longer periods of time. Thus, the computer readable media may include secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, solid state drives, compact-disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.

Claim 1:
A computer-implemented method comprising:
obtaining (i) an input image comprising a region to be inpainted, (ii) an indication of the region to be inpainted in the input image, and (iii) a guide image;
determining, by an encoder model, (i) a first latent representation of the input image and (ii) a second latent representation of the guide image;
generating a combined latent representation based on the first latent representation and the second latent representation;
generating, by a style generative adversarial network (StyleGAN) model and based on the combined latent representation, an intermediate output image comprising inpainted image content for the region to be inpainted in the input image; and
generating, based on the input image, the indication of the region to be inpainted, and the intermediate output image, an output image representing the input image with the region to be inpainted comprising the inpainted image content from the intermediate output image, wherein the StyleGAN model comprises a mapping network and a synthesis network, wherein the synthesis network comprises a plurality of style blocks,
characterized in that the generating the intermediate output image comprises:
generating, by the mapping network, an intermediate latent representation;
determining a similarity metric indicative of a similarity between the input image and the guide image;
generating, for a respective style block of the plurality of style blocks, an input similarity feature based on a comodulation of (i) an output of a preceding style block of the plurality of style blocks, (ii) the intermediate latent representation, and (iii) the first latent representation;
generating, for the respective style block, a fused similarity feature based on a sum of (i) the input similarity feature and (ii) a product of the similarity metric and a comodulation of the input similarity feature and the second latent representation;
generating, for the respective style block, an output similarity feature based on a convolution of (i) the input similarity feature and (ii) the fused similarity feature; and
providing the output similarity feature as input to the respective style block.