Patent ID: 12204610

DETAILED DESCRIPTION

This disclosure describes one or more embodiments of a structure-aware inpainting system that learns parameters for a generative inpainting neural network utilizing a novel training technique not found in prior systems. In practical scenarios, inpainting digital images often requires training generative neural networks to identify pixels for replacing missing, flawed, or otherwise undesirable regions (or “holes”) within digital images. To date, many existing digital image systems train generative neural networks with datasets that poorly represent real-world use cases and that, consequently, often generate inaccurate inpainted digital images when trained. Motivated by this issue, the structure-aware inpainting system utilizes a training technique that includes generating synthetic digital image masks for sample digital images in a dataset to resemble hole regions and/or that includes a masked regularization for improved accuracy. Specifically, the structure-aware inpainting system trains a generative inpainting neural network using object-aware training and/or masked regularization.

As just mentioned, in one or more embodiments, the structure-aware inpainting system learns parameters for a generative inpainting neural network utilizing object-aware training. In particular, the structure-aware inpainting system utilizes a mask generation scheme tailored for real-world use cases (e.g., object removal and completion). For example, the structure-aware inpainting system leverages instance-level segmentation to generate sample digital images with object-aware masks that simulate real distractor or clutter removal use cases. In some cases, the structure-aware inpainting system filters out sample digital images where an entire object (or a large part of it) is covered by a mask to prevent the generator from learning to produce distorted objects or color blobs. Furthermore, the structure-aware inpainting system provides precise object boundaries for depicted objects, and thus, prevents a trained generative inpainting neural network from leaking pixel colors (e.g., where non-object pixel colors bleed with object pixel colors or vice-versa) at object boundaries.

As also mentioned, in certain embodiments, the structure-aware inpainting system learns parameters for a generative inpainting neural network utilizing masked regularization. To elaborate, the structure-aware inpainting system utilizes a modified regularization technique such as R1 regularization that is tailored specifically for inpainting digital images. For instance, the structure-aware inpainting system modifies an R1 regularization term to avoid computing penalties on a partial image and to thus impose a better separation of input conditions from generated outputs. In some cases, the structure-aware inpainting system modifies R1 regularization utilizing a digital image mask to form a masked R1 regularization term. By utilizing masked regularization, in one or more embodiments, the structure-aware inpainting system reduces or eliminates harmful impacts of computing regularization on a background of a digital image.

In one or more embodiments, the structure-aware inpainting system utilizes a trained generative inpainting neural network to generate an inpainted digital image. More specifically, the structure-aware inpainting system trains a generative inpainting neural network using one or more of the aforementioned techniques (e.g., object-aware training and/or masked regularization) and further applies the trained generative inpainting neural network to generate an inpainted digital image. For example, the structure-aware inpainting system generates an inpainted digital image by utilizing the generative inpainting neural network to fill or replace a hole region with replacement pixels identified from the digital image (as dictated by network parameters learned via the training process).

As suggested above, many conventional digital image systems exhibit a number of shortcomings or disadvantages, particularly in accuracy and efficiency. For example, due to their limiting training processes, conventional systems often generate inaccurate inpainted digital images that include unwanted or jarring artifacts and/or that depict color bleeding. More particularly, because conventional systems usually only sample rectangular or irregularly shaped masks (or a combination of the two), the neural networks trained by these systems often struggle to generate accurate results when filling more complicated hole regions beyond simple shapes or blobs. Indeed, experimenters have demonstrated that, due to their training limitations, conventional systems often generate inpainted digital images with unexpected and visually jarring artifacts within hole regions (e.g., floating heads or other pixel bodies misplaced in various regions). Even certain existing systems that attempt to remediate these issues with saliency annotation continue to show issues because saliency annotation only captures large dominant foreground objects and leaves background objects (possibly covered by large hole regions). To this point, saliency detection does not work well for object completion (e.g., reconstructing an object from a partially masked one) because it generally predicts only the most obvious objects while ignoring surrounding objects, leading to ambiguity during training.

In addition to their inaccuracy, some conventional digital image systems inefficiently consume computing resources such as processing power and memory. Indeed, training generative inpainting models is computationally expensive, often requiring hours, days, or weeks to complete. Existing digital image systems that train using conventional datasets with irregularly shaped masks and/or standard regularization take an especially long amount of time (and therefore an especially large amount of processing power and memory) to converge, expending computing resources that could otherwise be preserved with more efficient training techniques.

As suggested above, embodiments of the structure-aware inpainting system provide a variety of improvements or advantages over conventional image modification systems. For example, embodiments of the structure-aware inpainting system utilize a novel training technique not found in prior systems. To elaborate, the structure-aware inpainting system utilizes a training technique that involves object-aware training and/or masked regularization, neither of which are implemented by prior systems. For example, the structure-aware inpainting system generates a dataset of masked digital image from which to sample that includes masked digital images depicting object instance masks (that are further used for determining overlap ratios as part of training). In addition, the structure-aware inpainting system utilizes masked regularization to specifically focus the computation of gradient penalties on unmasked pixels and to avoid computing regularization outside masked pixels (therefore resulting in more stable training as well).

Due at least in part to implementing a new training technique, in some embodiments, the structure-aware inpainting system improves accuracy over conventional digital image systems. While some existing systems' training processes lead to generating unwanted artifacts in strange locations within hole regions (particularly larger hole regions), one or more embodiments of the object-aware training and masked regularization of the structure-aware inpainting system greatly improve the accuracy of generating inpainted digital images. As discussed in further detail below, experimenters have demonstrated the accuracy improvements that result from the training process of one or more embodiments of the structure-aware inpainting system, generating final results that do not depict unwanted artifacts and that appear more visually coherent.

Additionally, embodiments of the structure-aware inpainting system also improve efficiency over conventional digital image systems. For example, compared to conventional systems, the structure-aware inpainting system trains a generative neural network using fewer computing resources such as processing power and memory. By utilizing the object-aware training and/or masked regularization described herein, the structure-aware inpainting system converges faster than prior systems, thus preserving computing resources compared to prior systems. Indeed, in some cases, the novel training technique of the structure-aware inpainting system is faster and more stable than that of conventional systems, requiring fewer training iterations or epochs to converge.

Additional detail regarding the structure-aware inpainting system will now be provided with reference to the figures. For example,FIG.1illustrates a schematic diagram of an example system environment for implementing a structure-aware inpainting system102in accordance with one or more embodiments. An overview of the structure-aware inpainting system102is described in relation toFIG.1. Thereafter, a more detailed description of the components and processes of the structure-aware inpainting system102is provided in relation to the subsequent figures.

As shown, the environment includes server(s)104, a client device108, a database112, and a network114. Each of the components of the environment communicate via the network114, and the network114is any suitable network over which computing devices communicate. Example networks are discussed in more detail below in relation toFIG.12.

As mentioned, the environment includes a client device108. The client device108is one of a variety of computing devices, including a smartphone, a tablet, a smart television, a desktop computer, a laptop computer, a virtual reality device, an augmented reality device, or another computing device as described in relation toFIG.12. AlthoughFIG.1illustrates a single instance of the client device108, in some embodiments, the environment includes multiple different client devices, each associated with a different user (e.g., a digital image editor). The client device108communicates with the server(s)104via the network114. For example, the client device108provides information to server(s)104indicating client device interactions (e.g., digital image selections, user interactions requesting generation or modification of digital images, or other input) and receives information from the server(s)104such as generated inpainted digital images. Thus, in some cases, the structure-aware inpainting system102on the server(s)104provides and receives information based on client device interaction via the client device108.

As shown inFIG.1, the client device108includes a client application110. In particular, the client application110is a web application, a native application installed on the client device108(e.g., a mobile application, a desktop application, etc.), or a cloud-based application where all or part of the functionality is performed by the server(s)104. Based on instructions from the client application110, the client device108presents or displays information to a user, including digital images such as inpainted digital images, masked digital images, and/or selectable options for generating and editing digital images (e.g., to indicate objects to remove and/or inpaint). In some cases, the client application110includes all or part of the structure-aware inpainting system102and/or the generative inpainting neural network116.

As illustrated inFIG.1, the environment includes the server(s)104. The server(s)104generates, tracks, stores, processes, receives, and transmits electronic data, such as indications of client device interactions and/or pixels of digital images. For example, the server(s)104receives data from the client device108in the form of an indication of a client device interaction to generate an inpainted digital image. In response, the server(s)104transmits data to the client device108to cause the client device108to display or present an inpainted digital image based on the client device interaction.

In some embodiments, the server(s)104communicates with the client device108to transmit and/or receive data via the network114, including client device interactions, inpainted digital images, and/or other data. In some embodiments, the server(s)104comprises a distributed server where the server(s)104includes a number of server devices distributed across the network114and located in different physical locations. The server(s)104comprise a content server, an application server, a communication server, a web-hosting server, a multidimensional server, or a machine learning server. The server(s)104further access and utilize the database112to store and retrieve information such as a generative inpainted neural network (e.g., the generative inpainting neural network116), stored sample digital images for training, and/or generated inpainted digital images.

As further shown inFIG.1, the server(s)104also includes the structure-aware inpainting system102as part of a digital content editing system106. For example, in one or more implementations, the digital content editing system106is able to store, generate, modify, edit, enhance, provide, distribute, and/or share digital content, such as digital images. For example, the digital content editing system106provides tools for the client device108, via the client application110, to generate and modify digital images.

In one or more embodiments, the server(s)104includes all, or a portion of, the structure-aware inpainting system102. For example, the structure-aware inpainting system102operates on the server(s) to train a generative inpainted neural network to generate inpainted digital images. In some cases, the structure-aware inpainting system102utilizes, locally on the server(s)104or from another network location (e.g., the database112), a generative inpainting neural network116including one or more constituent neural networks such as an encoder neural network, a generator neural network, and/or a discriminator neural network.

In certain cases, the client device108includes all or part of the structure-aware inpainting system102. For example, the client device108generates, obtains (e.g., download), or utilizes one or more aspects of the structure-aware inpainting system102, such as the generative inpainting neural network116, from the server(s)104. Indeed, in some implementations, as illustrated inFIG.1, the structure-aware inpainting system102is located in whole or in part on the client device108. For example, the structure-aware inpainting system102includes a web hosting application that allows the client device108to interact with the server(s)104. To illustrate, in one or more implementations, the client device108accesses a web page supported and/or hosted by the server(s)104.

In one or more embodiments, the client device108and the server(s)104work together to implement the structure-aware inpainting system102. For example, in some embodiments, the server(s)104train one or more neural networks discussed herein and provide the one or more neural networks to the client device108for implementation (e.g., to generate inpainted digital images at the client device108). In some embodiments, the server(s)104train one or more neural networks, the client device108requests an inpainted digital image, the server(s)104generate an inpainted digital image utilizing the one or more neural networks and provide the inpainted digital image to the client device108. Furthermore, in some implementations, the client device108assists in training one or more neural networks.

AlthoughFIG.1illustrates a particular arrangement of the environment, in some embodiments, the environment has a different arrangement of components and/or may have a different number or set of components altogether. For instance, as mentioned, the structure-aware inpainting system102is implemented by (e.g., located entirely or in part on) the client device108. In addition, in one or more embodiments, the client device108communicates directly with the structure-aware inpainting system102, bypassing the network114. Further, in some embodiments, the digital image collaging neural network116is stored in the database112, maintained by the server(s)104, the client device108, or a third-party device.

As mentioned, in one or more embodiments, the structure-aware inpainting system102trains a generative inpainting neural network using a novel training technique that includes object-aware training and masked regularization. In particular, the structure-aware inpainting system102learns parameters for a generative inpainting neural network to accurately inpaint or fill missing, flawed, or otherwise undesirable pixels in one or more regions.FIG.2illustrates an overview of training a generative inpainting neural network via object-aware training and masked regularization to generate an inpainted digital image in accordance with one or more embodiments. Additional detail regarding the various acts ofFIG.2is provided thereafter with reference to subsequent figures.

As illustrated inFIG.2, the structure-aware inpainting system102performs an act202to identify a hole region in a sample digital image. More specifically, the structure-aware inpainting system102identifies or determines a region of pixels within a sample digital image to replace with replacement pixels. For instance, the structure-aware inpainting system102identifies a hole region based on user interaction indicating or defining a region of pixels to replace. In some case, the structure-aware inpainting system102identifies a hole region by generating a digital image mask via segmentation. In some embodiments, the structure-aware inpainting system102generates a set of sample digital images with hole regions indicating pixels to replace to be used during the training process. In one or more embodiments, a hole region includes a region, a portion, an area, or a set of one or more pixels within a digital image that are to be replaced (or filled) with replacement pixels. For instance, a hole region is defined or indicated by a digital image mask determined via user interaction (e.g., selecting an object or portion of a digital image to remove) or otherwise generated by the structure-aware inpainting system102.

As further illustrated inFIG.2, the structure-aware inpainting system102performs an act204to learn parameters for a generative inpainting neural network. In particular, the structure-aware inpainting system102learns parameters via a training or tuning process. As shown, the structure-aware inpainting system102utilizes one or more unique training methods such as object-aware training206and/or masked regularization208to learn the parameters for a generative inpainting neural network. To train a generative inpainting neural network, in some cases, the structure-aware inpainting system102utilizes an iterative training process that repeats for a number of iterations or epochs until the generative inpainting neural network (including its inner networks such as an encoder neural network and a generator neural network) satisfies a threshold measure of accuracy.

In some embodiments, the term neural network refers to a machine learning model that is trained and/or tuned based on inputs to generate predictions, determine classifications, or approximate unknown functions. For example, a neural network includes a model of interconnected artificial neurons (e.g., organized in layers) that communicate and learn to approximate complex functions and generate outputs (e.g., generated digital images) based on a plurality of inputs provided to the neural network. In some cases, a neural network refers to an algorithm (or set of algorithms) that implements deep learning techniques to model high-level abstractions in data. For example, a neural network includes a convolutional neural network, a recurrent neural network (e.g., an LSTM), a graph neural network, a generative adversarial neural network, or other architecture.

Relatedly, a generative adversarial neural network (sometimes simply GAN) includes a neural network that is tuned or trained via an adversarial process to generate an output digital image (e.g., from an input digital image). In some cases, a generative adversarial neural network includes multiple constituent neural networks such as an encoder neural network and one or more generator neural networks. For example, an encoder neural network extracts latent code from a noise vector or from a digital image. A generator neural network (or a combination of generator neural networks) generates a modified digital image by combining extracted latent code (e.g., from the encoder neural network). During training, a discriminator neural network, in competition with the generator neural network, analyzes a generated digital image to generate an authenticity prediction by determining whether the generated digital image is real (e.g., from a set of stored digital images) or fake (e.g., not from the set of stored digital images). The discriminator neural network also causes the structure-aware inpainting system102to modify parameters of the encoder neural network and/or the one or more generator neural networks to eventually generate digital images that fool the discriminator neural network into indicating that a generated digital image is a real digital image.

Along these lines, a generative adversarial neural network refers to a neural network having a specific architecture or a specific purpose such as a generative inpainting neural network. For example, a generative inpainting neural network includes a generative adversarial neural network that inpaints or fills pixels of a digital image with replacement pixels. In some cases, a generative inpainting neural network inpaints a digital image by filling hole regions (indicated by digital image masks) which include pixels determine to be, or otherwise designated as, flawed, missing, or otherwise undesirable. Indeed, as mentioned above, in some embodiments a digital image mask defines a hole region using a segmentation or a mask indicating, overlaying, covering, or outlining pixels to be removed or replaced within a digital image.

For each training iteration, the structure-aware inpainting system102implements the object-aware training206by performing one or more steps pertaining to objects identified within a sample digital image. For example, the structure-aware inpainting system102generates a set of object masks indicating or outlining objects identified within a sample digital image. In one or more embodiments, the structure-aware inpainting system102generates object masks utilizing a segmentation model, such as a segmentation neural network, to determine or generate object segmentations indicating boundaries of individual object instances (e.g., differentiating between instances of common object types). In one or more implementations, the segmentation model comprises a panoptic segmentation neural network.

As part of the object-aware training206, the structure-aware inpainting system102further selects a masked digital image from a set of masked digital images. For example, the structure-aware inpainting system102generates a set of masked digital images that include or depict different types of masks. In some cases, a masked digital image includes masked object instances that the structure-aware inpainting system102generates from object segmentations indicating boundaries of individual object instances. Indeed, a masked object instance, in one or more embodiments, includes an object instance that has been specifically masked according to its segmentation (as determined via a segmentation neural network), where the mask excludes other pixels outside of, or other than, those indicated by a specific object instance segmentation. In these or other cases, a masked digital image includes a random pattern mask that the structure-aware inpainting system102generates using random strokes and/or rectangles (or other polygons or non-polygon shapes). For a given training iteration, the structure-aware inpainting system102thus (randomly or probabilistically) selects a masked digital image from the set of masked digital images.

Additionally, in one or more implementations, the structure-aware inpainting system102determines or computes an overlap ratio associated with the masked digital image. More particularly, as part of a training iteration, the structure-aware inpainting system102determines an overlap ratio indicating a measure or an amount (e.g., a proportion or a percentage) of overlap between a digital image mask (indicating a hole region to inpaint) and each masked object instance in the digital image (indicating a particular object instance within a sample digital image). In some cases, the structure-aware inpainting system102further compares the overlap ratio with an overlap ratio threshold that indicates whether to exclude the object instance from the hole region (e.g., to prevent sampling pixels of the object when inpainting to avoid or prevent generating random nonsensical artifacts from the object/hole when inpainting). Additional detail regarding the object-aware training206is provided below with reference to subsequent figures.

In addition, for each training iteration, the structure-aware inpainting system102implements masked regularization208to modify parameters of a generative inpainting neural network. To elaborate, the structure-aware inpainting system102utilizes a regularization technique to penalize a discriminator neural network during training to prevent or reduce overfitting. For instance, the structure-aware inpainting system102leverages a digital image mask (indicating a hole region) within a digital image as part of a regularization technique to avoid computing gradient penalties inside the mask, thereby reducing potential harmful impact of computing the regularization outside the hole region. In some cases, the structure-aware inpainting system102utilizes a particular type of regularization such as R1 regularization that also incorporates the digital image mask. Additional detail regarding the masked regularization208is provided below with reference to subsequent figures.

In one or more embodiments, the structure-aware inpainting system102repeats a training process for multiple iterations or epochs. For example, the structure-aware inpainting system102repeats, for each iteration, the process of: i) sampling a masked digital image from a set of masked digital images (including masked object instances and/or random pattern masks), ii) determining an overlap ratio between a digital image mask of the masked digital image and each object instance within the masked digital image, iii) comparing the overlap ratio with an overlap ratio threshold (and modifying any masks motivated by the comparison), iv) generating an inpainted digital image utilizing the generative inpainting neural network, v) comparing the inpainted digital image with a stored (e.g., real) digital image utilizing a discriminator neural network as dictated by masked regularization, vi) generating an authenticity prediction designating the inpainted digital image as real or fake based on the comparison, and vii) modifying or updating parameters of the generative inpainting neural network and/or discriminator based on the authenticity prediction. In some embodiments, the structure-aware inpainting system102repeats the training process until, based on its learned parameters, the generator neural network (of the generative inpainting neural network) fools the discriminator neural network into predicting that an inpainted digital image is real (at least threshold number of consecutive or non-consecutive times). In some cases, the structure-aware inpainting system102may omit or reorder one or more of the aforementioned steps of the training process for one or more iterations.

As further illustrated inFIG.2, the structure-aware inpainting system102performs an act210to generate an inpainted digital image. In particular, the structure-aware inpainting system102utilizes a trained generative inpainting neural network with parameters learned via the act204to fill or inpaint pixels in on or more regions of a masked digital image. As shown, the structure-aware inpainting system102receives or otherwise identifies a masked digital image with a rectangular mask designating pixels to inpaint from an image of a koala. In turn, the structure-aware inpainting system102further applies the trained generative inpainting neural network to identify replacement pixels from the digital image and generates content to “inpaint” the hole region indicated by the rectangular digital image mask.

As mentioned above, in certain described embodiments, the structure-aware inpainting system102utilizes object-aware training techniques as part of learning parameters for a generative inpainting neural network. In particular, the structure-aware inpainting system102generates object masks for individual object instances and utilizes masked object instances as part of the parameter learning process.FIGS.3A-3Eillustrate example steps involved in object-aware training in accordance with one or more embodiments. Specifically,FIG.3Aillustrates a sequence of acts for object-aware training in accordance with one or more embodiments. Additionally,FIG.3Billustrates an example of generating digital image masks and sampling masked digital images in accordance with one or more embodiments. Further,FIG.3Cillustrates an example of determining an overlap ratio and modifying digital image masks based on the overlap ratio in accordance with one or more embodiments. Thereafter,FIG.3Dillustrates additional object-aware training techniques for dilating and translating a masked object instance in accordance with one or more embodiments.FIG.3Eillustrates an additional object-aware training technique for dilating a digital image mask along a segmentation boundary in accordance with one or more embodiments.

As illustrated inFIG.3A, the structure-aware inpainting system102identifies or accesses a sample digital image302. For example, the structure-aware inpainting system102retrieves the sample digital image302from a dataset of training digital images stored within the database112. In some cases, the structure-aware inpainting system102utilizes a particular dataset such as the Places dataset or the Places2 dataset (or some other dataset). While the images used throughoutFIG.3Aare of different scenes and objects, this is merely illustrative and exemplary, and the images and masks may vary across different embodiments or implementations.

As further illustrated inFIG.3A, the structure-aware inpainting system102performs an act304to generate object masks. More specifically, the structure-aware inpainting system102generates object masks that mask individual instances of various objects depicted within the sample digital image302. For instance, the structure-aware inpainting system102utilizes a segmentation model to determine or generate object instance segmentations within the sample digital image302. As mentioned above, in one or more embodiments, the structure-aware inpainting system102generates the object instance segmentations utilizing a segmentation model, such as a panoptic segmentation neural network305. A panoptic segmentation neural network305segments all object instances in a digital image not only foreground or salient objects.

By utilizing a panoptic segmentation neural network305, in one or more implementations, the structure-aware inpainting system102ensures that foreground objects are not always occluded during training (thereby preventing the generative inpainting neural network from learning accurate object competition). In one or more implementations, the panoptic segmentation neural network305comprises a panoptic segmentation neural network as described in U.S. patent application Ser. No. 17/319,979, filed on May 13, 2021 and entitled “GENERATING IMPROVED PANOPTIC SEGMENTED DIGITAL IMAGES BASED ON PANOPTIC SEGMENTATION NEURAL NETWORKS THAT UTILIZE EXEMPLAR UNKNOWN OBJECT CLASSES,” the entire contents of which are hereby incorporated by reference. In still further implementations, the panoptic segmentation neural network305comprises a class-agnostic object segmentation neural network as described in U.S. patent application Ser. No. 17/151,111, filed on Jan. 15, 2021 and entitled “GENERATING CLASS-AGNOSTIC SEGMENTATION MASKS IN DIGITAL IMAGES,” the entire contents of which are hereby incorporated by reference. In still further implementations, the panoptic segmentation neural network305comprises the panoptic segmentation neural network (“PanopticFCN”) described by Yanwei Li et al. inFully Convolutional Networks for Panoptic Segmentation, Proceedings of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (2021), the entire contents of which are hereby incorporated by reference.

Having generated the object instance segmentations, the structure-aware inpainting system102converts one or more of the object instance segmentations into a mask to generate an object mask. A mask refers to an indication of a plurality of pixels portraying an object. For example, an object mask includes a segmentation boundary (e.g., a boundary line or curve indicating the borders of one or more objects) or a segmentation mask (e.g., a binary mask identifying pixels corresponding to an object vs those that do not).

In some cases, the structure-aware inpainting system102generates digital image masks other than (or in addition to) object masks. For some sample digital images for example, the structure-aware inpainting system102generates random pattern masks that depict masks in the shape of random strokes, rectangles (or other shapes), or a combination of random strokes and rectangles. By generating digital image masks including both object masks and random pattern masks, the structure-aware inpainting system generates a set of masked digital images to use as a basis for training a generative inpainting neural network.

As further shown, the structure-aware inpainting system102performs an act306to generate a masked digital image. In particular, the structure-aware inpainting system102randomly (or according to some probability or sampling technique) selects one or more masks by sampling from among the set of masks that includes object masks, random masks, and optionally combinations thereof. Thus, in some iterations, the structure-aware inpainting system102selects a masked object instance, in other iterations the structure-aware inpainting system102selects a random pattern mask, and in still further iterations, the structure-aware inpainting system102selections a combination thereof.

Additionally, the structure-aware inpainting system102performs an act308to determine an overlap ratio and modify masks based on the overlap ratio. More specifically, the structure-aware inpainting system102determines an overlap ratio between a hole region (or a digital image mask indicating a hole region) and each object instance identified within a selected masked digital image (or the sample digital image302). For example, the structure-aware inpainting system102determines an amount or a percentage of an object that is occluded or covered by a mask or a hole to be inpainted or filled. Indeed, the structure-aware inpainting system102determines an overlap ratio to identify one or more object instances that are substantially or significantly covered by a mask and that might impact pixel sampling for inpainting as a result (e.g., for completion of an object that is partially occluded and/or to prevent generating nonsensical artifacts when inpainting).

In some cases, the structure-aware inpainting system102further compares the overlap ratio with an overlap ratio threshold. For instance, the structure-aware inpainting system102compares the overlap ratio with the overlap ratio threshold to determine whether to exclude the object instance from the mask or hole. As an example, as shown inFIG.3A, the structure-aware inpainting system102has determined that an overlap ratio of the girl in the foreground and to object mask exceeds the overlap ratio threshold and therefore modifies the masked digital image by excluding the girl in the foreground from the object mask. By so doing, the structure-aware inpainting system102prevents sampling pixels of objects that are largely covered by a mask and which might therefore cause nonsensical artifacts to be inpainted in the hole. If the structure-aware inpainting system102determines that the overlap ratio is less than the overlap ratio threshold, on the other hand, the structure-aware inpainting system102leaves the mask unchanged to mimic or perform object completion (e.g., by inpainting pixels to inpaint the covered portion of an object).

As further illustrated inFIG.3A, the structure-aware inpainting system102performs an act310to generate an inpainted digital image from a masked digital image. For instance, the structure-aware inpainting system102generates an inpainted digital image utilizing a generative inpainting neural network116to fill or inpaint a hole region (indicated by a digital image mask) by generating replacement pixels. Indeed, the structure-aware inpainting system102utilizes a generative inpainting neural network116to, according to its internal parameters such as weights and biases, generates replacement pixels and to fill the hole region.

In one or more implementations, the generative inpainting neural network116comprises the ProFill model described by Y. Zeng et al. inHigh-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling, European Conf. on Computer Vision, 1-17 (2020)) or the DeepFillv2 model described by J. Yu et al., inFree-Form Image Inpainting with Gated Convolution, Proceedings of IEEE Int'l Conf. on Computer Vision, 4471-80 (2019)), the entire contents of which are hereby incorporated by reference. In still further implementations, the generative inpainting neural network116comprises one of the models referenced in relation toFIG.7below.

Additionally, the structure-aware inpainting system102performs an act312to determine an authenticity prediction. In particular, the structure-aware inpainting system102utilizes a discriminator neural network to determine whether the inpainted digital image generated via the act310is real (e.g., a captured digital image) or fake (e.g., a generated digital image). For instance, the structure-aware inpainting system102determines or utilizes an adversarial loss as the discriminator neural network competes with a generator neural network of the generative inpainting neural network. In some cases, the structure-aware inpainting system102utilizes a perceptual loss (in addition to the adversarial loss) to compare the inpainted digital image with a sample digital image such as the sample digital image corresponding to the inpainted digital image (e.g., the sample digital image302that depicts objects which were later masked via the acts304and306) stored in the database112.

As further illustrated inFIG.3A, the structure-aware inpainting system102performs an act314to modify parameters based on the authenticity prediction. For example, the structure-aware inpainting system102modifies parameters (e.g., weights and biases) of the generative inpainting neural network116to adjust how the network processes data and improve the inpainting for subsequent iterations. To modify the parameters, in some embodiments, the structure-aware inpainting system102backpropagates based on an adversarial loss and/or a perceptual loss. In some cases, the structure-aware inpainting system102modifies parameters of an encoder neural network, a generator neural network, and/or a discriminator neural network that are part of the generative inpainting neural network116.

In one or more embodiments, the structure-aware inpainting system102repeats one or more of the acts ofFIG.3Afor successive training iterations. For example, the structure-aware inpainting system102generates a plurality of masked digital image with object masks, random masks, masks modified based on an overlap ratio etc. For each iteration, the structure-aware inpainting system102samples a masked digital image, generates a new inpainted digital image, and modifies parameters based on comparing the new inpainted digital image with a corresponding sample (e.g., real) digital image. In one or more embodiments, the structure-aware inpainting system102repeats the training process for many iterations until the generative inpainting neural network generates an inpainted digital image that fools the discriminator neural network into predicting that the inpainted digital image is real.

As mentioned above, in certain described embodiments, the structure-aware inpainting system102generates masked digital images for use in training a generative inpainting neural network. In particular, the structure-aware inpainting system102generates a set of masked digital images from which to sample for inpainting during training.FIG.3Billustrates generating and sampling masked digital images in accordance with one or more embodiments. WhileFIG.3Billustrates generating different types of digital image masks for a single sample digital image, the structure-aware inpainting system102also generates digital image masks of different types for other sample digital images to generate a large corpus of training images.

As illustrated inFIG.3B, the structure-aware inpainting system102identifies, accesses, or receives a sample digital image316. For example, the structure-aware inpainting system102accesses the sample digital image from a database (e.g., the database112). In addition, the structure-aware inpainting system102generates object instance segmentations318from which the structure-aware inpainting system102generates object masks320. The structure-aware inpainting system102also generates random pattern masks322for the sample digital image316.

For example, the structure-aware inpainting system102generates object instance segmentations318utilizing a segmentation model to determine object instances within the sample digital image316as described above. For instance, the structure-aware inpainting system102analyzes pixels of the sample digital image316to determine probabilities of different objects appearing within the sample digital image316and further labels each instance of each object type based on their respective probabilities. As shown, the structure-aware inpainting system102identifies and outlines individual object instances within the sample digital image to generate the object instance segmentations318. The structure-aware inpainting system102further generates object masks320that align with one or more object instance segmentations318.

In addition, the structure-aware inpainting system102generates the random pattern masks322. More specifically, the structure-aware inpainting system102generates the random pattern masks322by utilizing one or more types of non-object masks. In some cases, the structure-aware inpainting system102utilizes random strokes, rectangles (or other shapes), a combination of random strokes and rectangles (or other shapes), or some other type of mask such as those proposed by Shengyu Zhao et al. inLarge Scale Image Completion via Co-Modulated Generative Adversarial Networks, ArXiv:2103:10428 (2021), the entire contents of which are hereby incorporated by reference. As shown, in one or more embodiments, the structure-aware inpainting system102utilizes rectangles to generate the random pattern masks322to mask out a portion of the sample digital image316.

In some embodiments, the structure-aware inpainting system102further generates a set of masked digital images for use in training a generative inpainting neural network116. For example, the structure-aware inpainting system102stores the object masks320, and optionally, the random pattern masks322within the database112. The structure-aware inpainting system102further performs an act325to sample masked digital images. For example, the structure-aware inpainting system102samples an initial mask, which can be a random pattern mask322or an object mask320. In particular, to sample a random pattern mask322, the structure-aware inpainting system102simulates random brush strokes and rectangles as mentioned above. To sample an object mask, the structure-aware inpainting system102randomly selections an object mask from the database112and randomly scales, translates, and/or dilates the selected object mask. The structure-aware inpainting system102also computes the overlap ratio between each object instance and the generated mask. If the overlap ratio is larger than an overlap threshold, the structure-aware inpainting system102excludes the object instance from the mask. One will appreciate, that because the structure-aware inpainting system102samples object masks from a database of object masks, a sampled object mask may not correspond to object instance in a training digital image to which it is applied (e.g., the sampled object mask will often comprise an object mask generated from another digital image). The structure-aware inpainting system102then applies the sampled mask to a training digital image and utilizes the training digital image and sampled mask combination for training a generative inpainting neural network116.

In one or alternative implementations, the structure-aware inpainting system102optionally generates combined masks324for use in training the generative inpainting neural network116. For example, the generative inpainting neural network116samples one or more random pattern masks322and one or more object masks320. In such implementations, masked digital image includes one or more object masks320together with one or more random pattern masks322.

As mentioned, in certain embodiments, the structure-aware inpainting system102determines an overlap ratio. In particular, the structure-aware inpainting system102compares a digital image mask with the object instance segmentations for a digital image to determine an overlap ratio.FIG.3Cillustrates modifying (or refraining from modifying) digital image masks based on an overlap ratio in accordance with one or more embodiments.

As illustrated inFIG.3C, the structure-aware inpainting system102performs an act326ato determine an overlap ratio for a masked digital image depicting part of a tractor (or combine) in a field. As shown, the structure-aware inpainting system102determines a relatively large overlap ratio by comparing the area occupied by the square mask to the area occupied by the tractor object instance as indicated by the dashed outline. Indeed, the structure-aware inpainting system102determines how much (or what percentage or proportion) of the object instance (the tractor) is covered by the square mask. To determine an overlap ratio, in some embodiments, the structure-aware inpainting system102determines, for each object instance siwithin a sample digital image x, an overlap ratio given by:

ri=Area(m,si)Area(si)
where rirepresents the overlap ratio, Area(m, si) represents an area occupied by the initial mask m (e.g., a digital image mask indicating a hole region to inpaint), and Area(si) represents an area occupied by an object instance si.

The structure-aware inpainting system102further compares the overlap ratio with an overlap ratio threshold. More particularly, the structure-aware inpainting system102compares the overlap ratio with a threshold that indicates whether to exclude the occluded object instance or include with occluded object instance from the digital image mask. Indeed, if the structure-aware inpainting system102determines that the overlap ratio meets or exceeds the overlap ratio threshold, the structure-aware inpainting system102excludes the object instance from the mask, as given by: m←m−sito mimic the distractor removal use case. More specifically, the structure-aware inpainting system102compares the overlap ratio with an overlap ratio threshold of 0.5 (or 50%) or another threshold such as 0.2, 0.3, 0.6, 0.7, etc. As shown, the structure-aware inpainting system102determines that the overlap ratio as determine via the act326ais greater than the overlap threshold. Consequently, the structure-aware inpainting system102performs an act327to exclude the object instance from the mask. As depicted, the structure-aware inpainting system102thus modifies the mask to carve out the portion occupied by the pixels of the object instance, masking only the remaining pixels not occupied by the formerly occluded object instance. The structure-aware inpainting system102thus refrains from sampling pixels of the occluded object when inpainting, thereby preventing generation of nonsensical artifacts and improving the quality of the result.

As further illustrated inFIG.3C, the structure-aware inpainting system102performs an act326bto determine an overlap ratio for a masked digital image of a tractor with a smaller square mask. As shown, the structure-aware inpainting system102determines a smaller overlap ratio via the act326bthan for the act326a. In addition, the structure-aware inpainting system102compares the overlap ratio with an overlap ratio threshold. As a result of the comparison, the structure-aware inpainting system102determines that the overlap ratio determined via the act326bis not greater than the overlap ratio threshold. The structure-aware inpainting system102therefore performs the act328to include the object instance. Specifically, the structure-aware inpainting system102leaves the digital image mask as-is, refraining from modifying the mask and including the portion of the object instance occluded by the mask as part of the mask. In some cases, the structure-aware inpainting system102performs more accurate object completion by utilizing the overlap ratio and the overlap ratio threshold to sample pixels of occluded objects when completing them (e.g., in cases where they are covered less than a threshold amount).

As mentioned above, in certain embodiments, the structure-aware inpainting system102further improves or modifies the object-aware training by translating and/or dilating masks of individual objects within digital images. In particular, the structure-aware inpainting system102dilates and/or translates an object mask (or a masked object instance) to prevent or reduce sampling pixels within a hole region (e.g., to avoid overfitting).FIG.3Dillustrates example dilations and translations of an object mask in accordance with one or more embodiments.

As illustrated inFIG.3D, the structure-aware inpainting system102dilates an object mask, as indicated in the digital image330. To elaborate, the structure-aware inpainting system102dilates a digital image mask (e.g., indicated by the differently sized masks shaped like blobs) within the digital image330to help train a generative inpainting neural network to generate inpainted digital images by sampling from regions of different sizes (over various iterations) to avoiding overfitting. In some cases, the structure-aware inpainting system102randomly dilates a digital image mask such as an object instance mask or a random pattern mask (e.g., to increase or decrease its size by a random width and/or random number of times) to prevent the generative inpainting neural network from inpainting pixels that look too much like the masked object. In these or other cases, the structure-aware inpainting system102dilates a digital image mask within particular size bounds (e.g., not smaller than a lower limit and not larger than an upper limit). As shown, the digital image330includes three masks of different sizes, though this is merely illustrative (and each dilation would be for a different training iteration).

To further prevent overfitting, as further illustrated inFIG.3D, the structure-aware inpainting system102translates a digital image mask (in addition or alternatively to dilating), as indicated in the digital image332. To elaborate, the structure-aware inpainting system102uses random circular translation to translate (and/or rotate) a digital image mask such as an object instance mask or a random pattern mask. In some embodiments, the structure-aware inpainting system102uses a different translation to, for example, translate a mask by a random number of pixels in a random direction. In some cases, the structure-aware inpainting system102translates a mask within certain distance bounds (e.g., beneath an upper distance limit in a given direction) to avoid sampling pixels that differ too greatly from those of the masked object while also preventing the generative inpainting neural network from inpainting pixels that match the masked object too closely. Indeed, by translating an object mask, the structure-aware inpainting system102avoids rigid or strict boundaries for inpainting the hole region of a digital image mask for smoother, more accurate results. As shown, the digital image332includes three masks in different locations, though this is merely illustrative, and more or fewer masks are possible in different locations.

As further mentioned above, in certain embodiments, the structure-aware inpainting system102improves or modifies object-aware training by dilating a digital image mask along a segmentation boundary. In particular, the structure-aware inpainting system102randomly (or by a specified amount) dilates a hole region to prevent color bleeding or leaking of background pixels into object pixels of an inpainted region.FIG.3Eillustrates an example of dilating a digital image mask or a hole region along a segmentation boundary in accordance with one or more embodiments. The dilated digital image masks ofFIG.3Eare an example for discussion purposes, and different shapes, numbers, sizes, and other variations are possible.

As illustrated inFIG.3E, the structure-aware inpainting system102randomly (or by a specified amount) dilates a hole region or a digital image mask within the digital image334. For instance, the structure-aware inpainting system102dilates a digital image mask along a segmentation boundary (e.g., as indicated by a segmentation neural network). Thus, for successive training iterations, the structure-aware inpainting system102utilizes masks of varying size and/or shape (as indicated by the variation of the different dashed lines along a segmentation boundary for the dump truck object instance in the digital image334) for inpainting the same hole region to ensure more robust pixel sampling, especially in areas near the boundary of the hole or mask. By dilating the digital image mask in this way, the structure-aware inpainting system102prevents or reduces pixel bleeding or leaking and generates more accurate, higher quality inpainted digital images.

As mentioned, in certain described embodiments, the structure-aware inpainting system102utilizes masked regularization in addition (or alternatively) to object-aware training. In particular, the structure-aware inpainting system102utilizes masked regularization to penalize a discriminator neural network from overfitting during training.FIG.4illustrates a sequence of acts for utilizing masked regularization as part of training a generative inpainting neural network in accordance with one or more embodiments.

As illustrated inFIG.4, the structure-aware inpainting system102accesses (e.g., receives or retrieves from the database112) a sample digital image402. In addition, the structure-aware inpainting system102generates a masked digital image404from the sample digital image402by, for example, generating an object mask for masking one or more object instances or generating a random pattern mask from random (free-form) strokes and/or rectangles. As shown, the structure-aware inpainting system102further inputs the masked digital image404into an encoder neural network406(as part of a generative inpainting neural network), whereupon the encoder neural network406extracts or encodes a feature vector from the masked digital image404.

In addition, the structure-aware inpainting system102passes the encoded feature vector to a generator neural network408(as part of the generative inpainting neural network). The generator neural network408further generates an inpainted digital image410from the encoded feature vector extracted by the encoder neural network406. Additionally, the structure-aware inpainting system102utilizes a discriminator neural network412to compare the inpainted digital image410with the sample digital image402. By comparing the inpainted digital image410with the sample digital image402the discriminator neural network412generates an authenticity prediction416that indicates whether the inpainted digital image410is real or fake. Indeed, the structure-aware inpainting system102utilizes an adversarial loss to compare the inpainted digital image410and the sample digital image402. In some cases, the structure-aware inpainting system102further utilizes a perceptual loss in addition (or alternatively) to the adversarial loss. Indeed, the perceptual loss and/or the adversarial loss is optionally part of the object-aware training and/or the masked regularization for modifying parameters of a generative inpainting neural network.

To generate the authenticity prediction416, in some case, the structure-aware inpainting system102utilizes masked regularization414to regularize how the discriminator neural network412processes data for comparing the inpainted digital image410with the sample digital image402. To elaborate, the structure-aware inpainting system102utilizes a masked regularization to stabilize adversarial training by penalizing the discriminator neural network412from overfitting.

For example, the structure-aware inpainting system102utilizes an R1 regularization but modifies the R1 regularization utilizing a digital image mask. Specifically, the structure-aware inpainting system102utilizes a masked R1 regularization specifically designed for inpainting, where incorporating the digital image mask into the regularization avoids computing a gradient penalty inside the mask region and reduces the harmful impact of computing regularization outside of holes. In some cases, the structure-aware inpainting system102utilizes a masked R1 regularization given by:

R¯1=γ2⁢𝔼pd⁢a⁢t⁢a[m⊙∇D⁡(x)2]
whereR1represents an R1 regularization term, m represents a digital image mask indicating a hole region to inpaint, γ represents a balancing weight,pdatarepresents a sampling of images (e.g., sample digital images) from real images, and D(x) represents an output of the discriminator neural network412(e.g., the authenticity prediction416).

Based on the authenticity prediction416, in certain embodiments, the structure-aware inpainting system102back propagates to modify or update parameters of the encoder neural network406, the generator neural network408, and/or the discriminator neural network412. For example, the structure-aware inpainting system102modifies internal weights and biases associated with one or more layers or neurons of the encoder neural network406, the generator neural network408, and/or the discriminator neural network412to reduce a measure of loss (e.g., adversarial loss and/or perceptual loss). By reducing one or more measures of loss, the structure-aware inpainting system102improves the inpainting of the generative inpainting neural network (by improving the encoder neural network406and/or the generator neural network408) to reduce one or more measures of loss for fooling the discriminator neural network412.

As mentioned above, in certain described embodiments, the structure-aware inpainting system102generates an inpainted digital image by inpainting a hole region of an initial digital image. In particular, the structure-aware inpainting system102utilizes a trained generative inpainting neural network with parameters learned via one or more of object-aware training and/or masked regularization.FIG.5illustrates generating an inpainted digital image utilizing a trained generative inpainting neural network in accordance with one or more embodiments.

As illustrated inFIG.5, the structure-aware inpainting system102accesses or receives (e.g., from the client device108) a digital image502that depicts a hole or a mask of pixels to inpaint. For example, the structure-aware inpainting system102receives the digital image502via upload and/or in response to user interaction selecting an object or some other portion of the digital image502to remove or replace (e.g., as indicated by the rectangular hole). In some cases, the structure-aware inpainting system102utilizes a segmentation neural network to indicate object segmentations or object masks (for each object instance) within the digital image502. The structure-aware inpainting system102further receives a user selection of an object mask (or an object segment) from among the instance-specific object masks as a region to inpaint. In some embodiments, the structure-aware inpainting system102receives a different user selection (e.g., a click and drag of a rectangle or a lasso of a particular area of pixels) indicating a region to inpaint.

In addition, the structure-aware inpainting system102utilizes a trained generative inpainting neural network504(e.g., the generative inpainting neural network116) to generate an inpainted digital image506from the digital image502. Indeed, the trained generative inpainting neural network504accurately generates replacement pixels for filling the hole region and inpaints the hole region with the replacement pixels according to internal network parameters learned via one or more of object-aware training and/or masked regularization. As shown, the inpainted digital image506depicts a seamless scene of a koala in a tree.

As mentioned above, in some embodiments, the structure-aware inpainting system102improves accuracy over prior systems. Indeed, experimenters have demonstrated that the object-aware training and the masked regularization improve the accuracy of generative inpainting models (of various architectures) in generating inpainted digital images.FIG.6illustrates an example table602comparing a generative inpainting neural network without masked regularization against a generative inpainting neural network with masked regularization in accordance with one or more embodiments.

As illustrated inFIG.6, the table602indicates results of an ablation study using the Places2 evaluation set. Indeed, each neural network of the table602was trained using the same dataset, one with masked regularization and one without. As shown, utilizing masked regularization as part of training results in more favorable results during evaluation. For example, the generative inpainting neural network with masked regularization has a lower Frechet Inception Distance (FID) and a higher paired inception discriminative score (P-IDS), both of which indicate a higher degree of accuracy and/or increased image quality. Looking at the results for perceptual image patch similarity distance (LPIPS) and unpaired inception discriminative distance (U-IDS), each neural network has comparable or nearly identical metrics. Based on the table602, the neural network trained with the masked regularization exhibits better performance.FIG.7particularly illustrates visual improvements that result from utilizing the object instance masks described above as part of a training process.

Additionally, in certain embodiments, the structure-aware inpainting system102trains neural network with improved accuracy for higher quality results. In particular, the structure-aware inpainting system102utilizes object-aware training and/or masked regularization to generate high quality inpainted digital images.FIG.7illustrates example inpainted digital images output by several different systems, each with different training procedures in accordance with one or more embodiments.FIG.7particularly illustrates visual improvements that result from utilizing the object instance masks described above as part of a training process.

As illustrated inFIG.7, the structure-aware inpainting system102generates the inpainted digital image710from the input digital image702. Indeed, the structure-aware inpainting system102generates the inpainted digital image710using a generative inpainting neural network with parameters learned via object-aware training. For instance, the structure-aware inpainting system102identifies the hole region indicated by the digital image mask, generates replacement pixels utilizing a generative inpainting neural network, and generates the inpainted digital image710by filling the hole region with the replacement pixels. By contrast, the inpainted digital images704,706, and708are generated by systems that do not utilize object-aware training.

Specifically, the inpainted digital image704is generated by ProFill as described by Yu Zheng et al inHigh-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling. In addition, the inpainted digital image706is generated by LaMa as described by Roman Suvorov et al. inResolution-Robust Large Mask Inpainting with Fourier Convolutions, arXiv:2109:07161 (2021). In some cases, the LaMa model utilizes salient object masks which, as mentioned above, results in particular issues especially in object completion applications (e.g., because saliency annotation only captures large dominant foreground objects and ignores background objects). Further, the inpainted digital image708is generated by CoModGAN as described by Shengyu Zhao et al inLarge Scale Image Completion via Co-Modulated Generative Adversarial Networks.

As shown, the inpainted digital image704includes nonsensical artifacts in the inpainted region, with part of a tree floating in air without a trunk, in addition to unrealistic clouds in a virtually straight line through the inpainted region. Similarly, the inpainted digital image706includes an artifact in the form of a floating portion of a tree along with blurry tree colors mixed with sky colors in areas near the tree portion. Additionally, the inpainted digital image708depicts multiple floating tree portions disconnected from one another and hovering in the sky. By contrast, the inpainted digital image710generated by the structure-aware inpainting system102includes high quality detail without artifacts or blurring, where a tree is generated and inpainted with no floating parts and a trunk connecting it to the ground for better visual coherence.

Looking now toFIG.8, additional detail will be provided regarding components and capabilities of the structure-aware inpainting system102. Specifically,FIG.8illustrates an example schematic diagram of the structure-aware inpainting system102on an example computing device800(e.g., one or more of the client device108and/or the server(s)104). As shown inFIG.8, the structure-aware inpainting system102includes an object-aware training manager802, a masked regularization training manager804, an image inpainting manager806, and a storage manager808.

As just mentioned, the structure-aware inpainting system102includes an object-aware training manager802. In particular, the object-aware training manager802manages, maintains, performs, implements, applies, or utilizes object-aware training techniques to train a generative inpainting neural network812. For example, the object-aware training manager802learns parameters for the object-aware training manager802by generating object masks in sample digital images, sampling from masked digital images, determining an overlap ratio, and modifying parameters of the generative inpainting neural network812according to the overlap ratio. Additional detail regarding object-aware training is provided above.

As further mentioned, the structure-aware inpainting system102includes a masked regularization training manager804. In particular, the masked regularization training manager804manages, maintains, performs, implements, applies, or utilizes masked regularization techniques for training the generative inpainting neural network812. For example, the masked regularization training manager804utilizes the above-described techniques to penalize a discriminator neural network from overfitting by applying a regularization that incorporates a digital image mask for an object instance within a sample digital image.

As shown, the structure-aware inpainting system102also includes an image inpainting manager806. In particular, the image inpainting manager806manages, maintains, performs, implements, or applies digital image inpainting to generate an inpainted digital image. For example, the image inpainting manager806inpaint or fills one or more hole regions with replacement pixels utilizing the generative inpainting neural network812with parameters learned via object-aware training and/or masked regularization.

The structure-aware inpainting system102further includes a storage manager808. The storage manager808operates in conjunction with, or includes, one or more memory devices such as the database810(e.g., the database112) that stores various data such as sample digital images for training and/or the generative inpainting neural network812.

In one or more embodiments, each of the components of the structure-aware inpainting system102are in communication with one another using any suitable communication technologies. Additionally, the components of the structure-aware inpainting system102is in communication with one or more other devices including one or more client devices described above. It will be recognized that although the components of the structure-aware inpainting system102are shown to be separate inFIG.8, any of the subcomponents may be combined into fewer components, such as into a single component, or divided into more components as may serve a particular implementation. Furthermore, although the components ofFIG.8are described in connection with the structure-aware inpainting system102, at least some of the components for performing operations in conjunction with the structure-aware inpainting system102described herein may be implemented on other devices within the environment.

The components of the structure-aware inpainting system102include software, hardware, or both. For example, the components of the structure-aware inpainting system102include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices (e.g., the computing device800). When executed by the one or more processors, the computer-executable instructions of the structure-aware inpainting system102cause the computing device800to perform the methods described herein. Alternatively, the components of the structure-aware inpainting system102comprise hardware, such as a special purpose processing device to perform a certain function or group of functions. Additionally, or alternatively, the components of the structure-aware inpainting system102include a combination of computer-executable instructions and hardware.

Furthermore, the components of the structure-aware inpainting system102performing the functions described herein may, for example, be implemented as part of a stand-alone application, as a module of an application, as a plug-in for applications including content management applications, as a library function or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components of the structure-aware inpainting system102may be implemented as part of a stand-alone application on a personal computing device or a mobile device. Alternatively, or additionally, the components of the structure-aware inpainting system102may be implemented in any application that allows creation and delivery of content to users, including, but not limited to, applications in ADOBE® EXPERIENCE MANAGER and CREATIVE CLOUD®, such as PHOTOSHOP®, LIGHTROOM®, and INDESIGN®. “ADOBE,” “ADOBE EXPERIENCE MANAGER,” “CREATIVE CLOUD,” “PHOTOSHOP,” “LIGHTROOM,” and “INDESIGN” are either registered trademarks or trademarks of Adobe Inc. in the United States and/or other countries.

FIGS.1-8the corresponding text, and the examples provide a number of different systems, methods, and non-transitory computer readable media for training a generative inpainting neural network via object-aware training and/or masked regularization for accurate digital image inpainting. In addition to the foregoing, embodiments can also be described in terms of flowcharts comprising acts for accomplishing a particular result. For example,FIGS.9-11illustrate flowcharts of example sequences or series of acts in accordance with one or more embodiments.

WhileFIGS.9-11illustrate acts according to particular embodiments, alternative embodiments may omit, add to, reorder, and/or modify any of the acts shown inFIGS.9-11. The acts ofFIGS.9-11can be performed as part of a method. Alternatively, a non-transitory computer readable medium can comprise instructions, that when executed by one or more processors, cause a computing device to perform the acts ofFIGS.9-11. In still further embodiments, a system can perform the acts ofFIGS.9-11. Additionally, the acts described herein may be repeated or performed in parallel with one another or in parallel with different instances of the same or other similar acts.

FIG.9illustrates an example series of acts900for learning parameters for a generative inpainting neural network via object-aware training. In particular, the series of acts900includes an act902of generating object masks. For example, the act902involves an act904of generating instance-specific object segmentations. In some cases, the act902involves generating a set of object masks for objects within a digital image utilizing a segmentation neural network. In certain embodiments, the act902involves generating object masks corresponding to individual object instances depicted within the digital image utilizing a panoptic segmentation neural network. In one or more implementations, the series of acts900includes an act of generating the set of masked digital images to include masked digital images depicting masked object instances and additional masked digital images depicting random pattern masks.

As shown, the series of acts900includes an act906of selecting a masked digital image. In particular, the act906includes an act908of generating a random pattern mask and/or an act910of generating a masked object instance. In one or more embodiments, the act906involves selecting a masked digital image from a set of masked digital images depicting masked object instances indicated by the set of object masks for the digital image. Indeed, the act906sometimes involves generating a set of masked digital images including random pattern masks and/or masked object instances from which to select.

As further illustrated inFIG.9, the series of acts900includes an act912of generating an inpainted digital image from the masked digital image utilizing a generative inpainting neural network. In particular, act912involves generating, utilizing the generative inpainting neural network, an inpainted digital image by filling the hole region with the replacement pixels.

As shown, the series of acts900includes an act914of modifying network parameters of the generative inpainting neural network. In particular, the act914includes modifying parameters of the generative inpainting neural network based on a comparison of the inpainted digital image and the digital image (prior to any modification). For example, the act914includes backpropagating a loss using gradient based algorithm to update the parameters of the generative inpainting neural network.

In one or more embodiments, the series of acts900includes an act of determining an overlap ratio between a digital image mask of the masked digital image and a masked object instance of the masked object instances. The series of acts900optionally further involves comparing the overlap ratio with an overlap ratio threshold. In some cases, the series of acts900also includes an act of modifying the digital image mask to exclude the masked object instance based on comparing the overlap ratio with the overlap ratio threshold. In these or other embodiments, the series of acts900also includes acts of comparing the inpainted digital image with the digital image and modifying the parameters of the generative inpainting neural network according to comparing the inpainted digital image with the digital image.

In certain cases, the series of acts900includes an act of reducing overfitting by the generative inpainting neural network by dilating and translating the masked object instance. In these or other cases, the series of acts900includes an act of reducing leaking of background pixels into a hole region of the digital image indicated by the digital image mask by dilating the digital image mask along a segmentation boundary indicated by the set of object masks.

FIG.10illustrates an example series of acts1000for learning parameters for a generative inpainting neural network via masked regularization. In particular, the series of acts1000includes an act1002of generating a digital image mask. For example, the act1002involves generating a digital image mask indicating a hole region within the digital image. In some embodiments, the act1002involves generating one or more of a random pattern mask comprising a combination of random strokes and shapes or a masked object instance shaped like an object mask of an object instance depicted within the digital image.

As shown, the series of acts1000includes an act1004of generating an inpainted digital image. In particular, the act1004involves generating an inpainted digital image from the digital image by inpainting the hole region utilizing the generative inpainting neural network.

Additionally, the series of acts1000includes an act1006of penalizing a discriminator neural network with masked regularization. In particular, the act1006includes an act1007of utilizing an R1 regularization that incorporates the digital image mask. For instance, the act1006involves comparing the inpainted digital image with a digital image utilizing a masked regularization from the digital image mask to penalize the discriminator neural network from overfitting. In certain embodiments, the act1006involves comparing the inpainted digital image with an unmodified version of the digital image without the hole region.

Further, the series of acts1000includes an act1008of modifying parameters of a generative inpainting neural network. In particular, the act1008involves modifying parameters of the generative inpainting neural network based on comparing the inpainted digital image with the digital image.

In some cases, the series of acts1000includes an act of generating a set of object masks indicating objects depicted within the digital image and an act of generating the digital image mask by generating a masked object instance corresponding to an object instance from among the objects depicted within the digital image. In one or more embodiments, the series of acts1000includes acts of determining an overlap ratio between the digital image mask and the masked object instance, generating a modified digital image mask from the digital image mask according to the overlap ratio, and generating the inpainted digital image by inpainting a modified hole region indicated by the modified digital image mask. In certain embodiments, determine the overlap ratio involves comparing mask pixels occupied by the digital image mask with segmentation pixels occupied by the masked object instance.

FIG.11illustrates an example series of acts1100for generating an inpainted digital image utilizing a generative inpainting neural network trained via object-aware training and/or masked regularization. In particular, the series of acts1100includes an act1102of identifying a hole region within a digital image. For example, the act1102involves utilizing a segmentation neural network to generate object masks for objects within the digital image, receiving a user selection of an object mask, and identifying pixels within the object mask as the hole region.

In addition, the series of acts1100includes an act1004of generating replacement pixels. For example, the act1104includes an act1106of utilizing a generative inpainting neural network trained with object-aware training and/or masked regularization. Indeed, the act1104involves generating replacement pixels from the digital image to replace the hole region utilizing a generative inpainting neural network comprising parameters learned via one or more of object-aware training or masked regularization.

In some embodiments, the object-aware training includes generating, from a digital image, a set of masked digital images that includes masked digital images depicting object instance masks and masked digital images depicting random pattern masks, selecting a masked digital image from the set of masked digital images, generating an inpainted digital image from the masked digital image, comparing the inpainted digital image with the digital image, and modifying the parameters of the generative inpainting neural network according to comparing the inpainted digital image with the digital image.

In these or other embodiments, the object-aware training involves determining a set of object masks for a digital image utilizing a segmentation neural network, determining an overlap ratio between a digital image mask of the digital image and an object mask from among the set of object masks, and modifying the parameters of the generative inpainting neural network according to the overlap ratio. Comparing the inpainted digital image with the digital image utilizing the masked regularization can include utilizing a discriminator neural network to generate an authenticity prediction associated with the inpainted digital image according to the masked regularization to avoid determining a gradient penalty inside the digital image mask.

In some embodiments, the series of acts1100includes an act of learning parameters for the generative inpainting neural network by: generating a digital image mask for a digital image, generating an inpainted digital image from the digital image by inpainting a hole region indicated by the digital image mask, comparing the inpainted digital image with a digital image utilizing a masked regularization obtained from the digital image mask, and modifying the parameters of the generative inpainting neural network according to comparing the inpainted digital image with the digital image.

Further, the series of acts1100includes an act1108of generating an inpainted digital image. In particular, the act1108involves generating, utilizing the generative inpainting neural network, an inpainted digital image by filling the hole region with the replacement pixels.

Embodiments of the present disclosure may comprise or utilize a special purpose or general-purpose computer including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. In particular, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). In general, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium, (e.g., a memory, etc.), and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.

Computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer-executable instructions are non-transitory computer-readable storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments of the disclosure can comprise at least two distinctly different kinds of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

Non-transitory computer-readable storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

A “network” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and/or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media.

Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to non-transitory computer-readable storage media (devices) (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a “NIC”), and then eventually transferred to computer system RAM and/or to less volatile computer storage media (devices) at a computer system. Thus, it should be understood that non-transitory computer-readable storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general-purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. In some embodiments, computer-executable instructions are executed on a general-purpose computer to turn the general-purpose computer into a special purpose computer implementing elements of the disclosure. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.

Those skilled in the art will appreciate that the disclosure may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, and the like. The disclosure may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

Embodiments of the present disclosure can also be implemented in cloud computing environments. In this description, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be employed in the marketplace to offer ubiquitous and convenient on-demand access to the shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization and released with low management effort or service provider interaction, and then scaled accordingly.

A cloud-computing model can be composed of various characteristics such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and so forth. A cloud-computing model can also expose various service models, such as, for example, Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud-computing model can also be deployed using different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, and so forth. In this description and in the claims, a “cloud-computing environment” is an environment in which cloud computing is employed.

FIG.12illustrates, in block diagram form, an example computing device1200(e.g., the computing device800, the client device108, and/or the server(s)104) that may be configured to perform one or more of the processes described above. One will appreciate that the structure-aware inpainting system102can comprise implementations of the computing device1200. As shown byFIG.12, the computing device can comprise a processor1202, memory1204, a storage device1206, an I/O interface1208, and a communication interface1210. Furthermore, the computing device1200can include an input device such as a touchscreen, mouse, keyboard, etc. In certain embodiments, the computing device1200can include fewer or more components than those shown inFIG.12. Components of computing device1200shown inFIG.12will now be described in additional detail.

In particular embodiments, processor(s)1202includes hardware for executing instructions, such as those making up a computer program. As an example, and not by way of limitation, to execute instructions, processor(s)1202may retrieve (or fetch) the instructions from an internal register, an internal cache, memory1204, or a storage device1206and decode and execute them.

The computing device1200includes memory1204, which is coupled to the processor(s)1202. The memory1204may be used for storing data, metadata, and programs for execution by the processor(s). The memory1204may include one or more of volatile and non-volatile memories, such as Random-Access Memory (“RAM”), Read Only Memory (“ROM”), a solid-state disk (“SSD”), Flash, Phase Change Memory (“PCM”), or other types of data storage. The memory1204may be internal or distributed memory.

The computing device1200includes a storage device1206includes storage for storing data or instructions. As an example, and not by way of limitation, storage device1206can comprise a non-transitory storage medium described above. The storage device1206may include a hard disk drive (HDD), flash memory, a Universal Serial Bus (USB) drive or a combination of these or other storage devices.

The computing device1200also includes one or more input or output (“I/O”) devices/interfaces1208, which are provided to allow a user to provide input to (such as user strokes), receive output from, and otherwise transfer data to and from the computing device1200. These I/O devices/interfaces1208may include a mouse, keypad or a keyboard, a touch screen, camera, optical scanner, network interface, modem, other known I/O devices or a combination of such I/O devices/interfaces1208. The touch screen may be activated with a writing device or a finger.

The I/O devices/interfaces1208may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, devices/interfaces1208is configured to provide graphical data to a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and/or any other graphical content as may serve a particular implementation.

The computing device1200can further include a communication interface1210. The communication interface1210can include hardware, software, or both. The communication interface1210can provide one or more interfaces for communication (such as, for example, packet-based communication) between the computing device and one or more other computing devices1200or one or more networks. As an example, and not by way of limitation, communication interface1210may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI. The computing device1200can further include a bus1212. The bus1212can comprise hardware, software, or both that couples components of computing device1200to each other.

In the foregoing specification, the invention has been described with reference to specific example embodiments thereof. Various embodiments and aspects of the invention(s) are described with reference to details discussed herein, and the accompanying drawings illustrate the various embodiments. The description above and drawings are illustrative of the invention and are not to be construed as limiting the invention. Numerous specific details are described to provide a thorough understanding of various embodiments of the present invention.

The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. For example, the methods described herein may be performed with less or more steps/acts or the steps/acts may be performed in differing orders. Additionally, the steps/acts described herein may be repeated or performed in parallel with one another or in parallel with different instances of the same or similar steps/acts. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.