Patent Description:
Unsupervised and weakly supervised object segmentation. In [<NUM>] authors propose a GAN-based [<NUM>] technique to generate object segmentation masks from bounding boxes. Their training pipeline consists of taking two crops of the same image: one with object and one without any object. Objects are detected using Faster R-CNN [<NUM>].

Then they train a GAN to produce a segmentation mask so that these two crops merged with that mask result into a plausible image. Authors use a combination of adversarial loss, existence loss (which verifies that an object is present on an image) and cut loss (which verifies that no object part left after an object has been cut). They experiment with only some classes from Cityscapes [<NUM>] and all classes from MS COCO [<NUM>] datasets. Authors report that their approach achieves higher mean intersection-over-union values than classic GrabCut [<NUM>] algorithm and recent Simple-Does-It [<NUM>]. That approach requires a pretrained Faster R-CNN and a special policy for foreground and background patch selection. It also experiences difficulties with properly segmenting some object classes (e.g. kite, giraffe etc). Their approach also works well only with small resolution images (<NUM> X <NUM>).

In [<NUM>] authors propose an annotation-free framework to learn segmentation network for homogeneous objects. They use an adaptive synthetic data generation process to create a training data set.

While being traditionally tackled with superpixel clustering, unsupervised image segmentation recently has been addressed with deep learning [<NUM>]. In the latter paper authors propose to maximize information between two clustered vectors obtained by fully convolutional network from nearby patches of the same image. A similar technique, but constrained with reconstruction loss, has been proposed in [<NUM>]. Authors describe W-Net (autoencoder with U-Netlike encoder and decoder), which tries to cluster pixels at inner layer and then reconstruct image from pixel clusters. Their segmentation result is unaware of object classes.

Visual grounding. Methods for visual grounding aim on unsupervised or weakly supervised matching of freeform text queries and regions of images. Usually super-vision takes form of pairs of (Image; Caption). Model performance is usually measured as intersection-over-union against ground truth labels. The most popular datasets are Visual Genome [<NUM>], Flickr30k [<NUM>], Refer-It-Game [<NUM>] and MS COCO [<NUM>]. General approach to grounding consists in predicting if the given caption and image corresponds to each other. Negative samples are obtained by shuffling captions and images independently. Text-image attention is the core feature of most models for visual grounding. Obviously, using more fine-grained supervision (e.g. region-level annotations instead of image-level) allows to achieve higher scores [<NUM>].

Trimap generation. Trimap generation is a problem of producing a segmentation of an image into three classes: foreground, background and unknown (transparent foreground). Most algorithms require human intervention to propose trimap, but recently superpixel and clustering based approaches have been proposed for automatic trimap generation [<NUM>]. However, their approach requires executing multiple optimization steps for each image. Deep learning is used to produce alpha matting mask given image and a trimap [<NUM>]. There is also some work on video matting and background substitution in video [<NUM>]. They use per-frame superpixel segmentation and then optimize energy in conditional random field of Gaussian mixture models to separate foreground and background frame-by-frame.

Generative adversarial networks. In the latest years, GANs [<NUM>] are probably the most frequently used approach to train a generative model. Yet powerful, they prone to unstable training process and inconsistent performance on higher resolution images. A more recently proposed approach, CycleGAN [<NUM>] trains two GANs together to establish bidirectional mapping between two domains. Their approach offers much greater stability and consistency. On contrary, it requires the dataset to visualize a kind of invertible operation. A plenty of modifications and applications to CycleGAN have been published, including semantic image manipulation [<NUM>], domain adaptation [<NUM>], unsupervised image-to-image translation [<NUM>], multi-domain translation [<NUM>] and many others. There is also a problem that such a mapping between domains may be ambiguous. BicycleGAN [<NUM>] and augmented CycleGAN [<NUM>] address that problem by requiring that mapping must preserve latent representations.

In that paper we base on ideas of Cut&Paste [<NUM>] and CycleGAN [<NUM>] and propose a novel architecture and pipeline, which addresses a different problem (background swapping) and achieve better results on unsupervised object segmentation, inpainting and image blending.

The present invention presents a novel approach to visual understanding by simultaneously learning to segment object masks and remove objects from background (aka cut and paste).

Proposed is a computing system as defined in claim <NUM>, with embodiments as defined in claims <NUM>-<NUM>.

Proposed is a computer-implemented method as defined in claim <NUM>, with an embodiment as defined in claim <NUM>.

The above and/or other aspects will be more apparent by describing exemplary embodiments with reference to the accompanying drawings, in which:.

The proposed invention can be useful hardware comprising software products and devices that perform automatic or automated image processing, including:.

The symbols used in the application materials are explained below.

The proposed image processing functions require less detailed control on the part of the person, compared to the existing analogues at the moment.

The proposed solution can be implemented in software, which in turn can be run on any devices with sufficient computing power.

Throughout the paper, we denote images as object background tuples, e.g. x=<O,Bx> means that image x contains object O and background B x, and y=<Ø,By> means that image y contains background B y and no objects.

The main problem that we address in this work can be formulated as follows. Given a dataset of background images y={<Ø,By>} y∈Y and a dataset of objects on different backgrounds x= {<O,Bx>} x∈X (unpaired, i.e., with no mapping between X and Y ), train a model to take an object from an image x∈X and paste it onto a new background defined by an image y∈Y, while at the same time deleting it from the original background. In other words, the problem is to transform a pair of images x=<O,Bx> and y=<Ø,By> into a new pair x̂= <∅, B̂x> and ŷ =<Ô, B̂y>, where B̂x≈B̂, B̂y≈B̂, and Ô≈O , and Ô≈O but the object and both backgrounds are changed so that the new images look natural.

This general problem can be decomposed into three subtasks:.

For each of these tasks we construct a separate neural network that accepts an image or a pair of images and outputs new image or images of the same dimensions. However, our main hypothesis that we explore in this work is that in the absence of large paired and labeled datasets (which is the normal state of affairs in most applications), it is highly beneficial to train all these neural networks together.

Thus, we present our SEIGAN (Segment-Enhance-Inpaint) architecture that combines all three components in a novel and previously unexplored way. In the <FIG> boxes with dotted outline denote data (images); ellipses denote objects contained in the data; boxes with sharp corners denote subprograms implementing neural networks; boxes with rounded corners denote subprograms which control the process of tuning neural network parameters during the training procedure; lines denote flows of data during training procedure (the fact that an arrow points from one box to another means that the results of the first box are passed as input to the second). We outline the general flow of our architecture on <FIG>; the "swap network" module there combines segmentation and enhancement. Since cut-and-paste is a partially reversible operation, it is natural to organize the training procedure in a way similar to CycleGAN [<NUM>]: the swap and inpainting networks are applied twice in order to complete the cycle and be able to use the idempotency property for the loss functions. We denote by x̂ and ŷ and y the results of the first application, and by x̂ and ŷ the results of the second application, moving the object back from x̂ and ŷ (see <FIG>).

The architecture, showed in <FIG>, combines five different neural networks, three used as generators, which create an image and convert it, and two as discriminators, which estimate plausibility of the image:.

Generators G seg and G enh constitute the so-called "swap network" depicted as a single unit on <FIG> and explained in detail on <FIG>. This figure depicts the architecture of the "swap network" (a box named "Swap network" on <FIG>) along with a minimal set of other entities needed to describe how the "swap network" is used. Boxes with dotted outline denote data (images); ellipses denote objects contained in the data; boxes with sharp corners denote subprograms implementing neural networks; boxes with rounded corners denote subprograms which control the process of tuning neural network parameters during the training procedure; lines denote flows of data during training procedure (the fact that an arrow points from one box to another means that the results of the first box are passed as input to the second). Segmentation network is a neural network which takes an image and outputs a segmentation mask of the same size. Refinement network takes an image and outputs an improved its version (i.e. with more realistic colors, with artifacts removed, etc.) of the same size.

Compared to [<NUM>], the training procedure in SEIGAN has proven to be more stable and able to work in higher resolutions. Furthermore, our architecture allows to address more tasks (inpainting and blending) simultaneously rather than only predicting segmentation masks. As usual in GAN design, the secret sauce of the architecture lies in a good combination of different loss functions. In SEIGAN, we use a combination of adversarial, reconstruction, and regularization losses.

The inpainting network Ginp aims to produce a plausible background B̂x given a source image (<NUM>-m)⊙x, which represents the original image x with the object subtracted according to segmentation mask m obtained by applying the segmentation network, m=G seg(x); in practice, we fill the pixels of m⊙x with white. Parameters of inpainting networks are optimized during the end-to-end training according to the following loss functions (shown by rounded rectangles on <FIG>).

The adversarial background loss aims to improve the plausibility of the resulting image. It is implemented with a dedicated discriminator network Dbg. For Dbg, we use the same architecture as in the original CycleGAN [<NUM>] except for the number of layers; our experiments have shown that a deeper discriminator works better in our setup. As the loss function Dbg uses the MSE adversarial loss suggested in Least Squares GAN (LSGAN) [<NUM>], as in practice it is byfar more stable than other types of GAN loss functions:
<MAT>
where y=<Ø,By> is the original background image,
x̂ =<∅, B̂x> is the background image resulting from x after the first swap, and <MAT> is the background image resulting from ŷ after the second swap.

The background reconstruction loss aims to preserve information about the original background Bx. It is implemented using texture loss [<NUM>], the mean absolute difference between Gram matrices of feature maps after the first <NUM> layers of VGG-<NUM> networks:
<MAT>
where VGG(y) denotes the matrix of features of a pretrained image classification neural network (e.g. VGG but not limited to), and Gram(A)ij= ΣkAikAjk is the Gram matrix.

Our choice of loss functions is motivated by the fact that there are plenty of possible plausible reconstructions of the background, so the loss functions must allow for a certain degree of freedom that mean absolute error or mean squared error would not permit but which texture loss does. In our experiments, optimizing MAE or MSE has usually led to the generated image being filled with median or mean pixel values, with no objects or texture. Note that the background reconstruction loss is applied only to y because we do not have the ground truth background for x (see <FIG>).

Another important remark is that before feeding the image to the inpainting network Ginp, we subtract a part of image according to segmentation mask m, and we do it in a differentiable way, without any thresholding applied to m. Thus, gradients can propagate back through the segmentation mask to the segmentation network Gseg. Joint training of inpainting and segmentation has a regularization effect. First, the inpainting network Ginp wants the mask to be as accurate as possible: if it is too small then Ginp will have to erase the remaining parts of the objects, which is a much order problem, and if it is too large then Ginp will have more empty area to inpainting. Second, Ginp wants the segmentation mask m to be high-contrast (with values close to <NUM> and <NUM>) even without thresholding: if much of m is low-contrast (close to <NUM>) then Ginp will have to learn to remove the "ghost" of the object (again, much harder than just inpainting on empty space), and it will most probably be much easier for the discriminator Dbg to tell that the resulting picture is fake.

Showed in <FIG> is an example of data consumed and produced by the proposed method. The meanings of the images, from left to right, top-down:.

For Ginp, we use a neural network consisting of two residual blocks connected sequentially (see <FIG>). We also experimented with ShiftNet [<NUM>]. depicts architecture of ResNet neural network used as "inpainting network" and "segmentation network". Ellipses denote data; rectangles - layers of neural networks. The overall architecture is present in the left part of the Figure. The right part of the figure contains a more detailed description of blocks used in the left part. Arrows denote data flow (i.e. output of one block is fed as input to another block). Conv2d denote convolutional layer; BatchNorm2d denote batch normalization layer; ReLU - linear rectification unit; ReflectionPad - padding of pixels with reflection; ConvTranspose2d - deconvolutional layer.

The swap network aims to generate a new image ŷ=<Ô, B̂y> ; from two original images, x=<O,Bx> with an object O and y=<Ø,By> with a different background B y.

The swap network consists of two major steps: segmentation G seg and enhancement G enh (see <FIG>).

The segmentation network Gseg produces a soft segmentation mask m=G seg(x) from x. With the mask m, we can extract the object O from its source image x and paste it on By to produce a "coarse" version of the target image z=m⊙x+(<NUM>-m)⊙y; z is not the end result, though: it lacks anti-aliasing, color or lightning correction, and other improvements. Note that in the ideal case, pasting an object in a natural way might also require a more involved understanding of the target background; e.g., if we want to paste a dog onto a grass field then we should probably put some of the background grass in front of the dog, hiding its paws as they would not be seen behind the grass in reality.

To address this, we introduce the so-called enhancement neural network G enh whose purpose is to generate a "smoother", more natural image
ŷ=<Ô,B̂y> given original images x and y, and segmentation mask m, which lead to the coarse result z=m⊙x+(<NUM>-m)⊙y= <O, B y>. We have experimented with the enhancement network implemented in four different ways:.

In any case, we denote by G enh(x,y,m) the final improved image after all outputs of Genh have been applied to z accordingly.

We train the swap network end-to-end with the following loss functions (shown by rounded rectangles on <FIG>).

The object reconstruction loss <MAT> aims to ensure consistency and training stability. It is implemented as the mean absolute difference between the source image x = (O, Bx) and x̂ =Genh(ŷ, x̂, Gseg(ŷ)):
<MAT>,
where ŷ =Genh(x,y,Gseg(x)) and
where ŷ =Genh(x,y,Gseg(x)) and x̂ =Ginp((<NUM>-Gseg(x))⊙x, i.e.
<MAT> is the result of applying the swap network to x and y twice.

The adversarial object loss <MAT> aims to increase the plausibility of ŷ =<Ô, B̂y>. It is implemented with a dedicated discriminator network Dobj. It also has the side effect of maximizing the area covered by segmentation mask m=G seg (x). We apply this loss to all images with objects: real image x and "fake" images ŷ and x̂. Again, the discriminator has the same architecture as in CycleGAN [<NUM>] except for the number of layers, where we have found that a deeper discriminator works better. We again use the MSK loss inspired by LSGAN [<NUM>]:
<MAT>.

The mask consistency loss aims to make the segmentation network invariant against the background. It is implemented as the mean absolute distance between m = Gseg(x), the mask extracted from x = (O, Bx), and m = Gseg(y), the mask extracted from ŷ=<Ô, B̂y>:
<MAT>.

The mask is essentially black-white picture of the same size as the picture from which this mask was extracted. White pixels on the mask correspond to the selected areas of the image (pixels in which the object is depicted in this case), black ones - to the background. Mean absolute distance is the modulus of the difference in pixel values, averaged over all pixels. The mask is re-extracted to make sure that the neural network that extracts the mask responds precisely to the shape of the object, and does not respond to the background behind it (in other words, the masks for the same object must always be the same).

Finally, apart from the loss functions defined above we have used the identity loss, an idea put forward in CycleGAN [<NUM>]. We introduce two different instances of identity loss:.

The overall SEIGAN loss function is a linear combination of all loss functions defined above:
<MAT>
with coefficients chosen empirically.

During experiments, we have noticed several interesting effects. First, original images x=<O,Bx> and y=<Ø,By> might have different scale and aspect ratios before merging. Rescaling them to the same shape with bilinear interpolation would introduce significant differences in low-level textures that would be very easy to identify as fake for the discriminator, thus preventing GAN from convergence.

The authors of [<NUM>] faced the same problem and addressed it by a special procedure they use to create training samples: they took foreground and background patches only from the same image to ensure the same scale and aspect ratios, which reduces diversity and makes fewer images suitable for the training set. In our setup this problem is addressed by a separate enhancement network, so we have fewer limitations when finding appropriate training data.

Another interesting effect is the low contrast in segmentation masks when inpainting is optimized against MAE or MSE reconstruction loss. A low-contrast mask (i.e., m with many values around <NUM> rather than close to <NUM> or <NUM>) allows information about the object from the original image to "leak through" and facilitate reconstruction. A similar effect has been noticed before by other researchers, and in the CycleGAN architecture it has even been used for steganography [<NUM>]. We first addressed this issue by converting the soft segmentation mask to a hard mask by simple thresholding. Later we found that optimizing inpainting against the texture loss <MAT> is a more elegant solution that leads to better results than thresholding.

For the segmentation network Gseg, we used the architecture from CycleGAN [<NUM>], which itself is an adaptation of the architecture from [<NUM>]. For better performance, we replaced ConvTranspose layers with bilinear upsampling. Also, after the final layer of the network we used the logistic sigmoid as the activation function.

For the enhancement network Genh, we used the U-net architecture [<NUM>] since it is able both to work with images in high resolution and to make small changes in the source image. This is important for our setup because we do not want to significantly change the image content in the enhancement network but rather just "smooth" the boundaries of the pasted image in a smarter way.

<FIG> This figure depicts architecture of U-Net neural network used as "inpainting network" and "refinement network". Ellipses denote data; rectangles - layers of neural networks. The overall architecture is present in the left part of the Figure. The right part of the figure contains a more detailed description of blocks used in the left part. Arrows denote data flow (i.e. output of one block is fed as input to another block). Conv2d denote convolutional layer; BatchNorm2d denote batch normalization layer; ReLU - linear rectification unit; ReflectionPad - padding of pixels with reflection; ConvTranspose2d - deconvolutional layer.

Major part of our experiments is carried out on images, publicly available on Flickr under Creative Commons license. We used query "dog" to collect initial image. Then we used a pretrained Faster R-CNN to detect all objects (including dogs) and all regions without any object. Then we constructed two datasets {<O, B <NUM>>} (from regions with dogs) and {(B <NUM>)} (from regions without objects of any class). After data collection, we conducted data filtering procedure in order to get regions of images without any extraneous objects.

The filtering procedure was carried out as follows. First of all, we used a Faster R-CNN [<NUM>] (pretrained on MS COCO (<NUM>]) to detect all objects on an image. Then, we get crops of the input image according to the following rules:.

Claim 1:
A computing system for performing automated image processing, comprising:
means for carrying out, by use of a first neural network, a step of
forming a coarse image by segmenting an object from a first image containing the object and a first background by a segmentation mask, and, using the mask, cutting off the segmented object from the first image and pasting the segmented object onto a second image containing only a second background,
means for carrying out, by use of a second neural network, a step of
constructing an enhanced version of the second image with the pasted segmented object by enhancing the coarse image based on the first image and the second image and the mask;
means for carrying out, by use of a third neural network, a step of restoring a first background-only image of the first image without the removed segmented object by inpainting an image obtained by zeroing out pixels of the first image using the mask;
wherein the first, second and third neural networks are combined into a common architecture of neural networks for sequentially performing segmentation, enhancing and inpainting and for simultaneously learning, wherein the common architecture of neural networks accepts the images and outputs processed images of the same dimensions.