# Masked Style Transfer for Source-Coherent Image-to-Image Translation
## 1 Introduction  
Image-to-image translation (I2I) aims to generate an output image with a different style while preserving the content information of the input [1]. More specifically, the goal of I2I is to convert an image \(x_{A}\) belonging to a _source_ domain \(A\) into an image \(y_{B}\) belonging to a _target_ domain \(B\) by preserving its intrinsic content belonging to the source domain and modifying its extrinsic content by making it as similar as possible to that characterizing the target domain. A lot of frameworks that use generative models to perform I2I translation are emerging in a variety of areas: from face editing [2] to style transfer [3] and the automotive field [4]. Focusing on style transfer, StarGANv2 [5] introduces an innovative approach. Specifically, StarGANv2 incorporates a style encoder that is designed to extract the style characteristics of an image, which is referred to as the _reference image_. Subsequently, the extracted style is applied to the input image using a single generator that is able to perform image translation across multiple domains. StarGANv2 also has a mapping network that is in charge of generating styles for the generator from random noise. In their work, the creators of StarGANv2 introduce diversity as the characteristic of each image within a domain to be different, despite the images coming from the same domain. By this definition, the authors show how the output changes as the reference changes, even if the images are picked from the same domain. However, this architecture design tends to apply global changes to the entire input image without preserving its intrinsic content representation. This canbe described as a heavy form of _reference-based_ style transfer, which can be seen in the generated output images by extracting the style from the same reference. In such cases, the output images tend to closely mirror the reference image, and the generated images collapse to the reference image, as can be seen in Figure 1.  
To address this limitation, we present a novel architecture that is able to perform _source-coherent_ IZI translation across multiple domains. Our solution involves adding a segmentation layer before the style encoder: this layer computes segmentation masks that are used to separate the subjects within the reference image and to select only the desired part of the image. In this way, we can remove all of the unnecessary content in the reference, like the background or out-of-domain parts. Ultimately, the style encoder extracts the style only from the relevant part of the reference image, and the generator produces images with the style of the reference image without collapsing to it. Our network architecture takes inspiration from StarGANv2 [5], though it has some fundamental changes; in particular, we change the style application by using cross attention layers [6] and not adaptive instance normalization (AdaIN) [7]; then, we adapt the style encoder by feeding it with both the image and its corresponding mask. Moreover, a crucial aspect lies in the utilization of an unsupervised architecture for extracting masks from reference images. Specifically, we choose to use the STEGO [8] model, which is an architecture that can perform unsupervised semantic segmentation, in order to produce masks. With STEGO, we produce binary images that are used in order to separate information regarding where to extract the style code inside the style encoder. To summarize, the main contributions of the proposed work are as follows:  
* Innovative architecture for style transfer: We introduce a novel architecture that is able to perform source-coherent IZI translation between multiple domains by preserving input details and increasing diversity during generation. * Semantic style separation: The model utilizes an unsupervised segmentation architecture to produce masks in order to localize the style only for specific subjects of the images and to remove useless areas like backgrounds or out-of-domain details.