DSI2I: Dense Style for Unpaired Exemplar-based Image-to- Image Translation
Authors: Baran Ozaydin
Tong Zhang
Sabine Susstrunk
Mathieu Salzmann
Venue: Accepted by TMLR ()

Abstract: Unpaired exemplar-based image-to-image (UEI2I) translation aims to translate a source
image to a target image domain with the style of a target image exemplar, without ground-
truth input-translation pairs. Existing UEI2I methods represent style using one vector per
image or rely on semantic supervision to define one style vector per object. Here, in contrast,
we propose to represent style as a dense feature map, allowing for a finer-grained transfer
to the source image without requiring any external semantic information. We then rely on
perceptual and adversarial losses to disentangle our dense style and content representations.
To stylize the source content with the exemplar style, we extract unsupervised cross-domain
semantic correspondences and warp the exemplar style to the source content. We demon-
strate the effectiveness of our method on four datasets using standard metrics together with
a localized style metric we propose, which measures style similarity in a class-wise man-
ner. Our results show that the translations produced by our approach are more diverse,
preserve the source content better, and are closer to the exemplars when compared to the
state-of-the-art methods.

Published in Transactions on Machine Learning Research (04/2024)
DSI2I: Dense Style for Unpaired Exemplar-based Image-to- Image Translation
Baran Ozaydin
wwnnoozaydinneel..cc
School of Computer and Communication Sciences, EPFL, Switzerland
Tong Zhang
tong.zhang
School of Computer and Communicattion Sciences, EPFL, Swiitzerland
Sabine Süsstrunk
sabine.sustruukkeepff...c
School of Computer and Communication Sciences, EPFL, Switzerland
Sabine Süsstrunk
subine.susttrukkeephh
School of Computer and Commmnneeation Sciences, EPFL, Switzerland
Reviewed on OpenReview: /ttps://openreen net/ forun? id= JrJa5kdKA4
Abstract
Unpaired exemplar-based image-to-image (UEI2I) translation aims to translate source image to larget image domain with the style of target image exemplar, without ground- truth input-translation pairs. Existing UEI2T methods represent style using one vector per image or rely on semantic supervision to define one style vector per object. Here, in contrast, we propose to represent style as a dense feature map, allowing for a finer- bher-grained transfer to the source image without requiring any external semantic information. We then rely on perceptual and adversarial losses disentangle our dense style and content representations. To stylize the source content with the exemplar style, we extract unsupervised cross-domain semantic correspondences and warp the exemplar style to the source content. We demon- strate the effectiveness of our method on four datascts using standard metrics together with a localized style metric we propose, which measures style similarity in a class-wise man- ner. Our results show that the translations produced by our approach are more diverse, preserve the source content better, and are closer to the exemplars when compared to the statc-of-the-art methods. Project page: https://githuut com/ IVRL/dsi21
1 Introduction
Unpaired image-to-image (UI2I) translation aims to translate source image to target image domain by training deep network using images from the source and target domains without ground-truth input- translation pairs. In the exemplar-based scenario (UEI2I), an additional target image exemplar is provided as input so as to further guide the style translation. Ultimately, the resulting translation should 1) preserve the content/semant /semantics of the source image; 2) convincingly seem belong the target domain; and 3) adopt the specific style of the target exemplar image.
Some existing UEI2I strategies Huang et al. (2018) Lee et al. 2018 encode the style of the exemplar using global, image-level feature vector. While this has proven to be effective for relatively simple scenes, it leads to undesirable artifacts for complex, multi-object ones, illustrated in Fig. 1 where appearance information of the dominating semantic regions, such as sky, unnaturally bleeds into other semantic areas, such as the road, trees and buildings. Other UE121 methods Bhattacharjee et al. (2020); Jeong et al. (001]); Kim et al. (2022) Shen et al. (2019) address this by computing instancc- or class- wise style representations. However, they require knowledge of the scene semantics, e.g., segmentation masks or bounding boxes during training, which limits their applicability.

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Baseline
DS121
Exemplar
Figure 1: Global style vs dense style representations. The baseline method (MUNTT) Huang et al. (0018) represents the exemplar style with single feature vector image. As such, some appearance information from the exemplar bleeds inlo semaantcalll-inncoee regions, giving, for example, an unnalural bluish taint to the road and the buildings the second row, first image. By modeling style densely, our approach better respects the semantics when applying the style from the exemplar the source content. Our method also has finer-grained control over style. The color of the road and center line in the third row reflect the exemplar appearance accurately.
By contrast, we propose to model style densely. That is, we represent the style of an image with a feature tensor that has the same spatial resolution as the content one. The difficulty of having spatial information in style that style information can more easily pollute the content one, and vice versa. To prevent this and encourage the disentanglement of style and content, we utilize perceptual and adversarial losses, which encourages the model to preserve the source content and semantics.
A dense style representation alone is not beneficial for UEI21 as the spatial arrangement of each dense style component is only applicable for its own image. Hence, we propose a cross-domain semantic correspondence module spatially arrange/ warp the dense style of the target image the source content. To that end, we utilize the CLIP Radford al. (2021) vision backbone as feature extractor and establish correspondences between the features of the source and larget images using Optimal Transport Cuturi (2013): Liu el. al.) (2020).
As a consequence, and as shown in Fig. 1 our approach transfers the local style of the exemplar to the source content in a more natural manner than the global-style Lechniques. Yet, in contrast to Bhattacharjee et al. 20200g Jeong et al. (2021): Kim et al. (2022); Shen et al. (2019), we do not require semantic supervision during training, thanks our dense modeling of style. To quantitatively evaluate the benefits of our approach, we introduce metric that better reflects the stylistic similarity between the translations and the exemplars than the image-level metrics used in the literature such FTD Heusel al. (2017), TS Salimans
et al. 2016), and CIS Huang et al. (2018).
Our contributions can be summarized as follows:
We propose a dense style representation for UE121, Our method retains the source content in the translation while providing finer-grain stylistic control.
We show that adversarial and perceptual losses encourage the disentanglement of our dense style and content representations.
We develop a cross-domain semantic correspondence module to warp the exemplar style to the source content.
We propose a localized style metric measure the stylistic accuracy of the translation.
Our experiments show both qualitatively and quantitatively the benefits of our method over global, image- level style representations. We will make code publicly available upon acceptance.
2

---

Published in Transactions on Machine Learning Research (04/2024)
Label Free
Multi -modal
Exemplar Local guided
Method
Unpaired
style
MUNIT
Huang et al. 2018)
X
DRIT
Lee et al. 2018
x
CUT
Park et al. 2020
X
X
FSeSim Zheng et
al.
22021
X
X
INIT Shen ct al
(2019)
X
J
V
DUNIT Bacharecerjee et al.
2020
A
X
MGUIT Jeong et al. (2021)
EX
CoCosNet Zhang al. (2020)
MCLNet. Zhan et al. (022b
MATEBIT Jiang
2023
DSI21
Table 1: Comparison of I2I methods. Unpaired methods do not require ground-truth translation pairs. Label free methods do not require object or segmentation annotations. Multimodal methods can produce multiple translations for one content. Excmplar guided methods can stylize the translations based on an exemplar image. The methods that represent style object-wise or densely have local style control.
2 Related Work
Our method primarily relates to three lines of research: saage-to-image (121) translation, Style Transfer, and Semantic Correspondence. Our main source of inspiration is I2I rescarch it deals with content preservation and domain fidelity. However, we borrow concepts from Style Transfer when it comes to adopting exemplar style and evaluating stylistic accuracy. Furthermore, our approach to swapping styles across semantically relevant parts of different images is related to semantic correspondences.
2.1
Image-to-image Translation
We focus the discussion of I2I methods on the unpaired scenario, as our method does not utilize paired data. CycleGAN Zhu al. (2017) was the first work to address this by utilizing cycle consistency. Recent works Hu ct al. 2022 Jung et al. (2022): Park et al. 2020 Zheng et al. (2021) lift the cycle consistency requirement and perform one-sided translation using contrastive losses and/o or self-similarity between the source and the translation. Many 12I methods, however, are unimodal, in that they produce single translation per input image, thus not reflecting the diversity of the target domain, especially in the presence of high within-domain variance. Although some works Jung et al. (2022): Zheng et al. (2021) extend this to multimodal outputs, they cannot adopt the style of a specific larget exemplar, which is what we address.
Some effort has nonetheless been made to develop exemplar-guider 121 methods. For example, Huang et al. (2018) Lee et al. (2018) decompose the images into content and style components, and generate exemplar- based translations by merging the exemplar style with the content of the source image. However, these models define a single style representation for the whole image, which does not reflect the complexity of multi-object scenes. By contrast. Bhattacharjeo et al. [2020); Jeong et al. (0021) ; Kim et al. (2022) : Mo et al. (2018) 0 Shen
et al. 2019 reason about object instances for 12I translation. Their goal is thus similar to ours, but their style representations focus on foreground objects only, and they require object-level (pseudo) annotations during training. Moreover, these methods do not report how stylistically close their translations are to the exemplars. Here, achieve dense style transfer for more categories without requiring annotations and show that our method generates translations closer to the exemplar style while having comparable domain fidelity with that of the state-of-the-art methods.
2.2 Style Transfer
Style transfer aims to bring the appearance of a content image closer a target image. The seminal work of Gatys et al. Gatys al. (2016) achieves by matching the Gram matrices of the two images via image-
3

---

Published in Transactions on Machine Learning Research (04/2024)
based optimization. Li et al. (2017b) provides an analytical solution to Gram matrix alignment, enabling arbitrary style transfer without image based optimization. [Huang & Belongie (2017) only matches the diagonal of the Gram matrices by adjusting the channel means and standard deviations. Li et al. (2017a) shows that matching the Gram matrices minimizes the Maximum Mean Discrepancy between the two feature distribut Inspired by this ributiut interpretation, Kolkin et al. (2019) proposes to minimize a relaxed Earth Movers Distance between the two distributions, showing the effectiveness of Optimal Transport style transfer. Zhang et al. (2019) defines multiple styles per image via GrabCut and exchanges styles between the local regions in two images. Kolkin et al. (211)) Zhang et al. (2019) are particularly relevant to our work as they account for the spatial aspect of style. Chiu & Gurari [022]; Li et al. (2018); Yoo et al. (2019) aim to achieve photorealistic stylization using a pre-trained VGG based autoencoder. Kim et al. (2020); Liu
et al. (2021); Yang et. al.] (2022) model texture- and geometry-based style separately and learn to warp the texture- based style to the geometry of another image. However, the geometric warping module they rely on makes their methods only applicable to images depicting single objects. Our dense style representation and our evaluation metric are inspired by this research on style transfer. Unlike these works, our Image-lo-imas translation method operates complex scenes, deals with domain transfer and does not require image based optimization.
2.3 Semantic Correspondence
Semantic correspondence methods aim to find semantically related regions across two different images. This involves the challenging task matching object parts and fine-grained keypoints. Early approaches [Barnes eet al. (2009); ; Liu et al. (2010) used hand-crafted features. These features, however, are not invariant changes in illumination, appearance, and other low-level factors that do not affect semantics. Hence, they
have limited ability to generalize across different scenes. Aberman et al. 2018); Liu et al. (2020); Min
et al. (2019) use ImageNNtt Simonyan & Zisserman (2044) pre-trained features to address this issue and find correspondences between images containing similar objects. However, these methods do not generalize to finding accurate correspondences across images from different modalities/domaan domains.
Semantic correspondences have been explored in the context of image to image translation as well. Tn particular, Zhan et al.] (2021 2022b); Zhang et al. (2020); Zhou et al. (202I); Zhan et al. (2022a) use cross- domain correspondencces to guide paired exemplar-based 12I translation. These methods are applicable to single dataset where the two paired domains consist of segmentation labels and corresponding images, Specifically, they aim to translate segmentation labels to real images. In this case, both the I2I and semantic correspondence tasks benefit from the paired data, i.e., semantic supervision. We also use cross-domain correspondences, but unlike these works, our method is 1) unpaired and unsupervised, i.e., the ground-truth translation is unknown; 2) unsupervised in terms of semantics, i.e., we do not use segmentation labels during training: 3) applicable translation between two datasets from different domains.
3 Method
Let us now introduce our UEI2T approach using dense style representations. To this end, we first define the main architectural components of our model. It largely follows the architecture of [Hang et al. (2018) and model consists of two style encoders EXx, E R3 RS&HW, two content encoders Fx, Fy 33" + RCCHH, two generators Gx
is depicted in Fig 2 Given two image domains X, Y C 33
Gy RCXHW RS#WW R3xH''' and two patch discriminators Dx, Dy B3::'' W
The content and style representations are then defined as follows. The content of image is computed C, := (x), and its dense style as Sdense := (x). Note that the latter departs from the definition of style in Huang et al. (2018); here, instead of a global style vector, we use a dense style map with spatial dimensions, which will let us transfer style in a finer -reingg manner. Nevertheless, we also compute a global style for image x as Sglobal == Aug(Sgense), where Aug denotes spatial averaging and repeating a vector across spatial dimensions. Furthermore, we define mixed style smir == 0.5Sgiobal + 0.55 As will be shown later, this mixed style will allow to preserve the content without sacrificing stylistic control.
4

---

Published in Transactions on Machine Learning Research (04/2024)
y
Exemplar image
Cy
Content of
y
Sgense
Sy
Syense
Syntx
s
golobal
Training
Source image x
Avg
Synix
Lads
Leato
Cx
Content
Sglobal
Cy
Gy
Lpere
Cz
Gy
E
Sy
Sdense
Style o
Lrccon
Lads
Sy Sdense
Flatten
Syense
x
Matrix multiplication
By
SxHxW
Optimal Transport
SxHW
Reshape
y
Flatten
CLIP
Fy
Testing
Frozen
FxHxW
S
Shared
SxHxW
CLIP
Flatten
Fx
Awwxww
FxHxW
x
Ex
Cx
Gy
CxHxW
Figure 2: Overview of method. We represent style feature map with spatial dimensions and constrain it via adversarial and perceptual losses for disentanglement. Our method does not require any labels or paired images during training. In test time, we warp the style of the exemplar for the source content using semantic correspondence. test time, we utilize the CLIP Radford et al. (2021) vision backbone to build semantic correspondences. See Section for definitions and explanations.
In the remainder of this section, we first introduce our approach learning meaningful dense style repre- sentations during training, shown in the top portion of Fig. 2 We then discuss how dense style is injected architecturally, and finally how to exchange the dense styles of the source and exemplar images at inference time, illustrated in the bottom portion of Fig. 2
3.1 Learning Dense Style
We define style as low level attributes that not affect the semantics of the image. These low level attributes can include lighting, color, appcarance, and texture. We also believe that change in style should not lead to unrealistic image and should not modify the semantics of the scene. In this work, we argue that, based this definition, style should be 1) represented densely to reflect finer grained stylistic attributes (stylistic accuracy); 2) constrained by adversarial loss to encourage fidelity to the target domain (domain fidelity); 3) constrained by perceptual loss to preserve semantics (content preservation).
To learn a dense style representation that accurately reflects the stylistic attributes of the exemplar, we Sdense to enable the flow of fine-grained dense information into the utilize the L1 reconstruction loss with Sy style representation. This is expressed as
Lrecon Ln (C,, Sdense),
(1)
Such an image reconstruction loss encourages the content and dense style representation to contain all the information in the input image. However, on its own, it does prevent style from modeling content and leading unrealistic semantic changes when edited. To encourage rich content. representation that
5

---

Published in Transactions on Machine Learning Research (04/2024)
preserves semantics, use adversarial and perceptual losses with random style vector Wo Sx1
Lado Wadv_randone random = LG,,
(2)
Leee_random random = V(Vr)),
(3)
These losses prevent our model from relying too much on dense style for reconstruction and translation by leading a richer content representation.
Having rich content does not prevent dense style from polluting it. To prevent style from modeling content, constrain the dense style using the adversarial and perceptual losses
Lady global LGAN(Gr(CC, ( Sglabal), ), Dr)
(4)
I Lpper global (x(V(x), V(r (Cz. S Sglobal)))
where LGAN denotes a standard adversarial loss, and V represents the VGG16 backbone up to but excluding the Global Average Pooling layer. The adversarial loss above encourages the fidelity of the translations to the target domain Goodlellow et al. (2014); Zhu et al. (2017) whereas the perceptual losses help preserve the semantics Huang et al. (2018): Johnson et al. (2016); Zhu et al.) (2017). While the global losses in Eqs. constrain dense style via the spatial averaging operation, there is no loss that involves Sdense, Involving Sdense in the adversarial and perceptual losses tends to make the model learn to ignore the style representation, referred to as style collapse. Also, note that all the constraints Eqs. 2 constant style representation. Hence, to involve a spatially varying style during the training and to avoid style collapse, we introduce two losses computed the mixed style Sy given by
use spatially
Lude = LGAN(GY Sp Smir), Dy), Iper miz Li(V(y), V(Y (Cy, )))
(6)
(7)
3.2 Injecting Dense Style
Let us now describe how we inject a dense style map, Sceense in our framework to produce an image. Accurate stylization requires the removal of the existing style as an initial step Li et al. (2017h a). Thus, for our dense style to be effective, we incorporate a dense normalization that first removes the style of each region. To this end, inspired by Li et al. (2019); ; Park et al. (2019) Zhu et al. (2020), we utilize Positional Normalization Layer Li et al. 2019) followed by dense modulation. These operations are performed on the generator activations that produce the images.
Formally, let. P E HW denote the generator activations, with C' the number of channels. We compute the position-wing wise means and standard deviations P, μ, a E HW We then replace the existing style by our dense one via the Dense Normalization (DNorm) function
P P-
FDN
- 3+a.
(8)
where the arithmetic operations are performed in an element wise manner and by replicating and a C" times to match the channel dimension of P. The tensors a, 8 E R HW are obtained by applying x convolutions to the dense style gdense.
Up to now, we have discussed how to inject dense style an image and how to learn meaningful dense style representation in the training stage. However, one problem remains unaddressed in the test stage: The dense style extracted from an image is only applicable to that same image because its spatial arrangement corresponds to that image. In this section, we therefore propose approach to swapping dense style maps across two images from different domains.
Our approach is motivated by the intuition that style should be exchanged between semantically similar regions both images. To achieve this, we leverage an auxiliary pre-trained network that generalizes well across various image modalities Radford et al. (2021). Specifically, we extract middle layer features F,, F, FxHW by passing the source and exemplar images through the CLIP-RN50 backbone Radford

---

Published in Transactions on Machine Learning Research (04/2024)
et al. 2021)) J. We then compute the cosine similarity between these features, clipping the negative similarity values to zero. We denote this matrix as Zyz E RHWWHW and use it to solve an optimal transport problem as described in Liu et al. (2000] : Zhang et al. (2020). We construct our cost matrix C as
C 1 yy with
(9)
y max(cos O).
(10)
We then use Sinkhorn's algorithm Cuturi 20133 to compute a doubly stochastic optimal transportation matrix yo E HRWW which corresponds to solving
A pur arg min C)r X1(A)
(11)
8.t A1 = Py A A"1 = Pz
(12)
where h(A) denotes the entropy of A and is the entropy regularization parameter. Pas Py R constrain the and column sums of Aya: which are chosen uniform distributions (sce the supplementary material for other choices). Optimal Transport returns transportation plan Ayr to warp Sy
Sdense
Sy
Sgenee Agr
(13)
so that Sy-ra is semantically aligned with x instead of with y. This plan transports style across semantically similar regions with the constraint that each region receives an equal mass. With this operation, each spatial clement Sy-z(h, w| can be soen as a weighted sum of spatial elements of Sdense [h', w'] with the weights being proportional to the semantic similarity between Fh. and FMW' Hence, we can trade the style across semantically similar regions.
Our semantic correspondence module can also be thought of as a cross attention mechanism across two images with the queries being F,, the keys Fy and the values S Syense, Note also that global style transfer, as done in MUNIT Huang et al. (2018). is actually a special case of this formalism where Ayz is a constant uniform matrix.
3.3 Discussion on Losses and Components
Discussion on the model components. The semantic correspondence matrices Ayz built from CLIP Rad- ford et al. (2021) features are 1) expensive in terms of computation and memory, and 2) noisy as cach point corresponds to the others with some non-negative weight. For example, a self-correspondence ma- trix Aez (HWxHW) computed with CLIP Radford et al. (2021) would have large diagonal entries, positive but smaller off- entries for related semantic pixel pairs, and ideally zero off-diagonal entries for semantically unrelated pixel pairs. Instead of computing and storing these noisy and costly matricos with CLIP Radford et al. (2021) during training, we provide the losses with Smir and Salb
Our intuition for Smiz and S9lh is that these two style components replace noisy correspondence matrices of CLIP Radford et al. (2021) during training. Sath is used to imitate coosccorrespondences Ayz and can be seen as the output of a uniform, constant HWxHW correspondence matrix of 1/HW (each content pixel corresponding to all the exemplar pixels equally) as shown in 3 parts a) and d); Sdense can be seen as the output of an identity self-correspondence matrix (each pixel corresponding only to itself) as shown in 3 parts b); and Smiz is the output a noisy correspecreeeeeccee matrix, imitating Azz: with large diagonal entries and uniform non-diagonal entries (each pixel corresponds mainly to itself but also to all the others) as shown in 3 part c).
Additionally, randomly sampled style codes simulate and enable our model to generalize to the cases where no style information (other than the random style vector) is available. This intuition is linked to the previous works on VAE Kingma & Welling 2013) and
correspondence matrix, as shown in 3 part e).
utilized for image translation in Liu et al. (017); Huang et al. (0118)
Finally, these style components and analytical correspondence matrices enable our model to generalize to the HWxHW crossccoreepooddence matrices of CLIP Radford et al. (202)), without needing to use CLIP Radford et al. 2022)) during training.
7

---

Published in Transactions on Machine Learning Research (04/2024)
a) Agx simulated by Sath used for Smis
b) Azz simulated by Sdense used for Simke and Eq.
c) Azz simulated by Smiz used for Eqs.
d) Azy simulated by Sath used for Eqs.
A simulated by Srond used for Eqs.
f) Any created by CLIP used during test time
Figure 3: Style components and correspondence matrices. Example for the simulated and created correspondence matrices Aga, Azy E [0, 13x3, Top row includes the self-corresoondence Azz between three pixels from an image in the purple domain, whereas the bottom row displays cross-domain correspondence Ary between an image from the purple domain and another image from the yellow domain. Using a)-e) during training enables our model generalize to during test time.
Discussion on the adversarial and perceptual losses. Adversarial losses in our framework are mainly intended to produce translations that to preserve the content and semantics.
have high domain fidelity, whereas the perceptual losses are intended
4 Experiments
4.1 Evaluation Metrics
In this UF12T work, we have three goals and we evaluate these three goals with different metrics. To evaluate stylistic accuracy, we propose novel metric assess classwise stylistic distance that takes semantic information into account. To evaluate domain fidelily and how well the translations seem belong to the target domain, report the standard FID Heusel et al. (2017) between the translations and the targets. Lastly, to evaluate content preservation, report segmentation accuracy with segmentation model, DRN Yu et al. (2017), trained on the target domain and tested on the translations.
4.2 Classwise Stylistic Distance
Our local style metric, Classwise Stylistic Distance (CSD), computes the stylistic distance between the corresponding semantic classes in two images. We use VGG until its first pooling layer, denoted as V. to extract features of size VVHW from the input image x, exemplar y, and translation - y. Our metric uses binary segmentation masks M, KW to compute the style similarity across corresponding classes. Using the mask for class k. M IR HR we compute the Gram matrix Q of the VGG features for class

---

Published in Transactions on Machine Learning Research (04/2024)
Baseline
Ours
Exemplar
Figure 4: Effect of the exemplar. Our method can change the appearance of each semantic region differently, yet has realistic output. The colors of the road and car in the translations match the exemplar road and car styles better than the baseline (MUNIT) Huang al. (2018) does. Content image can be seen in Figure
k in image
Q k
(V(x) (V(x) o M
(14)
M.M.
This operation equivalent treating each class as separate image and computing their Gram matrices. We then compute the distance between the Gram matrices of corresponding classes in two images, i.e.,
L(x,y,k) 1 Q.1
(15)
Note that L(x, y, k) denotes the Maximum Mean Discrepancy (MMD) between the features of the masked regions with a degree polynomial kernel Li et al. (2017a). Since this distance is computed based on VGG features from early layer, it implies stylistic distance between the two images Gatys et al. (2016).
However, L(x, k) is not very informative its scale is arbitrary and depends the stylistic distance between the input image pair x,y. Hence, we propose a metric that takes x, y, and x + at the same time for better interpretability. We express Classwise Stylistic Distance (CSD) as
H(x,y, = L(x + vyy,kk) L(x,y,k) (M 1 (M;>0)
(16)
where 10 is the indicator function and M E, M
Unlike L, H is more interpretable because its value would be equal to one if the translation outputs the content image. In an ideal translation scenario, we would expect the feature distributions of the translation and exemplar to be close to each other Kolkin et al. 2019 Hence, we expect small values for more successful translations.
Note that Zhang et al. (2200) also proposes a metric assess classwisc stylistic similarity. Instead of the L2 distance between the classwise Gram matrices, it computes the cosine distance between the average features of corresponding regions in the exemplars and the translations. However, exemplar guided translation involves
9

---

Published in Transactions on Machine Learning Research (04/2024)
three images; the source, the exemplar, and the translation. We believe that evaluating UE121 should take all three images into account. because stylistic distance between the source and exemplar affects the stylistic distance between the translation and exemplar, i.e. they are positively correlated. Our metric normalizes the stylistic distance between the exemplar and translation by the stylistic distance between the source and exemplar. By doing s0, we obtain an interpretable value that shows which portion of the stylistic gap is closed for each translation, regardless of the initial style gap.
4.3 Implementation Details
We evaluate our method on real-to-synthotic and synthetic-to-rcal translations using the GTA Richter et al. (2016), Cityscapes Cordts et al. (2016), and KITTI Geiger el al. (2012) datasets. We use the code published
by the baseline works Huang et al. (2018); Jeong et al. (2021): |5 Lee et al. (2018): Park et al. (2020): Zheng
let al. (2021). Images are resized to have a short side of 256. We borrow the hyperparameters from Huang et al. (2018) but scale the adversarial losses by half since our method receives gradients from three ad advessarial losses for one source image. We do not change the hyperparameters for the perceptual losses. The entropy regularization term in Sinkhorn's algorithm in Eq. [2] is set to 0.05. During training, we crop the center 224x224 pixels of the images. During test time, we report single scale evaluation with the same resolution for all the metrics. We use a pre-trained DRNYu et al. (2017) to report the segmentation results.
We also evaluate our method on real-to-rcal translation using the sunny and night splits of the INIT Shen et al. (2019) dataset. We use the same setup as previous works and the results of the baselines are taken from the respective papers.
4.4 Results
buil-
side-
buil
side
GTT
Avg
KUTTI
sky
tation
diug
walks
tation
diug
walk
MUNIT
Huang es al
2018
0.43
0.78
0.21
0.28
0.13
0.06
0.32
MUNIT Heans et al.
2018
0.46
0.17
0.59
0.39
0.64
0.53
0.46
DRIT Lee
0.41
1.21
0.27
0.27
0.12
0.08
0.38
DRIT Le
0.52
0.22
0.81
0:44
0.65
0.55
0.53
CUT
0.92
0.24
0.36
0.15
0.38
20p0r
0.53
0.21
0.63
0.76
FS-Sin Zaroo el d. 3
0.60
0.96
0.25
0.38
0.15
0.13
0.38
FSehan Zheng al 2121
0.50
0.25
0.76
0.19
0.58
0.88
0.61
MGUT pronig
0.45
1.42
0.29
0.18
0.49
MGUI Teang
0.40
0.74
0.84
0.55
DS121
0.29
0.22
0.16
0.26
0.08
0.03
0.17
DS121
0.29
0.08
0.42
0.34
0.59
0.23
0.32
Table 2: Stylistic Accuracy. Classwise Stylistic Distance between translation-cxemplar pairs. Our trans- lations mathh the classwise style of the exemplars better (lower is better).
Stylistic Accuracy. Firstly, we evaluate the stylistic distance between the exemplars and the translations using our metric CSD. We report this metric for the most frequent six classes of GTA Richter et al. (2016) and Cityscapes Cordts et al. (2016). The trend with other classes is similar and can be seen in our supplementary material. As shown in Table 2 our method outperforms the baselines in the synthetic-to-rea and real-to- synthetic scenarios. Note that in the synthetic domains, stylistic diversity is overall higher because the images are more saturated. The results for translations in the opposite directions can be seen in our supplementary material. Our dense style and semantic correspondence modules bring style of corresponding classes closer each other.
GTA
KITTI GTA
sumny
night
night
sunn
Method
FID
Seg Ace 1
FID
Seg Ace
Method
CIS
18 t
CIS
IS
MUNIT
Huang
al.
2018
47.76
0.79
53.48
0.73
MUNIT
Huang al.
2018
1.159
1.278
1.036
1.051
DRIT
Loe et al
2018
42.93
0.70
52.12
0.62
DRIT
Loe et RL
2013
1.058
1.224
1.024
1.099
CUT
Park et al
2120
49.82
0.65
62,30
0.59
INIT
Shen et al
2019
1.060
1.118
1.015
1.080
FS4Smm
Zhong el al. 2021
48.77
0.71
63.04
0.60
DUNIT
Blaftacia jee ot
2020
1.166
1.259
1.083
1.108
MGUIT
Joong
44.36
57.00
0.57
MGUIT
Jeong
1.176
1.271
1.115
1.130
DS121
42.61
0.82
48.30
0.75
DS121
1.204
1.283
1.138
1.149
Table 3: Content preservation and domain fi- Table 4: Diversity. The translations produced by delity. Our method generates translations with high our method have higher diversity than those of the fidelity and preserves the content.
baselines.
10

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Exemplar
Ours
MUNIT
DRIT
CUT
LSeSimi
MGUIT
Figure 5: Qualitative comparison with other methods. CS - GTA translations. In the first columi, our method disentangles the road from the sky and preserves the dark color for the road. In the second column, the appearance of the road and roadlines in translation closest to those in the exemplar. In the last two columns, our model preserves the semantics better, especially for tree and building classes
Domain Fidelity, We then evaluate the domain fidelity of the translations using FID Heusel et al. 2017 in Table 3 Our method generates translations with high fidelity in the synthetic-to-real and real-to-synthetic scenarios, which pose large domain gaps.
Content preservation. We also evaluate how well our model preserves the content via segmentation accuracy Table Our method preserves content better than other 121 methods.
Diversity. Although our main goal is not diversity but stylistic accuracy, having a finer-grained dense style representation brings about diversity by product. We evaluate the diversity and quality of
11

---

Published in Transactions on Machine Learning Research (04/2024)
translations using the IS Salimans et al 2016) and CIS Huang et al. (2018) metrics in real-to-real translation in Table 4 Our results are better than those reported in the baseline papers. Even though the baselines (020 Jeong et al. (002)) Shen et al. (2019) use object detection labels during training to guide style, we outperform them without using labels. Note that we do not use semantic correspondences, T.e. CLIP Radford el al. (2021), during training either. Hence, the performance increase is not due to
Bhattacharjee et al.
dense somantic correspondences or the use of CLIP Radford et al. (2021) during training. Our dense style
representation leads to greater stylistic control and diversity.
Method
Test time label
FID +
Styl. Dist.
Method
FID
Styl. Dist.
CoCosNetv2
GT Label
46.32
0.34
CoCusNetv2 Zhon et al 2021
51.32
0.37
MCLNet
Zhan et ml. 20226
50.42
0.38
CoCosNetv2
Pred Label
51.32
0.37
MATEBIT
Jannn al. 2025
49.25
0.36
DSI2I
No Label
45.12
0.32
DS121
45.12
0.32
Table 5: Quantitative Comparison with CoCos- Netv2 Zhou et al. (2021). Our method (without train-test time labels) outperform CoCosNetv2 (with train-test time labels). When the ground truth labels are replaced with the prodicted labels (%95 accurate) 2 performance drops drastically.
Table 6: Quantitative Comparison with Seman- tic Image Synthesis Methods Our method out-
performs the image synthesis baselines that use pre- dicted labels.
Comparison to exemplar guided semantic image synthesis. Several works use semantic correspon- dence in 121 Zhan et al. [2021] 2022b;; Zhang al. (2020); Zhou et al. (2021); Zhan et al. (2022a); Jiang ct al. 2023) to synthosize an image based on a given exemplar. As mentioned in Table our method differs from this line of research in terms of training resources in three ways; 1) we do not require any semantic labels during training (Tabel Free), 2) our image translation task is not guided by ground truth translations during training (Unpaired), and additionally, 3) our method does not rely on highly similar exemplar-targe -taget pairs within the same domain.
To demonstrate the effectiveness of the unsupervised aspect of our method, we provide comparisons with the exemplar based image synthesis works. To that end, train CoCosNetv2 Zhou et al. (2021) the GTA dataset using the GTA labels. We test them with GTA images as the exemplars by giving 1) ground-truth labels of a CS image, 2) segmentation predictions of CS image (%95 accurate) as inputs. Our method outperforms CoCosnetv2 Zhou et al. (2021) that use labels both during training and test time. Our mothod also outperformn more recent works Zhan et al.] (2022b); Jiang et al. (2023) even though we do not use any labels or pretrained segmentation models neither during training nor during test time as seen in Table [
User study. We conduct user study Amazon Mechanical Turk and ask the users which translation is closer to the exemplar in lerms of classwise style, color and appearance. We show the users one larget image, and translations (CS + GTA) from the six methods in Fig. 5 Out of 3003 votes, our method received the most votes (1062), see Table 7 MUNIT Huang et al. (2018) is the second best model with 860 votes. Our method brings the style of semantically relevant regions closer to each other and is preferred by humans.
4.5 Ablation Study
Our ablations in Table show that the losses on Smir and Solh encourage our model to preserve content and generate high-quality translations. The elfects of adversarial and perceptual losses are shown in Table g
Additional analysis the model components can be found in the Appendix in Tables I2 14
Tables and 0 include ablations on GTA -> CS for our losses and style components (w/o Sath is equivalent to w/o Lodt Leper__gtbi w// Smir is equivalent to w/o Leate_mis miz, Lipere_mis) mir). The advensarial losses are mainly helpful for domain fidelity (Table g FID column). The perceptual losses are mainly beneficial for content preservation (Table Seg. Acc. column). Smiz and Sulb provide analytical noisy correspondences during training time and lead better FID and Seg. Acc. in Table & during test time, when CLIP
correspondences used with OT. Altogether, our ablations in Tables g 13 14 show that the adversarial
12

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Ours
CoCosNetv2
Exemplar
Figure 6: Qualitative results from Table 5 CS - GTA. CoCosNetv2 fails when Source and Exemplar
images are from different domains and have uncommon classes. The human in the 2nd row, the car in the 1st and 3rd rows and the buildings in all rows are preserved better with our method. Our translations are more realistic and better represent the source content.
Method
DSI21
MUNIT
DRIT
CUT
FSeSim
MGUIT
Ratio
35%
28%
18%
5%
7%
5%
Table 7: User study on similarity of translations with exemplars.
and perceptual losses on Salb and Smiz are useful in terms domain fidelity (FID), content preservation (Seg. Acc.), and stylistic accuracy (Styl. Dis.).
5 Limitations
The main advantage of our method compared to the basclines the dense modeling of style. Hence, our method loses its advantage for simple scenes with fewer objects where dense style is not necessary.
13

---

Published in Transactions on Machine Learning Research (04/2024)
GTA + CS
FID
Seg Acc 1
GTA CS
FID 4
Seg Ace f
DS121
42.61
0.82
DSI21
42.61
0.82
DSI2I w/o Sulb
43.52
0.80
DSI2I w/o Ladex
48.30
0.82
DSI21 w/o Swai
45.61
0.78
DS121 w/o Liperes
42.96
0.73
DSI2I w w/o Smar Sals
50.63
0.72
DS121 w/o Ladvx: Lperes
50.63
0.72
Table 8:
Ablation study Smir and Sutb.
Our Table 9:
Ablation study on adversarial and perceptual losses with S and Salb, Adversarial loss encourages domain fidelity whereas perceptual loss helps preserve the content.
method benefits from both.
6 Conclusion
We present a framework for UEI2I that densely represents style and show how such a dense style repre- sentation can learned and exchanged across images. This formalism allows local stylistic changes across semantic regions, while not requiring any labels. We demonstrate the effectiveness of our dense style repre- sentation in the synthetic-to-real, real-to-synthetic and real-to-real scenarios by showing that translations match the style of the exemplar better, more diverse, better preserve the content, and have high fidelity.
Acknowledgements. This work was supported by the Swiss National Science Foundation via the Sinergia grant CRS115-180399. We also thank Ehsan Pajouheshgar for valuable discussions and contributions.
14

---

Published in Transactions on Machine Learning Research (04/2024)
References
Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. Neural best- buddies: Sparse cross-domain correspondence. ACM Transactions on Graphics (TOG), 37(4):1 14, 2018.
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized corrospondence algorithm for structural image editing. ACM Trans. Graph., 28(3):24, 2009.
Deblina Bhattacharjoe, Seungryong Kim, Guillaume Vizier, and Mathieu Salzmann. Dunit: Detection-based
unsupervised image-to-image translation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4787-4796, 2020.
Xinyuan Chen, Chang Xu, Xiaokang Yang, and Dacheng Tao. Attention-gan for object transfiguration in wild images. In Proceedings of the European conference on computer vision (ECCV), pp. 164-180, 2018.
Tai- Chiu and Danna Gurari. Photowet2: Compact auloencoder for photorealistic style transfer resulting from blockwise training and skip connections of nighfrreuency residuals. In Proceedings of the IEEE/CVP Winter Conference on Applications of Computer Vision, pp.
2022.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson,
Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understand-
ing. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Marco Cuturi. Sinkhorn distances: Lightspoed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2414- 2423, 2016.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition, 2012.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint 1r0i::1406.6601 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural infor- mation processing systems, 30, 2017.
Judy Hoffman, Eric Tzeng, Tacsung Park, Yun Zhu, Phillip Isola, Kate Sacnko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference on Machine Learning, pp. 1989-1998. PMLR, 2018
Xueqi Hu, Xinyue Zhou, Qiusheng Huang, Zhengyi Shi, Li Sun, and Qingli Li. Qs-attn: Query-sclected atlention for cont rastive learning in i2i translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18291-18300 2022.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEFE International Conference Computer Vision pp. 1501-1510, 2017.
Xun Huang, Ming-Vi Yu Täu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image trans- lation. In Proceedings of the European Conference on Computer Vision, pp. 172 189, 2018
Somi Jeong, Youngjung Kim, Fungbean Lee, and Kwanghoon Sohn. Memory-guided unsupervised image- to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6558 6567, 2021.
Chang Jiang, Fei Gao, Biao Ma, Yuhao Lin, Nannan Wang, and Gang Xu. Masked and adaptive transformer for exemplar based image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 22418 22427, 2023.
15

---

Published in Transactions on Machine Learning Research (04/2024)
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Proceedingg of the Europenn Conference on Compuler Vision, pp. 694-711. Springer, 2016.
Chanyong Jung, Gihyun Kwon, and Jong Chul Ye. Exploring match-wise semantic relation for contrastive learning in inage-to-image translation tasks. Tn Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognation, pp. 18260-18269, 2022.
Soohyun Kim, Jongbeom Baek, Jihye Park, Gyeongnyeon Kim, and Seungryong Kim. Instaformer: Instance- aware image-to-image translation with transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18321-18331, 2022.
Sunnie SY Kim, Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. Deformable style transfer. In Proceedings of the European Conference on Computer Vision, pp. 246-261. Springer, 2020.
Diederik P Kingma and Max Welling. Auto-cncoding variational baycs. arXiv preprint arXiv:1312.6114 2013.
Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. Style transfer by relaxed optimal transport and self-similarity. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10051-10060, 2019.
Hsin- Ying Lee, Hung- Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming- Hsuan Yang. Diverse image- to-image translation via disentangled representations. Tn Proceedings of the European Conference on Computer Vision, pp. 35-51, 2018.
Boyi Li, Felix Wu, Kilian Q Weinberger, and Serge Belongie. Positional normalization. In Advances in Newral Information Processing Systems, pp. 1620-1632, 2019.
Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. Demystifying neural style transfer. arXiv preprint 1eXiv:1701.0111 2017a.
Yijun Lí, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Universal style transfer via feature transforms. Advances in Neural Information Processing Systems, 30, 2017b.
Yijun Li, Ming- Yu Liu, Xueting Li, Ming- Hssng Yang, and Jan Kautz. closed-form solution to photore- alistic image stylization. In Proceedings of the European Conference on Computer Vision, pp. 453-468, 2018.
Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its applica- tions. IEEE transactions on pattern analysis and machine intelligence, 33(5))978-999) 2010.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised taago-to-image translation networks. Advances in neural information processing systems, 30, 2017.
Xiao- Kiao-Chang Liu, Yong-Liang Yang, and Peter Hall. Learning to warp for style transfer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognétion, pp. 3702-31 8702-3711, 2021.
Yanbin Liu, Linchao Zhu, Makoto Yamada, and Yi Yang. Semantic correspondence as an optimal transport problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 1463-4472 2020.
Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Hyperpixel flow: Semantic correspondence with multi-layer neural features. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 3395-3404, 2019.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Instagan: Instancc-awarc image-to-image translation. arXiv preprint arxi 1812. 108888888 2018
Tacsung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- niltion, 2019.
16

---

Published in Transactions on Machine Learning Research (04/2024)
Taesung Park, Alexci A Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to- image translation. In Proceedings of the European Conference Computer Vision, pp. 119-345. Springer, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Proceedings of the European Conference on Computer Vision, volume 9906 of LNCS, pp. 102-118. Springer International Publishing, 2016.
Tim Salimans, Tan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, Advances in neural information processing systems, 29, 2016.
Zhiqiang Shen, Mingyang Huang, Jianping Shi, Xiangyang Xue, and Thomas S Huang. Towards instance- level image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognilion, pp. 3683 3692, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image rocognition. arXiv preprinl Xrxi 1556, 2014.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive lcarning for solf- supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3024-3033, 2021.
Jinchao Yang, Fei Guo, Shuo Chen, Jun Li, and Jian Yang. Industrial style transfer with large-scale geometric warping and content preservation. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pattern Recognition, pp. 7834 7843, 2022.
Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, and Jung-Woo Ha. Photorealistic style transfer via wavelet transforms. In Proceedings of the IEEE/CVP International Conference Computer Vision, pp.
2019.
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated rosidual networks. In Proceedings of the IEEE conference Compuler Vision and Pallern Recognilion, pp. 472 220,480 2017.
Fangneng Zhan, Yingchen Yu, Kaiwen Cui, Gongjic Zhang, Shijian Lu, Jianxiong Pan, Changgong Zhang, Feiying Ma, Xuansong Xie, and Chunyan Miao. Unbalanced feature transport for exemplar-based image translation. In Proceedings of the JEEE/CC /CF Conference on Computer Vision and Pattern Recognition, pp. 15028 15038, 2021.
Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Kaiwen Cui, Aoran Xiao, Shijian Lu, and Chunyan Miao. Bi-level feature aligmment for versatile image translation and manipulation. In Computer Dision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVI, pp. 224-241. Springer, 2022a.
Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Shijian Lu, and Changgong Zhang. Marginal contrastive correspondence for guided image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognétion, pp. 10663-10672 2022b.
Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. Cross-domain correspondence learning for exemplar-basod image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 5143 5153, 2020.
Yulun Zhang, Chen Fang, Yilin Wang, Zhaowen Wang, Zhe Lin, Yun Fu, and Jimei Yang. Multimodal style transfer via graph cuts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5943 5951, 2019.
17

---

Published in Transactions on Machine Learning Research (04/2024)
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. The spatially-correlative loss for various image translation tasks. In Proceedings of the IEEE/CVE Conference. on Compuler Vision and Pallern Recognation, pp. 2021.
Xingran Zhou, Bo Zhang, Ting Zhang, Pan Zhang, Jianmin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen. Cocosnet v2: Full-resolution correspondence learning for image translation. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 11465 11475, 2021.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexci A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference Compuler Vision, pp. 2223 2232,, 2017.
Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region- adaptive normalization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 5104 5113, 2020.
A Results on Unlabeled Datasets
We evaluated our method on datasets which have ground truth segmentation labels. The reason behind this is that, even though our method is applicable to datasets without labels, the metries care about (Seg. Acc. and Styl. Dist.) rely on ground-truth segmentation labels. KITTI, GTA and Cityscapes satisfy this label requirement for quantitative evaluation. Furthermore, some of the baselines, CoCosNetv2, MCLNet, MATEBIT and MGUIT, strietly rely on semantic segmentation labels for training, which makes them inapplicable to unlabeled datasets.
Our method, however, is applicable to scenes without semantic labels. Hence, as requested by the reviewers, we provide results the summer2winter and monet2photo datasets Zhhu et al. (2017); in both directions. Our qualitative results show that dense modeling of style enables more accurate transportation of style between semantically relevant regions.
Also, we would like to mention that, to our knowledge, our method is the first GAN-based T2T method to model style densely and exchange it accuratoly across semantic regions, in unlabeled datascts.
B Limitations
Our mothod is effective at preserving the content for a semantically distinct image pair from two semantically related datasets. The second row of Figure 5 is a good example, where our method preserves the content and styles of the pedestrian, bike and rider classes even though there are no such classes in the exemplar. Another example provided in monet2photo in Figure 6, where the yellow leaf stylizes the vegetation in the ground (a semantically relevant but distinct class) but the other semantic classes are less affected and retain their style. However, our method is not effective for translation between semantically distinct dataset pairs. For example, in horsc2zebra dataset, where the image translation requires semantic changes, our method often fails to add the stripes to the horses animals. In Figure S we show a cherry-picked result in the first row and another example that reflects the general performance of our method in the second row. The limitation might be partly due the perceptual loss with VGG, which too conscrvative for the horze2zebra task.
In our work, the attributes to be swapped are matched via CLIP-based correspondence whereas the attributes to be preserved are constrained via VGG-based perceptual loss. The choice of the VGG backbone reflects what kind of content we aim to preserve whereas the choice of the CLIP backbones reflects among which content
regions we aim to exchange the dense style. Hence, for the applications whose definitions differ from ours, a possible solution could be to experiment with other backbones instead of CLIP and VGG. An example for horse2zebra dataset would be to use a background focused perceptual loss instead of VGG-based perceptual loss and an animal-part focused correspondence backbone instead of the CLIP based correspondence. One reference for such a solution could be AttentionGAN Chen al. (2018).
18

---

Published in Transactions on Machine Learning Research (04/2024)
Photo
LSeSim
MUNIT
DSI21
Monet
Monet
LSeSim
MUNIT
DSI21
Photo
Winter
LSeSim
MUNIT
DSI2I
Summer
Winter
LSeSim
MUNIT
DSI2I
Summer
Winter
LSeSim
MUNIT
DSI21
Summer
Summer
LSeSim
MUNIT
DSI21
Winter
Figure 7: Applicability of our method. DSI2I is effective in challenging datasets that do not contain any semantic labels. Our method is the first to model the style densely in these datasets. In the first, third, fourth, fifth and sixth rows, sky in our translations reflect the exemplar style of sky more accurately. In the second row, the yellow leaf the exemplar stylizes the vegetation (grass) whereas sky less affected by the style of the yellow leaf.
19

---

Published in Transactions on Machine Learning Research (04/2024)
Horse
MUNIT
DS121
Exemplar
Horse
MUNIT
DSI21
Exemplar
Figure Limitation on I2I tasks that require semantic changes. On the horze2zebra dataset that requires semantic changes, our method fails to make the required changes. We present a cherry picked result the first row. The quality of the outputs of our method reflected more accurately in the second where the stripes are not added properly.
C Technical Details
In this section, we describe the technical details of the 12T methods that we used in our comparisons. For each method, we adopt the default hyperparameters of the method. All the models are trained for 800K iterations with 224x224 images. We use linear learning rate decay after 400K iterations as suggested in these works.
For training, we resize the input images to have the shorter side size 256 without changing the aspect ratio and then crop random 224x224 region. At. test time, we generate translations without. cropping the images. We use the evaluation code of FSeSim Zheng al. (2021) for computing the FID. We resize the images to have a shorter side of size 299 without changing the aspect ratio when computing the FID. Wo borrow the code for IS/ CIS from MUNIT Huang et al. (2018). We report the exponential of IS and CIS as done in Huang et al. (2018). The images are resized to have the shorter side of size 299 followed by taking a center crop size 299x299 as done in Huang et al. (2018). FID, IS and CIS computations, we sample 100 random source images and target images for each source image. We generate 1900 exemplar based translations as done in Huang
(2018).
We use two pre-trained DRN models Yu et al. (0117 for segmentation. We use the pre-trained models for GTA and CS from Hoffman et al. (2018) and Yu et al. 2017 respectively. The former is a DRN-C 26 model whereas the latter is DRN-D 22.
D Results
We provide additional results in this section on CS GTA. Our method outperforms the baselines in CS
GTA.
20

---

Published in Transactions on Machine Learning Research (04/2024)
buil-
side-
CS - GTA
car
sky
tation
road
Avg
ding
walk
MUNIT Huang dt 2018
0.57
0.44
0.59
0.53
0.47
2018
0.63
CUT
Park
2020
0.88
0.58
0.75
0.67
0.74
FSeSim
Zheng et Gl.
2021
0.77
0.68
0.75
0.71
MGUIT
Jooug et
2022
0.76
0.69
0.69
0.65
0.68
0.57
0.67
DSI21
0.35
0.17
0.44
0.41
0.50
0.29
0.36
Table 10: Classwise stylistic distance
CS GTA
FID 4
Seg Acc
MUNIT
Huang et al.
(2018)
48.91
0.79
DRIT
Lee et al. (2018)
48.18
0.72
CUT
Park et al.
20200]
65.68
0.61
FSeSim
Zheng et al.
[202]
64.81
0.74
MGUIT
Jeong
2021
55.72
0.68
DSI2I
45.12
0.81
Table 11: Fidelity and diversity of the translations. Our method outperforms all others on all metrics.
E Semantic Correspondence
E.1 Marginal Distributions in Optimal Transport
As mentioned in line 497 in the main paper, we discuss a better choice for the marginal distributions for Sinkhorn's Algorithm Cuturi (2013). The most straightforward choice for transportation masses Pa and Py is the uniform distribution. However, doing so transports equal mass from every location in the images. This is problematic for because can in Fig. α that when translation pairs have unbalanced classes, the largest semantic region can dominate the style representation and lead to undesired artifacts. In our example in Fig. [ the content image expects to receive style vectors for roads, buildings, and tree but the exemplar image provides style for sky and road. This results in building and tree regions being stylizes by sky attributes.
To solve the unbalanced class problem, we first assume that segmentation labels M, M, E {0, kx HW for K classes are available. We define M the binary mask for the k-th class. The number of pixels in class k is defined M = E, Mrk where I indexes the spatial dimension. We, then, define M, M Myg.M ya E RKKHW
My M, and My M.
(17)
where M IR Kx1 is the concatenation of M We propose dividing the transportation mass Py of each semantic region in y by the area of that semantic region to normalize the style based on class distribution of y. We also multiply the mass of each semantic region in y by the area of the same semantic class in x match the expectations of x. We set px to be the uniform distribution and compute
By Myz M
(18)
where denotes Hadamard (element-wise) division. However, we perform correspondence only during test time and cannot rely on labels. Hence, we do not know the area of any of the classes. To that end, we propose estimating My and M, yz. based on features Fr and F,- As such, we define Zrz as self-similarity of similarly Zye and estimate Myy and Mgn with Ryy and Ryy respectively.
R
Σ z Zgv and Rg Σ Z
(19)
21

---

Published in Transactions on Machine Learning Research (04/2024)
Source image
Exemplar image
Cosine Similarity
Figure Visualization of Cosine Similarity across domains. We choose a region centered al the red point from the source image in the first column and display the cosine similarity between the chosen source region with all the other target regions. Our correspondence module is able to relate the object parts that are not labeled in semantic segmentation annotations which is demonstrated by the correspondence of roadlines in the second row and wheel in the third row.
where Z E HWx1 x1 and / indexes to the second dimension of Z. We then compute By
Ryg
py Rgy
(20)
which is linearly scaled to obtain probability distribution py. Lastly, we compute Aga as to warp S deense
as
Sy-z ReshapeeSS dense Ayz)
(21)
Corr Acc
GTA + CS
CS - GTA
Ours
0.59
0.59
Ours w/o Ps
0.57
0.56
Table 12:
Accuracy of semantic correspondence. Our unsupervised Py increases the accuracy of correspon-
dence.
E.2 Effect of the Backbones
We use CLIP Radford et al. (2221] to build semantic correspondences between the two images. CLIP Radford ct al. (2021) is trained with image- caption pairs from the internet, to match the global representation of the image with the language representation of the corresponding caption. Hence, it has never received pixel-level supervision segmentation masks.
22

---

Published in Transactions on Machine Learning Research (04/2024)
We also experiment with pre-trained DenseCL Wang al. (2021) model. DenseCL Wang et al. (002 is trained in self supervised way to predict the intersection of two crops from two augmentations of the same
image. We measure the accuracy of correspondence by warping the segmentation labels of the exemplar via Ayx and then dividing the correctly classified pixels by the total number pixels. We observe that semanntic correspondence with DenseCI. Wang et al. (0021) less accurale, hence we stick to using CLIP.
We use pre-trained weights for the ResNet50 architecture. Specifically, we extract features from the end of the 'layerl' and 'layer3' stages of ResNet50 architecture.
Corr Acc
GTA + CS
CS
GTA
Ours w/ CLIP Radford et al. (2021) 0.59 0.55
0.59
Ours w/ Dense Wang et al. [2021]
0.54
Table 13: Accuracy of semantic correspondence with different backbones. Our method uses CLIP Radford et al. (202 unless otherwise mentioned
E.3 Ablation on Semantic Correspondence
The contributions of OT are analyzed in the supplementary material in Table 12 and the last two rows of Table 14 Table 12 shows that controlling the marginal distributions in OT leads to more accurate semantic correspondences. The last two rows of Table 14 demonstrate that OT contributes the performance of our 12I method. OT encourages one-to-one matches and increases the accuracy of these matches (correspondence accuracy in Table 12h. As a result, OT leads to better transportation of dense style from the exemplar to the target images (Stylistic distance in Table [4] last two rows), more realistic translations with less artifacts (FID score in Table 14| last two rows), and better content preservation (Segmentation Accuracy in Table 14 last two rows). In addition to Table 12 using softmax instead of OT leads to lower correspondence accuracy in both directions (0.57 -> 0.55 and 0.56 -> 0.53, compared to the last row of Table 12), which supports the theoretical advantage of OT experimentally.
F
Qualitative Results
F.1
Ablation Study
We show the qualitative effect of Smir and Sath in Fig. 10 and adversarial losses, the content component encodes less unrealistic translations with the exemplar style.
Without using Smin or S with perceptual information about the image, which leads to
Lyand
Lyth
Linia
OT
Styl. Dis. 4
FID |
Seg. t
X
X
/
0.36
59.72
0.73
X
/
/
/
0.38
52.82
0.75
/
0.32
45.12
0.81
0.35
45.72
0.77
Table 14: Ablation studics for loss terms and OT.
F.2
User Study
We conduct study Amazon Mechanical Turk. We GTA Richter et al. (0066) images that the exemplars because they have greater varicty in style and appcarance. Cityscapes Cordts et al. (2016) images are used as source images. We removed samples which do not contain road and sky. We tried lo include complex scenes with multiple objects in the exemplar so that it contains style for many classes. We form 90 sourco-cxemplar pairs. The users are shown one exemplar image and translations from models, ours and five other baselines Huang et al. (2018): Jeong al. (2021): Lee et al. (2018): Park et al. (2020): Zheng et al. (022), shown in Fig. 11 They asked to choose the image that looks the most similar to
23

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Exemplar
DSI2I
w// S Sas
1 Sume w// S9lh Smiz
Figure 10: Effect of m and Soth The adversarial and perceptual losses Smiz and Suib constrain the dense style representation and, thus, encourage the preservation of content, semantics, and details of the source image. As mentioned in the main paper, the labels are used to swap style across classes in our ablation study instead of the semantic correspondence module. (CS Cordts et al. (2016) to GTA Richter ct al. 20166)
the exemplar image. The question they received was: 'Which image more similar the target image (T)? Similar images would have closer road and sky colors and would reflect the same time of the day.' We did not provide the users with the source image S because most users were choosing the translations that were closest to the content. We randomized the order of the choices (six methods) in our user study, We also filtered the responses with mock question in which users are shown the translation of one content image to the style of six exemplars using MUNIT Huang et al. (2018). Only one of the six exemplars is provided in the question as the exemplar. We ask users the same question and accept the answers of those who pick the translation that matches the exemplar displayed in the question. We received answers from 77 users and each user answered questions (+1 mock question). Out of 3003 answers, 1082 picked our method as the best, followed by MUNIT Huang al. (008)) with 860 votes.
F.3 Comparison with Other Methods
We provide qualitative examples for our model in the end of our supplementary material.
24

---

Published in Transactions on Machine Learning Research (04/2024)
S
T
(4)
(6)
Figure 11: Screenshot from our user study The users are asked to pick the translation (from CS Cordus
et al. (016)) to GTA Richter et al. (016)) that looks the most similar to the exemplar image T. We do not provide the users with the source image S and we randomized the order of the choices in our user study
Here, (1): DSI2I, (2): MUNIT Huang et al. (2018), (3): DRIT Lee et al. 20088) (5): CUT Park et al. (2020)
(5): FSeSim Zheng et al. (2021) ). (6): MGUIT Jeong et al. 2021)
25

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Exemplar
Ours
MUNIT
DRIT
CUT
LSeSim
MGUIT
Figure 12:
Qualitative comparison with other methods. CS + GTA. The road and sky appearance in all the columns are closer to the exemplar road and sky with our method. In the second column, our method is more accurate in the appearance of cars. In the second and third columns, the roadlines are yellow in our translations, which is closer to the exemplar appearance.
26

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Exemplar
Ours
MUNIT
DRIT
CUT
LSeSim
MGUIT
Figure 13: Qualitative comparison with other methods. CS + GTA. Our method brings sky and road appearances closer to those of the exemplar in all cases. In the second column, our method preserves the tree whereas the other methods remove it and display sky.
27

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Exemplar
Ours
MUNIT
DRIT
CUT
LSeSim
MGUIT
Figure 14: Qualitative comparison with other methods. CS - GTA. Our method changes the roadlines based the exemplar. In the first and second columns, the appearance of the roadlines is adjusted based the exemplar whereas the other methods either leave them white or change them to yellow for all the exemplars. In columns three and four, we can see that our method preserves the tree better than the other methods do.
28

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Exemplar
Ours
MUNIT
DRIT
CUT
LSeSim
MGUIT
Figure 15: Qualitative comparison with other methods. CS GTA. Our method preserves the building pixels in the first two colunns. In the last two columns tree and sky are better preserved with our method and reflect closer appearance to the exemplar.
29

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Exemplar
Ours
MUNIT
DRIT
CUT
LSeSim
MGUIT
Figure 16: Qualitative comparison with other methods. CS - GTA. Our method yields a high output diversity, yet preserves the trees in the first, second, and third columns. The road has the closest appearance to the exemplar with our method in the last column.
30

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Exemplar
Ours
MUNIT
DRIT
CUT
LSeSim MGUIT
Figure 17: Qualitative comparison with other methods. GTA CS. In this figure, and in the following ones, show translations in the opposite direction, namely from GTA to CS. Even though stylistic diversity is less in the real image domain, the advantage of our method is still visible. Our method is better at matching the road and sky colors. In the second column, our method does not introduce trees instead of sky, which is common in translations of GTA images.
31

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Exemplar
Ours
MUNIT
DRIT
CUT
LSeSim
MGUIT
Figure 18: Qualitative comparison with other methods. GTA + CS. Aside from road and sky colors, our method is better at preserving the sky regions whereas other methods introduce trees.
32

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Exemplar
Ours
MUNIT
DRIT
CUT
LSeSim
MGUIT
Figure 19: Qualitative comparison with other methods. GTA + CS. In all the columns, sky is flipped to tree with other methods. Our method is better at preserving the semantics, yet has diverse outputs.
33

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Exemplar
Ours
MUNIT
DRIT
CUT
LSeSim
MGUIT
Figure 20: Qualitative comparison with other methods. GTA CS. Our method has much less artifacts in sky in the first three columns. In the last column, the road has closer appearance to the exemplar with our method.
34

---

Published in Transactions on Machine Learning Research (04/2024)
Source
Exemplar
Ours
MUNIT
DRIT
CUT
LSeSim
MGUIT
Qualitative comparison with other methods. KITTI - GTA. The road and sky appearance in all the colunns are closer to the exemplar road and sky with our method. In the second colunn, red colors from the truck pollute the style of the other areas with MUNIT
Figure 21:
35


Reviews:
[{"review_id": "GCGLl9d72H", "reviewer": "Reviewer_Lmht", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Improved quantitative performance relative to prior work across multiple metrics, including the proposed classwise stylistic distance. Qualitatively it is easy to observe the improved precision in the local style transfer as opposed to how prior work appears to globally (and inaccurately) alter all colors of an image. Figure 4 is a great example.\n- The approach is well motivated: it makes sense to model exemplar style using dense features rather than single global features to preserve source content and achieve high style fidelity. The results show that this is effective.\n- The experiments, analysis and ablation study are very thorough. Additional details in the appendix should aid with reproducibility if code is not released.\n\nWeaknesses\n- The writing in the method section can be improved (the training diagram of figure 2 can be improved so that it's easier to follow from the text)\n- While for autonomous driving-like domains where there are many shared semantics the proposed idea makes a lot of sense, but would this still be a viable strategy using other types of data as exemplars (for example artist paintings)? It would be good to expand on the limitations of dense modeling, perhaps relating to generalization across domains that may be very different semantically. This is relevant for broader applicability to creative applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a novel approach for exemplar-based image-to-image translation with state-of-the-art performance called DSI2I. The key idea behind DSI2I is modeling the style of the exemplar image using a dense feature grid, that then gets warped to the the content of the source image based on optimal transport. This allows for content to be preserved while making style transfer is more accurate. Disentangled representations through style and content encoder are enabled by perceptual and adversarial losses. Quantitative evaluation is performed on KITTI, GTA and CityScapes, and a user study further confirms the improved quality of DSI2I compared with baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Revise/clarify the writing in the method section, particularly the training portion of figure 2. There are 7 loss terms in total being defined, and understanding them entirely from the text is difficult.\n- Expand on possible limitations of the approach where the semantics of the exemplar and source image may be significantly different.\n- Fix the vertical spacing of labels in figure 4", "broader_impact_concerns": "There are no ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rgq4ZTOQJ3", "reviewer": "Reviewer_wKEx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: \n- The idea of dense style representation and transferring style locally between semantic regions is novel. Additionally, the method's ability to establish style correspondences without the need for explicit semantic labels or segmentation masks during training is a commendable advancement, showcasing a pioneering approach in fully unsupervised style disentanglement.\n- The proposed method is technically sound with appropriate loss formulations and architectures. Experiments follow standard protocols and compare fairly to state-of-the-art approaches. The metrics cover different desirable properties.\n- Modeling style densely could be impactful for controllable image synthesis and editing applications.\n\n**Weaknesses**\n- The method relies on off-the-shelf components like CLIP features and optimal transport for correspondence, however the interplay with the proposed components is not extensively analyzed.\n- The experiments, though extensive in demonstrating content preservation, appear to focus on a somewhat limited range of styles, which may question the generalizability of the work.\n- The examples presented in Figure 9 implies a possible issue in the disentanglement of style and content within the model. Specifically, in row3, column 2/4, cars in these translated image in variably adopt the color of the car in the exemplar, regardless of their original color. This potentially indicates overlap or confusion between style and content elements. This observation raises concerns about the model’s ability to distinctly separate stylistic attributes (like color or texture) from the inherent content features (like the shape or type of objects).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel approach for UEI2I translation using dense style representations. Existing methods in this area represent style using a single global vector per image or per object instance. In contrast, this work proposes modeling style as a dense feature map with spatial dimensions equal to the content representation. This allows for fine-grained control over style transfer to different semantic regions in a complex scene without requiring external semantic supervision.\nThe method is evaluated on multiple datasets for tasks like synthetic-to-real, real-to-synthetic and real-to-real image translation, showing that the translations are more diverse, better preserve source content, and are closer to exemplar styles compared to state-of-the-art methods​​.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Expanding the range of examined styles, especially those with distinct or challenging characteristics, would provide a more comprehensive assessment of the model's versatility.  This could involve testing the method on painting to photograph, summer to winner, label to photo, and their respective reversals.", "broader_impact_concerns": "I have no broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EELZwWp5G0", "reviewer": "Reviewer_wcQB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n- strong empirical results, in both qualitative and quantitative evaluation\n- inclusion of a user study helps ground the results \n\n*Weaknesses*\n- The method has many components (3 different adversarial and perceptual losses), separate backbone for style warping.\n         - It's hard to follow why it's all necessary and the ablation studies don't have much discussion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a new method for Unpaired Exemplar based Image-to-Image translation (UE2I2I), replying on dense representations instead of a global feature vectors. \n\nThe method use an L1 reconstruction loss to ensure the content (C) and style (S) contain all the information needed to generate an image.\nThe method uses an adversarial loss and perceptual losses to constraint the behavior of the C repesenations, the global S represnetations and mixed global/dense S_{mix} representations. \n\nTo inject a new style into an image given an examplar, the method warmps the style of the target image onto the spatial layout of the source image by solving an optimal transport problem on CLIP dense representations. Given the warp map (which is HWxHW), the method can obtained a warped dense style vector, and feed it into the generators with the the source content vector to get a final generation. \n\nThey demonstrate qualitivately and quantitiatively improved results across multiple datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The work would be strengthened if:\n- A figure was added showing the setup if ablations on the method design were worked into the main text of the paper.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The proposed dense style representation and the use of semantic correspondence for fine-grained style control without seem to be technically sound, and experimental results sufficiently demonstrate improved image-to-image translation and benefits from each component of the proposed method. The authors address most of reviewers' concerns and properly revise the paper to incorporate feedbacks. Therefore, I recommend to accept the paper. However, a few things, especially about the explanation of the proposed training losses, need to be improved in the final version.

- Some training losses need to be more explained. For example, in Eq. (2), what is the role of a random style vector and how is the loss related to rich content representation? More detailed descriptions on motivations, intuitions, and differences between Eq. (2) - (7) are necessary.
- The explanation of S^global and S^mix from the perspective of approximated simulation of semantic correspondence is still unclear. More thorough description with some examples would be necessary.
- Why S^mix is used instead of S^dense?
- There is no explanation of the results in Table 3.
Decision: Accept (poster)